[PATCH v5 0/6] Fully support standalone Clang/LLVM toolchains

[Bin Meng]

On Sun, Jul 11, 2021 at 9:41 PM Jessica Clarke <jrtc27 at jrtc27.com> wrote:
>
> On 11 Jul 2021, at 14:14, Bin Meng <bmeng.cn at gmail.com> wrote:
> >
> > On Sun, Jul 11, 2021 at 10:29 AM Jessica Clarke <jrtc27 at jrtc27.com> wrote:
> >>
> >> This patch series is comprised of six parts.
> >>
> >> The first patch fixes a warning seen when building with LLVM due to a
> >> bogus combination of assembly directives.
> >>
> >> The second patch fixes errors seen when trying to build OpenSBI as a
> >> position-independent binary using LLD that may or may not be an LLD bug
> >> (the exact meaning of -N/--omagic isn't clear) but can easily be worked
> >> around without any issue either way.
> >>
> >> The third patch bypasses the Clang driver's helpful nature of not
> >> honouring -pie for bare-metal binaries as it's normally not something
> >> you want (arguably that should be an error though, or passed on, since
> >> just giving a -Wunused-command-line-argument warning can get lost, and
> >> may be disabled).
> >>
> >> The fourth patch fixes the FW_PIC support added recently to be disabled
> >> by default when unsupported, since bare-metal GNU ld does not support
> >> PIE and so currently building with a bare-metal GNU ld will fail unless
> >> FW_PIC is manually disabled. Disabling FW_PIC is suboptimal but restores
> >> the pre-FW_PIC behaviour so is no longer a regression.
> >>
> >> The fifth patch adds build system support for using Clang and LLVM
> >> binutils, provided your Clang is able to locate a libgcc.a in its search
> >> path.
> >>
> >> However, pure LLVM toolchains do not use libgcc, they use compiler-rt
> >> (libclang_rt.builtins.<arch>.a). We could change the Makefile to not
> >> hard-code -lgcc and instead use -print-libgcc-file-name, but that still
> >> requires a bare metal compiler-rt built for the right -march/-mabi to be
> >> present, which is often not the case. Moreover, we need very little from
> >> libgcc/compiler-rt; RV64 needs nothing, and RV32 only needs 64-bit
> >> division. Thus, the sixth patch vendors part of FreeBSD's libquad and
> >> stops linking against libgcc entirely, allowing OpenSBI to be built with
> >> just a cross-compiler. This means that building with any distro-provided
> >> LLVM just works, as does compiling with the system Clang compiler and
> >> LLD linker on FreeBSD without any external packages needed (beyond GNU
> >> make).
> >>
> >> Note that using Clang with the current bleeding-edge riscv64 glibc
> >> toolchain from Bootlin's linker is currently broken; around 60
> >> relocations, or about a third of the number expected, go missing, with a
> >> big gap in the relocations emitted, causing various global data
> >> structures to not be correctly initialised. In particular, root's
> >> possible_harts and regions fields lose their relocations during linking
> >> so are left as NULL, causing sanitize_domain to immediately fail with
> >> SBI_EINVAL. The relocations exist in the object files and look
> >> completely sane so this appears to be a GNU ld bug, but I have not
> >> narrowed it down enough to be sure it's not something weird that OpenSBI
> >> or Clang are doing. I do not think this should be a blocker, and have
> >> added a note about this currently-broken combination to the README.
> >>
> >> Changes in v5:
> >> * Disable FW_PIC by default when unsupported
> >> * Pulled out -mno-relax and -fuse-ld flags into variables for reusing
> >>   in the code for the above change, and to deduplicate the existing
> >>   -mno-relax code
> >> * Note in the README that riscv64-linux-gnu-ld + Clang is currently
> >>   broken due to what appears to be a GNU ld bug
> >
> > There is still one build error in the following environment:
> >
> > If /opt/riscv/bin/riscv64-unknown-linux-gnu- is not in my $PATH, the
> > following command fails:
> >
> > $ make CC=clang
> > CROSS_COMPILE=/opt/riscv/bin/riscv64-unknown-linux-gnu-
> > PLATFORM=generic V=1 2>&1
> >
> > mkdir -p `dirname
> > /home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf`;
> > echo " ELF       platform/generic/firmware/payloads/test.elf"; clang
> > -g -Wall -Werror -ffreestanding -nostdlib -fno-stack-protector
> > -fno-strict-aliasing -O2 -fno-omit-frame-pointer
> > -fno-optimize-sibling-calls -mno-save-restore -mstrict-align
> > -mabi=lp64 -march=rv64imafdc -mcmodel=medany
> > --target=riscv64-unknown-linux-gnu -Wno-unused-command-line-argument
> > -I/home/test/git/opensbi/platform/generic/include
> > -I/home/test/git/opensbi/include
> > -DOPENSBI_VERSION_GIT="\"v0.9-111-g401461d\""
> > -I/home/test/git/opensbi/lib/utils/libfdt/
> > -DFW_TEXT_START=0x80000000 -DFW_JUMP_ADDR=0x80200000
> > -DFW_JUMP_FDT_ADDR=0x82200000
> > -DFW_PAYLOAD_PATH=\"/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.bin\"
> > -DFW_PAYLOAD_OFFSET=0x200000 -DFW_PAYLOAD_FDT_ADDR=0x82200000
> > -fno-pie -no-pie
> > /home/test/git/opensbi/build/platform/generic/firmware/payloads/test.o
> > /home/test/git/opensbi/build/platform/generic/lib/libplatsbi.a
> > -fuse-ld=bfd -Wl,--build-id=none -Wl,-N
> > -Wl,-T/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf.ld
> > -o /home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf
> > ELF       platform/generic/firmware/payloads/test.elf
> > /usr/bin/ld.bfd: unrecognised emulation mode: elf64lriscv
> > Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu
> > elf_l1om elf_k1om i386pep i386pe
> > clang-12: error: linker command failed with exit code 1 (use -v to see
> > invocation)
> > make: *** [Makefile:406:
> > /home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf]
> > Error 1
>
> The same problem exists if you do:
>
>   make CC=gcc CROSS_COMPILE=/opt/riscv/bin/riscv64-unknown-linux-gnu-
>
> today (CC=gcc can be anything so long as it’s not “next to” the linker).
> Do you also want:
>
>   make LD=/path/to/ld/not/in/path
>
> to work? Because that doesn’t get used for the ELF rules with GCC
> today, and the same will be true of Clang too. For the former,
> -B$(abspath $(dir CROSS_COMPILE)) if it has a / in it is probably
> sufficient. For the latter, there is no way to support that with GCC
> (without hacking up a temporary directory with a symlink that you can
> use with -B). It can be supported with Clang, though it requires
> version parsing because Clang 12 changed how it does it (it deprecates
> reusing -fuse-ld for this and adds a new --ld-path specifically for
> specifying a path to the linker). Since these are not regressions, nor
> problems only faced by Clang (though it’s more likely you might hit
> them once you have two different toolchain families you’re mixing), I
> would prefer to see the current patch series land so that users and
> distributions can start building with LLVM today as it supports most of
> the common uses without (as far as we know) introducing any regressions
> for purely GNU toolchains, and then support for any additional more
> niche use cases can be added as follow-up patches, otherwise this patch
> series is just going to keep growing and growing with feature creep.

I am just here for beta testing, so relax :) I did not meant to block
the patch series. For this version, I felt it is in nice shape now.

Again, some additional use case as documented in patch 5. For this one
both CC and LLVM are in my $PATH:

$ make CC=riscv64-unknown-elf-gcc LLVM=1 PLATFORM=generic V=1

mkdir -p `dirname
/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf`;
echo " ELF       platform/generic/firmware/payloads/test.elf";
riscv64-unknown-elf-gcc -g -Wall -Werror -ffreestanding -nostdlib
-fno-stack-protector -fno-strict-aliasing -O2 -fno-omit-frame-pointer
-fno-optimize-sibling-calls -mno-save-restore -mstrict-align
-mabi=lp64 -march=rv64imafdc -mcmodel=medany -mno-relax
-I/home/test/git/opensbi/platform/generic/include
-I/home/test/git/opensbi/include
-DOPENSBI_VERSION_GIT="\"v0.9-111-g401461d\""
-I/home/test/git/opensbi/lib/utils/libfdt/
-DFW_TEXT_START=0x80000000 -DFW_JUMP_ADDR=0x80200000
-DFW_JUMP_FDT_ADDR=0x82200000
-DFW_PAYLOAD_PATH=\"/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.bin\"
-DFW_PAYLOAD_OFFSET=0x200000 -DFW_PAYLOAD_FDT_ADDR=0x82200000
-fno-pie -no-pie
/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.o
/home/test/git/opensbi/build/platform/generic/lib/libplatsbi.a
-fuse-ld=lld -Wl,--build-id=none -Wl,-N
-Wl,-T/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf.ld
-o /home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf
 ELF       platform/generic/firmware/payloads/test.elf
collect2: fatal error: cannot find 'ld'
compilation terminated.
make: *** [Makefile:406:
/home/test/git/opensbi/build/platform/generic/firmware/payloads/test.elf]
Error 1

Regards,
Bin