eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Alexei Starovoitov]

On Fri, Aug 7, 2020 at 11:41 AM Andrii Nakryiko
<andrii.nakryiko at gmail.com> wrote:
>
> On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker
> <jean-philippe at linaro.org> wrote:
> >
> > Hi,
> >
> > [Adding the linux-arm-kernel list on Cc]
> >
> > On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> > > Hi everyone,
> > >
> > > recently we have begun extensive research into eBPF and related
> > > technologies. Seeking an easier development process, we have switched over
> > > to using the eBPF CO-RE [0] approach internally which has enabled us to
> > > simplify most aspects of eBPF development, especially those related to
> > > cross-compilation.
> > >
> > > However, as part of these efforts we have stumbled upon several problems
> > > that we feel would benefit from a community discussion where we may share
> > > our solutions and discuss alternatives moving forward.
> > >
> > > As a reference point, we have started researching and modifying several eBPF
> > > CO-RE samples that have been developed or migrated from existing `bcc`
> > > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> > > directory [1]. Some of these samples have just recently been converted to
> > > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> > > proven to be very interesting.
> > >
> > > First showstopper for cross-compiling aforementioned example on the ARM
> > > 32-bit platform has been with regards to generation of the required
> > > `vmlinux.h` kernel header from the BTF information. More specifically, our
> > > initial approach to have e.g. a compilation target dependency which would
> > > invoke `bpftool` at configure time was not appropriate due to several
> > > issues: a) CO-RE requires host kernel to have been compiled in such a way to
> > > expose BTF information which may not available, and b) the generated
> > > `vmlinux.h` was actually architecture-specific.
> > >
> > > The second point proved interesting because `tcpconnect` makes use of the
> > > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> > > the first function parameter. The `pt_regs` structure is defined by the
> > > kernel and is architecture-specific. Since `libbpf` does have
> > > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> > > resulted in cross-compilation failure as `libbpf` provided macros that work
> > > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> > > resolve this issue, we have resorted to including pre-generated
> > > `<arch>_vmlinux.h` files in our CO-RE build system.
> > >
> > > However, there are certainly drawbacks to this approach: a) (relatively)
> > > large file size of the generated headers, b) regular maintenance to
> > > re-generate the header files for various architectures and kernel versions,
> > > and c) incompatible definitions being generated, to name a few. This last
> > > point relates to the the fact that our `aarch64`/`arm64` kernel generates
> > > the following definition using `bpftool`, which has resulted in compilation
> > > failure:
> > >
> > > ```
> > > typedef __Poly8_t poly8x16_t[16];
> > > ```
> > >
> > > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > > have opted to comment out this line as there was no additional `poly8x16_t`
> > > usage in the header file.
> >
> > It looks like this "__Poly8_t" type is internal to GCC (provided in
> > arm_neon.h) and clang has its own internals. I managed to reproduce this
> > with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> > the moment. Maybe libbpf should generate defines to translate these
> > intrinsics between clang and gcc? Not very elegant. I'll take another
> > look next week.
>
> libbpf is already blacklisting __builtin_va_list for GCC, so we can
> just add __Poly8_t to the list. See [0].
> Are there any other types like that? If you guys can provide me this,
> I'll gladly update libbpf to take those compiler-provided
> types/built-ins into account.

Shouldn't __Int8x16_t and friends cause the same trouble?
There is a bunch more in gcc/config/arm/arm-simd-builtin-types.def.
May be there is a way to detect compiler builtin types by pattern matching
their dwarf/btf shape and skip them automatically?
The simplest, of course, is to only add a few that caused this known
trouble to blocklist.