eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Andrii Nakryiko]

On Fri, Aug 7, 2020 at 10:24 AM Jean-Philippe Brucker
<jean-philippe at linaro.org> wrote:
>
> Hi,
>
> [Adding the linux-arm-kernel list on Cc]
>
> On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> > Hi everyone,
> >
> > recently we have begun extensive research into eBPF and related
> > technologies. Seeking an easier development process, we have switched over
> > to using the eBPF CO-RE [0] approach internally which has enabled us to
> > simplify most aspects of eBPF development, especially those related to
> > cross-compilation.
> >
> > However, as part of these efforts we have stumbled upon several problems
> > that we feel would benefit from a community discussion where we may share
> > our solutions and discuss alternatives moving forward.
> >
> > As a reference point, we have started researching and modifying several eBPF
> > CO-RE samples that have been developed or migrated from existing `bcc`
> > tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> > directory [1]. Some of these samples have just recently been converted to
> > respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> > proven to be very interesting.
> >
> > First showstopper for cross-compiling aforementioned example on the ARM
> > 32-bit platform has been with regards to generation of the required
> > `vmlinux.h` kernel header from the BTF information. More specifically, our
> > initial approach to have e.g. a compilation target dependency which would
> > invoke `bpftool` at configure time was not appropriate due to several
> > issues: a) CO-RE requires host kernel to have been compiled in such a way to
> > expose BTF information which may not available, and b) the generated
> > `vmlinux.h` was actually architecture-specific.
> >
> > The second point proved interesting because `tcpconnect` makes use of the
> > `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> > the first function parameter. The `pt_regs` structure is defined by the
> > kernel and is architecture-specific. Since `libbpf` does have
> > architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> > resulted in cross-compilation failure as `libbpf` provided macros that work
> > with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> > resolve this issue, we have resorted to including pre-generated
> > `<arch>_vmlinux.h` files in our CO-RE build system.
> >
> > However, there are certainly drawbacks to this approach: a) (relatively)
> > large file size of the generated headers, b) regular maintenance to
> > re-generate the header files for various architectures and kernel versions,
> > and c) incompatible definitions being generated, to name a few. This last
> > point relates to the the fact that our `aarch64`/`arm64` kernel generates
> > the following definition using `bpftool`, which has resulted in compilation
> > failure:
> >
> > ```
> > typedef __Poly8_t poly8x16_t[16];
> > ```
> >
> > AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> > have opted to comment out this line as there was no additional `poly8x16_t`
> > usage in the header file.
>
> It looks like this "__Poly8_t" type is internal to GCC (provided in
> arm_neon.h) and clang has its own internals. I managed to reproduce this
> with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
> the moment. Maybe libbpf should generate defines to translate these
> intrinsics between clang and gcc? Not very elegant. I'll take another
> look next week.

libbpf is already blacklisting __builtin_va_list for GCC, so we can
just add __Poly8_t to the list. See [0].
Are there any other types like that? If you guys can provide me this,
I'll gladly update libbpf to take those compiler-provided
types/built-ins into account.

  [0] https://github.com/torvalds/linux/blob/master/tools/lib/bpf/btf_dump.c#L585-L598

>
> > Given various issues we have encountered so far (among which is a kernel
> > panic/crash on a specific device), additional input and feedback regarding
> > cross-compilation of the eBPF utilities would be greatly appreciated.
>
> I don't know if there is a room for improvement regarding your a) and b)
> points, as I think the added complexity is inherent to cross-building. But
> kernel crashes definitely need to be fixed, as well as the above problem.
>
> Thanks,
> Jean