eBPF CO-RE cross-compilation for 32-bit ARM platforms

[Jean-Philippe Brucker]

Hi,

[Adding the linux-arm-kernel list on Cc]

On Fri, Aug 07, 2020 at 04:20:58PM +0200, Jakov Petrina wrote:
> Hi everyone,
> 
> recently we have begun extensive research into eBPF and related
> technologies. Seeking an easier development process, we have switched over
> to using the eBPF CO-RE [0] approach internally which has enabled us to
> simplify most aspects of eBPF development, especially those related to
> cross-compilation.
> 
> However, as part of these efforts we have stumbled upon several problems
> that we feel would benefit from a community discussion where we may share
> our solutions and discuss alternatives moving forward.
> 
> As a reference point, we have started researching and modifying several eBPF
> CO-RE samples that have been developed or migrated from existing `bcc`
> tooling. Most notable examples are those present in `bcc`'s `libbpf-tools`
> directory [1]. Some of these samples have just recently been converted to
> respective eBPF CO-RE variants, of which the `tcpconnect` tracing sample has
> proven to be very interesting.
> 
> First showstopper for cross-compiling aforementioned example on the ARM
> 32-bit platform has been with regards to generation of the required
> `vmlinux.h` kernel header from the BTF information. More specifically, our
> initial approach to have e.g. a compilation target dependency which would
> invoke `bpftool` at configure time was not appropriate due to several
> issues: a) CO-RE requires host kernel to have been compiled in such a way to
> expose BTF information which may not available, and b) the generated
> `vmlinux.h` was actually architecture-specific.
> 
> The second point proved interesting because `tcpconnect` makes use of the
> `BPF_KPROBE` and `BPF_KRETPROBE` macros, which pass `struct pt_regs *ctx` as
> the first function parameter. The `pt_regs` structure is defined by the
> kernel and is architecture-specific. Since `libbpf` does have
> architecture-specific conditionals, pairing it with an "invalid" `vmlinux.h`
> resulted in cross-compilation failure as `libbpf` provided macros that work
> with ARM `pt_regs`, and `vmlinux.h` had an x86 `pt_regs` definition. To
> resolve this issue, we have resorted to including pre-generated
> `<arch>_vmlinux.h` files in our CO-RE build system.
> 
> However, there are certainly drawbacks to this approach: a) (relatively)
> large file size of the generated headers, b) regular maintenance to
> re-generate the header files for various architectures and kernel versions,
> and c) incompatible definitions being generated, to name a few. This last
> point relates to the the fact that our `aarch64`/`arm64` kernel generates
> the following definition using `bpftool`, which has resulted in compilation
> failure:
> 
> ```
> typedef __Poly8_t poly8x16_t[16];
> ```
> 
> AFAICT these are ARM NEON intrinsic definitions which are GCC-specific. We
> have opted to comment out this line as there was no additional `poly8x16_t`
> usage in the header file.

It looks like this "__Poly8_t" type is internal to GCC (provided in
arm_neon.h) and clang has its own internals. I managed to reproduce this
with an arm64 allyesconfig kernel (+BTF), but don't know how to fix it at
the moment. Maybe libbpf should generate defines to translate these
intrinsics between clang and gcc? Not very elegant. I'll take another
look next week.

> Given various issues we have encountered so far (among which is a kernel
> panic/crash on a specific device), additional input and feedback regarding
> cross-compilation of the eBPF utilities would be greatly appreciated.

I don't know if there is a room for improvement regarding your a) and b)
points, as I think the added complexity is inherent to cross-building. But
kernel crashes definitely need to be fixed, as well as the above problem.

Thanks,
Jean