Arm + KASAN + syzbot

[Dmitry Vyukov]

On Tue, Jan 19, 2021 at 9:37 AM Krzysztof Kozlowski <krzk at kernel.org> wrote:
>
> On Mon, 18 Jan 2021 at 17:31, Dmitry Vyukov <dvyukov at google.com> wrote:
> >
> > Hello Arm maintainers,
> >
> > We are considering setting up an Arm 32-bit instance on syzbot for
> > continuous testing using qemu emulation and I have several questions
> > related to that.
> >
> > 1. Is there interest in this on your end?
>
> Sure, the more, the better.
>
> > What git tree/branch should
> > be used for testing (contains latest development and is regularly
> > updated with fixes)?
>
> Depends on your testing capabilities, whether you can deal with every
> sub-maintainer's tree. 0-day kernel robot tests everything possible
> and this allows each submaintanier to early receive feedback about his
> tree. It can be around 30 Git trees, though... If you want only few, I
> would start with:
>  - https://git.kernel.org/pub/scm/linux/kernel/git/soc/soc.git/
>  - linux-next
>  - and Russell's for-next
> (http://git.armlinux.org.uk/cgit/linux-arm.git/log/?h=for-next)

Hi Krzysztof,

We need to start with just 1 tree. What syzbot is doing is slightly
different from 0-day. 0-day is unit testing, while syzbot is fuzzing.
One caveat is that majority of bugs won't be arm-specific, hundreds of
bugs will be just generic kernel bugs, so the tested tree needs to be
regularly updated to pick up fixes for all these generic bugs.
Otherwise the instance will be just re-hitting these known and already
fixed bugs all the time without having time to discover any new
arm-specific bugs.
I see that for-next branch of
git://git.armlinux.org.uk/~rmk/linux-arm.git is last updated on Dec
21, so it does not even include v5.11-rc11 created on Dec 27, and we
are now on rc4.
We could use linux-next, but sometimes it's broken or pulls in bugs
that cause crashes all the time. So it's not ideal as well.
Maybe we should just use the upstream tree?



> > 2. I see KASAN has just become supported for Arm, which is very
> > useful, but I can't boot a kernel with KASAN enabled. I am using
> > v5.11-rc4 and this config without KASAN boots fine:
> > https://gist.githubusercontent.com/dvyukov/12de2905f9479ba2ebdcc603c2fec79b/raw/c8fd3f5e8328259fe760ce9a57f3e6c6f5a95c8f/gistfile1.txt
>
> Maybe try first with a kernel based on vexpress defconfig. Yours looks
> closer to multi_v7 which enables a lot of stuff also as modules and
> this by itself brought up few issues (mostly with order of probes).

The first config I provided above works fine, so there is no need to
reduce it. The problem is with KASAN.

syzbot also needs a number of debugging configs, a number of configs
that allow to run in qemu, sandboxing/isolation configs, etc. Plus it
enables configs for tested subsystems. All syzbot configs:
https://github.com/google/syzkaller/tree/master/dashboard/config/linux
are produced from the same fragments:
https://github.com/google/syzkaller/tree/master/dashboard/config/linux/bits
That's the plan for Arm as well, we don't want to do 100% custom
things for each new tree/configuration. That's not
scalable/maintainable.


> You could also try other QEMU machine (I don't know many of them, some
> time ago I was using exynos defconfig on smdkc210, but without KASAN).

vexpress-a15 seems to be the most widely used and more maintained. It
works without KASAN. Is there a reason to switch to something else?

> > using the following qemu command line:
> > qemu-system-arm \
> >   -machine vexpress-a15 -cpu max -smp 2 -m 2G \
> >   -device virtio-blk-device,drive=hd0 \
> >   -drive if=none,format=raw,id=hd0,file=image-arm -snapshot \
> >   -kernel arch/arm/boot/zImage \
> >   -dtb arch/arm/boot/dts/vexpress-v2p-ca15-tc1.dtb \
> >   -nographic \
> >   -netdev user,host=10.0.2.10,hostfwd=tcp::10022-:22,id=net0 -device
> > virtio-net-device,netdev=net0 \
> >   -append "root=/dev/vda earlycon earlyprintk=serial console=ttyAMA0
> > oops=panic panic_on_warn=1 panic=86400 vmalloc=512M"
> >
> > However, when I enable KASAN and get this config:
> > https://gist.githubusercontent.com/dvyukov/a7e3edd35cc39a1b69b11530c7d2e7ac/raw/7cbda88085d3ccd11227224a1c9964ccb8484d4e/gistfile1.txt
> >
> > kernel does not boot, qemu only prints the following output and then silence:
> > pulseaudio: set_sink_input_volume() failed
> > pulseaudio: Reason: Invalid argument
> > pulseaudio: set_sink_input_mute() failed
> > pulseaudio: Reason: Invalid argument
> >
> > What am I doing wrong?
>
> No clue but I just tried KASAN on my ARMv7 Exynos5422 board (real
> hardware) and it works (although kernel log appeared with a bigger
> delay):
>
> [    0.000000] Booting Linux on physical CPU 0x100
> [    0.000000] Linux version
> 5.11.0-rc3-next-20210115-00001-g77140600eeec (kozik at kozik-lap)
> (arm-linux-gnueabi-gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0, GNU ld
> (GNU Binutils for Ubuntu) 2.34) #144 SMP PREEMPT Tue Jan 19 09:23:24
> CET 2021
> [    0.000000] CPU: ARMv7 Processor [410fc073] revision 3 (ARMv7), cr=10c5387d
> ...
> [    0.000000] kasan: Truncating shadow for memory block at
> 0x40000000-0xbea00000 to lowmem region at 0x70000000
> [    0.000000] kasan: Mapping kernel virtual memory block:
> c0000000-f0000000 at shadow: b7000000-bd000000
> [    0.000000] kasan: Mapping kernel virtual memory block:
> bf000000-c0000000 at shadow: b6e00000-b7000000
> [    0.000000] kasan: Kernel address sanitizer initialized
>
> Best regards,
> Krzysztof