RISC-V Zbb QEMU Boot Failures (was Re: [PATCH v1] RISC-V: take text_mutex during alternative patching)

[Palmer Dabbelt]

On Wed, 22 Feb 2023 06:27:43 PST (-0800), ajones at ventanamicro.com wrote:
> On Thu, Feb 16, 2023 at 07:33:55AM -0800, Guenter Roeck wrote:
>> On 2/15/23 14:00, Heiko Stübner wrote:
>> [ ... ]
>> >
>> > So now I've also tested Palmer's for-next at
>> >    commit ec6311919ea6 ("Merge patch series "riscv: Optimize function trace"")
>> >
>> > again with the same variants
>> > - qemu-riscv32 without zbb
>> > - qemu-riscv32 with zbb
>> > - qemu-riscv64 without zbb
>> > - qemu-riscv64 with zbb
>> >
>> > And all of them booted fine into a nfs-root (debian for riscv64 and a
>> > buildroot for riscv32).
>> >
>> > I even forced a bug into the zbb code to make sure the patching worked
>> > correctly (where the kernel failed as expected).
>> >
>> > Qemu-version for me was 7.2.50 (v7.2.0-744-g5a3633929a-dirty)
>> >
>> > I did try the one from Debian-stable (qemu-5.2) but that was too old and
>> > didn't support Zbb yet.
>> >
>> > One thing of note, the "active" 32bit config I had, somehow didn't produce
>> > working images and I needed to start a new build using the rv32_defconfig.
>> >
>> > So right now, I'm not sure what more to test though.
>> >
>>
>> Another example:
>>
>> - build defconfig
>> - run
>>   qemu-system-riscv64 -M virt -m 512M -no-reboot -kernel arch/riscv/boot/Image \
>>     -snapshot -device virtio-blk-device,drive=d0 -drive file=rootfs.ext2,if=none,id=d0,format=raw \
>>     -append "root=/dev/vda console=ttyS0,115200 earlycon=uart8250,mmio,0x10000000,115200" \
>>     -nographic -monitor none
>>
>> With CONFIG_RISCV_ISA_ZBB=y, that results in
>>
>> [    0.818263] /dev/root: Can't open blockdev
>> [    0.818856] VFS: Cannot open root device "/dev/vda" or unknown-block(0,0): error -6
>> [    0.819177] Please append a correct "root=" boot option; here are the available partitions:
>> [    0.819808] fe00           16384 vda
>> [    0.819944]  driver: virtio_blk
>> [    0.820534] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
>> [    0.821101] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.2.0-rc8-next-20230216-00002-g80332825e240 #4
>> [    0.821672] Hardware name: riscv-virtio,qemu (DT)
>> [    0.822050] Call Trace:
>> [    0.822427] [<ffffffff800053e4>] dump_backtrace+0x1c/0x24
>> [    0.822834] [<ffffffff807f90e4>] show_stack+0x2c/0x38
>> [    0.823085] [<ffffffff80803aea>] dump_stack_lvl+0x3c/0x54
>> [    0.823351] [<ffffffff80803b16>] dump_stack+0x14/0x1c
>> [    0.823601] [<ffffffff807f944c>] panic+0x102/0x29e
>> [    0.823834] [<ffffffff80a015e2>] mount_block_root+0x18c/0x23e
>> [    0.824148] [<ffffffff80a0187c>] mount_root+0x1e8/0x218
>> [    0.824398] [<ffffffff80a019ee>] prepare_namespace+0x142/0x184
>> [    0.824655] [<ffffffff80a01182>] kernel_init_freeable+0x236/0x25a
>> [    0.824934] [<ffffffff80804602>] kernel_init+0x1e/0x10a
>> [    0.825201] [<ffffffff80003560>] ret_from_exception+0x0/0x16
>> [    0.826180] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
>>
>> This works fine if CONFIG_RISCV_ISA_ZBB is not enabled.
>>
>> Tested with gcc 11.3, binutils 2.39, qemu v7.2.0 and qemu built from mainline.
>>
>
> Just to +1 this, I get the same result (unable to mount root fs) with
>
> $QEMU -cpu rv64,zbb=on \
>         -nographic \
>         -machine virt \
>         -kernel $KERNEL \
>         -append 'root=/dev/vda console=ttyS0' \
>         -drive file=disk.ext4,format=raw,id=hd0 \
>         -device virtio-blk-pci,drive=hd0
>
> kernel:   latest riscv-linux/for-next (8658db0a4a0f), defconfig
> gcc:      riscv-gnu-toolchain (12.1.0)
> binutils: riscv-gnu-toolchain (2.39)
> qemu:     latest master (79b677d658d3)
>
> Flipping the QEMU cpu zbb property off allows boot to succeed, i.e. it's
> not necessary to compile out the CONFIG_RISCV_ISA_ZBB code from the
> kernel, it's just necessary to avoid using it.

Thanks for finding this.  I'm changing the subject line and adding Paul, 
as it came up during the patchwork call.  I added zbb=on for rv32 and 
rv64 to my tests and they pass, but I'm not mounting a rootfs...

>
> Thanks,
> drew