arm64 regression in kernel 5.12 related to the (n)VHE

Thu Aug 12 01:24:14 PDT 2021

On 12.08.2021 09:57, Marc Zyngier wrote:
> On Thu, 12 Aug 2021 08:32:02 +0100,
> Rafał Miłecki <zajec5 at gmail.com> wrote:
>>
>> On 12.08.2021 08:51, Marc Zyngier wrote:
>>> Interestingly, all your CPUs are booting at EL2. Which is great.  Can
>>> you try and enable KVM on your existing 5.10 kernel? Just selecting
>>> CONFIG_KVM should be enough. Does it boot correctly with KVM enabled?
>>>
>>> My suspicion is that the firmware doesn't set SCR_EL3.HCE, and that
>>> the HVC instruction UNDEFs at EL1. That would be bad news.
>>
>> Interesting! I had to enable CONFIG_VIRTUALIZATION and CONFIG_NET first.
>> First I verified kernel built with those options still boots. It does.
>>
>> Then I enabled CONFIG_KVM and kernel seems to hang around switching from
>> bootconsole to the console.
>>
>> Starting program at 0x0000000000080000
>> /memory = 0x40000000
>> WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined
>> WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined
>> WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined
>> Booting Linux on physical CPU 0x0000000000 [0x420f1000]
>> Linux version 5.11.22-g0453a426c37b (rmilecki at localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #8 SMP Thu Aug 12 09:25:55 CEST 2021
>> Machine model: Asus GT-AC5300
>> earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
>> printk: bootconsole [bcm63xx_uart0] enabled
>> efi: UEFI not found.
>> [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
>> Zone ranges:
>>    DMA      [mem 0x0000000000000000-0x000000003fffffff]
>>    DMA32    empty
>>    Normal   empty
>> Movable zone start for each node
>> Early memory node ranges
>>    node   0: [mem 0x0000000000000000-0x000000003fffffff]
>> Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
>> percpu: Embedded 18 pages/cpu s43904 r0 d29824 u73728
>> Detected VIPT I-cache on CPU0
>> CPU features: detected: ARM erratum 843419
>> Built 1 zonelists, mobility grouping on.  Total pages: 258048
>> Kernel command line: earlycon=bcm63xx_uart,0xff800640
>> Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
>> Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
>> mem auto-init: stack:off, heap alloc:off, heap free:off
>> Memory: 1019556K/1048576K available (4352K kernel code, 678K rwdata, 860K rodata, 2496K init, 232K bss, 29020K reserved, 0K cma-reserved)
>> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
>> rcu: Hierarchical RCU implementation.
>> rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
>> NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>> GIC: Using split EOI/Deactivate mode
>> random: get_random_bytes called from start_kernel+0x33c/0x52c with crng_init=0
>> arch_timer: cp15 timer(s) running at 50.00MHz (phys).
>> clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
>> sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
>> Console: colour dummy device 80x25
>> printk: console [tty0] enabled
>> printk: bootconsole [bcm63xx_uart0] disabled
>>
>>
>> (Unless it's a false conclusion and CONFIG_KVM just breaks console
>> somehow)
> 
> No, that's because you don't pass the right console to your
> kernel. Add something like "console=ttyS0,115200" to the kernel
> command line, which will show what you are missing, as well as stop
> the double-logging.
> 
> Anyway, the fact that it stops booting when you enable KVM confirms my
> suspicion. The firmware on this system is probably crap enough not to
> enable HVC. Let's confirm it further: please apply the patch below on
> top of mainline and tell me that it now boots fine...

Thanks for the patch! It workarounds the issue. See below.

> Are you in a position where you can actually fix the firmware? Or is
> it some closed-source blob?

I'm just an end-user with no access to CFE sources and without any
business contact as Broadcom :(

> diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
> index 43d212618834..fc95b103ef42 100644
> --- a/arch/arm64/kernel/hyp-stub.S
> +++ b/arch/arm64/kernel/hyp-stub.S
> @@ -238,7 +238,7 @@ SYM_FUNC_START(switch_to_vhe)
>   
>   	// Turn the world upside down
>   	mov	x0, #HVC_VHE_RESTART
> -	hvc	#0
> +//	hvc	#0
>   1:
>   	ret
>   SYM_FUNC_END(switch_to_vhe)

This allows me to boot 5.13.9 and 5.14-rc5 without any reverts!

Enabling CONFIG_KVM still results in the:
Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt

Starting program at 0x0000000000080000
/memory = 0x40000000
WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined
WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined
WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined
Booting Linux on physical CPU 0x0000000000 [0x420f1000]
Linux version 5.14.0-rc5-g9c6405c34362-dirty (rmilecki at localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #4 SMP Thu Aug 12 10:11:31 CEST 2021
Machine model: Asus GT-AC5300
earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
printk: bootconsole [bcm63xx_uart0] enabled
efi: UEFI not found.
[Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
Zone ranges:
   DMA      [mem 0x0000000000000000-0x000000003fffffff]
   DMA32    empty
   Normal   empty
Movable zone start for each node
Early memory node ranges
   node   0: [mem 0x0000000000000000-0x000000003fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
percpu: Embedded 17 pages/cpu s38112 r0 d31520 u69632
Detected VIPT I-cache on CPU0
CPU features: detected: ARM erratum 843419
Built 1 zonelists, mobility grouping on.  Total pages: 258048
Kernel command line: earlycon=bcm63xx_uart,0xff800640
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 1020452K/1048576K available (3648K kernel code, 654K rwdata, 708K rodata, 2432K init, 228K bss, 28124K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
Root IRQ handler: gic_handle_irq
GIC: Using split EOI/Deactivate mode
random: get_random_bytes called from start_kernel+0x4a0/0x6dc with crng_init=0
arch_timer: cp15 timer(s) running at 50.00MHz (phys).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
Console: colour dummy device 80x25
printk: console [tty0] enabled
printk: bootconsole [bcm63xx_uart0] disabled
Booting Linux on physical CPU 0x0000000000 [0x420f1000]
Linux version 5.14.0-rc5-g9c6405c34362-dirty (rmilecki at localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #4 SMP Thu Aug 12 10:11:31 CEST 2021
Machine model: Asus GT-AC5300
earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
printk: bootconsole [bcm63xx_uart0] enabled
efi: UEFI not found.
[Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
Zone ranges:
   DMA      [mem 0x0000000000000000-0x000000003fffffff]
   DMA32    empty
   Normal   empty
Movable zone start for each node
Early memory node ranges
   node   0: [mem 0x0000000000000000-0x000000003fffffff]
Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
percpu: Embedded 17 pages/cpu s38112 r0 d31520 u69632
Detected VIPT I-cache on CPU0
CPU features: detected: ARM erratum 843419
Built 1 zonelists, mobility grouping on.  Total pages: 258048
Kernel command line: earlycon=bcm63xx_uart,0xff800640
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 1020452K/1048576K available (3648K kernel code, 654K rwdata, 708K rodata, 2432K init, 228K bss, 28124K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
Root IRQ handler: gic_handle_irq
GIC: Using split EOI/Deactivate mode
random: get_random_bytes called from start_kernel+0x4a0/0x6dc with crng_init=0
arch_timer: cp15 timer(s) running at 50.00MHz (phys).
clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
Console: colour dummy device 80x25
printk: console [tty0] enabled
printk: bootconsole [bcm63xx_uart0] disabled
Calibrating delay loop (skipped), value calculated using timer frequency.. 100.00 BogoMIPS (lpj=200000)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes, linear)
rcu: Hierarchical SRCU implementation.
EFI services will not be available.
smp: Bringing up secondary CPUs ...
Detected VIPT I-cache on CPU1
CPU1: Booted secondary processor 0x0000000001 [0x420f1000]
Detected VIPT I-cache on CPU2
CPU2: Booted secondary processor 0x0000000002 [0x420f1000]
Detected VIPT I-cache on CPU3
CPU3: Booted secondary processor 0x0000000003 [0x420f1000]
smp: Brought up 1 node, 4 CPUs
SMP: Total of 4 processors activated.
CPU features: detected: 32-bit EL0 Support
CPU features: detected: CRC32 instructions
CPU: All CPU(s) started at EL2
alternatives: patching kernel code
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
pinctrl core: initialized pinctrl subsystem
DMI not present or invalid.
DMA: preallocated 128 KiB GFP_KERNEL pool for atomic allocations
DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
DMA: preallocated 128 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
thermal_sys: Registered thermal governor 'step_wise'
ASID allocator initialised with 65536 entries
iommu: Default domain type: Translated
vgaarb: loaded
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
clocksource: Switched to clocksource arch_sys_counter
PCI: CLS 0 bytes, default 64
workingset: timestamp_bits=62 max_order=18 bucket_order=0
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler mq-deadline registered
io scheduler kyber registered
basic-mmio-gpio: probe of ff800500.gpio-controller failed with error -22
ff800640.serial: ttyS0 at MMIO 0xff800640 (irq = 24, base_baud = 1562500) is a bcm63xx_uart
printk: console [ttyS0] enabled
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ehci-platform: EHCI generic platform driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci-pci: OHCI PCI platform driver
ohci-platform: OHCI generic platform driver
i2c /dev entries driver
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
brcmstb-usb-phy 8000c200.usb-phy: Clock not found in Device Tree
brcmstb-usb-phy 8000c200.usb-phy: USB3.0 clock not found in Device Tree
brcmstb-usb-phy 8000c200.usb-phy: Suspend Clock not found in Device Tree
brcmstb-usb-phy 8000c200.usb-phy: IRQ wake not found
brcmstb-usb-phy 8000c200.usb-phy: IRQ wakeup not found
brcmstb-usb-phy 8000c200.usb-phy: Wake interrupt missing, system wake not supported
ehci-platform 8000c300.usb: EHCI Host Controller
ehci-platform 8000c300.usb: new USB bus registered, assigned bus number 1
ehci-platform 8000c300.usb: irq 19, io mem 0x8000c300
ehci-platform 8000c300.usb: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 2 ports detected
ohci-platform 8000c400.usb: Generic Platform OHCI controller
ohci-platform 8000c400.usb: new USB bus registered, assigned bus number 2
ohci-platform 8000c400.usb: irq 20, io mem 0x8000c400
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 2 ports detected
xhci-hcd 8000d000.usb: xHCI Host Controller
xhci-hcd 8000d000.usb: new USB bus registered, assigned bus number 3
xhci-hcd 8000d000.usb: hcc params 0x0250f17c hci version 0x100 quirks 0x0000000000010010
xhci-hcd 8000d000.usb: irq 21, io mem 0x8000d000
hub 3-0:1.0: USB hub found
hub 3-0:1.0: config failed, hub doesn't have any ports! (err -19)
xhci-hcd 8000d000.usb: xHCI Host Controller
xhci-hcd 8000d000.usb: new USB bus registered, assigned bus number 4
xhci-hcd 8000d000.usb: Host supports USB 3.0 SuperSpeed
usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
Freeing unused kernel memory: 2432K
Run /init as init process
tmpfs: Unknown parameter 'mode'
mount: mounting tmpfs: Unknown parameter 'mode'
tmpfs on /dev/shtmpfs: Unknown parameter 'mode'
m failed: Invalid argument
mount: mounting tmpfs on /tmp failed: Invalid argument
mount: mounting tmpfs on /run failed: Invalid argument
Starting syslogd: OK
Starting klogd: OK
Runnirandom: dd: uninitialized urandom read (512 bytes read)
ng sysctl: OK
Saving random seed: OK
Starting network: ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
ip: socket: Function not implemented
FAIL

Welcome to Buildroot
buildroot login: