arm64 regression in kernel 5.12 related to the (n)VHE
Marc Zyngier
maz at kernel.org
Thu Aug 12 03:13:16 PDT 2021
On Thu, 12 Aug 2021 09:24:14 +0100,
Rafał Miłecki <zajec5 at gmail.com> wrote:
>
> On 12.08.2021 09:57, Marc Zyngier wrote:
> > On Thu, 12 Aug 2021 08:32:02 +0100,
> > Rafał Miłecki <zajec5 at gmail.com> wrote:
> >>
> >> On 12.08.2021 08:51, Marc Zyngier wrote:
> >>> Interestingly, all your CPUs are booting at EL2. Which is great. Can
> >>> you try and enable KVM on your existing 5.10 kernel? Just selecting
> >>> CONFIG_KVM should be enough. Does it boot correctly with KVM enabled?
> >>>
> >>> My suspicion is that the firmware doesn't set SCR_EL3.HCE, and that
> >>> the HVC instruction UNDEFs at EL1. That would be bad news.
> >>
> >> Interesting! I had to enable CONFIG_VIRTUALIZATION and CONFIG_NET first.
> >> First I verified kernel built with those options still boots. It does.
> >>
> >> Then I enabled CONFIG_KVM and kernel seems to hang around switching from
> >> bootconsole to the console.
> >>
> >> Starting program at 0x0000000000080000
> >> /memory = 0x40000000
> >> WARNING: Node's property /reserved-memory/dt_reserved_buffer is not defined
> >> WARNING: Node's property /reserved-memory/dt_reserved_flow is not defined
> >> WARNING: Node's property /reserved-memory/dt_reserved_dhd2 is not defined
> >> Booting Linux on physical CPU 0x0000000000 [0x420f1000]
> >> Linux version 5.11.22-g0453a426c37b (rmilecki at localhost.localdomain) (aarch64-buildroot-linux-uclibc-gcc.br_real (Buildroot -g91617ed) 9.3.0, GNU ld (GNU Binutils) 2.33.1) #8 SMP Thu Aug 12 09:25:55 CEST 2021
> >> Machine model: Asus GT-AC5300
> >> earlycon: bcm63xx_uart0 at MMIO 0x00000000ff800640 (options '')
> >> printk: bootconsole [bcm63xx_uart0] enabled
> >> efi: UEFI not found.
> >> [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
> >> Zone ranges:
> >> DMA [mem 0x0000000000000000-0x000000003fffffff]
> >> DMA32 empty
> >> Normal empty
> >> Movable zone start for each node
> >> Early memory node ranges
> >> node 0: [mem 0x0000000000000000-0x000000003fffffff]
> >> Initmem setup node 0 [mem 0x0000000000000000-0x000000003fffffff]
> >> percpu: Embedded 18 pages/cpu s43904 r0 d29824 u73728
> >> Detected VIPT I-cache on CPU0
> >> CPU features: detected: ARM erratum 843419
> >> Built 1 zonelists, mobility grouping on. Total pages: 258048
> >> Kernel command line: earlycon=bcm63xx_uart,0xff800640
> >> Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)
> >> Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
> >> mem auto-init: stack:off, heap alloc:off, heap free:off
> >> Memory: 1019556K/1048576K available (4352K kernel code, 678K rwdata, 860K rodata, 2496K init, 232K bss, 29020K reserved, 0K cma-reserved)
> >> SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
> >> rcu: Hierarchical RCU implementation.
> >> rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
> >> NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
> >> GIC: Using split EOI/Deactivate mode
> >> random: get_random_bytes called from start_kernel+0x33c/0x52c with crng_init=0
> >> arch_timer: cp15 timer(s) running at 50.00MHz (phys).
> >> clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb8812736b, max_idle_ns: 440795202655 ns
> >> sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100ns
> >> Console: colour dummy device 80x25
> >> printk: console [tty0] enabled
> >> printk: bootconsole [bcm63xx_uart0] disabled
> >>
> >>
> >> (Unless it's a false conclusion and CONFIG_KVM just breaks console
> >> somehow)
> >
> > No, that's because you don't pass the right console to your
> > kernel. Add something like "console=ttyS0,115200" to the kernel
> > command line, which will show what you are missing, as well as stop
> > the double-logging.
> >
> > Anyway, the fact that it stops booting when you enable KVM confirms my
> > suspicion. The firmware on this system is probably crap enough not to
> > enable HVC. Let's confirm it further: please apply the patch below on
> > top of mainline and tell me that it now boots fine...
>
> Thanks for the patch! It workarounds the issue. See below.
>
>
> > Are you in a position where you can actually fix the firmware? Or is
> > it some closed-source blob?
>
> I'm just an end-user with no access to CFE sources and without any
> business contact as Broadcom :(
I feared that would be the case. Florian's reply seems to indicate
that the "upstream" firmware implementation is correct, so the OEM
must have fumbled it somehow...
> > diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
> > index 43d212618834..fc95b103ef42 100644
> > --- a/arch/arm64/kernel/hyp-stub.S
> > +++ b/arch/arm64/kernel/hyp-stub.S
> > @@ -238,7 +238,7 @@ SYM_FUNC_START(switch_to_vhe)
> > // Turn the world upside down
> > mov x0, #HVC_VHE_RESTART
> > - hvc #0
> > +// hvc #0
> > 1:
> > ret
> > SYM_FUNC_END(switch_to_vhe)
>
> This allows me to boot 5.13.9 and 5.14-rc5 without any reverts!
>
> Enabling CONFIG_KVM still results in the:
> Kernel panic - not syncing: Oops - BUG: Fatal exception in interrupt
That's expected. Can you please check the patch below? It should
result in a booting kernel which actually survives having KVM compiled
in. It should even display a warning telling you that your setup is
completely buggered.
That's obviously not the final version, but probably a good enough
approximation.
Thanks,
M.
diff --git a/arch/arm64/kernel/hyp-stub.S b/arch/arm64/kernel/hyp-stub.S
index 43d212618834..74a130808b38 100644
--- a/arch/arm64/kernel/hyp-stub.S
+++ b/arch/arm64/kernel/hyp-stub.S
@@ -46,6 +46,14 @@ SYM_CODE_END(__hyp_stub_vectors)
.align 11
SYM_CODE_START_LOCAL(elx_sync)
+ mrs x4, CurrentEL
+ cmp x4, #CurrentEL_EL1
+ b.ne 0f
+ mrs x4, esr_el1
+ eor x4, x4, #ESR_ELx_IL
+ cbz x4, el1_undef
+
+0:
cmp x0, #HVC_SET_VECTORS
b.ne 1f
msr vbar_el2, x1
@@ -71,6 +79,18 @@ SYM_CODE_START_LOCAL(elx_sync)
9: mov x0, xzr
eret
+
+el1_undef:
+ # Downgrade the sucker to EL1...
+ adr_l x1, __boot_cpu_mode
+ mov w0, #BOOT_CPU_MODE_EL1
+ str w0, [x1]
+
+ mrs x0, elr_el1
+ add x0, x0, #4
+ msr elr_el1, x0
+ mov_q x0, HVC_STUB_ERR
+ eret
SYM_CODE_END(elx_sync)
// nVHE? No way! Give me the real thing!
@@ -236,6 +256,19 @@ SYM_FUNC_START(switch_to_vhe)
cmp x0, #CurrentEL_EL1
b.ne 1f
+ // Check that HVC actually works...
+ adr_l x1, __hyp_stub_vectors
+ msr vbar_el1, x1
+ isb
+ mov x0, #HVC_RESET_VECTORS
+ hvc #0
+ adr_l x1, vectors
+ msr vbar_el1, x1
+ isb
+ mov_q x1, HVC_STUB_ERR
+ cmp x0, x1
+ b.eq 1f
+
// Turn the world upside down
mov x0, #HVC_VHE_RESTART
hvc #0
--
Without deviation from the norm, progress is not possible.
More information about the linux-arm-kernel
mailing list