[PATCH v4 7/7] ARM: implement support for vmap'ed stacks

[Ard Biesheuvel]

On Tue, 28 Dec 2021 at 17:13, Geert Uytterhoeven <geert at linux-m68k.org> wrote:
>
> On Tue, Dec 28, 2021 at 3:39 PM Geert Uytterhoeven <geert at linux-m68k.org> wrote:
> > On Thu, Dec 23, 2021 at 3:30 PM Ard Biesheuvel <ardb at kernel.org> wrote:
> > > On Tue, 21 Dec 2021 at 22:56, Marek Szyprowski <m.szyprowski at samsung.com> wrote:
> > > > On 21.12.2021 17:20, Ard Biesheuvel wrote:
> > > > > On Tue, 21 Dec 2021 at 14:51, Marek Szyprowski <m.szyprowski at samsung.com> wrote:
> > > > >> On 21.12.2021 14:34, Ard Biesheuvel wrote:
> > > > >>> On Tue, 21 Dec 2021 at 12:15, Marek Szyprowski <m.szyprowski at samsung.com> wrote:
> > > > >>>> On 21.12.2021 11:44, Ard Biesheuvel wrote:
> > > > >>>>> On Tue, 21 Dec 2021 at 11:39, Marek Szyprowski <m.szyprowski at samsung.com> wrote:
> > > > >>>>>> On 22.11.2021 10:28, Ard Biesheuvel wrote:
> > > > >>>>>>> Wire up the generic support for managing task stack allocations via vmalloc,
> > > > >>>>>>> and implement the entry code that detects whether we faulted because of a
> > > > >>>>>>> stack overrun (or future stack overrun caused by pushing the pt_regs array)
> > > > >>>>>>>
> > > > >>>>>>> While this adds a fair amount of tricky entry asm code, it should be
> > > > >>>>>>> noted that it only adds a TST + branch to the svc_entry path. The code
> > > > >>>>>>> implementing the non-trivial handling of the overflow stack is emitted
> > > > >>>>>>> out-of-line into the .text section.
> > > > >>>>>>>
> > > > >>>>>>> Since on ARM, we rely on do_translation_fault() to keep PMD level page
> > > > >>>>>>> table entries that cover the vmalloc region up to date, we need to
> > > > >>>>>>> ensure that we don't hit such a stale PMD entry when accessing the
> > > > >>>>>>> stack. So we do a dummy read from the new stack while still running from
> > > > >>>>>>> the old one on the context switch path, and bump the vmalloc_seq counter
> > > > >>>>>>> when PMD level entries in the vmalloc range are modified, so that the MM
> > > > >>>>>>> switch fetches the latest version of the entries.
> > > > >>>>>>>
> > > > >>>>>>> Note that we need to increase the per-mode stack by 1 word, to gain some
> > > > >>>>>>> space to stash a GPR until we know it is safe to touch the stack.
> > > > >>>>>>> However, due to the cacheline alignment of the struct, this does not
> > > > >>>>>>> actually increase the memory footprint of the struct stack array at all.
> > > > >>>>>>>
> > > > >>>>>>> Signed-off-by: Ard Biesheuvel <ardb at kernel.org>
> > > > >>>>>>> Tested-by: Keith Packard <keithpac at amazon.com>
> > > > >>>>>> This patch landed recently in linux-next 20211220 as commit a1c510d0adc6
> > > > >>>>>> ("ARM: implement support for vmap'ed stacks"). Sadly it breaks
> > > > >>>>>> suspend/resume operation on all ARM 32bit Exynos SoCs. Probably the
> > > > >>>>>> suspend/resume related code must be updated somehow (it partially works
> > > > >>>>>> on physical addresses and disabled MMU), but I didn't analyze it yet. If
> > > > >>>>>> you have any hints, let me know.
> > > > >>>>>>
> > > > >>>>> Are there any such systems in KernelCI? We caught a suspend/resume
> > > > >>>>> related issue in development, which is why the hunk below was added.
> > > > >>>> I think that some Exynos-based Odroids (U3 and XU3) were some time ago
> > > > >>>> available in KernelCI, but I don't know if they are still there.
> > > > >>>>
> > > > >>>>
> > > > >>>>> In general, any virt-to-phys translation involving and address on the
> > > > >>>>> stack will become problematic.
> > > > >>>>>
> > > > >>>>> Could you please confirm whether the issue persists with the patch
> > > > >>>>> applied but with CONFIG_VMAP_STACK turned off? Just so we know we are
> > > > >>>>> looking in the right place?
> > > > >>>> I've just checked. After disabling CONFIG_VMAP_STACK suspend/resume
> > > > >>>> works fine both on commit a1c510d0adc6 and linux-next 20211220.
> > > > >>>>
> > > > >>> Thanks. Any other context you can provide beyond 'does not work' ?
> > > > >> Well, the board properly suspends, but it doesn't wake then (tested
> > > > >> remotely with rtcwake command). So far I cannot provide anything more.
> > > > >>
> > > > > Thanks. Does the below help? Or otherwise, could you try doubling the
> > > > > size of the overflow stack at arch/arm/include/asm/thread_info.h:34?
> > > >
> > > > I've tried both (but not at the same time) on the current linux-next and
> > > > none helped. This must be something else... :/
> > > >
> > >
> > > Thanks.
> > >
> > > As i don't have access to this hardware, I am going to have to rely on
> > > someone who does to debug this further. The only alternative is
> > > marking CONFIG_VMAP_STACK broken on MACH_EXYNOS but that would be
> > > unfortunate.
> >
> > Wish I had seen this thread before...
> >
> > I've just bisected a resume after s2ram failure on R-Car Gen2 to the same
> > commit a1c510d0adc604bb ("ARM: implement support for vmap'ed stacks")
> > in arm/for-next.
> >
> > Expected output:
> >
> >     PM: suspend entry (deep)
> >     Filesystems sync: 0.000 seconds
> >     Freezing user space processes ... (elapsed 0.010 seconds) done.
> >     OOM killer disabled.
> >     Freezing remaining freezable tasks ... (elapsed 0.009 seconds) done.
> >     Disabling non-boot CPUs ...
> >
> > [system suspended, this is also where it hangs on failure]
> >
> >     Enabling non-boot CPUs ...
> >     CPU1 is up
> >     sh-eth ee700000.ethernet eth0: Link is Down
> >     Micrel KSZ8041RNLI ee700000.ethernet-ffffffff:01: attached PHY
> > driver (mii_bus:phy_addr=ee700000.ethernet-ffffffff:01, irq=193)
> >     OOM killer enabled.
> >     Restarting tasks ... done.
> >     PM: suspend exit
> >
> > Both wake-on-LAN and wake-up by gpio-keys fail.
> > Nothing interesting in the kernel log, cfr. above.
> >
> > Disabling CONFIG_VMAP_STACK fixes the issue for me.
>
> Enabling CONFIG_ARM_LPAE also fixes the issue, but is not an option
> for shmobile_defconfig, as that would break systems with a Cortex-A9.
>

Thanks Geert.

As you have confirmed on #armlinux, the issue also goes away when
booting with 'nosmp'. So this looks like an issue with the virtual
mapping of the stack in the secondary boot path on !LPAE. That really
narrows it down, so hopefully I will be able to fix this shortly.

Marek: could you please confirm whether or not enabling LPAE (on cores
that support it, of course) and/or booting with 'nosmp' make the issue
go away?