LPA2 on non-LPA2 hardware broken with 16K pages

[Ard Biesheuvel]

On Wed, 24 Jul 2024 at 13:33, Will Deacon <will at kernel.org> wrote:
>
> On Tue, Jul 23, 2024 at 06:28:16PM +0200, Ard Biesheuvel wrote:
> > On Tue, 23 Jul 2024 at 18:05, Will Deacon <will at kernel.org> wrote:
> > >
> > > On Tue, Jul 23, 2024 at 05:02:15PM +0200, Ard Biesheuvel wrote:
> > > > On Tue, 23 Jul 2024 at 16:52, Will Deacon <will at kernel.org> wrote:
> > > > > On Fri, Jul 19, 2024 at 11:02:29AM -0700, Ard Biesheuvel wrote:
> > ...
> > > > > >
> > > > > > We might add
> > > > > >
> > > > > > if (pgtable_l4_enabled())
> > > > > >     pgdp = &pgd;
> > > > > >
> > > > > > here to preserve the existing 'lockless' behavior when PUDs are not
> > > > > > folded.
> > > > >
> > > > > The code still needs to be 'lockless' for the 5-level case, so I don't
> > > > > think this is necessary.
> > > >
> > > > The 5-level case is never handled here.
> > >
> > > Urgh, yes, sorry. I've done a fantasticly bad job of explaining myself.
> > >
> > > > There is the 3-level case, where the runtime PUD folding needs the
> > > > actual address in order to recalculate the descriptor address using
> > > > the correct shift. In this case, we don't dereference the pointer
> > > > anyway so the 'lockless' thing doesn't matter (afaict)
> > > >
> > > > In the 4-level case, we want to preserve the original behavior, where
> > > > pgd is not reloaded from pgdp. Setting pgdp to &pgd achieves that.
> > >
> > > Right. What I'm trying to get at is the case where we have folding. For
> > > example, with my patch applied, if we have 3 levels then the lockless
> > > GUP walk looks like:
> > >
> > >
> > > pgd_t pgd = READ_ONCE(*pgdp);
> > >
> > > p4dp = p4d_offset_lockless(pgdp, pgd, addr);
> > >         => Returns pgdp
> > > p4d_t p4d = READ_ONCE(*p4dp);
> > >
> > > pudp = pud_offset_lockless(p4dp, p4d, addr);
> > >         => Returns &p4d, which is again the pgdp
> > > pud_t pud = READ_ONCE(*pudp);
> > >
> > >
> > > So here we're reloading the same pointer multiple times and my argument
> > > is that if we need to add logic to avoid this for the
> > > pgtable_l4_enabled() case, then we have bigger problems.
> > >
> >
> > The 3-level case is not relevant here. My suggestion only affects the
> > 4-level case:
>
> That's exactly what I'm uneasy about :/
>

Right.

> > if (pgtable_l4_enabled())
> >    pgdp = &pgd;
> >
> > which prevents us from evaluating *pgdp twice, which seems to me to be
> > the reason these routines exist in the first place. Given that the
> > 3-level runtime-folded case is the one we are trying to fix here, I'd
> > argue that keeping the 4-level case the same as before is important.
>
> I think consistency between 4-level and 3-level is far more important.
> Adding code to avoid reloading the entry for one specific case (the
> pgtable_l4_enabled() case), whilst requiring other cases (e.g. the
> 3-level compile-time folded case) to reload from the pointer is
> inconsistent. Either they both need it or neither of them need it, no?
>

The thing to keep in mind here is that the path via
p4d_to_folded_pud() does not dereference the same pointer either. It
just converts a p4d_t* to a pud_t* by deriving the page tables address
from the p4d_t*, and applying the PUD_SHIFT rather than the P4D_shift
which was applied one level up.

So a) it does not dereference, and b) it refers to a different entry
so the prior dereference loaded the wrong entry.

OTOH, the 4-level path is not only used by 16k+lpa2 (or 16k/48bits),
it is also used by 4k/48bits where pgtable_l4_enabled() is a
compile-time constant TRUE, and this case will no longer be 'lockless'
as before.

However, if I am understanding you correctly, you are saying that

a) p4d_offset_lockless() for <5 levels cannot race in the way that
these macros are intended to address, as the folding implies that the
next-level load is in reality a reload of the same entry, and so we
will be using the latest value
b) reloading the same value is not an issue because this is not a
performance optimization but a concurrency/correctness thing.

I suppose it would be good to clarify this in a comment, as doing the
reload in the implementation of a helper that exists to omit it looks
rather dodgy. But I agree with your analysis, and the additional bits
I suggested are not needed.