[RFC PATCH 0/2] Optimize S2 page splitting

[Marc Zyngier]

On Thu, 28 May 2026 18:00:48 +0100,
Leonardo Bras <leo.bras at arm.com> wrote:
> 
> On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote:
> > On Fri, 15 May 2026 20:59:01 +0100,
> > Leonardo Bras <leo.bras at arm.com> wrote:
> > > 
> > > While playing with dirty-bit tracking, I decided to take a look on how page
> > > splitting works. Found out all entries are walked, even though we can infer,
> > > for instance that:
> > > - If a level-3 entry is walked, it means the parent level-2 entry is split
> > > - If a split just succeeded in an table entry, it means all children nodes
> > >   are already split
> > > 
> > > So I tried to optimize it in a way that it does not break other users.
> > > 
> > > My main idea is to introduce positive return values that hint to the
> > > pagetable walking mechanism that either siblings or children can be 
> > > skipped. That should be contained to the visitor function, that returns
> > > zero if no error was detected.
> > > 
> > > Numbers on above optimization are promising:
> > > A 1GB VM, running on the model, splitting all at the beginning 
> > > (no manual protect):
> > > - Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
> > > - THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
> > > - 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs
> > >
> > 
> > I haven't looked at the changes in details, but the methodology is
> > quite flawed. For a start, measuring anything on a software model
> > (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs
> > are completely different from a HW implementation, and even the notion
> > of time is pretty inconsistent.
> > 
> > Please run this on actual HW. I'm sure your employer can give you
> > access to one of these mythical arm64 toys. Measure things from
> > userspace, not from the kernel, so that you have all the overheads.
> > Don't add console output, because that will make things far worse.
> > 
> > I'm sure you can hack one of the selftests for this purpose.
> 
> Hello Marc,
> 
> I have ran some tests in real hardware (AmpereOne) and calling from 
> userspace as you suggested.
> 
> The tests create a 64GB VM, using either regular pages (no split needed), 
> THP backed memory or Hugetlb. 
> 
> Those tests measured the times for the whole syscall that happens to do
> eager page splitting, in this case,
> 1 - Manual protect and init set: On every KVM_CLEAR_DIRTY_LOG, if the page
>     was requested to be cleaned, it is also split.
> 2 - Otherwise, on dirty log enable (aka KVM_SET_USER_MEMORY_REGION2 with
>     KVM_MEM_LOG_DIRTY_PAGES set in flags), for the whole memslot is split
>     at once.
> 
> Tests reflect new suggestions I got in this patchset: instead of using 
> return values, I am using flags to skip whole levels, or skipping children 
> nodes. I tested those 2 kernels, on top of vanilla:
> a - Skip levels
> b - Skip levels, and skip children node
> 
> Results averaged across 4k, 16k and 64k pages, a percentage of time 
> saved, compared to vanilla kernel. More is better.
> 
> For (1), we have
> regular pages:	26.7% (a) and 25.3% (b)
> THP:		13.2% (a) and 11.9% (b)
> Hugetlb:	14.3% (a) and 13.2% (b)
> 
> For (2), we have:
> regular pages:	33.1% (a) and 35.0% (b)
> THP:		11.9% (a) and 10.7% (b)
> Hugetlb:	13.4% (a) and 13.2% (b)
> 
> On above results, I could notice about 1% overhead showing that skipping 
> child nodes (testing flag) ends up being counter-productive compared to 
> just skipping levels. The only case this does not happen is (2a), but it's 
> not clear on why that happens.
> 
> Based on that, I plan to remove the skip_child patch, and send a v2 with a 
> flag-based mechanism, which results are shown above.
> 
> Please let me know of any thoughts, suggestions or ideas you might have 
> about it.

Right, that seems compelling. I'd suggest you send a new version with:

- a pointer to the test and its the exact invocation parameters so
  that people other than you can reproduce it

- these numbers with a bit of analysis explaining where the gains are

With that, we can look at it again and compare results on different
workloads and HW.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.