[RFC PATCH 5/5] KVM: arm64: Enable HAFDBS for guests not on migration

[Tian Zheng]

On 10/27/2026 7:46 PM, Leonardo Bras wrote:
> On Tue, Oct 27, 2026 at 10:02:55AM +0800, Tian Zheng wrote:
>>
>>
>> On 10/23/2026 8:01 PM, Leonardo Bras wrote:
>>> On Fri, Oct 23, 2026 at 03:23:04PM +0800, Tian Zheng wrote:
>>>>
>>>>
>>>> On 10/14/2026 11:56 PM, Leonardo Bras wrote:
>>>>> On Thu, Sep 17, 2026 at 02:40:42PM +0100, Leonardo Bras wrote:
>>>>>> On Wed, Sep 16, 2026 at 04:27:19PM -0700, Oliver Upton wrote:
>>>>>>> On Wed, Sep 16, 2026 at 03:00:46PM +0100, Leonardo Bras wrote:
>>>>>>>> On Tue, Sep 15, 2026 at 05:10:02PM -0700, Oliver Upton wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On Tue, Sep 01, 2026 at 06:15:56PM +0100, Leonardo Bras wrote:
>>>>>>>>>> When dirty-logging is disabled, even non-write faults make a page dirty,
>>>>>>>>>> which avoids a second fault when the page is actually written to.
>>>>>>>>>>
>>>>>>>>>> On dirty-logging enable, this approach causes all (writable) pages on the
>>>>>>>>>> memslot to be marked clean, even if they were not written to, which can
>>>>>>>>>> take a lot of time, while holding the MMU lock, doing atomic writes to
>>>>>>>>>> PTEs.
>>>>>>>>>
>>>>>>>>> Do you have any performance numbers for this? Enabling HAFDBS seems a
>>>>>>>>> bit involved to avoid some stores on the first pass.
>>>>>>>>
>>>>>>>> Not yet, but if the idea does not look too crazy I can find hardware and
>>>>>>>> collect some data :)
>>>>>>>
>>>>>>> TBH this looks like a micro-optimization so I'm not expecting the
>>>>>>> performance gains to justify the behavior change.
>>>>>>>
>>>>>>
>>>>>> Okay then, I will get a machine to figure the performance gains :)
>>>>>
>>>>> Got the performance numbers here.
>>>>> I tested both the case where all PTEs are dirty, and the case that 50% of
>>>>> the PTEs are dirty.
>>>>>
>>>>> For that test, actually patch #5 could make kvm_pgtable_stage2_wrprotect()
>>>>> slower, which is counter-intuitive, in all platforms tested:
>>>>>
>>>>> When we walk the S2 pagetable to wrprotect, using the new walker, the
>>>>> write to the PTE itself is so cheap* that the cost of the branch
>>>>> misprediction (write PTE or not) for the 50/50 case is significantly higher
>>>>> than the cost of the 50% extra writes, which makes the 50/50 up to 60%
>>>>> slower than the 100% dirty case. It is still slower in the '25% dirty' as
>>>>> well, and starts to get even at '10% dirty'.
>>>>>
>>>>> * (non-atomic, everything is in order, walker is small, cpu will probably
>>>>> bundle it all in a single cacheline write)
>>>>>
>>>>> Since the idea of patch #5 was to reduce PTE writes in the first wrprotect
>>>>> walk, it becomes faster only if only a small portion (<10%) of the VM was
>>>>> write-faulted while the remaining was read-faulted, which seems to be a
>>>>> really reduced audience.
>>>>>
>>>>> Since this comes with increased complexity, and also carry costs of
>>>>> having HAFDBS itself enabled outside dirty-tracking, I don't think it's a
>>>>> good idea to further pursue patch #5. It's only value is that it served as
>>>>> a base for HDBSS implementation.
>>>>>
>>>>> Thanks for reading!
>>>>> Leo
>>>>>
>>>> Hi Leo,
>>>>
>>>> Thanks for running this, the numbers are really useful.
>>>>
>>>> It means the hardware dirty mode (HD=1) should only be enabled while
>>>> dirty logging is active with HDBSS support, and stays off otherwise.
>>>
>>> Hi Tian,
>>>
>>> Correct, it's actually counter-productive for it to stay on without HDBSS
>>> before dirty-logging.
>>>
>>>>
>>>> With HD only enabled during dirty logging, we also keep the DBM-based
>>>> encoding for the HD=0 case. On systems that support FEAT_HAFDBS, DBM
>>>> remains the write bit and S2AP[1] the dirty bit even with HD=0. This
>>>> works because kvm_s2_fault_compute_prot() maps any writable page of a
>>>> non-tracked memslot writable-dirty (WD), including on read faults:
>>>>
>>>> ```c
>>>> if (s2vi->map_writable) {
>>>>           *prot |= KVM_PGTABLE_PROT_W;
>>>>
>>>>           if (s2vi->device ||
>>>>               !memslot_is_logging(s2fd->memslot) ||
>>>>               kvm_is_write_fault(s2fd->vcpu))
>>>>                   *prot |= KVM_PGTABLE_PROT_DIRTY;
>>>> }
>>>> ```
>>>>
>>>> So there is no WC entry for hardware to promote, and HD is simply not
>>>> needed outside dirty logging.
>>>>
>>>> In the next HDBSS version, we'll only turn on HD=1 when dirty logging
>>>> is active.
>>>>
>>>
>>> Right!
>>> I will also send a new version of the pre-patch (4 patches) for us to
>>> discuss. The first 2 patches will change meaning a bit, but should still
>>> work as they do today.
>>>
>>> Thanks!
>>> Leo
>>>
>> Hi Leo,
>>
>> OK, I've seen the new DBM RFC patchset. HDBSS v6 will be based on it.
> 
> 
> Hi Tian, thanks for reading it, and for accepting it in your series!
> 
> Before rebasing, please wait for the ongoing discussion on it's
> correctness. Maybe I will have to send new versions before it can be used.
> 
> Thanks!
> Leo

Hi Leo,

I've read your discussion with Oliver on DBM RFC v2. As a suggestion,
maybe we should go with Oliver's approach, i.e. revert to the v1 scheme
and continue fixing from there.

Thanks,
Tian