[RESEND][PATCH 0/2] Batch register access for live migration optimization

[Yize Wang]

kindly ping

在 2026/9/20 20:12, Yize Wang 写道:
>
> 在 2026/9/18 20:08, Marc Zyngier 写道:
>> On Fri, 18 Sep 2026 09:18:13 +0100,
>> Yize Wang <wangyize7 at huawei.com> wrote:
>>> This series adds batch register access support to KVM/arm64 to reduce
>>> syscall overhead during VM live migration.
>>>
>>> Currently, QEMU issues one ioctl per register when saving/restoring 
>>> VGIC
>>> state. On large VM configurations this means tens of thousands of 
>>> syscalls,
>>> where lock acquisition and context switch overhead dominates migration
>>> downtime. Thus, we provide a batch register method to allow userspace
>>> read/write multiple distributor and redistributor registers in a 
>>> single call.
>>> In this way, we can significantly reduce syscalls and migration 
>>> downtime.
>>>
>>> Test the VM migration time under pressure conditions.
>>> The VM specifications for migration are as follows:
>>> - VM use 4-K page;
>>> - the number of VCPU is 160;
>>> - the total memory is 320Gigabit;
>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>
>>> Performance results (3-run average, ms):
>>>      | Metric              | Without patch | With patch | Improvement |
>>> |---------------------|---------------|------------|-------------|
>>>      | Migration downtime  |        536    |     321    | 40%     |
>>>      | Source (total)      |        344    |     230    | 33%     |
>>>      |   - VGIC put        |        158    |      40    | 75%     |
>>>      |   - VGIC get        |        120    |      19    | 84%     |
>>>      | Destination (total) |        192    |      91    | 53%     |
>>>      |   - VGIC put        |        132    |      27    | 80%     |
>>>
>>> Yize Wang (2):
>>>    KVM: arm64: Add batch group constant and data structure to UAPI 
>>> header
>>>    KVM: arm64: Add VGIC v3 batch register access implementation
>> Questions:
>>
>> - Why only the MMIO registers?
>>
>> - Why not the sysregs?
>>
>> - Why only the GIC?
>>
>> - Why not all of the state?
>>
>> - Where is the corresponding userspace code?
>>
>> More importantly, since this is about batching system calls:
>>
>> - Why can't this be done with io_uring instead?
>>
>>     M.
>
>
> Hi, Marc! Thank you for the review.
>
>
> These patches focus on optimizing GICv3 register access during live 
> migration. We found that there are a large number of locks (kvm->lock, 
> vcpus, config_lock) in the GIC, these lock operations wil cost large 
> time waste. The batches of sysreg for vcpu optimization will come in 
> follow as a separate series. And let me address these questions one by 
> one.
>
>
> 1. Why only the MMIO registers?
>
> In vgic_v3_batch_access(), we use 'entries' structure to implement 
> batch read/write of register status. The structure is 'struct 
> kvm_dev_arm_vgic_batch_entry', where the group information can be 
> freely specified by userspace. Thus, vgic_v3_batch_access() supports 
> all VGIC device attr groups, not only MMIO registers.
>
>
> 2. Why not the sysregs?
>
> The newly added vgic_v3_batch_access() just forwards the groups 
> received fromQEMU in batches to 'vgic_v3_attr_regs_access()'. And this 
> function already has 'KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS' to handle 
> sysregs. Our patch does not modify the existing sysreg handling logic.
>
>
> 3. Why only the GIC?
>
> During live migration, the GIC is the device with the largest number 
> of registers, and we found that there are a large number of locks in 
> it. If each register is locked and unlocked individually, it would 
> cause large time consumption. Thus, we want to optimize the GIC 
> register time consumption during live migrate. The experimental 
> results also show that batch processing significantly reduces VGIC 
> handling time during migration.
>
> Similarly, vCPU register save/restore also costs significantly time. 
> The patch of vCPU sysreg batch access will be submitted separately in 
> the future.
>
>
> 4. Why not all of the state?
>
> As different devices have different lock hierarchies and access 
> paths(e.g., GIC goes through the device fd, CPU regs go through the 
> vcpu fd), it's difficult for us to realize in a single batch handler. 
> Additionally, introducing too many changes at once would make review 
> harder. These patches focus on GIC-related optimization, and a 
> separate series for vCPU batch processing will follow.
>
>
> 5. Where is the corresponding userspace code?
>
> The QEMU-side implementation has been posted to qemu-devel. The link 
> is below:
>
> https://lore.kernel.org/qemu-devel/20260918092120.370805-1-wangyize7@huawei.com/T/#t 
>
>
>
> 6. Why can't this be done with io_uring instead?
>
> The core idea of io_uring is to reduce the number of context switches 
> between user space and kernel space by utilizing two ring buffers. 
> However, each SQE is still processed independently in kernel space. 
> For the VGIC registers, each SQE still require lock -> read/write -> 
> unlock, so the per-register lock overhead remains unchanged. We hope 
> to read/write a set of register states with a single lock operation to 
> reduce the time cost. Therefore, io_uring does not meet our needs.