[RESEND][PATCH 0/2] Batch register access for live migration optimization
Yize Wang
wangyize7 at huawei.com
Mon Oct 12 04:04:46 PDT 2026
Kindly Ping
在 2026/10/8 20:50, Yize Wang 写道:
>
> 在 2026/10/3 22:53, Marc Zyngier 写道:
>> On Sun, 20 Sep 2026 10:15:25 +0100,
>> Yize Wang <wangyize7 at huawei.com> wrote:
>>> 在 2026/9/18 20:08, Marc Zyngier 写道:
>>>> On Fri, 18 Sep 2026 09:18:13 +0100,
>>>> Yize Wang<wangyize7 at huawei.com> wrote:
>>>>> This series adds batch register access support to KVM/arm64 to reduce
>>>>> syscall overhead during VM live migration.
>>>>>
>>>>> Currently, QEMU issues one ioctl per register when
>>>>> saving/restoring VGIC
>>>>> state. On large VM configurations this means tens of thousands of
>>>>> syscalls,
>>>>> where lock acquisition and context switch overhead dominates
>>>>> migration
>>>>> downtime. Thus, we provide a batch register method to allow userspace
>>>>> read/write multiple distributor and redistributor registers in a
>>>>> single call.
>>>>> In this way, we can significantly reduce syscalls and migration
>>>>> downtime.
>>>>>
>>>>> Test the VM migration time under pressure conditions.
>>>>> The VM specifications for migration are as follows:
>>>>> - VM use 4-K page;
>>>>> - the number of VCPU is 160;
>>>>> - the total memory is 320Gigabit;
>>>>> - use 'Redis SET-benchmark' to pressurize VM;
>>>>>
>>>>> Performance results (3-run average, ms):
>>>>> | Metric | Without patch | With patch |
>>>>> Improvement |
>>>>> |---------------------|---------------|------------|-------------|
>>>>> | Migration downtime | 536 | 321 | 40% |
>>>>> | Source (total) | 344 | 230 | 33% |
>>>>> | - VGIC put | 158 | 40 | 75% |
>>>>> | - VGIC get | 120 | 19 | 84% |
>>>>> | Destination (total) | 192 | 91 | 53% |
>>>>> | - VGIC put | 132 | 27 | 80% |
>>>>>
>>>>> Yize Wang (2):
>>>>> KVM: arm64: Add batch group constant and data structure to
>>>>> UAPI header
>>>>> KVM: arm64: Add VGIC v3 batch register access implementation
>>>> Questions:
>>>>
>>>> - Why only the MMIO registers?
>>>>
>>>> - Why not the sysregs?
>>>>
>>>> - Why only the GIC?
>>>>
>>>> - Why not all of the state?
>>>>
>>>> - Where is the corresponding userspace code?
>>>>
>>>> More importantly, since this is about batching system calls:
>>>>
>>>> - Why can't this be done with io_uring instead?
>>>>
>>>> M.
>>>
>>> Hi, Marc! Thank you for the review.
>>>
>>> These patches focus on optimizing GICv3 register access during live
>>> migration. We found that there are a large number of locks (kvm->lock,
>>> vcpus, config_lock) in the GIC, these lock operations wil cost large
>>> time waste. The batches of sysreg for vcpu optimization will come in
>>> follow as a separate series. And let me address these questions one by
>>> one.
>> All these locks should be non-blocking by the time you save anything
>> related to the GIC, because:
>>
>> - none of the vcpu can be running
>>
>> - this must be a single threaded operation
>>
>> So if you are seeing anything contended, this is either the sign of a
>> bad KVM bug, or an indication that you are violating the above
>> requirements.
>>
>> M.
>
>
> Hi, Marc! Thanks for your review.
>
> You are right that when saving the GIC state during live migration
> process, all vCPUs are not running and it is a single threaded
> operation. Our patches process the packaged register states from the
> userspace under the same lock. These operations only aim to reduce the
> lock opreation downtime and do not involve lock contention.
>
> In addition, these patches focus on reducing the syscalls of ioctl.
> Now, every register requires an ioctl, resulting in (num_cpu *
> registers_per_cpu) system calls. This will cost lots of time during
> the large-scale VM migration. Our patches batch registers by category
> in qemu and then send them to kernel in batches. This operations
> significantly reduce the number of ioctl syscalls and save a great
> amount of downtime.
>
> The comparison is as follow:
> BEFORE:
>
> QEMU (Userspace) Kernel (Kernelspace)
> +------------------------+ +----------------------------+
> | | | |
> | kvm_arm_gicv3_get() | | vgic_v3_attr_regs_access |
> | | | |
> | for ncpu in num_cpu: | | |
> | for each register: | | |
> | | | |
> | GICR_CTLR | ioctl->| mutex_lock(&kvm->lock) |
> | | | kvm_trylock_all_vcpus |
> | | | mutex_lock(&config_lock) |
> | | <-val | read/write register |
> | | | mutex_unlock |
> | | | kvm_unlock_all_vcpus |
> | | | mutex_unlock(&kvm->lock) |
> | | | |
> | GICR_STATUSR | ioctl->| (repeat lock/read |
> | | <-val | -> unlock) |
> | ... | | |
> | | | |
> | GICR_WAKER | ioctl->| (repeat lock/read |
> | | <-val | -> unlock) |
> | ... | | |
> | | | |
> | ICC_SRE_EL1 | ioctl->| (repeat lock/read |
> | | <-val | -> unlock) |
> | ... | | |
> | | | |
> +------------------------+ +----------------------------+
>
>
> AFTER:
>
> QEMU (Userspace) Kernel (Kernelspace)
> +------------------------+ +----------------------------+
> | | | |
> | kvm_arm_gicv3_get() | | vgic_v3_batch_access |
> | | | |
> | Collect Phase: | | |
> | add(GICR_CTLR) | | |
> | add(GICR_STATUSR) | | |
> | add(GICR_WAKER) | | |
> | add(GICR_IGROUPR0) | | |
> | ... | | |
> | add(ICC_SRE_EL1) | | |
> | ... | | |
> | | | |
> | Commit Phase: | BATCH->| copy_from_user |
> | batch_commit() | | mutex_lock(&kvm->lock) |
> | | | kvm_trylock_all_vcpus |
> | | | mutex_lock(&config_lock) |
> | | | |
> | | | Process Each Entry: |
> | | | entry[0]: GICR_CTLR |
> | | | entry[1]: GICR_STATUSR |
> | | | entry[2]: GICR_WAKER |
> | | | ... |
> | | | entry[N]: ICC_SRE |
> | | | FLAG_LOCKED: skip |
> | | | |
> | Unpack Phase: |<-result| mutex_unlock |
> | Write back | | kvm_unlock_all_vcpus |
> | Post-process: | | mutex_unlock(&kvm->lock) |
> | unshuffle,split | | copy_to_user |
> | | | |
> +------------------------+ +----------------------------+
More information about the linux-arm-kernel
mailing list