[PATCH 43/43] Documentation: KVM: Add the VGICv5 IRS save/restore sequences

[Peter Maydell]

On Mon, 27 Apr 2026 at 17:22, Sascha Bischoff <Sascha.Bischoff at arm.com> wrote:
>
> When saving/restoring the state of the GICv5 IRS, it is important that
> it happens in the correct order. Failure to do so will almost
> certainly result in failing to restore a guest that is capable of
> handling interrupts correctly.
>
> On a save, the ISTs must be saved prior to saving the guest's memory
> as the guest's LPI IST is written to guest memory. Conversely, on
> restore the guest's memory must be restored prior to restoring the
> ISTs.
>
> It is important to restore the IRS MMIO registers by first restoring
> the IRS_IDx registers as they define the capabilities of the IRS, and
> are used as part of creating and managing ISTs and SPIs.
>
> In order to restore the ISTs themselves, the IRS_IST_CFGR must be
> restored prior to the IRS_IST_BASER. This is because KVM extracts
> fields from the CFGR to determine the size and structure of the IRS
> created by the guest. The IST itself is created as part of the write
> to the IRS_IST_BASER. At this stage the remaining MMIO registers can
> be restored.
>
> Once the LPI IST has been created (by the aforementioned write to the
> IRS_IST_BASER), the IST state can be restored using
> KVM_DEV_ARM_VGIC_GRP_IST. The SPI IST gets extracted from a userspace
> provided buffer, and is transferred to the host-allocated SPI IST. The
> LPI IST is extracted from guest memory, and is written to the
> host-allocated LPI IST.
>
> As a general rule, the IRS_*_STATUSR registers can be ignored on
> restore. They are not userspace writable.
>
> Signed-off-by: Sascha Bischoff <sascha.bischoff at arm.com>
> ---
>  .../virt/kvm/devices/arm-vgic-v5.rst          | 63 +++++++++++++++++++
>  1 file changed, 63 insertions(+)
>
> diff --git a/Documentation/virt/kvm/devices/arm-vgic-v5.rst b/Documentation/virt/kvm/devices/arm-vgic-v5.rst
> index 38eef7cc63e3e..1c55f5040757d 100644
> --- a/Documentation/virt/kvm/devices/arm-vgic-v5.rst
> +++ b/Documentation/virt/kvm/devices/arm-vgic-v5.rst
> @@ -201,3 +201,66 @@ Groups:
>        -ENOMEM      Restoring IST state failed while tracking pending interrupts
>        -ETIMEDOUT   An IRS save/VM operation timed out
>        ===========  ============================================================
> +
> +IRS Save Sequence:
> +------------------
> +
> +The following ordering should be followed when saving the virtual GICv5 and
> +IRS:
> +
> +a) Save the ISTs by issuing KVM_GET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IST.
> +   This MUST happen before the guest's memory is serialised as the LPI IST is
> +   stored directly to guest memory.
> +
> +b) Save the IRS MMIO register state in the following order by issuing
> +   KVM_GET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
> +
> +     1. Save IRS_IDR0-2 and IRS_IDR5-7 registers.
> +     2. Save IRS_IST_CFGR.
> +     3. Save IRS_IST_BASER.
> +     4. Save the remaining global IRS MMIO registers.
> +     5. For each PE:
> +        - write IRS_PE_SELR
> +        - save IRS_PE_CR0
> +     6. For each SPI:
> +        - write IRS_SPI_SELR
> +        - save IRS_SPI_CFGR
> +
> +IRS Restore Sequence:
> +---------------------
> +
> +The following ordering must be followed when restoring the virtual GICv5 and
> +IRS:
> +
> +a) restore all guest memory and create vcpus
> +b) provide the IRS base address by issuing KVM_SET_DEVICE_ATTR on
> +   KVM_DEV_ARM_VGIC_GRP_ADDR
> +c) initialise the GIC - this sets up the default state and creates the SPI
> +   IST - by issuing KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_CTRL with
> +   KVM_DEV_ARM_VGIC_CTRL_INIT

This isn't going to work for QEMU, if I understand it correctly.
QEMU always creates the whole VM first, including creating the
VCPUs and GIC, telling KVM what its base address is, initializing it,
etc, before it starts an inbound migration. So the memory read
is going to come in after step (c), not right at the start.

> +d) restore the IRS MMIO register state in the following order by issuing
> +   KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
> +
> +     1. Restore IRS_IDR0-2 and IRS_IDR5-7 registers.
> +     2. Restore IRS_IST_CFGR.
> +     3. Restore IRS_IST_BASER - this triggers KVM to create the LPI IST.
> +
> +e) restore the ISTs by issuing KVM_SET_DEVICE_ATTR on
> +   KVM_DEV_ARM_VGIC_GRP_IST.
> +f) restore the remaining IRS MMIO register state in the following order by
> +   issuing KVM_SET_DEVICE_ATTR on KVM_DEV_ARM_VGIC_GRP_IRS_REGS:
> +
> +     1. Restore the remaining global IRS MMIO registers.
> +     2. For each PE:
> +        - write IRS_PE_SELR
> +        - restore IRS_PE_CR0
> +     3. For each SPI:
> +        - write IRS_SPI_SELR
> +        - restore IRS_SPI_CFGR

More generally, if your API involves this much in the way
of complicated ordering dependencies, it's going to be
very bug prone.  From userspace's perspective, this is
not a very helpful way to design the interface :-)

thanks
-- PMM