[PATCH v3 00/24] vfio/pci: Base Live Update support for VFIO device files
David Matlack
dmatlack at google.com
Mon Mar 23 16:57:52 PDT 2026
This series can be found on GitHub:
https://github.com/dmatlack/linux/tree/liveupdate/vfio/cdev/v3
This series adds the base support to preserve a VFIO device file across
a Live Update. "Base support" means that this allows userspace to
safely preserve a VFIO device file with LIVEUPDATE_SESSION_PRESERVE_FD
and retrieve it with LIVEUPDATE_SESSION_RETRIEVE_FD, but the device
itself is not preserved in a fully running state across Live Update.
This series aims to provide a foundation on which to build the rest of
the device preservation infrastructure, including:
1. Preservation of iommufd files [1]
2. Preservation of IOMMU driver state
3. Preservation of PCI state (BAR resources, device state, bridge state, ...)
4. Preservation of vfio-pci driver state
Steps 1 and 2 are already in-progress on the mailing list. We are
working on a detailed roadmap for steps 3 and 4.
Testing
-------
The patches at the end of this series provide comprehensive selftests
for the new code added by this series. The selftests have been validated
in both a VM environment using a virtio-net PCIe device, and in a
baremetal environment on an Intel EMR server with an Intel DSA PCIe
device.
Here is an example of how to run the new selftests:
vfio_pci_liveupdate_uapi_test:
$ tools/testing/selftests/vfio/scripts/setup.sh 0000:00:04.0
$ tools/testing/selftests/vfio/vfio_pci_liveupdate_uapi_test 0000:00:04.0
$ tools/testing/selftests/vfio/scripts/cleanup.sh
vfio_pci_liveupdate_kexec_test:
$ tools/testing/selftests/vfio/scripts/setup.sh 0000:00:04.0
$ tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test --stage 1 0000:00:04.0
$ kexec ...
$ tools/testing/selftests/vfio/scripts/setup.sh 0000:00:04.0
$ tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test --stage 2 0000:00:04.0
$ tools/testing/selftests/vfio/scripts/cleanup.sh
It is also possible to run vfio_pci_liveupdate_kexec_test multiple times
to preserve multiple devices simultaneously across a Live Update. This
series has been tested with up to 8 devices concurrently preserved.
Dependencies
------------
This series is built on top of v7.0-rc4 plus a series from Pasha
Tatashin to fix the module refcounting in FLB:
https://lore.kernel.org/lkml/20260318141637.1870220-10-pasha.tatashin@soleen.com/
Changelog
---------
v3:
- Add logging & documentation for pci=assign-busses overrides (Pranjal)
- Use 2026 in drivers/pci/liveupdate.c (Pranjal)
- Use 2026 in drivers/vfio/pci/vfio_pci_liveupdate.c (Pranjal)
- Drop incoming/outgoing from PCI APIs (Sami)
- Eliminate duplicate extern declarations for vfio_device_fops
(Pranjal)
- Keep struct vfio_device_file private (Pranjal)
- Add comment about not supporting hot-plug (Pranjal)
- Add comment about not supporting VFs (Sami)
- Better error handling for liveupdate_flb_get_incoming() (Pranjal)
- Remove liveupdate_enabled() checks (Zhu)
- Remove liveupdate_enabled() checks in vfio_pci_liveupdate_init() (Pranjal)
- Drop IOMMU reference from bus number commit message (Bjorn)
- Add fabric rationale to commit message (Jason)
- Swap incoming ... outgoing ordering in commit message (Bjorn)
- Use vfio_device_cdev_opened() instead of df->group (Alex)
- Add comments for CONFIG_VFIO_PCI_ZDEV_KVM (Alex)
- Add comments for vfio_pci_is_intel_display() (Alex)
- Use pci_dev_try_lock() in freeze (Alex)
- Fix device reset locking in freeze() (me)
- Use u32 for domain in PCI (Bjorn)
- Use u32 for domain in VFIO (Bjorn)
- Make pci_liveupdate_incoming_nr_devices() private to drivers/pci/ (Bjorn)
- Fix dev->liveupdate_incoming readability (Bjorn)
- Take pci_ser_delete() out of WARN_ON_ONCE() (Bjorn)
- Drop reference to userspace & files from PCI commit message (Bjorn)
- Rename __vfio_device_fops_cdev_open() to vfio_device_cdev_open_file() (Alex)
- Fix NULL pointer dereference in release() (Alex)
- Handle return value of pci_liveupdate_outgoing_preserve() (Alex)
- Make pci_liveupdate_unregister_fh() unabe to fail. (Alex)
- Move vfio_liveupdate_incoming_is_preserved() to drivers/vfio/vfio.h (Alex)
- Add vfio_pci_core_probe_reset() (Alex)
- Forward declare ser struct in include/linux/vfio_pci_core.h (Alex)
- Bump compatibility string when adding reset_works (Alex)
- How will userspace detect partial preservation? (Alex)
- Require single device per iommu_group (Jason)
- Rename pci_liveupdate_register_fh() to pci_liveupdate_register_flb() (Vipin)
- Use ksft_exit_skip() and SKIP() (Vipin)
- Move documentation to code (Vipin)
- Use __u64 instead of int for token in Live Update selftest helpers (Gemini)
- Add documentation for drivers/pci/liveupdate.c (me)
v2: https://lore.kernel.org/kvm/20260129212510.967611-1-dmatlack@google.com/
v1: https://lore.kernel.org/kvm/20251126193608.2678510-1-dmatlack@google.com/
rfc: https://lore.kernel.org/kvm/20251018000713.677779-1-vipinsh@google.com/
Cc: Pranjal Shrivastava <praan at google.com>
Cc: Saeed Mahameed <saeedm at nvidia.com>
Cc: Adithya Jayachandran <ajayachandra at nvidia.com>
Cc: Jason Gunthorpe <jgg at nvidia.com>
Cc: Parav Pandit <parav at nvidia.com>
Cc: Leon Romanovsky <leonro at nvidia.com>
Cc: William Tu <witu at nvidia.com>
Cc: Jacob Pan <jacob.pan at linux.microsoft.com>
Cc: Lukas Wunner <lukas at wunner.de>
Cc: Pasha Tatashin <pasha.tatashin at soleen.com>
Cc: Mike Rapoport <rppt at kernel.org>
Cc: Pratyush Yadav <pratyush at kernel.org>
Cc: Samiullah Khawaja <skhawaja at google.com>
Cc: Chris Li <chrisl at kernel.org>
Cc: Josh Hilke <jrhilke at google.com>
Cc: David Rientjes <rientjes at google.com>
[1] https://lore.kernel.org/linux-iommu/20251202230303.1017519-1-skhawaja@google.com/
David Matlack (15):
liveupdate: Export symbols needed by modules
PCI: Add API to track PCI devices preserved across Live Update
PCI: Require Live Update preserved devices are in singleton
iommu_groups
PCI: Inherit bus numbers from previous kernel during Live Update
docs: liveupdate: Add documentation for PCI
vfio/pci: Notify PCI subsystem about devices preserved across Live
Update
vfio: Enforce preserved devices are retrieved via
LIVEUPDATE_SESSION_RETRIEVE_FD
vfio/pci: Store incoming Live Update state in struct
vfio_pci_core_device
docs: liveupdate: Add documentation for VFIO PCI
vfio: selftests: Add Makefile support for TEST_GEN_PROGS_EXTENDED
vfio: selftests: Add vfio_pci_liveupdate_uapi_test
vfio: selftests: Expose iommu_modes to tests
vfio: selftests: Expose low-level helper routines for setting up
struct vfio_pci_device
vfio: selftests: Verify that opening VFIO device fails during Live
Update
vfio: selftests: Add continuous DMA to vfio_pci_liveupdate_kexec_test
Vipin Sharma (9):
vfio/pci: Register a file handler with Live Update Orchestrator
vfio/pci: Preserve vfio-pci device files across Live Update
vfio/pci: Retrieve preserved device files after Live Update
vfio/pci: Skip reset of preserved device after Live Update
selftests/liveupdate: Move luo_test_utils.* into a reusable library
selftests/liveupdate: Add helpers to preserve/retrieve FDs
vfio: selftests: Build liveupdate library in VFIO selftests
vfio: selftests: Initialize vfio_pci_device using a VFIO cdev FD
vfio: selftests: Add vfio_pci_liveupdate_kexec_test
Documentation/PCI/liveupdate.rst | 23 +
.../admin-guide/kernel-parameters.txt | 6 +-
Documentation/core-api/liveupdate.rst | 2 +
.../driver-api/vfio_pci_liveupdate.rst | 23 +
MAINTAINERS | 2 +
drivers/pci/Kconfig | 11 +
drivers/pci/Makefile | 1 +
drivers/pci/liveupdate.c | 415 ++++++++++++++++++
drivers/pci/pci.h | 14 +
drivers/pci/probe.c | 37 +-
drivers/vfio/device_cdev.c | 63 ++-
drivers/vfio/group.c | 9 +
drivers/vfio/pci/Kconfig | 11 +
drivers/vfio/pci/Makefile | 1 +
drivers/vfio/pci/vfio_pci.c | 14 +-
drivers/vfio/pci/vfio_pci_core.c | 90 ++--
drivers/vfio/pci/vfio_pci_liveupdate.c | 328 ++++++++++++++
drivers/vfio/pci/vfio_pci_priv.h | 18 +
drivers/vfio/vfio.h | 18 +
drivers/vfio/vfio_main.c | 16 +-
include/linux/kho/abi/pci.h | 62 +++
include/linux/kho/abi/vfio_pci.h | 45 ++
include/linux/pci.h | 41 ++
include/linux/vfio.h | 13 +
include/linux/vfio_pci_core.h | 2 +
kernel/liveupdate/luo_core.c | 1 +
kernel/liveupdate/luo_file.c | 2 +
tools/testing/selftests/liveupdate/.gitignore | 1 +
tools/testing/selftests/liveupdate/Makefile | 14 +-
.../include/libliveupdate.h} | 11 +-
.../selftests/liveupdate/lib/libliveupdate.mk | 20 +
.../{luo_test_utils.c => lib/liveupdate.c} | 43 +-
.../selftests/liveupdate/luo_kexec_simple.c | 2 +-
.../selftests/liveupdate/luo_multi_session.c | 2 +-
tools/testing/selftests/vfio/Makefile | 23 +-
.../vfio/lib/include/libvfio/iommu.h | 2 +
.../lib/include/libvfio/vfio_pci_device.h | 8 +
tools/testing/selftests/vfio/lib/iommu.c | 4 +-
.../selftests/vfio/lib/vfio_pci_device.c | 60 ++-
.../vfio/vfio_pci_liveupdate_kexec_test.c | 256 +++++++++++
.../vfio/vfio_pci_liveupdate_uapi_test.c | 93 ++++
41 files changed, 1715 insertions(+), 92 deletions(-)
create mode 100644 Documentation/PCI/liveupdate.rst
create mode 100644 Documentation/driver-api/vfio_pci_liveupdate.rst
create mode 100644 drivers/pci/liveupdate.c
create mode 100644 drivers/vfio/pci/vfio_pci_liveupdate.c
create mode 100644 include/linux/kho/abi/pci.h
create mode 100644 include/linux/kho/abi/vfio_pci.h
rename tools/testing/selftests/liveupdate/{luo_test_utils.h => lib/include/libliveupdate.h} (80%)
create mode 100644 tools/testing/selftests/liveupdate/lib/libliveupdate.mk
rename tools/testing/selftests/liveupdate/{luo_test_utils.c => lib/liveupdate.c} (89%)
create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_kexec_test.c
create mode 100644 tools/testing/selftests/vfio/vfio_pci_liveupdate_uapi_test.c
base-commit: 3251ac3df4374e4e94e4fbecf49ad1573933018a
prerequisite-patch-id: 37ebd38e2247ccb02e6a6c7543a378534d69a038
prerequisite-patch-id: 19c8469a9ae5cd13618481dd75012444330f80f9
prerequisite-patch-id: 2ba04f598993e2c2d941c4cd3f2dc1e98905d68b
prerequisite-patch-id: fcab928f6ee32a145667822ceca5e4f1f567d530
prerequisite-patch-id: 6d11347278609426baa9eacc1726b32b48d09a25
prerequisite-patch-id: 94a9e8a8cb6004e12de90fcc0068e8a8b12652de
prerequisite-patch-id: d658019a7ac7c82ebe4a6c6086457e27460174d3
prerequisite-patch-id: 41e68c9fc8e8c5e493497e87ca13577b3167cf80
--
2.53.0.983.g0bb29b3bc5-goog
More information about the kexec
mailing list