[PATCH 0/7] Enable Qualcomm WiFi PCIe passthrough to VMs
Michael Kelley
mhklinux at outlook.com
Thu Nov 5 09:20:51 PST 2026
From: Michael Kelley <mhklinux at outlook.com> Sent: Wednesday, October 14, 2026 1:47 PM
>
> From: Jason Gunthorpe <jgg at ziepe.ca> Sent: Wednesday, October 14, 2026 8:01 AM
> >
> > On Tue, Oct 13, 2026 at 05:42:25PM +0200, Jose Ignacio Tornos Martinez wrote:
> > > Thank you for the direction on device MSI domains. The Hyper-V
> > > reference is difficult for me to follow for this case. Could
> > > you point me to some existing KVM/VFIO equivalent, or similar
> > > example for my use case? Any pointers to relevant code,
> > > documentation, or previous discussions on this topic would be
> > > very helpful to get started in the right direction.
> > >
> > > Also, where would this need to be implemented? Kernel only
> > > because it is already supported in QEMU in a general way, or
> > > kernel and QEMU for some specifics?
> >
> > I spent some time with AI to research this and drafted up some slop
> > for you that outlines a bunch of ideas and explains how some of the
> > bits work today. Not saying this is something to follow but it should
> > be a good starting point to come up with a good idea.
>
> This is a nice summary of what Hyper-V does. It looks right to me
> based on my experience with that driver (pci-hyperv.c) over the
> years.
>
> I wrote the documentation in Documentation/virt/hyperv/vpci.rst.
> With your permission, I'd like to adapt some of your material below
> and add it to vpci.rst, primarily to provide some broader context
> on the "why" and the implications of the overall approach. I'm
> not sure how soon that will be, but it will go on my "to do" list.
In working on this documentation update, I found one error in
what you have written. See comment below.
>
> Michael
>
> >
> > It is quite interesting that amd virtual interrupt remapping in the
> > iommu is very close to being able to solve the problem without any
> > guest changes. Maybe that is the right thread to pull on, if AMD can
> > be fixed fully then maybe there is an option to negotiate something
> > within the Intel spec, or build a similar idea with a 'virtio
> > interrupt remapping'
> >
> > Nevertheless this looks reasonably tractable to me, assuming you have
> > some strong AI to help understand the whole picture and baroque
> > hardware details it is even almost approachable..
> >
> > Jason
> >
> > Host-managed MSI: the Hyper-V flow and a virtio proposal
> > ========================================================
> >
> > Goal: make device-specific MSI storage work for devices assigned to KVM guests.
> > Such storage is outside the PCI MSI/MSI-X capability. In Linux it is driven
> > through a per-device MSI domain, today via
> > platform_device_msi_init_and_alloc_irqs(). A PCI device needs a PCI counterpart
> > of that helper, which does not exist yet and must be written. The VMM cannot
> > trap this storage, so the guest must write host-valid MSI messages into it
> > directly, as Hyper-V guests do for MSI-X. The guest obtains those messages, and
> > routes the interrupts, through a paravirtual (virtio) interface.
> >
> > Sections 1–4 describe how Hyper-V does this through the Linux IRQ domain
> > hierarchy. Section 5 maps that onto the KVM/VFIO host stack and assesses the
> > virtio proposal. Section 6 looks at an alternative with no paravirtual
> > interface: the host mirrors the guest's vIOMMU interrupt remapping (or vITS)
> > entries 1:1.
> >
> > 0. Why host-managed MSI exists at all
> > -------------------------------------
> >
> > On a normal VFIO/KVM system the VMM traps guest writes to a passthrough device's
> > MSI/MSI-X registers. The guest writes a virtual message (a guest APIC address
> > and a guest vector). The VMM then programs the physical device and the physical
> > interrupt remapping (IR) table on the guest's behalf. The guest never sees a
> > host-valid message.
> >
> > Hyper-V does not do that. In the vPCI and root-partition cases, the guest writes
> > the device's MSI capability and MSI-X table directly. Writes are not trapped and
> > emulated (see the comment above hv_pci_restore_msi_state()). The IOMMU and its
> > interrupt remapping table are owned by the hypervisor. The value the guest
> > writes must therefore already be a host-valid, remapped message, and only the
> > host can produce one. Hyper-V has the guest ask the host for each message.
> >
> > Every design decision below follows from that premise:
> >
> > The guest chooses a destination (vCPU + vector, or a guest interrupt line).
> > The host allocates a remapping entry for (source device → destination) and
> > returns the MSI address/data that selects it. A Linux IRQ domain hands that
> > pair to the MSI core, which writes it into the device using the normal PCI MSI
> > writer.
> >
> > For device-specific storage on KVM there is no trap to remove: the storage was
> > never trappable. Adopting this premise is the only way it can work in a guest.
> > Section 5.2 shows that the KVM half (stable host-composed messages, IRTE-only
> > retargeting, posting) already exists. The missing pieces are host IRQs without
> > host-side storage in VFIO, and an MSI parent in the guest that serves both the
> > PCI MSI-X domain and per-device MSI domains.
> >
> > 1. Separate the destination from the message
> > --------------------------------------------
> >
> > -------------------------------------------------------------------------------
> > Identity Meaning
> > ----------------------- -------------------------------------------------------
> > Linux IRQ number (virq) The guest kernel's software handle. Never visible to
> > the host.
> >
> > Destination What the guest wants to receive: on x86 a vCPU set +
> > IDT vector; on Hyper-V ARM64 a GIC SPI number
> >
> > Source identity The requester whose MSI writes the host will accept for
> > this mapping
> >
> > Host mapping Host state binding (source, message) → destination; on
> > hardware, an IR table entry
> >
> > MSI address/data What is written into the device; on IR hardware this
> > effectively selects the host mapping
> > -------------------------------------------------------------------------------
> >
> > In a hierarchical domain, one Linux IRQ has an irq_data at each level, linked by
> > parent_data, and each level keeps private state in chip_data. For host-managed
> > MSI:
> >
> > - the parent (receive) domain allocates the guest destination;
> > - the host-managed MSI parent domain obtains the message from the host and
> > owns the mapping's lifetime;
> > - the device MSI domain writes the message into the device.
> >
> > The guest must treat the returned pair as host-defined. It must not reconstruct
> > an APIC-format address from the CPU number, substitute its own vector into the
> > data, or assume an ITS doorbell. The only structure the guest may rely on is
> > what the protocol explicitly defines. Hyper-V's one example is conventional
> > multi-message MSI, where the data of vector n is base data + n (section 4).
> >
> > 2. The common Linux path: compose, then write
> > ---------------------------------------------
> >
> > Root-partition hierarchy on x86:
> >
> > PCI device MSI/MSI-X domain writes device registers / MSI-X table
> > |
> > Hyper-V MSI parent domain obtains message from host, owns mapping
> > |
> > x86 vector domain allocates guest CPU + vector
> >
> > Allocation walks down the hierarchy to reserve receive resources. Activation
> > walks parents first, then the MSI device domain's msi_domain_activate() does:
> >
> > msi_domain_activate(device_domain, irq_data)
> > +-- irq_chip_compose_msi_msg(irq_data, msg)
> > | walks irq_data -> parent_data to the first level with
> > | .irq_compose_msi_msg -> Hyper-V callback returns host pair
> > +-- irq_chip_write_msi_msg(irq_data, msg)
> > -> pci_msi_domain_write_msg() -> __pci_write_msi_msg()
> > MSI: config-space address/data registers
> > MSI-X: the entry in the device's MSI-X table
> >
> > Notes:
> >
> > - The Hyper-V compose callback never touches device registers; the write
> > belongs to the PCI MSI layer.
> > - __pci_write_msi_msg() caches the message in msi_desc->msg. If the device is
> > not in D0 or is disconnected, it skips the hardware write.
> > - irq_compose_msi_msg returns void. msi_domain_activate() and
> > msi_domain_set_affinity() start from a zeroed msg. If the provider fails, it
> > simply leaves (or sets) 0/0, and a zero message is written to the device.
> > Nothing reports the failure to the caller.
> > - msi_domain_set_affinity() asks the parent to change destination. Unless the
> > parent returns IRQ_SET_MASK_OK_DONE, it composes and writes again. Every
> > affinity change therefore re-runs the Hyper-V compose callback.
Actually, it doesn't. A new MSI message is not needed when changing the
effective affinity CPU. The new CPU is communicated to Hyper-V via
hv_irq_retarget_interrupt() as called by hv_irq_unmask() on x86, or by the
GICv3 gic_set_affinity() function writing the emulated GICD registers on arm64.
I don't have much understanding of what's going on the Hyper-V side that makes
all this work.
Because a new MSI message is not needed, the default msi_domain_set_affinity()
function is not used in the PCI device MSI/MSI-X domain. msi_lib_init_dev_msi_info()
does indeed set the .irq_set_affinity function to msi_domain_set_affinity(), but
then hv_pcie_init_dev_msi_info() overrides that and sets .irq_set_affinity
to irq_chip_set_affinity_parent(). However, given how msi_domain_set_affinity()
first calls the parent set affinity function, this override seems unnecessary.
I may look further into the history of that ....
This detailed Hyper-V behavior is probably not relevant to virtio, so I'm just
pointing it out for the record.
Michael
> > - Both Hyper-V x86 chips set IRQCHIP_MOVE_DEFERRED. On x86 the affinity change
> > is applied from the interrupt's own handler context (irq_move_masked_irq()).
> > Any host interaction done in compose therefore runs in hard-IRQ context with
> > the descriptor lock held.
> >
> > 3. Hyper-V root partition: direct hypercalls
> > --------------------------------------------
> >
> > Code: arch/x86/hyperv/irqdomain.c.
> >
> > Here Linux is the root partition. It sees the physical PCI topology, but the
> > hypervisor owns the IOMMU.
> >
> > Installing the domain
> > ~~~~~~~~~~~~~~~~~~~~~
> >
> > hv_init.c sets x86_init.irqs.create_pci_msi_domain = hv_create_pci_msi_domain
> > when running as root. That creates a single system-wide MSI parent domain over
> > x86_vector_domain. hv_msi_domain_alloc() only allocates parent vectors and
> > installs hv_pci_msi_controller. It does no host work; a TODO says the mapping
> > should move there.
> >
> > Supported flags are generic + MSI-X. Conventional multi-message MSI is not
> > supported, and each mapping has vector_count = 1, edge-triggered.
> >
> > Mapping: all work happens in compose
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > hv_irq_compose_msi_msg(irqd, msg)
> > if irqd->chip_data: # already mapped (affinity change)
> > HVCALL_UNMAP_DEVICE_INTERRUPT(old entry); kfree
> > entry = kzalloc(GFP_ATOMIC)
> > hv_map_msi_interrupt()
> > source = physical segment/BDF, after walking DMA aliases;
> > PCI-X bridges add a bus-range "shadow"
> > cpu = first CPU of the effective affinity (single CPU)
> > vector = parent vector (irqd_cfg()->vector)
> > HVCALL_MAP_DEVICE_INTERRUPT -> hv_interrupt_entry
> > irqd->chip_data = entry
> > msg = { address_hi = 0,
> > address_lo = entry.msi_entry.address,
> > data = entry.msi_entry.data }
> >
> > The hypercall returns the 32-bit address and 32-bit data. Linux copies them
> > verbatim. address_hi = 0 is a property of this interface, not a guest encoding
> > choice.
> >
> > Affinity change
> > ~~~~~~~~~~~~~~~
> >
> > The hypervisor does not let the root partition change the vector or move the
> > interrupt to VPs outside the originally mapped set. For that reason, root does
> > not use RETARGET. Every affinity change is unmap, then map, followed by a device
> > rewrite with the possibly new pair.
> >
> > This has consequences:
> >
> > - Between the unmap and the device rewrite, the device holds a message that
> > refers to a released mapping.
> > - If the map fails, compose returns with chip_data == NULL and a zero msg, and
> > the device is programmed with 0/0. The IRQ is now dead and nothing reports
> > it.
> > - All of this runs in the deferred-move path, in hard-IRQ context (section 2).
> >
> > Teardown
> > ~~~~~~~~
> >
> > hv_msi_domain_free() → hv_teardown_msi_irq() issues
> > HVCALL_UNMAP_DEVICE_INTERRUPT with the stored entry. The mapping is a host
> > resource; forgetting the pair in the guest leaks it.
> >
> > 4. Hyper-V guest vPCI: CREATE over VMBus, RETARGET by hypercall
> > ---------------------------------------------------------------
> >
> > Code: drivers/pci/controller/pci-hyperv.c.
> >
> > Here Linux is an ordinary guest with passthrough devices exposed on a synthetic
> > PCI bus. Each vPCI bus is a VMBus device with its own channel.
> >
> > Two control paths
> > ~~~~~~~~~~~~~~~~~
> >
> > This split is the most important thing to understand before designing a virtio
> > version:
> >
> > CREATE (PCI_CREATE_INTERRUPT_MESSAGE{,2,3})
> >
> > - Transport: VMBus request/response
> > - Called from: hv_compose_msi_msg()
> > - Context: IRQ descriptor lock held; polls the channel
> >
> > DELETE (PCI_DELETE_INTERRUPT_MESSAGE)
> >
> > - Transport: VMBus, fire-and-forget
> > - Called from: compose (replace) and hv_msi_free()
> >
> > RETARGET (x86 only)
> >
> > - Transport: HVCALL_RETARGET_INTERRUPT, synchronous hypercall
> > - Called from: hv_irq_unmask()
> > - Context: atomic
> >
> > Slow, rare operations use the message transport. The operation that must
> > complete inside an IRQ chip callback uses a synchronous trap. VMBus is not used
> > for that.
> >
> > Domain setup
> > ~~~~~~~~~~~~
> >
> > hv_pcie_init_irq_domain() creates one MSI parent domain per vPCI bus, and
> > dev_set_msi_domain() binds it to that bus's host bridge. Device-to-domain
> > selection is therefore by topology: every device on that synthetic bus uses that
> > domain. Source identity is likewise implicit. The host already knows which
> > physical function sits behind each slot (wslot) on that channel.
> >
> > Parent receive domain:
> >
> > - x86: x86_vector_domain. Destination = vCPU(s) + IDT vector.
> > - ARM64: a private hv_vpci_arm64 domain over the GIC. It hands out SPIs
> > 64–1019 from a bitmap in power-of-two aligned blocks. Destination = SPI
> > number, sent to the host as the "vector". CPU routing of that SPI is done by
> > the guest's virtual GIC distributor, which Hyper-V already emulates. So
> > hv_arch_irq_unmask() is empty on ARM64, and no retarget is ever needed.
> >
> > CREATE
> > ~~~~~~
> >
> > hv_compose_msi_msg():
> >
> > 1. For multi-MSI with existing chip_data, return the cached pair (no host
> > call).
> > 2. If a previous int_desc exists (single MSI / MSI-X), send DELETE for it.
> > 3. Build a v1/v2/v3 CREATE request: slot, vector, vector count, and one "dummy"
> > vCPU (first online CPU of the affinity; round-robin for multi-MSI).
> > 4. Send it with vmbus_sendpacket_getid(). Disable the channel tasklet. Spin
> > calling hv_pci_onchannelcallback() with udelay(100) until the completion
> > fires, the device disappears (vendor ID 0xFFFF), or the channel is torn
> > down.
> > 5. Store the returned tran_int_desc in chip_data. Return
> > address_hi:lo = int_desc.address (64-bit) and data = int_desc.data.
> >
> > On any failure it returns a zero message.
> >
> > The vCPU in CREATE is not cosmetic. Hyper-V picks the physical CPU that services
> > the interrupt based on it, and (as of the comment in the driver) RETARGET does
> > not reselect the pCPU. The vector in CREATE is the real vector for single
> > MSI/MSI-X, but a dummy 32 for multi-MSI.
> >
> > RETARGET at unmask (x86)
> > ~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > hv_irq_unmask() → hv_arch_irq_unmask() → hv_irq_retarget_interrupt(). This
> > issues HVCALL_RETARGET_INTERRUPT with:
> >
> > - device id: built from the vPCI bus instance GUID + PCI function;
> > - the existing message (address_lo, data) used as the key that identifies the
> > mapping;
> > - the new vector and the full effective-affinity vCPU set.
> >
> > The message is unchanged; the host updates its IR entry behind it. Note that
> > Hyper-V uses the address/data pair itself as the mapping handle. There is no
> > separate handle.
> >
> > Nested root (Linux is root under a nested hypervisor) calls
> > hv_map_msi_interrupt(data, NULL) instead and discards the returned entry.
> >
> > Affinity in practice (x86)
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > An affinity change goes through msi_domain_set_affinity() → recompose. For
> > single MSI/MSI-X that means DELETE + CREATE over VMBus, polled in hard-IRQ
> > context, then a device rewrite, then RETARGET on the next unmask. Multi-MSI
> > skips the recompose (cached) and relies on RETARGET alone. All vectors of a
> > multi-MSI device therefore share whatever pCPU the host picked at CREATE.
> >
> > Multi-message MSI
> > ~~~~~~~~~~~~~~~~~
> >
> > Conventional MSI has one address and a base data value; the device ORs in the
> > vector index. The host allocates an aligned block for nvec_used vectors during
> > the first IRQ's CREATE. Later IRQs derive data = base + (irq - first_irq)
> > without asking the host. This works only because the host guarantees block
> > semantics. Independent per-vector pairs cannot be advertised as multi-MSI.
> >
> > Free and resume
> > ~~~~~~~~~~~~~~~
> >
> > hv_msi_free() sends DELETE for the stored descriptor. On resume, a cached MSI-X
> > table replay is useless because the host's IR entries are gone.
> > hv_pci_restore_msi_state() therefore calls hv_compose_msi_msg() for every
> > descriptor to re-CREATE.
> >
> > What the Hyper-V code itself admits is wrong
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > - Both providers have a TODO to move host allocation out of compose and into
> > .alloc.
> > - Failure in compose silently programs 0/0.
> > - vPCI CREATE runs a polled, unbounded message round-trip under IRQ locks,
> > including from the deferred affinity-move path.
> >
> > 5. Applying this to KVM/VFIO: device MSI storage in a guest
> > -----------------------------------------------------------
> >
> > 5.1 The functional problem
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > Many devices keep MSI address/data in device-specific storage: a context table
> > in device memory, or a field in a queue context that lives in DMA'd host memory.
> > The PCI MSI/MSI-X capability is not involved. (The PCI/IMS proposal used to
> > cover this; the current tree does it with per-device MSI domains, see below.) A
> > guest driver for an assigned device has to write a message into that storage,
> > and:
> >
> > - the VMM cannot trap it. The format is device-specific, and the storage may
> > be in guest RAM that the device reads by DMA;
> > - a guest-format message (guest APIC ID + guest vector) is wrong. On a host
> > with interrupt remapping, the device must emit a remappable-format message
> > that selects a host IRTE belonging to this device.
> >
> > So device-specific MSI storage cannot work in a KVM guest today. The only fix is
> > the Hyper-V model: the guest asks the host for a host-valid message and writes
> > it itself. This is "Model B": a host-provided message that the guest programs
> > into the storage.
> >
> > The generic Linux side already exists.
> > platform_device_msi_init_and_alloc_irqs(dev, nvec, write_msi_msg) creates a
> > DOMAIN_BUS_DEVICE_MSI domain on dev, parented on dev->msi.domain, and
> > allocates
> > nvec interrupts. The parent composes each message and the core hands it to the
> > driver's write_msi_msg(desc, msg), which stores it wherever the device wants it.
> > ARM ITS/MBI and the irq-msi-lib parents (e.g. RISC-V IMSIC) accept this bus
> > token. The x86 parents (x86_vector_msi_parent_ops, DMAR, AMDVI) accept only
> > DOMAIN_BUS_PCI_DEVICE_MSI{,X}, which is one of the gaps this work fills.
> >
> > Two properties of the helper shape the design:
> >
> > - The platform helper is the wrong API for a PCI device. It is built for
> > devices whose only MSI domain is the device MSI domain. It always uses
> > MSI_DEFAULT_DOMAIN, and MSI_MAX_DEVICE_IRQDOMAINS is 1. A PCI
> function's
> > own
> > MSI/MSI-X domain already occupies that slot. A PCI device therefore needs a
> > new pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) (and a
> > matching pci_device_msi_free_irqs_all()). Neither exists in the tree yet;
> > both must be written as part of this work. The new helper uses the same
> > DOMAIN_BUS_DEVICE_MSI template and driver write callback, but places the
> > domain in a second per-device domain slot that the core must add, so it
> > coexists with MSI-X on the same struct device. Both domains are then
> > parented on the function's dev->msi.domain, and the source identity is
> > simply the function itself.
> > - The device MSI chip has no mask of its own (irq_chip_mask_parent), no
> > affinity callback, and no feature flags. Masking and retargeting are the
> > parent's job, which suits a host-managed parent.
> >
> > 5.2 What the KVM/VFIO host already provides
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > VFIO_DEVICE_SET_IRQS(MSI-X, index i, eventfd)
> > host: pci_msix_alloc_irq_at(i) / request_irq(vfio_msihandler)
> > host IR domain composes a remappable msg (IRTE index [+ subhandle])
> > host writes that msg into the physical MSI-X entry i
> > irq_bypass_register_producer(eventfd, host_irq)
> > KVM_SET_GSI_ROUTING(gsi -> guest-format msg) + KVM_IRQFD(eventfd, gsi)
> > kvm_pi_update_irte(): post the IRTE to the vCPU PI descriptor + guest vector,
> > or leave it remapped to a host vector and let KVM inject
> >
> > Properties to reuse:
> >
> > - The host message is stable for the life of the host IRQ.
> > intel_ir_compose_msi_msg() returns the cached ir_data->msi_entry. Host
> > affinity changes (intel_ir_set_affinity(), amd_ir_set_affinity()) rewrite
> > only the IRTE and return IRQ_SET_MASK_OK_DONE. Posting changes
> > (irq_set_vcpu_affinity()) also rewrite only the IRTE.
> > - Guest retargeting is host-only. It is a KVM routing update followed by an
> > IRTE rewrite. The device is never touched.
> >
> > The KVM half (irqfd, GSI routing, posting) works on any host IRQ that has an IR
> > parent. It does not care whether the host IRQ came from MSI-X or from somewhere
> > else. The missing piece is entirely on the VFIO/host-IRQ side. VFIO can only
> > create host IRQs for MSI-X table indices. A device MSI interrupt needs a host
> > IRQ that has an IRTE but no host-side storage.
> >
> > 5.3 Security: what makes guest-written messages safe
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > The guest controls the device's DMA. It can already make the device write any
> > value to 0xFEEx_xxxx with its requester ID. VFIO's isolation therefore already
> > rests on interrupt remapping:
> >
> > - Intel: source-ID-checked IRTEs (set_msi_sid()), and compatibility-format
> > MSIs blocked;
> > - AMD: per-device remap tables.
> >
> > Letting the guest write host-valid messages into device storage adds no new
> > requirement, provided the whole requester ID belongs to that one guest. The
> > guest can only reach IRTEs whose source-ID check passes for its own RID, and all
> > of those are its own.
> >
> > This breaks for shared-RID assignment (SIOV/ADIs). Several guests' ADIs emit
> > MSIs with the same RID, and IR checks only the RID, not the PASID. A guest could
> > put another guest's IRTE handle into its own storage entry and inject interrupts
> > into that guest. That is why SIOV designs kept this storage host-programmed.
> > Guest-written messages are safe only for whole-function assignment (a PF or a VF
> > with its own RID). The proposal must state this scope, and the host must refuse
> > CREATE for devices whose RID is shared.
> >
> > 5.4 Protocol
> > ~~~~~~~~~~~~
> >
> > ----------------------------------------------------------------------------------
> > Op Request Response Transport
> > ------------ --------------------------- --------------------------- -------------
> > CREATE device, domain (MSI-X / per vector: handle, address virtqueue;
> > device MSI), index, count, (64-bit), data (32-bit); can sleep
> > storage address/data widths count granted
> >
> > RETARGET handle, guest-format MSI status; host message synchronous
> > message composed by the unchanged trapped
> > guest's parent vector register
> > domain
> >
> > DESTROY handle status; no further delivery virtqueue;
> > can sleep
> >
> > (host → config-change notification all handles stale; guest config
> > guest) must re-CREATE and rewrite interrupt
> > INVALIDATE storage
> > ----------------------------------------------------------------------------------
> >
> > Design points:
> >
> > - The message is stable and destination-independent. The host IR already
> > guarantees this (5.2). CREATE therefore carries no destination and can run
> > from the guest's .alloc, which can sleep and fail cleanly. Compose copies
> > the cached pair. Affinity changes never touch device storage. That matters
> > for device MSI in particular: writing a queue context from hard-IRQ context
> > is often impossible, and the device MSI chip has no affinity callback of its
> > own anyway.
> > - The RETARGET destination is the guest-format message. The guest's virtio-msi
> > level composes it by calling the parent x86_vector_domain (APIC ID, vector,
> > extended destination ID). The VMM puts it unchanged into KVM routing, so no
> > new destination encoding is needed.
> > - RETARGET is synchronous. It runs from irq_set_affinity (hard-IRQ context
> > with IRQCHIP_MOVE_DEFERRED) and from activation. Use a write to a trapped
> > virtio config/MMIO register and read back status. The vCPU waits in the exit
> > until the VMM has updated routing. A virtqueue cannot provide this. Hyper-V
> > uses the same split: VMBus for CREATE, a hypercall for RETARGET.
> > - The device MSI index is only a name. The host does not interpret the guest's
> > msi_index or storage slot. It only scopes handles, so that DESTROY and
> > INVALIDATE are well defined. (MSI-X differs, see 5.6(c).)
> > - Width negotiation. The x86 remappable message has a 32-bit address_lo
> > carrying the IRTE index, address_hi = 0, and data carrying the subhandle.
> > Device storage may be narrower. The guest declares the widths it has, and
> > the host fails CREATE if its message cannot fit.
> >
> > 5.5 Host-side work (VFIO + IR)
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > 1. Storage-less host IRQs. A VFIO device needs a way to allocate host IRQs with
> > an IR parent but no host writer. The same device MSI mechanism fits: VFIO
> > calls the new (not yet written) pci_device_msi_init_and_alloc_irqs() on the
> > assigned function, under its existing IR parent, with a write callback that
> > only records the composed message. The host cannot mask storage it doesn't
> > own; masking stays at the IR/eventfd level and the guest owns device-side
> > masking. This needs:
> > - writing pci_device_msi_init_and_alloc_irqs() and adding the second
> > per-device domain slot (5.1), shared with the guest side,
> > - the x86 vector, DMAR and AMDVI parents accepting DOMAIN_BUS_DEVICE_MSI
> > (x86_init_dev_msi_info(), x86_msi_prepare()), with an allocation type
> > that programs the PCI function's source ID (set_msi_sid()) as for MSI-X,
> > - the host refusing it without isolated MSI.
> > 2. New VFIO uAPI. "Allocate N storage-less interrupts on this device, attach
> > eventfds, return each host message." This could be a new IRQ index or a new
> > ioctl. It needs a quota, and the irq bypass producer is registered exactly
> > as for MSI-X, so KVM posting works unchanged.
> > 3. Scalability. Device-specific storage exists largely to go beyond MSI-X's
> > 2048 vectors. Each guest interrupt here costs:
> > - a host IRQ and a host vector, because the IR domain allocates a
> > vector-domain parent even when posted. That is about 200 per host CPU.
> > - a KVM GSI route. KVM_MAX_IRQ_ROUTES is 4096, and KVM_SET_GSI_ROUTING
> > replaces the whole table, with expedited SRCU, on every RETARGET.
> > - an IRTE. Intel has 64K per IOMMU, shared host-wide; AMD has a
> > 512/2048-entry per-device table.
> >
> > Thousands of device MSI interrupts per guest need posted-only IRTEs without
> > a host vector, and an incremental single-route KVM update (or a KVM-side
> > RETARGET). Neither exists today. Without them the design tops out at roughly
> > MSI-X scale, which defeats much of the point.
> >
> > 5.6 Guest-side work
> > ~~~~~~~~~~~~~~~~~~~
> >
> > (a) Device storage through a device MSI domain. Once the new helper exists
> > (5.1), the device driver calls
> > pci_device_msi_init_and_alloc_irqs(pdev, nvec, write_msi_msg) alongside its
> > normal pci_alloc_irq_vectors() MSI-X allocation. The callback receives the
> > msi_desc and uses desc->msi_index to find the queue/context to write. The domain
> > is a fixed-size block allocated up front and is torn down with
> > pci_device_msi_free_irqs_all(). Apart from the new PCI entry point and the
> > second slot, this is the existing platform MSI mechanism unchanged.
> >
> > (b) The virtio-msi domain is the device's MSI parent.
> > msi_create_device_irq_domain() parents every per-device domain on
> > dev->msi.domain, so the function's MSI-X domain and its device MSI domain share
> > one parent. For a served device that parent must be the virtio-msi domain. It
> > supplies its own init_dev_msi_info accepting DOMAIN_BUS_PCI_DEVICE_MSIX and
> > DOMAIN_BUS_DEVICE_MSI (the x86 helper rejects both a non-x86 real parent and
> the
> > device MSI token), its own msi_prepare that tags CREATE with the domain type,
> > and irq_mask/irq_unmask, because the device MSI chip forwards masking to the
> > parent. MSI-X is then also guest-written. VFIO's host writes to entry i write
> > the same value, and the VMM must stop emulating the MSI-X table for that device.
> > This is consistent and simpler than splitting MSI-X and device MSI across two
> > models.
> >
> > (c) MSI-X entries must be bound to the host index. The host masks, restores and
> > refreshes MSI-X entry i (in vfio_msi_set_vector_signal() and
> > pci_restore_msi_state()). CREATE for MSI-X therefore names index i, and the
> > guest must write that message only into entry i. Ownership must be defined:
> >
> > - address/data: the guest, with the host only rewriting the identical value;
> > - the per-vector mask bit: the guest, once set-up completes.
> >
> > Device MSI has no host writer, so there is no conflict there.
> >
> > (d) Domain binding on KVM. Hyper-V gives each passthrough device its own
> > synthetic bus, so dev_set_msi_domain() on the bridge selects the provider. A q35
> > guest mixes emulated devices, which keep normal MSI, with assigned ones, and x86
> > has no firmware mechanism such as DT msi-map or ACPI IORT to choose an MSI
> > parent per device. Put the assigned devices behind a dedicated host bridge (e.g.
> > pxb-pcie) and bind virtio-msi to that bridge. The device MSI domain lives on the
> > same function, so it gets the same parent.
> >
> > (e) Probe ordering and bootstrap. The virtio-msi control device takes its own
> > interrupts from ordinary emulated MSI, not from its own domain. Its domain must
> > exist before any served device allocates vectors. A dedicated bridge enumerated
> > after virtio-msi probes, or -EPROBE_DEFER on the domain's absence, handles that.
> >
> > (f) Callbacks.
> >
> > struct virtio_msi_irq {
> > u32 handle;
> > struct msi_msg msg; /* host message, stable until INVALIDATE */
> > };
> >
> > - .alloc: allocate parent vectors, then CREATE (sleeping). On failure, unwind
> > and return the error, so pci_alloc_irq_vectors() or
> > pci_device_msi_init_and_alloc_irqs() fails cleanly. Hyper-V instead programs
> > 0/0.
> > - .activate: the parent activates first. Compose the guest-format message from
> > the parent and RETARGET.
> > - .irq_set_affinity: call the parent, compose its new guest-format message,
> > RETARGET, and return IRQ_SET_MASK_OK_DONE so the device storage is never
> > rewritten. If RETARGET fails, restore the parent and return an error.
> > - .irq_compose_msi_msg: *msg = m->msg.
> > - .free: DESTROY (sleeping) after the core has shut the IRQ down.
> >
> > 5.7 Migration and invalidation
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > The destination host allocates different IRTEs, so every message the guest has
> > written becomes invalid. For MSI-X the guest can rewrite from cached
> > descriptors. For device MSI the storage is device-specific, possibly in guest
> > RAM. Only the driver's write_msi_msg callback can update it, and it must happen
> > before the device raises interrupts on the destination. Options:
> >
> > - INVALIDATE + re-CREATE + rewrite. The core walks each served domain's
> > descriptors, recomposes and calls irq_chip_write_msi_msg(), which reaches
> > the driver callback. That is a new requirement on device MSI drivers: the
> > callback must work outside the normal activation path, and the driver must
> > tolerate it while queues are live or quiesced.
> > - Host preserves the messages. This is impractical with Intel's global IRTE
> > table. It may be feasible with AMD's per-device remap table, because indices
> > are per device and the destination's table is empty.
> >
> > Device-state migration (VFIO migration v2) must also be ordered with this: the
> > device must not resume before its stored messages are valid. The same path
> > handles host suspend/resume and control-device reset.
> >
> > 5.8 Points the proposal must settle
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > 1. Scope: whole-function assignment only (5.3), x86 only. ARM64 with a vITS has
> > an architectural path (doorbell mapping plus guest EventIDs installed in the
> > physical ITT) and does not need a paravirtual message allocator.
> > 2. Host-side scale (5.5.3): without vector-less posted IRTEs and incremental
> > KVM routing, device MSI in a guest is limited to roughly MSI-X scale.
> > 3. The write_msi_msg callback contract for migration and invalidation (5.7).
> > 4. Domain binding: a dedicated bridge (5.6(d)).
> > 5. Whether RETARGET stays a VMM register (simple, the same cost as today's
> > trapped MSI-X write) or becomes a KVM-side operation (scales, new uAPI).
> >
> > 5.9 Prototype order
> > ~~~~~~~~~~~~~~~~~~~
> >
> > 1. Host: storage-less IR-backed IRQs in VFIO plus message readback. Verify that
> > posting through kvm_pi_update_irte works on them.
> > 2. Guest: virtio-msi parent with MSI-X only (index-bound, guest-written, VMM
> > table trap disabled). Verify that the written value equals the host's
> > composed message, and that affinity changes cause RETARGET without any
> > device rewrite.
> > 3. Device MSI: add the second per-device domain slot and write
> > pci_device_msi_init_and_alloc_irqs(), then allocate a device MSI domain next
> > to MSI-X on a test device, or on an emulated device with its own message
> > storage in QEMU backed by a real VFIO function.
> > 4. Failure injection (CREATE/RETARGET), CPU hot-unplug, FLR, guest driver
> > unbind with live interrupts.
> > 5. INVALIDATE/rewrite, then live migration.
> > 6. Scale tests at thousands of device MSI vectors per guest.
> >
> > 6. Alternative: mirror the guest's interrupt remapping 1:1
> > ----------------------------------------------------------
> >
> > 6.1 The idea
> > ~~~~~~~~~~~~
> >
> > Sections 4–5 make the host choose the message and teach the guest to ask for it.
> > The alternative turns this around: the guest chooses the message with its own,
> > unmodified interrupt remapping driver, and the host makes the physical remapping
> > hardware match.
> >
> > With a vIOMMU that does interrupt remapping (VT-d IR, AMD IR), or a vITS on ARM,
> > a guest driver's device MSI domain is parented on the guest's own remapping
> > domain. The message it composes is already in remappable form: an IRTE index
> > (x86) or an EventID (ITS). It names an entry in a guest-owned table, not a
> > destination. If the host installs, at the same index, a physical entry for the
> > same source device that delivers to the destination the guest's entry names,
> > then the guest-written message works on real hardware unchanged. The device
> > storage needs no trap and no host-provided message, so virtio-msi is
> > unnecessary.
> >
> > The destination translation already exists. KVM's posted-interrupt and GICv4
> > paths turn a guest-format destination into a physical IRTE or vLPI mapping
> > (5.2). The new requirement is that the host index is chosen by the guest rather
> > than by the host allocator.
> >
> > guest driver: device MSI domain -> guest IR domain (vIOMMU) -> vector domain
> > composes {IRTE index i}, writes it into device storage
> > guest IR: writes vIRTE[i] = (vBDF, APIC, vector); invalidates index i
> > VMM: traps the invalidation, reads vIRTE[i], maps vBDF -> VFIO device
> > host: allocates a storage-less host IRQ at IR index i for that RID,
> > KVM routing gsi -> (APIC, vector), irqfd, posting as today
> > device: emits {index i} with its RID -> physical IRTE[i] -> vCPU
> >
> > Retargeting needs nothing new. A guest affinity change rewrites vIRTE[i] and
> > invalidates it. The VMM updates the KVM route, and kvm_pi_update_irte() rewrites
> > the physical IRTE. The index, and so the device storage, never changes. That is
> > the stability property section 5.4 relied on, obtained here without a protocol.
> >
> > What the host needs in every variant:
> >
> > - Storage-less host IRQs (5.5.1), with the extra ability to allocate at a
> > caller-specified remapping index or EventID. Today alloc_irte() (Intel),
> > alloc_irq_index() (AMD) and its_alloc_device_irq() (ITS) all pick the first
> > free slot.
> > - A uAPI to request that index, e.g. VFIO "allocate a storage-less IRQ on this
> > device at remapping index i", with failure reported if i is not available.
> > - A guest that can parent a device MSI domain on its vIOMMU IR domain. That is
> > the same x86 gap as on the host (DMAR/AMDVI accepting
> > DOMAIN_BUS_DEVICE_MSI), plus the new pci_device_msi_init_and_alloc_irqs()
> > (5.1). No paravirtual driver is needed, apart from VT-d's index allocation
> > callback (6.2).
> >
> > Whether the scheme is viable depends entirely on whether the guest's index space
> > can be made to coincide with a host index space that the host can dedicate to
> > this device. That differs sharply by architecture.
> >
> > 6.2 Intel VT-d: one global index space per IOMMU
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > How the VMM learns the guest's entry. The guest writes IRTEs into a table in its
> > own memory, then issues an interrupt entry cache (IEC) invalidation through the
> > queued-invalidation interface (modify_irte() → qi_flush_iec(), index-selective).
> > QEMU already traps this. vtd_iec_notify_all() fans it out to IEC notifiers, and
> > today kvm_update_msi_routes_all() uses it to refresh KVM routes for trapped
> > MSI/MSI-X. A mirror would add a notifier that, for index i, reads vIRTE[i], uses
> > its SID (Linux always programs source-ID verification, set_msi_sid()) to find
> > the VFIO device, and allocates or updates the host IRQ at index i. The guest's
> > invalidation waits on a QI wait descriptor, so the host update completes before
> > the guest goes on to write device storage. Ordering is correct for free.
> >
> > The problem: index collisions. A VT-d message carries a 16-bit handle into the
> > IOMMU's single interrupt remapping table. There is no per-device or per-domain
> > table and no offset. The physical table on that IOMMU is shared by the host's
> > own interrupts, by every other assigned device behind it, and by every other
> > guest. The guest's IR driver allocates from index 0 upward
> > (bitmap_find_free_region()), exactly as the host's does and as every other
> > guest's does. Guest index i will often already be taken on the host.
> >
> > VT-d gives the vIOMMU no way to tell the guest "that index is busy". IEC
> > invalidation cannot fail. The best the VMM can do is leave the interrupt dead
> > and perhaps log a fault, which is the silent-failure mode section 4 criticised
> > in Hyper-V.
> >
> > The fix: a paravirtual index allocation callback. The guest must not pick the
> > index itself. It asks the host. The vIOMMU advertises a PV capability, and the
> > guest's alloc_irte() calls it instead of searching its own bitmap:
> >
> > guest alloc_irte(SID, count)
> > -> PV ALLOC(SID, count) trapped vIOMMU register (or hypercall)
> > host: find a free, count-aligned block on the physical IOMMU behind the
> > device with that SID; reserve it for that RID; return index i
> > guest marks i..i+count-1 used in its bitmap, writes vIRTE[i], flushes IEC
> > -> existing IEC trap: VMM reads vIRTE[i], programs physical IRTE[i]
> > guest clear_entries(i)
> > -> PV FREE(i) host releases the reservation
> >
> > - Only allocation is paravirtual. The IRTE format, the remappable message, IEC
> > invalidation, retargeting and posting all stay architectural VT-d, and they
> > reuse the QEMU trap points above. This is far thinner than virtio-msi: there
> > is no host-composed message, no CREATE/RETARGET protocol and no new guest
> > IRQ domain. The change is a hook in the guest's Intel IR driver.
> > - Allocation can fail cleanly. The PV call returns an error, so alloc_irte()
> > fails and the IRQ allocation unwinds. That is the failure path IEC
> > invalidation lacks.
> > - It must be synchronous and cheap. alloc_irte() runs under irq_2_ir_lock (a
> > raw spinlock) during IRQ domain allocation. A trapped vIOMMU register write
> > with status readback fits that context; a virtqueue does not. Alternatively
> > the guest calls out before taking the lock.
> > - All guest allocations go through it. Indices for emulated devices need no
> > physical IRTE, but they must not collide with indices the host later hands
> > out for assigned devices in the same vIR table. Letting the host hand out
> > every index keeps the guest's table and the reservations consistent. For an
> > emulated SID the host just allocates from a per-VM bitmap without touching
> > hardware.
> > - MSI-X can use the same path. If the host IRQ for MSI-X entry k is allocated
> > at the guest-obtained index, the guest-written MSI-X message equals the
> > host's, as in the AMD case (6.3).
> >
> > Alternatives considered, all worse:
> >
> > - One mirroring guest per physical IOMMU. The host allocates its own IRTEs
> > from the top of the table, and one guest gets the low indices. This is
> > deterministic but unscalable, and it depends on the platform's DRHD
> > topology.
> > - Pre-reserve a static window. Boot the vIOMMU with IR pre-enabled and every
> > vIRTE outside a host-reserved window marked present, so that the guest's
> > iommu_load_old_irte() avoids them. That path runs only in a kdump kernel, so
> > it still needs a guest change, and it fixes the window size at boot.
> > - Host IRTE window per guest, plus a fixed offset. Not possible: the device
> > emits the guest's index verbatim, and VT-d has no per-source index offset.
> >
> > Smaller issues:
> >
> > - Multi-vector blocks: alloc_irte() aligns count to a power of two and uses
> > the handle mask in invalidations. The host must reserve the same aligned
> > block.
> >
> > - The guest may use a global IEC invalidation (also issued when IRTA is
> > written). The VMM then has to rescan up to 64K vIRTEs.
> >
> > - Scale is unchanged from 5.5.3: each mirrored entry is still a host IRQ, a
> > host vector and a KVM route.
> >
> > - Migration: the indices are guest state (in vIRTEs and in device storage).
> > The destination host must reserve the same indices on its physical IOMMU,
> > and with a global table they may already be in use there. Fixing that needs
> > either a per-VM index range coordinated across the migration pool, or a host
> > → guest "reallocate" event that makes the guest rewrite device storage. The
> > latter is the 5.7 problem again.
> >
> > Verdict: VT-d needs one PV hook, an index allocation callback in the guest's IR
> > driver, because its handle space is global. With that hook, the rest of the
> > mirroring scheme is architectural and reuses existing QEMU and KVM paths.
> > Migration remains the weak point.
> >
> > 6.3 AMD-Vi: per-device index space
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > AMD remaps through a per-DeviceID interrupt table referenced from the DTE (512
> > entries, or 2048 with MAX_IRQS_PER_TABLE_2K). The message's data field carries
> > the index, and the IOMMU looks it up in the table belonging to the requester's
> > DeviceID. QEMU's AMD vIOMMU likewise gives the guest a per-device table and
> > traps INVALIDATE_INTERRUPT_TABLE (AMDVI_CMD_INVAL_INTR_TABLE), calling the
> > same
> > IEC notifier chain.
> >
> > For whole-function assignment (5.3) the physical table belongs to that one
> > device. No other device, guest or host consumer shares it. The guest's index i
> > and the host's index i live in the same private namespace, so the collision
> > problem from 6.2 does not arise for device MSI.
> >
> > The one sharer is the host itself, for the same device's MSI-X: VFIO allocates
> > host MSI-X IRTEs in the same per-device table, at indices chosen by
> > alloc_irq_index(). The guest's IR driver also allocates its MSI-X vIRTEs in its
> > per-device table. Two consistent options:
> >
> > - Mirror everything. When the guest writes a remappable MSI-X message (data =
> > vIRTE index i), VFIO allocates the host MSI-X IRQ at IRTE i too. The
> > host-composed message then equals the guest-written one, and MSI-X table
> > trapping is no longer needed for correctness either.
> > - Partition. The host keeps MSI-X in a range the guest's allocator will not
> > use. The guest allocator, like Intel's, gives no such guarantee without a
> > hint, so this is the weaker option.
> >
> > Caveats:
> >
> > - The table is per DeviceID, which is the alias after DMA aliasing. A function
> > behind a PCIe-to-PCI bridge shares its table with its siblings, so only
> > functions with their own DeviceID qualify. This is the same condition as
> > 5.3.
> > - Capacity is 512 or 2048 entries per device. That is enough for many
> > device-MSI uses, though well short of the "thousands per device" goal.
> > - The AMD vIOMMU in QEMU must emulate interrupt remapping completely enough
> > for the guest to use it (including x2APIC and the table-length encoding),
> > and the guest must enable IR on it.
> >
> > Verdict: AMD-Vi fits the mirroring model well. The host work is index-pinned,
> > storage-less host IRQs plus a VMM notifier. No guest PV interface is needed.
> >
> > 6.4 ARM GICv3/v4 ITS: per-device EventID space
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > The ITS is architecturally the closest match. A device's MSI is a write of an
> > EventID to the ITS doorbell (GITS_TRANSLATER). The DeviceID comes from the bus
> > (RID → StreamID → DeviceID through IORT / msi-map), not from the message.
> > Translation is per DeviceID: a device table entry points to that device's ITT,
> > indexed by EventID. EventIDs are therefore a private namespace per device, like
> > AMD-Vi and unlike VT-d.
> >
> > The guest drives a vITS. It issues MAPD (virtual DeviceID → ITT) and MAPTI
> > (DeviceID, EventID → vLPI, collection), then writes the EventID into device
> > storage. For that write to reach the guest on real hardware:
> >
> > 1. EventID. The host ITT for the physical DeviceID must contain the guest's
> > EventID e, mapped to a host LPI, and with GICv4 forwarded (VMAPTI) to the
> > guest's vLPI. The host ITS allocates events with its_alloc_device_irq(), the
> > first free region of lpi_map. It would need to allocate at a requested
> > EventID. The obvious rule for the device MSI domain is EventID = msi_index,
> > which VFIO sets to e.
> > 2. ITT sizing. its_msi_prepare() reuses an existing its_device for the DeviceID
> > ("shared") and keeps the ITT size from first creation. If MSI-X created the
> > ITT with 32 events, a device MSI domain on the same DeviceID cannot place
> > events beyond that. The ITT has to be sized for the whole guest-visible
> > EventID range at the first allocation, or be grown.
> > 3. Doorbell address. The device writes to the guest's vITS doorbell address.
> > That is a GPA (no vSMMU), or an IOVA the guest's stage 1 maps to that GPA
> > (nested vSMMU). Host stage 2 must map that GPA to the physical
> > GITS_TRANSLATER page of the ITS serving the device. Today the host maps the
> > physical doorbell at the reserved SW_MSI window (MSI_IOVA_BASE) that the
> > host itself chose (iommufd_sw_msi), and the guest's vITS doorbell sits
> > elsewhere. One of two things is needed: either the VMM places the vITS
> > doorbell where the host maps the physical one, or there is a new uAPI to map
> > the physical doorbell at a VMM-chosen IPA. iommufd has neither today.
> > 4. Learning the guest's events. With the in-kernel vITS, MAPTI is handled by
> > KVM (vgic_its_cmd_handle_mapi()), not QEMU. KVM already resolves (DeviceID,
> > EventID) → vLPI and calls its_map_vlpi() when an irqfd route is attached
> > (kvm_vgic_v4_set_forwarding()). But it does that lazily: forwarding happens
> > at route time, and MAPTI only creates the vLPI. Two options:
> > - The VMM pre-allocates host events 0..N-1 for the device, with routes for
> > each (DeviceID, e). This is a quota-sized upfront cost. KVM then
> > forwards whichever ones the guest maps.
> > - KVM (or a KVM → VMM exit on MAPTI/DISCARD for assigned DeviceIDs) drives
> > host event allocation on demand. That is new KVM ↔ VFIO plumbing.
> > 5. Retarget is MOVI/VMOVI, already handled by KVM for forwarded vLPIs. The
> > EventID, and so device storage, never changes.
> >
> > Security matches 5.3. The guest can only name EventIDs in its own device's ITT,
> > and the DeviceID is fixed by the hardware path. A shared DeviceID (aliasing, or
> > SIOV-style sharing) breaks it the same way.
> >
> > Verdict: the ITS model supports mirroring without collisions. The work lies in
> > the host ITS driver (allocating at an EventID, ITT sizing), in the doorbell
> > mapping uAPI, and in how KVM/VMM learn guest events. As 5.8 noted, this is also
> > the native ARM answer that never needed a message allocator.
> >
> > 6.5 Comparison with virtio-msi
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> > --------------------------------------------------------------------------------------
> > virtio-msi (5) Mirror on VT-d Mirror on AMD-Vi Mirror on ITS
> > ----------- -------------- -------------------- ----------------- --------------------
> > Guest PV virtio-msi IRQ index alloc/free none none
> > interface domain + callback in the IR
> > protocol driver
> >
> > Who picks host host, via PV guest guest
> > the index callback
> >
> > Collision none none: host allocates none: per-device none: per-device ITT
> > risk in the global table table
> >
> > Failure CREATE status PV ALLOC status none, but cannot none, but cannot
> > reporting collide collide
> > to guest
> >
> > Retarget trapped existing IEC trap → existing trap → existing KVM vITS
> > register KVM route KVM route
> >
> > Main new storage-less PV alloc/free, index-pinned EventID-pinned
> > host work IRQs, protocol index-pinned IRTEs, IRTEs, QEMU alloc, ITT sizing,
> > QEMU notifier notifier doorbell mapping,
> > event discovery
> >
> > Migration INVALIDATE + destination must destination destination
> > guest rewrite reproduce indices: reproduces reproduces
> > (5.7) fragile with a per-device per-device EventIDs:
> > global table, even indices: feasible feasible
> > with PV alloc
> > --------------------------------------------------------------------------------------
> >
> > Migration deserves emphasis. With mirroring, the guest's indices are part of the
> > guest state, held in guest memory and in device storage that only the guest
> > driver can rewrite. The destination host must recreate exactly those indices.
> > That works for per-device namespaces (AMD-Vi, ITS), because the destination
> > device's table is empty. With VT-d's global table it fails whenever the
> > destination IOMMU already uses one of them. This is the mirror image of 5.7,
> > where virtio-msi needs the guest to rewrite storage instead.
> >
> > Overall: 1:1 mirroring removes the need for virtio-msi where the remapping index
> > space is per device: AMD-Vi and the ARM ITS. On Intel VT-d the global handle
> > space needs one PV hook, an index allocation callback in the guest's IR driver.
> > That is still much smaller than virtio-msi, because the message format,
> > retargeting and posting stay architectural. Migration on VT-d remains fragile
> > without a coordinated per-VM index range, a guest rewrite path, or a hardware
> > change (per-device or per-domain IR tables).
> >
> > Source reading map
> > ------------------
> >
> > arch/x86/hyperv/hv_init.c
> >
> > - Functions: create_pci_msi_domain hook
> > - What they establish: Root-partition provider selection
> >
> > arch/x86/hyperv/irqdomain.c
> >
> > - Functions: hv_irq_compose_msi_msg, hv_map_interrupt,
> > hv_build_devid_type_pci, entry_to_msi_msg, hv_teardown_msi_irq
> > - What they establish: Hypercall output becomes the message; unmap+map on move
> >
> > drivers/pci/controller/pci-hyperv.c
> >
> > - Functions: hv_pcie_init_irq_domain, hv_compose_msi_msg,
> > hv_irq_retarget_interrupt, hv_arch_irq_unmask, hv_msi_free,
> > hv_pci_vec_irq_domain_alloc (ARM64), hv_pci_restore_msi_state
> > - What they establish: VMBus CREATE, hypercall RETARGET, per-bus binding, SPI
> > model, resume
> >
> > kernel/irq/chip.c
> >
> > - Functions: irq_chip_compose_msi_msg
> > - What they establish: Finding the composer in the hierarchy
> >
> > kernel/irq/msi.c
> >
> > - Functions: msi_domain_activate, msi_domain_set_affinity
> > - What they establish: Compose → write; recompose on move unless OK_DONE
> >
> > kernel/irq/migration.c
> >
> > - Functions: irq_move_masked_irq
> > - What they establish: Deferred moves run in interrupt context
> >
> > drivers/pci/msi/irqdomain.c
> >
> > - Functions: pci_msi_domain_write_msg
> > - What they establish: Device domain write callback
> >
> > drivers/pci/msi/msi.c
> >
> > - Functions: __pci_write_msi_msg
> > - What they establish: Config-space / MSI-X table programming
> >
> > drivers/vfio/pci/vfio_pci_intrs.c
> >
> > - Functions: vfio_msi_set_vector_signal, vfio_msi_alloc_irq
> > - What they establish: Host IRQ per MSI-X index, message refresh, irq bypass
> > producer
> >
> > drivers/iommu/intel/irq_remapping.c
> >
> > - Functions: intel_ir_compose_msi_msg, fill_msi_msg, intel_ir_set_affinity,
> > intel_ir_set_vcpu_affinity, set_msi_sid
> > - What they establish: Stable remappable message; IRTE-only retarget;
> > source-ID isolation
> >
> > drivers/iommu/amd/iommu.c
> >
> > - Functions: amd_ir_set_affinity, amd_ir_set_vcpu_affinity
> > - What they establish: Same properties on AMD
> >
> > drivers/base/platform-msi.c
> >
> > - Functions: platform_device_msi_init_and_alloc_irqs, platform_msi_template,
> > platform_device_msi_free_irqs_all
> > - What they establish: Device MSI domain with a driver write callback; masking
> > forwarded to the parent. Model for the new PCI helper
> >
> > kernel/irq/msi.c, include/linux/msi_api.h
> >
> > - Functions: msi_create_device_irq_domain, msi_domain_ids
> > - What they establish: Per-device domain parented on dev->msi.domain; only one
> > slot per device today
> >
> > drivers/irqchip/irq-msi-lib.c
> >
> > - Functions: msi_lib_init_dev_msi_info
> > - What they establish: How non-x86 parents accept DOMAIN_BUS_DEVICE_MSI
> >
> > arch/x86/kernel/apic/msi.c
> >
> > - Functions: x86_init_dev_msi_info, x86_msi_prepare
> > - What they establish: x86 parents accept only PCI MSI/MSI-X device domains
> >
> > arch/x86/kvm/irq.c
> >
> > - Functions: kvm_pi_update_irte, kvm_arch_update_irqfd_routing
> > - What they establish: Guest routing change → posted IRTE update
> >
> > drivers/iommu/intel/irq_remapping.c
> >
> > - Functions: alloc_irte, modify_irte, qi_flush_iec, iommu_load_old_irte,
> > fill_msi_msg
> > - What they establish: Global first-fit IRTE allocation; IEC invalidation per
> > index; pre-enabled IR copy (kdump only)
> >
> > drivers/iommu/amd/iommu.c
> >
> > - Functions: alloc_irq_index, irq_remapping_alloc
> > - What they establish: Per-DeviceID interrupt table, 512/2048 entries
> >
> > drivers/irqchip/irq-gic-v3-its.c
> >
> > - Functions: its_msi_prepare, its_alloc_device_irq, its_map_vlpi
> > - What they establish: Per-DeviceID ITT, first-fit EventIDs, ITT size fixed at
> > first use, GICv4 forwarding
> >
> > arch/arm64/kvm/vgic/vgic-its.c, arch/arm64/kvm/vgic/vgic-v4.c
> >
> > - Functions: vgic_its_cmd_handle_mapi, kvm_vgic_v4_set_forwarding
> > - What they establish: In-kernel vITS commands; vLPI forwarding set up at
> > irqfd route time
> >
> > QEMU hw/i386/intel_iommu.c, hw/i386/amd_iommu.c, target/i386/kvm/kvm.c
> >
> > - Functions: vtd_iec_notify_all, AMDVI_CMD_INVAL_INTR_TABLE,
> > kvm_update_msi_routes_all
> > - What they establish: VMM trap points for guest IRTE changes
More information about the ath12k
mailing list