[PATCH v10 00/17] KVM: arm64: GICv3 ITS emulation

[Auger Eric]

Hi Andre,

On 15/07/2016 13:43, Andre Przywara wrote:
> Hi,
> 
> this series allows those KVM guests that use an emulated GICv3 to use LPIs
> as well, though in the moment this is limited to emulated PCI devices.
> This is based on kvmarm/queue, which now only features the new VGIC
> implementation.
> 
> Another round of fixes:
> An optimization from Marc to only actually refcount LPIs. This avoids
> some churn on SPIs and private IRQs which don't need to be refcounted.
> Some bugs have been fixed in vgic_copy_lpi_list(), sync_pending_table(),
> and the CWRITER handler.
> Now we check for device IDs and collection IDs to be actually within
> the range the guest once advertised, also a check was added to make sure
> we don't try to propagate an unmapped collection to a struct vgic_irq.
> From now on we only support 64K sized pages for the ITS BASER tables.
> Some comments have been added or improved.
> The changelog below holds a more detailed list.
> 
> You can find all of this code (and the prerequisites) in the
> its-emul/v10 branch of my repository [1].
> This has been briefly tested on the model and on GICv3 hardware, with
> lockdep debugging enabled. If you have GICv3 capable hardware, please
> test it on your setup.
> Also of course any review comments are very welcome!
> 
> Cheers,
> Andre.
> 
> Changelog v9..v10:
> - Don't actually refcount SPIs, PPIs, SGIs (only LPIs)
> - add vgic-its.c to the Makefile later (avoids warnings in between)
> - fix vgic_copy_lpi_list() counter overflow
> - fix sync_pending_table optimization to actually work
> - drop support for 4K and 16K BASER tables
> - use consistent naming for BASER_CACHEABILITY_MASK definitions
> - check device ID and collection IDs to fit into the tables
> - extra check for indirect device tables to contain valid pointers
> - check for collection being mapped before updating target_vcpu
> - add and refine some comments
> - fix CWRITER handling: rework masking, take lock earlier, fix comparison
> - move some ITS initialization bits into vgic_its_register and rename it
> 
> Changelog v8..v9:
> - MMIO dispatching now no longer holds iodev type in each register
> - indirect bit is now cleared for collections
> - sanitisation functions reworked (no extra masking, taking
>   register specific masks)
> - make PROPBASER/PENDBASER updates atomic
> - move vgic_its_get_lpi to vgic.c
> - fix potential race in vgic_add_lpi
> - add comments on puts and gets if not obviously symmetric
> - simplify copy_lpi_list (no get/put anymore)
> - remove vgic_put_irq_locked()
> - remove update_lpi_config_filtered() wrapper
> - take its_lock in vgic_its_handle_command() already
> - move PENDBASER/PROPBASER RES0 masking into sanitise functions
> - properly mask *BASER registers to only support 48 bits
> - cache last read byte in its_sync_lpi_pending_table()
> 
> Changelog v7..v8:
> - rebase on old-VGIC removal patch
> - add missing vgic_put_irq()s
> - check and ensure proper ITS initialisation sequence
> - avoid double collection mapping
> - renaming vits_ function prefixes to vgic_its_
> - properly setup PENDBASER (for new VGIC now)
> - change vgic_v2_probe init order to allow clean exit
> 
> Changelog v6..v7:
> - use kref reference counting
> - remove RCU usage from lpi_list, use spinlock instead
> - copy list of LPIs before accessing guest memory
> - introduce kvm_io_bus_get_dev()
> - refactor parts of arm-gic-v3.h header file
> - provide proper initial values for redistributor and ITS base registers
> - rework sanitisation of base registers
> - rework VGIC MMIO dispatching to differentiate between VGIC parts
> - smaller fixes, also comments and commit messages amended
> 
> Changelog v5..v6:
> - remove its_list from VGIC code
> - add lpi_list and accessor functions
> - introduce reference counting to struct vgic_irq
> - replace its_lock spinlock with its_cmd and its_lock mutexes
> - simplify guest memory accesses (due to the new mutexes)
> - avoid unnecessary affinity updates
> - refine base register address masking
> - introduce sanity checks for PROPBASER and PENDBASER
> - implement BASER<n> registers
> - pass struct vgic_its directly into the MMIO handlers
> - convert KVM_SIGNAL_MSI ioctl into an MMIO write
> - add explicit INIT ioctl to the ITS KVM device
> - adjusting comments and commit messages
> 
> Changelog v4..v5:
> - adapting to final new VGIC (MMIO handlers, etc.)
> - new KVM device to model an ITS, multiple instances allowed
> - move redistributor data into struct vgic_cpu
> - separate distributor and ITS(es)
> - various bug fixes and amended comments after review comments
> 
> Changelog v3..v4:
> - adapting to new VGIC (changes in IRQ injection mechanism)
> 
> Changelog v2..v3:
> - adapt to 4.3-rc and Christoffer's timer rework
> - adapt spin locks on handling PROPBASER/PENDBASER registers
> - rework locking in ITS command handling (dropping dist where needed)
> - only clear LPI pending bit if LPI could actually be queued
> - simplify GICR_CTLR handling
> - properly free ITTEs (including our pending bitmap)
> - fix corner cases with unmapped collections
> - keep retire_lr() around
> - rename vgic_handle_base_register to vgic_reg64_access()
> - use kcalloc instead of kmalloc
> - minor fixes, renames and added comments
> 
> Changelog v1..v2
> - fix issues when using non-ITS GICv3 emulation
> - streamline frame address initialization (new patch 05/15)
> - preallocate buffer memory for reading from guest's memory
> - move locking into the actual command handlers
> -   preallocate memory for new structures if needed
> - use non-atomic __set_bit() and __clear_bit() when under the lock
> - add INT command handler to allow LPI injection from the guest
> - rewrite CWRITER handler to align with new locking scheme
> - remove unneeded CONFIG_HAVE_KVM_MSI #ifdefs
> - check memory table size against our LPI limit (65536 interrupts)
> - observe initial gap of 1024 interrupts in pending table
> - use term "configuration table" to be in line with the spec
> - clarify and extend documentation on API extensions
> - introduce new KVM_CAP_MSI_DEVID capability to advertise device ID requirement
> - update, fix and add many comments
> - minor style changes as requested by reviewers
> 
> ---------------
> 
> The GICv3 ITS (Interrupt Translation Service) is a part of the
> ARM GICv3 interrupt controller [3] used for implementing MSIs.
> It specifies a new kind of interrupts (LPIs), which are mapped to
> establish a connection between a device, its MSI payload value and
> the target processor the IRQ is eventually delivered to.
> In order to allow using MSIs in an ARM64 KVM guest, we emulate this
> ITS widget in the kernel.
> The ITS works by reading commands written by software (from the guest
> in our case) into a (guest allocated) memory region and establishing
> the mapping between a device, the MSI payload and the target CPU.
> We parse these commands and update our internal data structures to
> reflect those changes. On an MSI injection we iterate those
> structures to learn the LPI number we have to inject.
> For the time being we use simple lists to hold the data, this is
> good enough for the small number of entries each of the components
> currently have. Should this become a performance bottleneck in the
> future, those can be extended to arrays or trees if needed.
> 
> Most of the code lives in a separate source file (vgic-its.c), though
> there are some changes necessary in the existing VGIC files.
> 
> For the time being this series gives us the ability to use emulated
> PCI devices that can use MSIs in the guest. Those have to be
> triggered by letting the userland device emulation simulate the MSI
> write with the KVM_SIGNAL_MSI ioctl. This will be translated into
> the proper LPI by the ITS emulation and injected into the guest in
> the usual way (just with a higher IRQ number).
> 
> This series is based on kvmarm/queue and can be found at the
> its-emul/v10 branch of this repository [1].
> For this to be used you need a GICv3 host machine (a fast model would
> do), though it does not rely on any host ITS bits (neither in hardware
> or software).
> 
> To test this you can use the kvmtool patches available in the "its-v6"
> branch here [2].
> Start a guest with: "$ lkvm run --irqchip=gicv3-its --force-pci"
> and see the ITS being used for instance by the virtio devices.
> 
> [1]: git://linux-arm.org/linux-ap.git
>      http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v10
> [2]: git://linux-arm.org/kvmtool.git
>      http://www.linux-arm.org/git?p=kvmtool.git;a=log;h=refs/heads/its-v6
> [3]: http://infocenter.arm.com/help/topic/com.arm.doc.ihi0069b/index.html
> 
> Andre Przywara (17):
>   KVM: arm/arm64: move redistributor kvm_io_devices
>   KVM: arm/arm64: check return value for kvm_register_vgic_device
>   KVM: extend struct kvm_msi to hold a 32-bit device ID
>   KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
>   KVM: kvm_io_bus: add kvm_io_bus_get_dev() call
>   KVM: arm/arm64: VGIC: add refcounting for IRQs
>   irqchip: refactor and add GICv3 definitions
>   KVM: arm64: handle ITS related GICv3 redistributor registers
>   KVM: arm64: introduce ITS emulation file with MMIO framework
>   KVM: arm64: introduce new KVM ITS device
>   KVM: arm64: implement basic ITS register handlers
>   KVM: arm64: connect LPIs to the VGIC emulation
>   KVM: arm64: read initial LPI pending table
>   KVM: arm64: allow updates of LPI configuration table
>   KVM: arm64: implement ITS command queue command handlers
>   KVM: arm64: implement MSI injection in ITS emulation
>   KVM: arm64: enable ITS emulation as a virtual MSI controller
> 
>  Documentation/virtual/kvm/api.txt              |   14 +-
>  Documentation/virtual/kvm/devices/arm-vgic.txt |   25 +-
>  arch/arm/include/asm/kvm_host.h                |    2 +-
>  arch/arm/kvm/arm.c                             |    3 +-
>  arch/arm64/include/asm/kvm_host.h              |    2 +-
>  arch/arm64/include/uapi/asm/kvm.h              |    2 +
>  arch/arm64/kvm/Kconfig                         |    1 +
>  arch/arm64/kvm/Makefile                        |    1 +
>  arch/arm64/kvm/reset.c                         |    8 +-
>  include/kvm/arm_vgic.h                         |   74 +-
>  include/linux/irqchip/arm-gic-v3.h             |  180 ++-
>  include/linux/kvm_host.h                       |    2 +
>  include/uapi/linux/kvm.h                       |    7 +-
>  virt/kvm/arm/vgic/vgic-init.c                  |    9 +-
>  virt/kvm/arm/vgic/vgic-its.c                   | 1484 ++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic-kvm-device.c            |   22 +-
>  virt/kvm/arm/vgic/vgic-mmio-v2.c               |    8 +
>  virt/kvm/arm/vgic/vgic-mmio-v3.c               |  247 +++-
>  virt/kvm/arm/vgic/vgic-mmio.c                  |   62 +-
>  virt/kvm/arm/vgic/vgic-mmio.h                  |   31 +-
>  virt/kvm/arm/vgic/vgic-v2.c                    |   12 +-
>  virt/kvm/arm/vgic/vgic-v3.c                    |   29 +-
>  virt/kvm/arm/vgic/vgic.c                       |  127 +-
>  virt/kvm/arm/vgic/vgic.h                       |   30 +-
>  virt/kvm/kvm_main.c                            |   24 +
>  25 files changed, 2265 insertions(+), 141 deletions(-)
>  create mode 100644 virt/kvm/arm/vgic/vgic-its.c
> 

Tested-by: Eric Auger <eric.auger at redhat.com>

The code was tested on Cavium ThunderX with qemu virtio-net-pci and
vhost-net. Also tested on AMD overdrive with qemu vhost-net.

Cheers

Eric