[PATCH v6 01/24] KVM: arm/arm64: Add ITS save/restore API documentation
Christoffer Dall
cdall at linaro.org
Thu May 4 09:52:39 PDT 2017
On Thu, May 04, 2017 at 04:50:23PM +0200, Auger Eric wrote:
> Hi,
>
> On 04/05/2017 15:23, Christoffer Dall wrote:
> > On Thu, May 04, 2017 at 01:44:21PM +0200, Eric Auger wrote:
> >> Add description for how to access ITS registers and how to save/restore
> >> ITS tables into/from memory.
> >>
> >> Signed-off-by: Eric Auger <eric.auger at redhat.com>
> >>
> >> ---
> >> v5 -> v6:
> >> - add restoration ordering
> >> - 256B -> 256 Byte aligned
> >> - DTE Size is number of bits for the EVENTID
> >> - s/GITS_READR/GITS_CREADR
> >>
> >> v4 -> v5:
> >> - take into account Christoffer's comments
> >> - pending table save on GICV3 side now
> >>
> >> v3 -> v4:
> >> - take into account Peter's comments:
> >> - typos
> >> - KVM_DEV_ARM_VGIC_GRP_ITS_TABLES kvm_device_attr = 0
> >> - add a validity bit in DTE
> >> - document all fields in CTE and ITE
> >> - document ABI revision
> >> - take into account Andre's comments:
> >> - document restrictions about GITS_CREADR writing and GITS_IIDR
> >> - document -EBUSY error if one or more VCPUS are runnning
> >> - document 64b registers only can be accessed with 64b access
> >> - itt_addr field matches bits [51:8] of the itt_addr
> >>
> >> v1 -> v2:
> >> - DTE and ITE now are 8 bytes
> >> - DTE and ITE now indexed by deviceid/eventid
> >> - use ITE name instead of ITTE
> >> - mentions ITT_addr matches bits [51:8] of the actual address
> >> - mentions LE layout
> >> ---
> >> Documentation/virtual/kvm/devices/arm-vgic-its.txt | 117 +++++++++++++++++++++
> >> 1 file changed, 117 insertions(+)
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/arm-vgic-its.txt b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> index 6081a5b..863c359 100644
> >> --- a/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> +++ b/Documentation/virtual/kvm/devices/arm-vgic-its.txt
> >> @@ -32,7 +32,124 @@ Groups:
> >> KVM_DEV_ARM_VGIC_CTRL_INIT
> >> request the initialization of the ITS, no additional parameter in
> >> kvm_device_attr.addr.
> >> +
> >> + KVM_DEV_ARM_ITS_SAVE_TABLES
> >> + save the ITS table data into guest RAM, at the location provisioned
> >> + by the guest in corresponding registers/table entries.
> >> +
> >> + The layout of the tables in guest memory defines an ABI. The entries
> >> + are laid out in little endian format as described in the last paragraph.
> >> +
> >> + KVM_DEV_ARM_ITS_RESTORE_TABLES
> >> + restore the ITS tables from guest RAM to ITS internal structures.
> >> +
> >> + The GICV3 must be restored before the ITS and all ITS registers but
> >> + the GITS_CTLR must be restored before restoring the ITS tables.
> >> +
> >> + The GITS_IIDR read-only register must also be restored before
> >> + the table restore as the IIDR revision field encodes the ABI revision.
> >
> > the table restore? "before calling KVM_DEV_ARM_ITS_RESTORE_TABLES" ?
> > (there's a potential confusion between calling this ioctl and simply
> > restoring the memory containing the tables).
> OK
> >
> >> +
> >> + The expected ordering when restoring the GICv3/ITS is described in section
> >> + "ITS Restoration Sequence".
> >> +
> >> Errors:
> >> -ENXIO: ITS not properly configured as required prior to setting
> >> this attribute
> >> -ENOMEM: Memory shortage when allocating ITS internal data
> >> + -EINVAL: Inconsistent restored data
> >> + -EFAULT: Invalid guest ram access
> >> + -EBUSY: One or more VCPUS are running
> >> +
> >> + KVM_DEV_ARM_VGIC_GRP_ITS_REGS
> >> + Attributes:
> >> + The attr field of kvm_device_attr encodes the offset of the
> >> + ITS register, relative to the ITS control frame base address
> >> + (ITS_base).
> >> +
> >> + kvm_device_attr.addr points to a __u64 value whatever the width
> >> + of the addressed register (32/64 bits). 64 bit registers can only
> >> + be accessed with full length.
> >> +
> >> + Writes to read-only registers are ignored by the kernel except for:
> >> + - GITS_CREADR. It needs to be restored otherwise commands in the queue
> >> + will be re-executed after restoring CWRITER. GITS_CREADR must be
> >> + restored before restoring the GITS_CTLR which is likely to enable the
> >> + ITS. Also it needs to be restored after GITS_CBASER since a write to
> >> + GITS_CBASER resets GITS_CREADR.
> >
> > uber nit: s/needs to be/must be/g in all occurences above.
> OK
> >
> >> + - GITS_IIDR. Its Revision field encodes the table layout ABI revision.
> >
> > s/Its/The/
> OK. Argh, I meant its (~his) ;-)
You wrote it correctly, it's just that it can be misleading, so I'm
really in the business of beutifying the ABI text here :)
> > the revision of the table layout ABI.
> >
> >> + In the future we might implement direct injection of virtual LPIS.
> >
> > LPIs
> OK
> >
> >> + This will require an upgrade of the table layout and an evolution of
> >> + the ABI. GITS_IIDR must be restored before the table restoration.
> >
> > before calling KVM_DEV_ARM_ITS_RESTORE_TABLES.
> OK
> >
> >> +
> >> + For other registers, getting or setting a register has the same
> >> + effect as reading/writing the register on real hardware.
> >> + Errors:
> >> + -ENXIO: Offset does not correspond to any supported register
> >> + -EFAULT: Invalid user pointer for attr->addr
> >> + -EINVAL: Offset is not 64-bit aligned
> >> + -EBUSY: one or more VCPUS are running
> >> +
> >> + ITS Restoration Sequence:
> >
> > s/Restoration/restore/
> OK
> >
> >> + -------------------------
> >> +
> >> +The following ordering must be followed when restoring the GIC and the ITS:
> >> +a) restore all guest memory and create vcpus
> >> +b) initialize the GIC and ITS (KVM_DEV_ARM_VGIC_CTRL_INIT)
> >
> > are these really coupled or can you do everything related to the gic
> > first, and then everything related to the ITS afterwards?
> no they aren't.
>
> >
> >> +c) provide GIC and ITS base addresses (KVM_DEV_ARM_VGIC_GRP_ADDR)
> >
> > same
> no they aren't
>
>
> >
> >> +d) restore all redistributors
> >> +e) restore the ITS in the following order:
> >> + 1. Restore GITS_CBASER
> >> + 2. Restore all other GITS_ registers, except GITS_CTLR!
> >> + 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> >> + 4. Restore GITS_CTLR
> >> +f) start vcpus
> >
> > I would omit (f) from here.
>
> In that case I think it would make sense to describe the restore
> sequence for GICV3 RDIST in GICV3 device doc and not here. This would
> alleviate this very description.
>
> restoring RDIST regs require a previous GICV3 KVM_DEV_ARM_VGIC_CTRL_INIT
> and KVM_DEV_ARM_VGIC_GRP_ADDR I think.
>
> So description here would become
>
> "
> a) restore all guest memory and create vcpus
> b) restore all redistributors
> c) initialize the ITS and then provide its base address
> (KVM_DEV_ARM_VGIC_CTRL_INIT, KVM_DEV_ARM_VGIC_GRP_ADDR)
> c) restore the ITS in the following order:
> 1. Restore GITS_CBASER
> 2. Restore all other GITS_ registers, except GITS_CTLR!
> 3. Load the ITS table data (KVM_DEV_ARM_ITS_RESTORE_TABLES)
> 4. Restore GITS_CTLR
>
> Then vcpus can be started.
> "
>
> What do you think?
Yes, looks good.
>
>
> >
> >> +
> >> + ITS Table ABI REV0:
> >> + -------------------
> >> +
> >> + Revision 0 of the ABI only supports physical LPIs.
> >> +
> >> + The device table and ITT are indexed by the deviceid and eventid,
> >> + respectively. The collection table is not indexed by collectionid:
> >> + CTE are written in the table in the order of collection creation. All
> >
> > nit: CTEs
> OK
> >
> >> + entries are 8 bytes.
> >> +
> >> + Device Table Entry (DTE):
> >> +
> >> + bits: | 63| 62 ... 49 | 48 ... 5 | 4 ... 0 |
> >> + values: | V | next | ITT_addr | Size |
> >> +
> >> + where;
> >> + - V indicates whether the entry is valid. If not, other fields
> >> + are not meaningful.
> >> + - next: equals to 0 if this entry is the last one; otherwise it
> >> + corresponds to the deviceid offset to the next DTE, capped by
> >> + 2^14 -1.
> >> + - ITT_addr matches bits [51:8] of the ITT address (256 Byte aligned).
> >> + - Size specifies the supported number of bits for the eventid,
> >> + minus one
> >> +
> >> + Collection Table Entry (CTE):
> >> +
> >> + bits: | 63| 62 .. 52 | 51 ... 16 | 15 ... 0 |
> >> + values: | V | RES0 | RDBase | ICID |
> >> +
> >> + where:
> >> + - V indicates whether the entry is valid. If not, other fields are
> >> + not meaningful.
> >> + - RES0: reserved field with Should-Be-Zero-or-Preserved behavior.
> >> + - RDBase is the PE number (GICR_TYPER.Processor_Number semantic),
> >> + - ICID is the collection ID
> >> +
> >> + Interrupt Translation Entry (ITE):
> >> +
> >> + bits: | 63 ... 48 | 47 ... 16 | 15 ... 0 |
> >> + values: | next | pINTID | ICID |
> >> +
> >> + where:
> >> + - next: equals to 0 if this entry is the last one; otherwise it corresponds
> >> + to the eventid offset to the next ITE capped by 2^16 -1.
> >> + - pINTID is the physical LPI ID; if zero, it means the entry is not valid
> >> + and other fields are not meaningful.
> >> + - ICID is the collection ID
> >> +
> >> --
> >> 2.5.5
> >>
> >
> > The only outstanding issue here is the restore order part. I don't wont
> > to specify a stronger order than absolutely necessary here.
> see above
>
Yep, I think we're there.
Thanks,
-Christoffer
More information about the linux-arm-kernel
mailing list