[RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

[Radovanovic, Aleksandar]

[AMD Official Use Only - General]

> -----Original Message-----
> From: Gupta, Nipun <Nipun.Gupta at amd.com>
> Sent: 12 October 2022 11:04
> To: Marc Zyngier <maz at kernel.org>; Robin Murphy
> <robin.murphy at arm.com>
> Cc: robh+dt at kernel.org; krzysztof.kozlowski+dt at linaro.org;
> gregkh at linuxfoundation.org; rafael at kernel.org; eric.auger at redhat.com;
> alex.williamson at redhat.com; cohuck at redhat.com; Gupta, Puneet (DCG-
> ENG) <puneet.gupta at amd.com>; song.bao.hua at hisilicon.com;
> mchehab+huawei at kernel.org; f.fainelli at gmail.com;
> jeffrey.l.hugo at gmail.com; saravanak at google.com;
> Michael.Srba at seznam.cz; mani at kernel.org; yishaih at nvidia.com;
> jgg at ziepe.ca; jgg at nvidia.com; will at kernel.org; joro at 8bytes.org;
> masahiroy at kernel.org; ndesaulniers at google.com; linux-arm-
> kernel at lists.infradead.org; linux-kbuild at vger.kernel.org; linux-
> kernel at vger.kernel.org; devicetree at vger.kernel.org; kvm at vger.kernel.org;
> okaya at kernel.org; Anand, Harpreet <harpreet.anand at amd.com>; Agarwal,
> Nikhil <nikhil.agarwal at amd.com>; Simek, Michal <michal.simek at amd.com>;
> Radovanovic, Aleksandar <aleksandar.radovanovic at amd.com>; git (AMD-
> Xilinx) <git at amd.com>
> Subject: RE: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its
> domain as parent
> 
> [AMD Official Use Only - General]
> 
> 
> <snip>
> 
> >
> > > +}
> > > +
> > > +static void cdx_msi_write_msg(struct irq_data *irq_data,
> > > +                           struct msi_msg *msg) {
> > > +     /*
> > > +      * Do nothing as CDX devices have these pre-populated
> > > +      * in the hardware itself.
> > > +      */
> >
> > We talked about this in a separate thread. This is a major problem.
> 
> We discussed this further with the hardware design team and now have the
> correct and complete understanding here. As the CDX devices are FPGA
> based, they don't exist yet, so it would be possible to construct them in such
> a way that the eventid is programable.
> 
> To make it generic for CDX devices, we have added a firmware API which
> provide the mappings (MSI vector ID to eventID) to the fabric, that can be
> referred by the device while generating the MSI interrupt.
> 
> Also, there is an existing table to have GITS_TRANSLATOR iova address
> (address in
> msi_msg) for CDX devices, which can be programmed by the firmware. So,
> providing IOVA address to device would also not be a problem here.
> 
> We would be rolling out RFC v4 with these changes soon.
> 
> Regards,
> Nipun

Just to be clear, there will be some HW limitations with the proposed solution,
so let's just make sure that we're all OK with it.

For the MSI EventID, the HW interrupt logic assumes the MSI write value is 
equal to the MSI vector number. However, the vector number is programmable
for most (all) of the interrupt sources, which isn't exactly the same as saying
EventID is programmable for a vector number, but can be used to emulate the
desired behaviour, with a translation table in firmware. 

The limitation here is that we support at most 16 bits of EventID (and this still
needs to be confirmed for all interrupt sources)

As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which limits us
to 4 CDX devices (should be sufficient for current HW use-cases). Also, it means
that the address part must be the same for all vectors within a single CDX 
device. I'm assuming this is OK as it is going to be a single interrupt and IOMMU
domain anyway.

Thanks,
Aleksandar