[RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

[Radovanovic, Aleksandar]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Marc Zyngier <maz at kernel.org>
> Sent: 07 September 2022 12:17
> To: Jason Gunthorpe <jgg at nvidia.com>
> Cc: Gupta, Nipun <Nipun.Gupta at amd.com>; robh+dt at kernel.org;
> krzysztof.kozlowski+dt at linaro.org; gregkh at linuxfoundation.org;
> rafael at kernel.org; eric.auger at redhat.com; alex.williamson at redhat.com;
> cohuck at redhat.com; Gupta, Puneet (DCG-ENG)
> <puneet.gupta at amd.com>; song.bao.hua at hisilicon.com;
> mchehab+huawei at kernel.org; f.fainelli at gmail.com;
> jeffrey.l.hugo at gmail.com; saravanak at google.com;
> Michael.Srba at seznam.cz; mani at kernel.org; yishaih at nvidia.com;
> robin.murphy at arm.com; will at kernel.org; joro at 8bytes.org;
> masahiroy at kernel.org; ndesaulniers at google.com; linux-arm-
> kernel at lists.infradead.org; linux-kbuild at vger.kernel.org; linux-
> kernel at vger.kernel.org; devicetree at vger.kernel.org; kvm at vger.kernel.org;
> okaya at kernel.org; Anand, Harpreet <harpreet.anand at amd.com>; Agarwal,
> Nikhil <nikhil.agarwal at amd.com>; Simek, Michal <michal.simek at amd.com>;
> Radovanovic, Aleksandar <aleksandar.radovanovic at amd.com>; git (AMD-
> Xilinx) <git at amd.com>
> Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its
> domain as parent
> 
> [CAUTION: External Email]
> 
> On Tue, 06 Sep 2022 18:19:06 +0100,
> Jason Gunthorpe <jgg at nvidia.com> wrote:
> >
> > On Tue, Sep 06, 2022 at 07:17:58PM +0530, Nipun Gupta wrote:
> >
> > > +static void cdx_msi_write_msg(struct irq_data *irq_data,
> > > +                         struct msi_msg *msg) {
> > > +   /*
> > > +    * Do nothing as CDX devices have these pre-populated
> > > +    * in the hardware itself.
> > > +    */
> > > +}
> >
> > Huh?
> >
> > There is no way it can be pre-populated, the addr/data pair,
> > especially on ARM, is completely under SW control.
> 
> There is nothing in the GIC spec that says that.
> 
> > There is some commonly used IOVA base in Linux for the ITS page, but
> > no HW should hardwire that.
> 
> That's not strictly true. It really depends on how this block is integrated, and
> there is a number of existing blocks that know *in HW* how to signal an LPI.
> 
> See, as the canonical example, how the mbigen driver doesn't need to know
> about the address of GITS_TRANSLATER.
> 
> Yes, this messes with translation (the access is downstream of the
> SMMU) if you relied on it to have some isolation, and it has a "black hole"
> effect as nobody can have an IOVA that overlaps with the physical address of
> the GITS_TRANSLATER register.
> 
> But is it illegal as per the architecture? No. It's just stupid.
> 
>         M.
> 
> --
> Without deviation from the norm, progress is not possible.

To give some context, CDX devices are specific to embedded ARM CPUs on the FPGA and a lot of the CDX hardware core is under the control of the system firmware, not the application CPUs. 

That being said, the MSI address is always going to be the GIC GITS_TRANSLATER, which is known to the system firmware, as it is fixed per FPGA platform. At present, we do not allow the application CPU OS to change this - I believe this is for security reasons, but this may or may not be a good idea in general. As Marc mentions, CDX MSI writes are downstream of the SMMU and, if SMMU does not provide identity mapping for GITS_TRANSLATER, then we have a problem and may need to allow the OS to write the address part. However, even if we did, the CDX hardware is limited in that it can only take one GITS_TRANSLATER register target address per system, not per CDX device, nor per MSI vector.

As for the data part (EventID in GIC parlance), this is always going to be the CDX device-relative vector number - I believe this can't be changed, it is a hardware limitation (but I need to double-check). That should be OK, though, as I believe this is exactly what Linux would write anyway, as each CDX device should be in its own IRQ domain (i.e. have its own ITS device table).

The best I can propose is to pass the addr/data info to firmware here, which will then decide what to do with it. At least, it can assert that the values are what the hardware expects and fail loudly if not, rather than having a silently misconfigured system.

Aleksandar