[RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

[Radovanovic, Aleksandar]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Marc Zyngier <maz at kernel.org>
> Sent: 08 September 2022 09:08
> To: Radovanovic, Aleksandar <aleksandar.radovanovic at amd.com>
> Cc: Jason Gunthorpe <jgg at nvidia.com>; Gupta, Nipun
> <Nipun.Gupta at amd.com>; robh+dt at kernel.org;
> krzysztof.kozlowski+dt at linaro.org; gregkh at linuxfoundation.org;
> rafael at kernel.org; eric.auger at redhat.com; alex.williamson at redhat.com;
> cohuck at redhat.com; Gupta, Puneet (DCG-ENG)
> <puneet.gupta at amd.com>; song.bao.hua at hisilicon.com;
> mchehab+huawei at kernel.org; f.fainelli at gmail.com;
> jeffrey.l.hugo at gmail.com; saravanak at google.com;
> Michael.Srba at seznam.cz; mani at kernel.org; yishaih at nvidia.com;
> robin.murphy at arm.com; will at kernel.org; joro at 8bytes.org;
> masahiroy at kernel.org; ndesaulniers at google.com; linux-arm-
> kernel at lists.infradead.org; linux-kbuild at vger.kernel.org; linux-
> kernel at vger.kernel.org; devicetree at vger.kernel.org; kvm at vger.kernel.org;
> okaya at kernel.org; Anand, Harpreet <harpreet.anand at amd.com>; Agarwal,
> Nikhil <nikhil.agarwal at amd.com>; Simek, Michal <michal.simek at amd.com>;
> git (AMD-Xilinx) <git at amd.com>
> Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its
> domain as parent
> 
> [CAUTION: External Email]
>  
> OK, so you definitely need a mapping, but it cannot be a translation, and it
> needs to be in all the possible address spaces. OMG.

Could you elaborate why it needs to be in all the possible address spaces? I'm in no way familiar with kernel IOVA allocation, so not sure I understand this requirement. Note that each CDX device will have its own unique StreamID (in general case, equal to DeviceID sent to the GIC), so, from a SMMU perspective, the mapping can be specific to that device. As long as that IOVA is not allocated to any DMA region for _that_ device, things should be OK? But, I appreciate it might not be that simple from a kernel perspective.

> > > > As for the data part (EventID in GIC parlance), this is always
> > > > going to be the CDX device-relative vector number - I believe this
> > > > can't be changed, it is a hardware limitation (but I need to double-
> check).
> > > > That should be OK, though, as I believe this is exactly what Linux
> > > > would write anyway, as each CDX device should be in its own IRQ
> > > > domain (i.e. have its own ITS device table).
> > >
> > > But that's really the worse part. You have hardcoded what is the
> > > *current* Linux behaviour. Things change. And baking SW behaviour
> > > into a piece of HW looks incredibly shortsighted...
> >
> > For posterity, I'm not an RTL designer/architect, so share your
> > sentiment to a certain extent. That said, I expect the decision was
> > not based on Linux or any other SW behaviour, but because it is the
> > most straightforward and least expensive way to do it. Having an
> > EventID register for each and every MSI source just so you can program
> > it in any random order costs flops and all the associated complexity
> > of extra RTL logic (think timing closure, etc.), so trade-offs are
> > made. The fact that it matches current Linux behaviour means we just
> > got lucky...
> 
> Yeah, but that's not the only problem: there is no guarantee that we have
> enough LPIs to allocate for the device, so we'll perform a partial allocation (8
> instead of 32 LPIs, for example).

Why should that be a problem? The driver will know in advance the number of vectors required by the device. I expect it will need to call some equivalent of  platform_msi_domain_alloc_irqs(), shouldn't that fail if not enough IRQs are allocated (and ultimately fail the probe)? Even if not, we can still inform the firmware in write_msg, which will serve as an indication that a particular vector is enabled. The firmware can decide what to do with the device if not all of the vectors are enabled.

Aleksandar