[RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its domain as parent

[Radovanovic, Aleksandar]

[AMD Official Use Only - General]



> -----Original Message-----
> From: Jason Gunthorpe <jgg at ziepe.ca>
> Sent: 12 October 2022 15:38
> To: Radovanovic, Aleksandar <aleksandar.radovanovic at amd.com>
> Cc: Gupta, Nipun <Nipun.Gupta at amd.com>; Marc Zyngier
> <maz at kernel.org>; Robin Murphy <robin.murphy at arm.com>;
> robh+dt at kernel.org; krzysztof.kozlowski+dt at linaro.org;
> gregkh at linuxfoundation.org; rafael at kernel.org; eric.auger at redhat.com;
> alex.williamson at redhat.com; cohuck at redhat.com; Gupta, Puneet (DCG-
> ENG) <puneet.gupta at amd.com>; song.bao.hua at hisilicon.com;
> mchehab+huawei at kernel.org; f.fainelli at gmail.com;
> jeffrey.l.hugo at gmail.com; saravanak at google.com;
> Michael.Srba at seznam.cz; mani at kernel.org; yishaih at nvidia.com;
> will at kernel.org; joro at 8bytes.org; masahiroy at kernel.org;
> ndesaulniers at google.com; linux-arm-kernel at lists.infradead.org; linux-
> kbuild at vger.kernel.org; linux-kernel at vger.kernel.org;
> devicetree at vger.kernel.org; kvm at vger.kernel.org; okaya at kernel.org;
> Anand, Harpreet <harpreet.anand at amd.com>; Agarwal, Nikhil
> <nikhil.agarwal at amd.com>; Simek, Michal <michal.simek at amd.com>; git
> (AMD-Xilinx) <git at amd.com>
> Subject: Re: [RFC PATCH v3 4/7] bus/cdx: add cdx-MSI domain with gic-its
> domain as parent
> 
> Caution: This message originated from an External Source. Use proper
> caution when opening attachments, clicking links, or responding.
> 
> 
> On Wed, Oct 12, 2022 at 01:37:54PM +0000, Radovanovic, Aleksandar wrote:
> > > On Wed, Oct 12, 2022 at 10:34:23AM +0000, Radovanovic, Aleksandar
> wrote:
> > >
> > >
> > > > As for GITS_TRANSLATER, we can take up to 4 different IOVAs, which
> > > > limits us to 4 CDX devices (should be sufficient for current HW
> > > > use-cases). Also, it means that the address part must be the same
> > > > for all vectors within a single CDX device. I'm assuming this is
> > > > OK as it is going to be a single interrupt and IOMMU domain anyway.
> > >
> > > This is not at all how MSI is supposed to work.
> >
> > In the general case, no, they're not.
> 
> I don't mean that you can hack this to work - I mean that in MSI the
> addr/data is supposed to come from the end point itself, not from some kind
> of shared structure. This is important because the actual act of generating
> the write has to be coherent with the DMA the device is doing, as the MSI
> write must push any DMA data to visibility to meet the "producer /
> consumer" model.
> 

I'm not sure I follow your argument, the limitation here is that the MSI
address value is shared between vectors of the same device (requester id
or endpoint, whichever way you prefer to call it), not between devices. 
This in no way implies that it is unordered with respect to device DMA - 
it is ordered and takes the same AXI path into the CPU cluster, so the
producer/consumer semantics are preserved.

Thanks,
Aleksandar