[PATCH] arm64: Enable PCI write-combine resources under sysfs

[Lorenzo Pieralisi]

On Tue, Sep 15, 2020 at 09:25:57AM +1000, Benjamin Herrenschmidt wrote:
> On Mon, 2020-09-14 at 19:57 -0300, Jason Gunthorpe wrote:
> > On Tue, Sep 15, 2020 at 08:00:27AM +1000, Benjamin Herrenschmidt wrote:
> > > On Tue, 2020-09-15 at 07:42 +1000, Benjamin Herrenschmidt wrote:
> > > > 
> > > > > which is back to my original question, how do you do DMA using
> > > > > /sys/xx/resources? Why not use VFIO like everything else?
> > > > 
> > > > Note: All this doesnt' change the fact that sys/xx/resources_wc
> > > > exists
> > > > for other archs and I see no reasons so far not to have it on ARM...
> > > 
> > > Also... it looks like VFIO also doesn't provide a way to do WC yet
> > > unfortunately :-(
> > 
> > Yes, but if the driving reason for this patch is because a VFIO user
> > like EFA DPDK is trying to work around VFIO limitations, then I'd say
> > the VFIO mmap should be amended, and not so much worring about sysfs.
> 
> I don't think the two are exclusive.
> 
> > While there is no reason for ARM to not show the sysfs, it really
> > should never be used. Modern kernels in secure boot don't even show
> > it, for instance.
> 
> It's useful for random things, I've used it quite a bit in a previous
> life for things like in-lab hw testing etc...  There's tooling out
> there, esp. in the more 'embedded' side of thing that uses this, I
> don't see a good reason not to provide the same level of functionality.
> 
> So Lorenzo, imho, we should merge the patch.

To sum it up:

(1) RDMA drivers need a new mapping function/attribute to define their
    message push model. Actually the message model is not necessarily related
    to write combining a la x86, so we should probably come up with a better
    and consistent naming. Enabling this patchset may trigger performance
    regressions on mellanox drivers on arm64 - this ought to be addressed.
(2) User-space/passthrough drivers rely on VFIO for BAR mappings. Since
    it looks relevant, WC message model semantics should be introduced
    there, somehow.
(3) Given the above, the sysfs interface can be enabled. I still don't
    see the advantages (other than saying it is there for other arches, ie
    what can you really do with the sysfs mappings - see Jason's VFIO/DMA
    query) but we can do it. Still, I am not happy about the possible
    mellanox regressions - that should be tested/verified before this
    patch is merged IMHO. Do we really need it for v5.10 ?

Thoughts ?

Lorenzo

> As for fixing VFIO, definitely something to revive. The main contention
> point was which "interface" to use to request write combine.
> 
> Let's restart that conversation with the appropriate folks, the last I
> remember, the question was to figure out what interface to provide
> userspace for the functionality.
> 
> Clint, do you want to drive this as well ?
> 
> Cheers,
> Ben.
>