Neophyte questions about PCIe

[David Laight]

From: Mason
> Sent: 14 March 2017 12:06
> On 14/03/2017 11:23, David Laight wrote:
> 
> > Mason wrote:
> >
> >> I'd like to push support for this PCIe controller upstream.
> >>
> >> Is the code I posted on the right track?
> >> Maybe I can post a RFC patch tomorrow?
> >
> > I think you need to resolve the problem of config space (and IO) cycles
> > before the driver can be deemed usable.
> 
> You're alluding to the (unfortunate) muxing of config and mem spaces
> on my controller, where concurrent accesses by two different threads
> would blow the system up.
> 
> You've suggested sending IPIs in the config space accessor, in order
> to prevent other CPUs from starting a mem access. But this doesn't
> help if a mem access is already in flight, AFAIU.
> 
> I fear there is nothing that can be done in SW, short of rewriting
> drivers such that mem space accesses are handled by a driver-specific
> call-back which could take care of all required locking.
> 
> AFAICT, my only (reasonable) option is putting a big fat warning
> in the code, and pray that concurrent accesses never happen.
> (I'll test with a storage stress test on a USB3 drive.)

Since this is a host controller and you want to be able to use standard
PCIe cards (with their standard drivers) you need to fix the hardware
or do some kind of software workaround that is only in the config space
code.
You cannot assume than config space cycles won't happen at the same
time as other accesses.

While most driver accesses should use either the iowrite32() or writel()
families of functions I'm pretty sure they don't have inter-cpu locking.
(They might have annoying, excessive barriers.)
But it is also legal for drivers to allow pcie memory space be mmapped
directly into a processes address space.
You have no control over the bus cycles that generates.
This isn't common, but is done. We use it for bit-banging serial eeproms.

	David