[PATCH] arm64: Add architecture support for PCI

[Arnd Bergmann]

On Monday 03 February 2014 16:31:37 Jason Gunthorpe wrote:
> Specifying 'use EHCI, AHCI, etc' - which are all PCI based standards
> without clearly specifying exactly how PCI is suppose to work is
> completely bonkers.
> 
> What is needed is a spec that says:
>  1) Here is how you generate config TLPs. A MMIO region that
>     conforms to the already specified x86 ECAM would
>     be perfect
>  2) Here is a dword by dword break down of the entire config space in
>     a SOC. Here is where a on-board AHCI controller must show up in
>     config space. Here is how an external PCI-E port must show
>     up. Etc. Most of this is already specified, but it clearly needs
>     to be layed out explicitly for ARM SOCs to actually follow it.
>  3) Here is how you specify the aperture(s) associated with PCI BAR's
>     and bridge windows in config space. And yes: The CONFIG SPACE
>     BARS MUST WORK.
>  4) Here is how MSI works, these are the values you put in the
>     address/data and here is how you collect the interrupt.
>  5) Here is how Legacy INTx must be mapped into the GIC.
> 
> This is what x86 does, and they have been doing it well for 10
> years. If you want to play in the server game you have to properly
> implement PCI.

I'm pretty sure the authors of the SBSA actually thought that was
what they wrote, by referring to external specifications (pci-3.0,
ehci, ahci, ...).  However, it seems they were either foolish enough
to believe that hardware designers would follow these specs, or
they were intentionally misled and got talked into putting ambiguous
terminology in because there were already hardware designs that
are not exactly in line with the spirit of the SBSA but can be
argued to be conforming to the text for a lax interpretation.

I think EHCI is a much better example than PCI here, because the
spec has exactly one line to say about it, where it spends a whole
chapter on PCI.

Here is how a sane person would read SBSA to create a compliant
implementation:

  I have to use EHCI version 1.1 to provide USB-2.0 support. EHCI
  is a PCI device, so I'll put it behind a PCIe port that complies
  to the PCIe section of the SBSA. Since EHCI by itself only provides
  high-speed USB, and USB-2.0 mandates I provide low-speed and
  full-speed as well, I have to add a USB hub device. It would have
  been easier to just use OHCI for these, but SBSA says I can't.
  Now I want to integrate the EHCI into my SoC and not waste one
  of my precious PCIe root ports, so I have to create another PCI
  domain with its own ECAM compliant config space to put it into.
  Fortunately SBSA lets me add an arbitrary number of PCI domains,
  as long as they are all strictly compliant. To software it will
  look exactly as if it was on an external card, I just have to
  ensure the boot loader correctly sets up the clocks and the phy
  before an SBSA compliant OS gets loaded, all runtime power
  management will get handled through the EHCI-1.1 energy-efficiency
  extensions.

Here is how a crazy person would read the same sentence in the SBSA:

  I have an IP block that implements the EHCI register set, that
  should be good enough. It's not a fast device, so I can put it
  on a non-coherent bus. Since the SoC will be used for networking,
  I'll put the registers into big-endian configuration to make it
  easier for the OS to access. I'm not allowed to have USB-1.1
  according to SBSA, so I can get away without a hub or an extra
  OHCI. I can't support MSI because it's not a PCI device, and
  the GIC is pretty full, so I'll just connect the IRQ line to
  the GPIO controller. In order to do better power management,
  I'll design a fancy PHY that the device driver will manage
  for implementing autosuspend. I should also give the OS
  fine-grained control over the clocks, but it will have to share
  the clock domain with the other devices on the same edge of the
  chip. The EHCI device is not part of PCI, which measn I don't
  have to use the standard SMMU. However, my EHCI implementation
  can only do 32-bit DMA, and I'll have to design my own IOMMU
  to let it access the entire memory range. USB-OTG is a great
  feature and we already paid for having this in our EHCI
  implementation, better make sure it comes up in endpoint mode
  after waking up from power saving.

	Arnd