Neophyte questions about PCIe

[Bjorn Helgaas]

On Fri, Mar 10, 2017 at 04:05:50PM +0100, Mason wrote:
> On 10/03/2017 15:06, David Laight wrote:
> 
> > Robin Murphy wrote:
> >
> >> On 09/03/17 23:43, Mason wrote:
> >>
> >>> I think I'm making progress, in that I now have a better
> >>> idea of what I don't understand. So I'm able to ask
> >>> (hopefully) less vague questions.
> >>>
> >>> Take the USB3 PCIe adapter I've been testing with. At some
> >>> point during init, the XHCI driver request some memory
> >>> (via kmalloc?) in order to exchange data with the host, right?
> >>>
> >>> On my SoC, the RAM used by Linux lives at physical range
> >>> [0x8000_0000, 0x8800_0000[ => 128 MB
> >>>
> >>> How does the XHCI driver make the adapter aware of where
> >>> it can scribble data? The XHCI driver has no notion that
> >>> the device is behind a bus, does it?
> >>>
> >>> At some point, the physical addresses must be converted
> >>> to PCI bus addresses, right? Is it computed subtracting
> >>> the offset defined in the DT?
> > 
> > The driver should call dma_alloc_coherent() which returns both the
> > kernel virtual address and the device (xhci controller) has
> > to use to access it.
> > The cpu physical address is irrelevant (although it might be
> > calculated in the middle somewhere).
> 
> Thank you for that missing piece of the puzzle.
> I see some relevant action in drivers/usb/host/xhci-mem.c
> 
> And I now see this log:
> 
> [    2.499320] xhci_hcd 0000:01:00.0: // Device context base array address = 0x8e07e000 (DMA), d0855000 (virt)
> [    2.509156] xhci_hcd 0000:01:00.0: Allocated command ring at cfb04200
> [    2.515640] xhci_hcd 0000:01:00.0: First segment DMA is 0x8e07f000
> [    2.521863] xhci_hcd 0000:01:00.0: // Setting command ring address to 0x20
> [    2.528786] xhci_hcd 0000:01:00.0: // xHC command ring deq ptr low bits + flags = @00000000
> [    2.537188] xhci_hcd 0000:01:00.0: // xHC command ring deq ptr high bits = @00000000
> [    2.545002] xhci_hcd 0000:01:00.0: // Doorbell array is located at offset 0x800 from cap regs base addr
> [    2.554455] xhci_hcd 0000:01:00.0: // xHCI capability registers at d0852000:
> [    2.561550] xhci_hcd 0000:01:00.0: // @d0852000 = 0x1000020 (CAPLENGTH AND HCIVERSION)
> 
> I believe 0x8e07e000 is a CPU address, not a PCI bus address.

Sounds like you've made good progress since this email, but I think
0x8e07e000 is a PCI bus address, not a CPU address.  The code in
xhci_mem_init() is this:

  xhci->dcbaa = dma_alloc_coherent(dev, sizeof(*xhci->dcbaa), &dma, flags);
  xhci->dcbaa->dma = dma;
  xhci_dbg_trace("Device context base array address = 0x%llx (DMA), %p (virt)",
                 (unsigned long long)xhci->dcbaa->dma, xhci->dcbaa);

dma_alloc_coherent() allocates a buffer and returns two values: the
CPU virtual address ("xhci->dcbaa" here) for use by the driver and the
corresponding DMA address the device can use to reach the buffer
("dma").

Documentation/DMA-API-HOWTO.txt has more details that might be useful.