PCIe host controller behind IOMMU on ARM

[Arnd Bergmann]

On Friday 13 November 2015 13:03:11 Phil Edworthy wrote:
> 
> > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before
> > > calling of_pci_dma_configure(). I assume it does this on the basis that this is a
> > > good default for PCI drivers that don't call dma_set_mask().
> > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit
> > > this mask.
> > 
> > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure()
> > does this before calling arch_setup_dma_ops(). The PCI devices start out
> > with the 32-bit mask, but the limit should be whatever PCI host uses.
> Ok, so of_dma_configure() could walk up the tree and restrict the dma
> mask to whatever parents limit it to. Then it could be overridden by
> a dma-ranges entry in the DT node, right?

No, the dma-ranges properties tell you what the allowed masks are,
this is what of_dma_configure() looks at.

> If so, one problem I can see is PCI controllers already use the
> dma-ranges binding but with 3 address cells since it also specifies
> the PCI address range.
> 
> I noticed that of_dma_get_range() skips straight to the parent node.
> Shouldn't it attempt to get the dma-ranges for the device's node
> first?

No, the dma-ranges explain the capabilities of the bus, this is
what you have to look at. The device itself may have additional
restrictions, but those are what the driver knows based on the
compatibility value when it passes the device specific mask into
dma_set_mask()

> I mean most hardware is limited by the peripheral's
> capabilities, not the bus. If fact, of_dma_get_range() gets the number
> of address and size cells from the device node, but gets the dma-ranges
> from the parent. That seems a little odd to me.

of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get
the #address-cells/#size-cells property from the parent device (except
for the root, which is special).

> The only other problem I can see is that currently all PCI drivers can
> try to set their dma mask to 64 bits. At the moment that succeeds
> because there are no checks.

Right, this is the main bug we need to fix.

> Until devices using them have their DTs
> updated with dma-ranges, we would be limiting them to a 32 bit mask. I
> guess that's not much of an issue in practice.

Correct. I've tried to tell everyone about this when they added device
nodes for DMA capable devices. In most cases, they want 32-bit masks
anyway.

	Arnd