PCIe host controller behind IOMMU on ARM

[Phil Edworthy]

On 13 November 2015 14:00, Arnd Bergmann wrote:
> On Friday 13 November 2015 13:03:11 Phil Edworthy wrote:
> >
> > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB
> before
> > > > calling of_pci_dma_configure(). I assume it does this on the basis that this is
> a
> > > > good default for PCI drivers that don't call dma_set_mask().
> > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll
> hit
> > > > this mask.
> > >
> > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure()
> > > does this before calling arch_setup_dma_ops(). The PCI devices start out
> > > with the 32-bit mask, but the limit should be whatever PCI host uses.
> > Ok, so of_dma_configure() could walk up the tree and restrict the dma
> > mask to whatever parents limit it to. Then it could be overridden by
> > a dma-ranges entry in the DT node, right?
> 
> No, the dma-ranges properties tell you what the allowed masks are,
> this is what of_dma_configure() looks at.
Ok, I understand now.


> > If so, one problem I can see is PCI controllers already use the
> > dma-ranges binding but with 3 address cells since it also specifies
> > the PCI address range.
> >
> > I noticed that of_dma_get_range() skips straight to the parent node.
> > Shouldn't it attempt to get the dma-ranges for the device's node
> > first?
> 
> No, the dma-ranges explain the capabilities of the bus, this is
> what you have to look at. The device itself may have additional
> restrictions, but those are what the driver knows based on the
> compatibility value when it passes the device specific mask into
> dma_set_mask()
Ok, this is making sense now.


> > I mean most hardware is limited by the peripheral's
> > capabilities, not the bus. If fact, of_dma_get_range() gets the number
> > of address and size cells from the device node, but gets the dma-ranges
> > from the parent. That seems a little odd to me.
> 
> of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get
> the #address-cells/#size-cells property from the parent device (except
> for the root, which is special).
Right, I should have checked what of_n_addr/size_cell() actually did.


> > The only other problem I can see is that currently all PCI drivers can
> > try to set their dma mask to 64 bits. At the moment that succeeds
> > because there are no checks.
> 
> Right, this is the main bug we need to fix.
Yep.


> > Until devices using them have their DTs
> > updated with dma-ranges, we would be limiting them to a 32 bit mask. I
> > guess that's not much of an issue in practice.
> 
> Correct. I've tried to tell everyone about this when they added device
> nodes for DMA capable devices. In most cases, they want 32-bit masks
> anyway.

Thanks for your help & patience, much appreciated!
Phil