Neophyte questions about PCIe

[Robin Murphy]

On 13/03/17 10:59, Mason wrote:
> On 10/03/2017 17:00, Robin Murphy wrote:
> 
>> On 10/03/17 15:35, David Laight wrote:
>>
>>> Robin Murphy wrote:
>>>
>>>> The appropriate DT property would be "dma-ranges", i.e.
>>>>
>>>> pci at ... {
>>>> 	...
>>>> 	dma-ranges = <(PCI bus address) (CPU phys address) (size)>;
>>>> }
>>>
>>> Isn't that just saying which physical addresses the cpu can assign
>>> for buffers for those devices?
>>> There is also an offset between the 'cpu physical address' and the
>>> 'dma address'.
>>
>> That offset is inherent in what "dma-ranges" describes. Say (for ease of
>> calculation) that BAR0 has been put at a mem space address of 0x20000000
>> and maps the first 1GB of physical DRAM. That would give us:
>>
>> 	dma-ranges = <0x20000000 0x80000000 0x40000000>;
>>
>> Then a "virt = dma_alloc_coherent(..., &handle, ...)", borrowing the
>> numbers from earlier in the thread, would automatically end up with:
>>
>> 	virt == 0xd0855000;
>> 	handle == 0x2e07e000;
>>
>> (with the physical address of 0x8e07e000 in between being irrelevant to
>> the consuming driver)
>>
>> It is true that the device's DMA mask assignment is also part and parcel
>> of this, whereby we will limit what physical addresses the kernel
>> considers valid for DMA involving devices behind this range to the lower
>> 3GB (i.e. 0x80000000 + 0x40000000 - 1). With a bit of luck,
>> CONFIG_DMABOUNCE should do the rest of the job of working around that
>> where necessary.
> 
> AFAICT, the parser for the "dma-ranges" property is implemented
> in of_dma_get_range() in drivers/of/address.c
> 
> http://lxr.free-electrons.com/source/drivers/of/address.c#L808
> 
> /**
>  * of_dma_get_range - Get DMA range info
>  * @np:		device node to get DMA range info
>  * @dma_addr:	pointer to store initial DMA address of DMA range
>  * @paddr:	pointer to store initial CPU address of DMA range
>  * @size:	pointer to store size of DMA range
>  *
>  * Look in bottom up direction for the first "dma-ranges" property
>  * and parse it.
>  *  dma-ranges format:
>  *	DMA addr (dma_addr)	: naddr cells
>  *	CPU addr (phys_addr_t)	: pna cells
>  *	size			: nsize cells
>  *
>  * It returns -ENODEV if "dma-ranges" property was not found
>  * for this device in DT.
>  */
> 
> I didn't find anything relevant in Documentation/devicetree/bindings
> except Documentation/devicetree/bindings/iommu/iommu.txt but I'm not
> sure this applies to my SoC.
> 
> It's not clear to me how ranges and dma-ranges interact...
> Is it perhaps: ranges for cpu-to-bus, dma-ranges for bus-to-cpu?

That's it exactly. FWIW the in-kernel description can be found in
Documentation/devicetree/booting-without-of.txt, but the
www.devicetree.org spec (which succeeds ePAPR) should be the
authoritative reference now. Note that the property itself is fully
capable of describing multiple discontiguous ranges which might look
like the perfect solution to your multiple-DRAM-banks problem, but Linux
can't actually cope with that at all :(

Robin.

> ePAPR (Version 1.1 -- 08 April) provides a formal definition
> 
> 2.3.9 dma-ranges
> 
> Property: dma-ranges
> 
> Value type: <empty> or <prop-encoded-array> encoded as arbitrary number
> of triplets of (child-bus-address, parent-bus-address, length).
> 
> Description:
> 
> The dma-ranges property is used to describe the direct memory access (DMA) structure of a
> memory-mapped bus whose device tree parent can be accessed from DMA operations
> originating from the bus. It provides a means of defining a mapping or translation between the
> physical address space of the bus and the physical address space of the parent of the bus.
> The format of the value of the dma-ranges property is an arbitrary number of triplets of
> (child-bus-address, parent-bus-address, length). Each triplet specified describes a contiguous
> DMA address range.
> 
> o The child-bus-address is a physical address within the child bus' address space. The
> number of cells to represent the address depends on the bus and can be determined
> from the #address-cells of this node (the node in which the dma-ranges property
> appears).
> 
> o The parent-bus-address is a physical address within the parent bus' address space.
> The number of cells to represent the parent address is bus dependent and can be
> determined from the #address-cells property of the node that defines the parent's
> address space.
> 
> o The length specifies the size of the range in the child's address space. The number of
> cells to represent the size can be determined from the #size-cells of this node (the
> node in which the dma-ranges property appears).
> 
> 
> I'm still digging :-)
> 
> Regards.
>