dma_alloc_coherent versus streaming DMA, neither works satisfactory

[Mike Looijmans]

On 08-05-15 15:19, Arnd Bergmann wrote:
> On Friday 08 May 2015 10:31:53 Mike Looijmans wrote:
>> On 08-05-15 09:54, Arnd Bergmann wrote:
>>> On Friday 08 May 2015 07:55:26 Mike Looijmans wrote:
>>>> On 07-05-15 16:30, Russell King - ARM Linux wrote:
>>>>> On Thu, May 07, 2015 at 04:08:54PM +0200, Mike Looijmans wrote:
>>>>>> I read the rest of the thread, apparently it was never integrated.
>>>>>>
>>>>>> The patch for "non-consistent" is a BUG FIX, not some feature request or so.
>>>>>> I was already wondering why my driver had to kalloc pages to get proper
>>>>>> caching on it.
>>>>>
>>>>> I disagree.
>>>>>
>>>>>>    From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
>>>>>> """
>>>>>> DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either
>>>>>> consistent or non-consistent memory as it sees fit.  By using this API,
>>>>>> you are guaranteeing to the platform that you have all the correct and
>>>>>> necessary sync points for this memory in the driver.
>>>>>> """
>>>>>
>>>>> DMA attributes are something that came in _after_ the DMA API had been
>>>>> around for many years.  It's a "new feature" that was added to an
>>>>> existing subsystem, and because there have been no need for it to be
>>>>> implemented on ARM, the new feature was never implemented.
>>>>>
>>>>> More than that, the vast majority of ARM hardware can't provide this
>>>>> kind of memory, and there are _no_ kernel APIs to ensure that if
>>>>
>>>> By "non-coherent" memory I thought it meant the same kind of memory that
>>>> kalloc would return. But from your answer it seems I am mistaken and
>>>> this is something different?
>>>
>>> It depends: on a device that is actually cache-coherent,
>>> dma_alloc_coherent() and dma_alloc_noncoherent() both return normal
>>> memory.
>>>
>>> On some architectures (not ARM) that are not fully coherent,
>>> dma_alloc_coherent() has to return uncached memory, while
>>> dma_alloc_noncoherent() is allowed to return cached memory but
>>> requires a dma_cache_sync() operation.
>>>
>>> dma_alloc_attrs() with DMA_ATTR_NON_CONSISTENT is a variant of that,
>>> but I assume the idea is that you use dma_sync_single_fo_{cpu,device}()
>>> on that memory, which can actually work on  ARM, unlike dma_cache_sync().
>>
>> Ah, okay, I was misled by the names. I was under the impression that memory
>> would be either "coherent" or "non-coherent". But what is called
>> "non-coherent" here is actually something like "less-coherent", it isn't
>> normal memory as alloc_pages would return, but it also isn't completely
>> coherent. Is that a correct summary?
>>
>> In that case, I stand corrected.
>
> Almost. I think the part that you are still missing is that memory
> itself it not coherent or non-coherent. It's the device access to
> that memory that can be coherent or not with regard to the CPU.
>
> The memory that is returned by alloc_pages can be coherent with one
> device but non-coherent with another.

Yeah, I did understand, but never made that explicit.

The Zynq is even more funky than most hardware - I can actually enable/disable 
(hardware) coherency for each memory burst. If I wanted to. Currently I just 
turn it on/off for the whole system at boot time, since I need to write its 
coherency status into the devicetree for the kernel to understand how my 
device accesses memory.

>> I was looking for an interface that would allocate memory for access by my
>> device, but that would be just alloc_pages style memory. If my DMA controller
>> is limited to say only the first GB of RAM, I'd set the DMA mask to "30 bits".
>> If I just allocate memory using alloc_pages, the kernel doesn't know that I'd
>> want it to be in the lower 1GB range, and could allocate it in a spot my
>> device could not map.
>
> If you have that low-memory restriction, you also need to ensure that there
> is a ZONE_DMA that is large enough. ZONE_DMA should be sized to match the
> common subset that all devices can access, so a GFP_DMA request returns
> memory that is guaranteed to be accessible by all devices.

GFP_DMA still carries that scary remark: "on x86, GFP_DMA guarantees to be 
within the first 16MB of available bus addresses". Which is really scary if 
you want to allocate multimegabyte buffers.

>> Hence I'd expect there to be some "dma_alloc_pages(struct device* ...)" style
>> of call to get memory that my device could access (and I was under the false
>> impression that dma_alloc_noncoherent was the one I was looking for).
>>
>> Currently I can get away with just using alloc_pages or kmalloc since my DMA
>> controller happens to be able to access all memory. But I also want my device
>> driver to work on 64-bit platforms (e.g. arm64 for the MPSOC and x86-64 for
>> the PCIe version of the board).
>
> Those machines will have ZONE_DMA32, which refers to the first 4GB of memory,
> so that should work fine. Alternatively you can use the iommu to get
> all memory mapped into the space that is accessible by the device.

With my first tests on the PC version I ran into a problem with the PCIe 
interface that the IP block that does the address translation would do 
something like addr=base_addr|offset instead of addr=base_addr+offset. Since 
base_addr was assigned dynamically (BAR) the board would fail to work (write 
to the wrong spot in memory, that is) if the BAR was aligned on a 4MB boundary 
and my buffer was on an "odd" 4MB address. I hacked around that by calling 
dma_alloc_coherent() until I got one that was evenly aligned and then get rid 
of the failed ones. Just an anecdote that shows that requirements for buffer 
alignment can be weird. Of course, we software guys are here to clean up the 
mess because fixing the bug in hardware is out of the question.

 > Also, on 64-bit x86 or ARM machines, all memory tends to be coherent, so
 > dma_sync_* will turn into a nop.

Well, that should make things easier, right? I just can't wait until I get 
that first MPSOC on my desk :)


Kind regards,

Mike Looijmans
System Expert

TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans at topicproducts.com
Website: www.topicproducts.com

Please consider the environment before printing this e-mail