ARM64: Question: How to map non-shareable memory

[David Clear]

On Wed, 24 May 2023 at 23:59, Ard Biesheuvel <ardb at kernel.org> wrote:
> 
> Non-shareable cacheable mappings are problematic because they are not
> covered by the hardware coherency protocol that keeps caches
> synchronized between CPUs and cluster-level and system-level caches.
> (IOW, accesses to non-shareable mappings will have snooping disabled).
>
> This means that, unless your system only has a single CPU and does not
> support cache coherent DMA at all, the cached view of those RAM
> regions will go out of sync between CPUs and wrt other coherent
> masters, which is probably not what you're after.

Hi Ard. Thanks for the quick reply.

I understand your concerns. The general Linux memory within the
(multi-cluster) system is fully coherent, and there are no surprises
w.r.t normal SMP system operation and device DMA.

The non-coherent memories are outside of the general Linux pool, owned
by autonomous hardware units, and are used for product-specific purposes.
These memories are either internal to the units (far away from coherence
machinery) or purposefully avoid the system coherency controllers so as
to not incur the latecy tax in back-to-back dependent transactions. In
this product it would be a significant performance burden to maintain
coherence with ARM caches that are essentially nothing to do with these
unit's operations.

For the userspace software that needs to access this memory, the current
non-cached mapping is obtained via a device driver and the goal is
to minimize the number of discrete memory transactions by supporting
cached burst-reads and burst-writes, bracketed with appropriate cache
maintenance ops. There are already private caches within the hardware
pipelines that software needs to be explicitly flush or invalidate,
so this is just one more thing.

All that as prelude, it doesn't sound like you're objecting to the
notion of supporting a non-shared mapping, more just asking why, and I
hope I've given some insight into that.

Regards,
David.