ARM64: Question: How to map non-shareable memory

[Ard Biesheuvel]

(cc some folks that work on arm64 arch stuff)

Hello David,

On Wed, 24 May 2023 at 23:08, David Clear <dclear at amd.com> wrote:
>
> I'd like some advice on how a device driver could map normal, cacheable,
> non-shareable memory (currently not supported in the kernel).
>
> I have a device that contains areas of RAM and other internal memories
> that are outside of the coherency system, and it's a hardware requirement
> in this device that cacheable memory transactions to these areas be
> marked as non-shareable.
>
> In practical terms this means that Device or Normal_NC mappings work
> today, but Normal (cacheable) mappings will see transaction aborts
> (SErrors).
>
> An approach that appears to work is to define a pgprot_nonshared()
> macro in arch/arm64/include/asm/pgtable.h which sets the PTE SH bits to
> zero, and then define an arm64-specific pgprot_modify() that carries
> over the pgprot_nonshared() property, so the PTE changes aren't lost
> by vm_pgprot_modify().
>
> That's a bit low-level, so I wonder if there's a better approach. Ideally
> I'd like a successful patch to be upstreamed, so I hope there's no
> intrinsic resistance to supporting non-shareable mappings.
>

The code changes are rather straight-forward, but unfortunately, that
is not the real problem here.

Non-shareable cacheable mappings are problematic because they are not
covered by the hardware coherency protocol that keeps caches
synchronized between CPUs and cluster-level and system-level caches.
(IOW, accesses to non-shareable mappings will have snooping disabled).

This means that, unless your system only has a single CPU and does not
support cache coherent DMA at all, the cached view of those RAM
regions will go out of sync between CPUs and wrt other coherent
masters, which is probably not what you're after.