[PATCH 5/5] cxl/region: Manage CPU caches relative to DPA invalidation events

[Jonathan Cameron]

On Mon, 5 Dec 2022 12:10:22 -0800
Dan Williams <dan.j.williams at intel.com> wrote:

> [ add linux-arm-kernel at lists.infradead.org ]
> 
> Background for ARM folks, CXL can dynamically reconfigure the target
> devices that back a given physical memory region. When that happens the
> CPU cache can be holding cache data from a previous configuration. The
> mitigation for that scenario on x86 is wbinvd, ARM does not have an
> equivalent. The result, dynamic region creation is disabled on ARM. In
> the near term, most CXL is configured pre-boot, but going forward this
> restriction is untenable.
> 
> Davidlohr Bueso wrote:
> > On Thu, 01 Dec 2022, Dan Williams wrote:
> >   
> > >A "DPA invalidation event" is any scenario where the contents of a DPA
> > >(Device Physical Address) is modified in a way that is incoherent with
> > >CPU caches, or if the HPA (Host Physical Address) to DPA association
> > >changes due to a remapping event.
> > >
> > >PMEM security events like Unlock and Passphrase Secure Erase already
> > >manage caches through LIBNVDIMM,  
> > 
> > Just to be clear, is this is why you get rid of the explicit flushing
> > for the respective commands in security.c?  
> 
> Correct, because those commands can only be executed through libnvdimm.
> 
> >   
> > >so that leaves HPA to DPA remap events
> > >that need cache management by the CXL core. Those only happen when the
> > >boot time CXL configuration has changed. That event occurs when
> > >userspace attaches an endpoint decoder to a region configuration, and
> > >that region is subsequently activated.
> > >
> > >The implications of not invalidating caches between remap events is that
> > >reads from the region at different points in time may return different
> > >results due to stale cached data from the previous HPA to DPA mapping.
> > >Without a guarantee that the region contents after cxl_region_probe()
> > >are written before being read (a layering-violation assumption that
> > >cxl_region_probe() can not make) the CXL subsystem needs to ensure that
> > >reads that precede writes see consistent results.  
> > 
> > Hmm where does this leave us remaping under arm64 which is doesn't have
> > ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION?
> > 
> > Back when we were discussing this it was all related to the security stuff,
> > which under arm it could just be easily discarded as not available feature.  
> 
> I can throw out a few strawman options, but really need help from ARM
> folks to decide where to go next.

+Cc bunch of relevant people. There are discussions underway but I'm not sure
anyone will want to give more details here yet.

> 
> 1/ Map and loop cache flushing line by line. It works, but for Terabytes
>    of CXL the cost is 10s of seconds of latency to reconfigure a region.
>    That said, region configuration, outside of test scenarios, is typically
>    a "once per bare metal provisioning" event.
> 
> 2/ Set a configuration dependency that mandates that all CXL memory be
>    routed through the page allocator where it is guaranteed that the memory
>    will be written (zeroed) before use. This restricts some planned use
>    cases for the "Dynamic Capacity Device" capability.

This is the only case that's really a problem (to my mind) I hope we will have
a more general solution before there is much hardware out there, particularly
where sharing is involved. 

> 
> 3/ Work with the CXL consortium to extend the back-invalidate concept
>    for general purpose usage to make devices capable of invalidating caches
>    for a new memory region they joined, and mandate it for ARM. This one
>    has a long lead time and a gap for every device in flight currently.

There are significant disadvantages in doing this that I suspect will mean
this never happens for some classes of device, or is turned off for performance
reasons. For anyone curious, go look at the protocol requirements of back
invalidate in the CXL 3.0 spec.

Jonathan