[PATCH 4/5] iommu/arm-smmu-v3: Add set_dirty_tracking() support

[Shameerali Kolothum Thodi]

> -----Original Message-----
> From: Jason Gunthorpe <jgg at nvidia.com>
> Sent: Thursday, November 30, 2023 12:54 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi at huawei.com>
> Cc: iommu at lists.linux.dev; linux-arm-kernel at lists.infradead.org;
> robin.murphy at arm.com; will at kernel.org; joro at 8bytes.org;
> kevin.tian at intel.com; nicolinc at nvidia.com; mshavit at google.com;
> eric.auger at redhat.com; joao.m.martins at oracle.com; jiangkunkun
> <jiangkunkun at huawei.com>; zhukeqian <zhukeqian1 at huawei.com>;
> Linuxarm <linuxarm at huawei.com>
> Subject: Re: [PATCH 4/5] iommu/arm-smmu-v3: Add set_dirty_tracking()
> support
> 
> On Thu, Nov 30, 2023 at 08:56:32AM +0000, Shameerali Kolothum Thodi
> wrote:
> > > On Tue, Nov 28, 2023 at 09:49:39AM +0000, Shameer Kolothum wrote:
> > > > From: Joao Martins <joao.m.martins at oracle.com>
> > > >
> > > > Dirty tracking will always be enabled with DBM=1 modifier enabled
> > > > by default when HD is supported.
> > >
> > > Is this trying to say that ARM doesn't have a per-table global enable
> > > for dirty tracking but instead pre-sets the DBM bit to avoid the cost?
> > >
> >
> > Yes. SMMUv3 has per-PTE DBM control and I think the initial RFC had
> > it walking the PTEs and setting the DBM on set_dirty_tracking().
> 
> set_dirty_tracking doesn't have access to the necessary locking to
> touch the PTEs.
> 
> > > So on smmuv3 to enable we have to clear everything and disable
> > > continues to pay a penalty since we don't go and mark all things as
> > > dirty again?
> >
> > Yes we clear everything on enable. Sorry I didn't get the second part.
> > We don't mark dirty on disable. How is that different from Intel/AMD?
> 
> Intel/AMD have a global switch so they just turn off the tracking and
> stop paying the cost.
> 
> This approach on ARM means once the tracking is logically turned off
> the HW will continue to generate memory traffic to set dirty bits on
> DMAs. There is no way to back to the at-start state where their is 0
> memory traffic on DMAs. Not sure that it totally matters, but it is
> worth noting someplace.

Ah..you meant the HW penalty of keeping it on always.

> If we do want to solve this then ARM would need iommufd to make a pass
> over the page table to set for disable similar to how we have to clear
> for enable.

I will try to see if there is any overhead in keeping it on. Also we could limit
turning it on for DOMAIN_NESTED ?

Thanks,
Shameer