[PATCHv3] iommu/arm-smmu: Optimize ->tlb_flush_walk() for qcom implementation

[Will Deacon]

On Wed, Jun 23, 2021 at 07:12:01PM +0530, Sai Prakash Ranjan wrote:
> Currently for iommu_unmap() of large scatter-gather list with page size
> elements, the majority of time is spent in flushing of partial walks in
> __arm_lpae_unmap() which is a VA based TLB invalidation invalidating
> page-by-page on iommus like arm-smmu-v2 (TLBIVA).
> 
> For example: to unmap a 32MB scatter-gather list with page size elements
> (8192 entries), there are 16->2MB buffer unmaps based on the pgsize (2MB
> for 4K granule) and each of 2MB will further result in 512 TLBIVAs (2MB/4K)
> resulting in a total of 8192 TLBIVAs (512*16) for 16->2MB causing a huge
> overhead.
> 
> On qcom implementation, there are several performance improvements for
> TLB cache invalidations in HW like wait-for-safe (for realtime clients
> such as camera and display) and few others to allow for cache
> lookups/updates when TLBI is in progress for the same context bank.
> So the cost of over-invalidation is less compared to the unmap latency
> on several usecases like camera which deals with large buffers. So,
> ASID based TLB invalidations (TLBIASID) can be used to invalidate the
> entire context for partial walk flush thereby improving the unmap
> latency.
> 
> Non-strict mode can use this by default for all platforms given its
> all about over-invalidation saving time on individual unmaps and
> non-deterministic generally.
> 
> For this example of 32MB scatter-gather list unmap, this change results
> in just 16 ASID based TLB invalidations (TLBIASIDs) as opposed to 8192
> TLBIVAs thereby increasing the performance of unmaps drastically.
> 
> Test on QTI SM8150 SoC for 10 iterations of iommu_{map_sg}/unmap:
> (average over 10 iterations)
> 
> Before this optimization:
> 
>     size        iommu_map_sg      iommu_unmap
>       4K            2.067 us         1.854 us
>      64K            9.598 us         8.802 us
>       1M          148.890 us       130.718 us
>       2M          305.864 us        67.291 us
>      12M         1793.604 us       390.838 us
>      16M         2386.848 us       518.187 us
>      24M         3563.296 us       775.989 us
>      32M         4747.171 us      1033.364 us
> 
> After this optimization:
> 
>     size        iommu_map_sg      iommu_unmap
>       4K            1.723 us         1.765 us
>      64K            9.880 us         8.869 us
>       1M          155.364 us       135.223 us
>       2M          303.906 us         5.385 us
>      12M         1786.557 us        21.250 us
>      16M         2391.890 us        27.437 us
>      24M         3570.895 us        39.937 us
>      32M         4755.234 us        51.797 us
> 
> This is further reduced once the map/unmap_pages() support gets in which
> will result in just 1 TLBIASID as compared to 16 TLBIASIDs.
> 
> Real world data also shows big difference in unmap performance as below:
> 
> There were reports of camera frame drops because of high overhead in
> iommu unmap without this optimization because of frequent unmaps issued
> by camera of about 100MB/s taking more than 100ms thereby causing frame
> drops.
> 
> Signed-off-by: Sai Prakash Ranjan <saiprakash.ranjan at codeaurora.org>
> ---
> 
> Changes in v3:
>  * Move the logic to arm-smmu driver from io-pgtable (Robin)
>  * Use a new set of iommu_flush_ops->arm_smmu_s1_tlb_impl_ops and use it for qcom impl
> 
> Changes in v2:
>  * Add a quirk to choose tlb_flush_all in partial walk flush
>  * Set the quirk for QTI SoC implementation
> 
> ---
>  drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c | 13 +++++++++++++
>  drivers/iommu/arm/arm-smmu/arm-smmu.c      | 17 ++++++++++++++++-
>  2 files changed, 29 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> index 7771d40176de..218c71465819 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu-qcom.c
> @@ -10,6 +10,8 @@
>  
>  #include "arm-smmu.h"
>  
> +extern const struct iommu_flush_ops arm_smmu_s1_tlb_impl_ops;
> +
>  struct qcom_smmu {
>  	struct arm_smmu_device smmu;
>  	bool bypass_quirk;
> @@ -146,6 +148,8 @@ static int qcom_adreno_smmu_init_context(struct arm_smmu_domain *smmu_domain,
>  {
>  	struct adreno_smmu_priv *priv;
>  
> +	pgtbl_cfg->tlb = &arm_smmu_s1_tlb_impl_ops;
> +
>  	/* Only enable split pagetables for the GPU device (SID 0) */
>  	if (!qcom_adreno_smmu_is_gpu_device(dev))
>  		return 0;
> @@ -185,6 +189,14 @@ static const struct of_device_id qcom_smmu_client_of_match[] __maybe_unused = {
>  	{ }
>  };
>  
> +static int qcom_smmu_init_context(struct arm_smmu_domain *smmu_domain,
> +		struct io_pgtable_cfg *pgtbl_cfg, struct device *dev)
> +{
> +	pgtbl_cfg->tlb = &arm_smmu_s1_tlb_impl_ops;
> +
> +	return 0;
> +}
> +
>  static int qcom_smmu_cfg_probe(struct arm_smmu_device *smmu)
>  {
>  	unsigned int last_s2cr = ARM_SMMU_GR0_S2CR(smmu->num_mapping_groups - 1);
> @@ -308,6 +320,7 @@ static int qcom_smmu500_reset(struct arm_smmu_device *smmu)
>  }
>  
>  static const struct arm_smmu_impl qcom_smmu_impl = {
> +	.init_context = qcom_smmu_init_context,
>  	.cfg_probe = qcom_smmu_cfg_probe,
>  	.def_domain_type = qcom_smmu_def_domain_type,
>  	.reset = qcom_smmu500_reset,
> diff --git a/drivers/iommu/arm/arm-smmu/arm-smmu.c b/drivers/iommu/arm/arm-smmu/arm-smmu.c
> index d3c6f54110a5..f3845e822565 100644
> --- a/drivers/iommu/arm/arm-smmu/arm-smmu.c
> +++ b/drivers/iommu/arm/arm-smmu/arm-smmu.c
> @@ -341,6 +341,12 @@ static void arm_smmu_tlb_add_page_s1(struct iommu_iotlb_gather *gather,
>  				  ARM_SMMU_CB_S1_TLBIVAL);
>  }
>  
> +static void arm_smmu_tlb_inv_walk_impl_s1(unsigned long iova, size_t size,
> +				     size_t granule, void *cookie)
> +{
> +	arm_smmu_tlb_inv_context_s1(cookie);
> +}
> +
>  static void arm_smmu_tlb_inv_walk_s2(unsigned long iova, size_t size,
>  				     size_t granule, void *cookie)
>  {
> @@ -388,6 +394,12 @@ static const struct iommu_flush_ops arm_smmu_s1_tlb_ops = {
>  	.tlb_add_page	= arm_smmu_tlb_add_page_s1,
>  };
>  
> +const struct iommu_flush_ops arm_smmu_s1_tlb_impl_ops = {
> +	.tlb_flush_all	= arm_smmu_tlb_inv_context_s1,
> +	.tlb_flush_walk	= arm_smmu_tlb_inv_walk_impl_s1,
> +	.tlb_add_page	= arm_smmu_tlb_add_page_s1,
> +};

Hmm, dunno about this. Wouldn't it be a lot cleaner if the tlb_flush_walk
callbacks just did the right thing based on the smmu_domain (maybe in the
arm_smmu_cfg?) rather than having an entirely new set of ops just because
they're const and you can't overide the bit you want?

I don't think there's really an awful lot qcom-specific about the principle
here -- there's a trade-off between over-invalidation and invalidation
latency. That happens on the CPU as well.

Will