[PATCH] iommu/arm-smmu-v3: Poll for CMDQ drain completion more effectively

[Will Deacon]

On Wed, May 03, 2017 at 04:33:57PM +0100, Robin Murphy wrote:
> On 27/04/17 12:13, sunil.kovvuri at gmail.com wrote:
> > From: Sunil Goutham <sgoutham at cavium.com>
> > 
> > Modified polling on CMDQ consumer similar to how polling is done for TLB SYNC
> > completion in SMMUv2 driver. Code changes are done with reference to
> > 
> > 8513c8930069 iommu/arm-smmu: Poll for TLB sync completion more effectively
> > 
> > Poll timeout has been increased which addresses issue of 100us timeout not
> > sufficient, when command queue is full with TLB invalidation commands.
> > 
> > Signed-off-by: Sunil Goutham <sgoutham at cavium.com>
> > Signed-off-by: Geetha <gakula at cavium.com>
> > ---
> >  drivers/iommu/arm-smmu-v3.c | 15 ++++++++++++---
> >  1 file changed, 12 insertions(+), 3 deletions(-)
> > 
> > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
> > index d412bdd..34599d4 100644
> > --- a/drivers/iommu/arm-smmu-v3.c
> > +++ b/drivers/iommu/arm-smmu-v3.c
> > @@ -379,6 +379,9 @@
> >  #define CMDQ_SYNC_0_CS_NONE		(0UL << CMDQ_SYNC_0_CS_SHIFT)
> >  #define CMDQ_SYNC_0_CS_SEV		(2UL << CMDQ_SYNC_0_CS_SHIFT)
> >  
> > +#define CMDQ_DRAIN_TIMEOUT_US		1000
> > +#define CMDQ_SPIN_COUNT			10
> > +
> >  /* Event queue */
> >  #define EVTQ_ENT_DWORDS			4
> >  #define EVTQ_MAX_SZ_SHIFT		7
> > @@ -737,7 +740,8 @@ static void queue_inc_prod(struct arm_smmu_queue *q)
> >   */
> >  static int queue_poll_cons(struct arm_smmu_queue *q, bool drain, bool wfe)
> >  {
> > -	ktime_t timeout = ktime_add_us(ktime_get(), ARM_SMMU_POLL_TIMEOUT_US);
> > +	ktime_t timeout = ktime_add_us(ktime_get(), CMDQ_DRAIN_TIMEOUT_US);
> > +	unsigned int spin_cnt, delay = 1;
> >  
> >  	while (queue_sync_cons(q), (drain ? !queue_empty(q) : queue_full(q))) {
> >  		if (ktime_compare(ktime_get(), timeout) > 0)
> > @@ -746,8 +750,13 @@ static int queue_poll_cons(struct arm_smmu_queue *q, bool drain, bool wfe)
> >  		if (wfe) {
> >  			wfe();
> >  		} else {
> > -			cpu_relax();
> > -			udelay(1);
> > +			for (spin_cnt = 0;
> > +			     spin_cnt < CMDQ_SPIN_COUNT; spin_cnt++) {
> > +				cpu_relax();
> > +				continue;
> > +			}
> > +			udelay(delay);
> > +			delay *= 2;
> 
> Sorry, I can't make sense of this. The referenced commit uses the spin
> loop to poll opportunistically a few times before delaying. This loop
> just adds a short open-coded udelay to an exponential udelay, and it's
> not really clear that that's any better than a fixed udelay (especially
> as the two cases in which we poll are somewhat different).
> 
> What's wrong with simply increasing the timeout value alone?

I asked that the timeout is only increased for the drain case, and that
we fix the issue here where we udelat if cons didn't move immediately:

http://lists.infradead.org/pipermail/linux-arm-kernel/2017-April/503389.html

but I don't think the patch above actually achieves any of that.

Will