[PATCH v3 1/2] blk-mq: add async quiesce interface

[Sagi Grimberg]

>> +void blk_mq_quiesce_queue_async(struct request_queue *q)
>> +{
>> +	struct blk_mq_hw_ctx *hctx;
>> +	unsigned int i;
>> +
>> +	blk_mq_quiesce_queue_nowait(q);
>> +
>> +	queue_for_each_hw_ctx(q, hctx, i) {
>> +		init_completion(&hctx->rcu_sync.completion);
>> +		init_rcu_head(&hctx->rcu_sync.head);
>> +		if (hctx->flags & BLK_MQ_F_BLOCKING)
>> +			call_srcu(hctx->srcu, &hctx->rcu_sync.head,
>> +				wakeme_after_rcu);
>> +		else
>> +			call_rcu(&hctx->rcu_sync.head,
>> +				wakeme_after_rcu);
>> +	}
> 
> Looks not necessary to do anything in case of !BLK_MQ_F_BLOCKING, and single
> synchronize_rcu() is OK for all hctx during waiting.

That's true, but I want a single interface for both. v2 had exactly
that, but I decided that this approach is better.

Also, having the driver call a single synchronize_rcu isn't great
layering (as quiesce can possibly use a different mechanism in the 
future). So drivers assumptions like:

         /*
          * SCSI never enables blk-mq's BLK_MQ_F_BLOCKING flag so
          * calling synchronize_rcu() once is enough.
          */
         WARN_ON_ONCE(shost->tag_set.flags & BLK_MQ_F_BLOCKING);

         if (!ret)
                 synchronize_rcu();

Are not great...

>> +}
>> +EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_async);
>> +
>> +void blk_mq_quiesce_queue_async_wait(struct request_queue *q)
>> +{
>> +	struct blk_mq_hw_ctx *hctx;
>> +	unsigned int i;
>> +
>> +	queue_for_each_hw_ctx(q, hctx, i) {
>> +		wait_for_completion(&hctx->rcu_sync.completion);
>> +		destroy_rcu_head(&hctx->rcu_sync.head);
>> +	}
>> +}
>> +EXPORT_SYMBOL_GPL(blk_mq_quiesce_queue_async_wait);
>> +
>>   /**
>>    * blk_mq_quiesce_queue() - wait until all ongoing dispatches have finished
>>    * @q: request queue.
>> diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
>> index 23230c1d031e..5536e434311a 100644
>> --- a/include/linux/blk-mq.h
>> +++ b/include/linux/blk-mq.h
>> @@ -5,6 +5,7 @@
>>   #include <linux/blkdev.h>
>>   #include <linux/sbitmap.h>
>>   #include <linux/srcu.h>
>> +#include <linux/rcupdate_wait.h>
>>   
>>   struct blk_mq_tags;
>>   struct blk_flush_queue;
>> @@ -170,6 +171,7 @@ struct blk_mq_hw_ctx {
>>   	 */
>>   	struct list_head	hctx_list;
>>   
>> +	struct rcu_synchronize	rcu_sync;
>   
> The above struct takes at least 5 words, and I'd suggest to avoid it,
> and the hctx->srcu should be re-used for waiting BLK_MQ_F_BLOCKING.
> Meantime !BLK_MQ_F_BLOCKING doesn't need it.

It is at the end and contains exactly what is needed to synchronize. Not
sure what you mean by reuse hctx->srcu?