switch block layer polling to a bio based model v3

[Wunderlich, Mark]

As a data point, tested V2 and this V3 patch series, applied to my test initiator
currently running 5.10 branch of linux-nvme.

All results via FIO using the following options:  --time_based --runtime=60 --thread
--rw=randread --refill_buffers --direct=1 --ioengine=io_uring --hipri --fixedbufs --bs=4k
--iodepth=32 --iodepth_batch_complete_min=1 --iodepth_batch_complete_max=32
--iodepth_batch=8 --group_reporting --gtod_reduce=0 --disable_lat=0 , 
with only the queue depth, batch size, and number of job threads varied as indicated.

Tests are directed towards one or more remote nvme optane devices.

All data reported as:
IOPS (k), Avg Lat (usec), stdev (usec), 99.99 clat (usec)

For reference, baseline performance on this branch, running without nvme multipathing enabled, while using 'hipri' polling option:
[1 thread, QD 1, Batch 1]
33.1, 29.21, 1.42, 54.52
[1 thread, QD 32, Batch 8]
268, 101.17, 14.87, 139
[16 thread, QD 32, Batch 8]
1965, 247.25, 28.28, 449

This branch with nvme multipathing enabled, V2 of the patch series applied:
[1 thread, QD 1, Batch 1]
33, 29.22, 1.56, 54.01
[1 thread, QD 32, Batch 8]
259, 104.38, 15.04, 141
[16 thread, QD32, Batch 8]
1905, 255.52, 30.97, 461

The same config as above for V2, but FIO results when not using 'hipri' polling option:
[1 thread, QD 1, Batch 1]
22.9k, 41.66, 3.78, 78.33
[1 thread, QD 32, Batch 8]
224k, 103.88, 28.41, 163
[16 thread, QD32, Batch 8]
1910k, 245.23, 66.30, 502

The same branch but with V3 of the patch series applied.  Again using the 'hipri' option:
[1 thread, QD 1, Batch 1]
33.2, 29.12, 1.35, 54.53
[1 thread, QD 32, Batch 8]
258, 104.55, 15.01, 141
[16 thread, QD32, Batch 8]
1914, 254.19, 30.00,  457

So the data shows that this patch series clearly enables the use of 'hipri' polling when
kernel configured with nvme multipathing, which was not previously supported.

There is not a significant difference in performance measured with and without
multipathing enabled, with either patch series for bio based polling.  And V2 and V3
reporting virtually the same performance as expected.

So I tip my hat to this patch series - cheers

Tested-by: Mark Wunderlich <mark.wunderlich at intel.com>