nvme: split bios issued in reverse order

[Damien Le Moal]

On 5/25/22 05:32, Jonathan Nicklin wrote:
> On Tue, May 24, 2022 at 4:29 PM Keith Busch <kbusch at kernel.org> wrote:
>>
>> On Tue, May 24, 2022 at 03:37:56PM -0400, Jonathan Nicklin wrote:
>>> On Tue, May 24, 2022 at 3:25 PM Keith Busch <kbusch at kernel.org> wrote:
>>>>
>>>> On Tue, May 24, 2022 at 12:12:29PM -0400, Jonathan Nicklin wrote:
>>>>>
>>>>> The command lines you have are for read operations. The behavior seems
>>>>> only to appear with writes.
>>>>
>>>> Huh, I'll be darn...
>>>>
>>>> I think it's because the writes are plugged and the reads are not. The plug
>>>> appends requests to the head of the plug list, and unplugging will dispatch the
>>>> requests in the reverse order.
>>>
>>> Thanks for confirming! That's about where I got to. Do you have any
>>> ideas on what might explain the difference in behavior between
>>> fio/pvsync and fio/libaio? And, why does this not seem to occur when
>>> only one nvme queue is present? Perhaps the in-order cases are an
>>> indication of not being plugged?
>>
>> I actually didn't see a difference between libaio or psync, and also io_uring.
>> They all plugged and reversed the dispatch order. Do you have a scheduler
>> enabled?
> 
> Nope, there's no scheduler in the way.

If the drive is exercised at QD=1, e.g. psync() and libaio with iodepth=1,
then plugging does not matter as there will be no merge (so no at-head
insertion in the plug). Commands will be executed in the user submission
order. At higher qd, if merge happen while plugged, then order will be
reversed (libaio with iodepth > 1 only).

> 


-- 
Damien Le Moal
Western Digital Research