nvme: split bios issued in reverse order

[Jonathan Nicklin]

With the drive exercised as follows:

app: fio
engine: libaio
queue depth: 1
block size: 128K
device max_sectors_kb: 32K

A 128K user IO is split into four 32K I/Os which are then issued in
reverse order as follows:

             fio-12103   [001] ..... 89587.120514: nvme_setup_cmd:
nvme1: disk=nvme1c1n1, qid=2, cmdid=12328, nsid=1, flags=0x0,
meta=0x0, cmd=(nvme_cmd_write slba=192, len=63, ctrl=0x0, dsmgmt=0,
reftag=0)
             fio-12103   [001] ..... 89587.120515: nvme_setup_cmd:
nvme1: disk=nvme1c1n1, qid=2, cmdid=12327, nsid=1, flags=0x0,
meta=0x0, cmd=(nvme_cmd_write slba=128, len=63, ctrl=0x0, dsmgmt=0,
reftag=0)
             fio-12103   [001] ..... 89587.120518: nvme_setup_cmd:
nvme1: disk=nvme1c1n1, qid=2, cmdid=12326, nsid=1, flags=0x0,
meta=0x0, cmd=(nvme_cmd_write slba=64, len=63, ctrl=0x0, dsmgmt=0,
reftag=0)
             fio-12103   [001] ..... 89587.120518: nvme_setup_cmd:
nvme1: disk=nvme1c1n1, qid=2, cmdid=12325, nsid=1, flags=0x0,
meta=0x0, cmd=(nvme_cmd_write slba=0, len=63, ctrl=0x0, dsmgmt=0,
reftag=0)

On Tue, May 24, 2022 at 5:44 PM Damien Le Moal
<damien.lemoal at opensource.wdc.com> wrote:
>
> On 5/25/22 05:32, Jonathan Nicklin wrote:
> > On Tue, May 24, 2022 at 4:29 PM Keith Busch <kbusch at kernel.org> wrote:
> >>
> >> On Tue, May 24, 2022 at 03:37:56PM -0400, Jonathan Nicklin wrote:
> >>> On Tue, May 24, 2022 at 3:25 PM Keith Busch <kbusch at kernel.org> wrote:
> >>>>
> >>>> On Tue, May 24, 2022 at 12:12:29PM -0400, Jonathan Nicklin wrote:
> >>>>>
> >>>>> The command lines you have are for read operations. The behavior seems
> >>>>> only to appear with writes.
> >>>>
> >>>> Huh, I'll be darn...
> >>>>
> >>>> I think it's because the writes are plugged and the reads are not. The plug
> >>>> appends requests to the head of the plug list, and unplugging will dispatch the
> >>>> requests in the reverse order.
> >>>
> >>> Thanks for confirming! That's about where I got to. Do you have any
> >>> ideas on what might explain the difference in behavior between
> >>> fio/pvsync and fio/libaio? And, why does this not seem to occur when
> >>> only one nvme queue is present? Perhaps the in-order cases are an
> >>> indication of not being plugged?
> >>
> >> I actually didn't see a difference between libaio or psync, and also io_uring.
> >> They all plugged and reversed the dispatch order. Do you have a scheduler
> >> enabled?
> >
> > Nope, there's no scheduler in the way.
>
> If the drive is exercised at QD=1, e.g. psync() and libaio with iodepth=1,
> then plugging does not matter as there will be no merge (so no at-head
> insertion in the plug). Commands will be executed in the user submission
> order. At higher qd, if merge happen while plugged, then order will be
> reversed (libaio with iodepth > 1 only).
>
> >
>
>
> --
> Damien Le Moal
> Western Digital Research