nvme: split bios issued in reverse order

[Keith Busch]

On Tue, May 24, 2022 at 03:37:56PM -0400, Jonathan Nicklin wrote:
> On Tue, May 24, 2022 at 3:25 PM Keith Busch <kbusch at kernel.org> wrote:
> >
> > On Tue, May 24, 2022 at 12:12:29PM -0400, Jonathan Nicklin wrote:
> > >
> > > The command lines you have are for read operations. The behavior seems
> > > only to appear with writes.
> >
> > Huh, I'll be darn...
> >
> > I think it's because the writes are plugged and the reads are not. The plug
> > appends requests to the head of the plug list, and unplugging will dispatch the
> > requests in the reverse order.
> 
> Thanks for confirming! That's about where I got to. Do you have any
> ideas on what might explain the difference in behavior between
> fio/pvsync and fio/libaio? And, why does this not seem to occur when
> only one nvme queue is present? Perhaps the in-order cases are an
> indication of not being plugged?

I actually didn't see a difference between libaio or psync, and also io_uring.
They all plugged and reversed the dispatch order. Do you have a scheduler
enabled?