Sequential read from NVMe/XFS twice slower on Fedora 42 than on Rocky 9.5

[Dave Chinner]

On Mon, May 05, 2025 at 09:21:19AM -0400, Laurence Oberman wrote:
> On Mon, 2025-05-05 at 08:29 -0400, Laurence Oberman wrote:
> > On Mon, 2025-05-05 at 07:50 +1000, Dave Chinner wrote:
> > > So the MD block device shows the same read performance as the
> > > filesystem on top of it. That means this is a regression at the MD
> > > device layer or in the block/driver layers below it. i.e. it is not
> > > an XFS of filesystem issue at all.
> > > 
> > > -Dave.
> > 
> > I have a lab setup, let me see if I can also reproduce and then trace
> > this to see where it is spending the time
> > 
> 
> 
> Not seeing 1/2 the bandwidth but also significantly slower on Fedora42
> kernel.
> I will trace it
> 
> 9.5 kernel - 5.14.0-503.40.1.el9_5.x86_64
> 
> Run status group 0 (all jobs):
>    READ: bw=14.7GiB/s (15.8GB/s), 14.7GiB/s-14.7GiB/s (15.8GB/s-
> 15.8GB/s), io=441GiB (473GB), run=30003-30003msec
> 
> Fedora42 kernel - 6.14.5-300.fc42.x86_64
> 
> Run status group 0 (all jobs):
>    READ: bw=10.4GiB/s (11.2GB/s), 10.4GiB/s-10.4GiB/s (11.2GB/s-
> 11.2GB/s), io=313GiB (336GB), run=30001-30001msec

So is this MD chunk size related? i.e. what is the chunk size
the MD device? Is it smaller than the IO size (256kB) or larger?
Does the regression go away if the chunk size matches the IO size,
or if the IO size vs chunk size relationship is reversed?

-Dave.
-- 
Dave Chinner
david at fromorbit.com