[bug-report] 5-9% FIO randomwrite ext4 perf regression on 6.12.y kernel

[Saeed Mirzamohammadi]

Hi,

I’m reporting a performance regression of up to 9-10% with FIO randomwrite benchmark on ext4 comparing 6.12.0-rc2 kernel and v5.15.161. Also, standard deviation after this change grows up to 5-6%.

Bisect root cause commit
===================
- commit 63dfa1004322 ("nvme: move NVME_QUIRK_DEALLOCATE_ZEROES out of nvme_config_discard”)


Test details
=========
- readwrite=randwrite bs=4k size=1G ioengine=libaio iodepth=16 direct=1 time_based=1 ramp_time=180 runtime=1800 randrepeat=1 gtod_reduce=1
- Test is on ext4 filesystem
- System has 4 NVMe disks


5.15.y base
========
fio.test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.35
Starting 128 processes
fio.test: Laying out IO file (1 file / 1024MiB)
…[cut here]
fio.test: Laying out IO file (1 file / 1024MiB)

fio.test: (groupid=0, jobs=128): err= 0: pid=4226: Fri Sep 13 00:34:07 2024
write: IOPS=2550k, BW=9962MiB/s (10.4GB/s)(17.1TiB/1800006msec); 0 zone resets
bw ( MiB/s): min= 5326, max=15283, per=100.00%, avg=9972.35, stdev=12.37, samples=460672
iops : min=1363492, max=3912552, avg=2552897.74, stdev=3166.48, samples=460672
cpu : usr=4.00%, sys=13.81%, ctx=4730536027, majf=0, minf=71229
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4590594600,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
WRITE: bw=9962MiB/s (10.4GB/s), 9962MiB/s-9962MiB/s (10.4GB/s-10.4GB/s), io=17.1TiB (18.8TB), run=1800006-1800006msec

Disk stats (read/write):
dm-0: ios=0/5006840732, merge=0/0, ticks=0/3893644388, in_queue=3893644388, util=100.00%, aggrios=0/1251134829, aggrmerge=0/737033, aggrticks=0/973009387, aggrin_queue=973009387, aggrutil=100.00%
nvme3n1: ios=0/1251035509, merge=0/829083, ticks=0/1443792479, in_queue=1443792479, util=100.00%
nvme0n1: ios=0/1251231344, merge=0/638993, ticks=0/1011756001, in_queue=1011756002, util=100.00%
nvme1n1: ios=0/1251224162, merge=0/639192, ticks=0/672688952, in_queue=672688953, util=100.00%
nvme2n1: ios=0/1251048302, merge=0/840864, ticks=0/763800117, in_queue=763800117, util=100.00%

Throughput Results:
WRITE:10649.6:2550000:0


6.12.y test
========
fio.test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.35
Starting 128 processes
fio.test: Laying out IO file (1 file / 1024MiB)
…[cut here]
fio.test: Laying out IO file (1 file / 1024MiB)

fio.test: (groupid=0, jobs=128): err= 0: pid=4308: Fri Sep 13 08:03:37 2024
write: IOPS=2270k, BW=8868MiB/s (9299MB/s)(15.2TiB/1800006msec); 0 zone resets
bw ( MiB/s): min= 6, max=13343, per=100.00%, avg=9066.78, stdev=14.33, samples=451008
iops : min= 1743, max=3415839, avg=2321069.38, stdev=3669.20, samples=451008
cpu : usr=3.65%, sys=11.28%, ctx=3420577602, majf=0, minf=12682
IO depths : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4086517562,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
WRITE: bw=8868MiB/s (9299MB/s), 8868MiB/s-8868MiB/s (9299MB/s-9299MB/s), io=15.2TiB (16.7TB), run=1800006-1800006msec

Disk stats (read/write):
dm-0: ios=0/4404382767, merge=0/0, ticks=0/3614104412, in_queue=3614104412, util=99.91%, aggrios=0/1099847046, aggrmerge=0/1309757, aggrticks=0/908073333, aggrin_queue=908073333, aggrutil=82.28%
nvme3n1: ios=0/1099844453, merge=0/1309770, ticks=0/606919121, in_queue=606919121, util=82.28%
nvme0n1: ios=0/1099847110, merge=0/1310120, ticks=0/1007261464, in_queue=1007261464, util=80.97%
nvme1n1: ios=0/1099847041, merge=0/1309430, ticks=0/1327975386, in_queue=1327975386, util=81.71%
nvme2n1: ios=0/1099849583, merge=0/1309709, ticks=0/690137362, in_queue=690137362, util=80.29%

Throughput Results:
WRITE:9299:2270000:0


Thanks,
Saeed