[PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns
Chaitanya Kulkarni
chaitanyak at nvidia.com
Wed Feb 2 01:04:43 PST 2022
From: Chaitanya Kulkarni <kch at nvidia.com>
Hi,
In current implementation like NVMeOF target file backed namespace we
don't support buffered_io for block device backed namesapce. This allows
user to configure block device backend in buffered_io mode so user can
get the benefits of caching.
Please note that using this combination host needs to periodically sync
between cache and storage or set the system cache parameters
accordingly. NVMeOF target is not responsible for data consistency
between cache and storage.
fio data verification, rand read/write performance numbers :-
1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-
fio Verify job :-
----------------------
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)
write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);
fio Random Write :-
----------------------
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);
fio Random Read :-
----------------------
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)
2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-
fio Verify job :-
----------------------
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)
write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);
fio Random Write :-
----------------------
write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);
fio Random Read :-
----------------------
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)
-ck
changes from V2:
1. Add a comment in the nvmet_bdev_ns_enable() when returning early
for buffered_io case. Feel free to edit the comment at the time
of applying patch.
changes from v1:
1. Remove first two patches and don't check for stat mode for
block device.
2. use i_size_read() to get the size of the file.
Chaitanya Kulkarni (2):
nvmet: use i_size_read() to set size for file-ns
nvmet: allow bdev in buffered_io mode
drivers/nvme/target/io-cmd-bdev.c | 8 ++++++++
drivers/nvme/target/io-cmd-file.c | 17 ++++-------------
drivers/nvme/target/nvmet.h | 2 +-
3 files changed, 13 insertions(+), 14 deletions(-)
root at dev nvme (nvme-5.17) # ./bdev-buffered-io
bdev-buffered-io/ bdev-buffered-io-test.sh
root at dev nvme (nvme-5.17) # ./bdev-buffered-io-test.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.350s
user 0m0.003s
sys 0m0.002s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh 1
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to disconnect by NQN: testnqn
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/:
total 6.7M
-rw-r--r--. 1 root root 2.8M Feb 1 23:42 nvme-core.ko
-rw-r--r--. 1 root root 448K Feb 1 23:42 nvme-fabrics.ko
-rw-r--r--. 1 root root 1015K Feb 1 23:42 nvme-fc.ko
-rw-r--r--. 1 root root 745K Feb 1 23:42 nvme.ko
-rw-r--r--. 1 root root 958K Feb 1 23:42 nvme-rdma.ko
-rw-r--r--. 1 root root 902K Feb 1 23:42 nvme-tcp.ko
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//:
total 6.7M
-rw-r--r--. 1 root root 498K Feb 1 23:42 nvme-fcloop.ko
-rw-r--r--. 1 root root 439K Feb 1 23:42 nvme-loop.ko
-rw-r--r--. 1 root root 768K Feb 1 23:42 nvmet-fc.ko
-rw-r--r--. 1 root root 3.3M Feb 1 23:42 nvmet.ko
-rw-r--r--. 1 root root 929K Feb 1 23:42 nvmet-rdma.ko
-rw-r--r--. 1 root root 775K Feb 1 23:42 nvmet-tcp.ko
+ modprobe nvme
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)
real 0m0.001s
user 0m0.001s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│  └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
+++ echo 1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
+++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 2348.622209] nvme nvme0: 48/0/0 default/read/poll queues
[ 2350.605991] nvmet: adding nsid 1 to subsystem testnqn
[ 2351.616737] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bb4f2c46-9ac7-4283-902a-420883b62f3c.
[ 2351.616857] nvme nvme1: creating 48 I/O queues.
[ 2351.621168] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=20757: Tue Feb 1 23:42:57 2022
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)
slat (usec): min=3, max=422, avg= 3.96, stdev= 1.90
clat (nsec): min=1072, max=483948, avg=15610.29, stdev=5064.35
lat (usec): min=4, max=494, avg=19.64, stdev= 6.20
clat percentiles (nsec):
| 1.00th=[13248], 5.00th=[13376], 10.00th=[13504], 20.00th=[13632],
| 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60.00th=[14144],
| 70.00th=[14272], 80.00th=[14528], 90.00th=[23424], 95.00th=[26752],
| 99.00th=[28544], 99.50th=[32384], 99.90th=[42752], 99.95th=[47360],
| 99.99th=[82432]
write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);
slat (usec): min=4, max=221, avg= 7.36, stdev= 2.87
clat (nsec): min=841, max=222422, avg=24669.34, stdev=8735.58
lat (usec): min=6, max=263, avg=32.11, stdev=11.14
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19],
| 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 21],
| 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 38], 95.00th=[ 40],
| 99.00th=[ 51], 99.50th=[ 58], 99.90th=[ 70], 99.95th=[ 81],
| 99.99th=[ 155]
bw ( KiB/s): min=150160, max=525424, per=83.10%, avg=409600.00, stdev=174551.65, samples=4
iops : min=37540, max=131356, avg=102400.00, stdev=43637.91, samples=4
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=64.99%, 50=34.34%, 100=0.65%
lat (usec) : 250=0.02%, 500=0.01%
cpu : usr=27.97%, sys=32.66%, ctx=333157, majf=0, minf=3053
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=506MiB (531MB), run=663-663msec
WRITE: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=800MiB (839MB), run=1662-1662msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.2GiB/s][w=4236k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20761: Tue Feb 1 23:43:57 2022
write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
slat (nsec): min=1112, max=24135k, avg=6779.81, stdev=5320.08
clat (nsec): min=721, max=24222k, avg=53033.78, stdev=21465.11
lat (usec): min=6, max=24230, avg=59.89, stdev=23.52
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 57],
| 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 63], 95.00th=[ 88],
| 99.00th=[ 126], 99.50th=[ 133], 99.90th=[ 186], 99.95th=[ 223],
| 99.99th=[ 277]
bw ( MiB/s): min=11452, max=20170, per=100.00%, avg=16608.41, stdev=84.92, samples=3808
iops : min=2931936, max=5163618, avg=4251753.76, stdev=21739.07, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.82%, 100=42.95%
lat (usec) : 250=2.21%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.32%, sys=36.88%, ctx=254865246, majf=0, minf=439
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,254859948,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=972GiB (1044GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.3GiB/s][w=4015k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20878: Tue Feb 1 23:44:57 2022
write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
slat (nsec): min=1773, max=23957k, avg=6930.59, stdev=5690.00
clat (nsec): min=672, max=24058k, avg=54190.75, stdev=22713.63
lat (usec): min=6, max=24086, avg=61.20, stdev=24.89
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 94],
| 99.00th=[ 129], 99.50th=[ 137], 99.90th=[ 192], 99.95th=[ 225],
| 99.99th=[ 269]
bw ( MiB/s): min=11359, max=20085, per=100.00%, avg=16257.07, stdev=84.39, samples=3808
iops : min=2907927, max=5141948, avg=4161809.03, stdev=21605.08, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.02%, 100=42.99%
lat (usec) : 250=2.97%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.06%, sys=36.43%, ctx=249428122, majf=0, minf=464
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249423456,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.0GB/s), 15.9GiB/s-15.9GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1022GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4283k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20954: Tue Feb 1 23:45:57 2022
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
slat (nsec): min=1783, max=23922k, avg=6853.24, stdev=4821.07
clat (nsec): min=711, max=24147k, avg=53603.75, stdev=19747.55
lat (usec): min=5, max=24155, avg=60.53, stdev=21.64
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 85],
| 99.00th=[ 118], 99.50th=[ 133], 99.90th=[ 167], 99.95th=[ 190],
| 99.99th=[ 251]
bw ( MiB/s): min=11876, max=20042, per=100.00%, avg=16429.89, stdev=81.89, samples=3808
iops : min=3040376, max=5130956, avg=4206051.45, stdev=20963.19, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.72%, 100=45.12%
lat (usec) : 250=2.14%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.02%, sys=37.05%, ctx=252169523, majf=0, minf=473
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252164251,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.1GiB/s][w=3695k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=21089: Tue Feb 1 23:46:58 2022
write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
slat (nsec): min=1222, max=12690k, avg=6987.00, stdev=5499.44
clat (nsec): min=711, max=16691k, avg=54537.79, stdev=23234.80
lat (usec): min=5, max=16701, avg=61.61, stdev=25.56
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 93],
| 99.00th=[ 131], 99.50th=[ 141], 99.90th=[ 184], 99.95th=[ 204],
| 99.99th=[ 260]
bw ( MiB/s): min=11551, max=20105, per=100.00%, avg=16156.07, stdev=74.80, samples=3808
iops : min=2957123, max=5146990, avg=4135953.66, stdev=19148.37, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.25%, 100=44.71%
lat (usec) : 250=3.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.94%, sys=36.18%, ctx=247812660, majf=0, minf=472
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,247805833,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (16.9GB/s), 15.8GiB/s-15.8GiB/s (16.9GB/s-16.9GB/s), io=945GiB (1015GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4268k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22043: Tue Feb 1 23:47:58 2022
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
slat (nsec): min=1763, max=17410k, avg=6908.89, stdev=5766.44
clat (nsec): min=712, max=17505k, avg=54011.25, stdev=23339.67
lat (usec): min=5, max=17513, avg=61.00, stdev=25.50
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 87],
| 99.00th=[ 121], 99.50th=[ 135], 99.90th=[ 178], 99.95th=[ 204],
| 99.99th=[ 310]
bw ( MiB/s): min=11707, max=19992, per=100.00%, avg=16303.82, stdev=73.70, samples=3808
iops : min=2997012, max=5118104, avg=4173776.76, stdev=18867.69, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.47%, 100=45.42%
lat (usec) : 250=2.10%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.04%, sys=36.68%, ctx=250242780, majf=0, minf=461
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,250236670,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=955GiB (1025GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4186k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22119: Tue Feb 1 23:48:58 2022
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
slat (nsec): min=1162, max=13647k, avg=6841.84, stdev=4990.36
clat (nsec): min=832, max=17776k, avg=53604.58, stdev=21052.59
lat (usec): min=6, max=17783, avg=60.53, stdev=22.89
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 58],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 65], 95.00th=[ 79],
| 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 153], 99.95th=[ 176],
| 99.99th=[ 306]
bw ( MiB/s): min=11133, max=20082, per=100.00%, avg=16434.73, stdev=79.43, samples=3808
iops : min=2850202, max=5141134, avg=4207290.84, stdev=20334.84, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.62%
lat (usec) : 100=48.12%, 250=1.25%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.77%, sys=36.68%, ctx=252168793, majf=0, minf=462
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252163391,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.5GiB/s][w=4326k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22221: Tue Feb 1 23:49:58 2022
write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
slat (nsec): min=1804, max=16555k, avg=6911.03, stdev=4216.56
clat (nsec): min=731, max=16639k, avg=54075.05, stdev=17110.09
lat (usec): min=5, max=16647, avg=61.07, stdev=18.98
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 58],
| 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 81],
| 99.00th=[ 123], 99.50th=[ 135], 99.90th=[ 165], 99.95th=[ 184],
| 99.99th=[ 210]
bw ( MiB/s): min=11319, max=20001, per=100.00%, avg=16285.86, stdev=86.40, samples=3808
iops : min=2897836, max=5120368, avg=4169180.79, stdev=22118.90, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.04%
lat (usec) : 100=47.21%, 250=1.75%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.43%, sys=36.52%, ctx=249967922, majf=0, minf=464
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249962350,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=954GiB (1024GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4224k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22298: Tue Feb 1 23:50:59 2022
write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
slat (nsec): min=1372, max=18974k, avg=7033.73, stdev=4335.83
clat (nsec): min=721, max=19073k, avg=54854.61, stdev=19806.39
lat (usec): min=6, max=19082, avg=61.97, stdev=22.12
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 95],
| 99.00th=[ 133], 99.50th=[ 147], 99.90th=[ 184], 99.95th=[ 196],
| 99.99th=[ 219]
bw ( MiB/s): min=11060, max=20108, per=100.00%, avg=16051.21, stdev=92.86, samples=3808
iops : min=2831464, max=5147868, avg=4109110.77, stdev=23772.60, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=53.51%
lat (usec) : 100=42.99%, 250=3.49%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.98%, sys=35.65%, ctx=246371950, majf=0, minf=458
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,246367496,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.7GiB/s (16.8GB/s), 15.7GiB/s-15.7GiB/s (16.8GB/s-16.8GB/s), io=940GiB (1009GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.9GiB/s][w=3895k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22372: Tue Feb 1 23:51:59 2022
write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
slat (nsec): min=1623, max=16114k, avg=6882.50, stdev=4431.39
clat (nsec): min=712, max=16232k, avg=53879.81, stdev=16614.37
lat (usec): min=6, max=16241, avg=60.84, stdev=18.29
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 58],
| 70.00th=[ 60], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 80],
| 99.00th=[ 104], 99.50th=[ 129], 99.90th=[ 161], 99.95th=[ 182],
| 99.99th=[ 219]
bw ( MiB/s): min=12006, max=19952, per=100.00%, avg=16360.11, stdev=79.02, samples=3808
iops : min=3073692, max=5107912, avg=4188189.50, stdev=20228.57, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.59%, 100=49.27%
lat (usec) : 250=1.14%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.16%, sys=37.05%, ctx=250877181, majf=0, minf=498
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,250866850,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=957GiB (1028GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4230k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=23453: Tue Feb 1 23:52:59 2022
write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);
slat (nsec): min=1824, max=12568k, avg=6939.60, stdev=3999.18
clat (nsec): min=701, max=16670k, avg=54238.72, stdev=17869.32
lat (usec): min=6, max=16677, avg=61.26, stdev=19.85
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 84],
| 99.00th=[ 129], 99.50th=[ 141], 99.90th=[ 176], 99.95th=[ 194],
| 99.99th=[ 217]
bw ( MiB/s): min=11198, max=20030, per=100.00%, avg=16238.98, stdev=82.50, samples=3808
iops : min=2866772, max=5127914, avg=4157179.99, stdev=21119.50, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.21%, 100=45.55%
lat (usec) : 250=2.23%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.99%, sys=36.59%, ctx=249190314, majf=0, minf=490
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249183943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1021GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.9GiB/s][r=4960k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23530: Tue Feb 1 23:53:59 2022
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)
slat (nsec): min=1142, max=49607k, avg=8828.54, stdev=15569.98
clat (nsec): min=441, max=243040k, avg=612190.41, stdev=339908.18
lat (usec): min=6, max=243043, avg=621.12, stdev=343.94
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1926], 99.95th=[ 4817],
| 99.99th=[11469]
bw ( MiB/s): min= 1691, max=27419, per=100.00%, avg=19320.99, stdev=89.34, samples=5712
iops : min=433098, max=7019516, avg=4946173.24, stdev=22869.85, samples=5712
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=15.40%, 750=69.25%, 1000=4.03%
lat (msec) : 2=11.24%, 4=0.03%, 10=0.04%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=9.29%, sys=35.84%, ctx=296529230, majf=0, minf=3836
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296546544,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.2GB/s), 18.9GiB/s-18.9GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1215GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4834k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23882: Tue Feb 1 23:55:00 2022
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)
slat (usec): min=2, max=16996, avg= 8.84, stdev= 6.11
clat (nsec): min=911, max=21566k, avg=612493.08, stdev=234517.40
lat (usec): min=4, max=21574, avg=621.43, stdev=238.08
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1549], 99.95th=[ 2024],
| 99.99th=[ 3261]
bw ( MiB/s): min=12590, max=26589, per=100.00%, avg=19314.78, stdev=65.21, samples=5712
iops : min=3223108, max=6806844, avg=4944582.49, stdev=16693.22, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=14.27%, 750=69.32%, 1000=5.38%
lat (msec) : 2=10.98%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.29%, sys=36.21%, ctx=296412664, majf=0, minf=3763
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296403321,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1214GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.6GiB/s][r=4885k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24112: Tue Feb 1 23:56:00 2022
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)
slat (usec): min=2, max=22808, avg= 8.65, stdev= 6.86
clat (nsec): min=912, max=26105k, avg=600221.79, stdev=221404.88
lat (usec): min=5, max=26112, avg=608.96, stdev=224.71
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319],
| 99.99th=[ 2057]
bw ( MiB/s): min=16300, max=26946, per=100.00%, avg=19713.39, stdev=67.57, samples=5712
iops : min=4173052, max=6898318, avg=5046626.47, stdev=17299.17, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=12.91%, 750=73.53%, 1000=3.26%
lat (msec) : 2=10.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.37%, sys=36.74%, ctx=302479289, majf=0, minf=3758
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=302468143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1154GiB (1239GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.9GiB/s][r=4683k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24212: Tue Feb 1 23:57:00 2022
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)
slat (nsec): min=1953, max=52925k, avg=8751.89, stdev=11303.93
clat (nsec): min=912, max=56316k, avg=606515.90, stdev=246434.14
lat (usec): min=5, max=56326, avg=615.36, stdev=249.97
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1745], 99.95th=[ 1827],
| 99.99th=[ 4752]
bw ( MiB/s): min=13300, max=27673, per=100.00%, avg=19511.23, stdev=82.41, samples=5712
iops : min=3405011, max=7084445, avg=4994872.61, stdev=21096.46, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=14.24%, 750=70.95%, 1000=3.71%
lat (msec) : 2=11.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=9.31%, sys=36.30%, ctx=299345988, majf=0, minf=3730
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=299326708,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1142GiB (1226GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.8GiB/s][r=4674k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25485: Tue Feb 1 23:58:00 2022
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)
slat (nsec): min=1473, max=41255k, avg=8761.08, stdev=7916.28
clat (nsec): min=871, max=45519k, avg=607322.90, stdev=229101.48
lat (usec): min=5, max=45526, avg=616.18, stdev=232.51
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
| 99.99th=[ 1745]
bw ( MiB/s): min=15677, max=27351, per=100.00%, avg=19485.07, stdev=73.52, samples=5712
iops : min=4013328, max=7002048, avg=4988177.24, stdev=18821.75, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=14.54%, 750=69.53%, 1000=5.29%
lat (msec) : 2=10.62%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.33%, sys=36.13%, ctx=298932803, majf=0, minf=3800
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=298924191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1140GiB (1224GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4763k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25572: Tue Feb 1 23:59:01 2022
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)
slat (nsec): min=1985, max=21488k, avg=8673.54, stdev=5859.90
clat (nsec): min=902, max=22056k, avg=601397.58, stdev=214885.23
lat (usec): min=5, max=22064, avg=610.17, stdev=218.14
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 553], 90.00th=[ 971], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1631]
bw ( MiB/s): min=14781, max=27237, per=100.00%, avg=19676.38, stdev=75.29, samples=5712
iops : min=3784135, max=6972871, avg=5037153.12, stdev=19275.07, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=13.17%, 750=72.94%, 1000=4.22%
lat (msec) : 2=9.67%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.32%, sys=36.93%, ctx=301882480, majf=0, minf=3751
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=301872061,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1152GiB (1236GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.6GiB/s][r=4607k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25664: Wed Feb 2 00:00:01 2022
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
slat (usec): min=2, max=21123, avg= 8.83, stdev= 5.96
clat (nsec): min=942, max=22372k, avg=611470.57, stdev=236265.95
lat (usec): min=6, max=22391, avg=620.40, stdev=239.84
clat percentiles (usec):
| 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
| 99.99th=[ 1795]
bw ( MiB/s): min=15222, max=27543, per=100.00%, avg=19354.61, stdev=79.33, samples=5712
iops : min=3896862, max=7051174, avg=4954779.41, stdev=20309.68, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=16.87%, 750=66.30%, 1000=4.54%
lat (msec) : 2=12.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.19%, sys=35.87%, ctx=296904832, majf=0, minf=3743
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296894931,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.3GB/s), 18.9GiB/s-18.9GiB/s (20.3GB/s-20.3GB/s), io=1133GiB (1216GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4554k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25785: Wed Feb 2 00:01:01 2022
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)
slat (usec): min=2, max=28533, avg= 9.30, stdev= 7.03
clat (nsec): min=921, max=29089k, avg=640985.06, stdev=258045.84
lat (usec): min=5, max=29096, avg=650.39, stdev=261.95
clat percentiles (usec):
| 1.00th=[ 392], 5.00th=[ 404], 10.00th=[ 412], 20.00th=[ 510],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 906], 90.00th=[ 1156], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1401],
| 99.99th=[ 1876]
bw ( MiB/s): min=13222, max=27706, per=100.00%, avg=18458.01, stdev=98.55, samples=5712
iops : min=3384868, max=7092862, avg=4725251.34, stdev=25229.04, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=18.90%, 750=56.44%, 1000=10.71%
lat (msec) : 2=13.94%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=8.95%, sys=33.65%, ctx=283232675, majf=0, minf=3746
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=283217843,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.0GiB/s (19.3GB/s), 18.0GiB/s-18.0GiB/s (19.3GB/s-19.3GB/s), io=1080GiB (1160GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4774k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25897: Wed Feb 2 00:02:01 2022
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)
slat (usec): min=2, max=37297, avg= 9.06, stdev=11.53
clat (nsec): min=892, max=37911k, avg=626423.87, stdev=281337.79
lat (usec): min=5, max=37919, avg=635.59, stdev=285.46
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 429], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1156], 95.00th=[ 1205],
| 99.00th=[ 1549], 99.50th=[ 1680], 99.90th=[ 2835], 99.95th=[ 3392],
| 99.99th=[ 5407]
bw ( MiB/s): min= 6580, max=27466, per=100.00%, avg=18885.17, stdev=100.72, samples=5712
iops : min=1684599, max=7031464, avg=4834602.87, stdev=25783.81, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=12.90%, 750=70.01%, 1000=4.60%
lat (msec) : 2=12.35%, 4=0.13%, 10=0.02%, 20=0.01%, 50=0.01%
cpu : usr=9.17%, sys=37.36%, ctx=289835159, majf=0, minf=3741
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=289804479,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.4GiB/s (19.8GB/s), 18.4GiB/s-18.4GiB/s (19.8GB/s-19.8GB/s), io=1106GiB (1187GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.3GiB/s][r=4805k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=26489: Wed Feb 2 00:03:02 2022
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)
slat (nsec): min=1714, max=56074k, avg=8893.16, stdev=12899.01
clat (nsec): min=922, max=57329k, avg=615610.19, stdev=258832.65
lat (usec): min=5, max=57337, avg=624.60, stdev=262.51
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1418], 99.95th=[ 1811],
| 99.99th=[ 4948]
bw ( MiB/s): min=14198, max=26839, per=100.00%, avg=19217.89, stdev=63.10, samples=5712
iops : min=3634881, max=6870838, avg=4919778.50, stdev=16154.49, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=15.84%, 750=67.55%, 1000=3.63%
lat (msec) : 2=12.94%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=9.32%, sys=35.56%, ctx=294925914, majf=0, minf=3702
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=294900613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.7GiB/s (20.1GB/s), 18.7GiB/s-18.7GiB/s (20.1GB/s-20.1GB/s), io=1125GiB (1208GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.349s
user 0m0.000s
sys 0m0.003s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│  └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
+++ echo 0
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
+++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 3560.886773] nvme nvme0: 48/0/0 default/read/poll queues
[ 3562.872451] nvmet: adding nsid 1 to subsystem testnqn
[ 3563.883217] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bc654437-83be-4628-91bd-a439b8e830d5.
[ 3563.883353] nvme nvme1: creating 48 I/O queues.
[ 3563.887931] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][80.0%][r=99.2MiB/s][r=25.4k IOPS][eta 00m:03s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=27715: Wed Feb 2 00:03:19 2022
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)
slat (nsec): min=1032, max=1447.4k, avg=8870.69, stdev=27575.11
clat (usec): min=11, max=4669, avg=153.54, stdev=131.98
lat (usec): min=47, max=4672, avg=162.54, stdev=137.66
clat percentiles (usec):
| 1.00th=[ 62], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 106],
| 30.00th=[ 113], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 125],
| 70.00th=[ 135], 80.00th=[ 198], 90.00th=[ 221], 95.00th=[ 251],
| 99.00th=[ 709], 99.50th=[ 947], 99.90th=[ 1663], 99.95th=[ 2114],
| 99.99th=[ 2933]
write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);
slat (usec): min=2, max=3897, avg=13.26, stdev=21.42
clat (usec): min=15, max=5720, avg=129.39, stdev=79.93
lat (usec): min=37, max=6857, avg=142.77, stdev=84.05
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 77], 10.00th=[ 84], 20.00th=[ 96],
| 30.00th=[ 105], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 129],
| 70.00th=[ 137], 80.00th=[ 151], 90.00th=[ 169], 95.00th=[ 215],
| 99.00th=[ 273], 99.50th=[ 310], 99.90th=[ 1057], 99.95th=[ 1696],
| 99.99th=[ 2868]
bw ( KiB/s): min=82496, max=143312, per=98.04%, avg=109226.13, stdev=16354.90, samples=15
iops : min=20624, max=35828, avg=27306.67, stdev=4088.95, samples=15
lat (usec) : 20=0.01%, 50=0.28%, 100=19.29%, 250=77.38%, 500=2.09%
lat (usec) : 750=0.52%, 1000=0.20%
lat (msec) : 2=0.19%, 4=0.05%, 10=0.01%
cpu : usr=18.65%, sys=16.14%, ctx=141033, majf=0, minf=3053
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=94.9MiB/s (99.5MB/s), 94.9MiB/s-94.9MiB/s (99.5MB/s-99.5MB/s), io=506MiB (531MB), run=5335-5335msec
WRITE: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=800MiB (839MB), run=7353-7353msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=253MiB/s][w=64.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27744: Wed Feb 2 00:04:20 2022
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
slat (nsec): min=1072, max=42904k, avg=162051.61, stdev=467637.32
clat (nsec): min=752, max=72679k, avg=3588602.81, stdev=2243011.38
lat (usec): min=52, max=72681, avg=3750.87, stdev=2268.70
clat percentiles (usec):
| 1.00th=[ 562], 5.00th=[ 1029], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
| 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
| 99.00th=[11076], 99.50th=[12780], 99.90th=[18220], 99.95th=[21890],
| 99.99th=[30278]
bw ( KiB/s): min=193029, max=335346, per=100.00%, avg=273269.63, stdev=877.09, samples=3808
iops : min=48248, max=83826, avg=68310.92, stdev=219.29, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
lat (usec) : 100=0.04%, 250=0.09%, 500=0.46%, 750=1.38%, 1000=2.46%
lat (msec) : 2=18.84%, 4=43.56%, 10=31.38%, 20=1.55%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=0.63%, sys=1.57%, ctx=1935887, majf=0, minf=485
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4093399,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=251MiB/s][w=64.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27840: Wed Feb 2 00:05:20 2022
write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
slat (nsec): min=1052, max=52671k, avg=146009.86, stdev=449344.58
clat (nsec): min=641, max=64251k, avg=3581258.82, stdev=2231979.80
lat (usec): min=55, max=64976, avg=3727.49, stdev=2255.52
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
| 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
| 99.00th=[10945], 99.50th=[12649], 99.90th=[18220], 99.95th=[21627],
| 99.99th=[30016]
bw ( KiB/s): min=203172, max=338441, per=100.00%, avg=275054.71, stdev=839.49, samples=3808
iops : min=50788, max=84602, avg=68758.55, stdev=209.87, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.15%
lat (usec) : 100=0.03%, 250=0.08%, 500=0.45%, 750=1.40%, 1000=2.44%
lat (msec) : 2=18.96%, 4=43.62%, 10=31.29%, 20=1.51%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=0.63%, sys=1.56%, ctx=1832905, majf=0, minf=488
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4119099,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=15.7GiB (16.9GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=270MiB/s][w=69.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27949: Wed Feb 2 00:06:20 2022
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
slat (nsec): min=1082, max=41371k, avg=222396.07, stdev=539362.31
clat (nsec): min=631, max=67998k, avg=3529440.50, stdev=2240119.90
lat (usec): min=54, max=69361, avg=3752.10, stdev=2276.78
clat percentiles (usec):
| 1.00th=[ 529], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3523],
| 70.00th=[ 4113], 80.00th=[ 4883], 90.00th=[ 6194], 95.00th=[ 7570],
| 99.00th=[11076], 99.50th=[12780], 99.90th=[20055], 99.95th=[23987],
| 99.99th=[31327]
bw ( KiB/s): min=195270, max=336343, per=100.00%, avg=273029.60, stdev=880.99, samples=3808
iops : min=48813, max=84075, avg=68253.36, stdev=220.24, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.20%
lat (usec) : 100=0.05%, 250=0.13%, 500=0.52%, 750=1.45%, 1000=2.48%
lat (msec) : 2=19.07%, 4=44.45%, 10=30.09%, 20=1.45%, 50=0.10%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.69%, ctx=2408175, majf=0, minf=508
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4091921,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28031: Wed Feb 2 00:07:20 2022
write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
slat (nsec): min=1062, max=50611k, avg=206709.24, stdev=515844.58
clat (nsec): min=491, max=58217k, avg=3538847.51, stdev=2211954.38
lat (usec): min=46, max=58793, avg=3745.81, stdev=2241.07
clat percentiles (usec):
| 1.00th=[ 529], 5.00th=[ 1020], 10.00th=[ 1352], 20.00th=[ 1844],
| 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4883], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10945], 99.50th=[12649], 99.90th=[18744], 99.95th=[22414],
| 99.99th=[30540]
bw ( KiB/s): min=205666, max=333724, per=100.00%, avg=273520.75, stdev=810.61, samples=3808
iops : min=51408, max=83423, avg=68374.75, stdev=202.66, samples=3808
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.21%
lat (usec) : 100=0.05%, 250=0.12%, 500=0.50%, 750=1.42%, 1000=2.45%
lat (msec) : 2=18.99%, 4=44.21%, 10=30.54%, 20=1.43%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=0.66%, sys=1.67%, ctx=2291683, majf=0, minf=481
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4098881,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.6GiB (16.8GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28274: Wed Feb 2 00:08:21 2022
write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
slat (nsec): min=1062, max=68056k, avg=176848.85, stdev=488062.07
clat (nsec): min=692, max=77359k, avg=3532992.98, stdev=2215070.84
lat (usec): min=42, max=78568, avg=3710.06, stdev=2245.78
clat percentiles (usec):
| 1.00th=[ 545], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10814], 99.50th=[12518], 99.90th=[18482], 99.95th=[22414],
| 99.99th=[30540]
bw ( KiB/s): min=201233, max=338661, per=100.00%, avg=276170.71, stdev=819.64, samples=3808
iops : min=50302, max=84658, avg=69037.91, stdev=204.91, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
lat (usec) : 100=0.04%, 250=0.11%, 500=0.50%, 750=1.46%, 1000=2.53%
lat (msec) : 2=19.26%, 4=43.92%, 10=30.52%, 20=1.41%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.61%, ctx=2102892, majf=0, minf=494
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4138405,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=269MiB/s (283MB/s), 269MiB/s-269MiB/s (283MB/s-283MB/s), io=15.8GiB (16.9GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=277MiB/s][w=71.0k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29446: Wed Feb 2 00:09:21 2022
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
slat (nsec): min=1052, max=50297k, avg=138125.49, stdev=435715.13
clat (nsec): min=521, max=59414k, avg=3537010.84, stdev=2183018.58
lat (usec): min=51, max=61090, avg=3675.34, stdev=2211.08
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 1029], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10683], 99.50th=[12256], 99.90th=[17433], 99.95th=[21365],
| 99.99th=[30540]
bw ( KiB/s): min=206519, max=341031, per=100.00%, avg=278664.74, stdev=817.73, samples=3808
iops : min=51622, max=85253, avg=69660.15, stdev=204.42, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.12%
lat (usec) : 100=0.03%, 250=0.08%, 500=0.47%, 750=1.43%, 1000=2.51%
lat (msec) : 2=19.26%, 4=43.83%, 10=30.86%, 20=1.34%, 50=0.06%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.55%, ctx=1803573, majf=0, minf=455
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4177602,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=15.9GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29556: Wed Feb 2 00:10:21 2022
write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
slat (nsec): min=1052, max=35550k, avg=79344.28, stdev=323613.65
clat (nsec): min=771, max=53581k, avg=3588937.03, stdev=2176963.25
lat (usec): min=58, max=53842, avg=3668.46, stdev=2181.80
clat percentiles (usec):
| 1.00th=[ 586], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10683], 99.50th=[12125], 99.90th=[16057], 99.95th=[18744],
| 99.99th=[27395]
bw ( KiB/s): min=204774, max=339176, per=100.00%, avg=279266.54, stdev=802.82, samples=3808
iops : min=51191, max=84786, avg=69810.42, stdev=200.71, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.13%
lat (usec) : 100=0.03%, 250=0.06%, 500=0.44%, 750=1.37%, 1000=2.49%
lat (msec) : 2=18.96%, 4=43.01%, 10=32.04%, 20=1.42%, 50=0.04%
lat (msec) : 100=0.01%
cpu : usr=0.62%, sys=1.45%, ctx=1356660, majf=0, minf=493
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4185468,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=272MiB/s (286MB/s), 272MiB/s-272MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][98.4%][w=280MiB/s][w=71.7k IOPS][eta 00m:01s]
Jobs: 32 (f=32): [w(32)][100.0%][w=277MiB/s][w=70.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29643: Wed Feb 2 00:11:21 2022
write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
slat (nsec): min=1052, max=34378k, avg=59061.75, stdev=277238.20
clat (nsec): min=742, max=47279k, avg=3608224.03, stdev=2171757.46
lat (usec): min=48, max=47679, avg=3667.45, stdev=2172.85
clat percentiles (usec):
| 1.00th=[ 594], 5.00th=[ 1045], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10683], 99.50th=[12125], 99.90th=[15664], 99.95th=[17433],
| 99.99th=[23725]
bw ( KiB/s): min=201486, max=341296, per=100.00%, avg=279357.72, stdev=825.73, samples=3808
iops : min=50369, max=85319, avg=69833.71, stdev=206.44, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11%
lat (usec) : 100=0.03%, 250=0.05%, 500=0.42%, 750=1.37%, 1000=2.47%
lat (msec) : 2=18.87%, 4=42.63%, 10=32.55%, 20=1.47%, 50=0.02%
cpu : usr=0.60%, sys=1.42%, ctx=1195286, majf=0, minf=481
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4186642,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29748: Wed Feb 2 00:12:22 2022
write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
slat (nsec): min=1072, max=25141k, avg=14918.87, stdev=115673.29
clat (nsec): min=752, max=37983k, avg=3564960.33, stdev=2091487.49
lat (usec): min=57, max=37985, avg=3580.03, stdev=2089.00
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1057], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7570],
| 99.00th=[10421], 99.50th=[11600], 99.90th=[14615], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min=223459, max=340302, per=100.00%, avg=286170.92, stdev=703.56, samples=3808
iops : min=55856, max=85068, avg=71536.67, stdev=175.90, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.37%, 750=1.33%, 1000=2.48%
lat (msec) : 2=19.22%, 4=43.12%, 10=32.14%, 20=1.26%, 50=0.01%
cpu : usr=0.61%, sys=1.38%, ctx=872668, majf=0, minf=472
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4288951,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 2 (f=0): [_(12),f(2),_(18)][100.0%][w=266MiB/s][w=68.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29827: Wed Feb 2 00:13:22 2022
write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);
slat (nsec): min=1062, max=32432k, avg=19937.63, stdev=148066.81
clat (nsec): min=811, max=59246k, avg=3585601.00, stdev=2115468.91
lat (usec): min=52, max=59254, avg=3605.70, stdev=2113.21
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14877], 99.95th=[16188],
| 99.99th=[19792]
bw ( KiB/s): min=222503, max=339437, per=100.00%, avg=284148.08, stdev=711.74, samples=3808
iops : min=55619, max=84852, avg=71031.03, stdev=177.94, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.29%, 1000=2.44%
lat (msec) : 2=18.96%, 4=43.25%, 10=32.27%, 20=1.33%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.61%, sys=1.42%, ctx=918578, majf=0, minf=478
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4258348,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=30964: Wed Feb 2 00:14:22 2022
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)
slat (nsec): min=971, max=173108k, avg=39325.53, stdev=826441.69
clat (nsec): min=1894, max=316249k, avg=42166512.42, stdev=14952004.11
lat (usec): min=417, max=316251, avg=42205.96, stdev=15032.61
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 59], 95.00th=[ 68],
| 99.00th=[ 91], 99.50th=[ 99], 99.90th=[ 121], 99.95th=[ 140],
| 99.99th=[ 201]
bw ( KiB/s): min=142106, max=403311, per=100.00%, avg=291270.21, stdev=902.19, samples=5664
iops : min=35512, max=100811, avg=72801.07, stdev=225.55, samples=5664
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.05%, 4=0.22%, 10=1.02%, 20=3.07%, 50=76.23%
lat (msec) : 100=18.97%, 250=0.42%, 500=0.01%
cpu : usr=0.24%, sys=0.55%, ctx=261137, majf=0, minf=3757
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4368373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=300MiB/s][r=76.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31076: Wed Feb 2 00:15:23 2022
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)
slat (nsec): min=961, max=88836k, avg=17391.79, stdev=551047.29
clat (usec): min=22, max=221605, avg=42951.57, stdev=12630.75
lat (usec): min=383, max=222644, avg=42969.08, stdev=12673.43
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 32],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 63],
| 99.00th=[ 84], 99.50th=[ 93], 99.90th=[ 112], 99.95th=[ 123],
| 99.99th=[ 150]
bw ( KiB/s): min=170354, max=369108, per=100.00%, avg=286058.08, stdev=686.26, samples=5664
iops : min=42573, max=92262, avg=71497.39, stdev=171.56, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%, 10=0.36%, 20=1.78%, 50=77.37%
lat (msec) : 100=20.13%, 250=0.27%
cpu : usr=0.22%, sys=0.53%, ctx=218767, majf=0, minf=3743
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4290648,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=299MiB/s][r=76.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31174: Wed Feb 2 00:16:23 2022
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)
slat (nsec): min=941, max=113553k, avg=20605.09, stdev=608598.15
clat (usec): min=20, max=244312, avg=45449.46, stdev=13495.74
lat (usec): min=78, max=244315, avg=45470.18, stdev=13549.72
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 34],
| 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47],
| 70.00th=[ 50], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 66],
| 99.00th=[ 92], 99.50th=[ 101], 99.90th=[ 120], 99.95th=[ 129],
| 99.99th=[ 163]
bw ( KiB/s): min=147119, max=347537, per=100.00%, avg=270244.42, stdev=628.01, samples=5664
iops : min=36762, max=86866, avg=67544.04, stdev=157.00, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.08%, 10=0.40%, 20=1.57%, 50=70.55%
lat (msec) : 100=26.85%, 250=0.52%
cpu : usr=0.21%, sys=0.51%, ctx=217032, majf=0, minf=3833
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4054701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=264MiB/s (277MB/s), 264MiB/s-264MiB/s (277MB/s-277MB/s), io=15.5GiB (16.6GB), run=60033-60033msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31271: Wed Feb 2 00:17:23 2022
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)
slat (nsec): min=952, max=94814k, avg=15742.78, stdev=457046.68
clat (usec): min=38, max=202717, avg=41217.82, stdev=11676.36
lat (usec): min=65, max=202721, avg=41233.69, stdev=11708.54
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 56], 95.00th=[ 61],
| 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 105], 99.95th=[ 112],
| 99.99th=[ 140]
bw ( KiB/s): min=182011, max=362829, per=100.00%, avg=298134.45, stdev=603.05, samples=5664
iops : min=45487, max=90694, avg=74515.97, stdev=150.76, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.06%, 10=0.30%, 20=1.80%, 50=82.63%
lat (msec) : 100=15.05%, 250=0.14%
cpu : usr=0.23%, sys=0.55%, ctx=232008, majf=0, minf=3740
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4471696,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.3GB), run=60035-60035msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.2k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31366: Wed Feb 2 00:18:23 2022
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)
slat (nsec): min=952, max=131203k, avg=21974.73, stdev=596139.49
clat (usec): min=3, max=270950, avg=42409.20, stdev=13195.67
lat (usec): min=80, max=270961, avg=42431.29, stdev=13247.16
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 64],
| 99.00th=[ 86], 99.50th=[ 93], 99.90th=[ 109], 99.95th=[ 120],
| 99.99th=[ 161]
bw ( KiB/s): min=163161, max=388414, per=100.00%, avg=289753.58, stdev=752.52, samples=5664
iops : min=40771, max=97088, avg=72421.87, stdev=188.12, samples=5664
lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.14%, 10=0.67%, 20=2.25%, 50=77.40%
lat (msec) : 100=19.26%, 250=0.25%, 500=0.01%
cpu : usr=0.23%, sys=0.53%, ctx=232559, majf=0, minf=3758
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4345429,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=288MiB/s][r=73.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32520: Wed Feb 2 00:19:24 2022
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)
slat (nsec): min=961, max=155112k, avg=23329.58, stdev=600906.79
clat (usec): min=18, max=269118, avg=42721.27, stdev=13343.53
lat (usec): min=109, max=269121, avg=42744.72, stdev=13395.21
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 64],
| 99.00th=[ 88], 99.50th=[ 96], 99.90th=[ 113], 99.95th=[ 125],
| 99.99th=[ 176]
bw ( KiB/s): min=155002, max=377376, per=100.00%, avg=287593.81, stdev=736.00, samples=5664
iops : min=38739, max=94329, avg=71881.32, stdev=184.00, samples=5664
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.10%, 10=0.54%, 20=2.09%, 50=77.36%
lat (msec) : 100=19.56%, 250=0.33%, 500=0.01%
cpu : usr=0.22%, sys=0.54%, ctx=235545, majf=0, minf=3694
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4313415,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=16.5GiB (17.7GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32630: Wed Feb 2 00:20:24 2022
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)
slat (nsec): min=962, max=177671k, avg=38019.32, stdev=644096.49
clat (usec): min=29, max=287028, avg=41615.50, stdev=13238.93
lat (usec): min=94, max=319545, avg=41653.64, stdev=13291.04
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 57], 95.00th=[ 63],
| 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 112], 99.95th=[ 123],
| 99.99th=[ 176]
bw ( KiB/s): min=156946, max=373773, per=100.00%, avg=295110.41, stdev=718.25, samples=5664
iops : min=39217, max=93429, avg=73760.01, stdev=179.56, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.11%, 10=0.60%, 20=2.37%, 50=80.04%
lat (msec) : 100=16.54%, 250=0.31%, 500=0.01%
cpu : usr=0.23%, sys=0.56%, ctx=327850, majf=0, minf=3771
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4426283,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60035-60035msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=292MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32752: Wed Feb 2 00:21:24 2022
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)
slat (nsec): min=972, max=125387k, avg=20090.72, stdev=496977.06
clat (nsec): min=1563, max=210556k, avg=42435419.14, stdev=12289751.96
lat (usec): min=123, max=210559, avg=42455.63, stdev=12326.24
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 63],
| 99.00th=[ 82], 99.50th=[ 91], 99.90th=[ 108], 99.95th=[ 116],
| 99.99th=[ 146]
bw ( KiB/s): min=169789, max=367924, per=100.00%, avg=289626.85, stdev=669.34, samples=5664
iops : min=42433, max=91970, avg=72388.99, stdev=167.32, samples=5664
lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.07%, 10=0.34%, 20=1.74%, 50=79.17%
lat (msec) : 100=18.44%, 250=0.22%
cpu : usr=0.23%, sys=0.52%, ctx=248980, majf=0, minf=3751
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4343055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60039-60039msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32851: Wed Feb 2 00:22:25 2022
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)
slat (nsec): min=961, max=88402k, avg=16580.98, stdev=499828.64
clat (usec): min=16, max=190979, avg=42147.65, stdev=12291.43
lat (usec): min=76, max=200180, avg=42164.34, stdev=12331.85
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 46], 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 62],
| 99.00th=[ 83], 99.50th=[ 92], 99.90th=[ 108], 99.95th=[ 116],
| 99.99th=[ 142]
bw ( KiB/s): min=170567, max=369399, per=100.00%, avg=291617.48, stdev=663.74, samples=5664
iops : min=42628, max=92333, avg=72887.66, stdev=165.93, samples=5664
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.07%, 10=0.39%, 20=1.84%, 50=79.83%
lat (msec) : 100=17.63%, 250=0.22%
cpu : usr=0.23%, sys=0.53%, ctx=222791, majf=0, minf=3753
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4372949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=285MiB/s (298MB/s), 285MiB/s-285MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32948: Wed Feb 2 00:23:25 2022
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)
slat (nsec): min=982, max=143449k, avg=14518.04, stdev=453275.74
clat (usec): min=39, max=211504, avg=41151.34, stdev=11534.04
lat (usec): min=227, max=230931, avg=41165.97, stdev=11567.45
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 55], 95.00th=[ 60],
| 99.00th=[ 79], 99.50th=[ 87], 99.90th=[ 103], 99.95th=[ 112],
| 99.99th=[ 148]
bw ( KiB/s): min=182147, max=368337, per=100.00%, avg=298660.44, stdev=576.72, samples=5664
iops : min=45520, max=92066, avg=74647.02, stdev=144.17, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.07%, 10=0.33%, 20=1.75%, 50=83.09%
lat (msec) : 100=14.62%, 250=0.13%
cpu : usr=0.23%, sys=0.54%, ctx=233134, majf=0, minf=3732
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4479122,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=17.1GiB (18.3GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
--
2.29.0
More information about the Linux-nvme
mailing list