[PATCH 0/6] block: add support for REQ_OP_VERIFY
Chaitanya Kulkarni
kch at nvidia.com
Thu Jun 30 02:14:00 PDT 2022
Hi,
This adds support for the REQ_OP_VERIFY. In this version we add
support for block layer. NVMe host side, NVMeOF Block device backend,
and NVMeOF File device backend and null_blk driver.
In this version we also add a new blkverify userspace tool along with
the testcases patch for the util-linux, this patch will be followed
by the this series.
Below is the summary of testlog :-
1. NVMeOF bdev-ns null_blk verify=0 (triggering bdev emulation code) :-
-----------------------------------------------------------------------
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 1171.171536] nvmet: nvmet_bdev_emulate_verify_work 467
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 1199.322161] nvmet: nvmet_bdev_emulate_verify_work 467
2. NVMeOF bdev-ns null_blk verify=1.
-----------------------------------------------------------------------
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 1257.661548] nvmet: nvmet_bdev_execute_verify 506
[ 1257.661558] null_blk: null_process_cmd 1406
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 1269.613415] nvmet: nvmet_bdev_execute_verify 506
[ 1269.613425] null_blk: null_process_cmd 1406
3. NVMeOF file-ns :-
-----------------------------------------------------------------------
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 3452.675959] nvme_setup_verify 844
[ 3452.675969] nvmet: nvmet_file_execute_verify 525
[ 3452.675971] nvmet: nvmet_file_emulate_verify_work 502
[ 3452.675972] nvmet: do_direct_io_emulate_verify 431
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 3459.794385] nvmet: nvmet_file_execute_verify 525
[ 3459.794389] nvmet: nvmet_file_emulate_verify_work 502
[ 3459.794391] nvmet: do_direct_io_emulate_verify 431
4. NVMe PCIe device.
-----------------------------------------------------------------------
linux-block (for-next) # modprobe nvme
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme0n1
linux-block (for-next) # dmesg -c
[ 2763.432194] nvme nvme0: pci function 0000:00:04.0
[ 2763.473827] nvme nvme0: 48/0/0 default/read/poll queues
[ 2763.478868] nvme nvme0: Ignoring bogus Namespace Identifiers
[ 2766.583923] nvme_setup_verify 844
Here is a link for the complete cover-letter for the background to save
reviewer's time :-
https://patchwork.kernel.org/project/dm-devel/cover/20211104064634.4481-1-chaitanyak@nvidia.com/
-ck
Chaitanya Kulkarni (6):
block: add support for REQ_OP_VERIFY
nvme: add support for the Verify command
nvmet: add Verify command support for bdev-ns
nvmet: add Verify emulation support for bdev-ns
nvmet: add verify emulation support for file-ns
null_blk: add REQ_OP_VERIFY support
Documentation/ABI/stable/sysfs-block | 12 +++
block/blk-core.c | 5 +
block/blk-lib.c | 155 +++++++++++++++++++++++++++
block/blk-merge.c | 18 ++++
block/blk-settings.c | 17 +++
block/blk-sysfs.c | 8 ++
block/blk.h | 4 +
block/ioctl.c | 35 ++++++
drivers/block/null_blk/main.c | 20 +++-
drivers/block/null_blk/null_blk.h | 1 +
drivers/nvme/host/core.c | 33 ++++++
drivers/nvme/target/admin-cmd.c | 3 +-
drivers/nvme/target/core.c | 14 ++-
drivers/nvme/target/io-cmd-bdev.c | 75 +++++++++++++
drivers/nvme/target/io-cmd-file.c | 152 ++++++++++++++++++++++++++
drivers/nvme/target/nvmet.h | 4 +-
include/linux/bio.h | 9 +-
include/linux/blk_types.h | 2 +
include/linux/blkdev.h | 22 ++++
include/linux/nvme.h | 19 ++++
include/uapi/linux/fs.h | 1 +
21 files changed, 601 insertions(+), 8 deletions(-)
--
2.29.0
######################## NVMeOF bdev-ns null_blk verify=0 ######################
0 directories, 0 files
linux-block (for-next) # ./bdev.sh 1
++ FILE=/dev/nvme0n1
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe null_blk nr_devices=0 verify=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│  └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
++ sleep 1
++ echo transport=loop,nqn=testnqn
+++ shuf -i 1-1 -n 1
++ for i in `shuf -i 1-$NN -n $NN`
++ mkdir config/nullb/nullb1
++ echo 4096
++ echo 20971520
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ let IDX=IDX+1
++ echo ' ####### /dev/nullb1'
####### /dev/nullb1
++ echo -n /dev/nullb1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nullb1
++ echo 1
++ dmesg -c
[ 1150.985720] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:7f5b83f1-b258-4300-9a55-cd1902bea8c2.
[ 1150.985882] nvme nvme1: creating 48 I/O queues.
[ 1150.990654] nvme nvme1: new ctrl: "testnqn"
[ 1150.995681] null_blk: disk nullb1 created
[ 1150.998886] nvmet: adding nsid 1 to subsystem testnqn
[ 1151.000716] nvme nvme1: rescanning namespaces.
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 1171.171536] nvmet: nvmet_bdev_emulate_verify_work 467
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 1199.322161] nvmet: nvmet_bdev_emulate_verify_work 467
linux-block (for-next) # ./delete.sh 1
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.343s
user 0m0.002s
sys 0m0.003s
++ shuf -i 1-1 -n 1
+ for i in `shuf -i 1-$NN -n $NN`
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir config/nullb/nullb1
+ sleep 2
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ sleep 1
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ sleep 1
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
######################## NVMeOF bdev-ns null_blk verify=1 #####################
linux-block (for-next) # ./bdev.sh 1
++ FILE=/dev/nvme0n1
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe null_blk nr_devices=0 verify=1
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│  └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
++ sleep 1
++ echo transport=loop,nqn=testnqn
+++ shuf -i 1-1 -n 1
++ for i in `shuf -i 1-$NN -n $NN`
++ mkdir config/nullb/nullb1
++ echo 4096
++ echo 20971520
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ let IDX=IDX+1
++ echo ' ####### /dev/nullb1'
####### /dev/nullb1
++ echo -n /dev/nullb1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nullb1
++ echo 1
++ dmesg -c
[ 1250.311632] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:328bb18d-3662-48eb-8bc2-ca7d4ad73799.
[ 1250.311853] nvme nvme1: creating 48 I/O queues.
[ 1250.316251] nvme nvme1: new ctrl: "testnqn"
[ 1250.321710] null_blk: disk nullb1 created
[ 1250.324678] nvmet: adding nsid 1 to subsystem testnqn
[ 1250.326546] nvme nvme1: rescanning namespaces.
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 1257.661548] nvmet: nvmet_bdev_execute_verify 506
[ 1257.661558] null_blk: null_process_cmd 1406
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 1269.613415] nvmet: nvmet_bdev_execute_verify 506
[ 1269.613425] null_blk: null_process_cmd 1406
linux-block (for-next) #
linux-block (for-next) # ./delete.sh 1
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.339s
user 0m0.002s
sys 0m0.003s
++ shuf -i 1-1 -n 1
+ for i in `shuf -i 1-$NN -n $NN`
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir config/nullb/nullb1
+ sleep 2
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ sleep 1
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ sleep 1
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
################################### NVMeOF file-ns #############################
linux-block (for-next) #
linux-block (for-next) # ./file.sh 1
++ FILE=/mnt/backend/nvme1n1
++ SS=testnqn
++ SSPATH=/sys/kernel/config/nvmet/subsystems/testnqn/
++ PORTS=/sys/kernel/config/nvmet/ports
++ main
++ load_modules
++ dmesg -c
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ sleep 3
++ mount_fs
++ make_nullb
++ ./compile_nullb.sh
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block modules
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/5.18.0blk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.1M Jun 29 17:06 /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c
++ ./compile_nullb.sh
+ umount /mnt/nullb0
umount: /mnt/nullb0: not mounted.
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ dmesg -c
+ modprobe -r null_blk
+ lsmod
+ grep null_blk
++ nproc
+ make -j 48 M=drivers/block modules
+ HOST=drivers/block/null_blk/
++ uname -r
+ HOST_DEST=/lib/modules/5.18.0blk+/kernel/drivers/block/null_blk/
+ cp drivers/block/null_blk//null_blk.ko /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//
+ ls -lrth /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko
-rw-r--r--. 1 root root 1.1M Jun 29 17:06 /lib/modules/5.18.0blk+/kernel/drivers/block/null_blk//null_blk.ko
+ sleep 1
+ dmesg -c
++ modprobe null_blk nr_devices=0
++ sleep 1
++ mkdir config/nullb/nullb0
++ tree config/nullb/nullb0
config/nullb/nullb0
├── badblocks
├── blocking
├── blocksize
├── cache_size
├── completion_nsec
├── discard
├── home_node
├── hw_queue_depth
├── index
├── irqmode
├── max_sectors
├── mbps
├── memory_backed
├── poll_queues
├── power
├── queue_mode
├── size
├── submit_queues
├── use_per_node_hctx
├── verify
├── virt_boundary
├── zone_capacity
├── zoned
├── zone_max_active
├── zone_max_open
├── zone_nr_conv
└── zone_size
0 directories, 27 files
++ echo 1
++ echo 512
++ echo 20480
++ echo 1
++ sleep 2
+++ cat config/nullb/nullb0/index
++ IDX=0
++ lsblk
++ grep null0
++ sleep 1
++ mkfs.xfs -f /dev/nullb0
meta-data=/dev/nullb0 isize=512 agcount=4, agsize=1310720 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=0
data = bsize=4096 blocks=5242880, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
++ mount /dev/nullb0 /mnt/backend/
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ dd if=/dev/zero of=/mnt/backend/nvme1n1 count=2621440 bs=4096
2621440+0 records in
2621440+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 5.01608 s, 2.1 GB/s
++ file /mnt/backend/nvme1n1
/mnt/backend/nvme1n1: data
++ make_target
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│  ├── features
│  └── nullb0
│  ├── badblocks
│  ├── blocking
│  ├── blocksize
│  ├── cache_size
│  ├── completion_nsec
│  ├── discard
│  ├── home_node
│  ├── hw_queue_depth
│  ├── index
│  ├── irqmode
│  ├── max_sectors
│  ├── mbps
│  ├── memory_backed
│  ├── poll_queues
│  ├── power
│  ├── queue_mode
│  ├── size
│  ├── submit_queues
│  ├── use_per_node_hctx
│  ├── verify
│  ├── virt_boundary
│  ├── zone_capacity
│  ├── zoned
│  ├── zone_max_active
│  ├── zone_max_open
│  ├── zone_nr_conv
│  └── zone_size
└── nvmet
├── hosts
├── ports
└── subsystems
6 directories, 28 files
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/
++ for i in 1
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1
++ echo -n /mnt/backend/nvme1n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1/device_path
/mnt/backend/nvme1n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn//namespaces/1/buffered_io
0
++ echo 1
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn/ /sys/kernel/config/nvmet/ports/1/subsystems/
++ sleep 1
++ connect
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ dmesg -c
[ 3436.671070] null_blk: module loaded
[ 3437.678812] null_blk: disk nullb0 created
[ 3440.700250] XFS (nullb0): Mounting V5 Filesystem
[ 3440.701686] XFS (nullb0): Ending clean mount
[ 3440.701772] xfs filesystem being mounted at /mnt/backend supports timestamps until 2038 (0x7fffffff)
[ 3446.742777] nvmet: adding nsid 1 to subsystem testnqn
[ 3447.752282] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:55bf9c5a-2992-4ffc-ac53-18003029a0b9.
[ 3447.752423] nvme nvme1: creating 48 I/O queues.
[ 3447.758869] nvme nvme1: new ctrl: "testnqn"
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme1n1
linux-block (for-next) # dmesg -c
[ 3452.675959] nvme_setup_verify 844
[ 3452.675969] nvmet: nvmet_file_execute_verify 525
[ 3452.675971] nvmet: nvmet_file_emulate_verify_work 502
[ 3452.675972] nvmet: do_direct_io_emulate_verify 431
linux-block (for-next) # nvme verify -s 0 -c 1024 /dev/nvme1n1
NVME Verify Success
linux-block (for-next) # dmesg -c
[ 3459.794385] nvmet: nvmet_file_execute_verify 525
[ 3459.794389] nvmet: nvmet_file_emulate_verify_work 502
[ 3459.794391] nvmet: do_direct_io_emulate_verify 431
linux-block (for-next) #
linux-block (for-next) #
linux-block (for-next) # ./delete.sh 1
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.338s
user 0m0.001s
sys 0m0.004s
++ shuf -i 1-1 -n 1
+ for i in `shuf -i 1-$NN -n $NN`
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir config/nullb/nullb0
+ sleep 2
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ sleep 1
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ sleep 1
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
################################# NVMe PCIe device ###########################
linux-block (for-next) #
linux-block (for-next) #
linux-block (for-next) # modprobe nvme
linux-block (for-next) # blkverify -o 0 -l 40960 /dev/nvme0n1
linux-block (for-next) # dmesg -c
[ 2763.432194] nvme nvme0: pci function 0000:00:04.0
[ 2763.473827] nvme nvme0: 48/0/0 default/read/poll queues
[ 2763.478868] nvme nvme0: Ignoring bogus Namespace Identifiers
[ 2766.583923] nvme_setup_verify 844
More information about the Linux-nvme
mailing list