[bug report] most of blktests nvme/ failed on the latest linux tree

Tue Jun 27 21:12:16 PDT 2023

On 6/27/23 19:58, Yi Zhang wrote:
> Hi Chaitanya
>
> Here is the code I used, and I will try to bisect it later today.
>
> # git remote -v
> origin https://github.com/torvalds/linux.git (fetch)
> origin https://github.com/torvalds/linux.git (push)
> # git log -1 --oneline
> 98be618ad030 (HEAD -> master, origin/master, origin/HEAD) Merge tag
> 'Smack-for-6.5' of https://github.com/cschaufler/smack-next
>
>

I ran blktests on the above repo and it is still passing for me see [1].

The only testcases are not run those are nvme/auth since it seemed to
have a bug when kernel is compiled with git hash for the sake of
debugging. dmesg log is also clean :-

linux (master) #
linux (master) # dmesg | grep -i same
linux (master) # dmesg | grep -i hostid
linux (master) # dmesg | grep -i "found same hostid"
linux (master) # dmesg | grep -i "found same hostid"
linux (master) #

None of the testcases you mentioned in your log are failing :(.

Waiting for your bisection result.

-ck

linux (master) # uname -r
6.4.0linux-01691-g98be618ad030

linux (master) # git remote -v
origin git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
(fetch)
origin git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 
(push)

linux (master) # git log -1 --oneline
98be618ad030 (HEAD -> master, origin/master, origin/HEAD) Merge tag 
'Smack-for-6.5' of https://github.com/cschaufler/smack-next

linux (master) # ./compile_nvme.sh
+ umount /mnt/nvme0n1
+ clear_dmesg
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ 
HOST_DEST=/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/host/
++ uname -r
+ 
TARGET_DEST=/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko 
drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko 
drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko 
/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko 
drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko 
drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko 
/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/target//
+ ls -lrth 
/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/host/ 
/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/target//
/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/host/:
total 6.6M
-rw-r--r--. 1 root root  2.6M Jun 27 20:53 nvme-core.ko
-rw-r--r--. 1 root root  491K Jun 27 20:53 nvme-fabrics.ko
-rw-r--r--. 1 root root 1000K Jun 27 20:53 nvme-fc.ko
-rw-r--r--. 1 root root  780K Jun 27 20:53 nvme.ko
-rw-r--r--. 1 root root  928K Jun 27 20:53 nvme-rdma.ko
-rw-r--r--. 1 root root  899K Jun 27 20:53 nvme-tcp.ko

/lib/modules/6.4.0linux-01691-g98be618ad030/kernel/drivers/nvme/target//:
total 6.6M
-rw-r--r--. 1 root root 541K Jun 27 20:53 nvme-fcloop.ko
-rw-r--r--. 1 root root 471K Jun 27 20:53 nvme-loop.ko
-rw-r--r--. 1 root root 826K Jun 27 20:53 nvmet-fc.ko
-rw-r--r--. 1 root root 3.1M Jun 27 20:53 nvmet.ko
-rw-r--r--. 1 root root 902K Jun 27 20:53 nvmet-rdma.ko
-rw-r--r--. 1 root root 760K Jun 27 20:53 nvmet-tcp.ko
+ modprobe nvme
+ dmesg -c
[   74.521561] nvme nvme0: pci function 0000:00:04.0
[   74.545758] nvme nvme0: 48/0/0 default/read/poll queues
[   74.548527] nvme nvme0: Ignoring bogus Namespace Identifiers

linux (master) # date
Tue Jun 27 08:53:51 PM PDT 2023

linux (master) # cdblktests
blktests (master) # ./check nvme
nvme/002 (create many subsystems and test discovery) [passed]
     runtime    ...  18.920s
nvme/003 (test if we're sending keep-alives to a discovery controller) 
[passed]
     runtime  10.087s  ...  10.080s
nvme/004 (test nvme and nvmet UUID NS descriptors) [passed]
     runtime  1.471s  ...  1.437s
nvme/005 (reset local loopback target) [passed]
     runtime  1.210s  ...  1.784s
nvme/006 (create an NVMeOF target with a block device-backed ns) [passed]
     runtime  0.055s  ...  0.058s
nvme/007 (create an NVMeOF target with a file-backed ns) [passed]
     runtime  0.045s  ...  0.030s
nvme/008 (create an NVMeOF host with a block device-backed ns) [passed]
     runtime  1.164s  ...  1.462s
nvme/009 (create an NVMeOF host with a file-backed ns) [passed]
     runtime  1.120s  ...  1.421s
nvme/010 (run data verification fio job on NVMeOF block device-backed 
ns) [passed]
     runtime  93.167s  ...  58.078s
nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed]
     runtime  81.600s  ...  89.008s
nvme/012 (run mkfs and data verification fio job on NVMeOF block 
device-backed ns) [passed]
     runtime  95.264s  ...  52.273s
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed 
ns) [passed]
     runtime  66.072s  ...  81.176s
nvme/014 (flush a NVMeOF block device-backed ns) [passed]
     runtime  6.727s  ...  7.060s
nvme/015 (unit test for NVMe flush for file backed ns) [passed]
     runtime  5.914s  ...  6.356s
nvme/016 (create/delete many NVMeOF block device-backed ns and test 
discovery) [passed]
     runtime    ...  12.591s
nvme/017 (create/delete many file-ns and test discovery) [passed]
     runtime    ...  12.933s
nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed]
     runtime  1.121s  ...  1.419s
nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed]
     runtime  1.161s  ...  1.421s
nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed]
     runtime  1.122s  ...  1.417s
nvme/021 (test NVMe list command on NVMeOF file-backed ns) [passed]
     runtime  1.123s  ...  1.404s
nvme/022 (test NVMe reset command on NVMeOF file-backed ns) [passed]
     runtime  1.172s  ...  1.740s
nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed]
     runtime  1.144s  ...  1.425s
nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed]
     runtime  1.122s  ...  1.427s
nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed]
     runtime  1.119s  ...  1.428s
nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed]
     runtime  1.114s  ...  1.424s
nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed]
     runtime  1.140s  ...  1.417s
nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed]
     runtime  1.117s  ...  1.424s
nvme/029 (test userspace IO via nvme-cli read/write interface) [passed]
     runtime  1.258s  ...  1.547s
nvme/030 (ensure the discovery generation counter is updated 
appropriately) [passed]
     runtime  0.137s  ...  0.207s
nvme/031 (test deletion of NVMeOF controllers immediately after setup) 
[passed]
     runtime  0.849s  ...  3.837s
nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed]
     runtime  0.016s  ...  0.012s
nvme/040 (test nvme fabrics controller reset/disconnect operation during 
I/O) [passed]
     runtime  7.273s  ...  8.021s
nvme/041 (Create authenticated connections)                  [not run]
     runtime  0.440s  ...
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/042 (Test dhchap key types for authenticated connections) [not run]
     runtime  2.712s  ...
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/043 (Test hash and DH group variations for authenticated 
connections) [not run]
     runtime  0.731s  ...
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/044 (Test bi-directional authentication)                [not run]
     runtime  1.240s  ...
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/045 (Test re-authentication)                            [not run]
     runtime  3.630s  ...
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
common/rc: line 212: 0inux: value too great for base (error token is 
"0inux")
nvme/048 (Test queue count changes on reconnect)             [not run]
     runtime  6.244s  ...
     nvme_trtype=loop is not supported in this test
blktests (master) # nvme_trtype=tcp ./check nvme
nvme/002 (create many subsystems and test discovery)         [not run]
     runtime  18.920s  ...
     nvme_trtype=tcp is not supported in this test
nvme/003 (test if we're sending keep-alives to a discovery controller) 
[passed]
     runtime  10.080s  ...  10.086s
nvme/004 (test nvme and nvmet UUID NS descriptors) [passed]
     runtime  1.437s  ...  1.140s
nvme/005 (reset local loopback target) [passed]
     runtime  1.784s  ...  1.188s
nvme/006 (create an NVMeOF target with a block device-backed ns) [passed]
     runtime  0.058s  ...  0.060s
nvme/007 (create an NVMeOF target with a file-backed ns) [passed]
     runtime  0.030s  ...  0.035s
nvme/008 (create an NVMeOF host with a block device-backed ns) [passed]
     runtime  1.462s  ...  1.152s
nvme/009 (create an NVMeOF host with a file-backed ns) [passed]
     runtime  1.421s  ...  1.137s
nvme/010 (run data verification fio job on NVMeOF block device-backed 
ns) [passed]
     runtime  58.078s  ...  60.132s
nvme/011 (run data verification fio job on NVMeOF file-backed ns) [passed]
     runtime  89.008s  ...  85.854s
nvme/012 (run mkfs and data verification fio job on NVMeOF block 
device-backed ns) [passed]
     runtime  52.273s  ...  54.886s
nvme/013 (run mkfs and data verification fio job on NVMeOF file-backed 
ns) [passed]
     runtime  81.176s  ...  80.462s
nvme/014 (flush a NVMeOF block device-backed ns) [passed]
     runtime  7.060s  ...  6.527s
nvme/015 (unit test for NVMe flush for file backed ns) [passed]
     runtime  6.356s  ...  5.831s
nvme/016 (create/delete many NVMeOF block device-backed ns and test 
discovery) [not run]
     runtime  12.591s  ...
     nvme_trtype=tcp is not supported in this test
nvme/017 (create/delete many file-ns and test discovery)     [not run]
     runtime  12.933s  ...
     nvme_trtype=tcp is not supported in this test
nvme/018 (unit test NVMe-oF out of range access on a file backend) [passed]
     runtime  1.419s  ...  1.121s
nvme/019 (test NVMe DSM Discard command on NVMeOF block-device ns) [passed]
     runtime  1.421s  ...  1.135s
nvme/020 (test NVMe DSM Discard command on NVMeOF file-backed ns) [passed]
     runtime  1.417s  ...  1.109s
nvme/021 (test NVMe list command on NVMeOF file-backed ns) [passed]
     runtime  1.404s  ...  1.108s
nvme/022 (test NVMe reset command on NVMeOF file-backed ns) [passed]
     runtime  1.740s  ...  1.154s
nvme/023 (test NVMe smart-log command on NVMeOF block-device ns) [passed]
     runtime  1.425s  ...  1.133s
nvme/024 (test NVMe smart-log command on NVMeOF file-backed ns) [passed]
     runtime  1.427s  ...  1.107s
nvme/025 (test NVMe effects-log command on NVMeOF file-backed ns) [passed]
     runtime  1.428s  ...  1.115s
nvme/026 (test NVMe ns-descs command on NVMeOF file-backed ns) [passed]
     runtime  1.424s  ...  1.111s
nvme/027 (test NVMe ns-rescan command on NVMeOF file-backed ns) [passed]
     runtime  1.417s  ...  1.133s
nvme/028 (test NVMe list-subsys command on NVMeOF file-backed ns) [passed]
     runtime  1.424s  ...  1.101s
nvme/029 (test userspace IO via nvme-cli read/write interface) [passed]
     runtime  1.547s  ...  1.239s
nvme/030 (ensure the discovery generation counter is updated 
appropriately) [passed]
     runtime  0.207s  ...  0.116s
nvme/031 (test deletion of NVMeOF controllers immediately after setup) 
[passed]
     runtime  3.837s  ...  0.767s
nvme/038 (test deletion of NVMeOF subsystem without enabling) [passed]
     runtime  0.012s  ...  0.015s
nvme/040 (test nvme fabrics controller reset/disconnect operation during 
I/O) [passed]
     runtime  8.021s  ...  7.278s
nvme/041 (Create authenticated connections)                  [not run]
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/042 (Test dhchap key types for authenticated connections) [not run]
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/043 (Test hash and DH group variations for authenticated 
connections) [not run]
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/044 (Test bi-directional authentication)                [not run]
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
nvme/045 (Test re-authentication)                            [not run]
     kernel 6.4.0linux-01691-g98be618ad030 config not found
     kernel 6.4.0linux-01691-g98be618ad030 config not found
common/rc: line 212: 0inux: value too great for base (error token is 
"0inux")
nvme/048 (Test queue count changes on reconnect) [passed]
     runtime    ...  5.230s
blktests (master) #