blktests failures with v6.10-rc1 kernel

[Shinichiro Kawasaki]

Hi all,

(I added linux-rdma list to the To list since blktests nvme and srp groups
 depend on rdma drivers.)

I ran the latest blktests (git hash: 698f1a024cb4) with the v6.10-rc1 kernel,
and observed a couple of failures as listed below.

There are two notable differences from the result with kernel v6.9-rc1 [1].
The first one is srp/002,011 hangs with the rdma rxe driver, which was discussed
at LSF 2024. I no longer observe these hangs with v6.10-rc1 kernel. Great :) I
found Bob Pearson made a number of improvements in the driver. I guess these
changes avoided the hangs. Thank you very much!

The other difference is nbd/002 failure. CKI project still reports it for
v6.10-rc1 kernel [2]. Recently Josef provided blktests side fix [3] (Thanks!),
and it has not yet applied to the CKI test run set up. The fix was made for
nbd/001, but I expect that it will avoid the nbd/002 failure also.

[1] https://lore.kernel.org/linux-block/m6a437jvfwzq2jfytvvk62zpgu7e4bjvegr7x73pihhkp5me5c@sh6vs3s7w754/
[2] https://datawarehouse.cki-project.org/kcidb/tests/12631448
[3] https://lore.kernel.org/linux-block/9377610cbdc3568c172cd7c5d2e9d36da8dd2cf4.1716312272.git.josef@toxicpanda.com/


List of failures
================
#1: nvme/041 (fc transport)
#2: nvme/050

Failure description
===================

#1: nvme/041 (fc transport)

   With the trtype=fc configuration, nvme/041 fails:

  nvme/041 (Create authenticated connections)                  [failed]
      runtime  2.677s  ...  4.823s
      --- tests/nvme/041.out      2023-11-29 12:57:17.206898664 +0900
      +++ /home/shin/Blktests/blktests/results/nodev/nvme/041.out.bad     2024-03-19 14:50:56.399101323 +0900
      @@ -2,5 +2,5 @@
       Test unauthenticated connection (should fail)
       disconnected 0 controller(s)
       Test authenticated connection
      -disconnected 1 controller(s)
      +disconnected 0 controller(s)
       Test complete

   nvme/044 had same failure symptom until the kernel v6.9. A solution was
   suggested and discussed in Feb/2024 [4].

   [4] https://lore.kernel.org/linux-nvme/20240221132404.6311-1-dwagner@suse.de/

#2: nvme/050

   The test case fails occasionally with a QEMU NVME device. The failure cause
   is the lockdep WARN among ctrl->namespaces_rwsem, dev->shutdown_lock and
   workqueue work completion. After LSF 2024 discussion, Sagi and Keith worked
   on the solution and Keith provided the fix [5]. Thank you!

   [5] https://lore.kernel.org/linux-nvme/20240524155345.243814-1-kbusch@meta.com/