[LSF/MM/BPF TOPIC] Implementation of Cross-controller reset/Time-based recovery (CQT) in the linux nvme driver

[Randy Jennings]

Here is the presentation for next week:
https://drive.google.com/file/d/1IOcakxMCKC8KKrBl4YRWfzojP1_jFoal/

Current posted patchset:
https://lore.kernel.org/linux-nvme/20260328004518.1729186-1-mkhalfella@purestorage.com/T/

The target should be built with the code described here:
https://lore.kernel.org/linux-nvme/20260430232913.129271-1-randyj@purestorage.com/
The last patch should be included to fail CCR, excluded for normal operation

To conduct the test:
* on the target, enable cqt (if testing with cqt enabled)
    * echo 1 > /sys/kernel/config/nvmet/subsystems/mysubsystem/attr_cqt
* on the initiator, connect to the nvme device twice
    * the second time needs  --duplicate-connect if the ip address is the same
    * make sure the queue policy is round-robin
        * to make sure both paths get IO
* on the target, set the fatal opcode and delay time:
    * echo '1 2000' > /sys/kernel/debug/nvmet/mysubsystem/ctrl1/fopcode
* on the initiator, run the test program, rw_test
    * the test program is hardcoded to use /dev/nvme0n1

test program (courtesy of Mohamed Khalfella): rw_test.c:
#include <string.h>
#include <malloc.h>
#include <errno.h>

#define BUF_SIZE        4096
#define ITERATIONS      255
#define WRITE_COUNT     10

#define WRITE_OFFSET    0
#define READ_OFFSET     WRITE_OFFSET

int main() {
        int fd, i, w, off, ret;
        char *buff;

        fd = open("/dev/nvme0n1", O_RDWR | O_DIRECT);
        if (fd < 0) {
                fprintf(stderr, "failed to open device, errno = %d\n", errno);
                return 1;
        }

        ret = posix_memalign((void **)&buff, BUF_SIZE, BUF_SIZE);
        if (ret) {
                fprintf(stderr, "failed to allocate buffer, ret = %d\n", ret);
                return 1;
        }

        for (i = 0; i < 255; i++) {
                fprintf(stdout, "iteration number %d, writing data\n", i);

                for (w = 0; w < WRITE_COUNT; w++) {
                        memset(buff, w, BUF_SIZE);
                        ret = pwrite(fd, buff, BUF_SIZE, WRITE_OFFSET);
                        if (ret != BUF_SIZE) {
                                fprintf(stderr, "failed to write buff, "
                                                "ret = %d, errno = %d\n",
                                                ret, errno);
                                exit(1);
                        }
                }

                sleep(5);
                fprintf(stdout, "validating written data\n");

                ret = pread(fd, buff, BUF_SIZE, READ_OFFSET);
                if (ret != BUF_SIZE) {
                        fprintf(stderr, "failed to read buff, "
                                        "ret = %d, errno = %d\n",
                                        ret, errno);
                        exit(1);
                }

                for (off = 0; off < BUF_SIZE; off++) {
                        if (buff[off] != WRITE_COUNT - 1) {
                                fprintf(stderr, "validation failed\n");
                                exit(1);
                        }
                }

                fprintf(stdout, "successfully validated\n");
                sleep(3);
        }

        return 0;
}

On Fri, Feb 13, 2026 at 5:09 PM Randy Jennings <randyj at purestorage.com> wrote:
>
> I'd like to discuss the state of patchsets related to implementing
> Cross-controller Reset and Time-based recovery (CQT) in the linux nvme
> driver for fabrics transports.  I expect to present performance test
> results that show the impact on read, read/write, write workloads with
> packet drop and storage controller failover scenarios for different
> levels of implementation (none, CQT only, CCR+CQT, filtering some
> outstanding requests (probably just reads, maybe some admin
> requests)).
>
> Current patchset is:
> https://lore.kernel.org/all/35ae10c9-808a-4882-86dc-311b23c821e5@grimberg.me/
> Although, I expect it will be updated shortly.
>
> Sincerely,
> Randy Jennings