Data corruption when using multiple devices with NVMEoF TCP

[Sagi Grimberg]

> In my current setup, on the initiator side, nvme3n1 & nvme4n1 are 2
> nvme-tcp devices, schedulers for 3 is:
>   - cat /sys/block/nvme3n1/queue/scheduler: "none"
>   - cat /sys/block/nvme3c3n1/queue/scheduler: "[none] mq-deadline kyber"
> Not sure what is nvme3c3n1 here?
> 
> And disabling merges on nvme-tcp devices solves the data corruption issue!

Well, it actually confirms that we have an issue when we get a multi-bio
request that was merged. I'm assuming you also do not see the prints
I added in this case...

Would you mind adding these prints (they will overload probably, but
may be useful to shed some light on this):
--
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index 979ee31b8dd1..5a611ddb22ea 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -243,6 +243,16 @@ static void nvme_tcp_init_iter(struct 
nvme_tcp_request *req,
                 nsegs = bio_segments(bio);
                 size = bio->bi_iter.bi_size;
                 offset = bio->bi_iter.bi_bvec_done;
+               if (rq->bio != rq->biotail) {
+                       int bio_num = 1;
+                       struct bio *ptr = rq->bio;
+                       while (ptr != bio) {
+                               ptr = ptr->bi_next;
+                               bio_num++;
+                       };
+                       pr_info("rq %d (%s) data_len %d bio[%d/%d] 
sector %llx bvec: nsegs %d size %d offset %ld\n",
+                               rq->tag, dir == WRITE ? "WRITE" : 
"READ", req->data_len, bio_num, blk_rq_count_bios(rq), 
bio->bi_iter.bi_sector, nsegs, size, offset);
+               }
         }

         iov_iter_bvec(&req->iter, dir, vec, nsegs, size);
--

Thank you for helping isolating this issue.