[bug report] nvme sends invalid command capsule over rdma transport for 5KiB write when target supports MSDBD > 1

[Walker, Benjamin]

This bug was found using the iozone tool.

- Linux kernel initiator, SPDK target, RDMA (RoCEv2) transport
- iozone is performing a 5KiB write to a 512 byte block size nvme device
- The SPDK target has reported that it supports 4KiB of in-capsule data, MSDBD of 16 (number of SGL descriptors), and ICDOFF of 0.
- The Linux kernel sends an NVMe-oF capsule with a command that claims to have 5KiB of data in the command, but actually only has a single SGL element describing 4KiB of data in-capsule.
- The SPDK target correctly fails this I/O

This fails on at least 5.11 but worked prior to 5.4. A git bisect shows that this commit is responsible: 38e1800275d3af607e4df92ff49dc2cf442586a4

I believe the key is the use of MSDBD > 1 and in-capsule data support. This seems to trick the initiator into thinking it can do 5KiB in one command with two SGL elements, but then the initiator goes down the in-capsule data path and can only describe 4KiB that way.