NVMe CLI Invalid PRP Entry Status Failures

[Keith Busch]

On Sat, May 24, 2025 at 04:22:11AM +0000, Avinash M N wrote:
> The below function is causing the EINVAL to be returned to the userspace.
> 
> int blk_rq_append_bio(struct request *rq, struct bio *bio)
> {
>         const struct queue_limits *lim = &rq->q->limits;
>         unsigned int max_bytes = lim->max_hw_sectors << SECTOR_SHIFT;
>         unsigned int nr_segs = 0;
>         int ret;
> 
>         /* check that the data layout matches the hardware restrictions */
>         ret = bio_split_rw_at(bio, lim, &nr_segs, max_bytes);
>         if (ret) {
>                 /* if we would have to split the bio, copy instead */
>                 if (ret > 0) {
>                         ret = -EREMOTEIO;
>                 }
>                 return ret;
>         }
> 
> There is no issue if nvme-cli sends a transfer length of up to 128K.
> Anything more than 128K is failed as ENINVAL. I guess this is coming
> from the limitation of BIO_MAX_VECS as 256. Since this was working on
> older kernels, did anything change in this regard?

Well, it's passing a virtually contiguous address, so if we assume your
page size is 4k, 256 vectors would allow up to 1MB without a problem.

But the NVMe pci driver has its own limit of 128 vectors, so 512k is the
largest you can safely go before you need to increase the size of pages
via hugetlbfs.

But you say you're doing something smaller than 512k, and your device's
MDTS is bigger than that, so what else is limiting here your transfer
size here? Do you have some udev rule that is reducing the size of your
max sectors value?

Check the value of /sys/block/nvmeXnY/queue/max_sectors_kb

See if it matches /sys/block/nvmeXnY/queue/max_hw_sectors_kb

If not, echo the higher value into the "max_sectors_kb" attribute and
see if that fixes your problem.
 
> The failing command was attempting to transfer a length of ~320K. It
> seems that nvme-cli does not split the transfers and sends the full
> transfer length to the kernel.

Your command is vendor specific; nvme-cli has no idea what your command
does so it can't possibly know how to properly split it up. Try
something nvme-cli knows about, like a telemetry-log, and nvme-cli will
split it up for you.