"nvme nvmeX: IO queues not created" with "Amazon.com, Inc. NVMe SSD Controller" from 5.19.y (issue bisected)

[Christoph Hellwig]

Filippo,

can you help figuring out what is going on with this Amazon controller?

On Wed, Sep 07, 2022 at 08:05:05AM +0200, Jaroslav Pulchart wrote:
> Hello,
> 
> I would like to report a regression issue in 5.19.y in NVMe driver.
> 
> The issue is reproducible at AWS EC2 instances with local NVMe storage
> like "r5d.*". Kernel report "IO queues not created":
> [    2.936641] nvme nvme0: 2/0/0 default/read/poll queues  <- EBS volume
> [    2.939493] nvme nvme1: 2/0/0 default/read/poll queues  <- EBS volume
> [    2.940797] nvme nvme2: IO queues not created              <- Local volume
> with 5.19.y (kernel 5.19) and the nvme storage cannot be used.
> 
> I bisected the issue to commit
> "aa41d2fe60ee2e4452b0f9ca9f0f6d80a4ff9f9d" (nvme: set controller
> enable bit in a separate write). Reverting it makes the nvme device
> working again:
> [    3.025599] nvme nvme0: 2/0/0 default/read/poll queues
> [    3.032467] nvme nvme2: 8/0/0 default/read/poll queues
> [    3.040040] nvme nvme1: 2/0/0 default/read/poll queues
> 
> Best,
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData
---end quoted text---