"nvme nvmeX: IO queues not created" with "Amazon.com, Inc. NVMe SSD Controller" from 5.19.y (issue bisected)

[Thorsten Leemhuis]

Hi, this is your Linux kernel regression tracker.

On 07.09.22 11:58, Sironi, Filippo wrote:
> Adding more folks (Soumyaroop, Valentin, and Zorik) involved in the investigation, which is by now concluded.

Has any progress been made to get this regression fixed? I might be
missing something, but from here it looks like nothing happened since
two weeks. Thing is: ideally it shouldn't take this long to fix
regressions in production releases, as explained in
https://docs.kernel.org/process/handling-regressions.html

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

P.S.: As the Linux kernel's regression tracker I deal with a lot of
reports and sometimes miss something important when writing mails like
this. If that's the case here, don't hesitate to tell me in a public
reply, it's in everyone's interest to set the public record straight.

#regzbot poke

> On 07.09.22, 11:13, "Sironi, Filippo" <sironi at amazon.de> wrote:
> 
>     I'm aware of a customer contact regarding this issue and I know that the team responsible for the NVMe implementation is investigating.
>     Adding Aviv since he's closer to this space than I am and may have more insights into how the investigation is progressing. 
> 
>     On 07.09.22, 08:30, "Christoph Hellwig" <hch at lst.de> wrote:
> 
>         CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
>         Filippo,
> 
>         can you help figuring out what is going on with this Amazon controller?
> 
>         On Wed, Sep 07, 2022 at 08:05:05AM +0200, Jaroslav Pulchart wrote:
>         > Hello,
>         >
>         > I would like to report a regression issue in 5.19.y in NVMe driver.
>         >
>         > The issue is reproducible at AWS EC2 instances with local NVMe storage
>         > like "r5d.*". Kernel report "IO queues not created":
>         > [    2.936641] nvme nvme0: 2/0/0 default/read/poll queues  <- EBS volume
>         > [    2.939493] nvme nvme1: 2/0/0 default/read/poll queues  <- EBS volume
>         > [    2.940797] nvme nvme2: IO queues not created              <- Local volume
>         > with 5.19.y (kernel 5.19) and the nvme storage cannot be used.
>         >
>         > I bisected the issue to commit
>         > "aa41d2fe60ee2e4452b0f9ca9f0f6d80a4ff9f9d" (nvme: set controller
>         > enable bit in a separate write). Reverting it makes the nvme device
>         > working again:
>         > [    3.025599] nvme nvme0: 2/0/0 default/read/poll queues
>         > [    3.032467] nvme nvme2: 8/0/0 default/read/poll queues
>         > [    3.040040] nvme nvme1: 2/0/0 default/read/poll queues
>         >
>         > Best,
>         > --
>         > Jaroslav Pulchart
>         > Sr. Principal SW Engineer
>         > GoodData
>         ---end quoted text---
> 
> 
> 
> 
> 
> Amazon Development Center Germany GmbH
> Krausenstr. 38
> 10117 Berlin
> Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
> Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
> Sitz: Berlin
> Ust-ID: DE 289 237 879
> 
>