"nvme nvmeX: IO queues not created" with "Amazon.com, Inc. NVMe SSD Controller" from 5.19.y (issue bisected)

[Dutta, Soumyaroop]

Hello,

We apologize for the delayed reply. Our Investigation and testing of the proposed fix had concluded since then. 
We Identified a latent issue in our NVMe Controller firmware that, coupled with the recent changes in Linux 5.19 resulted into the IO queues not being created after controller enablement.
We have identified the fix, and have it tested through different versions. The new release, which includes this fix is now under deployment, and should be available across the AWS fleet over the next few weeks.

Thanks
Soumyaroop

On 9/20/22, 2:42 AM, "Thorsten Leemhuis" <regressions at leemhuis.info> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    Hi, this is your Linux kernel regression tracker.

    On 07.09.22 11:58, Sironi, Filippo wrote:
    > Adding more folks (Soumyaroop, Valentin, and Zorik) involved in the investigation, which is by now concluded.

    Has any progress been made to get this regression fixed? I might be
    missing something, but from here it looks like nothing happened since
    two weeks. Thing is: ideally it shouldn't take this long to fix
    regressions in production releases, as explained in
    https://docs.kernel.org/process/handling-regressions.html

    Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)

    P.S.: As the Linux kernel's regression tracker I deal with a lot of
    reports and sometimes miss something important when writing mails like
    this. If that's the case here, don't hesitate to tell me in a public
    reply, it's in everyone's interest to set the public record straight.

    #regzbot poke

    > On 07.09.22, 11:13, "Sironi, Filippo" <sironi at amazon.de> wrote:
    >
    >     I'm aware of a customer contact regarding this issue and I know that the team responsible for the NVMe implementation is investigating.
    >     Adding Aviv since he's closer to this space than I am and may have more insights into how the investigation is progressing.
    >
    >     On 07.09.22, 08:30, "Christoph Hellwig" <hch at lst.de> wrote:
    >
    >         CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
    >
    >
    >
    >         Filippo,
    >
    >         can you help figuring out what is going on with this Amazon controller?
    >
    >         On Wed, Sep 07, 2022 at 08:05:05AM +0200, Jaroslav Pulchart wrote:
    >         > Hello,
    >         >
    >         > I would like to report a regression issue in 5.19.y in NVMe driver.
    >         >
    >         > The issue is reproducible at AWS EC2 instances with local NVMe storage
    >         > like "r5d.*". Kernel report "IO queues not created":
    >         > [    2.936641] nvme nvme0: 2/0/0 default/read/poll queues  <- EBS volume
    >         > [    2.939493] nvme nvme1: 2/0/0 default/read/poll queues  <- EBS volume
    >         > [    2.940797] nvme nvme2: IO queues not created              <- Local volume
    >         > with 5.19.y (kernel 5.19) and the nvme storage cannot be used.
    >         >
    >         > I bisected the issue to commit
    >         > "aa41d2fe60ee2e4452b0f9ca9f0f6d80a4ff9f9d" (nvme: set controller
    >         > enable bit in a separate write). Reverting it makes the nvme device
    >         > working again:
    >         > [    3.025599] nvme nvme0: 2/0/0 default/read/poll queues
    >         > [    3.032467] nvme nvme2: 8/0/0 default/read/poll queues
    >         > [    3.040040] nvme nvme1: 2/0/0 default/read/poll queues
    >         >
    >         > Best,
    >         > --
    >         > Jaroslav Pulchart
    >         > Sr. Principal SW Engineer
    >         > GoodData
    >         ---end quoted text---
    >
    >
    >
    >
    >
    > Amazon Development Center Germany GmbH
    > Krausenstr. 38
    > 10117 Berlin
    > Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
    > Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
    > Sitz: Berlin
    > Ust-ID: DE 289 237 879
    >
    >