"nvme nvmeX: IO queues not created" with "Amazon.com, Inc. NVMe SSD Controller" from 5.19.y (issue bisected)

[Sironi, Filippo]

I'm aware of a customer contact regarding this issue and I know that the team responsible for the NVMe implementation is investigating.
Adding Aviv since he's closer to this space than I am and may have more insights into how the investigation is progressing. 

On 07.09.22, 08:30, "Christoph Hellwig" <hch at lst.de> wrote:

    CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



    Filippo,

    can you help figuring out what is going on with this Amazon controller?

    On Wed, Sep 07, 2022 at 08:05:05AM +0200, Jaroslav Pulchart wrote:
    > Hello,
    >
    > I would like to report a regression issue in 5.19.y in NVMe driver.
    >
    > The issue is reproducible at AWS EC2 instances with local NVMe storage
    > like "r5d.*". Kernel report "IO queues not created":
    > [    2.936641] nvme nvme0: 2/0/0 default/read/poll queues  <- EBS volume
    > [    2.939493] nvme nvme1: 2/0/0 default/read/poll queues  <- EBS volume
    > [    2.940797] nvme nvme2: IO queues not created              <- Local volume
    > with 5.19.y (kernel 5.19) and the nvme storage cannot be used.
    >
    > I bisected the issue to commit
    > "aa41d2fe60ee2e4452b0f9ca9f0f6d80a4ff9f9d" (nvme: set controller
    > enable bit in a separate write). Reverting it makes the nvme device
    > working again:
    > [    3.025599] nvme nvme0: 2/0/0 default/read/poll queues
    > [    3.032467] nvme nvme2: 8/0/0 default/read/poll queues
    > [    3.040040] nvme nvme1: 2/0/0 default/read/poll queues
    >
    > Best,
    > --
    > Jaroslav Pulchart
    > Sr. Principal SW Engineer
    > GoodData
    ---end quoted text---




Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879