[External] : Re: [bug-report] 5-9% FIO randomwrite ext4 perf regression on 6.12.y kernel

[Saeed Mirzamohammadi]

FYI, Tried disabling write zeros but still getting the same errors:
[ 326.097275] operation not supported error, dev nvme2n1, sector 10624 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
[ 338.496217] nvme0n1: Dataset Management(0x9) @ LBA 10928, 256 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
… 

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index d3bde17c818d5..ad2ce6008062e 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3425,7 +3425,8 @@ static const struct pci_device_id nvme_id_table[] = {
                .driver_data = NVME_QUIRK_STRIPE_SIZE |
                                NVME_QUIRK_DEALLOCATE_ZEROES |
                                NVME_QUIRK_IGNORE_DEV_SUBNQN |
-                               NVME_QUIRK_BOGUS_NID, },
+                               NVME_QUIRK_BOGUS_NID |
+                               NVME_QUIRK_DISABLE_WRITE_ZEROES, },
        { PCI_VDEVICE(INTEL, 0x0a55),   /* Dell Express Flash P4600 */
                .driver_data = NVME_QUIRK_STRIPE_SIZE |
                                NVME_QUIRK_DEALLOCATE_ZEROES, },


This is for:
        { PCI_VDEVICE(INTEL, 0x0a54),   /* Intel P4500/P4600 */

And as we can see from lspci:
$ lspci -nn | grep -i nvme 19:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
20:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
94:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
9b:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]


> On Nov 22, 2024, at 9:18 AM, Paul Webb <paul.x.webb at oracle.com> wrote:
> 
> 
> On 22/11/2024 12:13, Christoph Hellwig wrote:
>> On Thu, Nov 21, 2024 at 09:07:32PM +0000, Paul Webb wrote:
>>> Christoph:
>>> To check for weird lazy init code using write zeroes
>>> 
>>> Values in the 5.15 kernel baseline prior to the commit:
>>> $ cat /sys/block/nvme*n1/queue/write_zeroes_max_bytes
>>> 0
>>> 0
>>> 0
>>> 0
>>> 
>>> Values in the 6.11 kernel that contains the commit:
>>> $ cat /sys/block/nvme*n1/queue/write_zeroes_max_bytes
>>> 2199023255040
>>> 2199023255040
>>> 2199023255040
>>> 2199023255040
>> Thanks!  So 6.11 actually enables write zeroes for your controller.
>> 
>>> Another interesting datapoint is that while performing some runs I am
>>> seeing the following output on the console in the 6.11/6.12 kernels that
>>> contain the commit:
>>> 
>>> [  473.398188] operation not supported error, dev nvme2n1, sector 13952 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
>> .. which it doesn't handle well.
>> 
>>> [  473.534550] nvme0n1: Dataset Management(0x9) @ LBA 14000, 256 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
>> .. and interesting this is for a Deallocate, which should only happen
>> with the quirk for certain Intel controllers from the very first days of
>> nvme.
>> 
>> What controller do you have?  Can you post the output of lspci and
>> "nvme list"?
> 
> Hi Christoph,
> 
> The nvme related output from lspci is as follows:
> $ lspci | grep -i nvme
> 19:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 20:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 94:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 9b:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 
> 
> $ sudo nvme list
> Node                  Generic               SN Model                                    Namespace Usage                      Format           FW Rev
> --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
> /dev/nvme0n1          /dev/ng0n1            PHLN942100EQ6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme1n1          /dev/ng1n1            PHLN942100PE6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme2n1          /dev/ng2n1            PHLN9415002B6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme3n1          /dev/ng3n1            PHLN942100DQ6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
>