RK3576 Rock 4D: PCIe NVMe device not detected in mainline kernel despite working in U-Boot

[Leonard Lausen]

Hi linux-rockchip, Hi Nicolas,

I'm testing mainline kernel support on the Rock 4D board and 
encountering an issue where PCIe NVMe devices are not detected by the 
kernel, despite working correctly in U-Boot. That is despite Nicolas 
recently enabling PCIe support for Rock 4D in 
29ff4bbff793334d6aff2238fdc3ccf3859d60da.

U-Boot successfully detects and initializes the NVMe device:

pcie at 2a200000: PCIe Linking... LTSSM is 0x0
pcie at 2a200000: PCIe Link up, LTSSM is 0x130011
pcie at 2a200000: PCIE-0: Link up (Gen2-x1, Bus0)

Device 0: Vendor: 0x1344 Rev: HPS0050  Prod: UTFRQ0170J8QSZ
             Type: Hard Disk
             Capacity: 488386.3 MB = 476.9 GB (1000215216 x 512)


However, after kernel boot, only the PCIe root port is visible:

00:00.0 PCI bridge: Device 1d87:3576 (rev 01)


PCIe rescan debug output:

(initramfs) echo 1 > /sys/bus/pci/rescan
[  106.782392] pci_bus 0000:00: scanning bus
[  106.796249] pcieport 0000:00:00.0: restore config 0x2c: 0x00000000 -> 
0x00000000
[  106.797060] pcieport 0000:00:00.0: restore config 0x28: 0x00000000 -> 
0x00000000
[  106.797808] pcieport 0000:00:00.0: restore config 0x24: 0x0001fff1 -> 
0x0001fff1
[  106.798583] pcieport 0000:00:00.0: PME# disabled
[  106.799069] pcieport 0000:00:00.0: scanning [bus 01-ff] behind 
bridge, pass 0
[  106.799795] pci_bus 0000:01: scanning bus
[  106.803471] pci_bus 0000:01: bus scan returning with max=01
[  106.804073] pcieport 0000:00:00.0: scanning [bus 01-ff] behind 
bridge, pass 1
[  106.804829] pci_bus 0000:00: bus scan returning with max=ff
(initramfs) [  106.924154] pcieport 0000:00:00.0: save config 0x00: 
0x35761d87
[  106.924765] pcieport 0000:00:00.0: save config 0x04: 0x00100507
[  106.925364] pcieport 0000:00:00.0: save config 0x08: 0x06040001
[  106.925958] pcieport 0000:00:00.0: save config 0x0c: 0x00010000
[  106.926534] pcieport 0000:00:00.0: save config 0x10: 0x00000000
[  106.932142] pcieport 0000:00:00.0: save config 0x14: 0x00000000
[  106.934608] pcieport 0000:00:00.0: save config 0x18: 0x00ff0100
[  106.937206] pcieport 0000:00:00.0: save config 0x1c: 0x000000f0
[  106.939575] pcieport 0000:00:00.0: save config 0x20: 0x0000fff0
[  106.941883] pcieport 0000:00:00.0: save config 0x24: 0x0001fff1
[  106.944138] pcieport 0000:00:00.0: save config 0x28: 0x00000000
[  106.946335] pcieport 0000:00:00.0: save config 0x2c: 0x00000000
[  106.948490] pcieport 0000:00:00.0: save config 0x30: 0x00000000
[  106.950601] pcieport 0000:00:00.0: save config 0x34: 0x00000040
[  106.952698] pcieport 0000:00:00.0: save config 0x38: 0x00000000
[  106.954765] pcieport 0000:00:00.0: save config 0x3c: 0x0002012d
[  106.956847] pcieport 0000:00:00.0: PME# enabled
ls /sys/bus/pci/devices/
drwxr-xr-x 2 0 0 0 .
drwxr-xr-x 5 0 0 0 ..
lrwxrwxrwx 1 0 0 0 0000:00:00.0 -> 
../../../devices/platform/soc/22000000.pcie/pci0000:00/0000:00:00.0

The full boot log including the u-boot part and kernel PCI_DEBUG output 
is also attached.

Thanks for any insights!

Best regards,
Leonard