Using the generic host PCIe driver

Fri Mar 3 09:18:02 PST 2017

On 03/03/2017 16:46, Bjorn Helgaas wrote:

> On Fri, Mar 03, 2017 at 01:44:54PM +0100, Mason wrote:
>
>> For now, I have "hidden" the root's BAR0 from the system with:
>>
>> 	if (bus->number == 0 && where == PCI_BASE_ADDRESS_0) {
>> 		*val = 0;
>> 		return PCIBIOS_SUCCESSFUL;
>> 	}
> 
> I'm scratching my head about this a little.  Here's what your dmesg
> log contained originally:
> 
>   pci 0000:00:00.0: [1105:8758] type 01 class 0x048000
>   pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
>   pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000 64bit]
>   pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000 64bit]
>   pci 0000:00:00.0: PCI bridge to [bus 01]
>   pcieport 0000:00:00.0: enabling device (0140 -> 0142)
> 
> This device is a bridge (a Root Port, per your lspci output).  With a
> BAR, which is legal but unusual.  We couldn't assign space for the
> BAR, which means we can't use whatever vendor-specific functionality
> it provides.

I had several chats with the HW designer. I'll try to explain, only as
far as I could understand ;-)

We used to make devices, before implementing a root. Since at least
one BAR is required (?) for a device, it was decided to have one BAR
for the root, for symmetry.

In fact, I thought I could ignore that BAR, but it is apparently NOT
the case, as MSIs are supposed to be sent *within* the BAR of the root.
So I have removed my kludge hiding the BAR.

The weird twist is that the BAR advertizes a 64-bit memory zone,
but we will, in fact, map MMIO registers behind it. So all the
RAM Linux assigns to the area is wasted, IIUC.

Did the above make sense?

[    0.986762] OF: PCI: host bridge /soc/pcie at 50000000 ranges:
[    0.992478] OF: PCI:   No bus range found for /soc/pcie at 50000000, using [bus 00-ff]
[    1.000279] OF: PCI: Parsing ranges property...
[    1.004938] OF: PCI:   MEM 0x90000000..0x9fffffff -> 0x90000000
[    1.014047] pci-host-generic 50000000.pcie: ECAM at [mem 0x50000000-0x5fffffff] for [bus 00-ff]
[    1.023088] pci-host-generic 50000000.pcie: PCI host bridge to bus 0000:00
[    1.030112] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.035729] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffffff]
[    1.042737] pci_bus 0000:00: scanning bus
[    1.046895] pci 0000:00:00.0: [1105:0024] type 01 class 0x048000
[    1.053050] pci 0000:00:00.0: calling tango_pcie_fixup_class+0x0/0x10
[    1.059639] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
[    1.066583] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x40
[    1.072929] pci 0000:00:00.0: supports D1 D2
[    1.077318] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
[    1.083452] pci 0000:00:00.0: PME# disabled
[    1.087901] pci_bus 0000:00: fixups for bus
[    1.092212] PCI: bus0: Fast back to back transfers disabled
[    1.097913] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
[    1.104746] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[    1.112893] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
[    1.119800] pci_bus 0000:01: scanning bus
[    1.123972] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
[    1.130144] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
[    1.137147] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x40
[    1.143577] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[    1.149801] pci 0000:01:00.0: PME# disabled
[    1.154364] pci_bus 0000:01: fixups for bus
[    1.158671] PCI: bus1: Fast back to back transfers disabled
[    1.164368] pci_bus 0000:01: bus scan returning with max=01
[    1.170067] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.176814] pci_bus 0000:00: bus scan returning with max=01
[    1.182511] pci 0000:00:00.0: fixup irq: got 0
[    1.187072] pci 0000:00:00.0: assigning IRQ 00
[    1.191658] pci 0000:01:00.0: fixup irq: got 20
[    1.196307] pci 0000:01:00.0: assigning IRQ 20
[    1.200892] pci 0000:00:00.0: BAR 0: assigned [mem 0x90000000-0x90ffffff 64bit]
[    1.208344] pci 0000:00:00.0: BAR 8: assigned [mem 0x91000000-0x910fffff]
[    1.215267] pci 0000:01:00.0: BAR 0: assigned [mem 0x91000000-0x91001fff 64bit]
[    1.222726] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.227812] pci 0000:00:00.0:   bridge window [mem 0x91000000-0x910fffff]
[    1.234781] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
[    1.241013] pcieport 0000:00:00.0: enabling bus mastering
[    1.246634] pci 0000:01:00.0: calling quirk_usb_early_handoff+0x0/0x790
[    1.253389] pci 0000:01:00.0: enabling device (0140 -> 0142)
[    1.275215] pci 0000:01:00.0: xHCI HW did not halt within 16000 usec status = 0xd2f4167e

# /usr/sbin/lspci -v
00:00.0 PCI bridge: Sigma Designs, Inc. Device 0024 (rev 01) (prog-if 00 [Normal decode])
        Flags: bus master, fast devsel, latency 0
        Memory at 90000000 (64-bit, non-prefetchable) [size=16M]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00000000-00000fff
        Memory behind bridge: 91000000-910fffff
        Prefetchable memory behind bridge: 00000000-000fffff
        Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+
        Capabilities: [78] Power Management version 3
        Capabilities: [80] Express Root Port (Slot-), MSI 03
        Capabilities: [100] Virtual Channel
        Capabilities: [800] Advanced Error Reporting
        Kernel driver in use: pcieport

01:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
        Flags: fast devsel, IRQ 20
        Memory at 91000000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 3
        Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [90] MSI-X: Enable- Count=8 Masked-
        Capabilities: [a0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Latency Tolerance Reporting

> What's puzzling me is that pcieport was able to enable the device and
> turn on PCI_COMMAND_MEMORY (the 0x2 bit).  It seems like this should
> have failed because pci_enable_resources() checks to see that all the
> BARs have been assigned.
> 
> Since this is a bridge, we really *have* to turn on PCI_COMMAND_MEMORY
> in order for the bridge to forward memory transactions to its
> secondary bus (bus 01).  But we can't safely enable PCI_COMMAND_MEMORY
> unless all its memory BARs are assigned.
> 
> So it's not safe to hide BAR0 from the PCI core.  That makes Linux
> think the BAR doesn't exist, but of course it still exists in the
> hardware itself, and it will respond at whatever address it happens to
> contain.  In this case, that address happens to be zero, and the host
> bridge does not advertise a window that maps to bus address zero, so
> you probably won't see a conflict right away, but it's a latent issue
> that may come back to bite you some day.

OK, no more hiding! :-)

> The easiest fix would be for you to increase the host bridge memory
> window size.  It's currently [mem 0xa0000000-0xa03fffff], which is
> only 4MB, which is *tiny*.  You need 16MB just to contain the bridge
> BAR, plus at least 8K for the USB controller.
> 
> If you can't make space for a bigger window, it's possible the Root
> Port has some device-specific way to disable BAR0 in hardware, e.g.,
> some register your firmware or an early Linux quirk could write.  That
> would be much safer than enabling an unassigned BAR.

Apparently, disabling BAR0 is not an option.

Regards.