Using the generic host PCIe driver

[Ard Biesheuvel]

On 3 March 2017 at 23:23, Mason <slash.tmp at free.fr> wrote:
> On 03/03/2017 21:04, Bjorn Helgaas wrote:
>> On Fri, Mar 03, 2017 at 06:18:02PM +0100, Mason wrote:
>>> On 03/03/2017 16:46, Bjorn Helgaas wrote:
>>>> On Fri, Mar 03, 2017 at 01:44:54PM +0100, Mason wrote:
>>>>
>>>>> For now, I have "hidden" the root's BAR0 from the system with:
>>>>>
>>>>>    if (bus->number == 0 && where == PCI_BASE_ADDRESS_0) {
>>>>>            *val = 0;
>>>>>            return PCIBIOS_SUCCESSFUL;
>>>>>    }
>>>>
>>>> I'm scratching my head about this a little.  Here's what your dmesg
>>>> log contained originally:
>>>>
>>>>   pci 0000:00:00.0: [1105:8758] type 01 class 0x048000
>>>>   pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
>>>>   pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000 64bit]
>>>>   pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000 64bit]
>>>>   pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>   pcieport 0000:00:00.0: enabling device (0140 -> 0142)
>>>>
>>>> This device is a bridge (a Root Port, per your lspci output).  With a
>>>> BAR, which is legal but unusual.  We couldn't assign space for the
>>>> BAR, which means we can't use whatever vendor-specific functionality
>>>> it provides.
>>>
>>> I had several chats with the HW designer. I'll try to explain, only as
>>> far as I could understand ;-)
>>>
>>> We used to make devices, before implementing a root. Since at least
>>> one BAR is required (?) for a device, it was decided to have one BAR
>>> for the root, for symmetry.
>>
>> I'm not aware of a spec requirement for any BARs.  It's conceivable
>> that one could build a device that only uses config space.  And of
>> course, most bridges have windows but no BARs.  But that doesn't
>> matter; the hardware is what it is and we have to deal with it.
>
> I appreciate the compassion. RMK considered the DMA HW too screwy
> to bother supporting ;-)
>
>>> In fact, I thought I could ignore that BAR, but it is apparently NOT
>>> the case, as MSIs are supposed to be sent *within* the BAR of the root.
>>
>> I don't know much about this piece of the MSI puzzle, but maybe Marc
>> can enlighten us.  If this Root Port is the target of MSIs and the
>> Root Port turns them into some sort of interrupt on the CPU side, I
>> can see how this might make sense.
>>
>> I think it's unusual for the PCI core to assign the MSI target using a
>> BAR, though.  I think this means you'll have to implement your
>> arch_setup_msi_irq() or .irq_compose_msi_msg() method such that it
>> looks up that BAR value, since you won't know it at build-time.
>
> I'll hack the Altera driver to fit my purpose.
>
>>> The weird twist is that the BAR advertizes a 64-bit memory zone,
>>> but we will, in fact, map MMIO registers behind it. So all the
>>> RAM Linux assigns to the area is wasted, IIUC.
>>
>> I'm not sure what this means.  You have this:
>>
>>> OF: PCI:   MEM 0x90000000..0x9fffffff -> 0x90000000
>
> This means I've put 256 MB of system RAM aside for PCIe devices.
> This memory is no longer available for Linux "stuff".
>

No it doesn't. It is a physical memory *range* that is assigned to the
PCI host bridge. Any memory accesses by the CPU to that window will be
forwarded to the PCI bus by the host bridge. From the kernel driver's
POV, this range is a given, but your host bridge h/w may involve some
configuration to make the host bridge 'listen' to this range. This is
h/w specific, and as Bjorn pointed out, usually configured by the
firmware so that the kernel driver does not require any knowledge of
those internals.

>>> pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffffff]
>
> I suppose this is the PCI bus address. As we've discussed,
> I used the identity to map bus <-> CPU addresses.
>

Yes, that is fine

>> This [mem 0x90000000-0x9fffffff] host bridge window means there can't
>> be RAM in that region.  CPU accesses to 0x90000000-0x9fffffff have to
>> be claimed by the host bridge and forwarded to PCI.
>>
>> Linux doesn't "assign system RAM" anywhere; we just learn somehow
>> where that RAM is.  Linux *does* assign BARs of PCI devices, and they
>> have to be inside the host bridge windows(s).
>
> I'm confused, I thought I had understood that part...
> I thought the binding required me to specify (in the "ranges"
> property) a non-prefetchable zone of system RAM, and this
> memory is then "handed out" by Linux to different devices.
> Or do I just need to specify some address range that's not
> necessarily backed with actual RAM?
>

Yes. Each PCI device advertises its need of memory windows via its
BARs, but the actual placement of those windows inside the host
bridge's memory range is configured dynamically, usually by the
firmware (on PCs) but on ARM/arm64 systems, this is done from scratch
by the kernel. The *purpose* of those memory windows is device
specific, but whatever is behind it lives on the PCI device. So this
is *not* system RAM.

>> The BARs may contain registers, RAM, frame buffers, etc., that live
>> on the PCI device.  It's totally up to the device what it is.
>>
>>> OF: PCI: host bridge /soc/pcie at 50000000 ranges:
>>> OF: PCI:   No bus range found for /soc/pcie at 50000000, using [bus 00-ff]
>>
>> Tangent: the lack of a bus range is a defect in your DTS.
>
> What range should I specify? 0-1? 0-2? 0-255?
>
>>> [    1.042737] pci_bus 0000:00: scanning bus
>>> [    1.046895] pci 0000:00:00.0: [1105:0024] type 01 class 0x048000
>>> [    1.053050] pci 0000:00:00.0: calling tango_pcie_fixup_class+0x0/0x10
>>> [    1.059639] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
>>> [    1.066583] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x40
>>> [    1.072929] pci 0000:00:00.0: supports D1 D2
>>> [    1.077318] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
>>> [    1.083452] pci 0000:00:00.0: PME# disabled
>>> [    1.087901] pci_bus 0000:00: fixups for bus
>>> [    1.092212] PCI: bus0: Fast back to back transfers disabled
>>> [    1.097913] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
>>> [    1.104746] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>>> [    1.112893] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
>>> [    1.119800] pci_bus 0000:01: scanning bus
>>> [    1.123972] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
>>> [    1.130144] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
>>> [    1.137147] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x40
>>> [    1.143577] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>>> [    1.149801] pci 0000:01:00.0: PME# disabled
>>> [    1.154364] pci_bus 0000:01: fixups for bus
>>> [    1.158671] PCI: bus1: Fast back to back transfers disabled
>>> [    1.164368] pci_bus 0000:01: bus scan returning with max=01
>>> [    1.170067] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
>>> [    1.176814] pci_bus 0000:00: bus scan returning with max=01
>>> [    1.182511] pci 0000:00:00.0: fixup irq: got 0
>>> [    1.187072] pci 0000:00:00.0: assigning IRQ 00
>>> [    1.191658] pci 0000:01:00.0: fixup irq: got 20
>>> [    1.196307] pci 0000:01:00.0: assigning IRQ 20
>>> [    1.200892] pci 0000:00:00.0: BAR 0: assigned [mem 0x90000000-0x90ffffff 64bit]
>>> [    1.208344] pci 0000:00:00.0: BAR 8: assigned [mem 0x91000000-0x910fffff]
>>> [    1.215267] pci 0000:01:00.0: BAR 0: assigned [mem 0x91000000-0x91001fff 64bit]
>>> [    1.222726] pci 0000:00:00.0: PCI bridge to [bus 01]
>>> [    1.227812] pci 0000:00:00.0:   bridge window [mem 0x91000000-0x910fffff]
>>> [    1.234781] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
>>> [    1.241013] pcieport 0000:00:00.0: enabling bus mastering
>>> [    1.246634] pci 0000:01:00.0: calling quirk_usb_early_handoff+0x0/0x790
>>> [    1.253389] pci 0000:01:00.0: enabling device (0140 -> 0142)
>>> [    1.275215] pci 0000:01:00.0: xHCI HW did not halt within 16000 usec status = 0xd2f4167e
>>>
>>> # /usr/sbin/lspci -v
>>> 00:00.0 PCI bridge: Sigma Designs, Inc. Device 0024 (rev 01) (prog-if 00 [Normal decode])
>>>         Flags: bus master, fast devsel, latency 0
>>>         Memory at 90000000 (64-bit, non-prefetchable) [size=16M]
>>>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>>>         I/O behind bridge: 00000000-00000fff
>>
>> Something's wrong with this.  You have no I/O windows through the host
>> bridge, which implies that you can't generic PCI I/O transactions, so
>> this I/O window should be disabled.  This might be an lspci issue;
>> what does "lspci -xxx" show?
>
> I'll look on Monday. But I know that this revision of the controller
> does not support any I/O areas. I don't know why Linux sees this.
> Might be a bug in the controller, or a missing init in my code.
>
>>>         Memory behind bridge: 91000000-910fffff
>>>         Prefetchable memory behind bridge: 00000000-000fffff
>>
>> This prefetchable memory window is bogus, too.  It should probably be
>> disabled.  If the bridge doesn't support a prefetchable window, the
>> base and limit should be hardwired to zero.  If it supports a window
>> but it's disabled, the limit should be less than the base.  For
>> example, on my system I see this for a bridge with the window
>> disabled:
>>
>>   # setpci -s00:1c.0 PREF_MEMORY_BASE
>>   fff1
>>   # setpci -s00:1c.0 PREF_MEMORY_LIMIT
>>   0001
>
> OK, one more thing for me to check. Thanks for your thoroughness.
>
> Regards.
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel