Using the generic host PCIe driver

[Mason]

On 03/03/2017 21:04, Bjorn Helgaas wrote:
> On Fri, Mar 03, 2017 at 06:18:02PM +0100, Mason wrote:
>> On 03/03/2017 16:46, Bjorn Helgaas wrote:
>>> On Fri, Mar 03, 2017 at 01:44:54PM +0100, Mason wrote:
>>>
>>>> For now, I have "hidden" the root's BAR0 from the system with:
>>>>
>>>> 	if (bus->number == 0 && where == PCI_BASE_ADDRESS_0) {
>>>> 		*val = 0;
>>>> 		return PCIBIOS_SUCCESSFUL;
>>>> 	}
>>>
>>> I'm scratching my head about this a little.  Here's what your dmesg
>>> log contained originally:
>>>
>>>   pci 0000:00:00.0: [1105:8758] type 01 class 0x048000
>>>   pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
>>>   pci 0000:00:00.0: BAR 0: no space for [mem size 0x01000000 64bit]
>>>   pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x01000000 64bit]
>>>   pci 0000:00:00.0: PCI bridge to [bus 01]
>>>   pcieport 0000:00:00.0: enabling device (0140 -> 0142)
>>>
>>> This device is a bridge (a Root Port, per your lspci output).  With a
>>> BAR, which is legal but unusual.  We couldn't assign space for the
>>> BAR, which means we can't use whatever vendor-specific functionality
>>> it provides.
>>
>> I had several chats with the HW designer. I'll try to explain, only as
>> far as I could understand ;-)
>>
>> We used to make devices, before implementing a root. Since at least
>> one BAR is required (?) for a device, it was decided to have one BAR
>> for the root, for symmetry.
> 
> I'm not aware of a spec requirement for any BARs.  It's conceivable
> that one could build a device that only uses config space.  And of
> course, most bridges have windows but no BARs.  But that doesn't
> matter; the hardware is what it is and we have to deal with it.

I appreciate the compassion. RMK considered the DMA HW too screwy
to bother supporting ;-)

>> In fact, I thought I could ignore that BAR, but it is apparently NOT
>> the case, as MSIs are supposed to be sent *within* the BAR of the root.
> 
> I don't know much about this piece of the MSI puzzle, but maybe Marc
> can enlighten us.  If this Root Port is the target of MSIs and the
> Root Port turns them into some sort of interrupt on the CPU side, I
> can see how this might make sense.
> 
> I think it's unusual for the PCI core to assign the MSI target using a
> BAR, though.  I think this means you'll have to implement your
> arch_setup_msi_irq() or .irq_compose_msi_msg() method such that it 
> looks up that BAR value, since you won't know it at build-time.

I'll hack the Altera driver to fit my purpose.

>> The weird twist is that the BAR advertizes a 64-bit memory zone,
>> but we will, in fact, map MMIO registers behind it. So all the
>> RAM Linux assigns to the area is wasted, IIUC.
> 
> I'm not sure what this means.  You have this:
> 
>> OF: PCI:   MEM 0x90000000..0x9fffffff -> 0x90000000

This means I've put 256 MB of system RAM aside for PCIe devices.
This memory is no longer available for Linux "stuff".

>> pci_bus 0000:00: root bus resource [mem 0x90000000-0x9fffffff]

I suppose this is the PCI bus address. As we've discussed,
I used the identity to map bus <-> CPU addresses.

> This [mem 0x90000000-0x9fffffff] host bridge window means there can't
> be RAM in that region.  CPU accesses to 0x90000000-0x9fffffff have to
> be claimed by the host bridge and forwarded to PCI.
> 
> Linux doesn't "assign system RAM" anywhere; we just learn somehow
> where that RAM is.  Linux *does* assign BARs of PCI devices, and they
> have to be inside the host bridge windows(s).

I'm confused, I thought I had understood that part...
I thought the binding required me to specify (in the "ranges"
property) a non-prefetchable zone of system RAM, and this
memory is then "handed out" by Linux to different devices.
Or do I just need to specify some address range that's not
necessarily backed with actual RAM?

> The BARs may contain registers, RAM, frame buffers, etc., that live
> on the PCI device.  It's totally up to the device what it is.
> 
>> OF: PCI: host bridge /soc/pcie at 50000000 ranges:
>> OF: PCI:   No bus range found for /soc/pcie at 50000000, using [bus 00-ff]
> 
> Tangent: the lack of a bus range is a defect in your DTS.

What range should I specify? 0-1? 0-2? 0-255?

>> [    1.042737] pci_bus 0000:00: scanning bus
>> [    1.046895] pci 0000:00:00.0: [1105:0024] type 01 class 0x048000
>> [    1.053050] pci 0000:00:00.0: calling tango_pcie_fixup_class+0x0/0x10
>> [    1.059639] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x00ffffff 64bit]
>> [    1.066583] pci 0000:00:00.0: calling pci_fixup_ide_bases+0x0/0x40
>> [    1.072929] pci 0000:00:00.0: supports D1 D2
>> [    1.077318] pci 0000:00:00.0: PME# supported from D0 D1 D2 D3hot
>> [    1.083452] pci 0000:00:00.0: PME# disabled
>> [    1.087901] pci_bus 0000:00: fixups for bus
>> [    1.092212] PCI: bus0: Fast back to back transfers disabled
>> [    1.097913] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 0
>> [    1.104746] pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
>> [    1.112893] pci 0000:00:00.0: scanning [bus 00-00] behind bridge, pass 1
>> [    1.119800] pci_bus 0000:01: scanning bus
>> [    1.123972] pci 0000:01:00.0: [1912:0014] type 00 class 0x0c0330
>> [    1.130144] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x00001fff 64bit]
>> [    1.137147] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x40
>> [    1.143577] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
>> [    1.149801] pci 0000:01:00.0: PME# disabled
>> [    1.154364] pci_bus 0000:01: fixups for bus
>> [    1.158671] PCI: bus1: Fast back to back transfers disabled
>> [    1.164368] pci_bus 0000:01: bus scan returning with max=01
>> [    1.170067] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
>> [    1.176814] pci_bus 0000:00: bus scan returning with max=01
>> [    1.182511] pci 0000:00:00.0: fixup irq: got 0
>> [    1.187072] pci 0000:00:00.0: assigning IRQ 00
>> [    1.191658] pci 0000:01:00.0: fixup irq: got 20
>> [    1.196307] pci 0000:01:00.0: assigning IRQ 20
>> [    1.200892] pci 0000:00:00.0: BAR 0: assigned [mem 0x90000000-0x90ffffff 64bit]
>> [    1.208344] pci 0000:00:00.0: BAR 8: assigned [mem 0x91000000-0x910fffff]
>> [    1.215267] pci 0000:01:00.0: BAR 0: assigned [mem 0x91000000-0x91001fff 64bit]
>> [    1.222726] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    1.227812] pci 0000:00:00.0:   bridge window [mem 0x91000000-0x910fffff]
>> [    1.234781] pcieport 0000:00:00.0: enabling device (0140 -> 0142)
>> [    1.241013] pcieport 0000:00:00.0: enabling bus mastering
>> [    1.246634] pci 0000:01:00.0: calling quirk_usb_early_handoff+0x0/0x790
>> [    1.253389] pci 0000:01:00.0: enabling device (0140 -> 0142)
>> [    1.275215] pci 0000:01:00.0: xHCI HW did not halt within 16000 usec status = 0xd2f4167e
>>
>> # /usr/sbin/lspci -v
>> 00:00.0 PCI bridge: Sigma Designs, Inc. Device 0024 (rev 01) (prog-if 00 [Normal decode])
>>         Flags: bus master, fast devsel, latency 0
>>         Memory at 90000000 (64-bit, non-prefetchable) [size=16M]
>>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>>         I/O behind bridge: 00000000-00000fff
> 
> Something's wrong with this.  You have no I/O windows through the host
> bridge, which implies that you can't generic PCI I/O transactions, so
> this I/O window should be disabled.  This might be an lspci issue;
> what does "lspci -xxx" show?

I'll look on Monday. But I know that this revision of the controller
does not support any I/O areas. I don't know why Linux sees this.
Might be a bug in the controller, or a missing init in my code.

>>         Memory behind bridge: 91000000-910fffff
>>         Prefetchable memory behind bridge: 00000000-000fffff
> 
> This prefetchable memory window is bogus, too.  It should probably be
> disabled.  If the bridge doesn't support a prefetchable window, the
> base and limit should be hardwired to zero.  If it supports a window
> but it's disabled, the limit should be less than the base.  For
> example, on my system I see this for a bridge with the window
> disabled:
> 
>   # setpci -s00:1c.0 PREF_MEMORY_BASE
>   fff1
>   # setpci -s00:1c.0 PREF_MEMORY_LIMIT
>   0001

OK, one more thing for me to check. Thanks for your thoroughness.

Regards.