[PATCH v9 0/4] APM X-Gene PCIe host controller

Tue Sep 30 09:53:53 PDT 2014

On Mon, Sep 22, 2014 at 09:23:51AM +0800, Ming Lei wrote:
> Hi Tanmay,
> 
> On Sat, Sep 20, 2014 at 1:15 AM, Tanmay Inamdar <tinamdar at apm.com> wrote:
> > Hi Ming Lei,
> >
> > On Thu, Sep 18, 2014 at 8:08 PM, Ming Lei <tom.leiming at gmail.com> wrote:
> >> Hi Tanmay,
> >>
> >> On Wed, Sep 17, 2014 at 6:33 AM, Tanmay Inamdar <tinamdar at apm.com> wrote:
> >>> This patch adds support for AppliedMicro X-Gene PCIe host controller. The
> >>> driver is tested on X-Gene platform with different gen1/2/3 PCIe endpoint
> >>> cards.
> >>>
> >>> X-Gene PCIe controller driver has depedency on the pcie arm64 arch support.
> >>> Liviu Dudau from ARM has sent a patch set for pcie arm64 arch support and
> >>> support for creating generic pcie bridge from device tree. Liviu's patches
> >>> are available here
> >>> https://lkml.org/lkml/2014/9/8/333
> >>>
> >>> If someone wishes to test PCIe on X-Gene with this patch set, above mentioned
> >>> patches from Liviu must be applied before the patches in this patch set. Also
> >>> please use latest xgene u-boot firmware.
> >>>
> >>> changes since V8:
> >>> 1. Add 'dma-coherent' attribute in device node.
> >>
> >>
> >> I just tested your V9 patches plus Liviu's V10 PCI/ARM64 patches
> >> against 3.17-rc5 on Mustang, and the following failure is triggered:
> >>
> >> [    1.190131] bnx2x: Broadcom NetXtreme II 5771x/578xx 10/20-Gigabit
> >> Ethernet Driver bnx2x 1.78.19-0 (2014/02/10)
> >> [    1.200249] tg3.c:v3.137 (May 11, 2014)
> >> [    1.204087] ------------[ cut here ]------------
> >> [    1.208682] WARNING: CPU: 1 PID: 1 at drivers/pci/pci.c:131
> >> pci_ioremap_bar+0x70/0x78()
> >> [    1.216646] Modules linked in:
> >> [    1.219696] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.17.0-rc5+ #1
> >> [    1.226018] Call trace:
> >> [    1.228453] [<ffffffc0000884f0>] dump_backtrace+0x0/0x16c
> >> [    1.233826] [<ffffffc00008866c>] show_stack+0x10/0x1c
> >> [    1.238851] [<ffffffc0006e3210>] dump_stack+0x74/0x94
> >> [    1.243878] [<ffffffc0000a934c>] warn_slowpath_common+0x88/0xb0
> >> [    1.249767] [<ffffffc0000a9438>] warn_slowpath_null+0x14/0x20
> >> [    1.255485] [<ffffffc0003994c4>] pci_ioremap_bar+0x6c/0x78
> >> [    1.260942] [<ffffffc0005598cc>] tg3_init_one+0x148/0x16f4
> >> [    1.266401] [<ffffffc00039ccf0>] pci_device_probe+0x78/0xd4
> >> [    1.271946] [<ffffffc00042785c>] driver_probe_device+0x94/0x390
> >> [    1.277834] [<ffffffc000427c44>] __driver_attach+0x98/0xa0
> >> [    1.283293] [<ffffffc000425a54>] bus_for_each_dev+0x54/0x98
> >> [    1.288835] [<ffffffc00042730c>] driver_attach+0x1c/0x28
> >> [    1.294121] [<ffffffc000426f04>] bus_add_driver+0x164/0x240
> >> [    1.299663] [<ffffffc000428430>] driver_register+0x64/0x130
> >> [    1.305207] [<ffffffc00039c97c>] __pci_register_driver+0x3c/0x48
> >> [    1.311181] [<ffffffc000aa8a5c>] tg3_driver_init+0x1c/0x28
> >> [    1.316640] [<ffffffc0000814e0>] do_one_initcall+0xc4/0x1b4
> >> [    1.322186] [<ffffffc000a74b64>] kernel_init_freeable+0x1b8/0x25c
> >> [    1.328246] [<ffffffc0006de318>] kernel_init+0xc/0xd4
> >> [    1.333278] ---[ end trace f4dca5ab5b436080 ]---
> >> [    1.337871] tg3 0000:01:00.0: Cannot map device registers, aborting
> >> [    1.344122] tg3: probe of 0000:01:00.0 failed with error -12
> >>
> >>
> >
> > Thanks for trying out the patches. Which firmware version are you
> > using? We may have release new firmware version for PCI to work if not
> > released yet.
> 
> U-Boot 2013.04-mustang_sw_1.13.28-beta (Aug 25 2014 - 14:16:10)
> 
> I will check if there is newer fw available.
> 
> > Also can you please send out the entire boot log?
> 
> Please see below link:
> 
>       http://pastebin.com/fLqaDn6t

>From Ming's log, using the v9 patches:

  PCI host bridge /soc/pcie at 1f2b0000 ranges:
    No bus range found for /soc/pcie at 1f2b0000, using [??? 0xffffffc3eb4292c0-0xffffffc0005e3b8c flags 0x40]

What's going on here?  It looks like it's from
of_pci_get_host_bridge_resources(), but I don't see how this output could
happen.

     IO 0xe010000000..0xe01000ffff -> 0x00000000
    MEM 0xe180000000..0xe1ffffffff -> 0x80000000
  xgene-pcie 1f2b0000.pcie: (rc) x1 gen-1 link up
  xgene-pcie 1f2b0000.pcie: PCI host bridge to bus 0000:00
  pci_bus 0000:00: root bus resource [bus 00-ff]
  pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
  pci_bus 0000:00: root bus resource [mem 0xe180000000-0xe1ffffffff] (bus address [0x80000000-0xffffffff])
  pci_bus 0000:00: scanning bus
  pci 0000:00:00.0: [19aa:e008] type 01 class 0x060400
  pci 0000:00:00.0: reg 0x10: [mem 0x8000000000-0x807fffffff 64bit pref]

I guess this is the "inbound BAR" that you hid in the v10 patches.  Hiding
it will prevent some of BAR assignment problems below.  This is probably
what caused the tg3 mapping problem Ming initially reported above.  We
tried to reassign the inbound BAR, and it took up the entire space that we
thought was available for CPU accesses to the PCI bus, leaving none for the
tg3 device.

Was Ming just unlucky by having an unusual configuration, e.g., with a lot
of memory or something?  He certainly doesn't have a complicated PCIe
hierarchy.  I assume you've tested this and had it work *somewhere*.

  pci 0000:00:00.0: supports D1 D2
  pci_bus 0000:00: fixups for bus
  pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 0
  pci_bus 0000:01: scanning bus
  pci 0000:01:00.0: [14e4:165a] type 00 class 0x020000
  pci 0000:01:00.0: reg 0x10: [mem 0x00100000-0x0010ffff 64bit]
  pci 0000:01:00.0: PME# supported from D3hot
  pci 0000:01:00.0: PME# disabled
  pci_bus 0000:01: fixups for bus
  pci_bus 0000:01: bus scan returning with max=01
  pci 0000:00:00.0: scanning [bus 01-01] behind bridge, pass 1
  pci_bus 0000:00: bus scan returning with max=01
  pci 0000:00:00.0: BAR 0: assigned [mem 0xe180000000-0xe1ffffffff 64bit pref]
  pci 0000:00:00.0: BAR 8: no space for [mem size 0x00100000]
  pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x00100000]
  pci 0000:01:00.0: BAR 0: no space for [mem size 0x00010000 64bit]
  pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x00010000 64bit]
  pci 0000:00:00.0: PCI bridge to [bus 01]