pci_bus_for_each_resource, transparent bridges and rsrc_nonstatic.c

[Bjorn Helgaas]

On Monday 22 March 2010 11:27:10 am Dominik Brodowski wrote:
> Bjorn,
> 
> On Mon, Mar 22, 2010 at 09:34:28AM -0700, Bjorn Helgaas wrote:
> > On Monday 22 March 2010 06:00:41 am Dominik Brodowski wrote:
> > > Komuro reports the following issue:
> > > 
> > > 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 93) (prog-if 01 [Subtractive decode])
> > > ...
> > > 	Bus: primary=00, secondary=04, subordinate=05, sec-latency=32
> > > 	I/O behind bridge: 00001000-00001fff
> > > 	Memory behind bridge: d2000000-d40fffff
> > > 	Prefetchable memory behind bridge: 00000000d0000000-00000000d1ffffff
> > > 
> > > 04:06.0 CardBus bridge: Ricoh Co Ltd RL5c476 II (rev b6)
> > > 
> > > Iterating over the bus (4) resources used to report just IO 0x1000-0x1ffff,
> > > MEM d2...-d4..., prefetch d0...-d1..., and that's exactly what we need in
> > > drivers/pcmcia/rsrc_nonstatic.c . However, the newly added
> > > pci_bus_for_each_resource() seems to return _also_ additional resources,
> > > which we need to exclude in drivers/pcmcia/rssrc_nonstatic.c . Any ideas on
> > > how to fix this issue?
> > ...

> We did indeed iterate over all 16 bus resources, so also for the root PCI
> bridge resources. However, this used to just skip in the next check:
> 
> if (res == &ioport_resource)
> 	continue;
> 
> (or &iomem_resource) as there used to be no specific root I/O / Memory
> window resource set. But now, all of the I/O address space is split up into
> two, and a (previously non-existing) "PCI Bus 0000:00" appears in
> /proc/iomem . Do you know, by chance, which change or patchset specifically
> introduced this?

Yep, I think this is 7bc5e3f2be32a, which makes us pay attention to
the PCI host bridge windows reported by the BIOS instead of assuming
that "all of ioport_resource and iomem_resource are routed to the
single PCI root bus."

You should see the same behavior even before 7bc5e3f2be32a if you
boot with "pci=use_crs".

> > > > yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge I/O window: 0x0 - 0xcf7
> > > > yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge I/O window: 0xd00 - 0xffff
> 
> with just a very small hole in between...

This is pretty typical.  The BIOS is telling us that 0xcf8-0xcff
is consumed directly by the host bridge (it uses that for doing
config space accesses on the PCI bus), and the rest of the area
if forwarded to the downstream PCI bus and is available for PCI
devices to use.

> So, what should we do? 

Let's back up a bit.  I don't know enough about PCMCIA, and I don't
see the problem yet.  We have the 00:1e.0 bridge leading to bus 04,
and 04:06.0 is a CardBus bridge.  These three windows are the ranges
positively decoded and forwarded by the 00:1e.0 bridge:

  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge I/O window: 0x1000 - 0x1fff
  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge Memory window: 0xd2000000 - 0xd40fffff
  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge Memory window: 0xd0000000 - 0xd1ffffff

But the bridge is in subtractive-decode mode, so it *also* forwards
anything it sees that is unclaimed by other devices on bus 00, which
means the 04:06.0 CardBus bridge will also see these ranges:

  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge I/O window: 0x0 - 0xcf7
  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge I/O window: 0xd00 - 0xffff
  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge Memory window: 0xa0000 - 0xbffff
  yenta_cardbus 0000:04:06.0: pcmcia: parent PCI bridge Memory window: 0xc0000000 - 0xfebfffff

Why do you care whether these additional ranges are excluded?  They
should be just as usable as the first three.  Of course, the first
three ranges (the ones positively decoded by 00:1e.0) can only be
used by devices on bus 04, while the last four (subtractively decoded
by 00:1e.0) can be used by devices on bus 00 or bus 04, so there's
more competition for the last four.

But I don't understand why we care whether the upstream bridge
decoded a region positively or subtractively when deciding whether to
add it to the PCMCIA resource map.

Bjorn