Giving special alignment/size constraints to the Linux PCI core?

[Bjorn Helgaas]

On Thu, Feb 7, 2013 at 8:50 AM, Thomas Petazzoni
<thomas.petazzoni at free-electrons.com> wrote:
> Bjorn,
>
> I guess I need your advice on the below problem, but here is a summary.
>
> Basically, the PCI-to-PCI bridge specifications require that the memory
> address assigned to a PCI-to-PCI bridge is aligned on a 1 MB boundary.
>
> Unfortunately, on Marvell hardware, we can only create address decoding
> windows that are aligned on a multiple of their size (a 1 MB window
> must be 1 MB aligned, a 2 MB window must be 2 MB aligned, etc.).
>
> How can we teach the Linux PCI core about this requirement, so that it
> does a proper assignment of addresses at the PCI-to-PCI bridge level?
> For the I/O addresses, Russell suggested the pcibios_window_alignment()
> hook, but it doesn't receive the size of the resource, so we can't
> determine what alignment is needed.
>
> As Jason points out below, we need to be able to tell the PCI core that
> a given memory area needs some alignment, but also that its size is
> larger than what the PCIe device claims, because we cannot create
> address decoding windows of an arbitrary size. For example, an address
> decoding window of 3 MB is not possible, so if a device wants 3 MB,
> then we would need to extend this memory area to 4 MB so that the next
> device doesn't get an address decoding window that overlaps with the
> previous one.
>
> I was hoping that the emulated PCI-to-PCI bridge could, by its
> behavior, teach the Linux PCI core about these special constraints.
> However, reading the PCI-to-PCI bridge specification, I don't see how
> to achieve that.
>
> Do you have some suggestions?

Huh.  That hardware looks less and less like a P2P bridge all the time
:(  You can't configure it via standard PCI config accesses, and the
aperture alignment and size constraints sound completely non-standard.
 Are the specs for this thing public?

I could imagine changing pcibios_window_alignment() to take the
resource, so it could deal with the alignment question (though I
haven't looked in detail and there might be some implementation issue
with that).

With regard to the size issue (3MB window using 4MB of address space),
I can't think of a reasonable way to teach the PCI core about both
sizes.  But is there any reason to program the bridge for a 3MB window
instead of a 4MB window, given that there's nothing else we can do
with the extra 1MB anyway?  Is a 3MB window even possible?  I would
think something that must be aligned on its size would be restricted
to power-of-2 sizes anyway, just like PCI BARs are.  Maybe you can
just always round up window sizes to a power of 2?

Bjorn

> On Wed, 6 Feb 2013 10:50:19 -0700, Jason Gunthorpe wrote:
>
>> > > > Can't this be solved using the window_alignement() hook we've
>> > > > been discussing separately? Just like we teach the Linux PCI
>> > > > core about our alignment requirements of 64K for the I/O
>> > > > regions, we could teach it about our alignment requirement on
>> > > > memory regions as well. No?
>> > >
>> > > Hopefully :) As long as it can adjust the start and length you
>> > > should be fine.
>> >
>> > Why would you need to adjust the length? If Linux allocates a 2 MB
>> > resource on a 1 MB boundary, we simply increase the start address to
>> > the next 2 MB boundary, and that's it. Why would the length need to
>> > change?
>>
>> Well, lets say 3MB is the example. A 3mb region needs to fit inside a
>> 4mb MBUS window. If you align the start to 4mb then the pci-e core
>> needs to know that it can't use the extra 1mb covered by the mbus
>> window. mbus windows must not overlap.
>>
>> Adjusting the bridge window length to be 4mb communicates that dead
>> space to the PCI core, and presumably this shows up in lspci and
>> whatnot.
>>
>> I suppose if you align the end to 4mb (thus creating the gap before,
>> not after) things should work out OK, but the information that the gap
>> is routed to a specific PCI link is lost..
>>
>> Jason
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel at lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>
>
>
> --
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com