arm64 PCI resource allocation issue

[Benjamin Herrenschmidt]

Still hoping for opinions on this ... I'll probably have time to hack
on it next week...

On Fri, 2022-08-05 at 09:51 +1000, Benjamin Herrenschmidt wrote:

> That leaves us with 3 overall routes I can think of (we can figure out
> the details next):
> 
>  1) We can try to detect early those devices (easy with SPCR, are
> there more on aarch64 ? on x86 there is) and hammer them into place,
> flagging them somewhat and forcing them (and all their parents) to keep their
> resources.
> 
> Pros: It's rather easy to implement, we can "register" the addresses
> early and have the PCI probe code match detected devices againt that
> list & flag them (for example IORESOURCE_PCI_FIXES :-) and their
> parents.
> 
> Cons: It will force entire bus hierarchies to be fixed, which might
> not really help on firmwares that are known to setup sub-optimal
> apertures (or even completely b0rked ones). But we don't know who those are
> except maybe one or two if we dig down into the previous version of
> that discussion from a couple of years ago.
> 
>  2) We can try to "keep track" of them as they move. Variant A.
> 
> We do it the way efifb does it and wrap that in something a bit nicer
> as follow:
> 
>  - We add a helper to "record" a pci_dev/BAR#/offset combination and
> an
> other one to do the lookup & fixup of a FW originated address.
> 
>  - We make efifb quirk use that instead of its existing global
> "bar_resource".
> 
>  - We add a similar quirk to the ACPI code that parses SPCR and
> (maybe) another one for earlycon (hint they may be the same device, some
> deduplication would be useful).
>  
>  - We update 8250_pci (I assume pl01x are never PCI ?) to call this
> to "fixup" addresses obtained from earlycon. That's the easy bit. SPRC
> is trickier, we'd need to fixup addressed parsed from
> add_preferred_console() .. I'm not 100% sure there's a case where
> such an address would be added post-PCI-remap and we might incorrectly fix
> it up.  I don't think so but ...
> 
> Pros: It should (hopefully) not be overly complicated and reasonably
> self contained, low risk.
> 
> Cons: 
> 
>  - It's a bit more complicated than other solutions, though not
> insanely
> 
>  - This doesn't solve the problem of a driver such as earlycon being
> "live" accross the remapping (and thus means we'll probably still
> have verbose PCI probing with earlycon dying horribly). This is already
> partially broken since we temporarily disable decoding during probing
> but that's a small window ... We can look at solving that separately
> by adding on top of this registration mechanism: We *could* optionally
> register in our above helper a pair of callbacks that the PCI code
> would call for each registered "early device" before and after
> remapping to "suspend access" and "fixup address". Those would be
> ideally called around the remapping of the entire host bridge the
> device is on.
> 
> 3) Keeping track, Variant B
> 
> (note: the more I think about it, the more I prefer variant A but
> let's see what others think)
> 
> We generalize pcibios_save_fw_addr() and for the sake of it, we move
> that into pci_dev which simplifies everything and gets rid of that
> separate list.
> 
> Then, things like efifb, 8250_pci etc... do a lookup in there for
> addresses they obtain from screen_info, earlycon,
> add_preferred_console.. and on match, perform the necessary fixup.
> Assuming we are confident those addresses originate from before the
> PCI remapping that is.
> 
> Pros: It *seems* even simpler than the above other options and maybe
> even faster.
> 
> Cons: It's more resource intensive as we now backup original BARs for
> everything under the sun. It also doesn't provide a great path to
> address the case I mentioned earlier for dealing with "live" devices.
> 
> That's all I came up with ... Any better ideas and any preferences ?
> At this point I'm reasonably keen on (2) (tracking variant A).