[PATCH v1 1/4] dt-bindings: PCI: microchip: add fabric address translation properties

[Daire.McNamara at microchip.com]

Thanks Rob for taking the time to get back to me.  Yes, I thought 2
levels of translation might be the way to go. If you could point me to
a gadget that has 2-level translation working, that'd be great

On Fri, 2022-09-02 at 11:28 -0500, Rob Herring wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you
> know the content is safe
> 
> On Fri, Sep 2, 2022 at 9:22 AM <daire.mcnamara at microchip.com> wrote:
> > From: Conor Dooley <conor.dooley at microchip.com>
> > 
> > On PolarFire SoC both in- & out-bound address translations occur in
> > two
> > stages. The specific translations are tightly coupled to the FPGA
> > designs and supplement the {dma-,}ranges properties. The first
> > stage of
> > the translation is done by the FPGA fabric & the second by the root
> > port.
> > Add two properties so that the translation tables in the root
> > port's
> > bridge layer can be configured to account for the translation done
> > by
> > the FPGA fabric.
> 
> I'm skeptical that ranges/dma-ranges can't handle what you need.
> Anything in this area is going to need justification 'ranges doesn't
> work because x, y, z...'.
Cool.
> 
> > Signed-off-by: Conor Dooley <conor.dooley at microchip.com>
> > Signed-off-by: Daire McNamara <daire.mcnamara at microchip.com>
> > ---
> >  .../bindings/pci/microchip,pcie-host.yaml     | 107
> > ++++++++++++++++++
> >  1 file changed, 107 insertions(+)
> > 
> > diff --git a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > index 23d95c65acff..29bb1fe99a2e 100644
> > --- a/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > +++ b/Documentation/devicetree/bindings/pci/microchip,pcie-
> > host.yaml
> > @@ -71,6 +71,113 @@ properties:
> >      minItems: 1
> >      maxItems: 6
> > 
> > +  microchip,outbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The CPU-to-PCIe (outbound) address translation takes place
> > in two stages.
> > +      Depending on the FPGA bitstream, the outbound address
> > translation tables
> > +      in the PCIe root port's bridge layer will need to be
> > configured to account
> > +      for only its part of the overall outbound address
> > translation.
> > +
> > +      The first stage of outbound address translation occurs
> > between the CPU address
> > +      and an intermediate "FPGA address". The second stage of
> > outbound address
> > +      translation occurs between this FPGA address and the PCIe
> > address. Use this
> > +      property, in conjunction with the ranges property, to divide
> > the overall
> > +      address translation into these two stages so that the PCIe
> > address
> > +      translation tables can be correctly configured.
> 
> Sounds like you need 2 levels of ranges/dma-ranges.
> 
> / {
>     fpga-bus {
>         ranges = ...
>         dma-ranges = ...
>         pcie at ... {
>             ranges = ...
>             dma-ranges = ...
>         };
>     };
> };
> 
Very possibly! I'd prefer that approach, tbh, if I can get it working. 
I'll go back and try 2 levels again.


> > +      If this property is present, one entry is required per
> > range. This is so
> > +      FPGA designers can choose to route different address ranges
> > through different
> > +      Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > +      If this property is not present, the entire address
> > translation
> > +      in any ranges property is attempted by the root port driver
> > via its outbound
> > +      address translation tables.
> > +
> > +      Each element in this property has three components. The
> > first is a
> > +      PCIe address, the second is an FPGA address, and the third
> > is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one
> > correspondance between
> > +      range properties and this property.  For each pair of range
> > and
> > +      outbound-fabric-translation-range properties, the root port
> > driver will
> > +      subtract the FPGA address in this property from the CPU
> > address in the
> > +      corresponding range property and use the remainder to
> > program its
> > +      outbound address translation tables.
> > +
> > +      For each range, take its PCIe address and size - these are
> > the PCIe
> > +      address & size for the element. The FPGA address is derived
> > from a given
> > +      FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > +      design to the Core Complex. For a trivial configuration, it
> > is likely to be the
> > +      lower 32 bits of the PCIe address in the range property and
> > the upper
> > +      bits of the base address of the Fabric Interface Controller
> > the design uses.
> > +      Otherwise, it is tightly coupled with the data path
> > configured in the
> > +      FPGA fabric between the root port and the Core Complex.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +      
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 3
> > +      maxItems: 6
> > +
> > +  microchip,inbound-fabric-translation-ranges:
> > +    $ref: /schemas/types.yaml#/definitions/uint32-matrix
> > +    minItems: 1
> > +    maxItems: 32
> > +    description: |
> > +      The PCIe-to-CPU (inbound) address translation takes place in
> > two stages.
> > +      Depending on the FPGA bitstream, the inbound address
> > translation tables
> > +      in the PCIe root port's bridge layer will need to be
> > configured to account
> > +      for only its part of the overall inbound address
> > translation.
> > +
> > +      The first stage of address translation occurs between the
> > PCIe address and
> > +      an intermediate FPGA address. The second stage of address
> > translation
> > +      occurs between the FPGA address and the CPU address. Use
> > this property
> > +      in conjunction with the dma-ranges property to divide the
> > address
> > +      translation into these two stages.
> > +
> > +      If this property is present, one entry is required per dma-
> > range. This is so
> > +      FPGA designers can choose to route different address ranges
> > through different
> > +      Fabric Interface Controllers and other logic as they see
> > fit.
> > +
> > +      If this property is not present, the entire address
> > translation
> > +      in any dma-ranges property is attempted by the root port
> > driver via its
> > +      inbound address translation tables.
> > +
> > +      Each element in this property has three components. The
> > first is a
> > +      PCIe address, the second is an FPGA address, and the third
> > is a size.
> > +      These properties may be 32 or 64 bit values.
> > +
> > +      In operation, the driver will expect a one-to-one
> > correspondance between
> > +      dma-range properties and this property.  For each pair of
> > dma-range and
> > +      inbound-fabric-translation-range properties, the root port
> > driver will
> > +      subtract the FPGA address in this property from the CPU
> > address in the
> > +      corresponding dma-range property and use the remainder to
> > program its
> > +      inbound address translation tables.
> > +
> > +      From each dma-range, take its PCIe address and size - these
> > are the PCIe
> > +      address & size for the element. The FPGA address is derived
> > from a given
> > +      FPGA fabric design and is the address delivered by that FPGA
> > fabric
> > +      design to the Core Complex. For a trivial configuration,
> > this property
> > +      is unlikely to be required (i.e. no fabric translation on
> > the inbound
> > +      interface).  Otherwise, it is tightly coupled with the
> > inbound data path
> > +      configured in the FPGA fabric between the root port and the
> > Core Complex.
> > +      It is expected that more than one translation range may be
> > added to
> > +      an FPGA fabric design, e.g. to deliver data to cached or
> > non-cached
> > +      DDR.
> > +
> > +      For more information on the tables, see Section 1.3.3,
> > +      "PCIe/AXI4 Address Translation" of the PolarFire SoC PCIe
> > User Guide:
> > +      
> > https://www.microsemi.com/document-portal/doc_download/1245812-polarfire-fpga-and-polarfire-soc-fpga-pci-express-user-guide
> > +
> > +    items:
> > +      minItems: 4
> > +      maxItems: 7
> > +
> >    msi-controller:
> >      description: Identifies the node as an MSI controller.
> > 
> > --
> > 2.25.1
> >