[PATCH v5 2/4] Documentation: arm64/arm: dt bindings for numa.

[Mark Rutland]

On Tue, Sep 29, 2015 at 09:38:04AM +0100, Ganapatrao Kulkarni wrote:
> (sending again, by mistake it was set to html mode)
> 
> On Tue, Sep 29, 2015 at 2:05 PM, Ganapatrao Kulkarni
> <gpkulkarni at gmail.com> wrote:
> > Hi Mark,
> >
> > I have tried to answer your comments, in the meantime we are waiting for Ben
> > to share the details.
> >
> > On Fri, Aug 28, 2015 at 6:02 PM, Mark Rutland <mark.rutland at arm.com> wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Aug 14, 2015 at 05:39:32PM +0100, Ganapatrao Kulkarni wrote:
> >> > DT bindings for numa map for memory, cores and IOs using
> >> > arm,associativity device node property.
> >>
> >> Given this is just a copy of ibm,associativity, I'm not sure I see much
> >> point in renaming the properties.
> >>
> >> However, (somewhat counter to that) I'm also concerned that this isn't
> >> sufficient for systems we're beginning to see today (more on that
> >> below), so I don't think a simple copy of ibm,associativity is good
> >> enough.
> >
> > it is just copy right now, however it can evolve when we come across more
> > arm64 numa platforms

Whatever we do I suspect we'll have to evolve it as new platforms
appear. As I mentioned there are contemporary NUMA ARM64 platforms (e.g.
those with CCN) that I don't think we can ignore now given we'll have to
cater for them.

> >> > +==============================================================================
> >> > +2 - arm,associativity
> >> >
> >> > +==============================================================================
> >> > +The mapping is done using arm,associativity device property.
> >> > +this property needs to be present in every device node which needs to
> >> > to be
> >> > +mapped to numa nodes.
> >>
> >> Can't there be some inheritance? e.g. all devices on a bus with an
> >> arm,associativity property being assumed to share that value?
> >
> > yes there is inheritance and respective bus drivers should take care of it,
> > like pci driver does at present.

Ok. 

That seems counter to my initial interpretation of the wording that the
property must be present on device nodes that need to be mapped to NUMA
nodes.

Is there any simple way of describing the set of nodes that need this
property?

> >> > +topology and boundary in the system at which a significant difference
> >> > in
> >> > +performance can be measured between cross-device accesses within
> >> > +a single location and those spanning multiple locations.
> >> > +The first cell always contains the broadest subdivision within the
> >> > system,
> >> > +while the last cell enumerates the individual devices, such as an SMT
> >> > thread
> >> > +of a CPU, or a bus bridge within an SoC".
> >>
> >> While this gives us some hierarchy, this doesn't seem to encode relative
> >> distances at all. That seems like an oversight.
> >
> >
> > distance is computed, will add the details to document.
> > local nodes will have distance as 10(LOCAL_DISTANCE) and every level, the
> > distance multiplies by 2.
> > for example, for level 1 numa topology, distance from local node to remote
> > node will be 20.

This seems arbitrary.

Why not always have this explicitly described?

> >> Additionally, I'm somewhat unclear on how what you'd be expected to
> >> provide for this property in cases like ring or mesh interconnects,
> >> where there isn't a strict hierarchy (see systems with ARM's own CCN, or
> >> Tilera's TILE-Mx), but there is some measure of closeness.
> >
> >
> > IIUC, as per ARMs CCN architecture, all core/clusters are at equal distance
> > of DDR, i dont see any NUMA topology.

The CCN is a ring interconnect, so CPU clusters (henceforth CPUs) can be
connected with differing distances to RAM instances (or devices).

Consider the simplified network below:

  +-------+      +--------+      +-------+
  | CPU 0 |------| DRAM A |------| CPU 1 |
  +-------+      +--------+      +-------+
      |                              |
      |                              |
  +--------+                     +--------+
  | DRAM B |                     | DRAM C |
  +--------+                     +--------+
      |                              |
      |                              |
  +-------+      +--------+      +-------+
  | CPU 2 |------| DRAM D |------| CPU 3 |
  +-------+      +--------+      +-------+

In this case CPUs and DRAMs are spaced evenly on the ring, but the
distance between an arbitrary CPU and DRAM is not uniform.

CPU 0 can access DRAM A or DRAM B with a single hop, but accesses to
DRAM C or DRAM D take three hops.

An access from CPU 0 to DRAM C could contend with accesses from CPU 1 to
DRAM D, as they share hops on the ring.

There is definitely a NUMA topology here, but there's not a strict
hierarchy. I don't see how you would represent this with the proposed
binding.

Likewise for the mesh networks (e.g. that of TILE-Mx)

> > however, if there are 2 SoC connected thorough the CCN, then it is very much
> > similar to cavium topology.
> >
> >> Must all of these have the same length? If so, why not have a
> >> #(whatever)-cells property in the root to describe the expected length?
> >> If not, how are they to be interpreted relative to each other?
> >
> >
> > yes, all are of default size.

Where that size is...?

> > IMHO, there is no need to add cells property.

That might be the case, but it's unclear from the documentation. I don't
see how one would parse / verify values currently.

> >> > +the arm,associativity nodes. The first integer is the most significant
> >> > +NUMA boundary and the following are progressively less significant
> >> > boundaries.
> >> > +There can be more than one level of NUMA.
> >>
> >> I'm not clear on why this is necessary; the arm,associativity property
> >> is already ordered from most significant to least significant per its
> >> description.
> >
> >
> > first entry in arm,associativity-reference-points is used to find which
> > entry in associativity defines node id.
> > also entries in arm,associativity-reference-points defines,
> > how many entries(depth) in associativity can be used to calculate node
> > distance
> > in both level 1 and  multi level(hierarchical) numa topology.

I think this needs a more thorough description; I don't follow the
current one.

> >> Is this only expected at the root of the tree? Can it be re-defined in
> >> sub-nodes?
> >
> > yes it is defined only at the root.

This needs to be stated explicitly.

I see that this being the case, *,associativity-reference-points would
be a more powerful property than the #(whatever)-cells property I
mentioned earlier, but a more thorough description is required.

Thanks,
Mark.