[PATCH v5 2/4] Documentation: arm64/arm: dt bindings for numa.

[Leizhen (ThunderTown)]

On 2015/8/29 18:37, Benjamin Herrenschmidt wrote:
> On Sat, 2015-08-29 at 17:46 +0800, Leizhen (ThunderTown) wrote:
>> Why not copy the method of ACPI numa? There only three elements
>> should be configured:
>> 1) a cpu belong to which node
>> 2) a memory block belong to which node
>> 3) the distance of each two nodes

Sorry, I forgot to write something:
4) a device(maybe a bus device) belongs to which node

For example:
device-name {
        ...
        numa-node = <&node0>;
};

To simplify the discussion, I will not mention device again. Treat both
cpus and devices as masters, memorys as slaves.

A bus is not a master, we allow binding numa node to a bus, because we may
want all devices on the bus to inherit its numa node-id without obvious configured one by one.

> 
> This means your are bolting into the DT representation the concept of
> "Node" which isn't necessarily very meaningful.
> 
> Your system is really a hierarchy of objects. You can have cores on a
> chip, already possibly sharing some level of cache or not, you can have
> chips on a module, modules linked at various distances, etc...
> 
> What is "a node" ?
> 
> For example, I have a P8 chip with 2 chips on a module (fast X-bus) and
> 2 modules (slightly slower A-bus). All 4 chips have 2 memory
> controllers each.
> 
> Is a "node" a chip or a module ?

A numa node is a abstract concept, it needn't related to a real hardware level.
A numa node normally contains both cpus and mems, but may only contains cpus or mems,
or maybe nothing(quite rare). We put cpus or mems into a node, because we want to use
node-distance to implement the nearest memory access, the nearest process schedule.

In your example:
On fast X-bus, have a module contains 2 chips.
On slightly slower A-bus, have 2 modules(treat them as 2 chips).
Each chip contains 2 memory controllers.

Suppose each chip access its local bus memory faster than another.

Case1:
Each chip access its 2 local memory controllers faster than others. Then we can define numa nodes:
node-xbus-0: a chip and 2 local memory.
node-xbus-1: a chip and 2 local memory.
node-abus-0: a chip(module) and 2 local memory.
node-abus-1: a chip(module) and 2 local memory.

Case2:
Each chip access any memory controllers on its local bus are the same. Then we can define numa nodes:
node-xbus: 2 chips and 4 local memory.
node-abus: 2 chips(modules) and 4 local memory.


> 
> The Linux concept of node is too restrictive. The associativity
> properties avoid this by allowing you to define as many "levels" of
> associativity as you wish. Also since it's right justified, a given
> component doesn't need to have all levels (a MC can stop at chip while
> cores can go down one more level for example).
> 
> The reference points property gives a hint as "interesting" levels can
> typically be used as a hint for chosing what Linux will use as a "node"
> at least until Linux gets smarter. It can also be used to calculate
> distances.
> 
> Cheers,
> Ben.
> 
> 
> .
>