[PATCH RFC v2 00/19] arm,fs/resctrl: ARM MPAM MB_NODE support
Richard Cheng
icheng at nvidia.com
Tue Sep 1 02:37:00 PDT 2026
On Mon, Aug 31, 2026 at 10:22:26AM +0800, Fenghua Yu wrote:
> This series enables MBA bandwidth control emulation on ARM MPAM when the
> default MB control is disabled, using a node-scoped MB_NODE control as
> the native backend. It extends memory-bandwidth monitoring and
> control to memory-level MSCs or CPU-less NUMA nodes, and adds
> documentation and tests.
>
> Background
>
> On x86, MBA bandwidth control is typically associated with the L3 cache:
> schemata lines use L3 cache ids and monitor domains appear as mon_L3_XX.
> On some ARM MPAM systems the MBA control is backed by a memory-level MSC
> rather than the L3 cache MSC. A memory-level MSC is represented as a
> NUMA node, and that node commonly has no CPUs of its own (a memory-only
> node that still participates in bandwidth control). For such a resource
> the control scope is NODE, schemata identifiers are NUMA node ids,
> monitor domains are named mon_NODE_XX, and a node-scoped MB_NODE control
> may be the only control with usable bandwidth hardware.
>
> Because the memory-level MSC lives on a CPU-less NUMA node, the driver
> must handle nodes with no local CPUs. mpam_ris_get_affinity() derives
> affinity from the component's NUMA node id, so a memory-only node would
> otherwise end up with an empty CPU mask and never be registered. When
> the derived mask is empty, the driver falls back to cpu_possible_mask
> (masked by MSC accessibility) so the node still gets a resctrl domain.
> The borrowed affinity is kept out of class->affinity: those CPUs already
> contribute through CPU-ful nodes in the class, and subtracting the
> borrowed mask on teardown would remove CPUs that other live nodes still
> depend on. Domain setup iterates all MSC components per CPU (keyed by
> component), so each component — including these CPU-less memory nodes —
> gets its own control and monitor domain.
>
> Examples of NODE and emulation interfaces on MPAM.
>
> ** On platform with CPU-less nodes, by default, control_mode is legacy.
> 1. By default, MB is emulated by MB_NODE in schemata.
> info/
> ├── MB
> │ ├── bandwidth_gran
> │ ├── control_mode # [legacy] native
> │ └── schemata
> │ └── MB
> │ ├── MB_NODE # emulated by "MB_NODE"
>
> schemata: Only show MB, which is emulated by MB_NODE. domain id is numa
> id
> MB:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
> L3:1=ffff;2=ffff
>
> 2. root can switch to native by writing "native" to "control_mode":
> info/
> ├── MB
> │ ├── bandwidth_gran
> │ ├── control_mode # legacy [native]
> │ └── schemata
> │ └── MB
> │ ├── MB_NODE # no emulation, native mode
>
> schemata: Only show MB_NODE. No "MB:" is emulated.
> MB_NODE:0=100;1=100;2=100;10=100;18=100;26=100;34=100;35=100
> L3:1=ffff;2=ffff
>
> 3. llc_occupancy in mon_L3_x where x is cache id
> mbm_total_bytes in mon_NODE_y where y is numa id
> mon_data/
> mon_L3_01 mon_NODE_00 mon_NODE_02 mon_NODE_18 mon_NODE_34
> mon_L3_02 mon_NODE_01 mon_NODE_10 mon_NODE_26 mon_NODE_35
>
> mon_data/mon_L3_01/llc_occupancy
> mon_data/mon_NODE_01/mbm_total_bytes
>
> ** On legacy machine which only has L3 MSCs. No NODE scope in schemata
> or mon_data
> schemata: MB's domain id is cache id, not numa node id
> MB:1=100;2=100
> L3:1=ffff;2=ffff
>
> mon_data/
> mon_L3_01 mon_L3_02
>
> mon_data/mon_L3_01/llc_occupancy
> mon_data/mon_L3_01/mbm_total_bytes
> mon_data/mon_L3_02/llc_occupancy
> mon_data/mon_L3_02/mbm_total_bytes
>
> The patches can be compiled but not tested on x86 yet.
>
> The 19 patches are grouped as follows:
>
> 1. Memory bandwidth monitoring and control on memory-level MSCs
> (patches 1–10, 12)
>
> Generalise the monitor infrastructure beyond hard-wired L3 paths:
> de-hardcode L3 monitor plumbing, expose MBA MBM counter assignment,
> name node-scoped monitor domains mon_NODE_<id>, add a node-scope MBM
> total event, and make MBM handling resource-aware.
>
> On the ARM MPAM side, add memory-level MSC and ABMC support, refine
> L3 topology and class selection, refactor domain setup to iterate all
> MSC components per CPU (keyed by component), and handle CPU-less NUMA
> memory nodes that have no local CPUs but still participate in MPAM
> bandwidth control. Memory hotplug locking and notifier are added.
>
> 4. MB_NODE emulates MB (patch 11) [1]
>
> Wire MB_NODE as the native backend that emulates a MB control
> in the ARM MPAM driver.
>
> If emulation is unnecessary, this patch can be removed.
>
> 5. Documentation (patches 14–17)
>
> Document memory-level MB control and NUMA nodes in the arm64 MPAM
> guide (patch 19), NODE-scoped MBA domains and mon_NODE_* monitoring
> in the resctrl documentation (patch 20), and an MB_NODE emulation
> example on ARM MPAM (patch 21).
>
> 6. Tests (patches 18–19)
>
> Add a KUnit test for CPU-less NUMA node affinity handling and a
> kselftest that exercises MB emulation mode switching,
> resource_schemata hierarchy, and schemata mirroring on ARM MPAM.
>
> Apply the patches on top of [2].
>
> Testing
>
> # KUnit (CPU-less affinity)
> sudo cat /sys/kernel/debug/kunit/mpam_devices_test_suite/results
>
>
> # kselftest (MB emulation; requires ARM MPAM with disabled MB control)
> cd tools/testing/selftests/resctrl
> sudo ./resctrl_tests -t mb_emulation
>
> This series is in https://github.com/fyu1/linux/ cpu_less.rfc.v2/
> which is on top of [2].
>
Hi Fenghua,
I think the high-level direction looks good.
Just have some question about lifecycle model and object model.
Now MBM event and ABMC state are still partly global. For example, mpam_resctrl_mbm_total_mon()
selects one preferred total-bandwidth event, while the assignment interface redirects or hides
mbm_L3_assignment when node monitoring is present. On a platform exposing both events, L3 init
can consume NODE state, and userspace has no independent endpoint for the real L3 assignment.
the idea of borrowing of cpumask is interesting, but it's now used for several different concept:
domain membership, HW accessibility and worker placement. Borrowing a CPU mask for a CPU-less node
thus will make multiple node domains overlap, while generic lookup and worker-migration code still
assumes that a CPU identifies one monitor domain.
And should we make domain lifetime have an owner ? it's now shared between CPU hotplug and memory-node
hotplug for the complete transition.
If we can make these concept fixed or even clearer, I think it would be amazing, thanks.
Best regards,
Richard Cheng.
> Change Log:
> RFC v2:
> - Remove fix patches 1-2 which are implemented in [2]
> - Remove the emulation patches 3-8 which are implemented in [2]
> - Update emulated_by to *emulated_by in patch 11. Ben seems
> doesn't like to emulate MB by MB_NODE. If that's case, I can
> remove this patch. But for now, I keep this patch for further
> discussion.
> - Add patch 13 to implemente legacy and native modes to emulate
> "mbm_L3_assignments" by "mbm_NODE_assignments" in patch 13.
> - Add patch 9 for memory hotplug locking (Ben)
> - Add patch 12 for memory hotplug notifier (Ben)
>
> RFC v1:
> https://lore.kernel.org/lkml/cover.1784217438.git.fenghuay@nvidia.com/
>
> [1] [RFC v2] mpam,x86,fs/resctrl: Generic schema description Proof of Concept https://lore.kernel.org/lkml/d258a32f-12d5-464d-abe9-4720fb3e44b3@intel.com/
> [2] Reinette's resctrl control branch RFC v2: git://git.kernel.org/pub/scm/linux/kernel/git/reinette/linux.git branch resctrl/controls_rfc_v2.4
>
> Fenghua Yu (17):
> resctrl: De-hardcode L3 monitor infrastructure
> resctrl: Expose MBA MBM counter assignment sysfs
> resctrl: name node-scoped monitor domains mon_NODE_<id>
> resctrl: Add node-scope MBM total event
> resctrl: Make MBM paths resource-aware
> arm_mpam: Support memory-level MSCs and ABMC per class
> arm_mpam: Refine L3 topology and class selection
> arm_mpam: Include all MSC components during domain setup
> arm_mpam: Handle CPU-less numa nodes
> arm_mpam: Emulate MB control with node-scoped MB_NODE control
> resctrl: Add mbm_assign_scope_mode for native assignment file names
> Documentation: resctrl: document mbm_assign_scope_mode
> Documentation: arm64: mpam: document memory-level MB control and NUMA
> nodes
> Documentation: resctrl: document NODE-scoped MBA domains and mon_NODE
> monitoring
> Documentation: resctrl: document MB_NODE emulation example on ARM MPAM
> arm_mpam: Add KUnit test for CPU-less NUMA node affinity
> selftests/resctrl: Add MB emulation test for ARM MPAM
>
> James Morse (2):
> fs/resctrl: Take memory hotplug lock whenever taking CPU hotplug lock
> arm_mpam: resctrl: Add NUMA node notifier for domain online/offline
>
> Documentation/arch/arm64/mpam.rst | 105 ++-
> Documentation/filesystems/resctrl.rst | 135 ++-
> drivers/resctrl/mpam_devices.c | 59 +-
> drivers/resctrl/mpam_internal.h | 8 +
> drivers/resctrl/mpam_resctrl.c | 879 +++++++++++++++---
> drivers/resctrl/test_mpam_devices.c | 67 ++
> fs/resctrl/ctrlmondata.c | 6 +-
> fs/resctrl/internal.h | 18 +-
> fs/resctrl/monitor.c | 355 +++++--
> fs/resctrl/pseudo_lock.c | 3 +
> fs/resctrl/rdtgroup.c | 157 +++-
> include/linux/memory.h | 1 +
> include/linux/resctrl.h | 36 +-
> include/linux/resctrl_types.h | 15 +-
> .../selftests/resctrl/mb_emulation_test.c | 314 +++++++
> tools/testing/selftests/resctrl/resctrl.h | 1 +
> .../testing/selftests/resctrl/resctrl_tests.c | 1 +
> 17 files changed, 1827 insertions(+), 333 deletions(-)
> create mode 100644 tools/testing/selftests/resctrl/mb_emulation_test.c
>
> --
> 2.53.0
>
More information about the linux-arm-kernel
mailing list