[EXT] Re: Testing DDR MPAM monitor (mbm_total_bytes)

[Peter Newman]

Hi Amit,

On Fri, Sep 22, 2023 at 9:08 PM Amit Singh Tomar <amitsinght at marvell.com> wrote:
> -----Original Message-----
> From: Peter Newman <peternewman at google.com>
> Sent: Tuesday, September 12, 2023 2:26 PM
> To: Amit Singh Tomar <amitsinght at marvell.com>
> Cc: james.morse at arm.com; Luck, Tony <tony.luck at intel.com>; Reinette Chatre <reinette.chatre at intel.com>; jonathan.cameron at huawei.com; carl at os.amperecomputing.com; Yu, Fenghua <fenghua.yu at intel.com>; George Cherian <gcherian at marvell.com>; linux-arm-kernel at lists.infradead.org; rohit.mathew at arm.com
 You explained earlier that two different MSCs have monitoring
capabilities on this hardware and this crash report is suggesting that
some monitoring events should have been assigned to the MBA resource.
> [>>] Yes, and in absence of event list (for MBA resource), this crash is triggered.
>
> The common resctrl code, especially the implementation of the RMID limbo list assumes a single monitoring resource in the system. That is, a single resource is expected to be able to tell you the memory bandwidth AND LLC occupancy of an RMID. That shouldn't be hard to fix, but if the LLC occupancy and MBM domains didn't align, fixing the limbo mechanism would become much more difficult.
>
> Can you start by explaining why monitoring resources should be assigned to different resources? What would have gone wrong if the DDR MSCs' bandwidth event counts were attached to the L3 resource? At a high level, the concept of a "resource" in resctrl seems abstract enough that it's difficult for me to understand why the MPAM code would choose to arrange things this way.
> [>>]
> But on ARM side, monitoring  capabilities are enumerated at individual MSC level, for instance, Cache storage usage monitor (LLC occupancy counter) is enumerated
> under L3 MSC, and DDR MPAM monitor (Memory BW) is enumerated under DDR MSC (corresponds to MBA resource).
>
> Monitor capability is set for both L3[1], and MBA[2] resource (from ARM specific resource control code).
>
> We may get away with this (avoid that crash, I mean) by doing something similarly to what X86 resource control does. But, as mentioned earlier, if DDR BW monitor is coupled with L3 resource, there would be feature mismatch as L3 resource class only has knowledge of the feature/control enumerated under L3 MSC. Unlike x86, there is no shared resource monitoring .i.e. L3 occupancy, and Memory BW on ARM side (at-least the way ARM specific resource control code is organized at the moment).
>
> Also, "I'd like to highlight another point, as per ARM MPAM spec, all these controls/monitors are optional, and if an implementation decide not to have L3 MSC, which effectively means there is no RDT_RESOURCE_L3 gets created (basically there is no CACHE class[3]). Then, how this whole thing would work? where it says
>
>      "MBM events are also part of RDT_RESOURCE_L3 resource because as per the SDM the total and local memory bandwidth
>       are enumerated as part of L3 monitoring".
>

It looks like associating the DDR MSCs' bandwidth counters with the
MBA resource is a choice made by James's prototype[1]. I'm sure we'll
be able to discuss it more when it becomes a topic of review.

It's my opinion that a resctrl driver could choose to associate
bandwidth counters residing on a memory controller with the L3
resource as long as the memory controller and L3 caches domains' are
the same. That seems to be the only implication of "level" and "scope"
in the resctrl fs code.

-Peter

[1] https://git.kernel.org/pub/scm/linux/kernel/git/morse/linux.git/commit/drivers/platform/mpam/mpam_resctrl.c?h=mpam/snapshot/v6.5-rc1&id=492bbaf1e09f9fcfe2ac37ddb3e6678014044453