Resctrl and I/O

[Zhanpeng Zhang]

Hi all,

On 9/23/26 12:51 AM, Luck, Tony wrote:
> Patches for ARM[1] and RISC-V[2] are both at RFC status.
>
> They appear to be based on similar h/w capabilities where architecture
> specific tags can be added to transactions from I/O devices by the IOMMU
> servicing the device. Device enumeration is handled outside of the resctrl
> file system.
>
> ARM docs say "the device name is the one listed under /sys/kernel/iommu_groups/<id>/devices/"
>
> RISC-V says architecture specific device tokens are "devices or device groups"
>
> In both cases assignments are made with:
>
> 	# echo {device} > {group}/devices
>
> So now I'm looking at the capabilities of Intel RDT hardware. I'm ignoring
> the implementation on the Granite Rapids CPU, and looking at what is described
> in the RDT Architecture Specification[3]. Here are the highlights:
>
> 1) Enumeration is split between two ACPI tables. IRDT enumerates device groups
> and provides MMIO addresses to assign CLOSID/RMID to each group. ERDT provides
> MMIO addresses to read event counts for RMIDs and to configure cache allocation
> bit masks and memory bandwidth limits.
>
> 2) All I/O monitoring and control is with respect to I/O caches, not to the L3
> caches that legacy RDT features operate on.
>
> 3) The number of RMIDs supported for I/O is lower (maybe considearbly lower)
> than the number supported for CPU monitoring.
>
> 4) Monitoring events are I/O cache occupancy, total i/o bandwidth, and i/o miss
> bandwidth.
>
> 5) Current revision (1.4) of the specification doesn't include details of
> I/O memory bandwidth control (The "IBAD" subtable of ERDT is reserved for this).
>
>
> Fitting this into resctrl
> -------------------------
>
> RISC-V description of "architecture specific device tokens" sounds great.
> I'll have to find some ares of /sys to provide the mapping of tokens to
> "device groups". There's no association with IOMMUs, so this will be some
> other path in /sys. Perhaps under /sys/firmware/acpi since the data comes
> from ACPI IRDT table.
>
> Need an info file to tell users the format and source for device tokens.
>
> I'd like to avoid limiting resctrl monitoring of CPU activity due to the small
> number of I/O RMIDs. Perhaps don't include a "devices" file or mon_data/{i/o
> events} files in resctrl group directories when rdtgroup::mon::rmid >=
> num_io_rmids? But that feels clumsy.
>
> Domains are based on I/O L3 cache instances. So a new rdt_resource with
> new line in the schemata file for IOL3 cache masks per domain. In the
> future when there are I/O MBA controls, new line(s) for those too.

Thanks for the feedback and the detailed discussion.

I have been occupied with some other work recently and have not had
enough time to follow all the replies closely. I will catch up on the
discussion and take the feedback into account before posting the next
version of the RISC-V patchset.

Thanks again for all the comments and suggestions.

>
> -Tony
>
>
> [1] https://lore.kernel.org/linux-iommu/20260901140802.1215508-1-xiaqinxin@huawei.com/
> [2] https://lore.kernel.org/lkml/20260714130657.46963-1-zhangzhanpeng.jasper@bytedance.com/
> [3] Linked from the bottom of https://www.intel.com/sdm under "Related Specifications", or
> direct link: https://cdrdv2.intel.com/v1/dl/getContent/789566?erpm_id=1158389_ts1790091453223


Best regards,
Zhanpeng