About add an A64FX cache control function into resctrl

[tan.shaopeng at fujitsu.com]

Hi Reinette,

I’m sorry for the late reply. 
I think I could not explain A64FX’s sector cache function well in 
my first mail. While answering the question, I will also explain 
this function in more detail. Though maybe you have already learned 
more about this function by reading specification and manual, 
in order to better understand this function, some contents may have 
duplicate explanations.

> >> The overview in section 12 was informative but very high level.
> >
> > I'm considering how to answer your questions from your email which I
> > received before, when I check the email again, I am sorry that the
> > information I provided before are insufficient.
> >
> > To understand the sector cache function of A64FX, could you please see
> > A64FX_Microarchitecture_Manual - section 12. Sector Cache
> >
> https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Microarchitectu
> > re_Manual_en_1.4.pdf
> > and,
> > A64FX_Specification_HPC_Extension ? section 1.2. Sector Cache
> >
> https://github.com/fujitsu/A64FX/blob/master/doc/A64FX_Specification_H
> > PC_Extension_v1_EN.pdf
> 
> Thank you for the direct links - I missed that there are two documents available.
> 
> After reading the spec portion it does seem to me even more as though
> "sectors" could be considered the same as the resctrl "classes of service". The
> Fujitsu hardware supports four sectors that can be configured with different
> number of ways using the registers you mention above. In resctrl this could be
> considered as hardware that supports four classes of service and each class of
> service can be allocated a different number of ways.

Fujitsu hardware supports four sectors that can be configured with 
different number of ways by using "IMP_SCCR" registers, and when this 
function is added into resctrl, the maximum ways of each sector are 
indicated by bitmap. 

However, A64FX's L2 cache setting registers are shared among PEs 
(Processor Element) in NUMA. If two PEs in the same NUMA are assigned 
to different resource groups, changing one PE's L2 setting on one 
resource group, the other PE's L2 setting on other resource groups 
will be influenced. So, adding this function into resctrl, we will 
assign NUMA to the resource group. (On F64FX, each NUMA has 12 PEs, 
and each PE has L1 cache setting registers, but these registers are 
not shared.) There are 4 NUMAs on A64FX, 4 NUMAs could be considered 
as hardware that supports four classes of service at most, and each 
class of service has 4 sectors (4 L1 sectors& 4 L2 sectors), 
and each sector can be allocated a different number of ways. 
And, when a running task on resource group, the [56:57] bits of 
virtual address are used for sector selection (cache affinity).

> The other part is how hardware knows which sector is being used at any
> moment in time. In resctrl that is programmed by writing the active class of
> service into needed register at the time the application is context switched
> (resctrl_sched_in()). This seems different here since as you describe the
> sector is chosen by bits in the address. Even so, which bits to set in the
> address needs to be programmed also and I also understand that there is a
> "default" sector that can be programmed via register. Could these be equivalent
> to what is done currently in resctrl?

Adding this function into resctrl, there is no need to write active 
class of service into needed register. When running a task, the sector 
id is decided by [56:57] bits of virtual address, and these bits are 
programed by users. When creating a resource group, the maximum number 
of ways of each sector are set by "IMP_SCCR" setting registers. 
As long as the task is running in a certain resource group, the sector 
and the maximum number of ways of sectors are used will not be changed. 
Therefore, we need not consider context switches on A64FX.

> (Could you please also consider my original questions?)
I will reply to the original questions mail. 


Best regards,
Tan Shaopeng