[PATCH v8 03/10] mm: thp: Introduce multi-size THP sysfs interface

[Ryan Roberts]

On 06/12/2023 13:18, Ryan Roberts wrote:
> On 05/12/2023 16:57, David Hildenbrand wrote:
>> On 04.12.23 11:20, Ryan Roberts wrote:
>>> In preparation for adding support for anonymous multi-size THP,
>>> introduce new sysfs structure that will be used to control the new
>>> behaviours. A new directory is added under transparent_hugepage for each
>>> supported THP size, and contains an `enabled` file, which can be set to
>>> "inherit" (to inherit the global setting), "always", "madvise" or
>>> "never". For now, the kernel still only supports PMD-sized anonymous
>>> THP, so only 1 directory is populated.
>>>
>>> The first half of the change converts transhuge_vma_suitable() and
>>> hugepage_vma_check() so that they take a bitfield of orders for which
>>> the user wants to determine support, and the functions filter out all
>>> the orders that can't be supported, given the current sysfs
>>> configuration and the VMA dimensions. If there is only 1 order set in
>>> the input then the output can continue to be treated like a boolean;
>>> this is the case for most call sites. The resulting functions are
>>> renamed to thp_vma_suitable_orders() and thp_vma_allowable_orders()
>>> respectively.
>>>
>>> The second half of the change implements the new sysfs interface. It has
>>> been done so that each supported THP size has a `struct thpsize`, which
>>> describes the relevant metadata and is itself a kobject. This is pretty
>>> minimal for now, but should make it easy to add new per-thpsize files to
>>> the interface if needed in future (e.g. per-size defrag). Rather than
>>> keep the `enabled` state directly in the struct thpsize, I've elected to
>>> directly encode it into huge_anon_orders_[always|madvise|inherit]
>>> bitfields since this reduces the amount of work required in
>>> thp_vma_allowable_orders() which is called for every page fault.
>>>
>>> See Documentation/admin-guide/mm/transhuge.rst, as modified by this
>>> commit, for details of how the new sysfs interface works.
>>>
>>> Signed-off-by: Ryan Roberts <ryan.roberts at arm.com>
>>> ---
>>
>> Some comments mostly regarding thp_vma_allowable_orders and friends. In general,
>> LGTM. I'll have to go over the order logic once again, I got a bit lost once we
>> started mixing anon and file orders.
>>
>> [...]
>>
>> Doc updates all looked good to me, skimming over them.
>>
>>> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
>>> index fa0350b0812a..bd0eadd3befb 100644
>>> --- a/include/linux/huge_mm.h
>>> +++ b/include/linux/huge_mm.h
>>
>> [...]
>>
>>> +static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
>>> +        unsigned long addr, unsigned long orders)
>>> +{
>>> +    int order;
>>> +
>>> +    /*
>>> +     * Iterate over orders, highest to lowest, removing orders that don't
>>> +     * meet alignment requirements from the set. Exit loop at first order
>>> +     * that meets requirements, since all lower orders must also meet
>>> +     * requirements.
>>> +     */
>>> +
>>> +    order = first_order(orders);
>>
>> nit: "highest_order" or "largest_order" would be more expressive regarding the
>> actual semantics.
> 
> Yep, will call it "highest_order".
> 
>>
>>> +
>>> +    while (orders) {
>>> +        unsigned long hpage_size = PAGE_SIZE << order;
>>> +        unsigned long haddr = ALIGN_DOWN(addr, hpage_size);
>>> +
>>> +        if (haddr >= vma->vm_start &&
>>> +            haddr + hpage_size <= vma->vm_end) {
>>> +            if (!vma_is_anonymous(vma)) {
>>> +                if (IS_ALIGNED((vma->vm_start >> PAGE_SHIFT) -
>>> +                        vma->vm_pgoff,
>>> +                        hpage_size >> PAGE_SHIFT))
>>> +                    break;
>>> +            } else
>>> +                break;
>>
>> Comment: Codying style wants you to use if () {} else {}
>>
>> But I'd recommend for the conditions:
>>
>> if (haddr < vma->vm_start ||
>>     haddr + hpage_size > vma->vm_end)
>>     continue;
>> /* Don't have to check pgoff for anonymous vma */
>> if (!vma_is_anonymous(vma))
>>     break;
>> if (IS_ALIGNED((...
>>     break;
> 
> OK I'll take this structure.

FYI I ended up NOT taking this, because I now have thp_vma_suitable_order() and
thp_vma_suitable_orders(). The former is essentially what was there before
(except I pass the order and derive the nr_pages, alignment, etc from that
rather than using the HPAGE_* macros. The latter is just a loop that calls the
former for each order in the bitfield.

> 
>>
>> [...]
>>
>>
>>> +/**
>>> + * thp_vma_allowable_orders - determine hugepage orders that are allowed for vma
>>> + * @vma:  the vm area to check
>>> + * @vm_flags: use these vm_flags instead of vma->vm_flags
>>> + * @smaps: whether answer will be used for smaps file
>>> + * @in_pf: whether answer will be used by page fault handler
>>> + * @enforce_sysfs: whether sysfs config should be taken into account
>>> + * @orders: bitfield of all orders to consider
>>> + *
>>> + * Calculates the intersection of the requested hugepage orders and the allowed
>>> + * hugepage orders for the provided vma. Permitted orders are encoded as a set
>>> + * bit at the corresponding bit position (bit-2 corresponds to order-2, bit-3
>>> + * corresponds to order-3, etc). Order-0 is never considered a hugepage order.
>>> + *
>>> + * Return: bitfield of orders allowed for hugepage in the vma. 0 if no hugepage
>>> + * orders are allowed.
>>> + */
>>> +unsigned long thp_vma_allowable_orders(struct vm_area_struct *vma,
>>> +                       unsigned long vm_flags, bool smaps,
>>> +                       bool in_pf, bool enforce_sysfs,
>>> +                       unsigned long orders)
>>> +{
>>> +    /* Check the intersection of requested and supported orders. */
>>> +    orders &= vma_is_anonymous(vma) ?
>>> +            THP_ORDERS_ALL_ANON : THP_ORDERS_ALL_FILE;
>>> +    if (!orders)
>>> +        return 0;
>>
>> Comment: if this is called from some hot path, we might want to move as much as
>> possible into a header, so we can avoid this function call here when e.g., THP
>> are completely disabled etc.
> 
> If THP is completely disabled (compiled out) then thp_vma_allowable_orders() is
> defined as a header inline that returns 0. I'm not sure there are any paths in
> practice which are likely to ask for a set of orders which are never supported
> (i.e. where this specific check would return 0). And the "are they run time
> enabled" check is further down and fairly involved, so not sure that's ideal for
> an inline.
> 
> I haven't changed the pattern from how it was previously, so I don't think it
> should be any more expensive. Which parts exactly do you want to move to a header?

As per my response against the next patch (#4), I have now implemented
thp_vma_allowable_orders() so that the order check is in the header with an
early exit if THP is completely disabled.

> 
> 
>>
>>> +
>>>       if (!vma->vm_mm)        /* vdso */
>>> -        return false;
>>> +        return 0;
>>>         /*
>>>        * Explicitly disabled through madvise or prctl, or some
>>> @@ -88,16 +141,16 @@ bool hugepage_vma_check(struct vm_area_struct *vma,
>>> unsigned long vm_flags,
>>>        * */
>>>       if ((vm_flags & VM_NOHUGEPAGE) ||
>>>           test_bit(MMF_DISABLE_THP, &vma->vm_mm->flags))
>>> -        return false;
>>> +        return 0;
>>>       /*
>>>        * If the hardware/firmware marked hugepage support disabled.
>>>        */
>>>       if (transparent_hugepage_flags & (1 << TRANSPARENT_HUGEPAGE_UNSUPPORTED))
>>> -        return false;
>>> +        return 0;
>>>         /* khugepaged doesn't collapse DAX vma, but page fault is fine. */
>>>       if (vma_is_dax(vma))
>>> -        return in_pf;
>>> +        return in_pf ? orders : 0;
>>>         /*
>>>        * khugepaged special VMA and hugetlb VMA.
>>> @@ -105,17 +158,29 @@ bool hugepage_vma_check(struct vm_area_struct *vma,
>>> unsigned long vm_flags,
>>>        * VM_MIXEDMAP set.
>>>        */
>>>       if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED))
>>> -        return false;
>>> +        return 0;
>>>         /*
>>> -     * Check alignment for file vma and size for both file and anon vma.
>>> +     * Check alignment for file vma and size for both file and anon vma by
>>> +     * filtering out the unsuitable orders.
>>>        *
>>>        * Skip the check for page fault. Huge fault does the check in fault
>>> -     * handlers. And this check is not suitable for huge PUD fault.
>>> +     * handlers.
>>>        */
>>> -    if (!in_pf &&
>>> -        !transhuge_vma_suitable(vma, (vma->vm_end - HPAGE_PMD_SIZE)))
>>> -        return false;
>>> +    if (!in_pf) {
>>> +        int order = first_order(orders);
>>> +        unsigned long addr;
>>> +
>>> +        while (orders) {
>>> +            addr = vma->vm_end - (PAGE_SIZE << order);
>>> +            if (thp_vma_suitable_orders(vma, addr, BIT(order)))
>>> +                break;
>>
>> Comment: you'd want a "thp_vma_suitable_order" helper here. But maybe the
>> compiler is smart enough to optimize the loop and everyything else out.
> 
> I'm happy to refactor so that thp_vma_suitable_order() is the basic primitive,
> then make thp_vma_suitable_orders() a loop that calls thp_vma_suitable_order()
> (that's basically how it is laid out already, just all in one function). Is that
> what you are requesting?
> 
>>
>> [...]
>>
>>> +
>>> +static ssize_t thpsize_enabled_store(struct kobject *kobj,
>>> +                     struct kobj_attribute *attr,
>>> +                     const char *buf, size_t count)
>>> +{
>>> +    int order = to_thpsize(kobj)->order;
>>> +    ssize_t ret = count;
>>> +
>>> +    if (sysfs_streq(buf, "always")) {
>>> +        set_bit(order, &huge_anon_orders_always);
>>> +        clear_bit(order, &huge_anon_orders_inherit);
>>> +        clear_bit(order, &huge_anon_orders_madvise);
>>> +    } else if (sysfs_streq(buf, "inherit")) {
>>> +        set_bit(order, &huge_anon_orders_inherit);
>>> +        clear_bit(order, &huge_anon_orders_always);
>>> +        clear_bit(order, &huge_anon_orders_madvise);
>>> +    } else if (sysfs_streq(buf, "madvise")) {
>>> +        set_bit(order, &huge_anon_orders_madvise);
>>> +        clear_bit(order, &huge_anon_orders_always);
>>> +        clear_bit(order, &huge_anon_orders_inherit);
>>> +    } else if (sysfs_streq(buf, "never")) {
>>> +        clear_bit(order, &huge_anon_orders_always);
>>> +        clear_bit(order, &huge_anon_orders_inherit);
>>> +        clear_bit(order, &huge_anon_orders_madvise);
>>
>> Note: I was wondering for a second if some concurrent cames could lead to an
>> inconsistent state. I think in the worst case we'll simply end up with "never"
>> on races.
> 
> You mean if different threads try to write different values to this file
> concurrently? Or if there is a concurrent fault that tries to read the flags
> while they are being modified?
> 
> I thought about this for a long time too and wasn't sure what was best. The
> existing global enabled store impl clears the bits first then sets the bit. With
> this approach you can end up with multiple bits set if there is a race to set
> diffierent values, and you can end up with a faulting thread seeing never if it
> reads the bits after they have been cleared but before setting them.
> 
> I decided to set the new bit before clearing the old bits, which is different; A
> racing fault will never see "never" but as you say, a race to set the file could
> result in "never" being set.
> 
> On reflection, it's probably best to set the bit *last* like the global control
> does?>
> 
>>
>> [...]
>>
>>>   static int __init hugepage_init_sysfs(struct kobject **hugepage_kobj)
>>>   {
>>>       int err;
>>> +    struct thpsize *thpsize;
>>> +    unsigned long orders;
>>> +    int order;
>>> +
>>> +    /*
>>> +     * Default to setting PMD-sized THP to inherit the global setting and
>>> +     * disable all other sizes. powerpc's PMD_ORDER isn't a compile-time
>>> +     * constant so we have to do this here.
>>> +     */
>>> +    huge_anon_orders_inherit = BIT(PMD_ORDER);
>>>         *hugepage_kobj = kobject_create_and_add("transparent_hugepage", mm_kobj);
>>>       if (unlikely(!*hugepage_kobj)) {
>>> @@ -434,8 +631,24 @@ static int __init hugepage_init_sysfs(struct kobject
>>> **hugepage_kobj)
>>>           goto remove_hp_group;
>>>       }
>>>   +    orders = THP_ORDERS_ALL_ANON;
>>> +    order = first_order(orders);
>>> +    while (orders) {
>>> +        thpsize = thpsize_create(order, *hugepage_kobj);
>>> +        if (IS_ERR(thpsize)) {
>>> +            pr_err("failed to create thpsize for order %d\n", order);
>>> +            err = PTR_ERR(thpsize);
>>> +            goto remove_all;
>>> +        }
>>> +        list_add(&thpsize->node, &thpsize_list);
>>> +        order = next_order(&orders, order);
>>> +    }
>>> +
>>>       return 0;
>>>   
>>
>> [...]
>>
>>>       page = compound_head(page);
>>> @@ -5116,7 +5116,8 @@ static vm_fault_t __handle_mm_fault(struct
>>> vm_area_struct *vma,
>>>           return VM_FAULT_OOM;
>>>   retry_pud:
>>>       if (pud_none(*vmf.pud) &&
>>> -        hugepage_vma_check(vma, vm_flags, false, true, true)) {
>>> +        thp_vma_allowable_orders(vma, vm_flags, false, true, true,
>>> +                     BIT(PUD_ORDER))) {
>>>           ret = create_huge_pud(&vmf);
>>>           if (!(ret & VM_FAULT_FALLBACK))
>>>               return ret;
>>> @@ -5150,7 +5151,8 @@ static vm_fault_t __handle_mm_fault(struct
>>> vm_area_struct *vma,
>>>           goto retry_pud;
>>>         if (pmd_none(*vmf.pmd) &&
>>> -        hugepage_vma_check(vma, vm_flags, false, true, true)) {
>>> +        thp_vma_allowable_orders(vma, vm_flags, false, true, true,
>>> +                     BIT(PMD_ORDER))) {
>>
>> Comment: A helper like "thp_vma_allowable_order(vma, PMD_ORDER)" might make this
>> easier to read -- and the implemenmtation will be faster.
> 
> I'm happy to do this and use it to improve readability:
> 
> #define thp_vma_allowable_order(..., order) \
> 	thp_vma_allowable_orders(..., BIT(order))
> 
> This wouldn't make the implementation any faster though; Are you suggesting a
> completely separate impl? Even then, I don't think there is much scope to make
> it faster for the case where there is only 1 order in the bitfield.
> 
>>
>>>           ret = create_huge_pmd(&vmf);
>>>           if (!(ret & VM_FAULT_FALLBACK))
>>>               return ret;
>>> diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c
>>> index e0b368e545ed..64da127cc267 100644
>>> --- a/mm/page_vma_mapped.c
>>> +++ b/mm/page_vma_mapped.c
>>> @@ -268,7 +268,8 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw)
>>>                * cleared *pmd but not decremented compound_mapcount().
>>>                */
>>>               if ((pvmw->flags & PVMW_SYNC) &&
>>> -                transhuge_vma_suitable(vma, pvmw->address) &&
>>> +                thp_vma_suitable_orders(vma, pvmw->address,
>>> +                            BIT(PMD_ORDER)) &&
>>
>> Comment: Similarly, a helper like "thp_vma_suitable_order(vma, PMD_ORDER)" might
>> make this easier to read.
> 
> Yep, will do this.
> 
>>
>>>                   (pvmw->nr_pages >= HPAGE_PMD_NR)) {
>>>                   spinlock_t *ptl = pmd_lock(mm, pvmw->pmd);
>>>   
>>
>