[PATCH 02/13] mm/vma: update do_mmap() to use vma_flags_t

[Zi Yan]

On Mon Jun 29, 2026 at 3:25 PM EDT, Lorenzo Stoakes wrote:
> The core do_mmap() function accepts a vm_flags_t parameter which it then
> manipulates before passing to mmap_region() to do the heavy lifting of the
> memory mapping.
>
> Update do_mmap() to instead accept a vma_flags_t parameter, and adjust all
> the logic within do_mmap() to manipulate this instead.
>
> This is as part of the ongoing effort to convert VMA flags from a system
> word size to a bitmap type which allows us to unrestrict the number of VMA
> flags, as well as gain control over how VMA flag manipulation occurs.
>
> We do not cascade these changes to all functions which accept vm_flags_t,
> but rather use vma_flags_to_legacy() where necessary, specifically
> deferring converting calc_vm_prot_bits(), calc_vm_flag_bits() and
> __get_unmapped_area() to vma_flags_t.
>
> Also utilise the new vma_flags_can_grow() predicate which correctly handles
> the case of architectures without upward growing stacks.
>
> As part of this change, introduce VMA_SHADOW_STACK so we can correctly
> handle the case of the shadow stack not being defined.
>
> No functional change intended.
>
> Signed-off-by: Lorenzo Stoakes <ljs at kernel.org>
> ---
>  arch/mips/kernel/vdso.c |  4 +--
>  fs/aio.c                |  2 +-
>  include/linux/memfd.h   |  6 ++--
>  include/linux/mm.h      |  6 ++--
>  ipc/shm.c               |  3 +-
>  mm/memfd.c              | 15 ++++-----
>  mm/mmap.c               | 67 ++++++++++++++++++++++++-----------------
>  mm/nommu.c              |  3 +-
>  mm/util.c               | 10 +++---
>  mm/vma.c                |  7 ++---
>  mm/vma.h                |  2 +-
>  11 files changed, 69 insertions(+), 56 deletions(-)
>

<snip>

>  
> -static int check_write_seal(vm_flags_t *vm_flags_ptr)
> +static int check_write_seal(vma_flags_t *vma_flags_ptr)
>  {
> -	vm_flags_t vm_flags = *vm_flags_ptr;
> -	vm_flags_t mask = vm_flags & (VM_SHARED | VM_WRITE);
> -
>  	/* If a private mapping then writability is irrelevant. */
> -	if (!(mask & VM_SHARED))
> +	if (!vma_flags_test(vma_flags_ptr, VMA_SHARED_BIT))
>  		return 0;
>  
>  	/*
>  	 * New PROT_WRITE and MAP_SHARED mmaps are not allowed when
>  	 * write seals are active.
>  	 */
> -	if (mask & VM_WRITE)
> +	if (vma_flags_test(vma_flags_ptr, VMA_WRITE_BIT))
>  		return -EPERM;
>  
>  	/*
>  	 * This is a read-only mapping, disallow mprotect() from making a
>  	 * write-sealed mapping writable in future.
>  	 */
> -	*vm_flags_ptr &= ~VM_MAYWRITE;
> +	vma_flags_clear(vma_flags_ptr, VMA_MAYWRITE_BIT);
>  
>  	return 0;
>  }

This function alone changed its original behavior, since vm_flags is a
snapshot of *vm_flags_ptr, but after the change this snapshot is gone.
But its only caller memfd_check_seals_mmap() gets vm_flags_ptr from the
input parameter of do_mmap(), so the overall behavior does not change.

<snip>

> +		case MAP_DROPPABLE: {
> +			vma_flags_t droppable = VMA_DROPPABLE;
> +
> +			if (vma_flags_empty(&droppable))
>  				return -EOPNOTSUPP;
> +			vma_flags_set_mask(&vma_flags, droppable);
> +
>  			/*
>  			 * A locked or stack area makes no sense to be droppable.
>  			 *
> @@ -515,23 +527,24 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
>  			 */
>  			if (flags & (MAP_LOCKED | MAP_HUGETLB))
>  			        return -EINVAL;
> -			if (vm_flags & (VM_GROWSDOWN | VM_GROWSUP))
> +			if (vma_flags_can_grow(&vma_flags))
>  			        return -EINVAL;
>  
> -			vm_flags |= VM_DROPPABLE;
> -

Lance pointed out the reordering of setting VMA_DROPPABLE and checking
of can_grow, but these flags are not overlapped and there is no parallel
writer to vma_flags. So it is still no functional change, just not
mechanical changes. :)

Otherwise, LGTM.

Reviewed-by: Zi Yan <ziy at nvidia.com>

-- 
Best Regards,
Yan, Zi