[PATCH v5 00/38] mm: make VMA flag semantics explicit, eliminate VM_SPECIAL

[Andrew Morton]

On Fri, 09 Oct 2026 19:03:54 +0100 "Lorenzo Stoakes (ARM)" <ljs at kernel.org> wrote:

> The VM_SPECIAL / VMA_SPECIAL_FLAGS mask conflates several unrelated
> properties:
> 
> * Is this managed by core mm, or by a driver mapping MMIO, kernel-allocated
>   pages, or even ordinary pages itself?
> * Can it be expanded or merged?
> * Is this a 'weird' case like mlock where migration might race and we
>   'have' to set invalid flags to notify?
> * Is it another 'weird' case where we just want to stop GUP from touching
>   it?
> 
> Driver writers have often been confused about this, and who can blame them?
> 
> It also interacts badly with the eternal edgecase known as hugetlb - which
> sets VMA_DONTEXPAND_BIT but doesn't also want to be treated like a
> 'special' flag.
> 
> Another issue is that we cannot make sensible assumptions about flag
> use. It's not possible to assume VMA_IO_BIT means iommu because drivers
> abuse it and mlock abuses it.
> 
> Special is also an overloaded term in mm. VDSO and VVAR mappings are also
> called 'special' but they're special in a... special way.
> 
> Sometimes things are called special that are a subset of
> VMA_SPECIAL_FLAGS (VMA_PFNMAP_BIT and VMA_MIXEDMAP_BIT for instance when it
> comes to zapping or vm_normal_folio()).
> 
> There's a specific kind of special for THP too, which considers
> PFN map, mixed map 'special' but DAX not.
> 
> It's all rather a mess.
> 
> This series brings some order to things by both limiting what drivers can
> do with VMA flags and switching to using predicates that describe
> behaviour, not arbitrary flags.

Thanks, I've updated mm-unstable to this version.

For some reason, new versions of this series aren't applying when I
drop the new series in place of the old.  I made the same old bunch of
reject-fixups, so the below interdiff will show if I messed any of them up.

> v5:
> * Applied tags (thanks everybody!)
> * Updated 3/38 to fix a bug in __mmap_new_file_vma() by zapping + closing
>   the VMA before setting vma->vm_file NULL to avoid an i_mmap_writable
>   imbalance. A repro was tested before/after and confirmed fixed.
> * Updated 7/38 to make __map_kernel_pages_prepare() use
>   mmap_assert_write_locked() rather than the weird
>   assert-on-mmap_read_trylock() pattern, as per Suren.
> * Also update vm_insert_page(), vm_insert_pages() to fix the same pattern
>   and to use the new VMA API, and converted a BUG_ON to a
>   VM_WARN_ON_ONCE().

Here's how v5 altered mm.git:

 mm/memory.c                       |   20 +++++++++++---------
 mm/vma.c                          |   16 ++++++++--------
 tools/testing/vma/include/stubs.h |    4 ++++
 3 files changed, 23 insertions(+), 17 deletions(-)

--- a/mm/memory.c~b
+++ a/mm/memory.c
@@ -2611,10 +2611,11 @@ int vm_insert_pages(struct vm_area_struc
 
 	if (!range_in_vma(vma, addr, end))
 		return -EFAULT;
-	if (!(vma->vm_flags & VM_MIXEDMAP)) {
-		VM_WARN_ON_ONCE(mmap_read_trylock(vma->vm_mm));
-		VM_WARN_ON_ONCE(vma->vm_flags & VM_PFNMAP);
-		vm_flags_set(vma, VM_MIXEDMAP);
+	if (!vma_test(vma, VMA_MIXEDMAP_BIT)) {
+		mmap_assert_write_locked(vma->vm_mm);
+		VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT));
+		vma_start_write(vma);
+		vma_set_flags(vma, VMA_MIXEDMAP_BIT);
 	}
 	/* Defer page refcount checking till we're about to map that page. */
 	return insert_pages(vma, addr, pages, num, vma->vm_page_prot);
@@ -2626,7 +2627,7 @@ static void __map_kernel_pages_prepare(s
 	if (vma_desc_test(desc, VMA_MIXEDMAP_BIT))
 		return;
 
-	VM_WARN_ON_ONCE(mmap_read_trylock(desc->mm));
+	mmap_assert_write_locked(desc->mm);
 	VM_WARN_ON_ONCE(vma_desc_test(desc, VMA_PFNMAP_BIT));
 	vma_desc_set_flags(desc, VMA_MIXEDMAP_BIT);
 }
@@ -2785,10 +2786,11 @@ int vm_insert_page(struct vm_area_struct
 {
 	if (addr < vma->vm_start || addr >= vma->vm_end)
 		return -EFAULT;
-	if (!(vma->vm_flags & VM_MIXEDMAP)) {
-		BUG_ON(mmap_read_trylock(vma->vm_mm));
-		BUG_ON(vma->vm_flags & VM_PFNMAP);
-		vm_flags_set(vma, VM_MIXEDMAP);
+	if (!vma_test(vma, VMA_MIXEDMAP_BIT)) {
+		mmap_assert_write_locked(vma->vm_mm);
+		VM_WARN_ON_ONCE(vma_test(vma, VMA_PFNMAP_BIT));
+		vma_start_write(vma);
+		vma_set_flags(vma, VMA_MIXEDMAP_BIT);
 	}
 	return insert_page(vma, addr, page, vma->vm_page_prot, false);
 }
--- a/mm/vma.c~b
+++ a/mm/vma.c
@@ -2631,21 +2631,21 @@ static int __mmap_new_file_vma(struct mm
 	map->vm_file = vma->vm_file;
 
 	if (error) {
-		struct file *file = vma->vm_file;
 		UNMAP_STATE(unmap, vmi, vma, vma->vm_start, vma->vm_end,
 			    map->prev, map->next);
 
+		if (vma->vm_ops->close) {
+			zap_vma(vma);
+			/* Safe, because VMA zapped. */
+			vma_close(vma);
+		}
+		if (map_same_file(map))
+			fput(map->vm_file);
+
 		vma->vm_file = NULL;
 		vma_iter_set(vmi, vma->vm_end);
 		/* Undo any partial mapping done by a device driver. */
 		unmap_region(&unmap);
-		vma->vm_file = file;
-		/* Only safe once unmapped. */
-		vma_close(vma);
-
-		if (map_same_file(map))
-			fput(map->vm_file);
-		vma->vm_file = NULL;
 		return error;
 	}
 
--- a/tools/testing/vma/include/stubs.h~b
+++ a/tools/testing/vma/include/stubs.h
@@ -329,6 +329,10 @@ static inline void vma_close(struct vm_a
 {
 }
 
+static inline void zap_vma(struct vm_area_struct *vma)
+{
+}
+
 static inline int mmap_file(struct file *file, struct vm_area_struct *vma)
 {
 	return 0;
_