[PATCH] mmap: Fix hugetlb accounting error in __split_vma()

[Liam Howlett]

When splitting a vma fails due to allocations of the maple tree nodes,
the error path in __split_vma() calls new->vm_ops->close(new).  The
page accounting is actually in the close() operation for hugetlb, so it
accounts for the removal of 1/2 of the VMA which was not adjusted.  This
results in a negative exit value.  To avoid the negative charge, set
vm_start = vm_end and vm_pgoff = 0.

At the same time, move the vma_adjust_trans_huge() call below the
allocation call for the maple tree to avoid any other issues that may be
caused in such a scenario.

There is also a potential accounting issue in special mappings from
insert_vm_struct() failing to allocate, so reverse the charge there as
well.

Reported-by: syzbot+28eb226ee1d37f08087a at syzkaller.appspotmail.com
Fixes: 2ee236fe53a8 ("mm: start tracking VMAs with maple tree")
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/mmap.c | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 4d7f7a39b926..fa54d3c2f0f6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -758,10 +758,11 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
 				return error;
 		}
 	}
-	vma_adjust_trans_huge(orig_vma, start, end, adjust_next);
+
 	if (mas_preallocate(&mas, vma, GFP_KERNEL))
 		return -ENOMEM;
 
+	vma_adjust_trans_huge(orig_vma, start, end, adjust_next);
 	if (file) {
 		mapping = file->f_mapping;
 		root = &mapping->i_mmap;
@@ -2283,6 +2284,9 @@ int __split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	if (!err)
 		return 0;
 
+	/* Avoid vm accounting in close() operation */
+	new->vm_start = new->vm_end;
+	new->vm_pgoff = 0;
 	/* Clean everything up if vma_adjust failed. */
 	if (new->vm_ops && new->vm_ops->close)
 		new->vm_ops->close(new);
@@ -3168,11 +3172,13 @@ void exit_mmap(struct mm_struct *mm)
  */
 int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
 {
+	unsigned long charged = vma_pages(vma);
+
 	if (find_vma_intersection(mm, vma->vm_start, vma->vm_end))
 		return -ENOMEM;
 
 	if ((vma->vm_flags & VM_ACCOUNT) &&
-	     security_vm_enough_memory_mm(mm, vma_pages(vma)))
+	     security_vm_enough_memory_mm(mm, charged))
 		return -ENOMEM;
 
 	/*
@@ -3192,8 +3198,10 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
 		vma->vm_pgoff = vma->vm_start >> PAGE_SHIFT;
 	}
 
-	if (vma_link(mm, vma))
+	if (vma_link(mm, vma)) {
+		vm_unacct_memory(charged);
 		return -ENOMEM;
+	}
 
 	return 0;
 }
-- 
2.35.1