From michel at lespinasse.org  Sat May  1 06:04:41 2021
From: michel at lespinasse.org (Michel Lespinasse)
Date: Fri, 30 Apr 2021 22:04:41 -0700
Subject: [PATCH 01/94] mm: Add vma_lookup()
In-Reply-To: <20210428153542.2814175-2-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-2-Liam.Howlett@Oracle.com>
Message-ID: <20210501050441.GB5188@lespinasse.org>

On Wed, Apr 28, 2021 at 03:35:43PM +0000, Liam Howlett wrote:
> Many places in the kernel use find_vma() to get a vma and then check the
> start address of the vma to ensure the next vma was not returned.
> 
> Other places use the find_vma_intersection() call with add, addr + 1 as
> the range; looking for just the vma at a specific address.
> 
> The third use of find_vma() is by developers who do not know that the
> function starts searching at the provided address upwards for the next
> vma.  This results in a bug that is often overlooked for a long time.
> 
> Adding the new vma_lookup() function will allow for cleaner code by
> removing the find_vma() calls which check limits, making
> find_vma_intersection() calls of a single address to be shorter, and
> potentially reduce the incorrect uses of find_vma().
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>

This seems like a good API to have, and I agree it's less error prone than
having every caller check the vma->vm_start address.

Minor nitpick, I would prefer if the implementation used find_vma()
and then checked the vma->vm_start address - I don't like using [i, i+1)
intervals to implement stabbing queries.

But other than that, I think this (and the other patches adding
corresponding call sites) is safe for merging.

--
Michel "walken" Lespinasse


From michel at lespinasse.org  Sat May  1 06:13:30 2021
From: michel at lespinasse.org (Michel Lespinasse)
Date: Fri, 30 Apr 2021 22:13:30 -0700
Subject: [PATCH 38/94] mm/gup: Add mm_populate_vma() for use when the vma
 is known
In-Reply-To: <20210428153542.2814175-39-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-39-Liam.Howlett@Oracle.com>
Message-ID: <20210501051330.GC5188@lespinasse.org>

On Wed, Apr 28, 2021 at 03:36:08PM +0000, Liam Howlett wrote:
> When a vma is known, avoid calling mm_populate to search for the vma to
> populate.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> ---
>  mm/gup.c      | 20 ++++++++++++++++++++
>  mm/internal.h |  4 ++++
>  2 files changed, 24 insertions(+)
> 
> diff --git a/mm/gup.c b/mm/gup.c
> index c3a17b189064..48fe98ab0729 100644
> --- a/mm/gup.c
> +++ b/mm/gup.c
> @@ -1468,6 +1468,26 @@ long populate_vma_page_range(struct vm_area_struct *vma,
>  				NULL, NULL, locked);
>  }
>  
> +/*
> + * mm_populate_vma() - Populate a single range in a single vma.
> + * @vma: The vma to populate.
> + * @start: The start address to populate
> + * @end: The end address to stop populating
> + *
> + * Note: Ignores errors.
> + */
> +void mm_populate_vma(struct vm_area_struct *vma, unsigned long start,
> +		unsigned long end)
> +{
> +	struct mm_struct *mm = current->mm;
> +	int locked = 1;
> +
> +	mmap_read_lock(mm);
> +	populate_vma_page_range(vma, start, end, &locked);
> +	if (locked)
> +		mmap_read_unlock(mm);
> +}
> +

This seems like a nonsensical API at first glance - VMAs that are found
in the vma tree might be modified, merged, split, or freed at any time
if the mmap lock is not held, so the API can not be safely used. I think
this applies to maple tree vmas just as much as it did for rbtree vmas ?

--
Michel "walken" Lespinasse


From michel at lespinasse.org  Sat May  1 07:13:35 2021
From: michel at lespinasse.org (Michel Lespinasse)
Date: Fri, 30 Apr 2021 23:13:35 -0700
Subject: [PATCH 94/94] mm: Move mas locking outside of munmap() path.
In-Reply-To: <20210428153542.2814175-95-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-95-Liam.Howlett@Oracle.com>
Message-ID: <20210501061335.GD5188@lespinasse.org>

On Wed, Apr 28, 2021 at 03:36:32PM +0000, Liam Howlett wrote:
> Now that there is a split variant that allows splitting to use a maple state,
> move the locks to a more logical position.

In this patch set, is the maple tree lock ever held outside of code
sections already protected by the mmap write lock ?

--
Michel "walken" Lespinasse


From liam.howlett at oracle.com  Mon May  3 16:53:58 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 3 May 2021 15:53:58 +0000
Subject: [PATCH 38/94] mm/gup: Add mm_populate_vma() for use when the vma
 is known
In-Reply-To: <20210501051330.GC5188@lespinasse.org>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-39-Liam.Howlett@Oracle.com>
 <20210501051330.GC5188@lespinasse.org>
Message-ID: <20210503155352.74xqhqgizaagdro5@revolver>

* Michel Lespinasse <michel at lespinasse.org> [210501 01:13]:
> On Wed, Apr 28, 2021 at 03:36:08PM +0000, Liam Howlett wrote:
> > When a vma is known, avoid calling mm_populate to search for the vma to
> > populate.
> > 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> >  mm/gup.c      | 20 ++++++++++++++++++++
> >  mm/internal.h |  4 ++++
> >  2 files changed, 24 insertions(+)
> > 
> > diff --git a/mm/gup.c b/mm/gup.c
> > index c3a17b189064..48fe98ab0729 100644
> > --- a/mm/gup.c
> > +++ b/mm/gup.c
> > @@ -1468,6 +1468,26 @@ long populate_vma_page_range(struct vm_area_struct *vma,
> >  				NULL, NULL, locked);
> >  }
> >  
> > +/*
> > + * mm_populate_vma() - Populate a single range in a single vma.
> > + * @vma: The vma to populate.
> > + * @start: The start address to populate
> > + * @end: The end address to stop populating
> > + *
> > + * Note: Ignores errors.
> > + */
> > +void mm_populate_vma(struct vm_area_struct *vma, unsigned long start,
> > +		unsigned long end)
> > +{
> > +	struct mm_struct *mm = current->mm;
> > +	int locked = 1;
> > +
> > +	mmap_read_lock(mm);
> > +	populate_vma_page_range(vma, start, end, &locked);
> > +	if (locked)
> > +		mmap_read_unlock(mm);
> > +}
> > +
> 
> This seems like a nonsensical API at first glance - VMAs that are found
> in the vma tree might be modified, merged, split, or freed at any time
> if the mmap lock is not held, so the API can not be safely used. I think
> this applies to maple tree vmas just as much as it did for rbtree vmas ?

This is correct - it cannot be used without having the mmap_sem lock.
This is a new internal mm code API and is used to avoid callers that use
mm_populate() on a range that is known to be in a single VMA and already
have that VMA.  So instead of re-walking the tree to re-find the VMAs,
this function can be used with the known VMA and range.

It is used as described in patch 39 and 40 of this series.

Thanks,
Liam

From liam.howlett at oracle.com  Mon May  3 17:05:09 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 3 May 2021 16:05:09 +0000
Subject: [PATCH 94/94] mm: Move mas locking outside of munmap() path.
In-Reply-To: <20210501061335.GD5188@lespinasse.org>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-95-Liam.Howlett@Oracle.com>
 <20210501061335.GD5188@lespinasse.org>
Message-ID: <20210503160456.avsjwschcfua7uzy@revolver>

* Michel Lespinasse <michel at lespinasse.org> [210501 02:13]:
> On Wed, Apr 28, 2021 at 03:36:32PM +0000, Liam Howlett wrote:
> > Now that there is a split variant that allows splitting to use a maple state,
> > move the locks to a more logical position.
> 
> In this patch set, is the maple tree lock ever held outside of code
> sections already protected by the mmap write lock ?


No, the maple tree lock is a currently a subset of the mmap write lock.

Thanks,
Liam

From willy at infradead.org  Mon May  3 17:00:20 2021
From: willy at infradead.org (Matthew Wilcox)
Date: Mon, 3 May 2021 17:00:20 +0100
Subject: [PATCH 38/94] mm/gup: Add mm_populate_vma() for use when the vma
 is known
In-Reply-To: <20210503155352.74xqhqgizaagdro5@revolver>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-39-Liam.Howlett@Oracle.com>
 <20210501051330.GC5188@lespinasse.org>
 <20210503155352.74xqhqgizaagdro5@revolver>
Message-ID: <20210503160020.GC1847222@casper.infradead.org>

On Mon, May 03, 2021 at 03:53:58PM +0000, Liam Howlett wrote:
> * Michel Lespinasse <michel at lespinasse.org> [210501 01:13]:
> > On Wed, Apr 28, 2021 at 03:36:08PM +0000, Liam Howlett wrote:
> > > When a vma is known, avoid calling mm_populate to search for the vma to
> > > populate.
> > > 
> > > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > > ---
> > >  mm/gup.c      | 20 ++++++++++++++++++++
> > >  mm/internal.h |  4 ++++
> > >  2 files changed, 24 insertions(+)
> > > 
> > > diff --git a/mm/gup.c b/mm/gup.c
> > > index c3a17b189064..48fe98ab0729 100644
> > > --- a/mm/gup.c
> > > +++ b/mm/gup.c
> > > @@ -1468,6 +1468,26 @@ long populate_vma_page_range(struct vm_area_struct *vma,
> > >  				NULL, NULL, locked);
> > >  }
> > >  
> > > +/*
> > > + * mm_populate_vma() - Populate a single range in a single vma.
> > > + * @vma: The vma to populate.
> > > + * @start: The start address to populate
> > > + * @end: The end address to stop populating
> > > + *
> > > + * Note: Ignores errors.
> > > + */
> > > +void mm_populate_vma(struct vm_area_struct *vma, unsigned long start,
> > > +		unsigned long end)
> > > +{
> > > +	struct mm_struct *mm = current->mm;
> > > +	int locked = 1;
> > > +
> > > +	mmap_read_lock(mm);
> > > +	populate_vma_page_range(vma, start, end, &locked);
> > > +	if (locked)
> > > +		mmap_read_unlock(mm);
> > > +}
> > > +
> > 
> > This seems like a nonsensical API at first glance - VMAs that are found
> > in the vma tree might be modified, merged, split, or freed at any time
> > if the mmap lock is not held, so the API can not be safely used. I think
> > this applies to maple tree vmas just as much as it did for rbtree vmas ?
> 
> This is correct - it cannot be used without having the mmap_sem lock.
> This is a new internal mm code API and is used to avoid callers that use
> mm_populate() on a range that is known to be in a single VMA and already
> have that VMA.  So instead of re-walking the tree to re-find the VMAs,
> this function can be used with the known VMA and range.
> 
> It is used as described in patch 39 and 40 of this series.

In patch 39, what you do is:

1 Take the mmap_sem for write
2 do stuff
3 Drop the mmap_sem
4 Call mm_populate_vm() with the vma, which takes the mmap_sem
   for read

The problem is that between 3 & 4, a racing thread might cause us to free
the vma and so we've now passed a bogus pointer into mm_populate_vm().

What we need instead is to downgrade the mmap_sem from write to read at
step 3, so the vma is guaranteed to still be good.


From liam.howlett at oracle.com  Mon May  3 17:08:47 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 3 May 2021 16:08:47 +0000
Subject: [PATCH 01/94] mm: Add vma_lookup()
In-Reply-To: <20210501050441.GB5188@lespinasse.org>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-2-Liam.Howlett@Oracle.com>
 <20210501050441.GB5188@lespinasse.org>
Message-ID: <20210503160841.2juynuimujnaajad@revolver>

* Michel Lespinasse <michel at lespinasse.org> [210501 01:04]:
> On Wed, Apr 28, 2021 at 03:35:43PM +0000, Liam Howlett wrote:
> > Many places in the kernel use find_vma() to get a vma and then check the
> > start address of the vma to ensure the next vma was not returned.
> > 
> > Other places use the find_vma_intersection() call with add, addr + 1 as
> > the range; looking for just the vma at a specific address.
> > 
> > The third use of find_vma() is by developers who do not know that the
> > function starts searching at the provided address upwards for the next
> > vma.  This results in a bug that is often overlooked for a long time.
> > 
> > Adding the new vma_lookup() function will allow for cleaner code by
> > removing the find_vma() calls which check limits, making
> > find_vma_intersection() calls of a single address to be shorter, and
> > potentially reduce the incorrect uses of find_vma().
> > 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> 
> This seems like a good API to have, and I agree it's less error prone than
> having every caller check the vma->vm_start address.
> 
> Minor nitpick, I would prefer if the implementation used find_vma()
> and then checked the vma->vm_start address - I don't like using [i, i+1)
> intervals to implement stabbing queries.

Okay, I will make that change.

> 
> But other than that, I think this (and the other patches adding
> corresponding call sites) is safe for merging.
> 

Thanks,
Liam

From liam.howlett at oracle.com  Tue May  4 00:01:38 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 3 May 2021 23:01:38 +0000
Subject: [PATCH 38/94] mm/gup: Add mm_populate_vma() for use when the vma
 is known
In-Reply-To: <20210503160020.GC1847222@casper.infradead.org>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-39-Liam.Howlett@Oracle.com>
 <20210501051330.GC5188@lespinasse.org>
 <20210503155352.74xqhqgizaagdro5@revolver>
 <20210503160020.GC1847222@casper.infradead.org>
Message-ID: <20210503230130.vmrwp3xnckdnintj@revolver>

* Matthew Wilcox <willy at infradead.org> [210503 12:02]:
> On Mon, May 03, 2021 at 03:53:58PM +0000, Liam Howlett wrote:
> > * Michel Lespinasse <michel at lespinasse.org> [210501 01:13]:
> > > On Wed, Apr 28, 2021 at 03:36:08PM +0000, Liam Howlett wrote:
> > > > When a vma is known, avoid calling mm_populate to search for the vma to
> > > > populate.
> > > > 
> > > > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > > > ---
> > > >  mm/gup.c      | 20 ++++++++++++++++++++
> > > >  mm/internal.h |  4 ++++
> > > >  2 files changed, 24 insertions(+)
> > > > 
> > > > diff --git a/mm/gup.c b/mm/gup.c
> > > > index c3a17b189064..48fe98ab0729 100644
> > > > --- a/mm/gup.c
> > > > +++ b/mm/gup.c
> > > > @@ -1468,6 +1468,26 @@ long populate_vma_page_range(struct vm_area_struct *vma,
> > > >  				NULL, NULL, locked);
> > > >  }
> > > >  
> > > > +/*
> > > > + * mm_populate_vma() - Populate a single range in a single vma.
> > > > + * @vma: The vma to populate.
> > > > + * @start: The start address to populate
> > > > + * @end: The end address to stop populating
> > > > + *
> > > > + * Note: Ignores errors.
> > > > + */
> > > > +void mm_populate_vma(struct vm_area_struct *vma, unsigned long start,
> > > > +		unsigned long end)
> > > > +{
> > > > +	struct mm_struct *mm = current->mm;
> > > > +	int locked = 1;
> > > > +
> > > > +	mmap_read_lock(mm);
> > > > +	populate_vma_page_range(vma, start, end, &locked);
> > > > +	if (locked)
> > > > +		mmap_read_unlock(mm);
> > > > +}
> > > > +
> > > 
> > > This seems like a nonsensical API at first glance - VMAs that are found
> > > in the vma tree might be modified, merged, split, or freed at any time
> > > if the mmap lock is not held, so the API can not be safely used. I think
> > > this applies to maple tree vmas just as much as it did for rbtree vmas ?
> > 
> > This is correct - it cannot be used without having the mmap_sem lock.
> > This is a new internal mm code API and is used to avoid callers that use
> > mm_populate() on a range that is known to be in a single VMA and already
> > have that VMA.  So instead of re-walking the tree to re-find the VMAs,
> > this function can be used with the known VMA and range.
> > 
> > It is used as described in patch 39 and 40 of this series.
> 
> In patch 39, what you do is:
> 
> 1 Take the mmap_sem for write
> 2 do stuff
> 3 Drop the mmap_sem
> 4 Call mm_populate_vm() with the vma, which takes the mmap_sem
>    for read
> 
> The problem is that between 3 & 4, a racing thread might cause us to free
> the vma and so we've now passed a bogus pointer into mm_populate_vm().
> 
> What we need instead is to downgrade the mmap_sem from write to read at
> step 3, so the vma is guaranteed to still be good.

Thank you.  I will remove these patches from the series and work on this
idea.

Regards,
Liam


From hdanton at sina.com  Tue May  4 05:29:30 2021
From: hdanton at sina.com (Hillf Danton)
Date: Tue,  4 May 2021 12:29:30 +0800
Subject: [PATCH 76/94] mm/khugepaged: Use maple tree iterators instead of
 vma
In-Reply-To: <20210428153542.2814175-77-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
Message-ID: <20210504042930.10436-1-hdanton@sina.com>

On Wed, 28 Apr 2021 15:36:24  Liam R. Howlett wrote:
> ---
>  mm/khugepaged.c | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)

Good work. And a nit below.
> 
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 33cf91529f0b..4983a25c5a90 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -2063,6 +2063,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> int pages,
>  	struct mm_struct *mm;
>  	struct vm_area_struct *vma;
>  	int progress = 0;
> +	MA_STATE(mas, NULL, 0, 0);
> 
>  	VM_BUG_ON(!pages);
>  	lockdep_assert_held(&khugepaged_mm_lock);
> @@ -2079,18 +2080,22 @@ static unsigned int khugepaged_scan_mm_slot(unsigne=
> d int pages,
>  	khugepaged_collapse_pte_mapped_thps(mm_slot);
> 
>  	mm = mm_slot->mm;
> +	mas.tree = &mm->mm_mt;
>  	/*
>  	 * Don't wait for semaphore (to avoid long wait times).  Just move to
>  	 * the next mm on the list.
>  	 */
>  	vma = NULL;
> +	mas_set(&mas, khugepaged_scan.address);
>  	if (unlikely(!mmap_read_trylock(mm)))
>  		goto breakouterloop_mmap_lock;
> +
> +	rcu_read_lock();
>  	if (likely(!khugepaged_test_exit(mm)))
> -		vma = find_vma(mm, khugepaged_scan.address);
> +		vma = mas_find(&mas, ULONG_MAX);
> 
>  	progress++;
> -	for (; vma; vma = vma->vm_next) {
> +	mas_for_each(&mas, vma, ULONG_MAX) {
>  		unsigned long hstart, hend;
> 
>  		cond_resched();

Given the added rcu_read_lock(), take another look at this scheduling
entry?

> @@ -2129,6 +2134,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> int pages,
>  				pgoff_t pgoff = linear_page_index(vma,
>  						khugepaged_scan.address);
> 
> +				rcu_read_unlock();
>  				mmap_read_unlock(mm);
>  				ret = 1;
>  				khugepaged_scan_file(mm, file, pgoff, hpage);
> @@ -2149,6 +2155,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> int pages,
>  		}
>  	}
>  breakouterloop:
> +	rcu_read_unlock();
>  	mmap_read_unlock(mm); /* exit_mmap will destroy ptes after this */
>  breakouterloop_mmap_lock:
> 
> --
> 2.30.2


From liam.howlett at oracle.com  Tue May  4 15:05:26 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Tue, 4 May 2021 14:05:26 +0000
Subject: [PATCH 76/94] mm/khugepaged: Use maple tree iterators instead of
 vma
In-Reply-To: <20210504042930.10436-1-hdanton@sina.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210504042930.10436-1-hdanton@sina.com>
Message-ID: <20210504140454.22isjwj66vydyzmy@revolver>

* Hillf Danton <hdanton at sina.com> [210504 00:29]:
> On Wed, 28 Apr 2021 15:36:24  Liam R. Howlett wrote:
> > ---
> >  mm/khugepaged.c | 11 +++++++++--
> >  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> Good work. And a nit below.
> > 
> > diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> > index 33cf91529f0b..4983a25c5a90 100644
> > --- a/mm/khugepaged.c
> > +++ b/mm/khugepaged.c
> > @@ -2063,6 +2063,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> > int pages,
> >  	struct mm_struct *mm;
> >  	struct vm_area_struct *vma;
> >  	int progress = 0;
> > +	MA_STATE(mas, NULL, 0, 0);
> > 
> >  	VM_BUG_ON(!pages);
> >  	lockdep_assert_held(&khugepaged_mm_lock);
> > @@ -2079,18 +2080,22 @@ static unsigned int khugepaged_scan_mm_slot(unsigne=
> > d int pages,
> >  	khugepaged_collapse_pte_mapped_thps(mm_slot);
> > 
> >  	mm = mm_slot->mm;
> > +	mas.tree = &mm->mm_mt;
> >  	/*
> >  	 * Don't wait for semaphore (to avoid long wait times).  Just move to
> >  	 * the next mm on the list.
> >  	 */
> >  	vma = NULL;
> > +	mas_set(&mas, khugepaged_scan.address);
> >  	if (unlikely(!mmap_read_trylock(mm)))
> >  		goto breakouterloop_mmap_lock;
> > +
> > +	rcu_read_lock();
> >  	if (likely(!khugepaged_test_exit(mm)))
> > -		vma = find_vma(mm, khugepaged_scan.address);
> > +		vma = mas_find(&mas, ULONG_MAX);
> > 
> >  	progress++;
> > -	for (; vma; vma = vma->vm_next) {
> > +	mas_for_each(&mas, vma, ULONG_MAX) {
> >  		unsigned long hstart, hend;
> > 
> >  		cond_resched();
> 
> Given the added rcu_read_lock(), take another look at this scheduling
> entry?
> 

Yes, thank you for pointing this out.  It is certainly an issue.

> > @@ -2129,6 +2134,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> > int pages,
> >  				pgoff_t pgoff = linear_page_index(vma,
> >  						khugepaged_scan.address);
> > 
> > +				rcu_read_unlock();
> >  				mmap_read_unlock(mm);
> >  				ret = 1;
> >  				khugepaged_scan_file(mm, file, pgoff, hpage);
> > @@ -2149,6 +2155,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned =
> > int pages,
> >  		}
> >  	}
> >  breakouterloop:
> > +	rcu_read_unlock();
> >  	mmap_read_unlock(mm); /* exit_mmap will destroy ptes after this */
> >  breakouterloop_mmap_lock:
> > 
> > --
> > 2.30.2

From liam.howlett at oracle.com  Mon May 10 17:58:54 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:54 +0000
Subject: [PATCH 12/22] net/ipv5/tcp: Use vma_lookup() in tcp_zerocopy_receive()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-13-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 net/ipv4/tcp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index e14fd0c50c10..d4781a514012 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -2094,8 +2094,8 @@ static int tcp_zerocopy_receive(struct sock *sk,
 
 	mmap_read_lock(current->mm);
 
-	vma = find_vma(current->mm, address);
-	if (!vma || vma->vm_start > address || vma->vm_ops != &tcp_vm_ops) {
+	vma = vma_lookup(current->mm, address);
+	if (!vma || vma->vm_ops != &tcp_vm_ops) {
 		mmap_read_unlock(current->mm);
 		return -EINVAL;
 	}
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:55 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:55 +0000
Subject: [PATCH 14/22] media: videobuf2: Use vma_lookup() in get_vaddr_frames()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-15-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 drivers/media/common/videobuf2/frame_vector.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/media/common/videobuf2/frame_vector.c b/drivers/media/common/videobuf2/frame_vector.c
index 91fea7199e85..b84b706073cb 100644
--- a/drivers/media/common/videobuf2/frame_vector.c
+++ b/drivers/media/common/videobuf2/frame_vector.c
@@ -64,7 +64,7 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames,
 	do {
 		unsigned long *nums = frame_vector_pfns(vec);
 
-		vma = find_vma_intersection(mm, start, start + 1);
+		vma = vma_lookup(mm, start);
 		if (!vma)
 			break;
 
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:59 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:59 +0000
Subject: [PATCH 21/22] mm/memory.c: Use vma_lookup() in __access_remote_vm()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-22-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/memory.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 86ba6c1f6821..e3b56903b111 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4954,8 +4954,8 @@ int __access_remote_vm(struct mm_struct *mm, unsigned long addr, void *buf,
 			 * Check if this is a VM_IO | VM_PFNMAP VMA, which
 			 * we can access using slightly different code.
 			 */
-			vma = find_vma(mm, addr);
-			if (!vma || vma->vm_start > addr)
+			vma = vma_lookup(mm, addr);
+			if (!vma)
 				break;
 			if (vma->vm_ops && vma->vm_ops->access)
 				ret = vma->vm_ops->access(vma, addr, buf,
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:54 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:54 +0000
Subject: [PATCH 13/22] drm/amdgpu: Use vma_lookup() in
 amdgpu_ttm_tt_get_user_pages()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-14-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 3251f6b67e23..00b7fa8b953b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -694,9 +694,9 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages)
 		return -ESRCH;
 
 	mmap_read_lock(mm);
-	vma = find_vma(mm, start);
+	vma = vma_lookup(mm, start);
 	mmap_read_unlock(mm);
-	if (unlikely(!vma || start < vma->vm_start)) {
+	if (unlikely(!vma)) {
 		r = -EFAULT;
 		goto out_putmm;
 	}
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:58 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:58 +0000
Subject: [PATCH 19/22] mm/migrate: Use vma_lookup() in do_pages_stat_array()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-20-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/migrate.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/migrate.c b/mm/migrate.c
index b234c3f3acb7..611781c0f9b5 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1833,8 +1833,8 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 		struct page *page;
 		int err = -EFAULT;
 
-		vma = find_vma(mm, addr);
-		if (!vma || addr < vma->vm_start)
+		vma = vma_lookup(mm, addr);
+		if (!vma)
 			goto set_status;
 
 		/* FOLL_DUMP to ignore special (like zero) pages */
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:59 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:59 +0000
Subject: [PATCH 22/22] mm/mempolicy: Use vma_lookup() in __access_remote_vm()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-23-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/mempolicy.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index d79fa299b70c..325771bef5e2 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -975,7 +975,7 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,
 		 * want to return MPOL_DEFAULT in this case.
 		 */
 		mmap_read_lock(mm);
-		vma = find_vma_intersection(mm, addr, addr+1);
+		vma = vma_lookup(mm, addr);
 		if (!vma) {
 			mmap_read_unlock(mm);
 			return -EFAULT;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:58 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:58 +0000
Subject: [PATCH 20/22] mm/mremap: Use vma_lookup() in vma_to_resize()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-21-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/mremap.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/mremap.c b/mm/mremap.c
index 47c255b60150..04143755cd1e 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -634,10 +634,10 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr,
 	unsigned long *p)
 {
 	struct mm_struct *mm = current->mm;
-	struct vm_area_struct *vma = find_vma(mm, addr);
+	struct vm_area_struct *vma = vma_lookup(mm, addr);
 	unsigned long pgoff;
 
-	if (!vma || vma->vm_start > addr)
+	if (!vma)
 		return ERR_PTR(-EFAULT);
 
 	/*
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:49 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:49 +0000
Subject: [PATCH 01/22] mm: Add vma_lookup()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-2-Liam.Howlett@Oracle.com>

Many places in the kernel use find_vma() to get a vma and then check the
start address of the vma to ensure the next vma was not returned.

Other places use the find_vma_intersection() call with add, addr + 1 as
the range; looking for just the vma at a specific address.

The third use of find_vma() is by developers who do not know that the
function starts searching at the provided address upwards for the next
vma.  This results in a bug that is often overlooked for a long time.

Adding the new vma_lookup() function will allow for cleaner code by
removing the find_vma() calls which check limits, making
find_vma_intersection() calls of a single address to be shorter, and
potentially reduce the incorrect uses of find_vma().

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 include/linux/mm.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 25b9041f9925..5f2a15e702ff 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2689,6 +2689,24 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m
 	return vma;
 }
 
+/**
+ * vma_lookup() - Find a VMA at a specific address
+ * @mm: The process address space.
+ * @addr: The user address.
+ *
+ * Return: The vm_area_struct at the given address, %NULL otherwise.
+ */
+static inline
+struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
+{
+	struct vm_area_struct * vma = find_vma(mm, addr);
+
+	if (vma && addr < vma->vm_start)
+		vma = NULL;
+
+	return vma;
+}
+
 static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
 {
 	unsigned long vm_start = vma->vm_start;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:49 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:49 +0000
Subject: [PATCH 02/22] drm/i915/selftests: Use vma_lookup() in __igt_mmap()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-3-Liam.Howlett@Oracle.com>

vma_lookup() will look up the vma at a specific address.  find_vma()
will start the search for a specific address and continue upwards.  This
fixes an issue with the selftest as the returned vma may not be the
newly created vma, but simply the vma at a higher address.

Fixes: 6fedafacae1b (drm/i915/selftests: Wrap vm_mmap() around GEM
objects
Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
index 5cf6df49c333..35c15ef1327d 100644
--- a/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
+++ b/drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c
@@ -871,7 +871,7 @@ static int __igt_mmap(struct drm_i915_private *i915,
 
 	pr_debug("igt_mmap(%s, %d) @ %lx\n", obj->mm.region->name, type, addr);
 
-	area = find_vma(current->mm, addr);
+	area = vma_lookup(current->mm, addr);
 	if (!area) {
 		pr_err("%s: Did not create a vm_area_struct for the mmap\n",
 		       obj->mm.region->name);
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:52 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:52 +0000
Subject: [PATCH 07/22] arch/mips/kernel/traps: Use vma_lookup() instead of
 find_vma()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-8-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/mips/kernel/traps.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 0b4e06303c55..6f07362de5ce 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -784,7 +784,6 @@ void force_fcr31_sig(unsigned long fcr31, void __user *fault_addr,
 int process_fpemu_return(int sig, void __user *fault_addr, unsigned long fcr31)
 {
 	int si_code;
-	struct vm_area_struct *vma;
 
 	switch (sig) {
 	case 0:
@@ -800,8 +799,7 @@ int process_fpemu_return(int sig, void __user *fault_addr, unsigned long fcr31)
 
 	case SIGSEGV:
 		mmap_read_lock(current->mm);
-		vma = find_vma(current->mm, (unsigned long)fault_addr);
-		if (vma && (vma->vm_start <= (unsigned long)fault_addr))
+		if (vma_lookup(current->mm, (unsigned long)fault_addr))
 			si_code = SEGV_ACCERR;
 		else
 			si_code = SEGV_MAPERR;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:52 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:52 +0000
Subject: [PATCH 09/22] x86/sgx: Use vma_lookup() in sgx_encl_find()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-10-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/x86/kernel/cpu/sgx/encl.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kernel/cpu/sgx/encl.h b/arch/x86/kernel/cpu/sgx/encl.h
index 6e74f85b6264..fec43ca65065 100644
--- a/arch/x86/kernel/cpu/sgx/encl.h
+++ b/arch/x86/kernel/cpu/sgx/encl.h
@@ -91,8 +91,8 @@ static inline int sgx_encl_find(struct mm_struct *mm, unsigned long addr,
 {
 	struct vm_area_struct *result;
 
-	result = find_vma(mm, addr);
-	if (!result || result->vm_ops != &sgx_vm_ops || addr < result->vm_start)
+	result = vma_lookup(mm, addr);
+	if (!result || result->vm_ops != &sgx_vm_ops)
 		return -EINVAL;
 
 	*vma = result;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:51 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:51 +0000
Subject: [PATCH 06/22] arch/powerpc/kvm/book3s: Use vma_lookup() in
 kvmppc_hv_setup_htab_rma()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-7-Liam.Howlett@Oracle.com>

Using vma_lookup() removes the requirement to check if the address is
within the returned vma.  The code is easier to understand and more
compact.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/powerpc/kvm/book3s_hv.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 28a80d240b76..a3a4b2179350 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -4759,8 +4759,8 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)
 	/* Look up the VMA for the start of this memory slot */
 	hva = memslot->userspace_addr;
 	mmap_read_lock(kvm->mm);
-	vma = find_vma(kvm->mm, hva);
-	if (!vma || vma->vm_start > hva || (vma->vm_flags & VM_IO))
+	vma = vma_lookup(kvm->mm, hva);
+	if (!vma || (vma->vm_flags & VM_IO))
 		goto up_out;
 
 	psize = vma_kernel_pagesize(vma);
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:50 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:50 +0000
Subject: [PATCH 04/22] arch/arm64/kvm: Use vma_lookup() instead of
 find_vma_intersection()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-5-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/arm64/kvm/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index c5d1f3c87dbd..8b112b594e09 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -855,7 +855,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
 	/* Let's check if we will get back a huge page backed by hugetlbfs */
 	mmap_read_lock(current->mm);
-	vma = find_vma_intersection(current->mm, hva, hva + 1);
+	vma = vma_lookup(current->mm, hva);
 	if (unlikely(!vma)) {
 		kvm_err("Failed to find VMA for hva 0x%lx\n", hva);
 		mmap_read_unlock(current->mm);
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:49 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:49 +0000
Subject: [PATCH 03/22] arch/arc/kernel/troubleshoot: use vma_lookup() instead
 of find_vma()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-4-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/arc/kernel/troubleshoot.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arc/kernel/troubleshoot.c b/arch/arc/kernel/troubleshoot.c
index a331bb5d8319..7654c2e42dc0 100644
--- a/arch/arc/kernel/troubleshoot.c
+++ b/arch/arc/kernel/troubleshoot.c
@@ -83,12 +83,12 @@ static void show_faulting_vma(unsigned long address)
 	 * non-inclusive vma
 	 */
 	mmap_read_lock(active_mm);
-	vma = find_vma(active_mm, address);
+	vma = vma_lookup(active_mm, address);
 
-	/* check against the find_vma( ) behaviour which returns the next VMA
-	 * if the container VMA is not found
+	/* Lookup the vma at the address and report if the container VMA is not
+	 * found
 	 */
-	if (vma && (vma->vm_start <= address)) {
+	if (vma) {
 		char buf[ARC_PATH_MAX];
 		char *nm = "?";
 
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:53 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:53 +0000
Subject: [PATCH 10/22] virt/kvm: Use vma_lookup() instead of
 find_vma_intersection()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-11-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 virt/kvm/kvm_main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 2799c6660cce..a7703b11407a 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2154,7 +2154,7 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async,
 	}
 
 retry:
-	vma = find_vma_intersection(current->mm, addr, addr + 1);
+	vma = vma_lookup(current->mm, addr);
 
 	if (vma == NULL)
 		pfn = KVM_PFN_ERR_FAULT;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:52 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:52 +0000
Subject: [PATCH 08/22] arch/m68k/kernel/sys_m68k: Use vma_lookup() in
 sys_cacheflush()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-9-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/m68k/kernel/sys_m68k.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/m68k/kernel/sys_m68k.c b/arch/m68k/kernel/sys_m68k.c
index f55bdcb8e4f1..bd0274c7592e 100644
--- a/arch/m68k/kernel/sys_m68k.c
+++ b/arch/m68k/kernel/sys_m68k.c
@@ -402,8 +402,8 @@ sys_cacheflush (unsigned long addr, int scope, int cache, unsigned long len)
 		 * to this process.
 		 */
 		mmap_read_lock(current->mm);
-		vma = find_vma(current->mm, addr);
-		if (!vma || addr < vma->vm_start || addr + len > vma->vm_end)
+		vma = vma_lookup(current->mm, addr);
+		if (!vma || addr + len > vma->vm_end)
 			goto out_unlock;
 	}
 
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:57 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:57 +0000
Subject: [PATCH 18/22] mm/ksm: Use vma_lookup() in find_mergeable_vma()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-19-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/ksm.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/mm/ksm.c b/mm/ksm.c
index 6bbe314c5260..ced6830d0ff4 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -521,10 +521,8 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,
 	struct vm_area_struct *vma;
 	if (ksm_test_exit(mm))
 		return NULL;
-	vma = find_vma(mm, addr);
-	if (!vma || vma->vm_start > addr)
-		return NULL;
-	if (!(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
+	vma = vma_lookup(mm, addr);
+	if (!vma || !(vma->vm_flags & VM_MERGEABLE) || !vma->anon_vma)
 		return NULL;
 	return vma;
 }
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:56 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:56 +0000
Subject: [PATCH 15/22] misc/sgi-gru/grufault: Use vma_lookup() in
 gru_find_vma()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-16-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 drivers/misc/sgi-gru/grufault.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c
index 723825524ea0..d7ef61e602ed 100644
--- a/drivers/misc/sgi-gru/grufault.c
+++ b/drivers/misc/sgi-gru/grufault.c
@@ -49,8 +49,8 @@ struct vm_area_struct *gru_find_vma(unsigned long vaddr)
 {
 	struct vm_area_struct *vma;
 
-	vma = find_vma(current->mm, vaddr);
-	if (vma && vma->vm_start <= vaddr && vma->vm_ops == &gru_vm_ops)
+	vma = vma_lookup(current->mm, vaddr);
+	if (vma && vma->vm_ops == &gru_vm_ops)
 		return vma;
 	return NULL;
 }
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:48 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:48 +0000
Subject: [PATCH 00/21] mm: Add vma_lookup()
Message-ID: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>

Many places in the kernel use find_vma() to get a vma and then check the
start address of the vma to ensure the next vma was not returned.

Other places use the find_vma_intersection() call with add, addr + 1 as
the range; looking for just the vma at a specific address.

The third use of find_vma() is by developers who do not know that the
function starts searching at the provided address upwards for the next
vma.  This results in a bug that is often overlooked for a long time.

Adding the new vma_lookup() function will allow for cleaner code by
removing the find_vma() calls which check limits, making
find_vma_intersection() calls of a single address to be shorter, and
potentially reduce the incorrect uses of find_vma().

This patch set was originally part of the maple tree patch set.
Changes since being broken out on its own:
- Changed initial implementation of vma_lookup() to use find_vma() as requested
  by Michel Lespinasse.
- Updated commit comments to be more descriptive

These patches are based on next-20210506

Liam R. Howlett (22):
  mm: Add vma_lookup()
  drm/i915/selftests: Use vma_lookup() in __igt_mmap()
  arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma()
  arch/arm64/kvm: Use vma_lookup() instead of find_vma_intersection()
  arch/powerpc/kvm/book3s_hv_uvmem: Use vma_lookup() instead of
    find_vma_intersection()
  arch/powerpc/kvm/book3s: Use vma_lookup() in
    kvmppc_hv_setup_htab_rma()
  arch/mips/kernel/traps: Use vma_lookup() instead of find_vma()
  arch/m68k/kernel/sys_m68k: Use vma_lookup() in sys_cacheflush()
  x86/sgx: Use vma_lookup() in sgx_encl_find()
  virt/kvm: Use vma_lookup() instead of find_vma_intersection()
  vfio: Use vma_lookup() instead of find_vma_intersection()
  net/ipv5/tcp: Use vma_lookup() in tcp_zerocopy_receive()
  drm/amdgpu: Use vma_lookup() in amdgpu_ttm_tt_get_user_pages()
  media: videobuf2: Use vma_lookup() in get_vaddr_frames()
  misc/sgi-gru/grufault: Use vma_lookup() in gru_find_vma()
  kernel/events/uprobes: Use vma_lookup() in find_active_uprobe()
  lib/test_hmm: Use vma_lookup() in dmirror_migrate()
  mm/ksm: Use vma_lookup() in find_mergeable_vma()
  mm/migrate: Use vma_lookup() in do_pages_stat_array()
  mm/mremap: Use vma_lookup() in vma_to_resize()
  mm/memory.c: Use vma_lookup() in __access_remote_vm()
  mm/mempolicy: Use vma_lookup() in __access_remote_vm()

 arch/arc/kernel/troubleshoot.c                 |  8 ++++----
 arch/arm64/kvm/mmu.c                           |  2 +-
 arch/m68k/kernel/sys_m68k.c                    |  4 ++--
 arch/mips/kernel/traps.c                       |  4 +---
 arch/powerpc/kvm/book3s_hv.c                   |  4 ++--
 arch/powerpc/kvm/book3s_hv_uvmem.c             |  2 +-
 arch/x86/kernel/cpu/sgx/encl.h                 |  4 ++--
 drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c        |  4 ++--
 .../gpu/drm/i915/gem/selftests/i915_gem_mman.c |  2 +-
 drivers/media/common/videobuf2/frame_vector.c  |  2 +-
 drivers/misc/sgi-gru/grufault.c                |  4 ++--
 drivers/vfio/vfio_iommu_type1.c                |  2 +-
 include/linux/mm.h                             | 18 ++++++++++++++++++
 kernel/events/uprobes.c                        |  4 ++--
 lib/test_hmm.c                                 |  5 ++---
 mm/ksm.c                                       |  6 ++----
 mm/memory.c                                    |  4 ++--
 mm/mempolicy.c                                 |  2 +-
 mm/migrate.c                                   |  4 ++--
 mm/mremap.c                                    |  4 ++--
 net/ipv4/tcp.c                                 |  4 ++--
 virt/kvm/kvm_main.c                            |  2 +-
 22 files changed, 54 insertions(+), 41 deletions(-)

-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:56 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:56 +0000
Subject: [PATCH 16/22] kernel/events/uprobes: Use vma_lookup() in
 find_active_uprobe()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-17-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 kernel/events/uprobes.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index 6addc9780319..907d4ee00cb2 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -2046,8 +2046,8 @@ static struct uprobe *find_active_uprobe(unsigned long bp_vaddr, int *is_swbp)
 	struct vm_area_struct *vma;
 
 	mmap_read_lock(mm);
-	vma = find_vma(mm, bp_vaddr);
-	if (vma && vma->vm_start <= bp_vaddr) {
+	vma = vma_lookup(mm, bp_vaddr);
+	if (vma) {
 		if (valid_vma(vma, false)) {
 			struct inode *inode = file_inode(vma->vm_file);
 			loff_t offset = vaddr_to_offset(vma, bp_vaddr);
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:53 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:53 +0000
Subject: [PATCH 11/22] vfio: Use vma_lookup() instead of
 find_vma_intersection()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-12-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 drivers/vfio/vfio_iommu_type1.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
index a0747c35a778..fb695bf0b1c4 100644
--- a/drivers/vfio/vfio_iommu_type1.c
+++ b/drivers/vfio/vfio_iommu_type1.c
@@ -567,7 +567,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr,
 	vaddr = untagged_addr(vaddr);
 
 retry:
-	vma = find_vma_intersection(mm, vaddr, vaddr + 1);
+	vma = vma_lookup(mm, vaddr);
 
 	if (vma && vma->vm_flags & VM_PFNMAP) {
 		ret = follow_fault_pfn(vma, mm, vaddr, pfn, prot & IOMMU_WRITE);
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:57 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:57 +0000
Subject: [PATCH 17/22] lib/test_hmm: Use vma_lookup() in dmirror_migrate()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-18-Liam.Howlett@Oracle.com>

Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
will return NULL if the address is not within any VMA, the start address
no longer needs to be validated.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 lib/test_hmm.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/lib/test_hmm.c b/lib/test_hmm.c
index 80a78877bd93..15f2e2db77bc 100644
--- a/lib/test_hmm.c
+++ b/lib/test_hmm.c
@@ -686,9 +686,8 @@ static int dmirror_migrate(struct dmirror *dmirror,
 
 	mmap_read_lock(mm);
 	for (addr = start; addr < end; addr = next) {
-		vma = find_vma(mm, addr);
-		if (!vma || addr < vma->vm_start ||
-		    !(vma->vm_flags & VM_READ)) {
+		vma = vma_lookup(mm, addr);
+		if (!vma || !(vma->vm_flags & VM_READ)) {
 			ret = -EINVAL;
 			goto out;
 		}
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 17:58:50 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 16:58:50 +0000
Subject: [PATCH 05/22] arch/powerpc/kvm/book3s_hv_uvmem: Use vma_lookup()
 instead of find_vma_intersection()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210510165839.2692974-6-Liam.Howlett@Oracle.com>

vma_lookup() finds the vma of a specific address with a cleaner
interface and is more readable.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/powerpc/kvm/book3s_hv_uvmem.c b/arch/powerpc/kvm/book3s_hv_uvmem.c
index 84e5a2dc8be5..34720b79588f 100644
--- a/arch/powerpc/kvm/book3s_hv_uvmem.c
+++ b/arch/powerpc/kvm/book3s_hv_uvmem.c
@@ -614,7 +614,7 @@ void kvmppc_uvmem_drop_pages(const struct kvm_memory_slot *slot,
 
 		/* Fetch the VMA if addr is not in the latest fetched one */
 		if (!vma || addr >= vma->vm_end) {
-			vma = find_vma_intersection(kvm->mm, addr, addr+1);
+			vma = vma_lookup(kvm->mm, addr);
 			if (!vma) {
 				pr_err("Can't find VMA for gfn:0x%lx\n", gfn);
 				break;
-- 
2.30.2


From liam.howlett at oracle.com  Mon May 10 20:50:22 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 19:50:22 +0000
Subject: [PATCH] mm/mmap: Introduce unlock_range() for code cleanup
Message-ID: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>

Both __do_munmap() and exit_mmap() unlock a range of VMAs using almost
identical code blocks.  Replace both blocks by a static inline function.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
---
 mm/mmap.c | 38 +++++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 19 deletions(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 81f5595a8490..ea556fc795d2 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -2801,6 +2801,21 @@ int split_vma(struct mm_struct *mm, struct vm_area_struct *vma,
 	return __split_vma(mm, vma, addr, new_below);
 }
 
+static inline void unlock_range(struct vm_area_struct *start, unsigned long limit)
+{
+	struct mm_struct *mm = start->vm_mm;
+	struct vm_area_struct *tmp = start;
+
+	while (tmp && tmp->vm_start < limit) {
+		if (tmp->vm_flags & VM_LOCKED) {
+			mm->locked_vm -= vma_pages(tmp);
+			munlock_vma_pages_all(tmp);
+		}
+
+		tmp = tmp->vm_next;
+	}
+}
+
 /* Munmap is split into 2 main parts -- this part which finds
  * what needs doing, and the areas themselves, which do the
  * work.  This now handles partial unmappings.
@@ -2889,17 +2904,8 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
 	/*
 	 * unlock any mlock()ed ranges before detaching vmas
 	 */
-	if (mm->locked_vm) {
-		struct vm_area_struct *tmp = vma;
-		while (tmp && tmp->vm_start < end) {
-			if (tmp->vm_flags & VM_LOCKED) {
-				mm->locked_vm -= vma_pages(tmp);
-				munlock_vma_pages_all(tmp);
-			}
-
-			tmp = tmp->vm_next;
-		}
-	}
+	if (mm->locked_vm)
+		unlock_range(vma, end);
 
 	/* Detach vmas from rbtree */
 	if (!detach_vmas_to_be_unmapped(mm, vma, prev, end))
@@ -3184,14 +3190,8 @@ void exit_mmap(struct mm_struct *mm)
 		mmap_write_unlock(mm);
 	}
 
-	if (mm->locked_vm) {
-		vma = mm->mmap;
-		while (vma) {
-			if (vma->vm_flags & VM_LOCKED)
-				munlock_vma_pages_all(vma);
-			vma = vma->vm_next;
-		}
-	}
+	if (mm->locked_vm)
+		unlock_range(mm->mmap, ULONG_MAX);
 
 	arch_exit_mmap(mm);
 
-- 
2.30.2


From willy at infradead.org  Mon May 10 20:56:54 2021
From: willy at infradead.org (Matthew Wilcox)
Date: Mon, 10 May 2021 20:56:54 +0100
Subject: [PATCH] mm/mmap: Introduce unlock_range() for code cleanup
In-Reply-To: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>
References: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>
Message-ID: <YJmQBp9r5UgvQiWr@casper.infradead.org>

On Mon, May 10, 2021 at 07:50:22PM +0000, Liam Howlett wrote:
> Both __do_munmap() and exit_mmap() unlock a range of VMAs using almost
> identical code blocks.  Replace both blocks by a static inline function.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>

Reviewed-by: Matthew Wilcox (Oracle) <willy at infradead.org>

> +static inline void unlock_range(struct vm_area_struct *start, unsigned long limit)

Seems like an unnecessary >80 column line ...

static inline
void unlock_range(struct vm_area_struct *start, unsigned long limit)



From liam.howlett at oracle.com  Mon May 10 22:01:16 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 10 May 2021 21:01:16 +0000
Subject: [PATCH] mm/mmap: Introduce unlock_range() for code cleanup
In-Reply-To: <YJmQBp9r5UgvQiWr@casper.infradead.org>
References: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>
 <YJmQBp9r5UgvQiWr@casper.infradead.org>
Message-ID: <20210510210057.mk7hgtdrxa5c5wt6@revolver>

* Matthew Wilcox <willy at infradead.org> [210510 15:57]:
> On Mon, May 10, 2021 at 07:50:22PM +0000, Liam Howlett wrote:
> > Both __do_munmap() and exit_mmap() unlock a range of VMAs using almost
> > identical code blocks.  Replace both blocks by a static inline function.
> > 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> 
> Reviewed-by: Matthew Wilcox (Oracle) <willy at infradead.org>
> 
> > +static inline void unlock_range(struct vm_area_struct *start, unsigned long limit)
> 
> Seems like an unnecessary >80 column line ...
> 
> static inline
> void unlock_range(struct vm_area_struct *start, unsigned long limit)
> 

Sorry about that, checkpatch also did not see this.  I will send a v2.

From geert at linux-m68k.org  Tue May 11 08:15:24 2021
From: geert at linux-m68k.org (Geert Uytterhoeven)
Date: Tue, 11 May 2021 09:15:24 +0200
Subject: [PATCH 08/22] arch/m68k/kernel/sys_m68k: Use vma_lookup() in
 sys_cacheflush()
In-Reply-To: <20210510165839.2692974-9-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-9-Liam.Howlett@Oracle.com>
Message-ID: <CAMuHMdViObRA1H-yggx5M-dn2j+7rrMR1Z_SFBYJE2cHmX2=zQ@mail.gmail.com>

On Mon, May 10, 2021 at 7:04 PM Liam Howlett <liam.howlett at oracle.com> wrote:
> Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
> will return NULL if the address is not within any VMA, the start address
> no longer needs to be validated.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>

Acked-by: Geert Uytterhoeven <geert at linux-m68k.org>

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds


From dave at stgolabs.net  Tue May 11 22:11:42 2021
From: dave at stgolabs.net (Davidlohr Bueso)
Date: Tue, 11 May 2021 14:11:42 -0700
Subject: [PATCH] mm/mmap: Introduce unlock_range() for code cleanup
In-Reply-To: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>
References: <20210510195020.2778501-1-Liam.Howlett@Oracle.com>
Message-ID: <20210511211142.hgti5kplyew4yrzl@offworld>

On Mon, 10 May 2021, Liam Howlett wrote:

>Both __do_munmap() and exit_mmap() unlock a range of VMAs using almost
>identical code blocks.  Replace both blocks by a static inline function.
>
>Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>

Reviewed-by: Davidlohr Bueso <dbueso at suse.de>


From david at redhat.com  Fri May 14 10:57:25 2021
From: david at redhat.com (David Hildenbrand)
Date: Fri, 14 May 2021 11:57:25 +0200
Subject: [PATCH 01/22] mm: Add vma_lookup()
In-Reply-To: <20210510165839.2692974-2-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-2-Liam.Howlett@Oracle.com>
Message-ID: <99e04b50-7210-8a2f-4eda-08747b8570fb@redhat.com>

On 10.05.21 18:58, Liam Howlett wrote:
> he third use of find_vma() is by developers who do not know that the
> function starts searching at the provided address upwards for the next
> vma.  This results in a bug that is often overlooked for a long time.
> 
> Adding the new vma_lookup() function will allow for cleaner code by

Sounds helpful to me

Acked-by: David Hildenbrand <david at redhat.com>

-- 
Thanks,

David / dhildenb



From peterz at infradead.org  Fri May 14 11:40:45 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 12:40:45 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
Message-ID: <YJ5TrReBMXwCqJFE@hirez.programming.kicks-ass.net>

On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:

> +/* Flags:
> + * MAPLE_ALLOC_RANGE	Use allocation ranges (tracks gaps) in this tree
> + * MAPLE_USE_RCU	Operate in read/copy/update mode for multi-readers.
> + * MAPLE_HEIGHT_OFFSET	The position of the tree height in the flags
> + * MAPLE_HEIGHT_MASK	The mask for the maple tree height value.
> + */

Regular comment style would be:

/*
 * Flags:
 ...
 */

> +#define MAPLE_ALLOC_RANGE	1	// Bit 0
> +#define MAPLE_USE_RCU		2	// Bit 1
> +#define	MAPLE_HEIGHT_OFFSET	2	// Bit 2
> +#define	MAPLE_HEIGHT_MASK	60	// Bits 2-5

Can we pretty please keep masks in hex, like:

#define MAPLE_HEIGHT_MASK	0x3c

Because then it's instantly obvious is it bits 2-5.



From peterz at infradead.org  Fri May 14 13:17:49 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 14:17:49 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
Message-ID: <YJ5qbS2KJcVp+ocY@hirez.programming.kicks-ass.net>

On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> +static void mas_set_height(struct ma_state *mas)
> +{
> +	unsigned int new_flags = mas->tree->ma_flags;
> +
> +	new_flags &= ~MAPLE_HEIGHT_MASK;
> +	new_flags |= mas->depth << MAPLE_HEIGHT_OFFSET;
> +	mas->tree->ma_flags = new_flags;
> +}

This goes sideways when mas->depth is too large; I didn't quickly see an
assertion / range checks for mas->depth anywhere.


From willy at infradead.org  Fri May 14 13:43:02 2021
From: willy at infradead.org (Matthew Wilcox)
Date: Fri, 14 May 2021 13:43:02 +0100
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ5qbS2KJcVp+ocY@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ5qbS2KJcVp+ocY@hirez.programming.kicks-ass.net>
Message-ID: <YJ5wVncTFHHmY762@casper.infradead.org>

On Fri, May 14, 2021 at 02:17:49PM +0200, Peter Zijlstra wrote:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +static void mas_set_height(struct ma_state *mas)
> > +{
> > +	unsigned int new_flags = mas->tree->ma_flags;
> > +
> > +	new_flags &= ~MAPLE_HEIGHT_MASK;
> > +	new_flags |= mas->depth << MAPLE_HEIGHT_OFFSET;
> > +	mas->tree->ma_flags = new_flags;
> > +}
> 
> This goes sideways when mas->depth is too large; I didn't quickly see an
> assertion / range checks for mas->depth anywhere.

Not arguing that we shouldn't have such an assertion (to catch bugs),
but with 4 bits, that's a height 15 tree.  At a minimum fanout of 5.5
per level (with a minimum of 8.5 in the leaf), that's 197 x 10^9 entries.
At a more typical fanout of 8 per level and 12 entries in each leaf, it's
52.8 x 10^12 entries in the tree (about 2^46).  You'll run out of memory
and address space first.


From peterz at infradead.org  Fri May 14 14:42:04 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 15:42:04 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
Message-ID: <YJ5+LN7FlNjAXQi0@hirez.programming.kicks-ass.net>

On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> +/*
> + * mte_set_parent() - Set the parent node and encode the slot.
> + * @enode: The encoded maple node.
> + * @parent: The encoded maple node that is the parent of @enode.
> + * @slot: The slot that @enode resides in @parent.
> + *
> + * Type is encoded in the enode->parent
> + * bit 0: 1 = root, 0 otherwise
> + * bit 1: Reserved.
> + * bit 2: 0 = range 32, 1 = [a]range 64
> + *
> + * Slot number is encoded in the enode->parent
> + * range_32, slot number is encoded in bits 3-6
> + * [a]range_64, slot number is encoded in bits 3-6
> + */
> +static inline void mte_set_parent(struct maple_enode *enode,
> +				 const struct maple_enode *parent,
> +				 unsigned char slot)
> +{
> +	unsigned long bitmask = 0x78;
> +	unsigned long val = (unsigned long) parent;
> +	unsigned long type = 0;
> +
> +	switch (mte_node_type(parent)) {
> +	case maple_range_64:
> +	case maple_arange_64:
> +		type = 6;

6 = 4 + 2, which has bit1 set, but the above sayeth bit1 is reserved.

It is also mighty confusing to have two different type fields, is there
no way we can merge the types into a single (shared) space?

> +		break;
> +	default:
> +		break;
> +	}
> +
> +	val &= ~bitmask; // Remove any old slot number.
> +	val |= (slot << MAPLE_PARENT_SHIFT); // Set the slot.
> +	val |= type;
> +	mte_to_node(enode)->parent = ma_parent_ptr(val);
> +}


From peterz at infradead.org  Fri May 14 16:32:15 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 17:32:15 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
Message-ID: <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>

On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> +enum maple_type {
> +	maple_dense,
> +	maple_leaf_64,
> +	maple_range_64,
> +	maple_arange_64,
> +};

> +static inline unsigned long *ma_pivots(struct maple_node *node,
> +					   enum maple_type type)
> +{
> +	switch (type) {
> +	case maple_arange_64:
> +		return node->ma64.pivot;
> +	case maple_range_64:
> +	case maple_leaf_64:
> +		return node->mr64.pivot;
> +	case maple_dense:
> +	default:
> +		return NULL;
> +	}
> +}

> +static inline unsigned long *ma_gaps(struct maple_node *node,
> +				     enum maple_type type)
> +{
> +	switch (type) {
> +	case maple_arange_64:
> +		return node->ma64.gap;
> +	case maple_range_64:
> +	case maple_leaf_64:
> +	case maple_dense:
> +	default:
> +		return NULL;
> +	}
> +}

> +static inline unsigned long mte_pivot(const struct maple_enode *mn,
> +				 unsigned char piv)
> +{
> +	struct maple_node *node = mte_to_node(mn);
> +
> +	switch (mte_node_type(mn)) {
> +	case maple_arange_64:
> +		return node->ma64.pivot[piv];
> +	case maple_range_64:
> +	case maple_leaf_64:
> +		return node->mr64.pivot[piv];
> +	case maple_dense:
> +	default:
> +		return 0;
> +	}
> +}

I would suggest removing the default: case. Without it the cases are
complete and the compiler should not complain. Then if you extend the
enum and forget to add a case, the switch is no longer complete and will
trigger a warning, forcing you to update it (hopefully correct).

If you have the default, you'll not get a warning and it'll do whatever.


From peterz at infradead.org  Fri May 14 16:34:52 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 17:34:52 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>
Message-ID: <YJ6YnJv99AEdu0it@hirez.programming.kicks-ass.net>

On Fri, May 14, 2021 at 05:32:15PM +0200, Peter Zijlstra wrote:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +static inline unsigned long mte_pivot(const struct maple_enode *mn,
> > +				 unsigned char piv)
> > +{
> > +	struct maple_node *node = mte_to_node(mn);
> > +
> > +	switch (mte_node_type(mn)) {
> > +	case maple_arange_64:
> > +		return node->ma64.pivot[piv];
> > +	case maple_range_64:
> > +	case maple_leaf_64:
> > +		return node->mr64.pivot[piv];
> > +	case maple_dense:
> > +	default:
> > +		return 0;
> > +	}
> > +}

Why is @piv 'unsigned char' ? Does that actually generate better code on
any architecture?

At the same time; there's no bounds check.


From peterz at infradead.org  Fri May 14 16:40:01 2021
From: peterz at infradead.org (Peter Zijlstra)
Date: Fri, 14 May 2021 17:40:01 +0200
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
Message-ID: <YJ6Z0X91NpuPTEre@hirez.programming.kicks-ass.net>

On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> +/*
> + * mas_set_alloc_req() - Set the requested number of allocations.
> + * @mas: the maple state
> + * @count: the number of allocations.
> + *
> + * If @mas->alloc has bit 1 set (0x1) or @mas->alloc is %NULL, then there are no
> + * nodes allocated and @mas->alloc should be set to count << 1 | 1.  If there is
> + * already nodes allocated, then @mas->alloc->request_count stores the request.
> + */
> +static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
> +{
> +	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
> +		if (!count)
> +			mas->alloc = NULL;
> +		else
> +			mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
> +		return;
> +	}
> +
> +	mas->alloc->request_count = count;
> +}
> +
> +/*
> + * mas_alloc_req() - get the requested number of allocations.
> + * @mas: The maple state
> + *
> + * The alloc count is either stored directly in @mas, or in
> + * @mas->alloc->request_count if there is at least one node allocated.
> + *
> + * Return: The allocation request count.
> + */
> +static inline unsigned int mas_alloc_req(const struct ma_state *mas)
> +{
> +	if ((unsigned long)mas->alloc & 0x1)
> +		return (unsigned long)(mas->alloc) >> 1;
> +	else if (mas->alloc)
> +		return mas->alloc->request_count;
> +	return 0;
> +}


I'm confuse.. and the comments fail to eludicate *why* the code is the
way it is, they simply explain exactly what the code does, which I can
already tell from reading the code.

Why can't we have ->request_count unconditionally be the requested
count, and have ->alloc be NULL or not. Why do we play games with low
pointer bits here? AFAICT there's no actual benefit to doing so.



From liam.howlett at oracle.com  Fri May 14 21:36:37 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 14 May 2021 20:36:37 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ5TrReBMXwCqJFE@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ5TrReBMXwCqJFE@hirez.programming.kicks-ass.net>
Message-ID: <20210514203620.ktklltqpco7ul5vi@revolver>


Hello Peter,

Thank you for looking at this patch set!

* Peter Zijlstra <peterz at infradead.org> [210514 06:41]:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> 
> > +/* Flags:
> > + * MAPLE_ALLOC_RANGE	Use allocation ranges (tracks gaps) in this tree
> > + * MAPLE_USE_RCU	Operate in read/copy/update mode for multi-readers.
> > + * MAPLE_HEIGHT_OFFSET	The position of the tree height in the flags
> > + * MAPLE_HEIGHT_MASK	The mask for the maple tree height value.
> > + */
> 
> Regular comment style would be:
> 
> /*
>  * Flags:
>  ...
>  */

I will fix this.

> 
> > +#define MAPLE_ALLOC_RANGE	1	// Bit 0
> > +#define MAPLE_USE_RCU		2	// Bit 1
> > +#define	MAPLE_HEIGHT_OFFSET	2	// Bit 2
> > +#define	MAPLE_HEIGHT_MASK	60	// Bits 2-5
> 
> Can we pretty please keep masks in hex, like:
> 
> #define MAPLE_HEIGHT_MASK	0x3c
> 
> Because then it's instantly obvious is it bits 2-5.
> 

Sure, I will change them all to two digit hex so they align.  I should
probably check that whitespace too.

From liam.howlett at oracle.com  Fri May 14 22:29:40 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 14 May 2021 21:29:40 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ5qbS2KJcVp+ocY@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ5qbS2KJcVp+ocY@hirez.programming.kicks-ass.net>
Message-ID: <20210514212930.3vprzwvuyhh4sdml@revolver>

* Peter Zijlstra <peterz at infradead.org> [210514 08:18]:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +static void mas_set_height(struct ma_state *mas)
> > +{
> > +	unsigned int new_flags = mas->tree->ma_flags;
> > +
> > +	new_flags &= ~MAPLE_HEIGHT_MASK;
> > +	new_flags |= mas->depth << MAPLE_HEIGHT_OFFSET;
> > +	mas->tree->ma_flags = new_flags;
> > +}
> 
> This goes sideways when mas->depth is too large; I didn't quickly see an
> assertion / range checks for mas->depth anywhere.
> 

I will add an assert here.  Thank you.

From liam.howlett at oracle.com  Fri May 14 22:42:32 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 14 May 2021 21:42:32 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ5+LN7FlNjAXQi0@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ5+LN7FlNjAXQi0@hirez.programming.kicks-ass.net>
Message-ID: <20210514214230.wpsim46hpnyyzoe7@revolver>

* Peter Zijlstra <peterz at infradead.org> [210514 09:42]:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +/*
> > + * mte_set_parent() - Set the parent node and encode the slot.
> > + * @enode: The encoded maple node.
> > + * @parent: The encoded maple node that is the parent of @enode.
> > + * @slot: The slot that @enode resides in @parent.
> > + *
> > + * Type is encoded in the enode->parent
> > + * bit 0: 1 = root, 0 otherwise
> > + * bit 1: Reserved.
> > + * bit 2: 0 = range 32, 1 = [a]range 64
> > + *
> > + * Slot number is encoded in the enode->parent
> > + * range_32, slot number is encoded in bits 3-6
> > + * [a]range_64, slot number is encoded in bits 3-6
> > + */
> > +static inline void mte_set_parent(struct maple_enode *enode,
> > +				 const struct maple_enode *parent,
> > +				 unsigned char slot)
> > +{
> > +	unsigned long bitmask = 0x78;
> > +	unsigned long val = (unsigned long) parent;
> > +	unsigned long type = 0;
> > +
> > +	switch (mte_node_type(parent)) {
> > +	case maple_range_64:
> > +	case maple_arange_64:
> > +		type = 6;
> 
> 6 = 4 + 2, which has bit1 set, but the above sayeth bit1 is reserved.

Yeah, that's not good.

> 
> It is also mighty confusing to have two different type fields, is there
> no way we can merge the types into a single (shared) space?
> 
> > +		break;
> > +	default:
> > +		break;
> > +	}
> > +
> > +	val &= ~bitmask; // Remove any old slot number.
> > +	val |= (slot << MAPLE_PARENT_SHIFT); // Set the slot.
> > +	val |= type;
> > +	mte_to_node(enode)->parent = ma_parent_ptr(val);
> > +}
> 


Sorry, this is more fallout from removing types for the initial merge..
I will re-examine the mte_set_paret/mte_parent_slot stuff and fix the
comments (note range 32 above is not even in this code base either).

The type space maple_range_64/maple_arange_64 cannot be merged into a
single type as they have different slot and pivot counts.  The type of
node it is depends on the immutable tree flag for alloc on creation
time.  Since both node types cannot exist in the same type of tree,
reusing the same value here allows for more node types to exist..
Merging them would cause multiple checks of the tree flag during a walk
and function pointers are not as fast as they used to be.

I can just set type to range/arange right now as it's the only option.

Thanks for highlighting this as confusing.  I'll rework this to be
easier to understand.

From liam.howlett at oracle.com  Fri May 14 22:43:44 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 14 May 2021 21:43:44 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>
Message-ID: <20210514214341.53bwviwoze2rzowj@revolver>

* Peter Zijlstra <peterz at infradead.org> [210514 11:32]:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +enum maple_type {
> > +	maple_dense,
> > +	maple_leaf_64,
> > +	maple_range_64,
> > +	maple_arange_64,
> > +};
> 
> > +static inline unsigned long *ma_pivots(struct maple_node *node,
> > +					   enum maple_type type)
> > +{
> > +	switch (type) {
> > +	case maple_arange_64:
> > +		return node->ma64.pivot;
> > +	case maple_range_64:
> > +	case maple_leaf_64:
> > +		return node->mr64.pivot;
> > +	case maple_dense:
> > +	default:
> > +		return NULL;
> > +	}
> > +}
> 
> > +static inline unsigned long *ma_gaps(struct maple_node *node,
> > +				     enum maple_type type)
> > +{
> > +	switch (type) {
> > +	case maple_arange_64:
> > +		return node->ma64.gap;
> > +	case maple_range_64:
> > +	case maple_leaf_64:
> > +	case maple_dense:
> > +	default:
> > +		return NULL;
> > +	}
> > +}
> 
> > +static inline unsigned long mte_pivot(const struct maple_enode *mn,
> > +				 unsigned char piv)
> > +{
> > +	struct maple_node *node = mte_to_node(mn);
> > +
> > +	switch (mte_node_type(mn)) {
> > +	case maple_arange_64:
> > +		return node->ma64.pivot[piv];
> > +	case maple_range_64:
> > +	case maple_leaf_64:
> > +		return node->mr64.pivot[piv];
> > +	case maple_dense:
> > +	default:
> > +		return 0;
> > +	}
> > +}
> 
> I would suggest removing the default: case. Without it the cases are
> complete and the compiler should not complain. Then if you extend the
> enum and forget to add a case, the switch is no longer complete and will
> trigger a warning, forcing you to update it (hopefully correct).
> 
> If you have the default, you'll not get a warning and it'll do whatever.


Thanks, I will do this.

From liam.howlett at oracle.com  Fri May 14 22:57:33 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 14 May 2021 21:57:33 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ6YnJv99AEdu0it@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ6X/wrP/El8KLdh@hirez.programming.kicks-ass.net>
 <YJ6YnJv99AEdu0it@hirez.programming.kicks-ass.net>
Message-ID: <20210514215716.k7lxgcnwuyk2isn4@revolver>

* Peter Zijlstra <peterz at infradead.org> [210514 11:37]:
> On Fri, May 14, 2021 at 05:32:15PM +0200, Peter Zijlstra wrote:
> > On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > > +static inline unsigned long mte_pivot(const struct maple_enode *mn,
> > > +				 unsigned char piv)
> > > +{
> > > +	struct maple_node *node = mte_to_node(mn);
> > > +
> > > +	switch (mte_node_type(mn)) {
> > > +	case maple_arange_64:
> > > +		return node->ma64.pivot[piv];
> > > +	case maple_range_64:
> > > +	case maple_leaf_64:
> > > +		return node->mr64.pivot[piv];
> > > +	case maple_dense:
> > > +	default:
> > > +		return 0;
> > > +	}
> > > +}
> 
> Why is @piv 'unsigned char' ? Does that actually generate better code on
> any architecture?

unsigned char is to match ma_state struct offset which is kept small
for cache efficiency.

> 
> At the same time; there's no bounds check.
> 

I will add an assert.  Thanks.


From liam.howlett at oracle.com  Sat May 15 01:02:12 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Sat, 15 May 2021 00:02:12 +0000
Subject: [PATCH 26/94] Maple Tree: Add new data structure
In-Reply-To: <YJ6Z0X91NpuPTEre@hirez.programming.kicks-ass.net>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-27-Liam.Howlett@Oracle.com>
 <YJ6Z0X91NpuPTEre@hirez.programming.kicks-ass.net>
Message-ID: <20210515000206.u5l44sgr6xzmpe5d@revolver>

* Peter Zijlstra <peterz at infradead.org> [210514 11:40]:
> On Wed, Apr 28, 2021 at 03:36:02PM +0000, Liam Howlett wrote:
> > +/*
> > + * mas_set_alloc_req() - Set the requested number of allocations.
> > + * @mas: the maple state
> > + * @count: the number of allocations.
> > + *
> > + * If @mas->alloc has bit 1 set (0x1) or @mas->alloc is %NULL, then there are no
> > + * nodes allocated and @mas->alloc should be set to count << 1 | 1.  If there is
> > + * already nodes allocated, then @mas->alloc->request_count stores the request.
> > + */
> > +static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
> > +{
> > +	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
> > +		if (!count)
> > +			mas->alloc = NULL;
> > +		else
> > +			mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
> > +		return;
> > +	}
> > +
> > +	mas->alloc->request_count = count;
> > +}
> > +
> > +/*
> > + * mas_alloc_req() - get the requested number of allocations.
> > + * @mas: The maple state
> > + *
> > + * The alloc count is either stored directly in @mas, or in
> > + * @mas->alloc->request_count if there is at least one node allocated.
> > + *
> > + * Return: The allocation request count.
> > + */
> > +static inline unsigned int mas_alloc_req(const struct ma_state *mas)
> > +{
> > +	if ((unsigned long)mas->alloc & 0x1)
> > +		return (unsigned long)(mas->alloc) >> 1;
> > +	else if (mas->alloc)
> > +		return mas->alloc->request_count;
> > +	return 0;
> > +}
> 
> 
> I'm confuse.. and the comments fail to eludicate *why* the code is the
> way it is, they simply explain exactly what the code does, which I can
> already tell from reading the code.
> 
> Why can't we have ->request_count unconditionally be the requested
> count, and have ->alloc be NULL or not. Why do we play games with low
> pointer bits here? AFAICT there's no actual benefit to doing so.
> 
> 


That comment was written by me before I coded the function.  I meant to
replace it.

The maple state stores the first node into alloc.  The alloc stores the
request_count if more nodes are needed than already allocated.  If there
is no nodes allocated, then the request count is stored directly into
alloc.  We do the low pointer bit game to reduce the maple state size.

In other words, the mas->alloc is overloaded to either hold an
allocation or the number of requested allocations.  Bit 1 indicates that
mas->alloc holds a request to alloc (mas->alloc >> 1) nodes.


From ldufour at linux.ibm.com  Mon May 17 18:28:37 2021
From: ldufour at linux.ibm.com (Laurent Dufour)
Date: Mon, 17 May 2021 19:28:37 +0200
Subject: [PATCH 00/21] mm: Add vma_lookup()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <1edcdd6f-af3b-560f-d10f-cbae5e911ea1@linux.ibm.com>

Le 10/05/2021 ? 18:58, Liam Howlett a ?crit?:
> Many places in the kernel use find_vma() to get a vma and then check the
> start address of the vma to ensure the next vma was not returned.
> 
> Other places use the find_vma_intersection() call with add, addr + 1 as
> the range; looking for just the vma at a specific address.
> 
> The third use of find_vma() is by developers who do not know that the
> function starts searching at the provided address upwards for the next
> vma.  This results in a bug that is often overlooked for a long time.
> 
> Adding the new vma_lookup() function will allow for cleaner code by
> removing the find_vma() calls which check limits, making
> find_vma_intersection() calls of a single address to be shorter, and
> potentially reduce the incorrect uses of find_vma().
> 
> This patch set was originally part of the maple tree patch set.

FWIW, for the whole series:

Reviewed-by: Laurent Dufour <ldufour at linux.ibm.com>


> Changes since being broken out on its own:
> - Changed initial implementation of vma_lookup() to use find_vma() as requested
>    by Michel Lespinasse.
> - Updated commit comments to be more descriptive
> 
> These patches are based on next-20210506
> 
> Liam R. Howlett (22):
>    mm: Add vma_lookup()
>    drm/i915/selftests: Use vma_lookup() in __igt_mmap()
>    arch/arc/kernel/troubleshoot: use vma_lookup() instead of find_vma()
>    arch/arm64/kvm: Use vma_lookup() instead of find_vma_intersection()
>    arch/powerpc/kvm/book3s_hv_uvmem: Use vma_lookup() instead of
>      find_vma_intersection()
>    arch/powerpc/kvm/book3s: Use vma_lookup() in
>      kvmppc_hv_setup_htab_rma()
>    arch/mips/kernel/traps: Use vma_lookup() instead of find_vma()
>    arch/m68k/kernel/sys_m68k: Use vma_lookup() in sys_cacheflush()
>    x86/sgx: Use vma_lookup() in sgx_encl_find()
>    virt/kvm: Use vma_lookup() instead of find_vma_intersection()
>    vfio: Use vma_lookup() instead of find_vma_intersection()
>    net/ipv5/tcp: Use vma_lookup() in tcp_zerocopy_receive()
>    drm/amdgpu: Use vma_lookup() in amdgpu_ttm_tt_get_user_pages()
>    media: videobuf2: Use vma_lookup() in get_vaddr_frames()
>    misc/sgi-gru/grufault: Use vma_lookup() in gru_find_vma()
>    kernel/events/uprobes: Use vma_lookup() in find_active_uprobe()
>    lib/test_hmm: Use vma_lookup() in dmirror_migrate()
>    mm/ksm: Use vma_lookup() in find_mergeable_vma()
>    mm/migrate: Use vma_lookup() in do_pages_stat_array()
>    mm/mremap: Use vma_lookup() in vma_to_resize()
>    mm/memory.c: Use vma_lookup() in __access_remote_vm()
>    mm/mempolicy: Use vma_lookup() in __access_remote_vm()
> 
>   arch/arc/kernel/troubleshoot.c                 |  8 ++++----
>   arch/arm64/kvm/mmu.c                           |  2 +-
>   arch/m68k/kernel/sys_m68k.c                    |  4 ++--
>   arch/mips/kernel/traps.c                       |  4 +---
>   arch/powerpc/kvm/book3s_hv.c                   |  4 ++--
>   arch/powerpc/kvm/book3s_hv_uvmem.c             |  2 +-
>   arch/x86/kernel/cpu/sgx/encl.h                 |  4 ++--
>   drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c        |  4 ++--
>   .../gpu/drm/i915/gem/selftests/i915_gem_mman.c |  2 +-
>   drivers/media/common/videobuf2/frame_vector.c  |  2 +-
>   drivers/misc/sgi-gru/grufault.c                |  4 ++--
>   drivers/vfio/vfio_iommu_type1.c                |  2 +-
>   include/linux/mm.h                             | 18 ++++++++++++++++++
>   kernel/events/uprobes.c                        |  4 ++--
>   lib/test_hmm.c                                 |  5 ++---
>   mm/ksm.c                                       |  6 ++----
>   mm/memory.c                                    |  4 ++--
>   mm/mempolicy.c                                 |  2 +-
>   mm/migrate.c                                   |  4 ++--
>   mm/mremap.c                                    |  4 ++--
>   net/ipv4/tcp.c                                 |  4 ++--
>   virt/kvm/kvm_main.c                            |  2 +-
>   22 files changed, 54 insertions(+), 41 deletions(-)
> 



From dave at stgolabs.net  Thu May 20 20:32:55 2021
From: dave at stgolabs.net (Davidlohr Bueso)
Date: Fri, 21 May 2021 03:32:55 -0000
Subject: [PATCH 01/22] mm: Add vma_lookup()
In-Reply-To: <20210510165839.2692974-2-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-2-Liam.Howlett@Oracle.com>
Message-ID: <20210521033232.dyams2dziycp4hv7@offworld>

On Mon, 10 May 2021, Liam Howlett wrote:

>Many places in the kernel use find_vma() to get a vma and then check the
>start address of the vma to ensure the next vma was not returned.
>
>Other places use the find_vma_intersection() call with add, addr + 1 as
>the range; looking for just the vma at a specific address.
>
>The third use of find_vma() is by developers who do not know that the
>function starts searching at the provided address upwards for the next
>vma.  This results in a bug that is often overlooked for a long time.
>
>Adding the new vma_lookup() function will allow for cleaner code by
>removing the find_vma() calls which check limits, making
>find_vma_intersection() calls of a single address to be shorter, and
>potentially reduce the incorrect uses of find_vma().
>
>Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
>---
> include/linux/mm.h | 18 ++++++++++++++++++
> 1 file changed, 18 insertions(+)
>
>diff --git a/include/linux/mm.h b/include/linux/mm.h
>index 25b9041f9925..5f2a15e702ff 100644
>--- a/include/linux/mm.h
>+++ b/include/linux/mm.h
>@@ -2689,6 +2689,24 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m
>	return vma;
> }

While at it can we clean up find_vma_intersection? I'm not particularly
user/fan of checkpatch.pl, but this one is kind of ridiculous.

Thanks,
Davidlohr

diff --git a/include/linux/mm.h b/include/linux/mm.h
index c274f75efcf9..16eddedf783f 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2678,9 +2678,14 @@ extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long add
  extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr,
					     struct vm_area_struct **pprev);

-/* Look up the first VMA which intersects the interval start_addr..end_addr-1,
-   NULL if none.  Assume start_addr < end_addr. */
-static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * mm, unsigned long start_addr, unsigned long end_addr)
+/*
+ * Look up the first VMA which intersects the interval start_addr..end_addr-1,
+ * NULL if none.  Assume start_addr < end_addr.
+ */
+static inline
+struct vm_area_struct *find_vma_intersection(struct mm_struct * mm,
+                                            unsigned long start_addr,
+                                            unsigned long end_addr)
  {
	struct vm_area_struct * vma = find_vma(mm,start_addr);


From dave at stgolabs.net  Thu May 20 20:47:43 2021
From: dave at stgolabs.net (Davidlohr Bueso)
Date: Fri, 21 May 2021 03:47:43 -0000
Subject: [PATCH 20/22] mm/mremap: Use vma_lookup() in vma_to_resize()
In-Reply-To: <20210510165839.2692974-21-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-21-Liam.Howlett@Oracle.com>
Message-ID: <20210521034719.fkaotqikwisie4kq@offworld>

On Mon, 10 May 2021, Liam Howlett wrote:

>Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
>will return NULL if the address is not within any VMA, the start address
>no longer needs to be validated.
>
>Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
>---
> mm/mremap.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
>diff --git a/mm/mremap.c b/mm/mremap.c
>index 47c255b60150..04143755cd1e 100644
>--- a/mm/mremap.c
>+++ b/mm/mremap.c
>@@ -634,10 +634,10 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr,
> 	unsigned long *p)
> {
> 	struct mm_struct *mm = current->mm;
>-	struct vm_area_struct *vma = find_vma(mm, addr);
>+	struct vm_area_struct *vma = vma_lookup(mm, addr);
> 	unsigned long pgoff;
>

Nit, but could the vma_lookup() call be separate from the declaration
of vma?

	vma = find_vma();
>-	if (!vma || vma->vm_start > addr)
>+	if (!vma)
> 		return ERR_PTR(-EFAULT);

Thanks,
Davidlohr


From dave at stgolabs.net  Thu May 20 21:22:22 2021
From: dave at stgolabs.net (Davidlohr Bueso)
Date: Fri, 21 May 2021 04:22:22 -0000
Subject: [PATCH 00/21] mm: Add vma_lookup()
In-Reply-To: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
Message-ID: <20210521042058.5urt6fr4idaosrue@offworld>

On Mon, 10 May 2021, Liam Howlett wrote:

>Adding the new vma_lookup() function will allow for cleaner code by
>removing the find_vma() calls which check limits, making
>find_vma_intersection() calls of a single address to be shorter, and
>potentially reduce the incorrect uses of find_vma().

I like this, specially implemented around find_vma(). For the series,
feel free to add:

Acked-by: Davidlohr Bueso <dbueso at suse.de>


From liam.howlett at oracle.com  Fri May 21 07:25:19 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 21 May 2021 14:25:19 -0000
Subject: [PATCH 01/22] mm: Add vma_lookup()
In-Reply-To: <20210521033232.dyams2dziycp4hv7@offworld>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-2-Liam.Howlett@Oracle.com>
 <20210521033232.dyams2dziycp4hv7@offworld>
Message-ID: <20210521142440.n6ejmeaxnxjus2to@revolver>

* Davidlohr Bueso <dave at stgolabs.net> [210520 23:32]:
> On Mon, 10 May 2021, Liam Howlett wrote:
> 
> > Many places in the kernel use find_vma() to get a vma and then check the
> > start address of the vma to ensure the next vma was not returned.
> > 
> > Other places use the find_vma_intersection() call with add, addr + 1 as
> > the range; looking for just the vma at a specific address.
> > 
> > The third use of find_vma() is by developers who do not know that the
> > function starts searching at the provided address upwards for the next
> > vma.  This results in a bug that is often overlooked for a long time.
> > 
> > Adding the new vma_lookup() function will allow for cleaner code by
> > removing the find_vma() calls which check limits, making
> > find_vma_intersection() calls of a single address to be shorter, and
> > potentially reduce the incorrect uses of find_vma().
> > 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> > include/linux/mm.h | 18 ++++++++++++++++++
> > 1 file changed, 18 insertions(+)
> > 
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index 25b9041f9925..5f2a15e702ff 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -2689,6 +2689,24 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m
> > 	return vma;
> > }
> 
> While at it can we clean up find_vma_intersection? I'm not particularly
> user/fan of checkpatch.pl, but this one is kind of ridiculous.

Agreed.  This addition is worth re-spin.  I will change the comment into
a kernel documentation style comment at the same time.

> 
> Thanks,
> Davidlohr
> 
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index c274f75efcf9..16eddedf783f 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2678,9 +2678,14 @@ extern struct vm_area_struct * find_vma(struct mm_struct * mm, unsigned long add
>  extern struct vm_area_struct * find_vma_prev(struct mm_struct * mm, unsigned long addr,
> 					     struct vm_area_struct **pprev);
> 
> -/* Look up the first VMA which intersects the interval start_addr..end_addr-1,
> -   NULL if none.  Assume start_addr < end_addr. */
> -static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * mm, unsigned long start_addr, unsigned long end_addr)
> +/*
> + * Look up the first VMA which intersects the interval start_addr..end_addr-1,
> + * NULL if none.  Assume start_addr < end_addr.
> + */
> +static inline
> +struct vm_area_struct *find_vma_intersection(struct mm_struct * mm,
> +                                            unsigned long start_addr,
> +                                            unsigned long end_addr)
>  {
> 	struct vm_area_struct * vma = find_vma(mm,start_addr);

From liam.howlett at oracle.com  Fri May 21 07:28:11 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 21 May 2021 14:28:11 -0000
Subject: [PATCH 20/22] mm/mremap: Use vma_lookup() in vma_to_resize()
In-Reply-To: <20210521034719.fkaotqikwisie4kq@offworld>
References: <20210510165839.2692974-1-Liam.Howlett@Oracle.com>
 <20210510165839.2692974-21-Liam.Howlett@Oracle.com>
 <20210521034719.fkaotqikwisie4kq@offworld>
Message-ID: <20210521142738.d7pbsxv453ebhrrz@revolver>

* Davidlohr Bueso <dave at stgolabs.net> [210520 23:47]:
> On Mon, 10 May 2021, Liam Howlett wrote:
> 
> > Use vma_lookup() to find the VMA at a specific address.  As vma_lookup()
> > will return NULL if the address is not within any VMA, the start address
> > no longer needs to be validated.
> > 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> > mm/mremap.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/mm/mremap.c b/mm/mremap.c
> > index 47c255b60150..04143755cd1e 100644
> > --- a/mm/mremap.c
> > +++ b/mm/mremap.c
> > @@ -634,10 +634,10 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr,
> > 	unsigned long *p)
> > {
> > 	struct mm_struct *mm = current->mm;
> > -	struct vm_area_struct *vma = find_vma(mm, addr);
> > +	struct vm_area_struct *vma = vma_lookup(mm, addr);
> > 	unsigned long pgoff;
> > 
> 
> Nit, but could the vma_lookup() call be separate from the declaration
> of vma?

Yes, I will make this change.

> 
> 	vma = find_vma();
> > -	if (!vma || vma->vm_start > addr)
> > +	if (!vma)
> > 		return ERR_PTR(-EFAULT);
> 
> Thanks,
> Davidlohr

From surenb at google.com  Fri May 28 10:31:10 2021
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 28 May 2021 17:31:10 -0000
Subject: [PATCH 21/94] radix tree test suite: Enhancements for Maple Tree
In-Reply-To: <20210428153542.2814175-22-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-22-Liam.Howlett@Oracle.com>
Message-ID: <CAJuCfpHto4kwjPuC-9iH9Pp45OO-6TJnHLGtfyJ-VtiaNstrAA@mail.gmail.com>

On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
>

I know you have v2 for the first part of this patchset, I'm just going
over the whole thing... There should be some description here of what
the new struct member and new function are for. Ideally you would also
split it in two because it introduces two seemingly independent
additions: non_kernel and kmem_cache_get_alloc.

> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Matthew Wilcox (Oracle) <willy at infradead.org>
> ---
>  tools/testing/radix-tree/linux.c        | 16 +++++++++++++++-
>  tools/testing/radix-tree/linux/kernel.h |  1 +
>  2 files changed, 16 insertions(+), 1 deletion(-)
>
> diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> index 2d9c59df60de..93f7de81fbe8 100644
> --- a/tools/testing/radix-tree/linux.c
> +++ b/tools/testing/radix-tree/linux.c
> @@ -24,15 +24,28 @@ struct kmem_cache {
>         int nr_objs;
>         void *objs;
>         void (*ctor)(void *);
> +       unsigned int non_kernel;
>  };
>
> +void kmem_cache_set_non_kernel(struct kmem_cache *cachep, unsigned int val)
> +{
> +       cachep->non_kernel = val;
> +}
> +
> +unsigned long kmem_cache_get_alloc(struct kmem_cache *cachep)
> +{
> +       return cachep->size * nr_allocated;

IIUC nr_allocated is incremented/decremented every time memory is
allocated/freed from *any* kmem_cache. Each kmem_cache has its own
size. So, nr_allocated counts allocated objects of potentially
different sizes. If that is so then I'm unclear what the result of
this multiplication would represent.

> +}
>  void *kmem_cache_alloc(struct kmem_cache *cachep, int gfp)
>  {
>         void *p;
>
> -       if (!(gfp & __GFP_DIRECT_RECLAIM))
> +       if (!(gfp & __GFP_DIRECT_RECLAIM) && !cachep->non_kernel)
>                 return NULL;
>
> +       if (!(gfp & __GFP_DIRECT_RECLAIM))
> +               cachep->non_kernel--;
> +
>         pthread_mutex_lock(&cachep->lock);
>         if (cachep->nr_objs) {
>                 struct radix_tree_node *node = cachep->objs;
> @@ -116,5 +129,6 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align,
>         ret->nr_objs = 0;
>         ret->objs = NULL;
>         ret->ctor = ctor;
> +       ret->non_kernel = 0;
>         return ret;
>  }
> diff --git a/tools/testing/radix-tree/linux/kernel.h b/tools/testing/radix-tree/linux/kernel.h
> index 39867fd80c8f..c5c9d05f29da 100644
> --- a/tools/testing/radix-tree/linux/kernel.h
> +++ b/tools/testing/radix-tree/linux/kernel.h
> @@ -14,6 +14,7 @@
>  #include "../../../include/linux/kconfig.h"
>
>  #define printk printf
> +#define pr_err printk
>  #define pr_info printk
>  #define pr_debug printk
>  #define pr_cont printk
> --
> 2.30.2


From surenb at google.com  Fri May 28 10:55:50 2021
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 28 May 2021 17:55:50 -0000
Subject: [PATCH 23/94] radix tree test suite: Add support for
 kmem_cache_free_bulk
In-Reply-To: <20210428153542.2814175-24-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-24-Liam.Howlett@Oracle.com>
Message-ID: <CAJuCfpFSE5MSDnMQ40fRx3mJ0eCLW0QDTMCbr05nrWDtC6wbaA@mail.gmail.com>

On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> ---
>  tools/testing/radix-tree/linux.c      | 9 +++++++++
>  tools/testing/radix-tree/linux/slab.h | 1 +
>  2 files changed, 10 insertions(+)
>
> diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> index 93f7de81fbe8..380bbc0a48d6 100644
> --- a/tools/testing/radix-tree/linux.c
> +++ b/tools/testing/radix-tree/linux.c
> @@ -91,6 +91,15 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp)
>         pthread_mutex_unlock(&cachep->lock);
>  }
>
> +void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
> +{
> +       if (kmalloc_verbose)
> +               printk("Bulk free %p[0-%lu]\n", list, size - 1);

nit: Printing the address of the "list" is meaningless IMHO unless you
output its value in kmem_cache_alloc_bulk, which you do not.
I would also suggest combining the patch introducing
kmem_cache_alloc_bulk with this one since they seem to be
compementary.

> +
> +       for (int i = 0; i < size; i++)
> +               kmem_cache_free(cachep, list[i]);
> +}
> +
>  void *kmalloc(size_t size, gfp_t gfp)
>  {
>         void *ret;
> diff --git a/tools/testing/radix-tree/linux/slab.h b/tools/testing/radix-tree/linux/slab.h
> index 2958830ce4d7..53b79c15b3a2 100644
> --- a/tools/testing/radix-tree/linux/slab.h
> +++ b/tools/testing/radix-tree/linux/slab.h
> @@ -24,4 +24,5 @@ struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
>                         unsigned int align, unsigned int flags,
>                         void (*ctor)(void *));
>
> +void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t, void **);
>  #endif         /* SLAB_H */
> --
> 2.30.2
>


From surenb at google.com  Fri May 28 11:17:57 2021
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 28 May 2021 18:17:57 -0000
Subject: [PATCH 24/94] radix tree test suite: Add keme_cache_alloc_bulk()
 support
In-Reply-To: <20210428153542.2814175-25-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-25-Liam.Howlett@Oracle.com>
Message-ID: <CAJuCfpEq3xVPrk0d_UNbgNOLQ7wN5rm4wx+CK2krc-bkmGpq1Q@mail.gmail.com>

On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> ---
>  tools/testing/radix-tree/linux.c      | 51 +++++++++++++++++++++++++++
>  tools/testing/radix-tree/linux/slab.h |  1 +
>  2 files changed, 52 insertions(+)
>
> diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> index 380bbc0a48d6..fb19a40ebb46 100644
> --- a/tools/testing/radix-tree/linux.c
> +++ b/tools/testing/radix-tree/linux.c
> @@ -99,6 +99,57 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
>         for (int i = 0; i < size; i++)
>                 kmem_cache_free(cachep, list[i]);
>  }
> +int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
> +                         void **p)
> +{
> +       size_t i;
> +
> +       if (kmalloc_verbose)
> +               printk("Bulk alloc %lu\n", size);
> +
> +       if (!(gfp & __GFP_DIRECT_RECLAIM) && cachep->non_kernel < size)
> +               return 0;
> +
> +       if (!(gfp & __GFP_DIRECT_RECLAIM))
> +               cachep->non_kernel -= size;
> +
> +       pthread_mutex_lock(&cachep->lock);
> +       if (cachep->nr_objs >= size) {
> +               struct radix_tree_node *node = cachep->objs;
> +

I don't think the loop below is correct because "node" is not being
changed on each iteration:

> +               for (i = 0; i < size; i++) {
> +                       cachep->nr_objs--;
> +                       cachep->objs = node->parent;

In the above assignment cachep->objs will be assigned the same value
on all iterations.

> +                       p[i] = cachep->objs;

p[0] should point to the node, however here it would point to the node->parent.

> +               }
> +               pthread_mutex_unlock(&cachep->lock);
> +               node->parent = NULL;

here you terminated the original cachep->objs which is not even inside
the "p" list at this point (it was skipped).

> +       } else {
> +               pthread_mutex_unlock(&cachep->lock);
> +               for (i = 0; i < size; i++) {
> +                       if (cachep->align) {
> +                               posix_memalign(&p[i], cachep->align,
> +                                              cachep->size * size);
> +                       } else {
> +                               p[i] = malloc(cachep->size * size);
> +                       }
> +                       if (cachep->ctor)
> +                               cachep->ctor(p[i]);
> +                       else if (gfp & __GFP_ZERO)
> +                               memset(p[i], 0, cachep->size);
> +               }
> +       }
> +
> +       for (i = 0; i < size; i++) {
> +               uatomic_inc(&nr_allocated);
> +               uatomic_inc(&nr_tallocated);

I don't see nr_tallocated even in linux-next branch. Was it introduced
in one of the previous patches and I missed it?

> +               if (kmalloc_verbose)
> +                       printf("Allocating %p from slab\n", p[i]);
> +       }
> +
> +       return size;
> +}
> +
>
>  void *kmalloc(size_t size, gfp_t gfp)
>  {
> diff --git a/tools/testing/radix-tree/linux/slab.h b/tools/testing/radix-tree/linux/slab.h
> index 53b79c15b3a2..ba42b8cc11d0 100644
> --- a/tools/testing/radix-tree/linux/slab.h
> +++ b/tools/testing/radix-tree/linux/slab.h
> @@ -25,4 +25,5 @@ struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
>                         void (*ctor)(void *));
>
>  void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t, void **);
> +int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t, size_t, void **);
>  #endif         /* SLAB_H */
> --
> 2.30.2


From liam.howlett at oracle.com  Fri May 28 11:55:49 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 28 May 2021 18:55:49 -0000
Subject: [PATCH 21/94] radix tree test suite: Enhancements for Maple Tree
In-Reply-To: <CAJuCfpHto4kwjPuC-9iH9Pp45OO-6TJnHLGtfyJ-VtiaNstrAA@mail.gmail.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-22-Liam.Howlett@Oracle.com>
 <CAJuCfpHto4kwjPuC-9iH9Pp45OO-6TJnHLGtfyJ-VtiaNstrAA@mail.gmail.com>
Message-ID: <20210528185449.vovlbet7omoex6ks@revolver>

* Suren Baghdasaryan <surenb at google.com> [210528 13:31]:
> On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
> >
> 
> I know you have v2 for the first part of this patchset, I'm just going
> over the whole thing... There should be some description here of what
> the new struct member and new function are for. Ideally you would also
> split it in two because it introduces two seemingly independent
> additions: non_kernel and kmem_cache_get_alloc.

Your comments are still valid and appreciated.

I did add a description to the patch:
--------------------------------
radix tree test suite: Add kmem_cache enhancements and pr_err

Add kmem_cache_set_non_kernel(), a mechanism to allow a certain number
of kmem_cache_alloc requests to succeed even when GFP_KERNEL is not set
in the flags.

Add kmem_cache_get_alloc() to see the size of the allocated kmem_cache.

Add a define of pr_err to printk.
--------------------------------

I did group these two changes together as they were both affecting
kmem_cache.  I will reorganize them into separate commits.

> 
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > Signed-off-by: Matthew Wilcox (Oracle) <willy at infradead.org>
> > ---
> >  tools/testing/radix-tree/linux.c        | 16 +++++++++++++++-
> >  tools/testing/radix-tree/linux/kernel.h |  1 +
> >  2 files changed, 16 insertions(+), 1 deletion(-)
> >
> > diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> > index 2d9c59df60de..93f7de81fbe8 100644
> > --- a/tools/testing/radix-tree/linux.c
> > +++ b/tools/testing/radix-tree/linux.c
> > @@ -24,15 +24,28 @@ struct kmem_cache {
> >         int nr_objs;
> >         void *objs;
> >         void (*ctor)(void *);
> > +       unsigned int non_kernel;
> >  };
> >
> > +void kmem_cache_set_non_kernel(struct kmem_cache *cachep, unsigned int val)
> > +{
> > +       cachep->non_kernel = val;
> > +}
> > +
> > +unsigned long kmem_cache_get_alloc(struct kmem_cache *cachep)
> > +{
> > +       return cachep->size * nr_allocated;
> 
> IIUC nr_allocated is incremented/decremented every time memory is
> allocated/freed from *any* kmem_cache. Each kmem_cache has its own
> size. So, nr_allocated counts allocated objects of potentially
> different sizes. If that is so then I'm unclear what the result of
> this multiplication would represent.

This is intended to only be used for testing with one kmem_cache, so it
hasn't been an issue.  Having this variable exist external to the
kmem_cache struct allows for checking if any allocations remain outside
the scope of the kmem_cache (ie: threads).  I think putting it in the
struct would cause issues with the IDR testing.  I could make this a new
variable and increment them together but this variable existed for the
IDR testing and I didn't need to support that additional functionality
for my testing.  I should at least add a comment about this limitation
though.

> 
> > +}
> >  void *kmem_cache_alloc(struct kmem_cache *cachep, int gfp)
> >  {
> >         void *p;
> >
> > -       if (!(gfp & __GFP_DIRECT_RECLAIM))
> > +       if (!(gfp & __GFP_DIRECT_RECLAIM) && !cachep->non_kernel)
> >                 return NULL;
> >
> > +       if (!(gfp & __GFP_DIRECT_RECLAIM))
> > +               cachep->non_kernel--;
> > +
> >         pthread_mutex_lock(&cachep->lock);
> >         if (cachep->nr_objs) {
> >                 struct radix_tree_node *node = cachep->objs;
> > @@ -116,5 +129,6 @@ kmem_cache_create(const char *name, unsigned int size, unsigned int align,
> >         ret->nr_objs = 0;
> >         ret->objs = NULL;
> >         ret->ctor = ctor;
> > +       ret->non_kernel = 0;
> >         return ret;
> >  }
> > diff --git a/tools/testing/radix-tree/linux/kernel.h b/tools/testing/radix-tree/linux/kernel.h
> > index 39867fd80c8f..c5c9d05f29da 100644
> > --- a/tools/testing/radix-tree/linux/kernel.h
> > +++ b/tools/testing/radix-tree/linux/kernel.h
> > @@ -14,6 +14,7 @@
> >  #include "../../../include/linux/kconfig.h"
> >
> >  #define printk printf
> > +#define pr_err printk
> >  #define pr_info printk
> >  #define pr_debug printk
> >  #define pr_cont printk
> > --
> > 2.30.2

From liam.howlett at oracle.com  Fri May 28 12:08:03 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 28 May 2021 19:08:03 -0000
Subject: [PATCH 23/94] radix tree test suite: Add support for
 kmem_cache_free_bulk
In-Reply-To: <CAJuCfpFSE5MSDnMQ40fRx3mJ0eCLW0QDTMCbr05nrWDtC6wbaA@mail.gmail.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-24-Liam.Howlett@Oracle.com>
 <CAJuCfpFSE5MSDnMQ40fRx3mJ0eCLW0QDTMCbr05nrWDtC6wbaA@mail.gmail.com>
Message-ID: <20210528190732.f7lxiyl6enmtidmi@revolver>

* Suren Baghdasaryan <surenb at google.com> [210528 13:55]:
> On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> >  tools/testing/radix-tree/linux.c      | 9 +++++++++
> >  tools/testing/radix-tree/linux/slab.h | 1 +
> >  2 files changed, 10 insertions(+)
> >
> > diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> > index 93f7de81fbe8..380bbc0a48d6 100644
> > --- a/tools/testing/radix-tree/linux.c
> > +++ b/tools/testing/radix-tree/linux.c
> > @@ -91,6 +91,15 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp)
> >         pthread_mutex_unlock(&cachep->lock);
> >  }
> >
> > +void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
> > +{
> > +       if (kmalloc_verbose)
> > +               printk("Bulk free %p[0-%lu]\n", list, size - 1);
> 
> nit: Printing the address of the "list" is meaningless IMHO unless you
> output its value in kmem_cache_alloc_bulk, which you do not.

The address has been rather useful for my testing when combined with
how the list is created and the LSAN_OPTIONS="report_objects=1".  When
this information is of interest is when a test fails, so the tree will
be dumped.  Combined with the list head and the report_objects output, I
am able to deduce if there is too much in the list or too few, which
operation caused the issue, and what calculation is of interest.

Adding the alloc_bulk counterpart is not very useful because the
prediction of how many nodes are necessary is the worst-case, so the
head of the list is almost never used and the request size is already
known.  Adding that print is just noise for my use case.

> I would also suggest combining the patch introducing
> kmem_cache_alloc_bulk with this one since they seem to be
> compementary.

Yes, I agree.  I noticed this and fixed it in v2.

> 
> > +
> > +       for (int i = 0; i < size; i++)
> > +               kmem_cache_free(cachep, list[i]);
> > +}
> > +
> >  void *kmalloc(size_t size, gfp_t gfp)
> >  {
> >         void *ret;
> > diff --git a/tools/testing/radix-tree/linux/slab.h b/tools/testing/radix-tree/linux/slab.h
> > index 2958830ce4d7..53b79c15b3a2 100644
> > --- a/tools/testing/radix-tree/linux/slab.h
> > +++ b/tools/testing/radix-tree/linux/slab.h
> > @@ -24,4 +24,5 @@ struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
> >                         unsigned int align, unsigned int flags,
> >                         void (*ctor)(void *));
> >
> > +void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t, void **);
> >  #endif         /* SLAB_H */
> > --
> > 2.30.2
> >

From liam.howlett at oracle.com  Fri May 28 12:28:58 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Fri, 28 May 2021 19:28:58 -0000
Subject: [PATCH 24/94] radix tree test suite: Add keme_cache_alloc_bulk()
 support
In-Reply-To: <CAJuCfpEq3xVPrk0d_UNbgNOLQ7wN5rm4wx+CK2krc-bkmGpq1Q@mail.gmail.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-25-Liam.Howlett@Oracle.com>
 <CAJuCfpEq3xVPrk0d_UNbgNOLQ7wN5rm4wx+CK2krc-bkmGpq1Q@mail.gmail.com>
Message-ID: <20210528192835.fqty5lk65dsg2q5c@revolver>

* Suren Baghdasaryan <surenb at google.com> [210528 14:17]:
> On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> >  tools/testing/radix-tree/linux.c      | 51 +++++++++++++++++++++++++++
> >  tools/testing/radix-tree/linux/slab.h |  1 +
> >  2 files changed, 52 insertions(+)
> >
> > diff --git a/tools/testing/radix-tree/linux.c b/tools/testing/radix-tree/linux.c
> > index 380bbc0a48d6..fb19a40ebb46 100644
> > --- a/tools/testing/radix-tree/linux.c
> > +++ b/tools/testing/radix-tree/linux.c
> > @@ -99,6 +99,57 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
> >         for (int i = 0; i < size; i++)
> >                 kmem_cache_free(cachep, list[i]);
> >  }
> > +int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
> > +                         void **p)
> > +{
> > +       size_t i;
> > +
> > +       if (kmalloc_verbose)
> > +               printk("Bulk alloc %lu\n", size);
> > +
> > +       if (!(gfp & __GFP_DIRECT_RECLAIM) && cachep->non_kernel < size)
> > +               return 0;
> > +
> > +       if (!(gfp & __GFP_DIRECT_RECLAIM))
> > +               cachep->non_kernel -= size;
> > +
> > +       pthread_mutex_lock(&cachep->lock);
> > +       if (cachep->nr_objs >= size) {
> > +               struct radix_tree_node *node = cachep->objs;
> > +
> 
> I don't think the loop below is correct because "node" is not being
> changed on each iteration:
> 
> > +               for (i = 0; i < size; i++) {
> > +                       cachep->nr_objs--;
> > +                       cachep->objs = node->parent;
> 
> In the above assignment cachep->objs will be assigned the same value
> on all iterations.
> 
> > +                       p[i] = cachep->objs;
> 
> p[0] should point to the node, however here it would point to the node->parent.
> 
> > +               }
> > +               pthread_mutex_unlock(&cachep->lock);
> > +               node->parent = NULL;
> 
> here you terminated the original cachep->objs which is not even inside
> the "p" list at this point (it was skipped).

I just verified that this code wasn't hit in my current test code.  I
will test and fix this.  Good catch.

> 
> > +       } else {
> > +               pthread_mutex_unlock(&cachep->lock);
> > +               for (i = 0; i < size; i++) {
> > +                       if (cachep->align) {
> > +                               posix_memalign(&p[i], cachep->align,
> > +                                              cachep->size * size);
> > +                       } else {
> > +                               p[i] = malloc(cachep->size * size);
> > +                       }
> > +                       if (cachep->ctor)
> > +                               cachep->ctor(p[i]);
> > +                       else if (gfp & __GFP_ZERO)
> > +                               memset(p[i], 0, cachep->size);
> > +               }
> > +       }
> > +
> > +       for (i = 0; i < size; i++) {
> > +               uatomic_inc(&nr_allocated);
> > +               uatomic_inc(&nr_tallocated);
> 
> I don't see nr_tallocated even in linux-next branch. Was it introduced
> in one of the previous patches and I missed it?

It was introduced with the maple tree itself.  I will spin this off as
its own patch with the same edits as nr_allocated.

> 
> > +               if (kmalloc_verbose)
> > +                       printf("Allocating %p from slab\n", p[i]);
> > +       }
> > +
> > +       return size;
> > +}
> > +
> >
> >  void *kmalloc(size_t size, gfp_t gfp)
> >  {
> > diff --git a/tools/testing/radix-tree/linux/slab.h b/tools/testing/radix-tree/linux/slab.h
> > index 53b79c15b3a2..ba42b8cc11d0 100644
> > --- a/tools/testing/radix-tree/linux/slab.h
> > +++ b/tools/testing/radix-tree/linux/slab.h
> > @@ -25,4 +25,5 @@ struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
> >                         void (*ctor)(void *));
> >
> >  void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t, void **);
> > +int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t, size_t, void **);
> >  #endif         /* SLAB_H */
> > --
> > 2.30.2

From surenb at google.com  Fri May 28 17:42:45 2021
From: surenb at google.com (Suren Baghdasaryan)
Date: Sat, 29 May 2021 00:42:45 -0000
Subject: [PATCH 32/94] kernel/fork: Convert dup_mmap to use maple tree
In-Reply-To: <20210428153542.2814175-33-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-33-Liam.Howlett@Oracle.com>
Message-ID: <CAJuCfpEBnrTShvYN9G9PTHjBWnEBeibuMwNOER66r1TdqBWNzg@mail.gmail.com>

On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
>
> Use the maple tree iterator to duplicate the mm_struct trees.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> ---
>  include/linux/mm.h       |  2 --
>  include/linux/sched/mm.h |  3 +++
>  kernel/fork.c            | 24 +++++++++++++++++++-----
>  mm/mmap.c                |  4 ----
>  4 files changed, 22 insertions(+), 11 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index e89bacfa9145..7f7dff6ad884 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2498,8 +2498,6 @@ extern bool arch_has_descending_max_zone_pfns(void);
>  /* nommu.c */
>  extern atomic_long_t mmap_pages_allocated;
>  extern int nommu_shrink_inode_mappings(struct inode *, size_t, size_t);
> -/* maple_tree */
> -void vma_store(struct mm_struct *mm, struct vm_area_struct *vma);
>
>  /* interval_tree.c */
>  void vma_interval_tree_insert(struct vm_area_struct *node,
> diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h
> index e24b1fe348e3..76cab3aea6ab 100644
> --- a/include/linux/sched/mm.h
> +++ b/include/linux/sched/mm.h
> @@ -8,6 +8,7 @@
>  #include <linux/mm_types.h>
>  #include <linux/gfp.h>
>  #include <linux/sync_core.h>
> +#include <linux/maple_tree.h>
>
>  /*
>   * Routines for handling mm_structs
> @@ -67,11 +68,13 @@ static inline void mmdrop(struct mm_struct *mm)
>   */
>  static inline void mmget(struct mm_struct *mm)
>  {
> +       mt_set_in_rcu(&mm->mm_mt);
>         atomic_inc(&mm->mm_users);
>  }
>
>  static inline bool mmget_not_zero(struct mm_struct *mm)
>  {
> +       mt_set_in_rcu(&mm->mm_mt);

Should you be calling mt_set_in_rcu() if atomic_inc_not_zero() failed?
I don't think mmput() is called after mmget_not_zero() fails and
mt_clear_in_rcu() will not be called.

>         return atomic_inc_not_zero(&mm->mm_users);
>  }
>
> diff --git a/kernel/fork.c b/kernel/fork.c
> index c37abaf28eb9..832416ff613e 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -477,7 +477,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>         struct vm_area_struct *mpnt, *tmp, *prev, **pprev;
>         struct rb_node **rb_link, *rb_parent;
>         int retval;
> -       unsigned long charge;
> +       unsigned long charge = 0;
> +       MA_STATE(old_mas, &oldmm->mm_mt, 0, 0);
> +       MA_STATE(mas, &mm->mm_mt, 0, 0);
>         LIST_HEAD(uf);
>
>         uprobe_start_dup_mmap();
> @@ -511,7 +513,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                 goto out;
>
>         prev = NULL;
> -       for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {
> +
> +       retval = mas_entry_count(&mas, oldmm->map_count);
> +       if (retval)
> +               goto fail_nomem;
> +
> +       rcu_read_lock();
> +       mas_for_each(&old_mas, mpnt, ULONG_MAX) {
>                 struct file *file;
>
>                 if (mpnt->vm_flags & VM_DONTCOPY) {
> @@ -525,7 +533,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                  */
>                 if (fatal_signal_pending(current)) {
>                         retval = -EINTR;
> -                       goto out;
> +                       goto loop_out;
>                 }
>                 if (mpnt->vm_flags & VM_ACCOUNT) {
>                         unsigned long len = vma_pages(mpnt);
> @@ -594,7 +602,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                 rb_parent = &tmp->vm_rb;
>
>                 /* Link the vma into the MT */
> -               vma_store(mm, tmp);
> +               mas.index = tmp->vm_start;
> +               mas.last = tmp->vm_end - 1;
> +               mas_store(&mas, tmp);
>
>                 mm->map_count++;
>                 if (!(tmp->vm_flags & VM_WIPEONFORK))
> @@ -604,14 +614,17 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                         tmp->vm_ops->open(tmp);
>
>                 if (retval)
> -                       goto out;
> +                       goto loop_out;
>         }
>         /* a new mm has just been created */
>         retval = arch_dup_mmap(oldmm, mm);
> +loop_out:
>  out:
> +       rcu_read_unlock();
>         mmap_write_unlock(mm);
>         flush_tlb_mm(oldmm);
>         mmap_write_unlock(oldmm);
> +       mas_destroy(&mas);
>         dup_userfaultfd_complete(&uf);
>  fail_uprobe_end:
>         uprobe_end_dup_mmap();
> @@ -1092,6 +1105,7 @@ static inline void __mmput(struct mm_struct *mm)
>  {
>         VM_BUG_ON(atomic_read(&mm->mm_users));
>
> +       mt_clear_in_rcu(&mm->mm_mt);
>         uprobe_clear_state(mm);
>         exit_aio(mm);
>         ksm_exit(mm);
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 929c2f9eb3f5..1bd43f4db28e 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -780,10 +780,6 @@ static inline void vma_mt_store(struct mm_struct *mm, struct vm_area_struct *vma
>                 GFP_KERNEL);
>  }
>
> -void vma_store(struct mm_struct *mm, struct vm_area_struct *vma) {
> -       vma_mt_store(mm, vma);
> -}
> -
>  static void
>  __vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
>         struct vm_area_struct *prev, struct rb_node **rb_link,
> --
> 2.30.2


From surenb at google.com  Fri May 28 18:26:48 2021
From: surenb at google.com (Suren Baghdasaryan)
Date: Sat, 29 May 2021 01:26:48 -0000
Subject: [PATCH 33/94] mm: Remove rb tree.
In-Reply-To: <20210428153542.2814175-34-Liam.Howlett@Oracle.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-34-Liam.Howlett@Oracle.com>
Message-ID: <CAJuCfpGD3EC_chwHWBETSJ9x7DR6umX6kFZES0sWyphtVNeNRw@mail.gmail.com>

On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
>
> Remove the RB tree and start using the maple tree for vm_area_struct
> tracking.
>
> Drop validate_mm() calls in expand_upwards() and expand_downwards() as
> the lock is not held.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> ---
>  arch/x86/kernel/tboot.c    |   1 -
>  drivers/firmware/efi/efi.c |   1 -
>  fs/proc/task_nommu.c       |  55 ++--
>  include/linux/mm.h         |   4 +-
>  include/linux/mm_types.h   |  26 +-
>  kernel/fork.c              |   8 -
>  mm/init-mm.c               |   2 -
>  mm/mmap.c                  | 525 ++++++++-----------------------------
>  mm/nommu.c                 |  96 +++----
>  mm/util.c                  |   8 +
>  10 files changed, 185 insertions(+), 541 deletions(-)
>
> diff --git a/arch/x86/kernel/tboot.c b/arch/x86/kernel/tboot.c
> index 6f978f722dff..121f28bb2209 100644
> --- a/arch/x86/kernel/tboot.c
> +++ b/arch/x86/kernel/tboot.c
> @@ -97,7 +97,6 @@ void __init tboot_probe(void)
>
>  static pgd_t *tboot_pg_dir;
>  static struct mm_struct tboot_mm = {
> -       .mm_rb          = RB_ROOT,
>         .mm_mt          = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
>         .pgd            = swapper_pg_dir,
>         .mm_users       = ATOMIC_INIT(2),
> diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
> index 271ae8c7bb07..8aaeaa824576 100644
> --- a/drivers/firmware/efi/efi.c
> +++ b/drivers/firmware/efi/efi.c
> @@ -54,7 +54,6 @@ static unsigned long __initdata mem_reserve = EFI_INVALID_TABLE_ADDR;
>  static unsigned long __initdata rt_prop = EFI_INVALID_TABLE_ADDR;
>
>  struct mm_struct efi_mm = {
> -       .mm_rb                  = RB_ROOT,
>         .mm_mt                  = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
>         .mm_users               = ATOMIC_INIT(2),
>         .mm_count               = ATOMIC_INIT(1),
> diff --git a/fs/proc/task_nommu.c b/fs/proc/task_nommu.c
> index a6d21fc0033c..8691a1216d1c 100644
> --- a/fs/proc/task_nommu.c
> +++ b/fs/proc/task_nommu.c
> @@ -22,15 +22,13 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
>  {
>         struct vm_area_struct *vma;
>         struct vm_region *region;
> -       struct rb_node *p;
>         unsigned long bytes = 0, sbytes = 0, slack = 0, size;
> -
> -       mmap_read_lock(mm);
> -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> +       MA_STATE(mas, &mm->mm_mt, 0, 0);
>
> +       mmap_read_lock(mm);
> +       rcu_read_lock();
> +       mas_for_each(&mas, vma, ULONG_MAX) {
>                 bytes += kobjsize(vma);
> -
>                 region = vma->vm_region;
>                 if (region) {
>                         size = kobjsize(region);
> @@ -53,7 +51,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
>                 sbytes += kobjsize(mm);
>         else
>                 bytes += kobjsize(mm);
> -
> +
>         if (current->fs && current->fs->users > 1)
>                 sbytes += kobjsize(current->fs);
>         else
> @@ -77,20 +75,21 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
>                 "Shared:\t%8lu bytes\n",
>                 bytes, slack, sbytes);
>
> +       rcu_read_unlock();
>         mmap_read_unlock(mm);
>  }
>
>  unsigned long task_vsize(struct mm_struct *mm)
>  {
>         struct vm_area_struct *vma;
> -       struct rb_node *p;
>         unsigned long vsize = 0;
> +       MA_STATE(mas, &mm->mm_mt, 0, 0);
>
>         mmap_read_lock(mm);
> -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> +       rcu_read_lock();
> +       mas_for_each(&mas, vma, ULONG_MAX)
>                 vsize += vma->vm_end - vma->vm_start;
> -       }
> +       rcu_read_unlock();
>         mmap_read_unlock(mm);
>         return vsize;
>  }
> @@ -101,12 +100,12 @@ unsigned long task_statm(struct mm_struct *mm,
>  {
>         struct vm_area_struct *vma;
>         struct vm_region *region;
> -       struct rb_node *p;
>         unsigned long size = kobjsize(mm);
> +       MA_STATE(mas, &mm->mm_mt, 0, 0);
>
>         mmap_read_lock(mm);
> -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> +       rcu_read_lock();
> +       mas_for_each(&mas, vma, ULONG_MAX) {
>                 size += kobjsize(vma);
>                 region = vma->vm_region;
>                 if (region) {
> @@ -119,6 +118,7 @@ unsigned long task_statm(struct mm_struct *mm,
>                 >> PAGE_SHIFT;
>         *data = (PAGE_ALIGN(mm->start_stack) - (mm->start_data & PAGE_MASK))
>                 >> PAGE_SHIFT;
> +       rcu_read_unlock();
>         mmap_read_unlock(mm);
>         size >>= PAGE_SHIFT;
>         size += *text + *data;
> @@ -190,17 +190,20 @@ static int nommu_vma_show(struct seq_file *m, struct vm_area_struct *vma)
>   */
>  static int show_map(struct seq_file *m, void *_p)
>  {
> -       struct rb_node *p = _p;
> -
> -       return nommu_vma_show(m, rb_entry(p, struct vm_area_struct, vm_rb));
> +       return nommu_vma_show(m, _p);
>  }
>
>  static void *m_start(struct seq_file *m, loff_t *pos)
>  {
>         struct proc_maps_private *priv = m->private;
>         struct mm_struct *mm;
> -       struct rb_node *p;
> -       loff_t n = *pos;
> +       struct vm_area_struct *vma;
> +       unsigned long addr = *pos;
> +       MA_STATE(mas, &priv->mm->mm_mt, addr, addr);
> +
> +       /* See m_next(). Zero at the start or after lseek. */
> +       if (addr == -1UL)
> +               return NULL;
>
>         /* pin the task and mm whilst we play with them */
>         priv->task = get_proc_task(priv->inode);
> @@ -216,14 +219,12 @@ static void *m_start(struct seq_file *m, loff_t *pos)
>                 return ERR_PTR(-EINTR);
>         }
>
> -       /* start from the Nth VMA */
> -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p))
> -               if (n-- == 0)
> -                       return p;
> +       /* start the next element from addr */
> +       vma = mas_find(&mas, ULONG_MAX);
>
>         mmap_read_unlock(mm);
>         mmput(mm);
> -       return NULL;
> +       return vma;
>  }
>
>  static void m_stop(struct seq_file *m, void *_vml)
> @@ -242,10 +243,10 @@ static void m_stop(struct seq_file *m, void *_vml)
>
>  static void *m_next(struct seq_file *m, void *_p, loff_t *pos)
>  {
> -       struct rb_node *p = _p;
> +       struct vm_area_struct *vma = _p;
>
> -       (*pos)++;
> -       return p ? rb_next(p) : NULL;
> +       *pos = vma->vm_end;
> +       return vma->vm_next;
>  }
>
>  static const struct seq_operations proc_pid_maps_ops = {
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 7f7dff6ad884..146976070fed 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2553,8 +2553,6 @@ extern int __split_vma(struct mm_struct *, struct vm_area_struct *,
>  extern int split_vma(struct mm_struct *, struct vm_area_struct *,
>         unsigned long addr, int new_below);
>  extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);
> -extern void __vma_link_rb(struct mm_struct *, struct vm_area_struct *,
> -       struct rb_node **, struct rb_node *);
>  extern void unlink_file_vma(struct vm_area_struct *);
>  extern struct vm_area_struct *copy_vma(struct vm_area_struct **,
>         unsigned long addr, unsigned long len, pgoff_t pgoff,
> @@ -2699,7 +2697,7 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m
>  static inline
>  struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
>  {
> -       return find_vma_intersection(mm, addr, addr + 1);
> +       return mtree_load(&mm->mm_mt, addr);
>  }
>
>  static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
> diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> index 51733fc44daf..41551bfa6ce0 100644
> --- a/include/linux/mm_types.h
> +++ b/include/linux/mm_types.h
> @@ -311,19 +311,6 @@ struct vm_area_struct {
>
>         /* linked list of VM areas per task, sorted by address */
>         struct vm_area_struct *vm_next, *vm_prev;
> -
> -       struct rb_node vm_rb;
> -
> -       /*
> -        * Largest free memory gap in bytes to the left of this VMA.
> -        * Either between this VMA and vma->vm_prev, or between one of the
> -        * VMAs below us in the VMA rbtree and its ->vm_prev. This helps
> -        * get_unmapped_area find a free area of the right size.
> -        */
> -       unsigned long rb_subtree_gap;
> -
> -       /* Second cache line starts here. */
> -
>         struct mm_struct *vm_mm;        /* The address space we belong to. */
>
>         /*
> @@ -333,6 +320,12 @@ struct vm_area_struct {
>         pgprot_t vm_page_prot;
>         unsigned long vm_flags;         /* Flags, see mm.h. */
>
> +       /* Information about our backing store: */
> +       unsigned long vm_pgoff;         /* Offset (within vm_file) in PAGE_SIZE
> +                                        * units
> +                                        */
> +       /* Second cache line starts here. */
> +       struct file *vm_file;           /* File we map to (can be NULL). */
>         /*
>          * For areas with an address space and backing store,
>          * linkage into the address_space->i_mmap interval tree.
> @@ -351,16 +344,14 @@ struct vm_area_struct {
>         struct list_head anon_vma_chain; /* Serialized by mmap_lock &
>                                           * page_table_lock */
>         struct anon_vma *anon_vma;      /* Serialized by page_table_lock */
> +       /* Third cache line starts here. */
>
>         /* Function pointers to deal with this struct. */
>         const struct vm_operations_struct *vm_ops;
>
> -       /* Information about our backing store: */
> -       unsigned long vm_pgoff;         /* Offset (within vm_file) in PAGE_SIZE
> -                                          units */
> -       struct file * vm_file;          /* File we map to (can be NULL). */
>         void * vm_private_data;         /* was vm_pte (shared mem) */
>
> +
>  #ifdef CONFIG_SWAP
>         atomic_long_t swap_readahead_info;
>  #endif
> @@ -389,7 +380,6 @@ struct mm_struct {
>         struct {
>                 struct vm_area_struct *mmap;            /* list of VMAs */
>                 struct maple_tree mm_mt;
> -               struct rb_root mm_rb;
>                 u64 vmacache_seqnum;                   /* per-thread vmacache */
>  #ifdef CONFIG_MMU
>                 unsigned long (*get_unmapped_area) (struct file *filp,
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 832416ff613e..83afd3007a2b 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -475,7 +475,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                                         struct mm_struct *oldmm)
>  {
>         struct vm_area_struct *mpnt, *tmp, *prev, **pprev;
> -       struct rb_node **rb_link, *rb_parent;
>         int retval;
>         unsigned long charge = 0;
>         MA_STATE(old_mas, &oldmm->mm_mt, 0, 0);
> @@ -502,8 +501,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>         mm->exec_vm = oldmm->exec_vm;
>         mm->stack_vm = oldmm->stack_vm;
>
> -       rb_link = &mm->mm_rb.rb_node;
> -       rb_parent = NULL;
>         pprev = &mm->mmap;
>         retval = ksm_fork(mm, oldmm);
>         if (retval)
> @@ -597,10 +594,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
>                 tmp->vm_prev = prev;
>                 prev = tmp;
>
> -               __vma_link_rb(mm, tmp, rb_link, rb_parent);
> -               rb_link = &tmp->vm_rb.rb_right;
> -               rb_parent = &tmp->vm_rb;
> -
>                 /* Link the vma into the MT */
>                 mas.index = tmp->vm_start;
>                 mas.last = tmp->vm_end - 1;
> @@ -1033,7 +1026,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
>         struct user_namespace *user_ns)
>  {
>         mm->mmap = NULL;
> -       mm->mm_rb = RB_ROOT;
>         mt_init_flags(&mm->mm_mt, MAPLE_ALLOC_RANGE);
>         mm->vmacache_seqnum = 0;
>         atomic_set(&mm->mm_users, 1);
> diff --git a/mm/init-mm.c b/mm/init-mm.c
> index 2014d4b82294..04bbe5172b72 100644
> --- a/mm/init-mm.c
> +++ b/mm/init-mm.c
> @@ -1,6 +1,5 @@
>  // SPDX-License-Identifier: GPL-2.0
>  #include <linux/mm_types.h>
> -#include <linux/rbtree.h>
>  #include <linux/maple_tree.h>
>  #include <linux/rwsem.h>
>  #include <linux/spinlock.h>
> @@ -28,7 +27,6 @@
>   * and size this cpu_bitmask to NR_CPUS.
>   */
>  struct mm_struct init_mm = {
> -       .mm_rb          = RB_ROOT,
>         .mm_mt          = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
>         .pgd            = swapper_pg_dir,
>         .mm_users       = ATOMIC_INIT(2),
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 1bd43f4db28e..7747047c4cbe 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -38,7 +38,6 @@
>  #include <linux/audit.h>
>  #include <linux/khugepaged.h>
>  #include <linux/uprobes.h>
> -#include <linux/rbtree_augmented.h>
>  #include <linux/notifier.h>
>  #include <linux/memory.h>
>  #include <linux/printk.h>
> @@ -290,93 +289,6 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
>         return origbrk;
>  }
>
> -static inline unsigned long vma_compute_gap(struct vm_area_struct *vma)
> -{
> -       unsigned long gap, prev_end;
> -
> -       /*
> -        * Note: in the rare case of a VM_GROWSDOWN above a VM_GROWSUP, we
> -        * allow two stack_guard_gaps between them here, and when choosing
> -        * an unmapped area; whereas when expanding we only require one.
> -        * That's a little inconsistent, but keeps the code here simpler.
> -        */
> -       gap = vm_start_gap(vma);
> -       if (vma->vm_prev) {
> -               prev_end = vm_end_gap(vma->vm_prev);
> -               if (gap > prev_end)
> -                       gap -= prev_end;
> -               else
> -                       gap = 0;
> -       }
> -       return gap;
> -}
> -
> -#ifdef CONFIG_DEBUG_VM_RB
> -static unsigned long vma_compute_subtree_gap(struct vm_area_struct *vma)
> -{
> -       unsigned long max = vma_compute_gap(vma), subtree_gap;
> -       if (vma->vm_rb.rb_left) {
> -               subtree_gap = rb_entry(vma->vm_rb.rb_left,
> -                               struct vm_area_struct, vm_rb)->rb_subtree_gap;
> -               if (subtree_gap > max)
> -                       max = subtree_gap;
> -       }
> -       if (vma->vm_rb.rb_right) {
> -               subtree_gap = rb_entry(vma->vm_rb.rb_right,
> -                               struct vm_area_struct, vm_rb)->rb_subtree_gap;
> -               if (subtree_gap > max)
> -                       max = subtree_gap;
> -       }
> -       return max;
> -}
> -
> -static int browse_rb(struct mm_struct *mm)
> -{
> -       struct rb_root *root = &mm->mm_rb;
> -       int i = 0, j, bug = 0;
> -       struct rb_node *nd, *pn = NULL;
> -       unsigned long prev = 0, pend = 0;
> -
> -       for (nd = rb_first(root); nd; nd = rb_next(nd)) {
> -               struct vm_area_struct *vma;
> -               vma = rb_entry(nd, struct vm_area_struct, vm_rb);
> -               if (vma->vm_start < prev) {
> -                       pr_emerg("vm_start %lx < prev %lx\n",
> -                                 vma->vm_start, prev);
> -                       bug = 1;
> -               }
> -               if (vma->vm_start < pend) {
> -                       pr_emerg("vm_start %lx < pend %lx\n",
> -                                 vma->vm_start, pend);
> -                       bug = 1;
> -               }
> -               if (vma->vm_start > vma->vm_end) {
> -                       pr_emerg("vm_start %lx > vm_end %lx\n",
> -                                 vma->vm_start, vma->vm_end);
> -                       bug = 1;
> -               }
> -               spin_lock(&mm->page_table_lock);
> -               if (vma->rb_subtree_gap != vma_compute_subtree_gap(vma)) {
> -                       pr_emerg("free gap %lx, correct %lx\n",
> -                              vma->rb_subtree_gap,
> -                              vma_compute_subtree_gap(vma));
> -                       bug = 1;
> -               }
> -               spin_unlock(&mm->page_table_lock);
> -               i++;
> -               pn = nd;
> -               prev = vma->vm_start;
> -               pend = vma->vm_end;
> -       }
> -       j = 0;
> -       for (nd = pn; nd; nd = rb_prev(nd))
> -               j++;
> -       if (i != j) {
> -               pr_emerg("backwards %d, forwards %d\n", j, i);
> -               bug = 1;
> -       }
> -       return bug ? -1 : i;
> -}
>  #if defined(CONFIG_DEBUG_MAPLE_TREE)
>  extern void mt_validate(struct maple_tree *mt);
>  extern void mt_dump(const struct maple_tree *mt);
> @@ -405,17 +317,25 @@ static void validate_mm_mt(struct mm_struct *mm)
>                         dump_stack();
>  #ifdef CONFIG_DEBUG_VM
>                         dump_vma(vma_mt);
> -                       pr_emerg("and next in rb\n");
> +                       pr_emerg("and vm_next\n");
>                         dump_vma(vma->vm_next);
> -#endif
> +#endif // CONFIG_DEBUG_VM
>                         pr_emerg("mt piv: %px %lu - %lu\n", vma_mt,
>                                  mas.index, mas.last);
>                         pr_emerg("mt vma: %px %lu - %lu\n", vma_mt,
>                                  vma_mt->vm_start, vma_mt->vm_end);
> -                       pr_emerg("rb vma: %px %lu - %lu\n", vma,
> +                       if (vma->vm_prev) {
> +                               pr_emerg("ll prev: %px %lu - %lu\n",
> +                                        vma->vm_prev, vma->vm_prev->vm_start,
> +                                        vma->vm_prev->vm_end);
> +                       }
> +                       pr_emerg("ll vma: %px %lu - %lu\n", vma,
>                                  vma->vm_start, vma->vm_end);
> -                       pr_emerg("rb->next = %px %lu - %lu\n", vma->vm_next,
> -                                       vma->vm_next->vm_start, vma->vm_next->vm_end);
> +                       if (vma->vm_next) {
> +                               pr_emerg("ll next: %px %lu - %lu\n",
> +                                        vma->vm_next, vma->vm_next->vm_start,
> +                                        vma->vm_next->vm_end);
> +                       }
>
>                         mt_dump(mas.tree);
>                         if (vma_mt->vm_end != mas.last + 1) {
> @@ -441,21 +361,6 @@ static void validate_mm_mt(struct mm_struct *mm)
>         rcu_read_unlock();
>         mt_validate(&mm->mm_mt);
>  }
> -#else
> -#define validate_mm_mt(root) do { } while (0)
> -#endif
> -static void validate_mm_rb(struct rb_root *root, struct vm_area_struct *ignore)
> -{
> -       struct rb_node *nd;
> -
> -       for (nd = rb_first(root); nd; nd = rb_next(nd)) {
> -               struct vm_area_struct *vma;
> -               vma = rb_entry(nd, struct vm_area_struct, vm_rb);
> -               VM_BUG_ON_VMA(vma != ignore &&
> -                       vma->rb_subtree_gap != vma_compute_subtree_gap(vma),
> -                       vma);
> -       }
> -}
>
>  static void validate_mm(struct mm_struct *mm)
>  {
> @@ -464,6 +369,8 @@ static void validate_mm(struct mm_struct *mm)
>         unsigned long highest_address = 0;
>         struct vm_area_struct *vma = mm->mmap;
>
> +       validate_mm_mt(mm);
> +
>         while (vma) {
>                 struct anon_vma *anon_vma = vma->anon_vma;
>                 struct anon_vma_chain *avc;
> @@ -488,80 +395,13 @@ static void validate_mm(struct mm_struct *mm)
>                           mm->highest_vm_end, highest_address);
>                 bug = 1;
>         }
> -       i = browse_rb(mm);
> -       if (i != mm->map_count) {
> -               if (i != -1)
> -                       pr_emerg("map_count %d rb %d\n", mm->map_count, i);
> -               bug = 1;
> -       }
>         VM_BUG_ON_MM(bug, mm);
>  }
> -#else
> -#define validate_mm_rb(root, ignore) do { } while (0)
> +
> +#else // !CONFIG_DEBUG_MAPLE_TREE
>  #define validate_mm_mt(root) do { } while (0)
>  #define validate_mm(mm) do { } while (0)
> -#endif
> -
> -RB_DECLARE_CALLBACKS_MAX(static, vma_gap_callbacks,
> -                        struct vm_area_struct, vm_rb,
> -                        unsigned long, rb_subtree_gap, vma_compute_gap)
> -
> -/*
> - * Update augmented rbtree rb_subtree_gap values after vma->vm_start or
> - * vma->vm_prev->vm_end values changed, without modifying the vma's position
> - * in the rbtree.
> - */
> -static void vma_gap_update(struct vm_area_struct *vma)
> -{
> -       /*
> -        * As it turns out, RB_DECLARE_CALLBACKS_MAX() already created
> -        * a callback function that does exactly what we want.
> -        */
> -       vma_gap_callbacks_propagate(&vma->vm_rb, NULL);
> -}
> -
> -static inline void vma_rb_insert(struct vm_area_struct *vma,
> -                                struct rb_root *root)
> -{
> -       /* All rb_subtree_gap values must be consistent prior to insertion */
> -       validate_mm_rb(root, NULL);
> -
> -       rb_insert_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
> -}
> -
> -static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root)
> -{
> -       /*
> -        * Note rb_erase_augmented is a fairly large inline function,
> -        * so make sure we instantiate it only once with our desired
> -        * augmented rbtree callbacks.
> -        */
> -       rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
> -}
> -
> -static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma,
> -                                               struct rb_root *root,
> -                                               struct vm_area_struct *ignore)
> -{
> -       /*
> -        * All rb_subtree_gap values must be consistent prior to erase,
> -        * with the possible exception of
> -        *
> -        * a. the "next" vma being erased if next->vm_start was reduced in
> -        *    __vma_adjust() -> __vma_unlink()
> -        * b. the vma being erased in detach_vmas_to_be_unmapped() ->
> -        *    vma_rb_erase()
> -        */
> -       validate_mm_rb(root, ignore);
> -
> -       __vma_rb_erase(vma, root);
> -}
> -
> -static __always_inline void vma_rb_erase(struct vm_area_struct *vma,
> -                                        struct rb_root *root)
> -{
> -       vma_rb_erase_ignore(vma, root, vma);
> -}
> +#endif // CONFIG_DEBUG_MAPLE_TREE
>
>  /*
>   * vma has some anon_vma assigned, and is already inserted on that
> @@ -595,38 +435,26 @@ anon_vma_interval_tree_post_update_vma(struct vm_area_struct *vma)
>                 anon_vma_interval_tree_insert(avc, &avc->anon_vma->rb_root);
>  }
>
> -static int find_vma_links(struct mm_struct *mm, unsigned long addr,
> -               unsigned long end, struct vm_area_struct **pprev,
> -               struct rb_node ***rb_link, struct rb_node **rb_parent)
> +/* Private
> + * range_has_overlap() - Check the @start - @end range for overlapping VMAs and
> + * sets up a pointer to the previous VMA
> + *
> + * @mm - the mm struct
> + * @start - the start address of the range
> + * @end - the end address of the range
> + * @pprev - the pointer to the pointer of the previous VMA
> + *
> + * Returns: True if there is an overlapping VMA, false otherwise
> + */
> +static bool range_has_overlap(struct mm_struct *mm, unsigned long start,
> +                             unsigned long end, struct vm_area_struct **pprev)
>  {
> -       struct rb_node **__rb_link, *__rb_parent, *rb_prev;
> -
> -       __rb_link = &mm->mm_rb.rb_node;
> -       rb_prev = __rb_parent = NULL;
> +       struct vm_area_struct *existing;
>
> -       while (*__rb_link) {
> -               struct vm_area_struct *vma_tmp;
> -
> -               __rb_parent = *__rb_link;
> -               vma_tmp = rb_entry(__rb_parent, struct vm_area_struct, vm_rb);
> -
> -               if (vma_tmp->vm_end > addr) {
> -                       /* Fail if an existing vma overlaps the area */
> -                       if (vma_tmp->vm_start < end)
> -                               return -ENOMEM;
> -                       __rb_link = &__rb_parent->rb_left;
> -               } else {
> -                       rb_prev = __rb_parent;
> -                       __rb_link = &__rb_parent->rb_right;
> -               }
> -       }
> -
> -       *pprev = NULL;
> -       if (rb_prev)
> -               *pprev = rb_entry(rb_prev, struct vm_area_struct, vm_rb);
> -       *rb_link = __rb_link;
> -       *rb_parent = __rb_parent;
> -       return 0;
> +       MA_STATE(mas, &mm->mm_mt, start, start);
> +       existing = mas_find(&mas, end - 1);
> +       *pprev = mas_prev(&mas, 0);
> +       return existing ? true : false;
>  }
>
>  /*
> @@ -653,8 +481,6 @@ static inline struct vm_area_struct *vma_next(struct mm_struct *mm,
>   * @start: The start of the range.
>   * @len: The length of the range.
>   * @pprev: pointer to the pointer that will be set to previous vm_area_struct
> - * @rb_link: the rb_node
> - * @rb_parent: the parent rb_node
>   *
>   * Find all the vm_area_struct that overlap from @start to
>   * @end and munmap them.  Set @pprev to the previous vm_area_struct.
> @@ -663,76 +489,41 @@ static inline struct vm_area_struct *vma_next(struct mm_struct *mm,
>   */
>  static inline int
>  munmap_vma_range(struct mm_struct *mm, unsigned long start, unsigned long len,
> -                struct vm_area_struct **pprev, struct rb_node ***link,
> -                struct rb_node **parent, struct list_head *uf)
> +                struct vm_area_struct **pprev, struct list_head *uf)
>  {
> -
> -       while (find_vma_links(mm, start, start + len, pprev, link, parent))
> +       // Needs optimization.
> +       while (range_has_overlap(mm, start, start + len, pprev))
>                 if (do_munmap(mm, start, len, uf))
>                         return -ENOMEM;
> -
>         return 0;
>  }
>  static unsigned long count_vma_pages_range(struct mm_struct *mm,
>                 unsigned long addr, unsigned long end)
>  {
>         unsigned long nr_pages = 0;
> -       unsigned long nr_mt_pages = 0;
>         struct vm_area_struct *vma;
> +       unsigned long vm_start, vm_end;
> +       MA_STATE(mas, &mm->mm_mt, addr, addr);
>
> -       /* Find first overlapping mapping */
> -       vma = find_vma_intersection(mm, addr, end);
> +       /* Find first overlaping mapping */

nit: I think the original comment was correct.

> +       vma = mas_find(&mas, end - 1);
>         if (!vma)
>                 return 0;
>
> -       nr_pages = (min(end, vma->vm_end) -
> -               max(addr, vma->vm_start)) >> PAGE_SHIFT;
> +       vm_start = vma->vm_start;
> +       vm_end = vma->vm_end;
> +       nr_pages = (min(end, vm_end) - max(addr, vm_start)) >> PAGE_SHIFT;
>
>         /* Iterate over the rest of the overlaps */
> -       for (vma = vma->vm_next; vma; vma = vma->vm_next) {
> -               unsigned long overlap_len;
> -
> -               if (vma->vm_start > end)
> -                       break;
> -
> -               overlap_len = min(end, vma->vm_end) - vma->vm_start;
> -               nr_pages += overlap_len >> PAGE_SHIFT;
> +       mas_for_each(&mas, vma, end) {
> +               vm_start = vma->vm_start;
> +               vm_end = vma->vm_end;
> +               nr_pages += (min(end, vm_end) - vm_start) >> PAGE_SHIFT;
>         }
>
> -       mt_for_each(&mm->mm_mt, vma, addr, end) {
> -               nr_mt_pages +=
> -                       (min(end, vma->vm_end) - vma->vm_start) >> PAGE_SHIFT;
> -       }
> -
> -       VM_BUG_ON_MM(nr_pages != nr_mt_pages, mm);
> -
>         return nr_pages;
>  }
>
> -void __vma_link_rb(struct mm_struct *mm, struct vm_area_struct *vma,
> -               struct rb_node **rb_link, struct rb_node *rb_parent)
> -{
> -       /* Update tracking information for the gap following the new vma. */
> -       if (vma->vm_next)
> -               vma_gap_update(vma->vm_next);
> -       else
> -               mm->highest_vm_end = vm_end_gap(vma);
> -
> -       /*
> -        * vma->vm_prev wasn't known when we followed the rbtree to find the
> -        * correct insertion point for that vma. As a result, we could not
> -        * update the vma vm_rb parents rb_subtree_gap values on the way down.
> -        * So, we first insert the vma with a zero rb_subtree_gap value
> -        * (to be consistent with what we did on the way down), and then
> -        * immediately update the gap to the correct value. Finally we
> -        * rebalance the rbtree after all augmented values have been set.
> -        */
> -       rb_link_node(&vma->vm_rb, rb_parent, rb_link);
> -       vma->rb_subtree_gap = 0;
> -       vma_gap_update(vma);
> -       vma_rb_insert(vma, &mm->mm_rb);
> -}
> -
>  static void __vma_link_file(struct vm_area_struct *vma)
>  {
>         struct file *file;
> @@ -780,19 +571,8 @@ static inline void vma_mt_store(struct mm_struct *mm, struct vm_area_struct *vma
>                 GFP_KERNEL);
>  }
>
> -static void
> -__vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> -       struct vm_area_struct *prev, struct rb_node **rb_link,
> -       struct rb_node *rb_parent)
> -{
> -       vma_mt_store(mm, vma);
> -       __vma_link_list(mm, vma, prev);
> -       __vma_link_rb(mm, vma, rb_link, rb_parent);
> -}
> -
>  static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> -                       struct vm_area_struct *prev, struct rb_node **rb_link,
> -                       struct rb_node *rb_parent)
> +                       struct vm_area_struct *prev)
>  {
>         struct address_space *mapping = NULL;
>
> @@ -801,7 +581,8 @@ static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
>                 i_mmap_lock_write(mapping);
>         }
>
> -       __vma_link(mm, vma, prev, rb_link, rb_parent);
> +       vma_mt_store(mm, vma);
> +       __vma_link_list(mm, vma, prev);
>         __vma_link_file(vma);
>
>         if (mapping)
> @@ -813,30 +594,18 @@ static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
>
>  /*
>   * Helper for vma_adjust() in the split_vma insert case: insert a vma into the
> - * mm's list and rbtree.  It has already been inserted into the interval tree.
> + * mm's list and the mm tree.  It has already been inserted into the interval tree.
>   */
>  static void __insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
>  {
>         struct vm_area_struct *prev;
> -       struct rb_node **rb_link, *rb_parent;
>
> -       if (find_vma_links(mm, vma->vm_start, vma->vm_end,
> -                          &prev, &rb_link, &rb_parent))
> -               BUG();
> -       __vma_link(mm, vma, prev, rb_link, rb_parent);
> +       BUG_ON(range_has_overlap(mm, vma->vm_start, vma->vm_end, &prev));
> +       vma_mt_store(mm, vma);
> +       __vma_link_list(mm, vma, prev);
>         mm->map_count++;
>  }
>
> -static __always_inline void __vma_unlink(struct mm_struct *mm,
> -                                               struct vm_area_struct *vma,
> -                                               struct vm_area_struct *ignore)
> -{
> -       vma_rb_erase_ignore(vma, &mm->mm_rb, ignore);
> -       __vma_unlink_list(mm, vma);
> -       /* Kill the cache */
> -       vmacache_invalidate(mm);
> -}
> -
>  /*
>   * We cannot adjust vm_start, vm_end, vm_pgoff fields of a vma that
>   * is already present in an i_mmap tree without adjusting the tree.
> @@ -854,13 +623,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>         struct rb_root_cached *root = NULL;
>         struct anon_vma *anon_vma = NULL;
>         struct file *file = vma->vm_file;
> -       bool start_changed = false, end_changed = false;
> +       bool vma_changed = false;
>         long adjust_next = 0;
>         int remove_next = 0;
>
> -       validate_mm(mm);
> -       validate_mm_mt(mm);
> -
>         if (next && !insert) {
>                 struct vm_area_struct *exporter = NULL, *importer = NULL;
>
> @@ -986,21 +752,23 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>         }
>
>         if (start != vma->vm_start) {
> -               unsigned long old_start = vma->vm_start;
> +               if (vma->vm_start < start)
> +                       vma_mt_szero(mm, vma->vm_start, start);
> +               else
> +                       vma_changed = true;
>                 vma->vm_start = start;
> -               if (old_start < start)
> -                       vma_mt_szero(mm, old_start, start);
> -               start_changed = true;
>         }
>         if (end != vma->vm_end) {
> -               unsigned long old_end = vma->vm_end;
> +               if (vma->vm_end > end)
> +                       vma_mt_szero(mm, end, vma->vm_end);
> +               else
> +                       vma_changed = true;
>                 vma->vm_end = end;
> -               if (old_end > end)
> -                       vma_mt_szero(mm, end, old_end);
> -               end_changed = true;
> +               if (!next)
> +                       mm->highest_vm_end = vm_end_gap(vma);
>         }
>
> -       if (end_changed || start_changed)
> +       if (vma_changed)
>                 vma_mt_store(mm, vma);
>
>         vma->vm_pgoff = pgoff;
> @@ -1018,25 +786,9 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>         }
>
>         if (remove_next) {
> -               /*
> -                * vma_merge has merged next into vma, and needs
> -                * us to remove next before dropping the locks.
> -                * Since we have expanded over this vma, the maple tree will
> -                * have overwritten by storing the value
> -                */
> -               if (remove_next != 3)
> -                       __vma_unlink(mm, next, next);
> -               else
> -                       /*
> -                        * vma is not before next if they've been
> -                        * swapped.
> -                        *
> -                        * pre-swap() next->vm_start was reduced so
> -                        * tell validate_mm_rb to ignore pre-swap()
> -                        * "next" (which is stored in post-swap()
> -                        * "vma").
> -                        */
> -                       __vma_unlink(mm, next, vma);
> +               __vma_unlink_list(mm, next);
> +               /* Kill the cache */
> +               vmacache_invalidate(mm);
>                 if (file)
>                         __remove_shared_vm_struct(next, file, mapping);
>         } else if (insert) {
> @@ -1046,15 +798,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>                  * (it may either follow vma or precede it).
>                  */
>                 __insert_vm_struct(mm, insert);
> -       } else {
> -               if (start_changed)
> -                       vma_gap_update(vma);
> -               if (end_changed) {
> -                       if (!next)
> -                               mm->highest_vm_end = vm_end_gap(vma);
> -                       else if (!adjust_next)
> -                               vma_gap_update(next);
> -               }
>         }
>
>         if (anon_vma) {
> @@ -1112,10 +855,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>                         remove_next = 1;
>                         end = next->vm_end;
>                         goto again;
> -               }
> -               else if (next)
> -                       vma_gap_update(next);
> -               else {
> +               } else if (!next) {
>                         /*
>                          * If remove_next == 2 we obviously can't
>                          * reach this path.
> @@ -1142,8 +882,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
>                 uprobe_mmap(insert);
>
>         validate_mm(mm);
> -       validate_mm_mt(mm);
> -
>         return 0;
>  }
>
> @@ -1290,7 +1028,6 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,
>         struct vm_area_struct *area, *next;
>         int err;
>
> -       validate_mm_mt(mm);
>         /*
>          * We later require that vma->vm_flags == vm_flags,
>          * so this tests vma->vm_flags & VM_SPECIAL, too.
> @@ -1366,7 +1103,6 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,
>                 khugepaged_enter_vma_merge(area, vm_flags);
>                 return area;
>         }
> -       validate_mm_mt(mm);
>
>         return NULL;
>  }
> @@ -1536,6 +1272,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
>         vm_flags_t vm_flags;
>         int pkey = 0;
>
> +       validate_mm(mm);
>         *populate = 0;
>
>         if (!len)
> @@ -1856,10 +1593,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
>         struct mm_struct *mm = current->mm;
>         struct vm_area_struct *vma, *prev, *merge;
>         int error;
> -       struct rb_node **rb_link, *rb_parent;
>         unsigned long charged = 0;
>
> -       validate_mm_mt(mm);
>         /* Check against address space limit. */
>         if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) {
>                 unsigned long nr_pages;
> @@ -1875,8 +1610,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
>                         return -ENOMEM;
>         }
>
> -       /* Clear old maps, set up prev, rb_link, rb_parent, and uf */
> -       if (munmap_vma_range(mm, addr, len, &prev, &rb_link, &rb_parent, uf))
> +       /* Clear old maps, set up prev and uf */
> +       if (munmap_vma_range(mm, addr, len, &prev, uf))
>                 return -ENOMEM;
>         /*
>          * Private writable mapping: check memory availability
> @@ -1984,7 +1719,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
>                         goto free_vma;
>         }
>
> -       vma_link(mm, vma, prev, rb_link, rb_parent);
> +       vma_link(mm, vma, prev);
>         /* Once vma denies write, undo our temporary denial count */
>         if (file) {
>  unmap_writable:
> @@ -2021,7 +1756,6 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
>
>         vma_set_page_prot(vma);
>
> -       validate_mm_mt(mm);
>         return addr;
>
>  unmap_and_free_vma:
> @@ -2041,7 +1775,6 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
>  unacct_error:
>         if (charged)
>                 vm_unacct_memory(charged);
> -       validate_mm_mt(mm);
>         return error;
>  }
>
> @@ -2324,9 +2057,6 @@ find_vma_prev(struct mm_struct *mm, unsigned long addr,
>
>         rcu_read_lock();
>         vma = mas_find(&mas, ULONG_MAX);
> -       if (!vma)
> -               mas_reset(&mas);
> -
>         *pprev = mas_prev(&mas, 0);
>         rcu_read_unlock();
>         return vma;
> @@ -2390,7 +2120,6 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
>         unsigned long gap_addr;
>         int error = 0;
>
> -       validate_mm_mt(mm);
>         if (!(vma->vm_flags & VM_GROWSUP))
>                 return -EFAULT;
>
> @@ -2437,15 +2166,13 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
>                         error = acct_stack_growth(vma, size, grow);
>                         if (!error) {
>                                 /*
> -                                * vma_gap_update() doesn't support concurrent
> -                                * updates, but we only hold a shared mmap_lock
> -                                * lock here, so we need to protect against
> -                                * concurrent vma expansions.
> -                                * anon_vma_lock_write() doesn't help here, as
> -                                * we don't guarantee that all growable vmas
> -                                * in a mm share the same root anon vma.
> -                                * So, we reuse mm->page_table_lock to guard
> -                                * against concurrent vma expansions.
> +                                * We only hold a shared mmap_lock lock here, so
> +                                * we need to protect against concurrent vma
> +                                * expansions.  anon_vma_lock_write() doesn't
> +                                * help here, as we don't guarantee that all
> +                                * growable vmas in a mm share the same root
> +                                * anon vma.  So, we reuse mm->page_table_lock
> +                                * to guard against concurrent vma expansions.
>                                  */
>                                 spin_lock(&mm->page_table_lock);
>                                 if (vma->vm_flags & VM_LOCKED)
> @@ -2453,10 +2180,9 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
>                                 vm_stat_account(mm, vma->vm_flags, grow);
>                                 anon_vma_interval_tree_pre_update_vma(vma);
>                                 vma->vm_end = address;
> +                               vma_mt_store(mm, vma);
>                                 anon_vma_interval_tree_post_update_vma(vma);
> -                               if (vma->vm_next)
> -                                       vma_gap_update(vma->vm_next);
> -                               else
> +                               if (!vma->vm_next)
>                                         mm->highest_vm_end = vm_end_gap(vma);
>                                 spin_unlock(&mm->page_table_lock);
>
> @@ -2466,8 +2192,6 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
>         }
>         anon_vma_unlock_write(vma->anon_vma);
>         khugepaged_enter_vma_merge(vma, vma->vm_flags);
> -       validate_mm(mm);
> -       validate_mm_mt(mm);
>         return error;
>  }
>  #endif /* CONFIG_STACK_GROWSUP || CONFIG_IA64 */
> @@ -2475,14 +2199,12 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
>  /*
>   * vma is the first one with address < vma->vm_start.  Have to extend vma.
>   */
> -int expand_downwards(struct vm_area_struct *vma,
> -                                  unsigned long address)
> +int expand_downwards(struct vm_area_struct *vma, unsigned long address)
>  {
>         struct mm_struct *mm = vma->vm_mm;
>         struct vm_area_struct *prev;
>         int error = 0;
>
> -       validate_mm(mm);
>         address &= PAGE_MASK;
>         if (address < mmap_min_addr)
>                 return -EPERM;
> @@ -2519,15 +2241,13 @@ int expand_downwards(struct vm_area_struct *vma,
>                         error = acct_stack_growth(vma, size, grow);
>                         if (!error) {
>                                 /*
> -                                * vma_gap_update() doesn't support concurrent
> -                                * updates, but we only hold a shared mmap_lock
> -                                * lock here, so we need to protect against
> -                                * concurrent vma expansions.
> -                                * anon_vma_lock_write() doesn't help here, as
> -                                * we don't guarantee that all growable vmas
> -                                * in a mm share the same root anon vma.
> -                                * So, we reuse mm->page_table_lock to guard
> -                                * against concurrent vma expansions.
> +                                * We only hold a shared mmap_lock lock here, so
> +                                * we need to protect against concurrent vma
> +                                * expansions.  anon_vma_lock_write() doesn't
> +                                * help here, as we don't guarantee that all
> +                                * growable vmas in a mm share the same root
> +                                * anon vma.  So, we reuse mm->page_table_lock
> +                                * to guard against concurrent vma expansions.
>                                  */
>                                 spin_lock(&mm->page_table_lock);
>                                 if (vma->vm_flags & VM_LOCKED)
> @@ -2539,7 +2259,6 @@ int expand_downwards(struct vm_area_struct *vma,
>                                 /* Overwrite old entry in mtree. */
>                                 vma_mt_store(mm, vma);
>                                 anon_vma_interval_tree_post_update_vma(vma);
> -                               vma_gap_update(vma);
>                                 spin_unlock(&mm->page_table_lock);
>
>                                 perf_event_mmap(vma);
> @@ -2548,7 +2267,6 @@ int expand_downwards(struct vm_area_struct *vma,
>         }
>         anon_vma_unlock_write(vma->anon_vma);
>         khugepaged_enter_vma_merge(vma, vma->vm_flags);
> -       validate_mm(mm);
>         return error;
>  }
>
> @@ -2681,16 +2399,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,
>         vma->vm_prev = NULL;
>         vma_mt_szero(mm, vma->vm_start, end);
>         do {
> -               vma_rb_erase(vma, &mm->mm_rb);
>                 mm->map_count--;
>                 tail_vma = vma;
>                 vma = vma->vm_next;
>         } while (vma && vma->vm_start < end);
>         *insertion_point = vma;
> -       if (vma) {
> +       if (vma)
>                 vma->vm_prev = prev;
> -               vma_gap_update(vma);
> -       } else
> +       else
>                 mm->highest_vm_end = prev ? vm_end_gap(prev) : 0;
>         tail_vma->vm_next = NULL;
>
> @@ -2821,11 +2537,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
>         if (len == 0)
>                 return -EINVAL;
>
> -       /*
> -        * arch_unmap() might do unmaps itself.  It must be called
> -        * and finish any rbtree manipulation before this code
> -        * runs and also starts to manipulate the rbtree.
> -        */
> +        /* arch_unmap() might do unmaps itself.  */
>         arch_unmap(mm, start, end);
>
>         /* Find the first overlapping VMA */
> @@ -2833,7 +2545,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
>         if (!vma)
>                 return 0;
>         prev = vma->vm_prev;
> -       /* we have  start < vma->vm_end  */
> +       /* we have start < vma->vm_end  */
>
>         /* if it doesn't overlap, we have nothing.. */
>         if (vma->vm_start >= end)
> @@ -2893,7 +2605,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
>         if (mm->locked_vm)
>                 unlock_range(vma, end);
>
> -       /* Detach vmas from rbtree */
> +       /* Detach vmas from the MM linked list and remove from the mm tree*/
>         if (!detach_vmas_to_be_unmapped(mm, vma, prev, end))
>                 downgrade = false;
>
> @@ -3041,11 +2753,11 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
>   *  anonymous maps.  eventually we may be able to do some
>   *  brk-specific accounting here.
>   */
> -static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long flags, struct list_head *uf)
> +static int do_brk_flags(unsigned long addr, unsigned long len,
> +                       unsigned long flags, struct list_head *uf)
>  {
>         struct mm_struct *mm = current->mm;
>         struct vm_area_struct *vma, *prev;
> -       struct rb_node **rb_link, *rb_parent;
>         pgoff_t pgoff = addr >> PAGE_SHIFT;
>         int error;
>         unsigned long mapped_addr;
> @@ -3064,8 +2776,8 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
>         if (error)
>                 return error;
>
> -       /* Clear old maps, set up prev, rb_link, rb_parent, and uf */
> -       if (munmap_vma_range(mm, addr, len, &prev, &rb_link, &rb_parent, uf))
> +       /* Clear old maps, set up prev and uf */
> +       if (munmap_vma_range(mm, addr, len, &prev, uf))
>                 return -ENOMEM;
>
>         /* Check against address space limits *after* clearing old maps... */
> @@ -3099,7 +2811,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
>         vma->vm_pgoff = pgoff;
>         vma->vm_flags = flags;
>         vma->vm_page_prot = vm_get_page_prot(flags);
> -       vma_link(mm, vma, prev, rb_link, rb_parent);
> +       vma_link(mm, vma, prev);
>  out:
>         perf_event_mmap(vma);
>         mm->total_vm += len >> PAGE_SHIFT;
> @@ -3219,26 +2931,10 @@ void exit_mmap(struct mm_struct *mm)
>  int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
>  {
>         struct vm_area_struct *prev;
> -       struct rb_node **rb_link, *rb_parent;
> -       unsigned long start = vma->vm_start;
> -       struct vm_area_struct *overlap = NULL;
>
> -       if (find_vma_links(mm, vma->vm_start, vma->vm_end,
> -                          &prev, &rb_link, &rb_parent))
> +       if (range_has_overlap(mm, vma->vm_start, vma->vm_end, &prev))
>                 return -ENOMEM;
>
> -       overlap = mt_find(&mm->mm_mt, &start, vma->vm_end - 1);
> -       if (overlap) {
> -
> -               pr_err("Found vma ending at %lu\n", start - 1);
> -               pr_err("vma : %lu => %lu-%lu\n", (unsigned long)overlap,
> -                               overlap->vm_start, overlap->vm_end - 1);
> -#if defined(CONFIG_DEBUG_MAPLE_TREE)
> -               mt_dump(&mm->mm_mt);
> -#endif
> -               BUG();
> -       }
> -
>         if ((vma->vm_flags & VM_ACCOUNT) &&
>              security_vm_enough_memory_mm(mm, vma_pages(vma)))
>                 return -ENOMEM;
> @@ -3260,7 +2956,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
>                 vma->vm_pgoff = vma->vm_start >> PAGE_SHIFT;
>         }
>
> -       vma_link(mm, vma, prev, rb_link, rb_parent);
> +       vma_link(mm, vma, prev);
>         return 0;
>  }
>
> @@ -3276,9 +2972,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
>         unsigned long vma_start = vma->vm_start;
>         struct mm_struct *mm = vma->vm_mm;
>         struct vm_area_struct *new_vma, *prev;
> -       struct rb_node **rb_link, *rb_parent;
>         bool faulted_in_anon_vma = true;
> -       unsigned long index = addr;
>
>         validate_mm_mt(mm);
>         /*
> @@ -3290,10 +2984,9 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
>                 faulted_in_anon_vma = false;
>         }
>
> -       if (find_vma_links(mm, addr, addr + len, &prev, &rb_link, &rb_parent))
> +       if (range_has_overlap(mm, addr, addr + len, &prev))
>                 return NULL;    /* should never get here */
> -       if (mt_find(&mm->mm_mt, &index, addr+len - 1))
> -               BUG();
> +
>         new_vma = vma_merge(mm, prev, addr, addr + len, vma->vm_flags,
>                             vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma),
>                             vma->vm_userfaultfd_ctx);
> @@ -3334,7 +3027,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
>                         get_file(new_vma->vm_file);
>                 if (new_vma->vm_ops && new_vma->vm_ops->open)
>                         new_vma->vm_ops->open(new_vma);
> -               vma_link(mm, new_vma, prev, rb_link, rb_parent);
> +               vma_link(mm, new_vma, prev);
>                 *need_rmap_locks = false;
>         }
>         validate_mm_mt(mm);
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 8848cf7cb7c1..c410f99203fb 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -566,13 +566,14 @@ static void put_nommu_region(struct vm_region *region)
>   */
>  static void add_vma_to_mm(struct mm_struct *mm, struct vm_area_struct *vma)
>  {
> -       struct vm_area_struct *pvma, *prev;
>         struct address_space *mapping;
> -       struct rb_node **p, *parent, *rb_prev;
> +       struct vm_area_struct *prev;
> +       MA_STATE(mas, &mm->mm_mt, vma->vm_start, vma->vm_end);
>
>         BUG_ON(!vma->vm_region);
>
>         mm->map_count++;
> +       printk("mm at %u\n", mm->map_count);

I think this was added while debugging and should be removed?

>         vma->vm_mm = mm;
>
>         /* add the VMA to the mapping */
> @@ -586,42 +587,12 @@ static void add_vma_to_mm(struct mm_struct *mm, struct vm_area_struct *vma)
>                 i_mmap_unlock_write(mapping);
>         }
>
> +       rcu_read_lock();
> +       prev = mas_prev(&mas, 0);
> +       rcu_read_unlock();
> +       mas_reset(&mas);
>         /* add the VMA to the tree */
> -       parent = rb_prev = NULL;
> -       p = &mm->mm_rb.rb_node;
> -       while (*p) {
> -               parent = *p;
> -               pvma = rb_entry(parent, struct vm_area_struct, vm_rb);
> -
> -               /* sort by: start addr, end addr, VMA struct addr in that order
> -                * (the latter is necessary as we may get identical VMAs) */
> -               if (vma->vm_start < pvma->vm_start)
> -                       p = &(*p)->rb_left;
> -               else if (vma->vm_start > pvma->vm_start) {
> -                       rb_prev = parent;
> -                       p = &(*p)->rb_right;
> -               } else if (vma->vm_end < pvma->vm_end)
> -                       p = &(*p)->rb_left;
> -               else if (vma->vm_end > pvma->vm_end) {
> -                       rb_prev = parent;
> -                       p = &(*p)->rb_right;
> -               } else if (vma < pvma)
> -                       p = &(*p)->rb_left;
> -               else if (vma > pvma) {
> -                       rb_prev = parent;
> -                       p = &(*p)->rb_right;
> -               } else
> -                       BUG();
> -       }
> -
> -       rb_link_node(&vma->vm_rb, parent, p);
> -       rb_insert_color(&vma->vm_rb, &mm->mm_rb);
> -
> -       /* add VMA to the VMA list also */
> -       prev = NULL;
> -       if (rb_prev)
> -               prev = rb_entry(rb_prev, struct vm_area_struct, vm_rb);
> -
> +       vma_mas_store(vma, &mas);
>         __vma_link_list(mm, vma, prev);
>  }
>
> @@ -634,6 +605,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
>         struct address_space *mapping;
>         struct mm_struct *mm = vma->vm_mm;
>         struct task_struct *curr = current;
> +       MA_STATE(mas, &vma->vm_mm->mm_mt, 0, 0);
>
>         mm->map_count--;
>         for (i = 0; i < VMACACHE_SIZE; i++) {
> @@ -643,7 +615,6 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
>                         break;
>                 }
>         }
> -
>         /* remove the VMA from the mapping */
>         if (vma->vm_file) {
>                 mapping = vma->vm_file->f_mapping;
> @@ -656,8 +627,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
>         }
>
>         /* remove from the MM's tree and list */
> -       rb_erase(&vma->vm_rb, &mm->mm_rb);
> -
> +       vma_mas_remove(vma, &mas);
>         __vma_unlink_list(mm, vma);
>  }
>
> @@ -681,24 +651,21 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma)
>  struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
>  {
>         struct vm_area_struct *vma;
> +       MA_STATE(mas, &mm->mm_mt, addr, addr);
>
>         /* check the cache first */
>         vma = vmacache_find(mm, addr);
>         if (likely(vma))
>                 return vma;
>
> -       /* trawl the list (there may be multiple mappings in which addr
> -        * resides) */
> -       for (vma = mm->mmap; vma; vma = vma->vm_next) {
> -               if (vma->vm_start > addr)
> -                       return NULL;
> -               if (vma->vm_end > addr) {
> -                       vmacache_update(addr, vma);
> -                       return vma;
> -               }
> -       }
> +       rcu_read_lock();
> +       vma = mas_walk(&mas);
> +       rcu_read_unlock();
>
> -       return NULL;
> +       if (vma)
> +               vmacache_update(addr, vma);
> +
> +       return vma;
>  }
>  EXPORT_SYMBOL(find_vma);
>
> @@ -730,26 +697,25 @@ static struct vm_area_struct *find_vma_exact(struct mm_struct *mm,
>  {
>         struct vm_area_struct *vma;
>         unsigned long end = addr + len;
> +       MA_STATE(mas, &mm->mm_mt, addr, addr);
>
>         /* check the cache first */
>         vma = vmacache_find_exact(mm, addr, end);
>         if (vma)
>                 return vma;
>
> -       /* trawl the list (there may be multiple mappings in which addr
> -        * resides) */
> -       for (vma = mm->mmap; vma; vma = vma->vm_next) {
> -               if (vma->vm_start < addr)
> -                       continue;
> -               if (vma->vm_start > addr)
> -                       return NULL;
> -               if (vma->vm_end == end) {
> -                       vmacache_update(addr, vma);
> -                       return vma;
> -               }
> -       }
> -
> -       return NULL;
> +       rcu_read_lock();
> +       vma = mas_walk(&mas);
> +       rcu_read_unlock();
> +       if (!vma)
> +               return NULL;
> +       if (vma->vm_start != addr)
> +               return NULL;
> +       if (vma->vm_end != end)
> +               return NULL;
> +
> +       vmacache_update(addr, vma);
> +       return vma;
>  }
>
>  /*
> diff --git a/mm/util.c b/mm/util.c
> index 0b6dd9d81da7..35deaa0ccac5 100644
> --- a/mm/util.c
> +++ b/mm/util.c
> @@ -287,6 +287,8 @@ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,
>         vma->vm_next = next;
>         if (next)
>                 next->vm_prev = vma;
> +       else
> +               mm->highest_vm_end = vm_end_gap(vma);
>  }
>
>  void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma)
> @@ -301,6 +303,12 @@ void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma)
>                 mm->mmap = next;
>         if (next)
>                 next->vm_prev = prev;
> +       else {
> +               if (prev)
> +                       mm->highest_vm_end = vm_end_gap(prev);
> +               else
> +                       mm->highest_vm_end = 0;
> +       }
>  }
>
>  /* Check if the vma is being used as a stack by this task */
> --
> 2.30.2


From liam.howlett at oracle.com  Mon May 31 06:40:18 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 31 May 2021 13:40:18 -0000
Subject: [PATCH 32/94] kernel/fork: Convert dup_mmap to use maple tree
In-Reply-To: <CAJuCfpEBnrTShvYN9G9PTHjBWnEBeibuMwNOER66r1TdqBWNzg@mail.gmail.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-33-Liam.Howlett@Oracle.com>
 <CAJuCfpEBnrTShvYN9G9PTHjBWnEBeibuMwNOER66r1TdqBWNzg@mail.gmail.com>
Message-ID: <20210531131341.c7weinlwxnhnnaqt@revolver>

* Suren Baghdasaryan <surenb at google.com> [210528 20:42]:
> On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
> >
> > Use the maple tree iterator to duplicate the mm_struct trees.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> >  include/linux/mm.h       |  2 --
> >  include/linux/sched/mm.h |  3 +++
> >  kernel/fork.c            | 24 +++++++++++++++++++-----
> >  mm/mmap.c                |  4 ----
> >  4 files changed, 22 insertions(+), 11 deletions(-)
> >
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index e89bacfa9145..7f7dff6ad884 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -2498,8 +2498,6 @@ extern bool arch_has_descending_max_zone_pfns(void);
> >  /* nommu.c */
> >  extern atomic_long_t mmap_pages_allocated;
> >  extern int nommu_shrink_inode_mappings(struct inode *, size_t, size_t);
> > -/* maple_tree */
> > -void vma_store(struct mm_struct *mm, struct vm_area_struct *vma);
> >
> >  /* interval_tree.c */
> >  void vma_interval_tree_insert(struct vm_area_struct *node,
> > diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h
> > index e24b1fe348e3..76cab3aea6ab 100644
> > --- a/include/linux/sched/mm.h
> > +++ b/include/linux/sched/mm.h
> > @@ -8,6 +8,7 @@
> >  #include <linux/mm_types.h>
> >  #include <linux/gfp.h>
> >  #include <linux/sync_core.h>
> > +#include <linux/maple_tree.h>
> >
> >  /*
> >   * Routines for handling mm_structs
> > @@ -67,11 +68,13 @@ static inline void mmdrop(struct mm_struct *mm)
> >   */
> >  static inline void mmget(struct mm_struct *mm)
> >  {
> > +       mt_set_in_rcu(&mm->mm_mt);
> >         atomic_inc(&mm->mm_users);
> >  }
> >
> >  static inline bool mmget_not_zero(struct mm_struct *mm)
> >  {
> > +       mt_set_in_rcu(&mm->mm_mt);
> 
> Should you be calling mt_set_in_rcu() if atomic_inc_not_zero() failed?
> I don't think mmput() is called after mmget_not_zero() fails and
> mt_clear_in_rcu() will not be called.

Good catch, but having it the way it is will be faster with the
possibility of re-entering RCU mode if there is a race during tear down.
Entering RCU mode during tear-down mean that the nodes that are not
already freed would remain for an RCU cycle before being freed.  I don't
think it is worth checking every time this is called for such a low
payoff.  I should probably add a comment about this though.

> 
> >         return atomic_inc_not_zero(&mm->mm_users);
> >  }
> >
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index c37abaf28eb9..832416ff613e 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -477,7 +477,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >         struct vm_area_struct *mpnt, *tmp, *prev, **pprev;
> >         struct rb_node **rb_link, *rb_parent;
> >         int retval;
> > -       unsigned long charge;
> > +       unsigned long charge = 0;
> > +       MA_STATE(old_mas, &oldmm->mm_mt, 0, 0);
> > +       MA_STATE(mas, &mm->mm_mt, 0, 0);
> >         LIST_HEAD(uf);
> >
> >         uprobe_start_dup_mmap();
> > @@ -511,7 +513,13 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                 goto out;
> >
> >         prev = NULL;
> > -       for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {
> > +
> > +       retval = mas_entry_count(&mas, oldmm->map_count);
> > +       if (retval)
> > +               goto fail_nomem;
> > +
> > +       rcu_read_lock();
> > +       mas_for_each(&old_mas, mpnt, ULONG_MAX) {
> >                 struct file *file;
> >
> >                 if (mpnt->vm_flags & VM_DONTCOPY) {
> > @@ -525,7 +533,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                  */
> >                 if (fatal_signal_pending(current)) {
> >                         retval = -EINTR;
> > -                       goto out;
> > +                       goto loop_out;
> >                 }
> >                 if (mpnt->vm_flags & VM_ACCOUNT) {
> >                         unsigned long len = vma_pages(mpnt);
> > @@ -594,7 +602,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                 rb_parent = &tmp->vm_rb;
> >
> >                 /* Link the vma into the MT */
> > -               vma_store(mm, tmp);
> > +               mas.index = tmp->vm_start;
> > +               mas.last = tmp->vm_end - 1;
> > +               mas_store(&mas, tmp);
> >
> >                 mm->map_count++;
> >                 if (!(tmp->vm_flags & VM_WIPEONFORK))
> > @@ -604,14 +614,17 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                         tmp->vm_ops->open(tmp);
> >
> >                 if (retval)
> > -                       goto out;
> > +                       goto loop_out;
> >         }
> >         /* a new mm has just been created */
> >         retval = arch_dup_mmap(oldmm, mm);
> > +loop_out:
> >  out:
> > +       rcu_read_unlock();
> >         mmap_write_unlock(mm);
> >         flush_tlb_mm(oldmm);
> >         mmap_write_unlock(oldmm);
> > +       mas_destroy(&mas);
> >         dup_userfaultfd_complete(&uf);
> >  fail_uprobe_end:
> >         uprobe_end_dup_mmap();
> > @@ -1092,6 +1105,7 @@ static inline void __mmput(struct mm_struct *mm)
> >  {
> >         VM_BUG_ON(atomic_read(&mm->mm_users));
> >
> > +       mt_clear_in_rcu(&mm->mm_mt);
> >         uprobe_clear_state(mm);
> >         exit_aio(mm);
> >         ksm_exit(mm);
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index 929c2f9eb3f5..1bd43f4db28e 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -780,10 +780,6 @@ static inline void vma_mt_store(struct mm_struct *mm, struct vm_area_struct *vma
> >                 GFP_KERNEL);
> >  }
> >
> > -void vma_store(struct mm_struct *mm, struct vm_area_struct *vma) {
> > -       vma_mt_store(mm, vma);
> > -}
> > -
> >  static void
> >  __vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> >         struct vm_area_struct *prev, struct rb_node **rb_link,
> > --
> > 2.30.2

From liam.howlett at oracle.com  Mon May 31 06:42:47 2021
From: liam.howlett at oracle.com (Liam Howlett)
Date: Mon, 31 May 2021 13:42:47 -0000
Subject: [PATCH 33/94] mm: Remove rb tree.
In-Reply-To: <CAJuCfpGD3EC_chwHWBETSJ9x7DR6umX6kFZES0sWyphtVNeNRw@mail.gmail.com>
References: <20210428153542.2814175-1-Liam.Howlett@Oracle.com>
 <20210428153542.2814175-34-Liam.Howlett@Oracle.com>
 <CAJuCfpGD3EC_chwHWBETSJ9x7DR6umX6kFZES0sWyphtVNeNRw@mail.gmail.com>
Message-ID: <20210531134157.gtue3q44gdra5oev@revolver>

* Suren Baghdasaryan <surenb at google.com> [210528 21:26]:
> On Wed, Apr 28, 2021 at 8:36 AM Liam Howlett <liam.howlett at oracle.com> wrote:
> >
> > Remove the RB tree and start using the maple tree for vm_area_struct
> > tracking.
> >
> > Drop validate_mm() calls in expand_upwards() and expand_downwards() as
> > the lock is not held.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> > ---
> >  arch/x86/kernel/tboot.c    |   1 -
> >  drivers/firmware/efi/efi.c |   1 -
> >  fs/proc/task_nommu.c       |  55 ++--
> >  include/linux/mm.h         |   4 +-
> >  include/linux/mm_types.h   |  26 +-
> >  kernel/fork.c              |   8 -
> >  mm/init-mm.c               |   2 -
> >  mm/mmap.c                  | 525 ++++++++-----------------------------
> >  mm/nommu.c                 |  96 +++----
> >  mm/util.c                  |   8 +
> >  10 files changed, 185 insertions(+), 541 deletions(-)
> >
> > diff --git a/arch/x86/kernel/tboot.c b/arch/x86/kernel/tboot.c
> > index 6f978f722dff..121f28bb2209 100644
> > --- a/arch/x86/kernel/tboot.c
> > +++ b/arch/x86/kernel/tboot.c
> > @@ -97,7 +97,6 @@ void __init tboot_probe(void)
> >
> >  static pgd_t *tboot_pg_dir;
> >  static struct mm_struct tboot_mm = {
> > -       .mm_rb          = RB_ROOT,
> >         .mm_mt          = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
> >         .pgd            = swapper_pg_dir,
> >         .mm_users       = ATOMIC_INIT(2),
> > diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
> > index 271ae8c7bb07..8aaeaa824576 100644
> > --- a/drivers/firmware/efi/efi.c
> > +++ b/drivers/firmware/efi/efi.c
> > @@ -54,7 +54,6 @@ static unsigned long __initdata mem_reserve = EFI_INVALID_TABLE_ADDR;
> >  static unsigned long __initdata rt_prop = EFI_INVALID_TABLE_ADDR;
> >
> >  struct mm_struct efi_mm = {
> > -       .mm_rb                  = RB_ROOT,
> >         .mm_mt                  = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
> >         .mm_users               = ATOMIC_INIT(2),
> >         .mm_count               = ATOMIC_INIT(1),
> > diff --git a/fs/proc/task_nommu.c b/fs/proc/task_nommu.c
> > index a6d21fc0033c..8691a1216d1c 100644
> > --- a/fs/proc/task_nommu.c
> > +++ b/fs/proc/task_nommu.c
> > @@ -22,15 +22,13 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
> >  {
> >         struct vm_area_struct *vma;
> >         struct vm_region *region;
> > -       struct rb_node *p;
> >         unsigned long bytes = 0, sbytes = 0, slack = 0, size;
> > -
> > -       mmap_read_lock(mm);
> > -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> > -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> > +       MA_STATE(mas, &mm->mm_mt, 0, 0);
> >
> > +       mmap_read_lock(mm);
> > +       rcu_read_lock();
> > +       mas_for_each(&mas, vma, ULONG_MAX) {
> >                 bytes += kobjsize(vma);
> > -
> >                 region = vma->vm_region;
> >                 if (region) {
> >                         size = kobjsize(region);
> > @@ -53,7 +51,7 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
> >                 sbytes += kobjsize(mm);
> >         else
> >                 bytes += kobjsize(mm);
> > -
> > +
> >         if (current->fs && current->fs->users > 1)
> >                 sbytes += kobjsize(current->fs);
> >         else
> > @@ -77,20 +75,21 @@ void task_mem(struct seq_file *m, struct mm_struct *mm)
> >                 "Shared:\t%8lu bytes\n",
> >                 bytes, slack, sbytes);
> >
> > +       rcu_read_unlock();
> >         mmap_read_unlock(mm);
> >  }
> >
> >  unsigned long task_vsize(struct mm_struct *mm)
> >  {
> >         struct vm_area_struct *vma;
> > -       struct rb_node *p;
> >         unsigned long vsize = 0;
> > +       MA_STATE(mas, &mm->mm_mt, 0, 0);
> >
> >         mmap_read_lock(mm);
> > -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> > -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> > +       rcu_read_lock();
> > +       mas_for_each(&mas, vma, ULONG_MAX)
> >                 vsize += vma->vm_end - vma->vm_start;
> > -       }
> > +       rcu_read_unlock();
> >         mmap_read_unlock(mm);
> >         return vsize;
> >  }
> > @@ -101,12 +100,12 @@ unsigned long task_statm(struct mm_struct *mm,
> >  {
> >         struct vm_area_struct *vma;
> >         struct vm_region *region;
> > -       struct rb_node *p;
> >         unsigned long size = kobjsize(mm);
> > +       MA_STATE(mas, &mm->mm_mt, 0, 0);
> >
> >         mmap_read_lock(mm);
> > -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p)) {
> > -               vma = rb_entry(p, struct vm_area_struct, vm_rb);
> > +       rcu_read_lock();
> > +       mas_for_each(&mas, vma, ULONG_MAX) {
> >                 size += kobjsize(vma);
> >                 region = vma->vm_region;
> >                 if (region) {
> > @@ -119,6 +118,7 @@ unsigned long task_statm(struct mm_struct *mm,
> >                 >> PAGE_SHIFT;
> >         *data = (PAGE_ALIGN(mm->start_stack) - (mm->start_data & PAGE_MASK))
> >                 >> PAGE_SHIFT;
> > +       rcu_read_unlock();
> >         mmap_read_unlock(mm);
> >         size >>= PAGE_SHIFT;
> >         size += *text + *data;
> > @@ -190,17 +190,20 @@ static int nommu_vma_show(struct seq_file *m, struct vm_area_struct *vma)
> >   */
> >  static int show_map(struct seq_file *m, void *_p)
> >  {
> > -       struct rb_node *p = _p;
> > -
> > -       return nommu_vma_show(m, rb_entry(p, struct vm_area_struct, vm_rb));
> > +       return nommu_vma_show(m, _p);
> >  }
> >
> >  static void *m_start(struct seq_file *m, loff_t *pos)
> >  {
> >         struct proc_maps_private *priv = m->private;
> >         struct mm_struct *mm;
> > -       struct rb_node *p;
> > -       loff_t n = *pos;
> > +       struct vm_area_struct *vma;
> > +       unsigned long addr = *pos;
> > +       MA_STATE(mas, &priv->mm->mm_mt, addr, addr);
> > +
> > +       /* See m_next(). Zero at the start or after lseek. */
> > +       if (addr == -1UL)
> > +               return NULL;
> >
> >         /* pin the task and mm whilst we play with them */
> >         priv->task = get_proc_task(priv->inode);
> > @@ -216,14 +219,12 @@ static void *m_start(struct seq_file *m, loff_t *pos)
> >                 return ERR_PTR(-EINTR);
> >         }
> >
> > -       /* start from the Nth VMA */
> > -       for (p = rb_first(&mm->mm_rb); p; p = rb_next(p))
> > -               if (n-- == 0)
> > -                       return p;
> > +       /* start the next element from addr */
> > +       vma = mas_find(&mas, ULONG_MAX);
> >
> >         mmap_read_unlock(mm);
> >         mmput(mm);
> > -       return NULL;
> > +       return vma;
> >  }
> >
> >  static void m_stop(struct seq_file *m, void *_vml)
> > @@ -242,10 +243,10 @@ static void m_stop(struct seq_file *m, void *_vml)
> >
> >  static void *m_next(struct seq_file *m, void *_p, loff_t *pos)
> >  {
> > -       struct rb_node *p = _p;
> > +       struct vm_area_struct *vma = _p;
> >
> > -       (*pos)++;
> > -       return p ? rb_next(p) : NULL;
> > +       *pos = vma->vm_end;
> > +       return vma->vm_next;
> >  }
> >
> >  static const struct seq_operations proc_pid_maps_ops = {
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index 7f7dff6ad884..146976070fed 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -2553,8 +2553,6 @@ extern int __split_vma(struct mm_struct *, struct vm_area_struct *,
> >  extern int split_vma(struct mm_struct *, struct vm_area_struct *,
> >         unsigned long addr, int new_below);
> >  extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);
> > -extern void __vma_link_rb(struct mm_struct *, struct vm_area_struct *,
> > -       struct rb_node **, struct rb_node *);
> >  extern void unlink_file_vma(struct vm_area_struct *);
> >  extern struct vm_area_struct *copy_vma(struct vm_area_struct **,
> >         unsigned long addr, unsigned long len, pgoff_t pgoff,
> > @@ -2699,7 +2697,7 @@ static inline struct vm_area_struct * find_vma_intersection(struct mm_struct * m
> >  static inline
> >  struct vm_area_struct *vma_lookup(struct mm_struct *mm, unsigned long addr)
> >  {
> > -       return find_vma_intersection(mm, addr, addr + 1);
> > +       return mtree_load(&mm->mm_mt, addr);
> >  }
> >
> >  static inline unsigned long vm_start_gap(struct vm_area_struct *vma)
> > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> > index 51733fc44daf..41551bfa6ce0 100644
> > --- a/include/linux/mm_types.h
> > +++ b/include/linux/mm_types.h
> > @@ -311,19 +311,6 @@ struct vm_area_struct {
> >
> >         /* linked list of VM areas per task, sorted by address */
> >         struct vm_area_struct *vm_next, *vm_prev;
> > -
> > -       struct rb_node vm_rb;
> > -
> > -       /*
> > -        * Largest free memory gap in bytes to the left of this VMA.
> > -        * Either between this VMA and vma->vm_prev, or between one of the
> > -        * VMAs below us in the VMA rbtree and its ->vm_prev. This helps
> > -        * get_unmapped_area find a free area of the right size.
> > -        */
> > -       unsigned long rb_subtree_gap;
> > -
> > -       /* Second cache line starts here. */
> > -
> >         struct mm_struct *vm_mm;        /* The address space we belong to. */
> >
> >         /*
> > @@ -333,6 +320,12 @@ struct vm_area_struct {
> >         pgprot_t vm_page_prot;
> >         unsigned long vm_flags;         /* Flags, see mm.h. */
> >
> > +       /* Information about our backing store: */
> > +       unsigned long vm_pgoff;         /* Offset (within vm_file) in PAGE_SIZE
> > +                                        * units
> > +                                        */
> > +       /* Second cache line starts here. */
> > +       struct file *vm_file;           /* File we map to (can be NULL). */
> >         /*
> >          * For areas with an address space and backing store,
> >          * linkage into the address_space->i_mmap interval tree.
> > @@ -351,16 +344,14 @@ struct vm_area_struct {
> >         struct list_head anon_vma_chain; /* Serialized by mmap_lock &
> >                                           * page_table_lock */
> >         struct anon_vma *anon_vma;      /* Serialized by page_table_lock */
> > +       /* Third cache line starts here. */
> >
> >         /* Function pointers to deal with this struct. */
> >         const struct vm_operations_struct *vm_ops;
> >
> > -       /* Information about our backing store: */
> > -       unsigned long vm_pgoff;         /* Offset (within vm_file) in PAGE_SIZE
> > -                                          units */
> > -       struct file * vm_file;          /* File we map to (can be NULL). */
> >         void * vm_private_data;         /* was vm_pte (shared mem) */
> >
> > +
> >  #ifdef CONFIG_SWAP
> >         atomic_long_t swap_readahead_info;
> >  #endif
> > @@ -389,7 +380,6 @@ struct mm_struct {
> >         struct {
> >                 struct vm_area_struct *mmap;            /* list of VMAs */
> >                 struct maple_tree mm_mt;
> > -               struct rb_root mm_rb;
> >                 u64 vmacache_seqnum;                   /* per-thread vmacache */
> >  #ifdef CONFIG_MMU
> >                 unsigned long (*get_unmapped_area) (struct file *filp,
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index 832416ff613e..83afd3007a2b 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -475,7 +475,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                                         struct mm_struct *oldmm)
> >  {
> >         struct vm_area_struct *mpnt, *tmp, *prev, **pprev;
> > -       struct rb_node **rb_link, *rb_parent;
> >         int retval;
> >         unsigned long charge = 0;
> >         MA_STATE(old_mas, &oldmm->mm_mt, 0, 0);
> > @@ -502,8 +501,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >         mm->exec_vm = oldmm->exec_vm;
> >         mm->stack_vm = oldmm->stack_vm;
> >
> > -       rb_link = &mm->mm_rb.rb_node;
> > -       rb_parent = NULL;
> >         pprev = &mm->mmap;
> >         retval = ksm_fork(mm, oldmm);
> >         if (retval)
> > @@ -597,10 +594,6 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
> >                 tmp->vm_prev = prev;
> >                 prev = tmp;
> >
> > -               __vma_link_rb(mm, tmp, rb_link, rb_parent);
> > -               rb_link = &tmp->vm_rb.rb_right;
> > -               rb_parent = &tmp->vm_rb;
> > -
> >                 /* Link the vma into the MT */
> >                 mas.index = tmp->vm_start;
> >                 mas.last = tmp->vm_end - 1;
> > @@ -1033,7 +1026,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
> >         struct user_namespace *user_ns)
> >  {
> >         mm->mmap = NULL;
> > -       mm->mm_rb = RB_ROOT;
> >         mt_init_flags(&mm->mm_mt, MAPLE_ALLOC_RANGE);
> >         mm->vmacache_seqnum = 0;
> >         atomic_set(&mm->mm_users, 1);
> > diff --git a/mm/init-mm.c b/mm/init-mm.c
> > index 2014d4b82294..04bbe5172b72 100644
> > --- a/mm/init-mm.c
> > +++ b/mm/init-mm.c
> > @@ -1,6 +1,5 @@
> >  // SPDX-License-Identifier: GPL-2.0
> >  #include <linux/mm_types.h>
> > -#include <linux/rbtree.h>
> >  #include <linux/maple_tree.h>
> >  #include <linux/rwsem.h>
> >  #include <linux/spinlock.h>
> > @@ -28,7 +27,6 @@
> >   * and size this cpu_bitmask to NR_CPUS.
> >   */
> >  struct mm_struct init_mm = {
> > -       .mm_rb          = RB_ROOT,
> >         .mm_mt          = MTREE_INIT(mm_mt, MAPLE_ALLOC_RANGE),
> >         .pgd            = swapper_pg_dir,
> >         .mm_users       = ATOMIC_INIT(2),
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index 1bd43f4db28e..7747047c4cbe 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -38,7 +38,6 @@
> >  #include <linux/audit.h>
> >  #include <linux/khugepaged.h>
> >  #include <linux/uprobes.h>
> > -#include <linux/rbtree_augmented.h>
> >  #include <linux/notifier.h>
> >  #include <linux/memory.h>
> >  #include <linux/printk.h>
> > @@ -290,93 +289,6 @@ SYSCALL_DEFINE1(brk, unsigned long, brk)
> >         return origbrk;
> >  }
> >
> > -static inline unsigned long vma_compute_gap(struct vm_area_struct *vma)
> > -{
> > -       unsigned long gap, prev_end;
> > -
> > -       /*
> > -        * Note: in the rare case of a VM_GROWSDOWN above a VM_GROWSUP, we
> > -        * allow two stack_guard_gaps between them here, and when choosing
> > -        * an unmapped area; whereas when expanding we only require one.
> > -        * That's a little inconsistent, but keeps the code here simpler.
> > -        */
> > -       gap = vm_start_gap(vma);
> > -       if (vma->vm_prev) {
> > -               prev_end = vm_end_gap(vma->vm_prev);
> > -               if (gap > prev_end)
> > -                       gap -= prev_end;
> > -               else
> > -                       gap = 0;
> > -       }
> > -       return gap;
> > -}
> > -
> > -#ifdef CONFIG_DEBUG_VM_RB
> > -static unsigned long vma_compute_subtree_gap(struct vm_area_struct *vma)
> > -{
> > -       unsigned long max = vma_compute_gap(vma), subtree_gap;
> > -       if (vma->vm_rb.rb_left) {
> > -               subtree_gap = rb_entry(vma->vm_rb.rb_left,
> > -                               struct vm_area_struct, vm_rb)->rb_subtree_gap;
> > -               if (subtree_gap > max)
> > -                       max = subtree_gap;
> > -       }
> > -       if (vma->vm_rb.rb_right) {
> > -               subtree_gap = rb_entry(vma->vm_rb.rb_right,
> > -                               struct vm_area_struct, vm_rb)->rb_subtree_gap;
> > -               if (subtree_gap > max)
> > -                       max = subtree_gap;
> > -       }
> > -       return max;
> > -}
> > -
> > -static int browse_rb(struct mm_struct *mm)
> > -{
> > -       struct rb_root *root = &mm->mm_rb;
> > -       int i = 0, j, bug = 0;
> > -       struct rb_node *nd, *pn = NULL;
> > -       unsigned long prev = 0, pend = 0;
> > -
> > -       for (nd = rb_first(root); nd; nd = rb_next(nd)) {
> > -               struct vm_area_struct *vma;
> > -               vma = rb_entry(nd, struct vm_area_struct, vm_rb);
> > -               if (vma->vm_start < prev) {
> > -                       pr_emerg("vm_start %lx < prev %lx\n",
> > -                                 vma->vm_start, prev);
> > -                       bug = 1;
> > -               }
> > -               if (vma->vm_start < pend) {
> > -                       pr_emerg("vm_start %lx < pend %lx\n",
> > -                                 vma->vm_start, pend);
> > -                       bug = 1;
> > -               }
> > -               if (vma->vm_start > vma->vm_end) {
> > -                       pr_emerg("vm_start %lx > vm_end %lx\n",
> > -                                 vma->vm_start, vma->vm_end);
> > -                       bug = 1;
> > -               }
> > -               spin_lock(&mm->page_table_lock);
> > -               if (vma->rb_subtree_gap != vma_compute_subtree_gap(vma)) {
> > -                       pr_emerg("free gap %lx, correct %lx\n",
> > -                              vma->rb_subtree_gap,
> > -                              vma_compute_subtree_gap(vma));
> > -                       bug = 1;
> > -               }
> > -               spin_unlock(&mm->page_table_lock);
> > -               i++;
> > -               pn = nd;
> > -               prev = vma->vm_start;
> > -               pend = vma->vm_end;
> > -       }
> > -       j = 0;
> > -       for (nd = pn; nd; nd = rb_prev(nd))
> > -               j++;
> > -       if (i != j) {
> > -               pr_emerg("backwards %d, forwards %d\n", j, i);
> > -               bug = 1;
> > -       }
> > -       return bug ? -1 : i;
> > -}
> >  #if defined(CONFIG_DEBUG_MAPLE_TREE)
> >  extern void mt_validate(struct maple_tree *mt);
> >  extern void mt_dump(const struct maple_tree *mt);
> > @@ -405,17 +317,25 @@ static void validate_mm_mt(struct mm_struct *mm)
> >                         dump_stack();
> >  #ifdef CONFIG_DEBUG_VM
> >                         dump_vma(vma_mt);
> > -                       pr_emerg("and next in rb\n");
> > +                       pr_emerg("and vm_next\n");
> >                         dump_vma(vma->vm_next);
> > -#endif
> > +#endif // CONFIG_DEBUG_VM
> >                         pr_emerg("mt piv: %px %lu - %lu\n", vma_mt,
> >                                  mas.index, mas.last);
> >                         pr_emerg("mt vma: %px %lu - %lu\n", vma_mt,
> >                                  vma_mt->vm_start, vma_mt->vm_end);
> > -                       pr_emerg("rb vma: %px %lu - %lu\n", vma,
> > +                       if (vma->vm_prev) {
> > +                               pr_emerg("ll prev: %px %lu - %lu\n",
> > +                                        vma->vm_prev, vma->vm_prev->vm_start,
> > +                                        vma->vm_prev->vm_end);
> > +                       }
> > +                       pr_emerg("ll vma: %px %lu - %lu\n", vma,
> >                                  vma->vm_start, vma->vm_end);
> > -                       pr_emerg("rb->next = %px %lu - %lu\n", vma->vm_next,
> > -                                       vma->vm_next->vm_start, vma->vm_next->vm_end);
> > +                       if (vma->vm_next) {
> > +                               pr_emerg("ll next: %px %lu - %lu\n",
> > +                                        vma->vm_next, vma->vm_next->vm_start,
> > +                                        vma->vm_next->vm_end);
> > +                       }
> >
> >                         mt_dump(mas.tree);
> >                         if (vma_mt->vm_end != mas.last + 1) {
> > @@ -441,21 +361,6 @@ static void validate_mm_mt(struct mm_struct *mm)
> >         rcu_read_unlock();
> >         mt_validate(&mm->mm_mt);
> >  }
> > -#else
> > -#define validate_mm_mt(root) do { } while (0)
> > -#endif
> > -static void validate_mm_rb(struct rb_root *root, struct vm_area_struct *ignore)
> > -{
> > -       struct rb_node *nd;
> > -
> > -       for (nd = rb_first(root); nd; nd = rb_next(nd)) {
> > -               struct vm_area_struct *vma;
> > -               vma = rb_entry(nd, struct vm_area_struct, vm_rb);
> > -               VM_BUG_ON_VMA(vma != ignore &&
> > -                       vma->rb_subtree_gap != vma_compute_subtree_gap(vma),
> > -                       vma);
> > -       }
> > -}
> >
> >  static void validate_mm(struct mm_struct *mm)
> >  {
> > @@ -464,6 +369,8 @@ static void validate_mm(struct mm_struct *mm)
> >         unsigned long highest_address = 0;
> >         struct vm_area_struct *vma = mm->mmap;
> >
> > +       validate_mm_mt(mm);
> > +
> >         while (vma) {
> >                 struct anon_vma *anon_vma = vma->anon_vma;
> >                 struct anon_vma_chain *avc;
> > @@ -488,80 +395,13 @@ static void validate_mm(struct mm_struct *mm)
> >                           mm->highest_vm_end, highest_address);
> >                 bug = 1;
> >         }
> > -       i = browse_rb(mm);
> > -       if (i != mm->map_count) {
> > -               if (i != -1)
> > -                       pr_emerg("map_count %d rb %d\n", mm->map_count, i);
> > -               bug = 1;
> > -       }
> >         VM_BUG_ON_MM(bug, mm);
> >  }
> > -#else
> > -#define validate_mm_rb(root, ignore) do { } while (0)
> > +
> > +#else // !CONFIG_DEBUG_MAPLE_TREE
> >  #define validate_mm_mt(root) do { } while (0)
> >  #define validate_mm(mm) do { } while (0)
> > -#endif
> > -
> > -RB_DECLARE_CALLBACKS_MAX(static, vma_gap_callbacks,
> > -                        struct vm_area_struct, vm_rb,
> > -                        unsigned long, rb_subtree_gap, vma_compute_gap)
> > -
> > -/*
> > - * Update augmented rbtree rb_subtree_gap values after vma->vm_start or
> > - * vma->vm_prev->vm_end values changed, without modifying the vma's position
> > - * in the rbtree.
> > - */
> > -static void vma_gap_update(struct vm_area_struct *vma)
> > -{
> > -       /*
> > -        * As it turns out, RB_DECLARE_CALLBACKS_MAX() already created
> > -        * a callback function that does exactly what we want.
> > -        */
> > -       vma_gap_callbacks_propagate(&vma->vm_rb, NULL);
> > -}
> > -
> > -static inline void vma_rb_insert(struct vm_area_struct *vma,
> > -                                struct rb_root *root)
> > -{
> > -       /* All rb_subtree_gap values must be consistent prior to insertion */
> > -       validate_mm_rb(root, NULL);
> > -
> > -       rb_insert_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
> > -}
> > -
> > -static void __vma_rb_erase(struct vm_area_struct *vma, struct rb_root *root)
> > -{
> > -       /*
> > -        * Note rb_erase_augmented is a fairly large inline function,
> > -        * so make sure we instantiate it only once with our desired
> > -        * augmented rbtree callbacks.
> > -        */
> > -       rb_erase_augmented(&vma->vm_rb, root, &vma_gap_callbacks);
> > -}
> > -
> > -static __always_inline void vma_rb_erase_ignore(struct vm_area_struct *vma,
> > -                                               struct rb_root *root,
> > -                                               struct vm_area_struct *ignore)
> > -{
> > -       /*
> > -        * All rb_subtree_gap values must be consistent prior to erase,
> > -        * with the possible exception of
> > -        *
> > -        * a. the "next" vma being erased if next->vm_start was reduced in
> > -        *    __vma_adjust() -> __vma_unlink()
> > -        * b. the vma being erased in detach_vmas_to_be_unmapped() ->
> > -        *    vma_rb_erase()
> > -        */
> > -       validate_mm_rb(root, ignore);
> > -
> > -       __vma_rb_erase(vma, root);
> > -}
> > -
> > -static __always_inline void vma_rb_erase(struct vm_area_struct *vma,
> > -                                        struct rb_root *root)
> > -{
> > -       vma_rb_erase_ignore(vma, root, vma);
> > -}
> > +#endif // CONFIG_DEBUG_MAPLE_TREE
> >
> >  /*
> >   * vma has some anon_vma assigned, and is already inserted on that
> > @@ -595,38 +435,26 @@ anon_vma_interval_tree_post_update_vma(struct vm_area_struct *vma)
> >                 anon_vma_interval_tree_insert(avc, &avc->anon_vma->rb_root);
> >  }
> >
> > -static int find_vma_links(struct mm_struct *mm, unsigned long addr,
> > -               unsigned long end, struct vm_area_struct **pprev,
> > -               struct rb_node ***rb_link, struct rb_node **rb_parent)
> > +/* Private
> > + * range_has_overlap() - Check the @start - @end range for overlapping VMAs and
> > + * sets up a pointer to the previous VMA
> > + *
> > + * @mm - the mm struct
> > + * @start - the start address of the range
> > + * @end - the end address of the range
> > + * @pprev - the pointer to the pointer of the previous VMA
> > + *
> > + * Returns: True if there is an overlapping VMA, false otherwise
> > + */
> > +static bool range_has_overlap(struct mm_struct *mm, unsigned long start,
> > +                             unsigned long end, struct vm_area_struct **pprev)
> >  {
> > -       struct rb_node **__rb_link, *__rb_parent, *rb_prev;
> > -
> > -       __rb_link = &mm->mm_rb.rb_node;
> > -       rb_prev = __rb_parent = NULL;
> > +       struct vm_area_struct *existing;
> >
> > -       while (*__rb_link) {
> > -               struct vm_area_struct *vma_tmp;
> > -
> > -               __rb_parent = *__rb_link;
> > -               vma_tmp = rb_entry(__rb_parent, struct vm_area_struct, vm_rb);
> > -
> > -               if (vma_tmp->vm_end > addr) {
> > -                       /* Fail if an existing vma overlaps the area */
> > -                       if (vma_tmp->vm_start < end)
> > -                               return -ENOMEM;
> > -                       __rb_link = &__rb_parent->rb_left;
> > -               } else {
> > -                       rb_prev = __rb_parent;
> > -                       __rb_link = &__rb_parent->rb_right;
> > -               }
> > -       }
> > -
> > -       *pprev = NULL;
> > -       if (rb_prev)
> > -               *pprev = rb_entry(rb_prev, struct vm_area_struct, vm_rb);
> > -       *rb_link = __rb_link;
> > -       *rb_parent = __rb_parent;
> > -       return 0;
> > +       MA_STATE(mas, &mm->mm_mt, start, start);
> > +       existing = mas_find(&mas, end - 1);
> > +       *pprev = mas_prev(&mas, 0);
> > +       return existing ? true : false;
> >  }
> >
> >  /*
> > @@ -653,8 +481,6 @@ static inline struct vm_area_struct *vma_next(struct mm_struct *mm,
> >   * @start: The start of the range.
> >   * @len: The length of the range.
> >   * @pprev: pointer to the pointer that will be set to previous vm_area_struct
> > - * @rb_link: the rb_node
> > - * @rb_parent: the parent rb_node
> >   *
> >   * Find all the vm_area_struct that overlap from @start to
> >   * @end and munmap them.  Set @pprev to the previous vm_area_struct.
> > @@ -663,76 +489,41 @@ static inline struct vm_area_struct *vma_next(struct mm_struct *mm,
> >   */
> >  static inline int
> >  munmap_vma_range(struct mm_struct *mm, unsigned long start, unsigned long len,
> > -                struct vm_area_struct **pprev, struct rb_node ***link,
> > -                struct rb_node **parent, struct list_head *uf)
> > +                struct vm_area_struct **pprev, struct list_head *uf)
> >  {
> > -
> > -       while (find_vma_links(mm, start, start + len, pprev, link, parent))
> > +       // Needs optimization.
> > +       while (range_has_overlap(mm, start, start + len, pprev))
> >                 if (do_munmap(mm, start, len, uf))
> >                         return -ENOMEM;
> > -
> >         return 0;
> >  }
> >  static unsigned long count_vma_pages_range(struct mm_struct *mm,
> >                 unsigned long addr, unsigned long end)
> >  {
> >         unsigned long nr_pages = 0;
> > -       unsigned long nr_mt_pages = 0;
> >         struct vm_area_struct *vma;
> > +       unsigned long vm_start, vm_end;
> > +       MA_STATE(mas, &mm->mm_mt, addr, addr);
> >
> > -       /* Find first overlapping mapping */
> > -       vma = find_vma_intersection(mm, addr, end);
> > +       /* Find first overlaping mapping */
> 
> nit: I think the original comment was correct.

Yes.  Not sure hwo this happened.

> 
> > +       vma = mas_find(&mas, end - 1);
> >         if (!vma)
> >                 return 0;
> >
> > -       nr_pages = (min(end, vma->vm_end) -
> > -               max(addr, vma->vm_start)) >> PAGE_SHIFT;
> > +       vm_start = vma->vm_start;
> > +       vm_end = vma->vm_end;
> > +       nr_pages = (min(end, vm_end) - max(addr, vm_start)) >> PAGE_SHIFT;
> >
> >         /* Iterate over the rest of the overlaps */
> > -       for (vma = vma->vm_next; vma; vma = vma->vm_next) {
> > -               unsigned long overlap_len;
> > -
> > -               if (vma->vm_start > end)
> > -                       break;
> > -
> > -               overlap_len = min(end, vma->vm_end) - vma->vm_start;
> > -               nr_pages += overlap_len >> PAGE_SHIFT;
> > +       mas_for_each(&mas, vma, end) {
> > +               vm_start = vma->vm_start;
> > +               vm_end = vma->vm_end;
> > +               nr_pages += (min(end, vm_end) - vm_start) >> PAGE_SHIFT;
> >         }
> >
> > -       mt_for_each(&mm->mm_mt, vma, addr, end) {
> > -               nr_mt_pages +=
> > -                       (min(end, vma->vm_end) - vma->vm_start) >> PAGE_SHIFT;
> > -       }
> > -
> > -       VM_BUG_ON_MM(nr_pages != nr_mt_pages, mm);
> > -
> >         return nr_pages;
> >  }
> >
> > -void __vma_link_rb(struct mm_struct *mm, struct vm_area_struct *vma,
> > -               struct rb_node **rb_link, struct rb_node *rb_parent)
> > -{
> > -       /* Update tracking information for the gap following the new vma. */
> > -       if (vma->vm_next)
> > -               vma_gap_update(vma->vm_next);
> > -       else
> > -               mm->highest_vm_end = vm_end_gap(vma);
> > -
> > -       /*
> > -        * vma->vm_prev wasn't known when we followed the rbtree to find the
> > -        * correct insertion point for that vma. As a result, we could not
> > -        * update the vma vm_rb parents rb_subtree_gap values on the way down.
> > -        * So, we first insert the vma with a zero rb_subtree_gap value
> > -        * (to be consistent with what we did on the way down), and then
> > -        * immediately update the gap to the correct value. Finally we
> > -        * rebalance the rbtree after all augmented values have been set.
> > -        */
> > -       rb_link_node(&vma->vm_rb, rb_parent, rb_link);
> > -       vma->rb_subtree_gap = 0;
> > -       vma_gap_update(vma);
> > -       vma_rb_insert(vma, &mm->mm_rb);
> > -}
> > -
> >  static void __vma_link_file(struct vm_area_struct *vma)
> >  {
> >         struct file *file;
> > @@ -780,19 +571,8 @@ static inline void vma_mt_store(struct mm_struct *mm, struct vm_area_struct *vma
> >                 GFP_KERNEL);
> >  }
> >
> > -static void
> > -__vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> > -       struct vm_area_struct *prev, struct rb_node **rb_link,
> > -       struct rb_node *rb_parent)
> > -{
> > -       vma_mt_store(mm, vma);
> > -       __vma_link_list(mm, vma, prev);
> > -       __vma_link_rb(mm, vma, rb_link, rb_parent);
> > -}
> > -
> >  static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> > -                       struct vm_area_struct *prev, struct rb_node **rb_link,
> > -                       struct rb_node *rb_parent)
> > +                       struct vm_area_struct *prev)
> >  {
> >         struct address_space *mapping = NULL;
> >
> > @@ -801,7 +581,8 @@ static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> >                 i_mmap_lock_write(mapping);
> >         }
> >
> > -       __vma_link(mm, vma, prev, rb_link, rb_parent);
> > +       vma_mt_store(mm, vma);
> > +       __vma_link_list(mm, vma, prev);
> >         __vma_link_file(vma);
> >
> >         if (mapping)
> > @@ -813,30 +594,18 @@ static void vma_link(struct mm_struct *mm, struct vm_area_struct *vma,
> >
> >  /*
> >   * Helper for vma_adjust() in the split_vma insert case: insert a vma into the
> > - * mm's list and rbtree.  It has already been inserted into the interval tree.
> > + * mm's list and the mm tree.  It has already been inserted into the interval tree.
> >   */
> >  static void __insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
> >  {
> >         struct vm_area_struct *prev;
> > -       struct rb_node **rb_link, *rb_parent;
> >
> > -       if (find_vma_links(mm, vma->vm_start, vma->vm_end,
> > -                          &prev, &rb_link, &rb_parent))
> > -               BUG();
> > -       __vma_link(mm, vma, prev, rb_link, rb_parent);
> > +       BUG_ON(range_has_overlap(mm, vma->vm_start, vma->vm_end, &prev));
> > +       vma_mt_store(mm, vma);
> > +       __vma_link_list(mm, vma, prev);
> >         mm->map_count++;
> >  }
> >
> > -static __always_inline void __vma_unlink(struct mm_struct *mm,
> > -                                               struct vm_area_struct *vma,
> > -                                               struct vm_area_struct *ignore)
> > -{
> > -       vma_rb_erase_ignore(vma, &mm->mm_rb, ignore);
> > -       __vma_unlink_list(mm, vma);
> > -       /* Kill the cache */
> > -       vmacache_invalidate(mm);
> > -}
> > -
> >  /*
> >   * We cannot adjust vm_start, vm_end, vm_pgoff fields of a vma that
> >   * is already present in an i_mmap tree without adjusting the tree.
> > @@ -854,13 +623,10 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >         struct rb_root_cached *root = NULL;
> >         struct anon_vma *anon_vma = NULL;
> >         struct file *file = vma->vm_file;
> > -       bool start_changed = false, end_changed = false;
> > +       bool vma_changed = false;
> >         long adjust_next = 0;
> >         int remove_next = 0;
> >
> > -       validate_mm(mm);
> > -       validate_mm_mt(mm);
> > -
> >         if (next && !insert) {
> >                 struct vm_area_struct *exporter = NULL, *importer = NULL;
> >
> > @@ -986,21 +752,23 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >         }
> >
> >         if (start != vma->vm_start) {
> > -               unsigned long old_start = vma->vm_start;
> > +               if (vma->vm_start < start)
> > +                       vma_mt_szero(mm, vma->vm_start, start);
> > +               else
> > +                       vma_changed = true;
> >                 vma->vm_start = start;
> > -               if (old_start < start)
> > -                       vma_mt_szero(mm, old_start, start);
> > -               start_changed = true;
> >         }
> >         if (end != vma->vm_end) {
> > -               unsigned long old_end = vma->vm_end;
> > +               if (vma->vm_end > end)
> > +                       vma_mt_szero(mm, end, vma->vm_end);
> > +               else
> > +                       vma_changed = true;
> >                 vma->vm_end = end;
> > -               if (old_end > end)
> > -                       vma_mt_szero(mm, end, old_end);
> > -               end_changed = true;
> > +               if (!next)
> > +                       mm->highest_vm_end = vm_end_gap(vma);
> >         }
> >
> > -       if (end_changed || start_changed)
> > +       if (vma_changed)
> >                 vma_mt_store(mm, vma);
> >
> >         vma->vm_pgoff = pgoff;
> > @@ -1018,25 +786,9 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >         }
> >
> >         if (remove_next) {
> > -               /*
> > -                * vma_merge has merged next into vma, and needs
> > -                * us to remove next before dropping the locks.
> > -                * Since we have expanded over this vma, the maple tree will
> > -                * have overwritten by storing the value
> > -                */
> > -               if (remove_next != 3)
> > -                       __vma_unlink(mm, next, next);
> > -               else
> > -                       /*
> > -                        * vma is not before next if they've been
> > -                        * swapped.
> > -                        *
> > -                        * pre-swap() next->vm_start was reduced so
> > -                        * tell validate_mm_rb to ignore pre-swap()
> > -                        * "next" (which is stored in post-swap()
> > -                        * "vma").
> > -                        */
> > -                       __vma_unlink(mm, next, vma);
> > +               __vma_unlink_list(mm, next);
> > +               /* Kill the cache */
> > +               vmacache_invalidate(mm);
> >                 if (file)
> >                         __remove_shared_vm_struct(next, file, mapping);
> >         } else if (insert) {
> > @@ -1046,15 +798,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >                  * (it may either follow vma or precede it).
> >                  */
> >                 __insert_vm_struct(mm, insert);
> > -       } else {
> > -               if (start_changed)
> > -                       vma_gap_update(vma);
> > -               if (end_changed) {
> > -                       if (!next)
> > -                               mm->highest_vm_end = vm_end_gap(vma);
> > -                       else if (!adjust_next)
> > -                               vma_gap_update(next);
> > -               }
> >         }
> >
> >         if (anon_vma) {
> > @@ -1112,10 +855,7 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >                         remove_next = 1;
> >                         end = next->vm_end;
> >                         goto again;
> > -               }
> > -               else if (next)
> > -                       vma_gap_update(next);
> > -               else {
> > +               } else if (!next) {
> >                         /*
> >                          * If remove_next == 2 we obviously can't
> >                          * reach this path.
> > @@ -1142,8 +882,6 @@ int __vma_adjust(struct vm_area_struct *vma, unsigned long start,
> >                 uprobe_mmap(insert);
> >
> >         validate_mm(mm);
> > -       validate_mm_mt(mm);
> > -
> >         return 0;
> >  }
> >
> > @@ -1290,7 +1028,6 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,
> >         struct vm_area_struct *area, *next;
> >         int err;
> >
> > -       validate_mm_mt(mm);
> >         /*
> >          * We later require that vma->vm_flags == vm_flags,
> >          * so this tests vma->vm_flags & VM_SPECIAL, too.
> > @@ -1366,7 +1103,6 @@ struct vm_area_struct *vma_merge(struct mm_struct *mm,
> >                 khugepaged_enter_vma_merge(area, vm_flags);
> >                 return area;
> >         }
> > -       validate_mm_mt(mm);
> >
> >         return NULL;
> >  }
> > @@ -1536,6 +1272,7 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
> >         vm_flags_t vm_flags;
> >         int pkey = 0;
> >
> > +       validate_mm(mm);
> >         *populate = 0;
> >
> >         if (!len)
> > @@ -1856,10 +1593,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
> >         struct mm_struct *mm = current->mm;
> >         struct vm_area_struct *vma, *prev, *merge;
> >         int error;
> > -       struct rb_node **rb_link, *rb_parent;
> >         unsigned long charged = 0;
> >
> > -       validate_mm_mt(mm);
> >         /* Check against address space limit. */
> >         if (!may_expand_vm(mm, vm_flags, len >> PAGE_SHIFT)) {
> >                 unsigned long nr_pages;
> > @@ -1875,8 +1610,8 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
> >                         return -ENOMEM;
> >         }
> >
> > -       /* Clear old maps, set up prev, rb_link, rb_parent, and uf */
> > -       if (munmap_vma_range(mm, addr, len, &prev, &rb_link, &rb_parent, uf))
> > +       /* Clear old maps, set up prev and uf */
> > +       if (munmap_vma_range(mm, addr, len, &prev, uf))
> >                 return -ENOMEM;
> >         /*
> >          * Private writable mapping: check memory availability
> > @@ -1984,7 +1719,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
> >                         goto free_vma;
> >         }
> >
> > -       vma_link(mm, vma, prev, rb_link, rb_parent);
> > +       vma_link(mm, vma, prev);
> >         /* Once vma denies write, undo our temporary denial count */
> >         if (file) {
> >  unmap_writable:
> > @@ -2021,7 +1756,6 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
> >
> >         vma_set_page_prot(vma);
> >
> > -       validate_mm_mt(mm);
> >         return addr;
> >
> >  unmap_and_free_vma:
> > @@ -2041,7 +1775,6 @@ unsigned long mmap_region(struct file *file, unsigned long addr,
> >  unacct_error:
> >         if (charged)
> >                 vm_unacct_memory(charged);
> > -       validate_mm_mt(mm);
> >         return error;
> >  }
> >
> > @@ -2324,9 +2057,6 @@ find_vma_prev(struct mm_struct *mm, unsigned long addr,
> >
> >         rcu_read_lock();
> >         vma = mas_find(&mas, ULONG_MAX);
> > -       if (!vma)
> > -               mas_reset(&mas);
> > -
> >         *pprev = mas_prev(&mas, 0);
> >         rcu_read_unlock();
> >         return vma;
> > @@ -2390,7 +2120,6 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
> >         unsigned long gap_addr;
> >         int error = 0;
> >
> > -       validate_mm_mt(mm);
> >         if (!(vma->vm_flags & VM_GROWSUP))
> >                 return -EFAULT;
> >
> > @@ -2437,15 +2166,13 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
> >                         error = acct_stack_growth(vma, size, grow);
> >                         if (!error) {
> >                                 /*
> > -                                * vma_gap_update() doesn't support concurrent
> > -                                * updates, but we only hold a shared mmap_lock
> > -                                * lock here, so we need to protect against
> > -                                * concurrent vma expansions.
> > -                                * anon_vma_lock_write() doesn't help here, as
> > -                                * we don't guarantee that all growable vmas
> > -                                * in a mm share the same root anon vma.
> > -                                * So, we reuse mm->page_table_lock to guard
> > -                                * against concurrent vma expansions.
> > +                                * We only hold a shared mmap_lock lock here, so
> > +                                * we need to protect against concurrent vma
> > +                                * expansions.  anon_vma_lock_write() doesn't
> > +                                * help here, as we don't guarantee that all
> > +                                * growable vmas in a mm share the same root
> > +                                * anon vma.  So, we reuse mm->page_table_lock
> > +                                * to guard against concurrent vma expansions.
> >                                  */
> >                                 spin_lock(&mm->page_table_lock);
> >                                 if (vma->vm_flags & VM_LOCKED)
> > @@ -2453,10 +2180,9 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
> >                                 vm_stat_account(mm, vma->vm_flags, grow);
> >                                 anon_vma_interval_tree_pre_update_vma(vma);
> >                                 vma->vm_end = address;
> > +                               vma_mt_store(mm, vma);
> >                                 anon_vma_interval_tree_post_update_vma(vma);
> > -                               if (vma->vm_next)
> > -                                       vma_gap_update(vma->vm_next);
> > -                               else
> > +                               if (!vma->vm_next)
> >                                         mm->highest_vm_end = vm_end_gap(vma);
> >                                 spin_unlock(&mm->page_table_lock);
> >
> > @@ -2466,8 +2192,6 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
> >         }
> >         anon_vma_unlock_write(vma->anon_vma);
> >         khugepaged_enter_vma_merge(vma, vma->vm_flags);
> > -       validate_mm(mm);
> > -       validate_mm_mt(mm);
> >         return error;
> >  }
> >  #endif /* CONFIG_STACK_GROWSUP || CONFIG_IA64 */
> > @@ -2475,14 +2199,12 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address)
> >  /*
> >   * vma is the first one with address < vma->vm_start.  Have to extend vma.
> >   */
> > -int expand_downwards(struct vm_area_struct *vma,
> > -                                  unsigned long address)
> > +int expand_downwards(struct vm_area_struct *vma, unsigned long address)
> >  {
> >         struct mm_struct *mm = vma->vm_mm;
> >         struct vm_area_struct *prev;
> >         int error = 0;
> >
> > -       validate_mm(mm);
> >         address &= PAGE_MASK;
> >         if (address < mmap_min_addr)
> >                 return -EPERM;
> > @@ -2519,15 +2241,13 @@ int expand_downwards(struct vm_area_struct *vma,
> >                         error = acct_stack_growth(vma, size, grow);
> >                         if (!error) {
> >                                 /*
> > -                                * vma_gap_update() doesn't support concurrent
> > -                                * updates, but we only hold a shared mmap_lock
> > -                                * lock here, so we need to protect against
> > -                                * concurrent vma expansions.
> > -                                * anon_vma_lock_write() doesn't help here, as
> > -                                * we don't guarantee that all growable vmas
> > -                                * in a mm share the same root anon vma.
> > -                                * So, we reuse mm->page_table_lock to guard
> > -                                * against concurrent vma expansions.
> > +                                * We only hold a shared mmap_lock lock here, so
> > +                                * we need to protect against concurrent vma
> > +                                * expansions.  anon_vma_lock_write() doesn't
> > +                                * help here, as we don't guarantee that all
> > +                                * growable vmas in a mm share the same root
> > +                                * anon vma.  So, we reuse mm->page_table_lock
> > +                                * to guard against concurrent vma expansions.
> >                                  */
> >                                 spin_lock(&mm->page_table_lock);
> >                                 if (vma->vm_flags & VM_LOCKED)
> > @@ -2539,7 +2259,6 @@ int expand_downwards(struct vm_area_struct *vma,
> >                                 /* Overwrite old entry in mtree. */
> >                                 vma_mt_store(mm, vma);
> >                                 anon_vma_interval_tree_post_update_vma(vma);
> > -                               vma_gap_update(vma);
> >                                 spin_unlock(&mm->page_table_lock);
> >
> >                                 perf_event_mmap(vma);
> > @@ -2548,7 +2267,6 @@ int expand_downwards(struct vm_area_struct *vma,
> >         }
> >         anon_vma_unlock_write(vma->anon_vma);
> >         khugepaged_enter_vma_merge(vma, vma->vm_flags);
> > -       validate_mm(mm);
> >         return error;
> >  }
> >
> > @@ -2681,16 +2399,14 @@ detach_vmas_to_be_unmapped(struct mm_struct *mm, struct vm_area_struct *vma,
> >         vma->vm_prev = NULL;
> >         vma_mt_szero(mm, vma->vm_start, end);
> >         do {
> > -               vma_rb_erase(vma, &mm->mm_rb);
> >                 mm->map_count--;
> >                 tail_vma = vma;
> >                 vma = vma->vm_next;
> >         } while (vma && vma->vm_start < end);
> >         *insertion_point = vma;
> > -       if (vma) {
> > +       if (vma)
> >                 vma->vm_prev = prev;
> > -               vma_gap_update(vma);
> > -       } else
> > +       else
> >                 mm->highest_vm_end = prev ? vm_end_gap(prev) : 0;
> >         tail_vma->vm_next = NULL;
> >
> > @@ -2821,11 +2537,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
> >         if (len == 0)
> >                 return -EINVAL;
> >
> > -       /*
> > -        * arch_unmap() might do unmaps itself.  It must be called
> > -        * and finish any rbtree manipulation before this code
> > -        * runs and also starts to manipulate the rbtree.
> > -        */
> > +        /* arch_unmap() might do unmaps itself.  */
> >         arch_unmap(mm, start, end);
> >
> >         /* Find the first overlapping VMA */
> > @@ -2833,7 +2545,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
> >         if (!vma)
> >                 return 0;
> >         prev = vma->vm_prev;
> > -       /* we have  start < vma->vm_end  */
> > +       /* we have start < vma->vm_end  */
> >
> >         /* if it doesn't overlap, we have nothing.. */
> >         if (vma->vm_start >= end)
> > @@ -2893,7 +2605,7 @@ int __do_munmap(struct mm_struct *mm, unsigned long start, size_t len,
> >         if (mm->locked_vm)
> >                 unlock_range(vma, end);
> >
> > -       /* Detach vmas from rbtree */
> > +       /* Detach vmas from the MM linked list and remove from the mm tree*/
> >         if (!detach_vmas_to_be_unmapped(mm, vma, prev, end))
> >                 downgrade = false;
> >
> > @@ -3041,11 +2753,11 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
> >   *  anonymous maps.  eventually we may be able to do some
> >   *  brk-specific accounting here.
> >   */
> > -static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long flags, struct list_head *uf)
> > +static int do_brk_flags(unsigned long addr, unsigned long len,
> > +                       unsigned long flags, struct list_head *uf)
> >  {
> >         struct mm_struct *mm = current->mm;
> >         struct vm_area_struct *vma, *prev;
> > -       struct rb_node **rb_link, *rb_parent;
> >         pgoff_t pgoff = addr >> PAGE_SHIFT;
> >         int error;
> >         unsigned long mapped_addr;
> > @@ -3064,8 +2776,8 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
> >         if (error)
> >                 return error;
> >
> > -       /* Clear old maps, set up prev, rb_link, rb_parent, and uf */
> > -       if (munmap_vma_range(mm, addr, len, &prev, &rb_link, &rb_parent, uf))
> > +       /* Clear old maps, set up prev and uf */
> > +       if (munmap_vma_range(mm, addr, len, &prev, uf))
> >                 return -ENOMEM;
> >
> >         /* Check against address space limits *after* clearing old maps... */
> > @@ -3099,7 +2811,7 @@ static int do_brk_flags(unsigned long addr, unsigned long len, unsigned long fla
> >         vma->vm_pgoff = pgoff;
> >         vma->vm_flags = flags;
> >         vma->vm_page_prot = vm_get_page_prot(flags);
> > -       vma_link(mm, vma, prev, rb_link, rb_parent);
> > +       vma_link(mm, vma, prev);
> >  out:
> >         perf_event_mmap(vma);
> >         mm->total_vm += len >> PAGE_SHIFT;
> > @@ -3219,26 +2931,10 @@ void exit_mmap(struct mm_struct *mm)
> >  int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
> >  {
> >         struct vm_area_struct *prev;
> > -       struct rb_node **rb_link, *rb_parent;
> > -       unsigned long start = vma->vm_start;
> > -       struct vm_area_struct *overlap = NULL;
> >
> > -       if (find_vma_links(mm, vma->vm_start, vma->vm_end,
> > -                          &prev, &rb_link, &rb_parent))
> > +       if (range_has_overlap(mm, vma->vm_start, vma->vm_end, &prev))
> >                 return -ENOMEM;
> >
> > -       overlap = mt_find(&mm->mm_mt, &start, vma->vm_end - 1);
> > -       if (overlap) {
> > -
> > -               pr_err("Found vma ending at %lu\n", start - 1);
> > -               pr_err("vma : %lu => %lu-%lu\n", (unsigned long)overlap,
> > -                               overlap->vm_start, overlap->vm_end - 1);
> > -#if defined(CONFIG_DEBUG_MAPLE_TREE)
> > -               mt_dump(&mm->mm_mt);
> > -#endif
> > -               BUG();
> > -       }
> > -
> >         if ((vma->vm_flags & VM_ACCOUNT) &&
> >              security_vm_enough_memory_mm(mm, vma_pages(vma)))
> >                 return -ENOMEM;
> > @@ -3260,7 +2956,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
> >                 vma->vm_pgoff = vma->vm_start >> PAGE_SHIFT;
> >         }
> >
> > -       vma_link(mm, vma, prev, rb_link, rb_parent);
> > +       vma_link(mm, vma, prev);
> >         return 0;
> >  }
> >
> > @@ -3276,9 +2972,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
> >         unsigned long vma_start = vma->vm_start;
> >         struct mm_struct *mm = vma->vm_mm;
> >         struct vm_area_struct *new_vma, *prev;
> > -       struct rb_node **rb_link, *rb_parent;
> >         bool faulted_in_anon_vma = true;
> > -       unsigned long index = addr;
> >
> >         validate_mm_mt(mm);
> >         /*
> > @@ -3290,10 +2984,9 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
> >                 faulted_in_anon_vma = false;
> >         }
> >
> > -       if (find_vma_links(mm, addr, addr + len, &prev, &rb_link, &rb_parent))
> > +       if (range_has_overlap(mm, addr, addr + len, &prev))
> >                 return NULL;    /* should never get here */
> > -       if (mt_find(&mm->mm_mt, &index, addr+len - 1))
> > -               BUG();
> > +
> >         new_vma = vma_merge(mm, prev, addr, addr + len, vma->vm_flags,
> >                             vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma),
> >                             vma->vm_userfaultfd_ctx);
> > @@ -3334,7 +3027,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
> >                         get_file(new_vma->vm_file);
> >                 if (new_vma->vm_ops && new_vma->vm_ops->open)
> >                         new_vma->vm_ops->open(new_vma);
> > -               vma_link(mm, new_vma, prev, rb_link, rb_parent);
> > +               vma_link(mm, new_vma, prev);
> >                 *need_rmap_locks = false;
> >         }
> >         validate_mm_mt(mm);
> > diff --git a/mm/nommu.c b/mm/nommu.c
> > index 8848cf7cb7c1..c410f99203fb 100644
> > --- a/mm/nommu.c
> > +++ b/mm/nommu.c
> > @@ -566,13 +566,14 @@ static void put_nommu_region(struct vm_region *region)
> >   */
> >  static void add_vma_to_mm(struct mm_struct *mm, struct vm_area_struct *vma)
> >  {
> > -       struct vm_area_struct *pvma, *prev;
> >         struct address_space *mapping;
> > -       struct rb_node **p, *parent, *rb_prev;
> > +       struct vm_area_struct *prev;
> > +       MA_STATE(mas, &mm->mm_mt, vma->vm_start, vma->vm_end);
> >
> >         BUG_ON(!vma->vm_region);
> >
> >         mm->map_count++;
> > +       printk("mm at %u\n", mm->map_count);
> 
> I think this was added while debugging and should be removed?

Yes, thank you.

> 
> >         vma->vm_mm = mm;
> >
> >         /* add the VMA to the mapping */
> > @@ -586,42 +587,12 @@ static void add_vma_to_mm(struct mm_struct *mm, struct vm_area_struct *vma)
> >                 i_mmap_unlock_write(mapping);
> >         }
> >
> > +       rcu_read_lock();
> > +       prev = mas_prev(&mas, 0);
> > +       rcu_read_unlock();
> > +       mas_reset(&mas);
> >         /* add the VMA to the tree */
> > -       parent = rb_prev = NULL;
> > -       p = &mm->mm_rb.rb_node;
> > -       while (*p) {
> > -               parent = *p;
> > -               pvma = rb_entry(parent, struct vm_area_struct, vm_rb);
> > -
> > -               /* sort by: start addr, end addr, VMA struct addr in that order
> > -                * (the latter is necessary as we may get identical VMAs) */
> > -               if (vma->vm_start < pvma->vm_start)
> > -                       p = &(*p)->rb_left;
> > -               else if (vma->vm_start > pvma->vm_start) {
> > -                       rb_prev = parent;
> > -                       p = &(*p)->rb_right;
> > -               } else if (vma->vm_end < pvma->vm_end)
> > -                       p = &(*p)->rb_left;
> > -               else if (vma->vm_end > pvma->vm_end) {
> > -                       rb_prev = parent;
> > -                       p = &(*p)->rb_right;
> > -               } else if (vma < pvma)
> > -                       p = &(*p)->rb_left;
> > -               else if (vma > pvma) {
> > -                       rb_prev = parent;
> > -                       p = &(*p)->rb_right;
> > -               } else
> > -                       BUG();
> > -       }
> > -
> > -       rb_link_node(&vma->vm_rb, parent, p);
> > -       rb_insert_color(&vma->vm_rb, &mm->mm_rb);
> > -
> > -       /* add VMA to the VMA list also */
> > -       prev = NULL;
> > -       if (rb_prev)
> > -               prev = rb_entry(rb_prev, struct vm_area_struct, vm_rb);
> > -
> > +       vma_mas_store(vma, &mas);
> >         __vma_link_list(mm, vma, prev);
> >  }
> >
> > @@ -634,6 +605,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
> >         struct address_space *mapping;
> >         struct mm_struct *mm = vma->vm_mm;
> >         struct task_struct *curr = current;
> > +       MA_STATE(mas, &vma->vm_mm->mm_mt, 0, 0);
> >
> >         mm->map_count--;
> >         for (i = 0; i < VMACACHE_SIZE; i++) {
> > @@ -643,7 +615,6 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
> >                         break;
> >                 }
> >         }
> > -
> >         /* remove the VMA from the mapping */
> >         if (vma->vm_file) {
> >                 mapping = vma->vm_file->f_mapping;
> > @@ -656,8 +627,7 @@ static void delete_vma_from_mm(struct vm_area_struct *vma)
> >         }
> >
> >         /* remove from the MM's tree and list */
> > -       rb_erase(&vma->vm_rb, &mm->mm_rb);
> > -
> > +       vma_mas_remove(vma, &mas);
> >         __vma_unlink_list(mm, vma);
> >  }
> >
> > @@ -681,24 +651,21 @@ static void delete_vma(struct mm_struct *mm, struct vm_area_struct *vma)
> >  struct vm_area_struct *find_vma(struct mm_struct *mm, unsigned long addr)
> >  {
> >         struct vm_area_struct *vma;
> > +       MA_STATE(mas, &mm->mm_mt, addr, addr);
> >
> >         /* check the cache first */
> >         vma = vmacache_find(mm, addr);
> >         if (likely(vma))
> >                 return vma;
> >
> > -       /* trawl the list (there may be multiple mappings in which addr
> > -        * resides) */
> > -       for (vma = mm->mmap; vma; vma = vma->vm_next) {
> > -               if (vma->vm_start > addr)
> > -                       return NULL;
> > -               if (vma->vm_end > addr) {
> > -                       vmacache_update(addr, vma);
> > -                       return vma;
> > -               }
> > -       }
> > +       rcu_read_lock();
> > +       vma = mas_walk(&mas);
> > +       rcu_read_unlock();
> >
> > -       return NULL;
> > +       if (vma)
> > +               vmacache_update(addr, vma);
> > +
> > +       return vma;
> >  }
> >  EXPORT_SYMBOL(find_vma);
> >
> > @@ -730,26 +697,25 @@ static struct vm_area_struct *find_vma_exact(struct mm_struct *mm,
> >  {
> >         struct vm_area_struct *vma;
> >         unsigned long end = addr + len;
> > +       MA_STATE(mas, &mm->mm_mt, addr, addr);
> >
> >         /* check the cache first */
> >         vma = vmacache_find_exact(mm, addr, end);
> >         if (vma)
> >                 return vma;
> >
> > -       /* trawl the list (there may be multiple mappings in which addr
> > -        * resides) */
> > -       for (vma = mm->mmap; vma; vma = vma->vm_next) {
> > -               if (vma->vm_start < addr)
> > -                       continue;
> > -               if (vma->vm_start > addr)
> > -                       return NULL;
> > -               if (vma->vm_end == end) {
> > -                       vmacache_update(addr, vma);
> > -                       return vma;
> > -               }
> > -       }
> > -
> > -       return NULL;
> > +       rcu_read_lock();
> > +       vma = mas_walk(&mas);
> > +       rcu_read_unlock();
> > +       if (!vma)
> > +               return NULL;
> > +       if (vma->vm_start != addr)
> > +               return NULL;
> > +       if (vma->vm_end != end)
> > +               return NULL;
> > +
> > +       vmacache_update(addr, vma);
> > +       return vma;
> >  }
> >
> >  /*
> > diff --git a/mm/util.c b/mm/util.c
> > index 0b6dd9d81da7..35deaa0ccac5 100644
> > --- a/mm/util.c
> > +++ b/mm/util.c
> > @@ -287,6 +287,8 @@ void __vma_link_list(struct mm_struct *mm, struct vm_area_struct *vma,
> >         vma->vm_next = next;
> >         if (next)
> >                 next->vm_prev = vma;
> > +       else
> > +               mm->highest_vm_end = vm_end_gap(vma);
> >  }
> >
> >  void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma)
> > @@ -301,6 +303,12 @@ void __vma_unlink_list(struct mm_struct *mm, struct vm_area_struct *vma)
> >                 mm->mmap = next;
> >         if (next)
> >                 next->vm_prev = prev;
> > +       else {
> > +               if (prev)
> > +                       mm->highest_vm_end = vm_end_gap(prev);
> > +               else
> > +                       mm->highest_vm_end = 0;
> > +       }
> >  }
> >
> >  /* Check if the vma is being used as a stack by this task */
> > --
> > 2.30.2

