From pfalcato at suse.de  Mon Sep  1 03:27:47 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Mon, 1 Sep 2025 11:27:47 +0100
Subject: [PATCH v2 1/4] tools/include: Implement a couple of atomic_t ops
In-Reply-To: <20250828-b4-vma-no-atomic-h-v2-1-02d146a58ed2@google.com>
References: <20250828-b4-vma-no-atomic-h-v2-0-02d146a58ed2@google.com>
 <20250828-b4-vma-no-atomic-h-v2-1-02d146a58ed2@google.com>
Message-ID: <tnbyxwmcwia3teewhqlpdjk7igsljfqcqbze2rbalzidccfndy@e3ybvfap4pbd>

On Thu, Aug 28, 2025 at 12:27:58PM +0000, Brendan Jackman wrote:
> The VMA tests need an operation equivalent to
> atomic_inc_unless_negative() to implement a fake mapping_map_writable().
> Adding it will enable them to switch to the shared atomic headers and
> simplify that fake implementation.
> 
> In order to add that, also add atomic_try_cmpxchg() which can be used to
> implement it. This is copied from Documentation/atomic_t.txt. Then,
> implement atomic_inc_unless_negative() itself based on the
> raw_atomic_dec_unless_positive() in
> include/linux/atomic/atomic-arch-fallback.h.
> 
> There's no present need for a highly-optimised version of this (nor any
> reason to think this implementation is sub-optimal on x86) so just
> implement this with generic C, no x86-specifics.
> 
> Signed-off-by: Brendan Jackman <jackmanb at google.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

LGTM, thanks!

-- 
Pedro


From pfalcato at suse.de  Mon Sep  1 03:29:05 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Mon, 1 Sep 2025 11:29:05 +0100
Subject: [PATCH v2 4/4] tools: testing: Use existing atomic.h for
 vma/maple tests
In-Reply-To: <20250828-b4-vma-no-atomic-h-v2-4-02d146a58ed2@google.com>
References: <20250828-b4-vma-no-atomic-h-v2-0-02d146a58ed2@google.com>
 <20250828-b4-vma-no-atomic-h-v2-4-02d146a58ed2@google.com>
Message-ID: <q5chl5fnmaj6wxlj3aclz7n3k4bz5sgjfliao37wdiefi32mgi@wxefcss6cmqp>

On Thu, Aug 28, 2025 at 12:28:01PM +0000, Brendan Jackman wrote:
> The shared userspace logic used for unit-testing maple tree and VMA code
> currently has its own replacements for atomics helpers. This is not
> needed as the necessary APIs already have userspace implementations in
> the tools tree. Switching over to that allows deleting a bit of code.
> 
> Note that the implementation is different; while the version being
> deleted here is implemented using liburcu, the existing version in tools
> uses either x86 asm or compiler builtins. It's assumed that both are
> equally likely to be correct.
> 
> The tools tree's version of atomic_t is a struct type while the version
> being deleted was just a typedef of an integer. This means it's no
> longer valid to call __sync_bool_compare_and_swap() directly on it. One
> option would be to just peek into the struct and call it on the field,
> but it seems a little cleaner to just use the corresponding atomic.h
> API whic has been added recently. Now the fake mapping_map_writable() is
> copied from the real one.
> 
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Brendan Jackman <jackmanb at google.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

> ---
>  tools/testing/shared/linux/maple_tree.h |  6 ++----
>  tools/testing/vma/linux/atomic.h        | 17 -----------------
>  tools/testing/vma/vma_internal.h        | 12 +++---------
>  3 files changed, 5 insertions(+), 30 deletions(-)

Yay, less code! :)

-- 
Pedro


From vbabka at suse.cz  Mon Sep  1 04:08:51 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:51 +0200
Subject: [PATCH 01/12] maple_tree: Fix check_bulk_rebalance() test locks
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-1-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

The check_bulk_rebalance() test was not correctly locking the tree which
caused issues with the sheaves testing in later patches.  Adding the
missing locks fixed the issue.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 172700fb7784d29f9403003b4484a5ebd7aa316b..159d5307b30a4b37e6cf2941848b8718e1b891d9 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -36465,6 +36465,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
 
 	build_full_tree(mt, 0, 2);
 
+	mas_lock(&mas);
 	/* erase every entry in the tree */
 	do {
 		/* set up bulk store mode */
@@ -36474,6 +36475,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
 	} while (mas_prev(&mas, 0) != NULL);
 
 	mas_destroy(&mas);
+	mas_unlock(&mas);
 }
 
 void farmer_tests(void)

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:50 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:50 +0200
Subject: [PATCH 00/12] maple_tree: slub sheaves conversion
Message-ID: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>

The previous version of this work was part of the patchset that
introduces slub sheaves, but for practical purposes was split out in v6
[1] while Liam did some further fixups and cleanups to the maple tree
and associated testing implementation, incorporating also patches from
Pedro [2] and then provided to me his branch [3] to include it. It
relies on [1] in the slab/for-next tree so the intent is to include this
in slab/for-next as well. More precisely speaking, patches 2 and 3 will
replace the last two patches of [1].

[1] https://lore.kernel.org/all/20250827-slub-percpu-caches-v6-0-f0f775a3f73f at suse.cz/
[2] https://lore.kernel.org/all/20250812162124.59417-1-pfalcato at suse.de/
[2] https://git.infradead.org/?p=users/jedix/linux-maple.git;a=shortlog;h=refs/heads/sheaves-v6r2_fixes

git version:
https://git.kernel.org/vbabka/l/maple-sheaves-v1

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
Liam R. Howlett (8):
      maple_tree: Fix check_bulk_rebalance() test locks
      tools: Add testing support for changes to slab for sheaves
      tools/testing/vma: Implement vm_refcnt reset
      testing/radix-tree/maple: Hack around kfree_rcu not existing
      tools: Add sheaf to slab testing
      maple_tree: Sheaf conversion and testing
      maple_tree: Add single node allocation support to maple state
      maple_tree: Convert forking to use the sheaf interface

Pedro Falcato (2):
      maple_tree: Use kfree_rcu in ma_free_rcu
      maple_tree: Replace mt_free_one() with kfree()

Vlastimil Babka (2):
      maple_tree: use percpu sheaves for maple_node_cache
      tools/testing: include maple-shared.h in maple.c

 include/linux/maple_tree.h          |   6 +-
 lib/maple_tree.c                    | 399 +++++++++--------------------
 lib/test_maple_tree.c               |   8 +
 tools/include/linux/slab.h          | 165 +++++++++++-
 tools/testing/radix-tree/maple.c    | 483 +++---------------------------------
 tools/testing/shared/linux.c        | 120 +++++++--
 tools/testing/shared/maple-shared.h |  11 +
 tools/testing/shared/maple-shim.c   |   7 +
 tools/testing/vma/vma_internal.h    |  98 +-------
 9 files changed, 439 insertions(+), 858 deletions(-)
---
base-commit: 2a2cada7250179353220bea9548acd7e7ed96e48
change-id: 20250901-maple-sheaves-63b9edc69bff

Best regards,
-- 
Vlastimil Babka <vbabka at suse.cz>



From vbabka at suse.cz  Mon Sep  1 04:08:53 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:53 +0200
Subject: [PATCH 03/12] maple_tree: use percpu sheaves for maple_node_cache
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-3-d6a1166b53f2@suse.cz>

Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
improve its performance. Note this will not immediately take advantage
of sheaf batching of kfree_rcu() operations due to the maple tree using
call_rcu with custom callbacks. The followup changes to maple tree will
change that and also make use of the prefilled sheaves functionality.

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
---
 lib/maple_tree.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a962ca374467d0533185f2db3d35ff..a0db6bdc63793b8bbd544e246391d99e880dede3 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -6302,9 +6302,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 
 void __init maple_tree_init(void)
 {
+	struct kmem_cache_args args = {
+		.align  = sizeof(struct maple_node),
+		.sheaf_capacity = 32,
+	};
+
 	maple_node_cache = kmem_cache_create("maple_node",
-			sizeof(struct maple_node), sizeof(struct maple_node),
-			SLAB_PANIC, NULL);
+			sizeof(struct maple_node), &args,
+			SLAB_PANIC);
 }
 
 /**

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:52 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:52 +0200
Subject: [PATCH 02/12] tools: Add testing support for changes to slab for
 sheaves
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-2-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The slab changes for sheaves requires more effort in the testing code.
Unite all the kmem_cache work into the tools/include slab header for
both the vma and maple tree testing.

The vma test code also requires importing more #defines to allow for
seamless use of the shared kmem_cache code.

This adds the pthread header to the slab header in the tools directory
to allow for the pthread_mutex in linux.c.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h        | 137 ++++++++++++++++++++++++++++++++++++--
 tools/testing/shared/linux.c      |  26 ++------
 tools/testing/shared/maple-shim.c |   1 +
 tools/testing/vma/vma_internal.h  |  94 +-------------------------
 4 files changed, 142 insertions(+), 116 deletions(-)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c87051e2b26f5a7fee0362697fae067076b8e84d..c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -4,11 +4,31 @@
 
 #include <linux/types.h>
 #include <linux/gfp.h>
+#include <pthread.h>
 
-#define SLAB_PANIC 2
 #define SLAB_RECLAIM_ACCOUNT    0x00020000UL            /* Objects are reclaimable */
 
 #define kzalloc_node(size, flags, node) kmalloc(size, flags)
+enum _slab_flag_bits {
+	_SLAB_KMALLOC,
+	_SLAB_HWCACHE_ALIGN,
+	_SLAB_PANIC,
+	_SLAB_TYPESAFE_BY_RCU,
+	_SLAB_ACCOUNT,
+	_SLAB_FLAGS_LAST_BIT
+};
+
+#define __SLAB_FLAG_BIT(nr)	((unsigned int __force)(1U << (nr)))
+#define __SLAB_FLAG_UNUSED	((unsigned int __force)(0U))
+
+#define SLAB_HWCACHE_ALIGN	__SLAB_FLAG_BIT(_SLAB_HWCACHE_ALIGN)
+#define SLAB_PANIC		__SLAB_FLAG_BIT(_SLAB_PANIC)
+#define SLAB_TYPESAFE_BY_RCU	__SLAB_FLAG_BIT(_SLAB_TYPESAFE_BY_RCU)
+#ifdef CONFIG_MEMCG
+# define SLAB_ACCOUNT		__SLAB_FLAG_BIT(_SLAB_ACCOUNT)
+#else
+# define SLAB_ACCOUNT		__SLAB_FLAG_UNUSED
+#endif
 
 void *kmalloc(size_t size, gfp_t gfp);
 void kfree(void *p);
@@ -23,6 +43,86 @@ enum slab_state {
 	FULL
 };
 
+struct kmem_cache {
+	pthread_mutex_t lock;
+	unsigned int size;
+	unsigned int align;
+	unsigned int sheaf_capacity;
+	int nr_objs;
+	void *objs;
+	void (*ctor)(void *);
+	bool non_kernel_enabled;
+	unsigned int non_kernel;
+	unsigned long nr_allocated;
+	unsigned long nr_tallocated;
+	bool exec_callback;
+	void (*callback)(void *);
+	void *private;
+};
+
+struct kmem_cache_args {
+	/**
+	 * @align: The required alignment for the objects.
+	 *
+	 * %0 means no specific alignment is requested.
+	 */
+	unsigned int align;
+	/**
+	 * @sheaf_capacity: The maximum size of the sheaf.
+	 */
+	unsigned int sheaf_capacity;
+	/**
+	 * @useroffset: Usercopy region offset.
+	 *
+	 * %0 is a valid offset, when @usersize is non-%0
+	 */
+	unsigned int useroffset;
+	/**
+	 * @usersize: Usercopy region size.
+	 *
+	 * %0 means no usercopy region is specified.
+	 */
+	unsigned int usersize;
+	/**
+	 * @freeptr_offset: Custom offset for the free pointer
+	 * in &SLAB_TYPESAFE_BY_RCU caches
+	 *
+	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
+	 * outside of the object. This might cause the object to grow in size.
+	 * Cache creators that have a reason to avoid this can specify a custom
+	 * free pointer offset in their struct where the free pointer will be
+	 * placed.
+	 *
+	 * Note that placing the free pointer inside the object requires the
+	 * caller to ensure that no fields are invalidated that are required to
+	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
+	 * details).
+	 *
+	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
+	 * is specified, %use_freeptr_offset must be set %true.
+	 *
+	 * Note that @ctor currently isn't supported with custom free pointers
+	 * as a @ctor requires an external free pointer.
+	 */
+	unsigned int freeptr_offset;
+	/**
+	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
+	 */
+	bool use_freeptr_offset;
+	/**
+	 * @ctor: A constructor for the objects.
+	 *
+	 * The constructor is invoked for each object in a newly allocated slab
+	 * page. It is the cache user's responsibility to free object in the
+	 * same state as after calling the constructor, or deal appropriately
+	 * with any differences between a freshly constructed and a reallocated
+	 * object.
+	 *
+	 * %NULL means no constructor.
+	 */
+	void (*ctor)(void *);
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -37,9 +137,38 @@ static inline void *kmem_cache_alloc(struct kmem_cache *cachep, int flags)
 }
 void kmem_cache_free(struct kmem_cache *cachep, void *objp);
 
-struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
-			unsigned int align, unsigned int flags,
-			void (*ctor)(void *));
+
+struct kmem_cache *
+__kmem_cache_create_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags);
+
+/* If NULL is passed for @args, use this variant with default arguments. */
+static inline struct kmem_cache *
+__kmem_cache_default_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags)
+{
+	struct kmem_cache_args kmem_default_args = {};
+
+	return __kmem_cache_create_args(name, size, &kmem_default_args, flags);
+}
+
+static inline struct kmem_cache *
+__kmem_cache_create(const char *name, unsigned int size, unsigned int align,
+		unsigned int flags, void (*ctor)(void *))
+{
+	struct kmem_cache_args kmem_args = {
+		.align	= align,
+		.ctor	= ctor,
+	};
+
+	return __kmem_cache_create_args(name, size, &kmem_args, flags);
+}
+
+#define kmem_cache_create(__name, __object_size, __args, ...)           \
+	_Generic((__args),                                              \
+		struct kmem_cache_args *: __kmem_cache_create_args,	\
+		void *: __kmem_cache_default_args,			\
+		default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__)
 
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 0f97fb0d19e19c327aa4843a35b45cc086f4f366..97b8412ccbb6d222604c7b397c53c65618d8d51b 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -16,21 +16,6 @@ int nr_allocated;
 int preempt_count;
 int test_verbose;
 
-struct kmem_cache {
-	pthread_mutex_t lock;
-	unsigned int size;
-	unsigned int align;
-	int nr_objs;
-	void *objs;
-	void (*ctor)(void *);
-	unsigned int non_kernel;
-	unsigned long nr_allocated;
-	unsigned long nr_tallocated;
-	bool exec_callback;
-	void (*callback)(void *);
-	void *private;
-};
-
 void kmem_cache_set_callback(struct kmem_cache *cachep, void (*callback)(void *))
 {
 	cachep->callback = callback;
@@ -234,23 +219,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 }
 
 struct kmem_cache *
-kmem_cache_create(const char *name, unsigned int size, unsigned int align,
-		unsigned int flags, void (*ctor)(void *))
+__kmem_cache_create_args(const char *name, unsigned int size,
+			  struct kmem_cache_args *args,
+			  unsigned int flags)
 {
 	struct kmem_cache *ret = malloc(sizeof(*ret));
 
 	pthread_mutex_init(&ret->lock, NULL);
 	ret->size = size;
-	ret->align = align;
+	ret->align = args->align;
+	ret->sheaf_capacity = args->sheaf_capacity;
 	ret->nr_objs = 0;
 	ret->nr_allocated = 0;
 	ret->nr_tallocated = 0;
 	ret->objs = NULL;
-	ret->ctor = ctor;
+	ret->ctor = args->ctor;
 	ret->non_kernel = 0;
 	ret->exec_callback = false;
 	ret->callback = NULL;
 	ret->private = NULL;
+
 	return ret;
 }
 
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 640df76f483e09f3b6f85612786060dd273e2362..9d7b743415660305416e972fa75b56824211b0eb 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -3,5 +3,6 @@
 /* Very simple shim around the maple tree. */
 
 #include "maple-shared.h"
+#include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 3639aa8dd2b06ebe5b9cfcfe6669994fd38c482d..09732eff8dd84555563b3d485805ebab7b204584 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -26,6 +26,7 @@
 #include <linux/mm.h>
 #include <linux/rbtree.h>
 #include <linux/refcount.h>
+#include <linux/slab.h>
 
 extern unsigned long stack_guard_gap;
 #ifdef CONFIG_MMU
@@ -509,65 +510,6 @@ struct pagetable_move_control {
 		.len_in = len_,						\
 	}
 
-struct kmem_cache_args {
-	/**
-	 * @align: The required alignment for the objects.
-	 *
-	 * %0 means no specific alignment is requested.
-	 */
-	unsigned int align;
-	/**
-	 * @useroffset: Usercopy region offset.
-	 *
-	 * %0 is a valid offset, when @usersize is non-%0
-	 */
-	unsigned int useroffset;
-	/**
-	 * @usersize: Usercopy region size.
-	 *
-	 * %0 means no usercopy region is specified.
-	 */
-	unsigned int usersize;
-	/**
-	 * @freeptr_offset: Custom offset for the free pointer
-	 * in &SLAB_TYPESAFE_BY_RCU caches
-	 *
-	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
-	 * outside of the object. This might cause the object to grow in size.
-	 * Cache creators that have a reason to avoid this can specify a custom
-	 * free pointer offset in their struct where the free pointer will be
-	 * placed.
-	 *
-	 * Note that placing the free pointer inside the object requires the
-	 * caller to ensure that no fields are invalidated that are required to
-	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
-	 * details).
-	 *
-	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
-	 * is specified, %use_freeptr_offset must be set %true.
-	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
-	 */
-	unsigned int freeptr_offset;
-	/**
-	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
-	 */
-	bool use_freeptr_offset;
-	/**
-	 * @ctor: A constructor for the objects.
-	 *
-	 * The constructor is invoked for each object in a newly allocated slab
-	 * page. It is the cache user's responsibility to free object in the
-	 * same state as after calling the constructor, or deal appropriately
-	 * with any differences between a freshly constructed and a reallocated
-	 * object.
-	 *
-	 * %NULL means no constructor.
-	 */
-	void (*ctor)(void *);
-};
-
 static inline void vma_iter_invalidate(struct vma_iterator *vmi)
 {
 	mas_pause(&vmi->mas);
@@ -652,40 +594,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_lock_seq = UINT_MAX;
 }
 
-struct kmem_cache {
-	const char *name;
-	size_t object_size;
-	struct kmem_cache_args *args;
-};
-
-static inline struct kmem_cache *__kmem_cache_create(const char *name,
-						     size_t object_size,
-						     struct kmem_cache_args *args)
-{
-	struct kmem_cache *ret = malloc(sizeof(struct kmem_cache));
-
-	ret->name = name;
-	ret->object_size = object_size;
-	ret->args = args;
-
-	return ret;
-}
-
-#define kmem_cache_create(__name, __object_size, __args, ...)           \
-	__kmem_cache_create((__name), (__object_size), (__args))
-
-static inline void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
-{
-	(void)gfpflags;
-
-	return calloc(s->object_size, 1);
-}
-
-static inline void kmem_cache_free(struct kmem_cache *s, void *x)
-{
-	free(x);
-}
-
 /*
  * These are defined in vma.h, but sadly vm_stat_account() is referenced by
  * kernel/fork.c, so we have to these broadly available there, and temporarily

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:54 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:54 +0200
Subject: [PATCH 04/12] tools/testing: include maple-shared.h in maple.c
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-4-d6a1166b53f2@suse.cz>

There's code duplication and we are about to add more functionality in
maple-shared.h that we will need in userspace maple test to be
available.

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c | 11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 159d5307b30a4b37e6cf2941848b8718e1b891d9..18db97a916f039bf72046c3ec3e7faffaeb5b755 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -8,14 +8,6 @@
  * difficult to handle in kernel tests.
  */
 
-#define CONFIG_DEBUG_MAPLE_TREE
-#define CONFIG_MAPLE_SEARCH
-#define MAPLE_32BIT (MAPLE_NODE_SLOTS > 31)
-#include "test.h"
-#include <stdlib.h>
-#include <time.h>
-#include <linux/init.h>
-
 #define module_init(x)
 #define module_exit(x)
 #define MODULE_AUTHOR(x)
@@ -23,6 +15,9 @@
 #define MODULE_LICENSE(x)
 #define dump_stack()	assert(0)
 
+#include "maple-shared.h"
+#include "test.h"
+
 #include "../../../lib/maple_tree.c"
 #include "../../../lib/test_maple_tree.c"
 

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:55 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:55 +0200
Subject: [PATCH 05/12] tools/testing/vma: Implement vm_refcnt reset
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-5-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the reset of the ref count in vma_lock_init().  This is needed if
the vma memory is not zeroed on allocation.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/vma/vma_internal.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 09732eff8dd84555563b3d485805ebab7b204584..972ab2686e0a3654cef611ce9f3409bc0c38dc80 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -1326,8 +1326,8 @@ static inline void ksm_exit(struct mm_struct *mm)
 
 static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)
 {
-	(void)vma;
-	(void)reset_refcnt;
+	if (reset_refcnt)
+		refcount_set(&vma->vm_refcnt, 0);
 }
 
 static inline void vma_numab_state_init(struct vm_area_struct *vma)

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:56 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:56 +0200
Subject: [PATCH 06/12] testing/radix-tree/maple: Hack around kfree_rcu not
 existing
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-6-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

liburcu doesn't have kfree_rcu (or anything similar). Despite that, we
can hack around it in a trivial fashion, by adding a wrapper.

The wrapper only works for maple_nodes because we cannot get the
kmem_cache pointer any other way in the test code.

Link: https://lore.kernel.org/all/20250812162124.59417-1-pfalcato at suse.de/
Suggested-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c    |  3 +--
 tools/testing/shared/maple-shared.h | 11 +++++++++++
 tools/testing/shared/maple-shim.c   |  6 ++++++
 3 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 18db97a916f039bf72046c3ec3e7faffaeb5b755..7fe91f24849b35723ec6aadbe45ec7d2abedcc11 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -15,10 +15,9 @@
 #define MODULE_LICENSE(x)
 #define dump_stack()	assert(0)
 
-#include "maple-shared.h"
 #include "test.h"
 
-#include "../../../lib/maple_tree.c"
+#include "../shared/maple-shim.c"
 #include "../../../lib/test_maple_tree.c"
 
 #define RCU_RANGE_COUNT 1000
diff --git a/tools/testing/shared/maple-shared.h b/tools/testing/shared/maple-shared.h
index dc4d30f3860b9bd23b4177c7d7926ac686887815..2a1e9a8594a2834326cd9374738b2a2c7c3f9f7c 100644
--- a/tools/testing/shared/maple-shared.h
+++ b/tools/testing/shared/maple-shared.h
@@ -10,4 +10,15 @@
 #include <time.h>
 #include "linux/init.h"
 
+void maple_rcu_cb(struct rcu_head *head);
+#define rcu_cb		maple_rcu_cb
+
+#define kfree_rcu(_struct, _memb)		\
+do {                                            \
+    typeof(_struct) _p_struct = (_struct);      \
+                                                \
+    call_rcu(&((_p_struct)->_memb), rcu_cb);    \
+} while(0);
+
+
 #endif /* __MAPLE_SHARED_H__ */
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 9d7b743415660305416e972fa75b56824211b0eb..16252ee616c0489c80490ff25b8d255427bf9fdc 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -6,3 +6,9 @@
 #include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
+
+void maple_rcu_cb(struct rcu_head *head) {
+	struct maple_node *node = container_of(head, struct maple_node, rcu);
+
+	kmem_cache_free(maple_node_cache, node);
+}

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:57 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:57 +0200
Subject: [PATCH 07/12] maple_tree: Use kfree_rcu in ma_free_rcu
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-7-d6a1166b53f2@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
possible to call it on non-kmalloc objects, but this restriction was
lifted ever since SLOB was dropped from the kernel, and since commit
6c6c47b063b5 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").

Thus, replace call_rcu + mt_free_rcu with kfree_rcu.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index a0db6bdc63793b8bbd544e246391d99e880dede3..d77e82362f03905040ac61630f92fe9af1e59f98 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -191,13 +191,6 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
-static void mt_free_rcu(struct rcu_head *head)
-{
-	struct maple_node *node = container_of(head, struct maple_node, rcu);
-
-	kmem_cache_free(maple_node_cache, node);
-}
-
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -208,7 +201,7 @@ static void mt_free_rcu(struct rcu_head *head)
 static void ma_free_rcu(struct maple_node *node)
 {
 	WARN_ON(node->parent != ma_parent_ptr(node));
-	call_rcu(&node->rcu, mt_free_rcu);
+	kfree_rcu(node, rcu);
 }
 
 static void mt_set_height(struct maple_tree *mt, unsigned char height)
@@ -5281,7 +5274,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_rcu(&node->rcu);
+	mt_free_one(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5365,7 +5358,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_rcu(&node->rcu);
+		mt_free_one(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:59 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:59 +0200
Subject: [PATCH 09/12] tools: Add sheaf to slab testing
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-9-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the sheaf structs to the slab header and the functions to the
testing/shared/linux.c file.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h   | 28 ++++++++++++++
 tools/testing/shared/linux.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08..94937a699402bd1f31887dfb52b6fd0a3c986f43 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -123,6 +123,18 @@ struct kmem_cache_args {
 	void (*ctor)(void *);
 };
 
+struct slab_sheaf {
+	union {
+		struct list_head barn_list;
+		/* only used for prefilled sheafs */
+		unsigned int capacity;
+	};
+	struct kmem_cache *cache;
+	unsigned int size;
+	int node; /* only used for rcu_sheaf */
+	void *objects[];
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -173,5 +185,21 @@ __kmem_cache_create(const char *name, unsigned int size, unsigned int align,
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 			  void **list);
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf **sheafp, unsigned int size);
+
+static inline unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
+{
+	return sheaf->size;
+}
 
 #endif		/* _TOOLS_SLAB_H */
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 97b8412ccbb6d222604c7b397c53c65618d8d51b..4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -137,6 +137,12 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
 	if (kmalloc_verbose)
 		pr_debug("Bulk free %p[0-%zu]\n", list, size - 1);
 
+	if (cachep->exec_callback) {
+		if (cachep->callback)
+			cachep->callback(cachep->private);
+		cachep->exec_callback = false;
+	}
+
 	pthread_mutex_lock(&cachep->lock);
 	for (int i = 0; i < size; i++)
 		kmem_cache_free_locked(cachep, list[i]);
@@ -242,6 +248,89 @@ __kmem_cache_create_args(const char *name, unsigned int size,
 	return ret;
 }
 
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
+{
+	struct slab_sheaf *sheaf;
+	unsigned int capacity;
+
+	if (s->exec_callback) {
+		if (s->callback)
+			s->callback(s->private);
+		s->exec_callback = false;
+	}
+
+	capacity = max(size, s->sheaf_capacity);
+
+	sheaf = calloc(1, sizeof(*sheaf) + sizeof(void *) * capacity);
+	if (!sheaf)
+		return NULL;
+
+	sheaf->cache = s;
+	sheaf->capacity = capacity;
+	sheaf->size = kmem_cache_alloc_bulk(s, gfp, size, sheaf->objects);
+	if (!sheaf->size) {
+		free(sheaf);
+		return NULL;
+	}
+
+	return sheaf;
+}
+
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf **sheafp, unsigned int size)
+{
+	struct slab_sheaf *sheaf = *sheafp;
+	int refill;
+
+	if (sheaf->size >= size)
+		return 0;
+
+	if (size > sheaf->capacity) {
+		sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
+		if (!sheaf)
+			return -ENOMEM;
+
+		kmem_cache_return_sheaf(s, gfp, *sheafp);
+		*sheafp = sheaf;
+		return 0;
+	}
+
+	refill = kmem_cache_alloc_bulk(s, gfp, size - sheaf->size,
+				       &sheaf->objects[sheaf->size]);
+	if (!refill)
+		return -ENOMEM;
+
+	sheaf->size += refill;
+	return 0;
+}
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf *sheaf)
+{
+	if (sheaf->size)
+		kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
+
+	free(sheaf);
+}
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf)
+{
+	void *obj;
+
+	if (sheaf->size == 0) {
+		printf("Nothing left in sheaf!\n");
+		return NULL;
+	}
+
+	obj = sheaf->objects[--sheaf->size];
+	sheaf->objects[sheaf->size] = NULL;
+
+	return obj;
+}
+
 /*
  * Test the test infrastructure for kem_cache_alloc/free and bulk counterparts.
  */

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:08:58 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:08:58 +0200
Subject: [PATCH 08/12] maple_tree: Replace mt_free_one() with kfree()
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-8-d6a1166b53f2@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree() is a little shorter and works with kmem_cache_alloc'd pointers
too. Also lets us remove one more helper.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index d77e82362f03905040ac61630f92fe9af1e59f98..b361b484cfcaacd99472dd4c2b8de9260b307425 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -181,11 +181,6 @@ static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
 	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
 }
 
-static inline void mt_free_one(struct maple_node *node)
-{
-	kmem_cache_free(maple_node_cache, node);
-}
-
 static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 {
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
@@ -5274,7 +5269,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_one(node);
+	kfree(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5358,7 +5353,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_one(node);
+		kfree(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }
@@ -5585,7 +5580,7 @@ void mas_destroy(struct ma_state *mas)
 			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
 			total -= count;
 		}
-		mt_free_one(ma_mnode_ptr(node));
+		kfree(ma_mnode_ptr(node));
 		total--;
 	}
 
@@ -6635,7 +6630,7 @@ static void mas_dup_free(struct ma_state *mas)
 	}
 
 	node = mte_to_node(mas->node);
-	mt_free_one(node);
+	kfree(node);
 }
 
 /*

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:09:00 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:09:00 +0200
Subject: [PATCH 10/12] maple_tree: Sheaf conversion and testing
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-10-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Use sheaves instead of bulk allocations.  This should speed up the
allocations and the return path of unused allocations.

Remove the push and pop of nodes from the maple state as this is now
handled by the slab layer with sheaves.

Testing has been removed as necessary since the features of the tree
have been reduced.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |   6 +-
 lib/maple_tree.c                 | 329 ++++++---------------------
 lib/test_maple_tree.c            |   8 +
 tools/testing/radix-tree/maple.c | 464 ++-------------------------------------
 tools/testing/shared/linux.c     |   5 +-
 5 files changed, 99 insertions(+), 713 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -442,7 +442,8 @@ struct ma_state {
 	struct maple_enode *node;	/* The node containing this entry */
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
-	struct maple_alloc *alloc;	/* Allocated nodes for this operation */
+	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
 	unsigned char offset;
@@ -490,7 +491,8 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.alloc = NULL,						\
+		.node_request= 0,					\
+		.sheaf = NULL,						\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b361b484cfcaacd99472dd4c2b8de9260b307425..cfe80c50a97e3118eefc24275fbcd2eec5e6e6e8 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -186,6 +186,22 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
+static void mt_return_sheaf(struct slab_sheaf *sheaf)
+{
+	kmem_cache_return_sheaf(maple_node_cache, GFP_KERNEL, sheaf);
+}
+
+static struct slab_sheaf *mt_get_sheaf(gfp_t gfp, int count)
+{
+	return kmem_cache_prefill_sheaf(maple_node_cache, gfp, count);
+}
+
+static int mt_refill_sheaf(gfp_t gfp, struct slab_sheaf **sheaf,
+		unsigned int size)
+{
+	return kmem_cache_refill_sheaf(maple_node_cache, gfp, sheaf, size);
+}
+
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -578,67 +594,6 @@ static __always_inline bool mte_dead_node(const struct maple_enode *enode)
 	return ma_dead_node(node);
 }
 
-/*
- * mas_allocated() - Get the number of nodes allocated in a maple state.
- * @mas: The maple state
- *
- * The ma_state alloc member is overloaded to hold a pointer to the first
- * allocated node or to the number of requested nodes to allocate.  If bit 0 is
- * set, then the alloc contains the number of requested nodes.  If there is an
- * allocated node, then the total allocated nodes is in that node.
- *
- * Return: The total number of nodes allocated
- */
-static inline unsigned long mas_allocated(const struct ma_state *mas)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1))
-		return 0;
-
-	return mas->alloc->total;
-}
-
-/*
- * mas_set_alloc_req() - Set the requested number of allocations.
- * @mas: the maple state
- * @count: the number of allocations.
- *
- * The requested number of allocations is either in the first allocated node,
- * located in @mas->alloc->request_count, or directly in @mas->alloc if there is
- * no allocated node.  Set the request either in the node or do the necessary
- * encoding to store in @mas->alloc directly.
- */
-static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
-		if (!count)
-			mas->alloc = NULL;
-		else
-			mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
-		return;
-	}
-
-	mas->alloc->request_count = count;
-}
-
-/*
- * mas_alloc_req() - get the requested number of allocations.
- * @mas: The maple state
- *
- * The alloc count is either stored directly in @mas, or in
- * @mas->alloc->request_count if there is at least one node allocated.  Decode
- * the request count if it's stored directly in @mas->alloc.
- *
- * Return: The allocation request count.
- */
-static inline unsigned int mas_alloc_req(const struct ma_state *mas)
-{
-	if ((unsigned long)mas->alloc & 0x1)
-		return (unsigned long)(mas->alloc) >> 1;
-	else if (mas->alloc)
-		return mas->alloc->request_count;
-	return 0;
-}
-
 /*
  * ma_pivots() - Get a pointer to the maple node pivots.
  * @node: the maple node
@@ -1142,77 +1097,15 @@ static int mas_ascend(struct ma_state *mas)
  */
 static inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
-	struct maple_alloc *ret, *node = mas->alloc;
-	unsigned long total = mas_allocated(mas);
-	unsigned int req = mas_alloc_req(mas);
+	struct maple_node *ret;
 
-	/* nothing or a request pending. */
-	if (WARN_ON(!total))
+	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
-	if (total == 1) {
-		/* single allocation in this ma_state */
-		mas->alloc = NULL;
-		ret = node;
-		goto single_node;
-	}
-
-	if (node->node_count == 1) {
-		/* Single allocation in this node. */
-		mas->alloc = node->slot[0];
-		mas->alloc->total = node->total - 1;
-		ret = node;
-		goto new_head;
-	}
-	node->total--;
-	ret = node->slot[--node->node_count];
-	node->slot[node->node_count] = NULL;
-
-single_node:
-new_head:
-	if (req) {
-		req++;
-		mas_set_alloc_req(mas, req);
-	}
-
+	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
 	memset(ret, 0, sizeof(*ret));
-	return (struct maple_node *)ret;
-}
-
-/*
- * mas_push_node() - Push a node back on the maple state allocation.
- * @mas: The maple state
- * @used: The used maple node
- *
- * Stores the maple node back into @mas->alloc for reuse.  Updates allocated and
- * requested node count as necessary.
- */
-static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
-{
-	struct maple_alloc *reuse = (struct maple_alloc *)used;
-	struct maple_alloc *head = mas->alloc;
-	unsigned long count;
-	unsigned int requested = mas_alloc_req(mas);
 
-	count = mas_allocated(mas);
-
-	reuse->request_count = 0;
-	reuse->node_count = 0;
-	if (count) {
-		if (head->node_count < MAPLE_ALLOC_SLOTS) {
-			head->slot[head->node_count++] = reuse;
-			head->total++;
-			goto done;
-		}
-		reuse->slot[0] = head;
-		reuse->node_count = 1;
-	}
-
-	reuse->total = count + 1;
-	mas->alloc = reuse;
-done:
-	if (requested > 1)
-		mas_set_alloc_req(mas, requested - 1);
+	return ret;
 }
 
 /*
@@ -1222,75 +1115,32 @@ static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	struct maple_alloc *node;
-	unsigned long allocated = mas_allocated(mas);
-	unsigned int requested = mas_alloc_req(mas);
-	unsigned int count;
-	void **slots = NULL;
-	unsigned int max_req = 0;
-
-	if (!requested)
-		return;
+	if (unlikely(mas->sheaf)) {
+		unsigned long refill = mas->node_request;
 
-	mas_set_alloc_req(mas, 0);
-	if (mas->mas_flags & MA_STATE_PREALLOC) {
-		if (allocated)
+		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
+			mas->node_request = 0;
 			return;
-		WARN_ON(!allocated);
-	}
-
-	if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) {
-		node = (struct maple_alloc *)mt_alloc_one(gfp);
-		if (!node)
-			goto nomem_one;
-
-		if (allocated) {
-			node->slot[0] = mas->alloc;
-			node->node_count = 1;
-		} else {
-			node->node_count = 0;
 		}
 
-		mas->alloc = node;
-		node->total = ++allocated;
-		node->request_count = 0;
-		requested--;
-	}
+		if (mt_refill_sheaf(gfp, &mas->sheaf, refill))
+			goto error;
 
-	node = mas->alloc;
-	while (requested) {
-		max_req = MAPLE_ALLOC_SLOTS - node->node_count;
-		slots = (void **)&node->slot[node->node_count];
-		max_req = min(requested, max_req);
-		count = mt_alloc_bulk(gfp, max_req, slots);
-		if (!count)
-			goto nomem_bulk;
-
-		if (node->node_count == 0) {
-			node->slot[0]->node_count = 0;
-			node->slot[0]->request_count = 0;
-		}
+		mas->node_request = 0;
+		return;
+	}
 
-		node->node_count += count;
-		allocated += count;
-		/* find a non-full node*/
-		do {
-			node = node->slot[0];
-		} while (unlikely(node->node_count == MAPLE_ALLOC_SLOTS));
-		requested -= count;
+	mas->sheaf = mt_get_sheaf(gfp, mas->node_request);
+	if (likely(mas->sheaf)) {
+		mas->node_request = 0;
+		return;
 	}
-	mas->alloc->total = allocated;
-	return;
 
-nomem_bulk:
-	/* Clean up potential freed allocations on bulk failure */
-	memset(slots, 0, max_req * sizeof(unsigned long));
-	mas->alloc->total = allocated;
-nomem_one:
-	mas_set_alloc_req(mas, requested);
+error:  
 	mas_set_err(mas, -ENOMEM);
 }
 
+
 /*
  * mas_free() - Free an encoded maple node
  * @mas: The maple state
@@ -1301,42 +1151,7 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
  */
 static inline void mas_free(struct ma_state *mas, struct maple_enode *used)
 {
-	struct maple_node *tmp = mte_to_node(used);
-
-	if (mt_in_rcu(mas->tree))
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
-}
-
-/*
- * mas_node_count_gfp() - Check if enough nodes are allocated and request more
- * if there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- * @gfp: the gfp flags
- */
-static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
-{
-	unsigned long allocated = mas_allocated(mas);
-
-	if (allocated < count) {
-		mas_set_alloc_req(mas, count - allocated);
-		mas_alloc_nodes(mas, gfp);
-	}
-}
-
-/*
- * mas_node_count() - Check if enough nodes are allocated and request more if
- * there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- *
- * Note: Uses GFP_NOWAIT | __GFP_NOWARN for gfp flags.
- */
-static void mas_node_count(struct ma_state *mas, int count)
-{
-	return mas_node_count_gfp(mas, count, GFP_NOWAIT | __GFP_NOWARN);
+	ma_free_rcu(mte_to_node(used));
 }
 
 /*
@@ -2511,10 +2326,7 @@ static inline void mas_topiary_node(struct ma_state *mas,
 	enode = tmp_mas->node;
 	tmp = mte_to_node(enode);
 	mte_set_node_dead(enode);
-	if (in_rcu)
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
+	ma_free_rcu(tmp);
 }
 
 /*
@@ -4162,7 +3974,7 @@ static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas)
  *
  * Return: Number of nodes required for preallocation.
  */
-static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
+static inline void mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 {
 	struct ma_state *mas = wr_mas->mas;
 	unsigned char height = mas_mt_height(mas);
@@ -4208,7 +4020,7 @@ static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 		WARN_ON_ONCE(1);
 	}
 
-	return ret;
+	mas->node_request = ret;
 }
 
 /*
@@ -4269,15 +4081,15 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
  */
 static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry)
 {
-	int request;
+	struct ma_state *mas = wr_mas->mas;
 
 	mas_wr_prealloc_setup(wr_mas);
-	wr_mas->mas->store_type = mas_wr_store_type(wr_mas);
-	request = mas_prealloc_calc(wr_mas, entry);
-	if (!request)
+	mas->store_type = mas_wr_store_type(wr_mas);
+	mas_prealloc_calc(wr_mas, entry);
+	if (!mas->node_request)
 		return;
 
-	mas_node_count(wr_mas->mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT | __GFP_NOWARN);
 }
 
 /**
@@ -5390,7 +5202,6 @@ static inline void mte_destroy_walk(struct maple_enode *enode,
  */
 void *mas_store(struct ma_state *mas, void *entry)
 {
-	int request;
 	MA_WR_STATE(wr_mas, mas, entry);
 
 	trace_ma_write(__func__, mas, 0, entry);
@@ -5420,11 +5231,11 @@ void *mas_store(struct ma_state *mas, void *entry)
 		return wr_mas.content;
 	}
 
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto store;
 
-	mas_node_count(mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT | __GFP_NOWARN);
 	if (mas_is_err(mas))
 		return NULL;
 
@@ -5512,20 +5323,19 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc);
 int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 {
 	MA_WR_STATE(wr_mas, mas, entry);
-	int ret = 0;
-	int request;
 
 	mas_wr_prealloc_setup(&wr_mas);
 	mas->store_type = mas_wr_store_type(&wr_mas);
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto set_flag;
 
 	mas->mas_flags &= ~MA_STATE_PREALLOC;
-	mas_node_count_gfp(mas, request, gfp);
+	mas_alloc_nodes(mas, gfp);
 	if (mas_is_err(mas)) {
-		mas_set_alloc_req(mas, 0);
-		ret = xa_err(mas->node);
+		int ret = xa_err(mas->node);
+
+		mas->node_request = 0;
 		mas_destroy(mas);
 		mas_reset(mas);
 		return ret;
@@ -5533,7 +5343,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 
 set_flag:
 	mas->mas_flags |= MA_STATE_PREALLOC;
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(mas_preallocate);
 
@@ -5547,9 +5357,6 @@ EXPORT_SYMBOL_GPL(mas_preallocate);
  */
 void mas_destroy(struct ma_state *mas)
 {
-	struct maple_alloc *node;
-	unsigned long total;
-
 	/*
 	 * When using mas_for_each() to insert an expected number of elements,
 	 * it is possible that the number inserted is less than the expected
@@ -5570,21 +5377,11 @@ void mas_destroy(struct ma_state *mas)
 	}
 	mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
 
-	total = mas_allocated(mas);
-	while (total) {
-		node = mas->alloc;
-		mas->alloc = node->slot[0];
-		if (node->node_count > 1) {
-			size_t count = node->node_count - 1;
-
-			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
-			total -= count;
-		}
-		kfree(ma_mnode_ptr(node));
-		total--;
-	}
+	mas->node_request = 0;
+	if (mas->sheaf)
+		mt_return_sheaf(mas->sheaf);
 
-	mas->alloc = NULL;
+	mas->sheaf = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -5634,7 +5431,8 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries)
 	/* Internal nodes */
 	nr_nodes += DIV_ROUND_UP(nr_nodes, nonleaf_cap);
 	/* Add working room for split (2 nodes) + new parents */
-	mas_node_count_gfp(mas, nr_nodes + 3, GFP_KERNEL);
+	mas->node_request = nr_nodes + 3;
+	mas_alloc_nodes(mas, GFP_KERNEL);
 
 	/* Detect if allocations run out */
 	mas->mas_flags |= MA_STATE_PREALLOC;
@@ -6281,7 +6079,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas_allocated(mas))
+	if (!mas->sheaf)
 		return false;
 
 	mas->status = ma_start;
@@ -7676,8 +7474,9 @@ void mas_dump(const struct ma_state *mas)
 
 	pr_err("[%u/%u] index=%lx last=%lx\n", mas->offset, mas->end,
 	       mas->index, mas->last);
-	pr_err("     min=%lx max=%lx alloc=" PTR_FMT ", depth=%u, flags=%x\n",
-	       mas->min, mas->max, mas->alloc, mas->depth, mas->mas_flags);
+	pr_err("     min=%lx max=%lx sheaf=" PTR_FMT ", request %lu depth=%u, flags=%x\n",
+	       mas->min, mas->max, mas->sheaf, mas->node_request, mas->depth,
+	       mas->mas_flags);
 	if (mas->index > mas->last)
 		pr_err("Check index & last\n");
 }
diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c
index cb3936595b0d56a9682ff100eba54693a1427829..1848d127eb50650e7cc2b9dfbb15ed93aa889f01 100644
--- a/lib/test_maple_tree.c
+++ b/lib/test_maple_tree.c
@@ -2746,6 +2746,7 @@ static noinline void __init check_fuzzer(struct maple_tree *mt)
 	mtree_test_erase(mt, ULONG_MAX - 10);
 }
 
+#if 0
 /* duplicate the tree with a specific gap */
 static noinline void __init check_dup_gaps(struct maple_tree *mt,
 				    unsigned long nr_entries, bool zero_start,
@@ -2770,6 +2771,7 @@ static noinline void __init check_dup_gaps(struct maple_tree *mt,
 		mtree_store_range(mt, i*10, (i+1)*10 - gap,
 				  xa_mk_value(i), GFP_KERNEL);
 
+	mt_dump(mt, mt_dump_dec);
 	mt_init_flags(&newmt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN);
 	mt_set_non_kernel(99999);
 	down_write(&newmt_lock);
@@ -2779,9 +2781,12 @@ static noinline void __init check_dup_gaps(struct maple_tree *mt,
 
 	rcu_read_lock();
 	mas_for_each(&mas, tmp, ULONG_MAX) {
+		printk("%lu nodes %lu\n", mas.index,
+		       kmem_cache_sheaf_count(newmas.sheaf));
 		newmas.index = mas.index;
 		newmas.last = mas.last;
 		mas_store(&newmas, tmp);
+		mt_dump(&newmt, mt_dump_dec);
 	}
 	rcu_read_unlock();
 	mas_destroy(&newmas);
@@ -2878,6 +2883,7 @@ static noinline void __init check_dup(struct maple_tree *mt)
 		cond_resched();
 	}
 }
+#endif
 
 static noinline void __init check_bnode_min_spanning(struct maple_tree *mt)
 {
@@ -4077,9 +4083,11 @@ static int __init maple_tree_seed(void)
 	check_fuzzer(&tree);
 	mtree_destroy(&tree);
 
+#if 0
 	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 	check_dup(&tree);
 	mtree_destroy(&tree);
+#endif
 
 	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 	check_bnode_min_spanning(&tree);
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 7fe91f24849b35723ec6aadbe45ec7d2abedcc11..da3e03d73b52162dab6fa5c368ad7b71b9e58521 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -57,430 +57,6 @@ struct rcu_reader_struct {
 	struct rcu_test_struct2 *test;
 };
 
-static int get_alloc_node_count(struct ma_state *mas)
-{
-	int count = 1;
-	struct maple_alloc *node = mas->alloc;
-
-	if (!node || ((unsigned long)node & 0x1))
-		return 0;
-	while (node->node_count) {
-		count += node->node_count;
-		node = node->slot[0];
-	}
-	return count;
-}
-
-static void check_mas_alloc_node_count(struct ma_state *mas)
-{
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 1, GFP_KERNEL);
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 3, GFP_KERNEL);
-	MT_BUG_ON(mas->tree, get_alloc_node_count(mas) != mas->alloc->total);
-	mas_destroy(mas);
-}
-
-/*
- * check_new_node() - Check the creation of new nodes and error path
- * verification.
- */
-static noinline void __init check_new_node(struct maple_tree *mt)
-{
-
-	struct maple_node *mn, *mn2, *mn3;
-	struct maple_alloc *smn;
-	struct maple_node *nodes[100];
-	int i, j, total;
-
-	MA_STATE(mas, mt, 0, 0);
-
-	check_mas_alloc_node_count(&mas);
-
-	/* Try allocating 3 nodes */
-	mtree_lock(mt);
-	mt_set_non_kernel(0);
-	/* request 3 nodes to be allocated. */
-	mas_node_count(&mas, 3);
-	/* Allocation request of 3. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 3);
-	/* Allocate failed. */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	mas_push_node(&mas, mn);
-	mas_reset(&mas);
-	mas_destroy(&mas);
-	mtree_unlock(mt);
-
-
-	/* Try allocating 1 node, then 2 more */
-	mtree_lock(mt);
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	/* Check Allocation request of 1. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	/* Eat the requested node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	mas.status = ma_start;
-	mas_destroy(&mas);
-	/* Allocate 3 nodes, will fail. */
-	mas_node_count(&mas, 3);
-	/* Drop the lock and allocate 3 nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	/* Ensure 3 are allocated. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Allocation request of 0. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 0);
-
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	/* Ensure we counted 3. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Free. */
-	mas_reset(&mas);
-	mas_destroy(&mas);
-
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	/* Check the node is only one node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	MT_BUG_ON(mt, mas.alloc->node_count);
-
-	mas_set_alloc_req(&mas, 2); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 2);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	for (i = 2; i >= 0; i--) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		MT_BUG_ON(mt, !mn);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 64;
-	mas_set_alloc_req(&mas, total); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != total);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	for (i = total; i > 0; i--) {
-		unsigned int e = 0; /* expected node_count */
-
-		if (!MAPLE_32BIT) {
-			if (i >= 35)
-				e = i - 34;
-			else if (i >= 5)
-				e = i - 4;
-			else if (i >= 2)
-				e = i - 1;
-		} else {
-			if (i >= 4)
-				e = i - 3;
-			else if (i >= 1)
-				e = i - 1;
-			else
-				e = 0;
-		}
-
-		MT_BUG_ON(mt, mas.alloc->node_count != e);
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		MT_BUG_ON(mt, mas_allocated(&mas) != i - 1);
-		MT_BUG_ON(mt, !mn);
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 100;
-	for (i = 1; i < total; i++) {
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = i; j > 0; j--) {
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			MT_BUG_ON(mt, !mn);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			nodes[j] = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-
-		while (j) {
-			j--;
-			mas_push_node(&mas, nodes[j]);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-		mas_reset(&mas);
-		MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL));
-		mas_destroy(&mas);
-
-	}
-
-	/* Set allocation request. */
-	total = 500;
-	mas_node_count(&mas, total);
-	/* Drop the lock and allocate the nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	MT_BUG_ON(mt, !mas.alloc);
-	i = 1;
-	smn = mas.alloc;
-	while (i < total) {
-		for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) {
-			i++;
-			MT_BUG_ON(mt, !smn->slot[j]);
-			if (i == total)
-				break;
-		}
-		smn = smn->slot[0]; /* next. */
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != total);
-	mas_reset(&mas);
-	mas_destroy(&mas); /* Free. */
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	for (i = 1; i < 128; i++) {
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = i; j > 0; j--) { /*Free the requests */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	}
-
-	for (i = 1; i < MAPLE_NODE_MASK + 1; i++) {
-		MA_STATE(mas2, mt, 0, 0);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas2, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-		MT_BUG_ON(mt, mas_allocated(&mas2) != i);
-
-		for (j = i; j > 0; j--) { /*Free the requests */
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-			mn = mas_pop_node(&mas2); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas2) != 0);
-	}
-
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	mn = mas_pop_node(&mas); /* get the next node. */
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
-
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	/* Check the limit of pop/push/pop */
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_alloc_req(&mas));
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count  != MAPLE_ALLOC_SLOTS);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-
-	for (i = 3; i < MAPLE_NODE_MASK * 3; i++) {
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put it back */
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mn3 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_push_node(&mas, mn3);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-	}
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 5); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 5);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 2);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 3 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 3 + 2);
-	mas_destroy(&mas);
-
-	mtree_unlock(mt);
-}
-
 /*
  * Check erasing including RCU.
  */
@@ -35452,8 +35028,7 @@ static void check_dfs_preorder(struct maple_tree *mt)
 	mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
 	mas_reset(&mas);
 	mt_zero_nr_tallocated();
-	mt_set_non_kernel(200);
-	mas_expected_entries(&mas, max);
+	mt_set_non_kernel(1000);
 	for (count = 0; count <= max; count++) {
 		mas.index = mas.last = count;
 		mas_store(&mas, xa_mk_value(count));
@@ -35518,6 +35093,13 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 	return vacant_height;
 }
 
+static int mas_allocated(struct ma_state *mas)
+{
+	if (mas->sheaf)
+		return kmem_cache_sheaf_size(mas->sheaf);
+
+	return 0;
+}
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)
 {
@@ -35536,7 +35118,10 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 
 	/* Spanning store */
 	mas_set_range(&mas, 470, 500);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
+
+	mas_wr_preallocate(&wr_mas, ptr);
+	MT_BUG_ON(mt, mas.store_type != wr_spanning_store);
+	MT_BUG_ON(mt, mas_is_err(&mas));
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
 	vacant_height = get_vacant_height(&wr_mas, ptr);
@@ -35546,6 +35131,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	allocated = mas_allocated(&mas);
 	MT_BUG_ON(mt, allocated != 0);
 
+	mas_wr_preallocate(&wr_mas, ptr);
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -35586,20 +35172,6 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	mn->parent = ma_parent_ptr(mn);
 	ma_free_rcu(mn);
 
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	allocated = mas_allocated(&mas);
-	height = mas_mt_height(&mas);
-	vacant_height = get_vacant_height(&wr_mas, ptr);
-	MT_BUG_ON(mt, allocated != 1 + (height - vacant_height) * 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	mas_destroy(&mas);
-	allocated = mas_allocated(&mas);
-	MT_BUG_ON(mt, allocated != 0);
-
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -36400,11 +35972,17 @@ static void check_nomem_writer_race(struct maple_tree *mt)
 	check_load(mt, 6, xa_mk_value(0xC));
 	mtree_unlock(mt);
 
+	mt_set_non_kernel(0);
 	/* test for the same race but with mas_store_gfp() */
 	mtree_store_range(mt, 0, 5, xa_mk_value(0xA), GFP_KERNEL);
 	mtree_store_range(mt, 6, 10, NULL, GFP_KERNEL);
 
 	mas_set_range(&mas, 0, 5);
+
+	/* setup writer 2 that will trigger the race condition */
+	mt_set_private(mt);
+	mt_set_callback(writer2);
+
 	mtree_lock(mt);
 	mas_store_gfp(&mas, NULL, GFP_KERNEL);
 
@@ -36546,10 +36124,6 @@ void farmer_tests(void)
 	check_erase_testset(&tree);
 	mtree_destroy(&tree);
 
-	mt_init_flags(&tree, 0);
-	check_new_node(&tree);
-	mtree_destroy(&tree);
-
 	if (!MAPLE_32BIT) {
 		mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 		check_rcu_simulated(&tree);
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1..8c72571559583759456c2b469a2abc2611117c13 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -64,7 +64,8 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
 
 	if (!(gfp & __GFP_DIRECT_RECLAIM)) {
 		if (!cachep->non_kernel) {
-			cachep->exec_callback = true;
+			if (cachep->callback)
+				cachep->exec_callback = true;
 			return NULL;
 		}
 
@@ -210,6 +211,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 		for (i = 0; i < size; i++)
 			__kmem_cache_free_locked(cachep, p[i]);
 		pthread_mutex_unlock(&cachep->lock);
+		if (cachep->callback)
+			cachep->exec_callback = true;
 		return 0;
 	}
 

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:09:01 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:09:01 +0200
Subject: [PATCH 11/12] maple_tree: Add single node allocation support to
 maple state
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-11-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The fast path through a write will require replacing a single node in
the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
special case the node store operation by just allocating one node in the
maple state.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |  4 +++-
 lib/maple_tree.c                 | 47 +++++++++++++++++++++++++++++++++++-----
 tools/testing/radix-tree/maple.c |  9 ++++++--
 3 files changed, 51 insertions(+), 9 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index 166fd67e00d882b1e6de1f80c1b590bba7497cd3..562a1e9e5132b5b1fa8f8402a7cadd8abb65e323 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -443,6 +443,7 @@ struct ma_state {
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
 	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	struct maple_node *alloc;	/* allocated nodes */
 	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
@@ -491,8 +492,9 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.node_request= 0,					\
 		.sheaf = NULL,						\
+		.alloc = NULL,						\
+		.node_request= 0,					\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index cfe80c50a97e3118eefc24275fbcd2eec5e6e6e8..61a322f945c28f5c3297c506923f00bcce5c7bca 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1095,16 +1095,23 @@ static int mas_ascend(struct ma_state *mas)
  *
  * Return: A pointer to a maple node.
  */
-static inline struct maple_node *mas_pop_node(struct ma_state *mas)
+static __always_inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
 	struct maple_node *ret;
 
+	if (mas->alloc) {
+		ret = mas->alloc;
+		mas->alloc = NULL;
+		goto out;
+	}
+
 	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
 	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
-	memset(ret, 0, sizeof(*ret));
 
+out:
+	memset(ret, 0, sizeof(*ret));
 	return ret;
 }
 
@@ -1115,9 +1122,34 @@ static inline struct maple_node *mas_pop_node(struct ma_state *mas)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	if (unlikely(mas->sheaf)) {
-		unsigned long refill = mas->node_request;
+	if (!mas->node_request)
+		return;
+
+	if (mas->node_request == 1) {
+		if (mas->sheaf)
+			goto use_sheaf;
+
+		if (mas->alloc)
+			return;
 
+		mas->alloc = mt_alloc_one(gfp);
+		if (!mas->alloc)
+			goto error;
+
+		mas->node_request = 0;
+		return;
+	}
+
+use_sheaf:
+	if (unlikely(mas->alloc)) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+
+	if (mas->sheaf) {
+		unsigned long refill;
+
+		refill = mas->node_request;
 		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
 			mas->node_request = 0;
 			return;
@@ -5380,8 +5412,11 @@ void mas_destroy(struct ma_state *mas)
 	mas->node_request = 0;
 	if (mas->sheaf)
 		mt_return_sheaf(mas->sheaf);
-
 	mas->sheaf = NULL;
+
+	if (mas->alloc)
+		kfree(mas->alloc);
+	mas->alloc = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6079,7 +6114,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas->sheaf)
+	if (!mas->sheaf && !mas->alloc)
 		return false;
 
 	mas->status = ma_start;
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index da3e03d73b52162dab6fa5c368ad7b71b9e58521..89da991e12cd97e44971757ddc105ef46c68ea4c 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -35095,10 +35095,15 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 
 static int mas_allocated(struct ma_state *mas)
 {
+	int total = 0;
+
+	if (mas->alloc)
+		total++;
+
 	if (mas->sheaf)
-		return kmem_cache_sheaf_size(mas->sheaf);
+		total += kmem_cache_sheaf_size(mas->sheaf);
 
-	return 0;
+	return total;
 }
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)

-- 
2.51.0



From vbabka at suse.cz  Mon Sep  1 04:09:02 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 01 Sep 2025 13:09:02 +0200
Subject: [PATCH 12/12] maple_tree: Convert forking to use the sheaf
 interface
In-Reply-To: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
Message-ID: <20250901-maple-sheaves-v1-12-d6a1166b53f2@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

Use the generic interface which should result in less bulk allocations
during a forking.

A part of this is to abstract the freeing of the sheaf or maple state
allocations into its own function so mas_destroy() and the tree
duplication code can use the same functionality to return any unused
resources.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 42 +++++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 19 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 61a322f945c28f5c3297c506923f00bcce5c7bca..5ef15e39fda8c7c65035fb7ed125b82dfa52ca69 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1172,6 +1172,19 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 	mas_set_err(mas, -ENOMEM);
 }
 
+static inline void mas_empty_nodes(struct ma_state *mas)
+{
+	mas->node_request = 0;
+	if (mas->sheaf) {
+		mt_return_sheaf(mas->sheaf);
+		mas->sheaf = NULL;
+	}
+
+	if (mas->alloc) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+}
 
 /*
  * mas_free() - Free an encoded maple node
@@ -5408,15 +5421,7 @@ void mas_destroy(struct ma_state *mas)
 		mas->mas_flags &= ~MA_STATE_REBALANCE;
 	}
 	mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
-
-	mas->node_request = 0;
-	if (mas->sheaf)
-		mt_return_sheaf(mas->sheaf);
-	mas->sheaf = NULL;
-
-	if (mas->alloc)
-		kfree(mas->alloc);
-	mas->alloc = NULL;
+	mas_empty_nodes(mas);
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6504,7 +6509,7 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	struct maple_node *node = mte_to_node(mas->node);
 	struct maple_node *new_node = mte_to_node(new_mas->node);
 	enum maple_type type;
-	unsigned char request, count, i;
+	unsigned char count, i;
 	void __rcu **slots;
 	void __rcu **new_slots;
 	unsigned long val;
@@ -6512,20 +6517,17 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	/* Allocate memory for child nodes. */
 	type = mte_node_type(mas->node);
 	new_slots = ma_slots(new_node, type);
-	request = mas_data_end(mas) + 1;
-	count = mt_alloc_bulk(gfp, request, (void **)new_slots);
-	if (unlikely(count < request)) {
-		memset(new_slots, 0, request * sizeof(void *));
-		mas_set_err(mas, -ENOMEM);
+	count = mas->node_request = mas_data_end(mas) + 1;
+	mas_alloc_nodes(mas, gfp);
+	if (unlikely(mas_is_err(mas)))
 		return;
-	}
 
-	/* Restore node type information in slots. */
 	slots = ma_slots(node, type);
 	for (i = 0; i < count; i++) {
 		val = (unsigned long)mt_slot_locked(mas->tree, slots, i);
 		val &= MAPLE_NODE_MASK;
-		((unsigned long *)new_slots)[i] |= val;
+		new_slots[i] = ma_mnode_ptr((unsigned long)mas_pop_node(mas) |
+					    val);
 	}
 }
 
@@ -6579,7 +6581,7 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 			/* Only allocate child nodes for non-leaf nodes. */
 			mas_dup_alloc(mas, new_mas, gfp);
 			if (unlikely(mas_is_err(mas)))
-				return;
+				goto empty_mas;
 		} else {
 			/*
 			 * This is the last leaf node and duplication is
@@ -6612,6 +6614,8 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 	/* Make them the same height */
 	new_mas->tree->ma_flags = mas->tree->ma_flags;
 	rcu_assign_pointer(new_mas->tree->ma_root, root);
+empty_mas:
+	mas_empty_nodes(mas);
 }
 
 /**

-- 
2.51.0



From aliceryhl at google.com  Tue Sep  2 01:35:10 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 02 Sep 2025 08:35:10 +0000
Subject: [PATCH v3 0/3] Add Rust abstraction for Maple Trees
Message-ID: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>

This will be used in the Tyr driver [1] to allocate from the GPU's VA
space that is not owned by userspace, but by the kernel, for kernel GPU
mappings.

Danilo tells me that in nouveau, the maple tree is used for keeping
track of "VM regions" on top of GPUVM, and that he will most likely end
up doing the same in the Rust Nova driver as well.

These abstractions intentionally do not expose any way to make use of
external locking. You are required to use the internal spinlock. For
now, we do not support loads that only utilize rcu for protection.

This contains some parts taken from Andrew Ballance's RFC [2] from
April. However, it has also been reworked significantly compared to that
RFC taking the use-cases in Tyr into account.

[1]: https://lore.kernel.org/r/20250627-tyr-v1-1-cb5f4c6ced46 at collabora.com
[2]: https://lore.kernel.org/r/20250405060154.1550858-1-andrewjballance at gmail.com

Signed-off-by: Alice Ryhl <aliceryhl at google.com>
---
Changes in v3:
- Change examples to avoid unwrap(), but not unwrap_err().
- Remove spurious MAINTAINERS change.
- Fix unnnecessary imports.
- Drop lockdep_map_p patch from this series.
- Reword MaState type invariant.
- Various docs improvements.
- Rename mas_find() to find().
- Add to existing MAINTAINERS entry.
- Link to v2: https://lore.kernel.org/r/20250819-maple-tree-v2-0-229b48657bab at google.com

Changes in v2:
- Add MaState abstraction. For now it only has a mas_find method. And
  use it in the destructor.
- Duplicate MA_STATE macro in Rust instead of using a C helper.
- Change maple_tree.h so that cast in ma_lock() is no longer needed.
- Add #[must_use] and #[inline] annotations.
- Rename MapleLock to MapleGuard.
- Change errors to use AllocError.
- Add MAINTAINERS file.
- Link to v1: https://lore.kernel.org/r/20250726-maple-tree-v1-0-27a3da7cb8e5 at google.com

---
Alice Ryhl (3):
      rust: maple_tree: add MapleTree
      rust: maple_tree: add lock guard for maple tree
      rust: maple_tree: add MapleTreeAlloc

 MAINTAINERS                |   4 +
 include/linux/maple_tree.h |   3 +
 rust/helpers/helpers.c     |   1 +
 rust/helpers/maple_tree.c  |   8 +
 rust/kernel/lib.rs         |   1 +
 rust/kernel/maple_tree.rs  | 648 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 665 insertions(+)
---
base-commit: 1b237f190eb3d36f52dffe07a40b5eb210280e00
change-id: 20250726-maple-tree-1af0803ac524

Best regards,
-- 
Alice Ryhl <aliceryhl at google.com>



From aliceryhl at google.com  Tue Sep  2 01:35:11 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 02 Sep 2025 08:35:11 +0000
Subject: [PATCH v3 1/3] rust: maple_tree: add MapleTree
In-Reply-To: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
Message-ID: <20250902-maple-tree-v3-1-fb5c8958fb1e@google.com>

The maple tree will be used in the Tyr driver to allocate and keep track
of GPU allocations created internally (i.e. not by userspace). It will
likely also be used in the Nova driver eventually.

This adds the simplest methods for additional and removal that do not
require any special care with respect to concurrency.

This implementation is based on the RFC by Andrew but with significant
changes to simplify the implementation.

Co-developed-by: Andrew Ballance <andrewjballance at gmail.com>
Signed-off-by: Andrew Ballance <andrewjballance at gmail.com>
Signed-off-by: Alice Ryhl <aliceryhl at google.com>
---
 MAINTAINERS                |   4 +
 include/linux/maple_tree.h |   3 +
 rust/helpers/helpers.c     |   1 +
 rust/helpers/maple_tree.c  |   8 ++
 rust/kernel/lib.rs         |   1 +
 rust/kernel/maple_tree.rs  | 350 +++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 367 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index fed6cd812d796a08cebc0c1fd540c8901d1bf448..c076f034562dd3d6b3679e8c2cd390adb312d483 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -14672,6 +14672,8 @@ F:	net/mctp/
 
 MAPLE TREE
 M:	Liam R. Howlett <Liam.Howlett at oracle.com>
+R:	Alice Ryhl <aliceryhl at google.com>
+R:	Andrew Ballance <andrewjballance at gmail.com>
 L:	maple-tree at lists.infradead.org
 L:	linux-mm at kvack.org
 S:	Supported
@@ -14680,6 +14682,8 @@ F:	include/linux/maple_tree.h
 F:	include/trace/events/maple_tree.h
 F:	lib/maple_tree.c
 F:	lib/test_maple_tree.c
+F:	rust/helpers/maple_tree.c
+F:	rust/kernel/maple_tree.rs
 F:	tools/testing/radix-tree/maple.c
 F:	tools/testing/shared/linux/maple_tree.h
 
diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index bafe143b1f783202e27b32567fffee4149e8e266..879cd5f24be49548d5fd9d6b19bf069e33c334a1 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -481,6 +481,9 @@ struct ma_wr_state {
 #define MA_ERROR(err) \
 		((struct maple_enode *)(((unsigned long)err << 2) | 2UL))
 
+/*
+ * When changing MA_STATE, remember to also change rust/kernel/maple_tree.rs
+ */
 #define MA_STATE(name, mt, first, end)					\
 	struct ma_state name = {					\
 		.tree = mt,						\
diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c
index 7cf7fe95e41dd51717050648d6160bebebdf4b26..c5d42e0f7ce6786c1e96f8c0f27710959ca3362b 100644
--- a/rust/helpers/helpers.c
+++ b/rust/helpers/helpers.c
@@ -26,6 +26,7 @@
 #include "io.c"
 #include "jump_label.c"
 #include "kunit.c"
+#include "maple_tree.c"
 #include "mm.c"
 #include "mutex.c"
 #include "of.c"
diff --git a/rust/helpers/maple_tree.c b/rust/helpers/maple_tree.c
new file mode 100644
index 0000000000000000000000000000000000000000..1dd9ac84a13feed53c0ed5eec6805517081d0673
--- /dev/null
+++ b/rust/helpers/maple_tree.c
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/maple_tree.h>
+
+void rust_helper_mt_init_flags(struct maple_tree *mt, unsigned int flags)
+{
+	mt_init_flags(mt, flags);
+}
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index ed53169e795c0badf548025a57f946fa18bc73e3..6b0a5689669fa691e366ab3f9d462692c12bd548 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -96,6 +96,7 @@
 #[cfg(CONFIG_KUNIT)]
 pub mod kunit;
 pub mod list;
+pub mod maple_tree;
 pub mod miscdevice;
 pub mod mm;
 #[cfg(CONFIG_NET)]
diff --git a/rust/kernel/maple_tree.rs b/rust/kernel/maple_tree.rs
new file mode 100644
index 0000000000000000000000000000000000000000..92930b408e9683b6615080a53800f1a393c5f31e
--- /dev/null
+++ b/rust/kernel/maple_tree.rs
@@ -0,0 +1,350 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Maple trees.
+//!
+//! C header: [`include/linux/maple_tree.h`](srctree/include/linux/maple_tree.h)
+//!
+//! Reference: <https://docs.kernel.org/core-api/maple_tree.html>
+
+use core::{
+    marker::PhantomData,
+    ops::{Bound, RangeBounds},
+    ptr,
+};
+
+use kernel::{
+    alloc::Flags,
+    error::to_result,
+    prelude::*,
+    types::{ForeignOwnable, Opaque},
+};
+
+/// A maple tree optimized for storing non-overlapping ranges.
+///
+/// # Invariants
+///
+/// Each range in the maple tree owns an instance of `T`.
+#[pin_data(PinnedDrop)]
+#[repr(transparent)]
+pub struct MapleTree<T: ForeignOwnable> {
+    #[pin]
+    tree: Opaque<bindings::maple_tree>,
+    _p: PhantomData<T>,
+}
+
+#[inline]
+fn to_maple_range(range: impl RangeBounds<usize>) -> Option<(usize, usize)> {
+    let first = match range.start_bound() {
+        Bound::Included(start) => *start,
+        Bound::Excluded(start) => start.checked_add(1)?,
+        Bound::Unbounded => 0,
+    };
+
+    let last = match range.end_bound() {
+        Bound::Included(end) => *end,
+        Bound::Excluded(end) => end.checked_sub(1)?,
+        Bound::Unbounded => usize::MAX,
+    };
+
+    if last < first {
+        return None;
+    }
+
+    Some((first, last))
+}
+
+impl<T: ForeignOwnable> MapleTree<T> {
+    /// Create a new maple tree.
+    ///
+    /// The tree will use the regular implementation with a higher branching factor, rather than
+    /// the allocation tree.
+    #[inline]
+    pub fn new() -> impl PinInit<Self> {
+        pin_init!(MapleTree {
+            // SAFETY: This initializes a maple tree into a pinned slot. The maple tree will be
+            // destroyed in Drop before the memory location becomes invalid.
+            tree <- Opaque::ffi_init(|slot| unsafe { bindings::mt_init_flags(slot, 0) }),
+            _p: PhantomData,
+        })
+    }
+
+    /// Insert the value at the given index.
+    ///
+    /// # Errors
+    ///
+    /// If the maple tree already contains a range using the given index, then this call will
+    /// return an [`InsertError`] with the [`Occupied`] kind. It may also fail if memory
+    /// allocation fails.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use kernel::maple_tree::{InsertErrorKind, MapleTree};
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<KBox<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = KBox::new(10, GFP_KERNEL)?;
+    /// let twenty = KBox::new(20, GFP_KERNEL)?;
+    /// let the_answer = KBox::new(42, GFP_KERNEL)?;
+    ///
+    /// // These calls will succeed.
+    /// tree.insert(100, ten, GFP_KERNEL)?;
+    /// tree.insert(101, twenty, GFP_KERNEL)?;
+    ///
+    /// // This will fail because the index is already in use.
+    /// assert_eq!(
+    ///     tree.insert(100, the_answer, GFP_KERNEL).unwrap_err().cause,
+    ///     InsertErrorKind::Occupied,
+    /// );
+    /// # Ok::<_, Error>(())
+    /// ```
+    #[inline]
+    pub fn insert(&self, index: usize, value: T, gfp: Flags) -> Result<(), InsertError<T>> {
+        self.insert_range(index..=index, value, gfp)
+    }
+
+    /// Insert a value to the specified range, failing on overlap.
+    ///
+    /// This accepts the usual types of Rust ranges using the `..` and `..=` syntax for exclusive
+    /// and inclusive ranges respectively. The range must not be empty, and must not overlap with
+    /// any existing range.
+    ///
+    /// # Errors
+    ///
+    /// If the maple tree already contains an overlapping range, then this call will return an
+    /// [`InsertError`] with the [`Occupied`] kind. It may also fail if memory allocation fails
+    /// or if the requested range is invalid (e.g. empty).
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use kernel::maple_tree::{InsertErrorKind, MapleTree};
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<KBox<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = KBox::new(10, GFP_KERNEL)?;
+    /// let twenty = KBox::new(20, GFP_KERNEL)?;
+    /// let the_answer = KBox::new(42, GFP_KERNEL)?;
+    /// let hundred = KBox::new(100, GFP_KERNEL)?;
+    ///
+    /// // Insert the value 10 at the indices 100 to 499.
+    /// tree.insert_range(100..500, ten, GFP_KERNEL)?;
+    ///
+    /// // Insert the value 20 at the indices 500 to 1000.
+    /// tree.insert_range(500..=1000, twenty, GFP_KERNEL)?;
+    ///
+    /// // This will fail due to overlap with the previous range on index 1000.
+    /// assert_eq!(
+    ///     tree.insert_range(1000..1200, the_answer, GFP_KERNEL).unwrap_err().cause,
+    ///     InsertErrorKind::Occupied,
+    /// );
+    ///
+    /// // When using .. to specify the range, you must be careful to ensure that the range is
+    /// // non-empty.
+    /// assert_eq!(
+    ///     tree.insert_range(72..72, hundred, GFP_KERNEL).unwrap_err().cause,
+    ///     InsertErrorKind::InvalidRequest,
+    /// );
+    /// # Ok::<_, Error>(())
+    /// ```
+    pub fn insert_range<R>(&self, range: R, value: T, gfp: Flags) -> Result<(), InsertError<T>>
+    where
+        R: RangeBounds<usize>,
+    {
+        let Some((first, last)) = to_maple_range(range) else {
+            return Err(InsertError {
+                value,
+                cause: InsertErrorKind::InvalidRequest,
+            });
+        };
+
+        let ptr = T::into_foreign(value);
+
+        // SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
+        let res = to_result(unsafe {
+            bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
+        });
+
+        if let Err(err) = res {
+            // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
+            let value = unsafe { T::from_foreign(ptr) };
+
+            let cause = if err == ENOMEM {
+                InsertErrorKind::AllocError(kernel::alloc::AllocError)
+            } else if err == EEXIST {
+                InsertErrorKind::Occupied
+            } else {
+                InsertErrorKind::InvalidRequest
+            };
+            Err(InsertError { value, cause })
+        } else {
+            Ok(())
+        }
+    }
+
+    /// Erase the range containing the given index.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use kernel::maple_tree::MapleTree;
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<KBox<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = KBox::new(10, GFP_KERNEL)?;
+    /// let twenty = KBox::new(20, GFP_KERNEL)?;
+    ///
+    /// tree.insert_range(100..500, ten, GFP_KERNEL)?;
+    /// tree.insert(67, twenty, GFP_KERNEL)?;
+    ///
+    /// assert_eq!(tree.erase(67).map(|v| *v), Some(20));
+    /// assert_eq!(tree.erase(275).map(|v| *v), Some(10));
+    ///
+    /// // The previous call erased the entire range, not just index 275.
+    /// assert!(tree.erase(127).is_none());
+    /// # Ok::<_, Error>(())
+    /// ```
+    #[inline]
+    pub fn erase(&self, index: usize) -> Option<T> {
+        // SAFETY: `self.tree` contains a valid maple tree.
+        let ret = unsafe { bindings::mtree_erase(self.tree.get(), index) };
+
+        // SAFETY: If the pointer is not null, then we took ownership of a valid instance of `T`
+        // from the tree.
+        unsafe { T::try_from_foreign(ret) }
+    }
+
+    /// Free all `T` instances in this tree.
+    ///
+    /// # Safety
+    ///
+    /// This frees Rust data referenced by the maple tree without removing it from the maple tree,
+    /// leaving it in an invalid state. The caller must ensure that this invalid state cannot be
+    /// observed by the end-user.
+    unsafe fn free_all_entries(self: Pin<&mut Self>) {
+        // SAFETY: The caller provides exclusive access to the entire maple tree, so we have
+        // exclusive access to the entire maple tree despite not holding the lock.
+        let mut ma_state = unsafe { MaState::new_raw(self.into_ref().get_ref(), 0, usize::MAX) };
+
+        loop {
+            // This uses the raw accessor because we're destroying pointers without removing them
+            // from the maple tree, which is only valid because this is the destructor.
+            let ptr = ma_state.mas_find_raw(usize::MAX);
+            if ptr.is_null() {
+                break;
+            }
+            // SAFETY: By the type invariants, this pointer references a valid value of type `T`.
+            // By the safety requirements, it is okay to free it without removing it from the maple
+            // tree.
+            drop(unsafe { T::from_foreign(ptr) });
+        }
+    }
+}
+
+#[pinned_drop]
+impl<T: ForeignOwnable> PinnedDrop for MapleTree<T> {
+    #[inline]
+    fn drop(mut self: Pin<&mut Self>) {
+        // We only iterate the tree if the Rust value has a destructor.
+        if core::mem::needs_drop::<T>() {
+            // SAFETY: Other than the below `mtree_destroy` call, the tree will not be accessed
+            // after this call.
+            unsafe { self.as_mut().free_all_entries() };
+        }
+
+        // SAFETY: The tree is valid, and will not be accessed after this call.
+        unsafe { bindings::mtree_destroy(self.tree.get()) };
+    }
+}
+
+/// A helper type used for navigating a [`MapleTree`].
+///
+/// # Invariants
+///
+/// For the duration of `'tree`:
+///
+/// * The `ma_state` references a valid `MapleTree<T>`.
+/// * The `ma_state` has read/write access to the tree.
+pub struct MaState<'tree, T: ForeignOwnable> {
+    state: bindings::ma_state,
+    _phantom: PhantomData<&'tree mut MapleTree<T>>,
+}
+
+impl<'tree, T: ForeignOwnable> MaState<'tree, T> {
+    /// Initialize a new `MaState` with the given tree.
+    ///
+    /// # Safety
+    ///
+    /// The caller must ensure that this `MaState` has read/write access to the maple tree.
+    #[inline]
+    unsafe fn new_raw(mt: &'tree MapleTree<T>, first: usize, end: usize) -> Self {
+        // INVARIANT:
+        // * Having a reference ensures that the `MapleTree<T>` is valid for `'tree`.
+        // * The caller ensures that we have read/write access.
+        Self {
+            state: bindings::ma_state {
+                tree: mt.tree.get(),
+                index: first,
+                last: end,
+                node: ptr::null_mut(),
+                status: bindings::maple_status_ma_start,
+                min: 0,
+                max: usize::MAX,
+                alloc: ptr::null_mut(),
+                mas_flags: 0,
+                store_type: bindings::store_type_wr_invalid,
+                ..Default::default()
+            },
+            _phantom: PhantomData,
+        }
+    }
+
+    #[inline]
+    fn as_raw(&mut self) -> *mut bindings::ma_state {
+        &raw mut self.state
+    }
+
+    #[inline]
+    fn mas_find_raw(&mut self, max: usize) -> *mut c_void {
+        // SAFETY: By the type invariants, the `ma_state` is active and we have read/write access
+        // to the tree.
+        unsafe { bindings::mas_find(self.as_raw(), max) }
+    }
+}
+
+/// Error type for failure to insert a new value.
+pub struct InsertError<T> {
+    /// The value that could not be inserted.
+    pub value: T,
+    /// The reason for the failure to insert.
+    pub cause: InsertErrorKind,
+}
+
+/// The reason for the failure to insert.
+#[derive(PartialEq, Eq, Copy, Clone, Debug)]
+pub enum InsertErrorKind {
+    /// There is already a value in the requested range.
+    Occupied,
+    /// Failure to allocate memory.
+    AllocError(kernel::alloc::AllocError),
+    /// The insertion request was invalid.
+    InvalidRequest,
+}
+
+impl From<InsertErrorKind> for Error {
+    #[inline]
+    fn from(kind: InsertErrorKind) -> Error {
+        match kind {
+            InsertErrorKind::Occupied => EEXIST,
+            InsertErrorKind::AllocError(kernel::alloc::AllocError) => ENOMEM,
+            InsertErrorKind::InvalidRequest => EINVAL,
+        }
+    }
+}
+
+impl<T> From<InsertError<T>> for Error {
+    #[inline]
+    fn from(insert_err: InsertError<T>) -> Error {
+        Error::from(insert_err.cause)
+    }
+}

-- 
2.51.0.338.gd7d06c2dae-goog



From aliceryhl at google.com  Tue Sep  2 01:35:12 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 02 Sep 2025 08:35:12 +0000
Subject: [PATCH v3 2/3] rust: maple_tree: add lock guard for maple tree
In-Reply-To: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
Message-ID: <20250902-maple-tree-v3-2-fb5c8958fb1e@google.com>

To load a value, one must be careful to hold the lock while accessing
it. To enable this, we add a lock() method so that you can perform
operations on the value before the spinlock is released.

This adds a MapleGuard type without using the existing SpinLock type.
This ensures that the MapleGuard type is not unnecessarily large, and
that it is easy to swap out the type of lock in case the C maple tree is
changed to use a different kind of lock.

There are two ways of using the lock guard: You can call load() directly
to load a value under the lock, or you can create an MaState to iterate
the tree with find().

The find() method does not have the mas_ prefix since it's a method on
MaState, and being a method on that struct serves a similar purpose to
the mas_ prefix in C.

Co-developed-by: Andrew Ballance <andrewjballance at gmail.com>
Signed-off-by: Andrew Ballance <andrewjballance at gmail.com>
Reviewed-by: Andrew Ballance <andrewjballance at gmail.com>
Reviewed-by: Danilo Krummrich <dakr at kernel.org>
Signed-off-by: Alice Ryhl <aliceryhl at google.com>
---
 rust/kernel/maple_tree.rs | 140 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 140 insertions(+)

diff --git a/rust/kernel/maple_tree.rs b/rust/kernel/maple_tree.rs
index 92930b408e9683b6615080a53800f1a393c5f31e..24b674ce07d0481702eccd86a7920f94ca000108 100644
--- a/rust/kernel/maple_tree.rs
+++ b/rust/kernel/maple_tree.rs
@@ -214,6 +214,23 @@ pub fn erase(&self, index: usize) -> Option<T> {
         unsafe { T::try_from_foreign(ret) }
     }
 
+    /// Lock the internal spinlock.
+    #[inline]
+    pub fn lock(&self) -> MapleGuard<'_, T> {
+        // SAFETY: It's safe to lock the spinlock in a maple tree.
+        unsafe { bindings::spin_lock(self.ma_lock()) };
+
+        // INVARIANT: We just took the spinlock.
+        MapleGuard(self)
+    }
+
+    #[inline]
+    fn ma_lock(&self) -> *mut bindings::spinlock_t {
+        // SAFETY: This pointer offset operation stays in-bounds.
+        let lock_ptr = unsafe { &raw mut (*self.tree.get()).__bindgen_anon_1.ma_lock };
+        lock_ptr.cast()
+    }
+
     /// Free all `T` instances in this tree.
     ///
     /// # Safety
@@ -257,6 +274,91 @@ fn drop(mut self: Pin<&mut Self>) {
     }
 }
 
+/// A reference to a [`MapleTree`] that owns the inner lock.
+///
+/// # Invariants
+///
+/// This guard owns the inner spinlock.
+#[must_use = "if unused, the lock will be immediately unlocked"]
+pub struct MapleGuard<'tree, T: ForeignOwnable>(&'tree MapleTree<T>);
+
+impl<'tree, T: ForeignOwnable> Drop for MapleGuard<'tree, T> {
+    #[inline]
+    fn drop(&mut self) {
+        // SAFETY: By the type invariants, we hold this spinlock.
+        unsafe { bindings::spin_unlock(self.0.ma_lock()) };
+    }
+}
+
+impl<'tree, T: ForeignOwnable> MapleGuard<'tree, T> {
+    /// Create a [`MaState`] protected by this lock guard.
+    pub fn ma_state(&mut self, first: usize, end: usize) -> MaState<'_, T> {
+        // SAFETY: The `MaState` borrows this `MapleGuard`, so it can also borrow the `MapleGuard`s
+        // read/write permissions to the maple tree.
+        unsafe { MaState::new_raw(self.0, first, end) }
+    }
+
+    /// Load the value at the given index.
+    ///
+    /// # Examples
+    ///
+    /// Read the value while holding the spinlock.
+    ///
+    /// ```
+    /// use kernel::maple_tree::MapleTree;
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<KBox<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = KBox::new(10, GFP_KERNEL)?;
+    /// let twenty = KBox::new(20, GFP_KERNEL)?;
+    /// tree.insert(100, ten, GFP_KERNEL)?;
+    /// tree.insert(200, twenty, GFP_KERNEL)?;
+    ///
+    /// let mut lock = tree.lock();
+    /// assert_eq!(lock.load(100).map(|v| *v), Some(10));
+    /// assert_eq!(lock.load(200).map(|v| *v), Some(20));
+    /// assert_eq!(lock.load(300).map(|v| *v), None);
+    /// # Ok::<_, Error>(())
+    /// ```
+    ///
+    /// Increment refcount under the lock, to keep value alive afterwards.
+    ///
+    /// ```
+    /// use kernel::maple_tree::MapleTree;
+    /// use kernel::sync::Arc;
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<Arc<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = Arc::new(10, GFP_KERNEL)?;
+    /// let twenty = Arc::new(20, GFP_KERNEL)?;
+    /// tree.insert(100, ten, GFP_KERNEL)?;
+    /// tree.insert(200, twenty, GFP_KERNEL)?;
+    ///
+    /// // Briefly take the lock to increment the refcount.
+    /// let value = tree.lock().load(100).map(Arc::from);
+    ///
+    /// // At this point, another thread might remove the value.
+    /// tree.erase(100);
+    ///
+    /// // But we can still access it because we took a refcount.
+    /// assert_eq!(value.map(|v| *v), Some(10));
+    /// # Ok::<_, Error>(())
+    /// ```
+    #[inline]
+    pub fn load(&mut self, index: usize) -> Option<T::BorrowedMut<'_>> {
+        // SAFETY: `self.tree` contains a valid maple tree.
+        let ret = unsafe { bindings::mtree_load(self.0.tree.get(), index) };
+        if ret.is_null() {
+            return None;
+        }
+
+        // SAFETY: If the pointer is not null, then it references a valid instance of `T`. It is
+        // safe to borrow the instance mutably because the signature of this function enforces that
+        // the mutable borrow is not used after the spinlock is dropped.
+        Some(unsafe { T::borrow_mut(ret) })
+    }
+}
+
 /// A helper type used for navigating a [`MapleTree`].
 ///
 /// # Invariants
@@ -310,6 +412,44 @@ fn mas_find_raw(&mut self, max: usize) -> *mut c_void {
         // to the tree.
         unsafe { bindings::mas_find(self.as_raw(), max) }
     }
+
+    /// Find the next entry in the maple tree.
+    ///
+    /// # Examples
+    ///
+    /// Iterate the maple tree.
+    ///
+    /// ```
+    /// use kernel::maple_tree::MapleTree;
+    /// use kernel::sync::Arc;
+    ///
+    /// let tree = KBox::pin_init(MapleTree::<Arc<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = Arc::new(10, GFP_KERNEL)?;
+    /// let twenty = Arc::new(20, GFP_KERNEL)?;
+    /// tree.insert(100, ten, GFP_KERNEL)?;
+    /// tree.insert(200, twenty, GFP_KERNEL)?;
+    ///
+    /// let mut ma_lock = tree.lock();
+    /// let mut iter = ma_lock.ma_state(0, usize::MAX);
+    ///
+    /// assert_eq!(iter.find(usize::MAX).map(|v| *v), Some(10));
+    /// assert_eq!(iter.find(usize::MAX).map(|v| *v), Some(20));
+    /// assert!(iter.find(usize::MAX).is_none());
+    /// # Ok::<_, Error>(())
+    /// ```
+    #[inline]
+    pub fn find(&mut self, max: usize) -> Option<T::BorrowedMut<'_>> {
+        let ret = self.mas_find_raw(max);
+        if ret.is_null() {
+            return None;
+        }
+
+        // SAFETY: If the pointer is not null, then it references a valid instance of `T`. It's
+        // safe to access it mutably as the returned reference borrows this `MaState`, and the
+        // `MaState` has read/write access to the maple tree.
+        Some(unsafe { T::borrow_mut(ret) })
+    }
 }
 
 /// Error type for failure to insert a new value.

-- 
2.51.0.338.gd7d06c2dae-goog



From aliceryhl at google.com  Tue Sep  2 01:35:13 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 02 Sep 2025 08:35:13 +0000
Subject: [PATCH v3 3/3] rust: maple_tree: add MapleTreeAlloc
In-Reply-To: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
Message-ID: <20250902-maple-tree-v3-3-fb5c8958fb1e@google.com>

To support allocation trees, we introduce a new type MapleTreeAlloc for
the case where the tree is created using MT_FLAGS_ALLOC_RANGE. To ensure
that you can only call mtree_alloc_range on an allocation tree, we
restrict thta method to the new MapleTreeAlloc type. However, all
methods on MapleTree remain accessible to MapleTreeAlloc as allocation
trees can use the other methods without issues.

Reviewed-by: Daniel Almeida <daniel.almeida at collabora.com>
Reviewed-by: Danilo Krummrich <dakr at kernel.org>
Signed-off-by: Alice Ryhl <aliceryhl at google.com>
---
 rust/kernel/maple_tree.rs | 158 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 158 insertions(+)

diff --git a/rust/kernel/maple_tree.rs b/rust/kernel/maple_tree.rs
index 24b674ce07d0481702eccd86a7920f94ca000108..31b6c13d08efab23f7fb20bf97e36c33bf9f6ad9 100644
--- a/rust/kernel/maple_tree.rs
+++ b/rust/kernel/maple_tree.rs
@@ -32,6 +32,26 @@ pub struct MapleTree<T: ForeignOwnable> {
     _p: PhantomData<T>,
 }
 
+/// A maple tree with `MT_FLAGS_ALLOC_RANGE` set.
+///
+/// All methods on [`MapleTree`] are also accessible on this type.
+#[pin_data]
+#[repr(transparent)]
+pub struct MapleTreeAlloc<T: ForeignOwnable> {
+    #[pin]
+    tree: MapleTree<T>,
+}
+
+// Make MapleTree methods usable on MapleTreeAlloc.
+impl<T: ForeignOwnable> core::ops::Deref for MapleTreeAlloc<T> {
+    type Target = MapleTree<T>;
+
+    #[inline]
+    fn deref(&self) -> &MapleTree<T> {
+        &self.tree
+    }
+}
+
 #[inline]
 fn to_maple_range(range: impl RangeBounds<usize>) -> Option<(usize, usize)> {
     let first = match range.start_bound() {
@@ -359,6 +379,107 @@ pub fn load(&mut self, index: usize) -> Option<T::BorrowedMut<'_>> {
     }
 }
 
+impl<T: ForeignOwnable> MapleTreeAlloc<T> {
+    /// Create a new allocation tree.
+    pub fn new() -> impl PinInit<Self> {
+        let tree = pin_init!(MapleTree {
+            // SAFETY: This initializes a maple tree into a pinned slot. The maple tree will be
+            // destroyed in Drop before the memory location becomes invalid.
+            tree <- Opaque::ffi_init(|slot| unsafe {
+                bindings::mt_init_flags(slot, bindings::MT_FLAGS_ALLOC_RANGE)
+            }),
+            _p: PhantomData,
+        });
+
+        pin_init!(MapleTreeAlloc { tree <- tree })
+    }
+
+    /// Insert an entry with the given size somewhere in the given range.
+    ///
+    /// The maple tree will search for a location in the given range where there is space to insert
+    /// the new range. If there is not enough available space, then an error will be returned.
+    ///
+    /// The index of the new range is returned.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// use kernel::maple_tree::{MapleTreeAlloc, AllocErrorKind};
+    ///
+    /// let tree = KBox::pin_init(MapleTreeAlloc::<KBox<i32>>::new(), GFP_KERNEL)?;
+    ///
+    /// let ten = KBox::new(10, GFP_KERNEL)?;
+    /// let twenty = KBox::new(20, GFP_KERNEL)?;
+    /// let thirty = KBox::new(30, GFP_KERNEL)?;
+    /// let hundred = KBox::new(100, GFP_KERNEL)?;
+    ///
+    /// // Allocate three ranges.
+    /// let idx1 = tree.alloc_range(100, ten, ..1000, GFP_KERNEL)?;
+    /// let idx2 = tree.alloc_range(100, twenty, ..1000, GFP_KERNEL)?;
+    /// let idx3 = tree.alloc_range(100, thirty, ..1000, GFP_KERNEL)?;
+    ///
+    /// assert_eq!(idx1, 0);
+    /// assert_eq!(idx2, 100);
+    /// assert_eq!(idx3, 200);
+    ///
+    /// // This will fail because the remaining space is too small.
+    /// assert_eq!(
+    ///     tree.alloc_range(800, hundred, ..1000, GFP_KERNEL).unwrap_err().cause,
+    ///     AllocErrorKind::Busy,
+    /// );
+    /// # Ok::<_, Error>(())
+    /// ```
+    pub fn alloc_range<R>(
+        &self,
+        size: usize,
+        value: T,
+        range: R,
+        gfp: Flags,
+    ) -> Result<usize, AllocError<T>>
+    where
+        R: RangeBounds<usize>,
+    {
+        let Some((min, max)) = to_maple_range(range) else {
+            return Err(AllocError {
+                value,
+                cause: AllocErrorKind::InvalidRequest,
+            });
+        };
+
+        let ptr = T::into_foreign(value);
+        let mut index = 0;
+
+        // SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
+        let res = to_result(unsafe {
+            bindings::mtree_alloc_range(
+                self.tree.tree.get(),
+                &mut index,
+                ptr,
+                size,
+                min,
+                max,
+                gfp.as_raw(),
+            )
+        });
+
+        if let Err(err) = res {
+            // SAFETY: As `mtree_alloc_range` failed, it is safe to take back ownership.
+            let value = unsafe { T::from_foreign(ptr) };
+
+            let cause = if err == ENOMEM {
+                AllocErrorKind::AllocError(kernel::alloc::AllocError)
+            } else if err == EBUSY {
+                AllocErrorKind::Busy
+            } else {
+                AllocErrorKind::InvalidRequest
+            };
+            Err(AllocError { value, cause })
+        } else {
+            Ok(index)
+        }
+    }
+}
+
 /// A helper type used for navigating a [`MapleTree`].
 ///
 /// # Invariants
@@ -488,3 +609,40 @@ fn from(insert_err: InsertError<T>) -> Error {
         Error::from(insert_err.cause)
     }
 }
+
+/// Error type for failure to insert a new value.
+pub struct AllocError<T> {
+    /// The value that could not be inserted.
+    pub value: T,
+    /// The reason for the failure to insert.
+    pub cause: AllocErrorKind,
+}
+
+/// The reason for the failure to insert.
+#[derive(PartialEq, Eq, Copy, Clone)]
+pub enum AllocErrorKind {
+    /// There is not enough space for the requested allocation.
+    Busy,
+    /// Failure to allocate memory.
+    AllocError(kernel::alloc::AllocError),
+    /// The insertion request was invalid.
+    InvalidRequest,
+}
+
+impl From<AllocErrorKind> for Error {
+    #[inline]
+    fn from(kind: AllocErrorKind) -> Error {
+        match kind {
+            AllocErrorKind::Busy => EBUSY,
+            AllocErrorKind::AllocError(kernel::alloc::AllocError) => ENOMEM,
+            AllocErrorKind::InvalidRequest => EINVAL,
+        }
+    }
+}
+
+impl<T> From<AllocError<T>> for Error {
+    #[inline]
+    fn from(insert_err: AllocError<T>) -> Error {
+        Error::from(insert_err.cause)
+    }
+}

-- 
2.51.0.338.gd7d06c2dae-goog



From aliceryhl at google.com  Tue Sep  2 01:36:11 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 02 Sep 2025 08:36:11 +0000
Subject: [PATCH] maple_tree: remove lockdep_map_p typedef
Message-ID: <20250902-maple-lockdep-p-v1-1-3ae5a398a379@google.com>

Having the ma_external_lock field exist when CONFIG_LOCKDEP=n isn't used
anywhere, so just get rid of it. This also avoids generating a typedef
called lockdep_map_p that could overlap with typedefs in other header
files.

Reviewed-by: Danilo Krummrich <dakr at kernel.org>
Signed-off-by: Alice Ryhl <aliceryhl at google.com>
---
Originally sent as
https://lore.kernel.org/all/20250819-maple-tree-v2-1-229b48657bab at google.com/

But I'm moving it out of that series as it did not have the intended
effect on the Rust bindgen output. However, I still think it makes sense
as a pure cleanup patch.
---
 include/linux/maple_tree.h | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index bafe143b1f783202e27b32567fffee4149e8e266..8244679ba1758235e049acbaedee62aae5c0e226 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -194,7 +194,6 @@ enum store_type {
 #define MAPLE_RESERVED_RANGE	4096
 
 #ifdef CONFIG_LOCKDEP
-typedef struct lockdep_map *lockdep_map_p;
 #define mt_lock_is_held(mt)                                             \
 	(!(mt)->ma_external_lock || lock_is_held((mt)->ma_external_lock))
 
@@ -207,7 +206,6 @@ typedef struct lockdep_map *lockdep_map_p;
 
 #define mt_on_stack(mt)			(mt).ma_external_lock = NULL
 #else
-typedef struct { /* nothing */ } lockdep_map_p;
 #define mt_lock_is_held(mt)		1
 #define mt_write_lock_is_held(mt)	1
 #define mt_set_external_lock(mt, lock)	do { } while (0)
@@ -230,8 +228,10 @@ typedef struct { /* nothing */ } lockdep_map_p;
  */
 struct maple_tree {
 	union {
-		spinlock_t	ma_lock;
-		lockdep_map_p	ma_external_lock;
+		spinlock_t		ma_lock;
+#ifdef CONFIG_LOCKDEP
+		struct lockdep_map	*ma_external_lock;
+#endif
 	};
 	unsigned int	ma_flags;
 	void __rcu      *ma_root;

---
base-commit: 1b237f190eb3d36f52dffe07a40b5eb210280e00
change-id: 20250902-maple-lockdep-p-a930e865c76b

Best regards,
-- 
Alice Ryhl <aliceryhl at google.com>



From dakr at kernel.org  Tue Sep  2 02:01:19 2025
From: dakr at kernel.org (Danilo Krummrich)
Date: Tue, 02 Sep 2025 11:01:19 +0200
Subject: [PATCH v3 1/3] rust: maple_tree: add MapleTree
In-Reply-To: <20250902-maple-tree-v3-1-fb5c8958fb1e@google.com>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
 <20250902-maple-tree-v3-1-fb5c8958fb1e@google.com>
Message-ID: <DCI6XGR65KH9.27TWYVKNZNGHV@kernel.org>

On Tue Sep 2, 2025 at 10:35 AM CEST, Alice Ryhl wrote:
> The maple tree will be used in the Tyr driver to allocate and keep track
> of GPU allocations created internally (i.e. not by userspace). It will
> likely also be used in the Nova driver eventually.
>
> This adds the simplest methods for additional and removal that do not
> require any special care with respect to concurrency.
>
> This implementation is based on the RFC by Andrew but with significant
> changes to simplify the implementation.
>
> Co-developed-by: Andrew Ballance <andrewjballance at gmail.com>
> Signed-off-by: Andrew Ballance <andrewjballance at gmail.com>
> Signed-off-by: Alice Ryhl <aliceryhl at google.com>

One nit below, otherwise:

	Reviewed-by: Danilo Krummrich <dakr at kernel.org>

> +    pub fn insert_range<R>(&self, range: R, value: T, gfp: Flags) -> Result<(), InsertError<T>>
> +    where
> +        R: RangeBounds<usize>,
> +    {
> +        let Some((first, last)) = to_maple_range(range) else {
> +            return Err(InsertError {
> +                value,
> +                cause: InsertErrorKind::InvalidRequest,
> +            });
> +        };
> +
> +        let ptr = T::into_foreign(value);
> +
> +        // SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
> +        let res = to_result(unsafe {
> +            bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
> +        });
> +
> +        if let Err(err) = res {
> +            // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
> +            let value = unsafe { T::from_foreign(ptr) };
> +
> +            let cause = if err == ENOMEM {
> +                InsertErrorKind::AllocError(kernel::alloc::AllocError)
> +            } else if err == EEXIST {
> +                InsertErrorKind::Occupied
> +            } else {
> +                InsertErrorKind::InvalidRequest
> +            };
> +            Err(InsertError { value, cause })
> +        } else {
> +            Ok(())
> +        }
> +    }

	// SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
	to_result(unsafe {
	    bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
	}).map_err(|err| {
	    // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
	    let value = unsafe { T::from_foreign(ptr) };
	
	    let cause = if err == ENOMEM {
	        InsertErrorKind::AllocError(kernel::alloc::AllocError)
	    } else if err == EEXIST {
	        InsertErrorKind::Occupied
	    } else {
	        InsertErrorKind::InvalidRequest
	    };
	    Err(InsertError { value, cause })
	})

I think that's a bit cleaner than the above (not compile tested).


From aliceryhl at google.com  Tue Sep  2 02:28:12 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Tue, 2 Sep 2025 09:28:12 +0000
Subject: [PATCH v3 1/3] rust: maple_tree: add MapleTree
In-Reply-To: <DCI6XGR65KH9.27TWYVKNZNGHV@kernel.org>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
 <20250902-maple-tree-v3-1-fb5c8958fb1e@google.com> <DCI6XGR65KH9.27TWYVKNZNGHV@kernel.org>
Message-ID: <aLa4rNojDIeShIrw@google.com>

On Tue, Sep 02, 2025 at 11:01:19AM +0200, Danilo Krummrich wrote:
> On Tue Sep 2, 2025 at 10:35 AM CEST, Alice Ryhl wrote:
> > The maple tree will be used in the Tyr driver to allocate and keep track
> > of GPU allocations created internally (i.e. not by userspace). It will
> > likely also be used in the Nova driver eventually.
> >
> > This adds the simplest methods for additional and removal that do not
> > require any special care with respect to concurrency.
> >
> > This implementation is based on the RFC by Andrew but with significant
> > changes to simplify the implementation.
> >
> > Co-developed-by: Andrew Ballance <andrewjballance at gmail.com>
> > Signed-off-by: Andrew Ballance <andrewjballance at gmail.com>
> > Signed-off-by: Alice Ryhl <aliceryhl at google.com>
> 
> One nit below, otherwise:
> 
> 	Reviewed-by: Danilo Krummrich <dakr at kernel.org>

Thanks!

> > +    pub fn insert_range<R>(&self, range: R, value: T, gfp: Flags) -> Result<(), InsertError<T>>
> > +    where
> > +        R: RangeBounds<usize>,
> > +    {
> > +        let Some((first, last)) = to_maple_range(range) else {
> > +            return Err(InsertError {
> > +                value,
> > +                cause: InsertErrorKind::InvalidRequest,
> > +            });
> > +        };
> > +
> > +        let ptr = T::into_foreign(value);
> > +
> > +        // SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
> > +        let res = to_result(unsafe {
> > +            bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
> > +        });
> > +
> > +        if let Err(err) = res {
> > +            // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
> > +            let value = unsafe { T::from_foreign(ptr) };
> > +
> > +            let cause = if err == ENOMEM {
> > +                InsertErrorKind::AllocError(kernel::alloc::AllocError)
> > +            } else if err == EEXIST {
> > +                InsertErrorKind::Occupied
> > +            } else {
> > +                InsertErrorKind::InvalidRequest
> > +            };
> > +            Err(InsertError { value, cause })
> > +        } else {
> > +            Ok(())
> > +        }
> > +    }
> 
> 	// SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
> 	to_result(unsafe {
> 	    bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
> 	}).map_err(|err| {
> 	    // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
> 	    let value = unsafe { T::from_foreign(ptr) };
> 	
> 	    let cause = if err == ENOMEM {
> 	        InsertErrorKind::AllocError(kernel::alloc::AllocError)
> 	    } else if err == EEXIST {
> 	        InsertErrorKind::Occupied
> 	    } else {
> 	        InsertErrorKind::InvalidRequest
> 	    };
> 	    Err(InsertError { value, cause })
> 	})
> 
> I think that's a bit cleaner than the above (not compile tested).

I don't love it. How about a match instead of if/else?

Alice


From dakr at kernel.org  Tue Sep  2 04:11:34 2025
From: dakr at kernel.org (Danilo Krummrich)
Date: Tue, 02 Sep 2025 13:11:34 +0200
Subject: [PATCH v3 1/3] rust: maple_tree: add MapleTree
In-Reply-To: <aLa4rNojDIeShIrw@google.com>
References: <20250902-maple-tree-v3-0-fb5c8958fb1e@google.com>
 <20250902-maple-tree-v3-1-fb5c8958fb1e@google.com>
 <DCI6XGR65KH9.27TWYVKNZNGHV@kernel.org> <aLa4rNojDIeShIrw@google.com>
Message-ID: <DCI9P6V88RHN.BX0BQWFDA8DO@kernel.org>

On Tue Sep 2, 2025 at 11:28 AM CEST, Alice Ryhl wrote:
> On Tue, Sep 02, 2025 at 11:01:19AM +0200, Danilo Krummrich wrote:
>> On Tue Sep 2, 2025 at 10:35 AM CEST, Alice Ryhl wrote:
>> > The maple tree will be used in the Tyr driver to allocate and keep track
>> > of GPU allocations created internally (i.e. not by userspace). It will
>> > likely also be used in the Nova driver eventually.
>> >
>> > This adds the simplest methods for additional and removal that do not
>> > require any special care with respect to concurrency.
>> >
>> > This implementation is based on the RFC by Andrew but with significant
>> > changes to simplify the implementation.
>> >
>> > Co-developed-by: Andrew Ballance <andrewjballance at gmail.com>
>> > Signed-off-by: Andrew Ballance <andrewjballance at gmail.com>
>> > Signed-off-by: Alice Ryhl <aliceryhl at google.com>
>> 
>> One nit below, otherwise:
>> 
>> 	Reviewed-by: Danilo Krummrich <dakr at kernel.org>
>
> Thanks!
>
>> > +    pub fn insert_range<R>(&self, range: R, value: T, gfp: Flags) -> Result<(), InsertError<T>>
>> > +    where
>> > +        R: RangeBounds<usize>,
>> > +    {
>> > +        let Some((first, last)) = to_maple_range(range) else {
>> > +            return Err(InsertError {
>> > +                value,
>> > +                cause: InsertErrorKind::InvalidRequest,
>> > +            });
>> > +        };
>> > +
>> > +        let ptr = T::into_foreign(value);
>> > +
>> > +        // SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
>> > +        let res = to_result(unsafe {
>> > +            bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
>> > +        });
>> > +
>> > +        if let Err(err) = res {
>> > +            // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
>> > +            let value = unsafe { T::from_foreign(ptr) };
>> > +
>> > +            let cause = if err == ENOMEM {
>> > +                InsertErrorKind::AllocError(kernel::alloc::AllocError)
>> > +            } else if err == EEXIST {
>> > +                InsertErrorKind::Occupied
>> > +            } else {
>> > +                InsertErrorKind::InvalidRequest
>> > +            };
>> > +            Err(InsertError { value, cause })
>> > +        } else {
>> > +            Ok(())
>> > +        }
>> > +    }
>> 
>> 	// SAFETY: The tree is valid, and we are passing a pointer to an owned instance of `T`.
>> 	to_result(unsafe {
>> 	    bindings::mtree_insert_range(self.tree.get(), first, last, ptr, gfp.as_raw())
>> 	}).map_err(|err| {
>> 	    // SAFETY: As `mtree_insert_range` failed, it is safe to take back ownership.
>> 	    let value = unsafe { T::from_foreign(ptr) };
>> 	
>> 	    let cause = if err == ENOMEM {
>> 	        InsertErrorKind::AllocError(kernel::alloc::AllocError)
>> 	    } else if err == EEXIST {
>> 	        InsertErrorKind::Occupied
>> 	    } else {
>> 	        InsertErrorKind::InvalidRequest
>> 	    };
>> 	    Err(InsertError { value, cause })
>> 	})
>> 
>> I think that's a bit cleaner than the above (not compile tested).
>
> I don't love it. How about a match instead of if/else?

I think if you don't like map_err(), the if/else is fine to keep as is.

Personally, I prefer map_err() over any "manual matching" in this case; it gets
us rid of `ref`, making to_result().map_err() a single statement returning
exactly what we expect. It makes it more self-contained.


From lorenzo.stoakes at oracle.com  Tue Sep  2 04:13:41 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Tue, 2 Sep 2025 12:13:41 +0100
Subject: [PATCH v6 08/10] mm, vma: use percpu sheaves for vm_area_struct
 cache
In-Reply-To: <20250827-slub-percpu-caches-v6-8-f0f775a3f73f@suse.cz>
References: <20250827-slub-percpu-caches-v6-0-f0f775a3f73f@suse.cz>
 <20250827-slub-percpu-caches-v6-8-f0f775a3f73f@suse.cz>
Message-ID: <195d00ab-5429-48ff-b5dd-a45b26cd9ea9@lucifer.local>

On Wed, Aug 27, 2025 at 10:26:40AM +0200, Vlastimil Babka wrote:
> Create the vm_area_struct cache with percpu sheaves of size 32 to
> improve its performance.
>
> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  mm/vma_init.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/mm/vma_init.c b/mm/vma_init.c
> index 8e53c7943561e7324e7992946b4065dec1149b82..52c6b55fac4519e0da39ca75ad018e14449d1d95 100644
> --- a/mm/vma_init.c
> +++ b/mm/vma_init.c
> @@ -16,6 +16,7 @@ void __init vma_state_init(void)
>  	struct kmem_cache_args args = {
>  		.use_freeptr_offset = true,
>  		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
> +		.sheaf_capacity = 32,

This breaks the VMA tests.

Please also update tools/testing/vma/vma_internal.h to add this field
(probably actually at the commit that adds the field in mainline :P)

I do wish we could have these tests be run by a bot :/ seems today I am
that bot :)

>  	};
>
>  	vm_area_cachep = kmem_cache_create("vm_area_struct",
>
> --
> 2.51.0
>
>
>


From sidhartha.kumar at oracle.com  Tue Sep  2 12:36:22 2025
From: sidhartha.kumar at oracle.com (Sidhartha Kumar)
Date: Tue, 2 Sep 2025 15:36:22 -0400
Subject: [PATCH 01/12] maple_tree: Fix check_bulk_rebalance() test locks
In-Reply-To: <20250901-maple-sheaves-v1-1-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
 <20250901-maple-sheaves-v1-1-d6a1166b53f2@suse.cz>
Message-ID: <bd2804ce-2d81-4a39-9623-32fb2fbf5351@oracle.com>

On 9/1/25 7:08 AM, Vlastimil Babka wrote:
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
> 
> The check_bulk_rebalance() test was not correctly locking the tree which
> caused issues with the sheaves testing in later patches.  Adding the
> missing locks fixed the issue.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

If needed,

Fixes: a6e0ceb7bf48 ("maple_tree: check for MA_STATE_BULK on setting 
wr_rebalance")

Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>

> ---
>   tools/testing/radix-tree/maple.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
> index 172700fb7784d29f9403003b4484a5ebd7aa316b..159d5307b30a4b37e6cf2941848b8718e1b891d9 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -36465,6 +36465,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
>   
>   	build_full_tree(mt, 0, 2);
>   
> +	mas_lock(&mas);
>   	/* erase every entry in the tree */
>   	do {
>   		/* set up bulk store mode */
> @@ -36474,6 +36475,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
>   	} while (mas_prev(&mas, 0) != NULL);
>   
>   	mas_destroy(&mas);
> +	mas_unlock(&mas);
>   }
>   
>   void farmer_tests(void)
> 



From sidhartha.kumar at oracle.com  Tue Sep  2 12:39:59 2025
From: sidhartha.kumar at oracle.com (Sidhartha Kumar)
Date: Tue, 2 Sep 2025 15:39:59 -0400
Subject: [PATCH 03/12] maple_tree: use percpu sheaves for maple_node_cache
In-Reply-To: <20250901-maple-sheaves-v1-3-d6a1166b53f2@suse.cz>
References: <20250901-maple-sheaves-v1-0-d6a1166b53f2@suse.cz>
 <20250901-maple-sheaves-v1-3-d6a1166b53f2@suse.cz>
Message-ID: <fbc20724-bec5-4a80-b77c-9bec445097cf@oracle.com>

On 9/1/25 7:08 AM, Vlastimil Babka wrote:
> Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
> improve its performance. Note this will not immediately take advantage
> of sheaf batching of kfree_rcu() operations due to the maple tree using
> call_rcu with custom callbacks. The followup changes to maple tree will
> change that and also make use of the prefilled sheaves functionality.
> 
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> Reviewed-by: Suren Baghdasaryan <surenb at google.com>

Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>

> ---
>   lib/maple_tree.c | 9 +++++++--
>   1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index b4ee2d29d7a962ca374467d0533185f2db3d35ff..a0db6bdc63793b8bbd544e246391d99e880dede3 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -6302,9 +6302,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
>   
>   void __init maple_tree_init(void)
>   {
> +	struct kmem_cache_args args = {
> +		.align  = sizeof(struct maple_node),
> +		.sheaf_capacity = 32,
> +	};
> +
>   	maple_node_cache = kmem_cache_create("maple_node",
> -			sizeof(struct maple_node), sizeof(struct maple_node),
> -			SLAB_PANIC, NULL);
> +			sizeof(struct maple_node), &args,
> +			SLAB_PANIC);
>   }
>   
>   /**
> 





From vbabka at suse.cz  Wed Sep  3 05:47:28 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 3 Sep 2025 14:47:28 +0200
Subject: [PATCH v6 08/10] mm, vma: use percpu sheaves for vm_area_struct
 cache
In-Reply-To: <195d00ab-5429-48ff-b5dd-a45b26cd9ea9@lucifer.local>
References: <20250827-slub-percpu-caches-v6-0-f0f775a3f73f@suse.cz>
 <20250827-slub-percpu-caches-v6-8-f0f775a3f73f@suse.cz>
 <195d00ab-5429-48ff-b5dd-a45b26cd9ea9@lucifer.local>
Message-ID: <814c7054-3a43-4190-893f-22e074dc93cd@suse.cz>

On 9/2/25 13:13, Lorenzo Stoakes wrote:
> On Wed, Aug 27, 2025 at 10:26:40AM +0200, Vlastimil Babka wrote:
>> Create the vm_area_struct cache with percpu sheaves of size 32 to
>> improve its performance.
>>
>> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
>> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
>> ---
>>  mm/vma_init.c | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/mm/vma_init.c b/mm/vma_init.c
>> index 8e53c7943561e7324e7992946b4065dec1149b82..52c6b55fac4519e0da39ca75ad018e14449d1d95 100644
>> --- a/mm/vma_init.c
>> +++ b/mm/vma_init.c
>> @@ -16,6 +16,7 @@ void __init vma_state_init(void)
>>  	struct kmem_cache_args args = {
>>  		.use_freeptr_offset = true,
>>  		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
>> +		.sheaf_capacity = 32,
> 
> This breaks the VMA tests.
> 
> Please also update tools/testing/vma/vma_internal.h to add this field
> (probably actually at the commit that adds the field in mainline :P)

Thanks, one of Liam's patches to support maple tree tests with sheaves does
that, we somehow missed that the vma test also needs it so will reorder
accordingly.
> I do wish we could have these tests be run by a bot :/ seems today I am
> that bot :)
Sorry :)

>>  	};
>>
>>  	vm_area_cachep = kmem_cache_create("vm_area_struct",
>>
>> --
>> 2.51.0
>>
>>
>>



From vbabka at suse.cz  Wed Sep  3 05:59:44 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:44 +0200
Subject: [PATCH v7 02/21] slab: simplify init_kmem_cache_nodes() error
 handling
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-2-71c114cdefef@suse.cz>

We don't need to call free_kmem_cache_nodes() immediately when failing
to allocate a kmem_cache_node, because when we return 0,
do_kmem_cache_create() calls __kmem_cache_release() which also performs
free_kmem_cache_nodes().

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 30003763d224c2704a4b93082b8b47af12dcffc5..9f671ec76131c4b0b28d5d568aa45842b5efb6d4 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5669,10 +5669,8 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
 		n = kmem_cache_alloc_node(kmem_cache_node,
 						GFP_KERNEL, node);
 
-		if (!n) {
-			free_kmem_cache_nodes(s);
+		if (!n)
 			return 0;
-		}
 
 		init_kmem_cache_node(n);
 		s->node[node] = n;

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:43 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:43 +0200
Subject: [PATCH v7 01/21] locking/local_lock: Expose dep_map in
 local_trylock_t.
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-1-71c114cdefef@suse.cz>

From: Alexei Starovoitov <ast at kernel.org>

lockdep_is_held() macro assumes that "struct lockdep_map dep_map;"
is a top level field of any lock that participates in LOCKDEP.
Make it so for local_trylock_t.

Reviewed-by: Sebastian Andrzej Siewior <bigeasy at linutronix.de>
Signed-off-by: Alexei Starovoitov <ast at kernel.org>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/local_lock_internal.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h
index d80b5306a2c0ccf95a3405b6b947b5f1f9a3bd38..949de37700dbc10feafc06d0b52382cf2e00c694 100644
--- a/include/linux/local_lock_internal.h
+++ b/include/linux/local_lock_internal.h
@@ -17,7 +17,10 @@ typedef struct {
 
 /* local_trylock() and local_trylock_irqsave() only work with local_trylock_t */
 typedef struct {
-	local_lock_t	llock;
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+	struct lockdep_map	dep_map;
+	struct task_struct	*owner;
+#endif
 	u8		acquired;
 } local_trylock_t;
 
@@ -31,7 +34,7 @@ typedef struct {
 	.owner = NULL,
 
 # define LOCAL_TRYLOCK_DEBUG_INIT(lockname)		\
-	.llock = { LOCAL_LOCK_DEBUG_INIT((lockname).llock) },
+	LOCAL_LOCK_DEBUG_INIT(lockname)
 
 static inline void local_lock_acquire(local_lock_t *l)
 {
@@ -81,7 +84,7 @@ do {								\
 	local_lock_debug_init(lock);				\
 } while (0)
 
-#define __local_trylock_init(lock) __local_lock_init(lock.llock)
+#define __local_trylock_init(lock) __local_lock_init((local_lock_t *)lock)
 
 #define __spinlock_nested_bh_init(lock)				\
 do {								\

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:42 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:42 +0200
Subject: [PATCH v7 00/21] SLUB percpu sheaves
Message-ID: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>

Hi,

I'm sending v7 as there were some non-trivial fixes needed since v6. I'm
including back the maple tree part that I sent separately the last time,
as some reordering between both parts was necessary to avoid
intermittently breaking the userspace vma tests. See the changelog at
the end of the cover letter for details and links.
The v7 will replace+extend the v6 in slab/for-next.

===

This series adds an opt-in percpu array-based caching layer to SLUB.
It has evolved to a state where kmem caches with sheaves are compatible
with all SLUB features (slub_debug, SLUB_TINY, NUMA locality
considerations). My hope is therefore that it can eventually be enabled
for all kmem caches and replace the cpu (partial) slabs.

Note the name "sheaf" was invented by Matthew Wilcox so we don't call
the arrays magazines like the original Bonwick paper. The per-NUMA-node
cache of sheaves is thus called "barn".

This caching may seem similar to the arrays in SLAB, but there are some
important differences:

- does not distinguish NUMA locality, thus there are no per-node
  "shared" arrays (with possible lock contention) and no "alien" arrays
  that would need periodical flushing
  - NUMA restricted allocations and strict_numa mode is still honoured,
    the percpu sheaves are bypassed for those allocations
  - a later patch (for separate evaluation) makes freeing remote objects
    bypass sheaves so sheaves contain mostly (not strictly) local objects
- improves kfree_rcu() handling by reusing whole sheaves
- there is an API for obtaining a preallocated sheaf that can be used
  for guaranteed and efficient allocations in a restricted context, when
  the upper bound for needed objects is known but rarely reached
- opt-in, not used for every cache (for now)

The motivation comes mainly from the ongoing work related to VMA locking
scalability and the related maple tree operations. This is why VMA and
maple nodes caches are sheaf-enabled in the patchset. In v5 I include
Liam's patches for full maple tree conversion that uses the improved
preallocation API.

A sheaf-enabled cache has the following expected advantages:

- Cheaper fast paths. For allocations, instead of local double cmpxchg,
  thanks to local_trylock() it becomes a preempt_disable() and no atomic
  operations. Same for freeing, which is otherwise a local double cmpxchg
  only for short term allocations (so the same slab is still active on the
  same cpu when freeing the object) and a more costly locked double
  cmpxchg otherwise.

- kfree_rcu() batching and recycling. kfree_rcu() will put objects to a
  separate percpu sheaf and only submit the whole sheaf to call_rcu()
  when full. After the grace period, the sheaf can be used for
  allocations, which is more efficient than freeing and reallocating
  individual slab objects (even with the batching done by kfree_rcu()
  implementation itself). In case only some cpus are allowed to handle rcu
  callbacks, the sheaf can still be made available to other cpus on the
  same node via the shared barn. The maple_node cache uses kfree_rcu() and
  thus can benefit from this.

- Preallocation support. A prefilled sheaf can be privately borrowed to
  perform a short term operation that is not allowed to block in the
  middle and may need to allocate some objects. If an upper bound (worst
  case) for the number of allocations is known, but only much fewer
  allocations actually needed on average, borrowing and returning a sheaf
  is much more efficient then a bulk allocation for the worst case
  followed by a bulk free of the many unused objects. Maple tree write
  operations should benefit from this.

- Compatibility with slub_debug. When slub_debug is enabled for a cache,
  we simply don't create the percpu sheaves so that the debugging hooks
  (at the node partial list slowpaths) are reached as before. The same
  thing is done for CONFIG_SLUB_TINY. Sheaf preallocation still works by
  reusing the (ineffective) paths for requests exceeding the cache's
  sheaf_capacity. This is in line with the existing approach where
  debugging bypasses the fast paths and SLUB_TINY preferes memory
  savings over performance.

GIT TREES:

this series: https://git.kernel.org/vbabka/l/slub-percpu-sheaves-v7r0
It is based on v6.17-rc3.

this series plus a microbenchmark hacked into slub_kunit:
https://git.kernel.org/vbabka/l/slub-percpu-sheaves-v7-benchmarking

It allows evaluating overhead of the added sheaves code, and benefits
for single-threaded allocation/frees of varying batch size. I plan to
look into adding multi-threaded scenarios too.

The last commit there also adds sheaves to every cache to allow
measuring effects on other caches than vma and maple node. Note these
measurements should be compared to slab_nomerge boots without sheaves,
as adding sheaves makes caches unmergeable.

RESULTS:

In order to get some numbers that should be only due to differences in
implementation and no cache layout side-effects in users of the slab
objects etc, I have started with a in-kernel microbenchmark that does
allocating and freeing from a slab cache with or without sheaves and/or
memcg. It's either alternating single object alloc and free, or
allocates 10 objects and frees them, then 100, then 1000
- in order to see the effects of exhausting percpu sheaves or barn, or
(without sheaves) the percpu slabs. The order of objects to free can
be also shuffled instead of FIFO - to stress the non-sheaf freeing
slowpath more.

Measurements done on Ryzen 7 5700, bare metal.

The first question was how just having the sheaves implementation affects
existing no-sheaf caches due to the extra (unused) code. I have experimented
with changing inlining and adding unlikely() to the sheaves case. The
optimum seems is what's currently in the implementation - fast-path sheaves
usage is inlined, any handling of main sheaf empty on alloc/full on free is
a separate function, and the if (s->sheaf_capacity) has neither likely() nor
unlikely(). When I added unlikely() it destroyed the performance of sheaves
completely.

So the result is that with batch size 10, there's 2.4% overhead, and the
other cases are all impacted less than this. Hopefully acceptable with the
plan that eventually there would be sheaves everywhere and the current
cpu (partial) slabs scheme removed.

As for benefits of enabling sheaves (capacity=32) see the results below,
looks all good here. Of course this microbenchmark is not a complete
story though for at least these reasons:

- no kfree_rcu() evaluation
- doesn't show barn spinlock contention effects. In theory shouldn't be
worse than without sheaves because after exhausting cpu (partial) slabs, the
list_lock has to be taken. Sheaf capacity vs capacity of partial slabs is a
matter of tuning.

---------------------------------
 BATCH SIZE: 1 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115660272
 bench: no memcg, sheaves
 average (excl. iter 0): 95734972
 sheaves better by 17.2%
 bench: memcg, no sheaves
 average (excl. iter 0): 163682964
 bench: memcg, sheaves
 average (excl. iter 0): 144792803
 sheaves better by 11.5%

 ---------------------------------
 BATCH SIZE: 10 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115496906
 bench: no memcg, sheaves
 average (excl. iter 0): 97781102
 sheaves better by 15.3%
 bench: memcg, no sheaves
 average (excl. iter 0): 162771491
 bench: memcg, sheaves
 average (excl. iter 0): 144746490
 sheaves better by 11.0%

 ---------------------------------
 BATCH SIZE: 100 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 151796052
 bench: no memcg, sheaves
 average (excl. iter 0): 104641753
 sheaves better by 31.0%
 bench: memcg, no sheaves
 average (excl. iter 0): 200733436
 bench: memcg, sheaves
 average (excl. iter 0): 151340989
 sheaves better by 24.6%

 ---------------------------------
 BATCH SIZE: 1000 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 187623118
 bench: no memcg, sheaves
 average (excl. iter 0): 130914624
 sheaves better by 30.2%
 bench: memcg, no sheaves
 average (excl. iter 0): 240239575
 bench: memcg, sheaves
 average (excl. iter 0): 181474462
 sheaves better by 24.4%

 ---------------------------------
 BATCH SIZE: 10 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115110219
 bench: no memcg, sheaves
 average (excl. iter 0): 100597405
 sheaves better by 12.6%
 bench: memcg, no sheaves
 average (excl. iter 0): 163573377
 bench: memcg, sheaves
 average (excl. iter 0): 144535545
 sheaves better by 11.6%

 ---------------------------------
 BATCH SIZE: 100 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 152457970
 bench: no memcg, sheaves
 average (excl. iter 0): 108720274
 sheaves better by 28.6%
 bench: memcg, no sheaves
 average (excl. iter 0): 203478732
 bench: memcg, sheaves
 average (excl. iter 0): 151241821
 sheaves better by 25.6%

 ---------------------------------
 BATCH SIZE: 1000 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 189950559
 bench: no memcg, sheaves
 average (excl. iter 0): 177934450
 sheaves better by 6.3%
 bench: memcg, no sheaves
 average (excl. iter 0): 242988187
 bench: memcg, sheaves
 average (excl. iter 0): 221609979
 sheaves better by 8.7%

Vlastimil

---
Changes in v7:
- Incorporate Alexei's patch for local_trylock_t to fix
  lockdep_assert_held() on RT, reported by Thorsten.
  Patch: https://lore.kernel.org/all/20250718021646.73353-2-alexei.starovoitov at gmail.com/
- Remove pcs->barn pointer to fix boot failures reported by Venkat:
  https://lore.kernel.org/all/866d7f30-7cde-4c88-87ba-bdad16075433 at linux.ibm.com/
  This was because initializing it for all possible cpus can get bogus
  nid values from cpu_to_mem() for non-online cpus.
  Instead introduce get_barn() to obtain it from node, the fast paths
  are not affected anyway. Otherwise cpu hotplug online callback would
  be necessary and I haven't found a cpuhp state that would satisfy all
  the constraints (including being safe to take slab_mutex).
  The draft fix patch from the bug report thread is effectively squashed
  into several of the sheaves patches.
- Incorporate maple tree patches posted separately here:
  https://lore.kernel.org/all/20250901-maple-sheaves-v1-0-d6a1166b53f2 at suse.cz/
  Everything is reordered so that vma and maple userspace tests compile
  and pass at every step (fixes a broken vma test report from Lorenzo).
  Add tags from Sid for two of the patches there.
- Link to v6: https://patch.msgid.link/20250827-slub-percpu-caches-v6-0-f0f775a3f73f at suse.cz

Changes in v6:
- Applied feedback and review tags from Suren and Harry (thanks!)
- Separate patch for init_kmem_cache_nodes() error handling change.
- Removed the more involved maple tree conversion to be posted as a
  separate followup series.
- Link to v5: https://patch.msgid.link/20250723-slub-percpu-caches-v5-0-b792cd830f5d at suse.cz

Changes in v5:
- Apply review tags (Harry, Suren) except where changed too much (first
  patch).
- Handle CONFIG_SLUB_TINY by not creating percpu sheaves (Harry)
- Apply review feedback (typos, comments).
- Extract handling sheaf slow paths to separate non-inline functions
  __pcs_handle_empty() and __pcs_handle_full().
- Fix empty sheaf leak in rcu_free_sheaf() (Suren)
- Add "allow NUMA restricted allocations to use percpu sheaves".
- Add Liam's maple tree full sheaf conversion patches for easier
  evaluation.
- Rebase to v6.16-rc1.
- Link to v4: https://patch.msgid.link/20250425-slub-percpu-caches-v4-0-8a636982b4a4 at suse.cz

Changes in v4:
- slub_debug disables sheaves for the cache in order to work properly
- strict_numa mode works as intended
- added a separate patch to make freeing remote objects skip sheaves
- various code refactoring suggested by Suren and Harry
- removed less useful stat counters and added missing ones for barn
  and prefilled sheaf events
- Link to v3: https://lore.kernel.org/r/20250317-slub-percpu-caches-v3-0-9d9884d8b643 at suse.cz

Changes in v3:
- Squash localtry_lock conversion so it's used immediately.
- Incorporate feedback and add tags from Suren and Harry - thanks!
  - Mostly adding comments and some refactoring.
  - Fixes for kfree_rcu_sheaf() vmalloc handling, cpu hotremove
    flushing.
  - Fix wrong condition in kmem_cache_return_sheaf() that may have
    affected performance negatively.
  - Refactoring of free_to_pcs()
- Link to v2: https://lore.kernel.org/r/20250214-slub-percpu-caches-v2-0-88592ee0966a at suse.cz

Changes in v2:
- Removed kfree_rcu() destructors support as VMAs will not need it
  anymore after [3] is merged.
- Changed to localtry_lock_t borrowed from [2] instead of an own
  implementation of the same idea.
- Many fixes and improvements thanks to Liam's adoption for maple tree
  nodes.
- Userspace Testing stubs by Liam.
- Reduced limitations/todos - hooking to kfree_rcu() is complete,
  prefilled sheaves can exceed cache's sheaf_capacity.
- Link to v1: https://lore.kernel.org/r/20241112-slub-percpu-caches-v1-0-ddc0bdc27e05 at suse.cz

---
Alexei Starovoitov (1):
      locking/local_lock: Expose dep_map in local_trylock_t.

Liam R. Howlett (8):
      tools/testing/maple_tree: Fix check_bulk_rebalance() locks
      tools/testing/vma: Implement vm_refcnt reset
      tools/testing: Add support for changes to slab for sheaves
      testing/radix-tree/maple: Hack around kfree_rcu not existing
      tools/testing: Add support for prefilled slab sheafs
      maple_tree: Prefilled sheaf conversion and testing
      maple_tree: Add single node allocation support to maple state
      maple_tree: Convert forking to use the sheaf interface

Pedro Falcato (2):
      maple_tree: Use kfree_rcu in ma_free_rcu
      maple_tree: Replace mt_free_one() with kfree()

Vlastimil Babka (10):
      slab: simplify init_kmem_cache_nodes() error handling
      slab: add opt-in caching layer of percpu sheaves
      slab: add sheaf support for batching kfree_rcu() operations
      slab: sheaf prefilling for guaranteed allocations
      slab: determine barn status racily outside of lock
      slab: skip percpu sheaves for remote object freeing
      slab: allow NUMA restricted allocations to use percpu sheaves
      mm, vma: use percpu sheaves for vm_area_struct cache
      maple_tree: use percpu sheaves for maple_node_cache
      tools/testing: include maple-shim.c in maple.c

 include/linux/local_lock_internal.h |    9 +-
 include/linux/maple_tree.h          |    6 +-
 include/linux/slab.h                |   47 +
 lib/maple_tree.c                    |  399 +++------
 lib/test_maple_tree.c               |    8 +
 mm/slab.h                           |    4 +
 mm/slab_common.c                    |   32 +-
 mm/slub.c                           | 1677 ++++++++++++++++++++++++++++++++++-
 mm/vma_init.c                       |    1 +
 tools/include/linux/slab.h          |  165 +++-
 tools/testing/radix-tree/maple.c    |  483 +---------
 tools/testing/shared/linux.c        |  120 ++-
 tools/testing/shared/maple-shared.h |   11 +
 tools/testing/shared/maple-shim.c   |    7 +
 tools/testing/vma/vma_internal.h    |   98 +-
 15 files changed, 2156 insertions(+), 911 deletions(-)
---
base-commit: 82efd569a8909f2b13140c1b3de88535aea0b051
change-id: 20231128-slub-percpu-caches-9441892011d7

Best regards,
-- 
Vlastimil Babka <vbabka at suse.cz>



From vbabka at suse.cz  Wed Sep  3 05:59:45 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:45 +0200
Subject: [PATCH v7 03/21] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-3-71c114cdefef@suse.cz>

Specifying a non-zero value for a new struct kmem_cache_args field
sheaf_capacity will setup a caching layer of percpu arrays called
sheaves of given capacity for the created cache.

Allocations from the cache will allocate via the percpu sheaves (main or
spare) as long as they have no NUMA node preference. Frees will also
put the object back into one of the sheaves.

When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn. If none
are available and the allocation is allowed to block, an empty sheaf is
refilled from slab(s) by an internal bulk alloc operation. When both
percpu sheaves are full during freeing, the barn can replace a full one
with an empty one, unless over a full sheaves limit. In that case a
sheaf is flushed to slab(s) by an internal bulk free operation. Flushing
sheaves and barns is also wired to the existing cpu flushing and cache
shrinking operations.

The sheaves do not distinguish NUMA locality of the cached objects. If
an allocation is requested with kmem_cache_alloc_node() (or a mempolicy
with strict_numa mode enabled) with a specific node (not NUMA_NO_NODE),
the sheaves are bypassed.

The bulk operations exposed to slab users also try to utilize the
sheaves as long as the necessary (full or empty) sheaves are available
on the cpu or in the barn. Once depleted, they will fallback to bulk
alloc/free to slabs directly to avoid double copying.

The sheaf_capacity value is exported in sysfs for observability.

Sysfs CONFIG_SLUB_STATS counters alloc_cpu_sheaf and free_cpu_sheaf
count objects allocated or freed using the sheaves (and thus not
counting towards the other alloc/free path counters). Counters
sheaf_refill and sheaf_flush count objects filled or flushed from or to
slab pages, and can be used to assess how effective the caching is. The
refill and flush operations will also count towards the usual
alloc_fastpath/slowpath, free_fastpath/slowpath and other counters for
the backing slabs.  For barn operations, barn_get and barn_put count how
many full sheaves were get from or put to the barn, the _fail variants
count how many such requests could not be satisfied mainly  because the
barn was either empty or full. While the barn also holds empty sheaves
to make some operations easier, these are not as critical to mandate own
counters.  Finally, there are sheaf_alloc/sheaf_free counters.

Access to the percpu sheaves is protected by local_trylock() when
potential callers include irq context, and local_lock() otherwise (such
as when we already know the gfp flags allow blocking). The trylock
failures should be rare and we can easily fallback. Each per-NUMA-node
barn has a spin_lock.

When slub_debug is enabled for a cache with sheaf_capacity also
specified, the latter is ignored so that allocations and frees reach the
slow path where debugging hooks are processed. Similarly, we ignore it
with CONFIG_SLUB_TINY which prefers low memory usage to performance.

[boot failure: https://lore.kernel.org/all/583eacf5-c971-451a-9f76-fed0e341b815 at linux.ibm.com/ ]
Reported-and-tested-by: Venkat Rao Bagalkote <venkat88 at linux.ibm.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/slab.h |   31 ++
 mm/slab.h            |    2 +
 mm/slab_common.c     |    5 +-
 mm/slub.c            | 1164 +++++++++++++++++++++++++++++++++++++++++++++++---
 4 files changed, 1143 insertions(+), 59 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index d5a8ab98035cf3e3d9043e3b038e1bebeff05b52..49acbcdc6696fd120c402adf757b3f41660ad50a 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -335,6 +335,37 @@ struct kmem_cache_args {
 	 * %NULL means no constructor.
 	 */
 	void (*ctor)(void *);
+	/**
+	 * @sheaf_capacity: Enable sheaves of given capacity for the cache.
+	 *
+	 * With a non-zero value, allocations from the cache go through caching
+	 * arrays called sheaves. Each cpu has a main sheaf that's always
+	 * present, and a spare sheaf that may be not present. When both become
+	 * empty, there's an attempt to replace an empty sheaf with a full sheaf
+	 * from the per-node barn.
+	 *
+	 * When no full sheaf is available, and gfp flags allow blocking, a
+	 * sheaf is allocated and filled from slab(s) using bulk allocation.
+	 * Otherwise the allocation falls back to the normal operation
+	 * allocating a single object from a slab.
+	 *
+	 * Analogically when freeing and both percpu sheaves are full, the barn
+	 * may replace it with an empty sheaf, unless it's over capacity. In
+	 * that case a sheaf is bulk freed to slab pages.
+	 *
+	 * The sheaves do not enforce NUMA placement of objects, so allocations
+	 * via kmem_cache_alloc_node() with a node specified other than
+	 * NUMA_NO_NODE will bypass them.
+	 *
+	 * Bulk allocation and free operations also try to use the cpu sheaves
+	 * and barn, but fallback to using slab pages directly.
+	 *
+	 * When slub_debug is enabled for the cache, the sheaf_capacity argument
+	 * is ignored.
+	 *
+	 * %0 means no sheaves will be created.
+	 */
+	unsigned int sheaf_capacity;
 };
 
 struct kmem_cache *__kmem_cache_create_args(const char *name,
diff --git a/mm/slab.h b/mm/slab.h
index 248b34c839b7ca39cf14e139c62d116efb97d30f..206987ce44a4d053ebe3b5e50784d2dd23822cd1 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -235,6 +235,7 @@ struct kmem_cache {
 #ifndef CONFIG_SLUB_TINY
 	struct kmem_cache_cpu __percpu *cpu_slab;
 #endif
+	struct slub_percpu_sheaves __percpu *cpu_sheaves;
 	/* Used for retrieving partial slabs, etc. */
 	slab_flags_t flags;
 	unsigned long min_partial;
@@ -248,6 +249,7 @@ struct kmem_cache {
 	/* Number of per cpu partial slabs to keep around */
 	unsigned int cpu_partial_slabs;
 #endif
+	unsigned int sheaf_capacity;
 	struct kmem_cache_order_objects oo;
 
 	/* Allocation and freeing of slabs */
diff --git a/mm/slab_common.c b/mm/slab_common.c
index bfe7c40eeee1a01c175766935c1e3c0304434a53..e2b197e47866c30acdbd1fee4159f262a751c5a7 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -163,6 +163,9 @@ int slab_unmergeable(struct kmem_cache *s)
 		return 1;
 #endif
 
+	if (s->cpu_sheaves)
+		return 1;
+
 	/*
 	 * We may have set a slab to be unmergeable during bootstrap.
 	 */
@@ -321,7 +324,7 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
 		    object_size - args->usersize < args->useroffset))
 		args->usersize = args->useroffset = 0;
 
-	if (!args->usersize)
+	if (!args->usersize && !args->sheaf_capacity)
 		s = __kmem_cache_alias(name, object_size, args->align, flags,
 				       args->ctor);
 	if (s)
diff --git a/mm/slub.c b/mm/slub.c
index 9f671ec76131c4b0b28d5d568aa45842b5efb6d4..42cb5848f1cecb17174967ff8b102b20a50110e3 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -363,8 +363,10 @@ static inline void debugfs_slab_add(struct kmem_cache *s) { }
 #endif
 
 enum stat_item {
+	ALLOC_PCS,		/* Allocation from percpu sheaf */
 	ALLOC_FASTPATH,		/* Allocation from cpu slab */
 	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
+	FREE_PCS,		/* Free to percpu sheaf */
 	FREE_FASTPATH,		/* Free to cpu slab */
 	FREE_SLOWPATH,		/* Freeing not to cpu slab */
 	FREE_FROZEN,		/* Freeing to frozen slab */
@@ -389,6 +391,14 @@ enum stat_item {
 	CPU_PARTIAL_FREE,	/* Refill cpu partial on free */
 	CPU_PARTIAL_NODE,	/* Refill cpu partial from node partial */
 	CPU_PARTIAL_DRAIN,	/* Drain cpu partial to node partial */
+	SHEAF_FLUSH,		/* Objects flushed from a sheaf */
+	SHEAF_REFILL,		/* Objects refilled to a sheaf */
+	SHEAF_ALLOC,		/* Allocation of an empty sheaf */
+	SHEAF_FREE,		/* Freeing of an empty sheaf */
+	BARN_GET,		/* Got full sheaf from barn */
+	BARN_GET_FAIL,		/* Failed to get full sheaf from barn */
+	BARN_PUT,		/* Put full sheaf to barn */
+	BARN_PUT_FAIL,		/* Failed to put full sheaf to barn */
 	NR_SLUB_STAT_ITEMS
 };
 
@@ -435,6 +445,32 @@ void stat_add(const struct kmem_cache *s, enum stat_item si, int v)
 #endif
 }
 
+#define MAX_FULL_SHEAVES	10
+#define MAX_EMPTY_SHEAVES	10
+
+struct node_barn {
+	spinlock_t lock;
+	struct list_head sheaves_full;
+	struct list_head sheaves_empty;
+	unsigned int nr_full;
+	unsigned int nr_empty;
+};
+
+struct slab_sheaf {
+	union {
+		struct rcu_head rcu_head;
+		struct list_head barn_list;
+	};
+	unsigned int size;
+	void *objects[];
+};
+
+struct slub_percpu_sheaves {
+	local_trylock_t lock;
+	struct slab_sheaf *main; /* never NULL when unlocked */
+	struct slab_sheaf *spare; /* empty or full, may be NULL */
+};
+
 /*
  * The slab lists for all objects.
  */
@@ -447,6 +483,7 @@ struct kmem_cache_node {
 	atomic_long_t total_objects;
 	struct list_head full;
 #endif
+	struct node_barn *barn;
 };
 
 static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
@@ -454,6 +491,12 @@ static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
 	return s->node[node];
 }
 
+/* Get the barn of the current cpu's memory node */
+static inline struct node_barn *get_barn(struct kmem_cache *s)
+{
+	return get_node(s, numa_mem_id())->barn;
+}
+
 /*
  * Iterator over all nodes. The body will be executed for each node that has
  * a kmem_cache_node structure allocated (which is true for all online nodes)
@@ -470,12 +513,19 @@ static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
  */
 static nodemask_t slab_nodes;
 
-#ifndef CONFIG_SLUB_TINY
 /*
  * Workqueue used for flush_cpu_slab().
  */
 static struct workqueue_struct *flushwq;
-#endif
+
+struct slub_flush_work {
+	struct work_struct work;
+	struct kmem_cache *s;
+	bool skip;
+};
+
+static DEFINE_MUTEX(flush_lock);
+static DEFINE_PER_CPU(struct slub_flush_work, slub_flush);
 
 /********************************************************************
  * 			Core slab cache functions
@@ -2473,6 +2523,360 @@ static void *setup_object(struct kmem_cache *s, void *object)
 	return object;
 }
 
+static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slab_sheaf *sheaf = kzalloc(struct_size(sheaf, objects,
+					s->sheaf_capacity), gfp);
+
+	if (unlikely(!sheaf))
+		return NULL;
+
+	stat(s, SHEAF_ALLOC);
+
+	return sheaf;
+}
+
+static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+	kfree(sheaf);
+
+	stat(s, SHEAF_FREE);
+}
+
+static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
+				   size_t size, void **p);
+
+
+static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
+			 gfp_t gfp)
+{
+	int to_fill = s->sheaf_capacity - sheaf->size;
+	int filled;
+
+	if (!to_fill)
+		return 0;
+
+	filled = __kmem_cache_alloc_bulk(s, gfp, to_fill,
+					 &sheaf->objects[sheaf->size]);
+
+	sheaf->size += filled;
+
+	stat_add(s, SHEAF_REFILL, filled);
+
+	if (filled < to_fill)
+		return -ENOMEM;
+
+	return 0;
+}
+
+
+static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slab_sheaf *sheaf = alloc_empty_sheaf(s, gfp);
+
+	if (!sheaf)
+		return NULL;
+
+	if (refill_sheaf(s, sheaf, gfp)) {
+		free_empty_sheaf(s, sheaf);
+		return NULL;
+	}
+
+	return sheaf;
+}
+
+/*
+ * Maximum number of objects freed during a single flush of main pcs sheaf.
+ * Translates directly to an on-stack array size.
+ */
+#define PCS_BATCH_MAX	32U
+
+static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p);
+
+/*
+ * Free all objects from the main sheaf. In order to perform
+ * __kmem_cache_free_bulk() outside of cpu_sheaves->lock, work in batches where
+ * object pointers are moved to a on-stack array under the lock. To bound the
+ * stack usage, limit each batch to PCS_BATCH_MAX.
+ *
+ * returns true if at least partially flushed
+ */
+static bool sheaf_flush_main(struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+	unsigned int batch, remaining;
+	void *objects[PCS_BATCH_MAX];
+	struct slab_sheaf *sheaf;
+	bool ret = false;
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return ret;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	sheaf = pcs->main;
+
+	batch = min(PCS_BATCH_MAX, sheaf->size);
+
+	sheaf->size -= batch;
+	memcpy(objects, sheaf->objects + sheaf->size, batch * sizeof(void *));
+
+	remaining = sheaf->size;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	__kmem_cache_free_bulk(s, batch, &objects[0]);
+
+	stat_add(s, SHEAF_FLUSH, batch);
+
+	ret = true;
+
+	if (remaining)
+		goto next_batch;
+
+	return ret;
+}
+
+/*
+ * Free all objects from a sheaf that's unused, i.e. not linked to any
+ * cpu_sheaves, so we need no locking and batching. The locking is also not
+ * necessary when flushing cpu's sheaves (both spare and main) during cpu
+ * hotremove as the cpu is not executing anymore.
+ */
+static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+	if (!sheaf->size)
+		return;
+
+	stat_add(s, SHEAF_FLUSH, sheaf->size);
+
+	__kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
+
+	sheaf->size = 0;
+}
+
+/*
+ * Caller needs to make sure migration is disabled in order to fully flush
+ * single cpu's sheaves
+ *
+ * must not be called from an irq
+ *
+ * flushing operations are rare so let's keep it simple and flush to slabs
+ * directly, skipping the barn
+ */
+static void pcs_flush_all(struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *spare;
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	spare = pcs->spare;
+	pcs->spare = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (spare) {
+		sheaf_flush_unused(s, spare);
+		free_empty_sheaf(s, spare);
+	}
+
+	sheaf_flush_main(s);
+}
+
+static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+	/* The cpu is not executing anymore so we don't need pcs->lock */
+	sheaf_flush_unused(s, pcs->main);
+	if (pcs->spare) {
+		sheaf_flush_unused(s, pcs->spare);
+		free_empty_sheaf(s, pcs->spare);
+		pcs->spare = NULL;
+	}
+}
+
+static void pcs_destroy(struct kmem_cache *s)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct slub_percpu_sheaves *pcs;
+
+		pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+		/* can happen when unwinding failed create */
+		if (!pcs->main)
+			continue;
+
+		/*
+		 * We have already passed __kmem_cache_shutdown() so everything
+		 * was flushed and there should be no objects allocated from
+		 * slabs, otherwise kmem_cache_destroy() would have aborted.
+		 * Therefore something would have to be really wrong if the
+		 * warnings here trigger, and we should rather leave objects and
+		 * sheaves to leak in that case.
+		 */
+
+		WARN_ON(pcs->spare);
+
+		if (!WARN_ON(pcs->main->size)) {
+			free_empty_sheaf(s, pcs->main);
+			pcs->main = NULL;
+		}
+	}
+
+	free_percpu(s->cpu_sheaves);
+	s->cpu_sheaves = NULL;
+}
+
+static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn)
+{
+	struct slab_sheaf *empty = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_empty) {
+		empty = list_first_entry(&barn->sheaves_empty,
+					 struct slab_sheaf, barn_list);
+		list_del(&empty->barn_list);
+		barn->nr_empty--;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return empty;
+}
+
+/*
+ * The following two functions are used mainly in cases where we have to undo an
+ * intended action due to a race or cpu migration. Thus they do not check the
+ * empty or full sheaf limits for simplicity.
+ */
+
+static void barn_put_empty_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_add(&sheaf->barn_list, &barn->sheaves_empty);
+	barn->nr_empty++;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+}
+
+static void barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_add(&sheaf->barn_list, &barn->sheaves_full);
+	barn->nr_full++;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+}
+
+/*
+ * If a full sheaf is available, return it and put the supplied empty one to
+ * barn. We ignore the limit on empty sheaves as the number of sheaves doesn't
+ * change.
+ */
+static struct slab_sheaf *
+barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty)
+{
+	struct slab_sheaf *full = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full) {
+		full = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
+					barn_list);
+		list_del(&full->barn_list);
+		list_add(&empty->barn_list, &barn->sheaves_empty);
+		barn->nr_full--;
+		barn->nr_empty++;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return full;
+}
+
+/*
+ * If an empty sheaf is available, return it and put the supplied full one to
+ * barn. But if there are too many full sheaves, reject this with -E2BIG.
+ */
+static struct slab_sheaf *
+barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full)
+{
+	struct slab_sheaf *empty;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full >= MAX_FULL_SHEAVES) {
+		empty = ERR_PTR(-E2BIG);
+	} else if (!barn->nr_empty) {
+		empty = ERR_PTR(-ENOMEM);
+	} else {
+		empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf,
+					 barn_list);
+		list_del(&empty->barn_list);
+		list_add(&full->barn_list, &barn->sheaves_full);
+		barn->nr_empty--;
+		barn->nr_full++;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return empty;
+}
+
+static void barn_init(struct node_barn *barn)
+{
+	spin_lock_init(&barn->lock);
+	INIT_LIST_HEAD(&barn->sheaves_full);
+	INIT_LIST_HEAD(&barn->sheaves_empty);
+	barn->nr_full = 0;
+	barn->nr_empty = 0;
+}
+
+static void barn_shrink(struct kmem_cache *s, struct node_barn *barn)
+{
+	struct list_head empty_list;
+	struct list_head full_list;
+	struct slab_sheaf *sheaf, *sheaf2;
+	unsigned long flags;
+
+	INIT_LIST_HEAD(&empty_list);
+	INIT_LIST_HEAD(&full_list);
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_splice_init(&barn->sheaves_full, &full_list);
+	barn->nr_full = 0;
+	list_splice_init(&barn->sheaves_empty, &empty_list);
+	barn->nr_empty = 0;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	list_for_each_entry_safe(sheaf, sheaf2, &full_list, barn_list) {
+		sheaf_flush_unused(s, sheaf);
+		free_empty_sheaf(s, sheaf);
+	}
+
+	list_for_each_entry_safe(sheaf, sheaf2, &empty_list, barn_list)
+		free_empty_sheaf(s, sheaf);
+}
+
 /*
  * Slab allocation and freeing
  */
@@ -3344,11 +3748,42 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu)
 	put_partials_cpu(s, c);
 }
 
-struct slub_flush_work {
-	struct work_struct work;
-	struct kmem_cache *s;
-	bool skip;
-};
+static inline void flush_this_cpu_slab(struct kmem_cache *s)
+{
+	struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab);
+
+	if (c->slab)
+		flush_slab(s, c);
+
+	put_partials(s);
+}
+
+static bool has_cpu_slab(int cpu, struct kmem_cache *s)
+{
+	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
+
+	return c->slab || slub_percpu_partial(c);
+}
+
+#else /* CONFIG_SLUB_TINY */
+static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
+static inline bool has_cpu_slab(int cpu, struct kmem_cache *s) { return false; }
+static inline void flush_this_cpu_slab(struct kmem_cache *s) { }
+#endif /* CONFIG_SLUB_TINY */
+
+static bool has_pcs_used(int cpu, struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	if (!s->cpu_sheaves)
+		return false;
+
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+	return (pcs->spare || pcs->main->size);
+}
+
+static void pcs_flush_all(struct kmem_cache *s);
 
 /*
  * Flush cpu slab.
@@ -3358,30 +3793,18 @@ struct slub_flush_work {
 static void flush_cpu_slab(struct work_struct *w)
 {
 	struct kmem_cache *s;
-	struct kmem_cache_cpu *c;
 	struct slub_flush_work *sfw;
 
 	sfw = container_of(w, struct slub_flush_work, work);
 
 	s = sfw->s;
-	c = this_cpu_ptr(s->cpu_slab);
-
-	if (c->slab)
-		flush_slab(s, c);
-
-	put_partials(s);
-}
 
-static bool has_cpu_slab(int cpu, struct kmem_cache *s)
-{
-	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
+	if (s->cpu_sheaves)
+		pcs_flush_all(s);
 
-	return c->slab || slub_percpu_partial(c);
+	flush_this_cpu_slab(s);
 }
 
-static DEFINE_MUTEX(flush_lock);
-static DEFINE_PER_CPU(struct slub_flush_work, slub_flush);
-
 static void flush_all_cpus_locked(struct kmem_cache *s)
 {
 	struct slub_flush_work *sfw;
@@ -3392,7 +3815,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)
 
 	for_each_online_cpu(cpu) {
 		sfw = &per_cpu(slub_flush, cpu);
-		if (!has_cpu_slab(cpu, s)) {
+		if (!has_cpu_slab(cpu, s) && !has_pcs_used(cpu, s)) {
 			sfw->skip = true;
 			continue;
 		}
@@ -3428,19 +3851,15 @@ static int slub_cpu_dead(unsigned int cpu)
 	struct kmem_cache *s;
 
 	mutex_lock(&slab_mutex);
-	list_for_each_entry(s, &slab_caches, list)
+	list_for_each_entry(s, &slab_caches, list) {
 		__flush_cpu_slab(s, cpu);
+		if (s->cpu_sheaves)
+			__pcs_flush_all_cpu(s, cpu);
+	}
 	mutex_unlock(&slab_mutex);
 	return 0;
 }
 
-#else /* CONFIG_SLUB_TINY */
-static inline void flush_all_cpus_locked(struct kmem_cache *s) { }
-static inline void flush_all(struct kmem_cache *s) { }
-static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
-static inline int slub_cpu_dead(unsigned int cpu) { return 0; }
-#endif /* CONFIG_SLUB_TINY */
-
 /*
  * Check if the objects in a per cpu structure fit numa
  * locality expectations.
@@ -4191,30 +4610,240 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 }
 
 /*
- * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
- * have the fastpath folded into their functions. So no function call
- * overhead for requests that can be satisfied on the fastpath.
- *
- * The fastpath works by first checking if the lockless freelist can be used.
- * If not then __slab_alloc is called for slow processing.
+ * Replace the empty main sheaf with a (at least partially) full sheaf.
  *
- * Otherwise we can simply pick the next object from the lockless free list.
+ * Must be called with the cpu_sheaves local lock locked. If successful, returns
+ * the pcs pointer and the local lock locked (possibly on a different cpu than
+ * initially called). If not successful, returns NULL and the local lock
+ * unlocked.
  */
-static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
-		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static struct slub_percpu_sheaves *
+__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
 {
-	void *object;
-	bool init = false;
+	struct slab_sheaf *empty = NULL;
+	struct slab_sheaf *full;
+	struct node_barn *barn;
+	bool can_alloc;
 
-	s = slab_pre_alloc_hook(s, gfpflags);
-	if (unlikely(!s))
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	if (pcs->spare && pcs->spare->size > 0) {
+		swap(pcs->main, pcs->spare);
+		return pcs;
+	}
+
+	barn = get_barn(s);
+
+	full = barn_replace_empty_sheaf(barn, pcs->main);
+
+	if (full) {
+		stat(s, BARN_GET);
+		pcs->main = full;
+		return pcs;
+	}
+
+	stat(s, BARN_GET_FAIL);
+
+	can_alloc = gfpflags_allow_blocking(gfp);
+
+	if (can_alloc) {
+		if (pcs->spare) {
+			empty = pcs->spare;
+			pcs->spare = NULL;
+		} else {
+			empty = barn_get_empty_sheaf(barn);
+		}
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (!can_alloc)
+		return NULL;
+
+	if (empty) {
+		if (!refill_sheaf(s, empty, gfp)) {
+			full = empty;
+		} else {
+			/*
+			 * we must be very low on memory so don't bother
+			 * with the barn
+			 */
+			free_empty_sheaf(s, empty);
+		}
+	} else {
+		full = alloc_full_sheaf(s, gfp);
+	}
+
+	if (!full)
+		return NULL;
+
+	/*
+	 * we can reach here only when gfpflags_allow_blocking
+	 * so this must not be an irq
+	 */
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	/*
+	 * If we are returning empty sheaf, we either got it from the
+	 * barn or had to allocate one. If we are returning a full
+	 * sheaf, it's due to racing or being migrated to a different
+	 * cpu. Breaching the barn's sheaf limits should be thus rare
+	 * enough so just ignore them to simplify the recovery.
+	 */
+
+	if (pcs->main->size == 0) {
+		barn_put_empty_sheaf(barn, pcs->main);
+		pcs->main = full;
+		return pcs;
+	}
+
+	if (!pcs->spare) {
+		pcs->spare = full;
+		return pcs;
+	}
+
+	if (pcs->spare->size == 0) {
+		barn_put_empty_sheaf(barn, pcs->spare);
+		pcs->spare = full;
+		return pcs;
+	}
+
+	barn_put_full_sheaf(barn, full);
+	stat(s, BARN_PUT);
+
+	return pcs;
+}
+
+static __fastpath_inline
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slub_percpu_sheaves *pcs;
+	void *object;
+
+#ifdef CONFIG_NUMA
+	if (static_branch_unlikely(&strict_numa)) {
+		if (current->mempolicy)
+			return NULL;
+	}
+#endif
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return NULL;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == 0)) {
+		pcs = __pcs_replace_empty_main(s, pcs, gfp);
+		if (unlikely(!pcs))
+			return NULL;
+	}
+
+	object = pcs->main->objects[--pcs->main->size];
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat(s, ALLOC_PCS);
+
+	return object;
+}
+
+static __fastpath_inline
+unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *main;
+	unsigned int allocated = 0;
+	unsigned int batch;
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return allocated;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == 0)) {
+
+		struct slab_sheaf *full;
+
+		if (pcs->spare && pcs->spare->size > 0) {
+			swap(pcs->main, pcs->spare);
+			goto do_alloc;
+		}
+
+		full = barn_replace_empty_sheaf(get_barn(s), pcs->main);
+
+		if (full) {
+			stat(s, BARN_GET);
+			pcs->main = full;
+			goto do_alloc;
+		}
+
+		stat(s, BARN_GET_FAIL);
+
+		local_unlock(&s->cpu_sheaves->lock);
+
+		/*
+		 * Once full sheaves in barn are depleted, let the bulk
+		 * allocation continue from slab pages, otherwise we would just
+		 * be copying arrays of pointers twice.
+		 */
+		return allocated;
+	}
+
+do_alloc:
+
+	main = pcs->main;
+	batch = min(size, main->size);
+
+	main->size -= batch;
+	memcpy(p, main->objects + main->size, batch * sizeof(void *));
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat_add(s, ALLOC_PCS, batch);
+
+	allocated += batch;
+
+	if (batch < size) {
+		p += batch;
+		size -= batch;
+		goto next_batch;
+	}
+
+	return allocated;
+}
+
+
+/*
+ * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
+ * have the fastpath folded into their functions. So no function call
+ * overhead for requests that can be satisfied on the fastpath.
+ *
+ * The fastpath works by first checking if the lockless freelist can be used.
+ * If not then __slab_alloc is called for slow processing.
+ *
+ * Otherwise we can simply pick the next object from the lockless free list.
+ */
+static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
+		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+{
+	void *object;
+	bool init = false;
+
+	s = slab_pre_alloc_hook(s, gfpflags);
+	if (unlikely(!s))
 		return NULL;
 
 	object = kfence_alloc(s, orig_size, gfpflags);
 	if (unlikely(object))
 		goto out;
 
-	object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
+	if (s->cpu_sheaves && node == NUMA_NO_NODE)
+		object = alloc_from_pcs(s, gfpflags);
+
+	if (!object)
+		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
 
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
@@ -4591,6 +5220,295 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab,
 	discard_slab(s, slab);
 }
 
+/*
+ * pcs is locked. We should have get rid of the spare sheaf and obtained an
+ * empty sheaf, while the main sheaf is full. We want to install the empty sheaf
+ * as a main sheaf, and make the current main sheaf a spare sheaf.
+ *
+ * However due to having relinquished the cpu_sheaves lock when obtaining
+ * the empty sheaf, we need to handle some unlikely but possible cases.
+ *
+ * If we put any sheaf to barn here, it's because we were interrupted or have
+ * been migrated to a different cpu, which should be rare enough so just ignore
+ * the barn's limits to simplify the handling.
+ *
+ * An alternative scenario that gets us here is when we fail
+ * barn_replace_full_sheaf(), because there's no empty sheaf available in the
+ * barn, so we had to allocate it by alloc_empty_sheaf(). But because we saw the
+ * limit on full sheaves was not exceeded, we assume it didn't change and just
+ * put the full sheaf there.
+ */
+static void __pcs_install_empty_sheaf(struct kmem_cache *s,
+		struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty)
+{
+	struct node_barn *barn;
+
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	/* This is what we expect to find if nobody interrupted us. */
+	if (likely(!pcs->spare)) {
+		pcs->spare = pcs->main;
+		pcs->main = empty;
+		return;
+	}
+
+	barn = get_barn(s);
+
+	/*
+	 * Unlikely because if the main sheaf had space, we would have just
+	 * freed to it. Get rid of our empty sheaf.
+	 */
+	if (pcs->main->size < s->sheaf_capacity) {
+		barn_put_empty_sheaf(barn, empty);
+		return;
+	}
+
+	/* Also unlikely for the same reason */
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		barn_put_empty_sheaf(barn, empty);
+		return;
+	}
+
+	/*
+	 * We probably failed barn_replace_full_sheaf() due to no empty sheaf
+	 * available there, but we allocated one, so finish the job.
+	 */
+	barn_put_full_sheaf(barn, pcs->main);
+	stat(s, BARN_PUT);
+	pcs->main = empty;
+}
+
+/*
+ * Replace the full main sheaf with a (at least partially) empty sheaf.
+ *
+ * Must be called with the cpu_sheaves local lock locked. If successful, returns
+ * the pcs pointer and the local lock locked (possibly on a different cpu than
+ * initially called). If not successful, returns NULL and the local lock
+ * unlocked.
+ */
+static struct slub_percpu_sheaves *
+__pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs)
+{
+	struct slab_sheaf *empty;
+	struct node_barn *barn;
+	bool put_fail;
+
+restart:
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	barn = get_barn(s);
+	put_fail = false;
+
+	if (!pcs->spare) {
+		empty = barn_get_empty_sheaf(barn);
+		if (empty) {
+			pcs->spare = pcs->main;
+			pcs->main = empty;
+			return pcs;
+		}
+		goto alloc_empty;
+	}
+
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		return pcs;
+	}
+
+	empty = barn_replace_full_sheaf(barn, pcs->main);
+
+	if (!IS_ERR(empty)) {
+		stat(s, BARN_PUT);
+		pcs->main = empty;
+		return pcs;
+	}
+
+	if (PTR_ERR(empty) == -E2BIG) {
+		/* Since we got here, spare exists and is full */
+		struct slab_sheaf *to_flush = pcs->spare;
+
+		stat(s, BARN_PUT_FAIL);
+
+		pcs->spare = NULL;
+		local_unlock(&s->cpu_sheaves->lock);
+
+		sheaf_flush_unused(s, to_flush);
+		empty = to_flush;
+		goto got_empty;
+	}
+
+	/*
+	 * We could not replace full sheaf because barn had no empty
+	 * sheaves. We can still allocate it and put the full sheaf in
+	 * __pcs_install_empty_sheaf(), but if we fail to allocate it,
+	 * make sure to count the fail.
+	 */
+	put_fail = true;
+
+alloc_empty:
+	local_unlock(&s->cpu_sheaves->lock);
+
+	empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+	if (empty)
+		goto got_empty;
+
+	if (put_fail)
+		 stat(s, BARN_PUT_FAIL);
+
+	if (!sheaf_flush_main(s))
+		return NULL;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return NULL;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	/*
+	 * we flushed the main sheaf so it should be empty now,
+	 * but in case we got preempted or migrated, we need to
+	 * check again
+	 */
+	if (pcs->main->size == s->sheaf_capacity)
+		goto restart;
+
+	return pcs;
+
+got_empty:
+	if (!local_trylock(&s->cpu_sheaves->lock)) {
+		barn_put_empty_sheaf(barn, empty);
+		return NULL;
+	}
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	__pcs_install_empty_sheaf(s, pcs, empty);
+
+	return pcs;
+}
+
+/*
+ * Free an object to the percpu sheaves.
+ * The object is expected to have passed slab_free_hook() already.
+ */
+static __fastpath_inline
+bool free_to_pcs(struct kmem_cache *s, void *object)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return false;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == s->sheaf_capacity)) {
+
+		pcs = __pcs_replace_full_main(s, pcs);
+		if (unlikely(!pcs))
+			return false;
+	}
+
+	pcs->main->objects[pcs->main->size++] = object;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat(s, FREE_PCS);
+
+	return true;
+}
+
+/*
+ * Bulk free objects to the percpu sheaves.
+ * Unlike free_to_pcs() this includes the calls to all necessary hooks
+ * and the fallback to freeing to slab pages.
+ */
+static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *main, *empty;
+	bool init = slab_want_init_on_free(s);
+	unsigned int batch, i = 0;
+	struct node_barn *barn;
+
+	while (i < size) {
+		struct slab *slab = virt_to_slab(p[i]);
+
+		memcg_slab_free_hook(s, slab, p + i, 1);
+		alloc_tagging_slab_free_hook(s, slab, p + i, 1);
+
+		if (unlikely(!slab_free_hook(s, p[i], init, false))) {
+			p[i] = p[--size];
+			if (!size)
+				return;
+			continue;
+		}
+
+		i++;
+	}
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto fallback;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (likely(pcs->main->size < s->sheaf_capacity))
+		goto do_free;
+
+	barn = get_barn(s);
+
+	if (!pcs->spare) {
+		empty = barn_get_empty_sheaf(barn);
+		if (!empty)
+			goto no_empty;
+
+		pcs->spare = pcs->main;
+		pcs->main = empty;
+		goto do_free;
+	}
+
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		goto do_free;
+	}
+
+	empty = barn_replace_full_sheaf(barn, pcs->main);
+	if (IS_ERR(empty)) {
+		stat(s, BARN_PUT_FAIL);
+		goto no_empty;
+	}
+
+	stat(s, BARN_PUT);
+	pcs->main = empty;
+
+do_free:
+	main = pcs->main;
+	batch = min(size, s->sheaf_capacity - main->size);
+
+	memcpy(main->objects + main->size, p, batch * sizeof(void *));
+	main->size += batch;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat_add(s, FREE_PCS, batch);
+
+	if (batch < size) {
+		p += batch;
+		size -= batch;
+		goto next_batch;
+	}
+
+	return;
+
+no_empty:
+	local_unlock(&s->cpu_sheaves->lock);
+
+	/*
+	 * if we depleted all empty sheaves in the barn or there are too
+	 * many full sheaves, free the rest to slab pages
+	 */
+fallback:
+	__kmem_cache_free_bulk(s, size, p);
+}
+
 #ifndef CONFIG_SLUB_TINY
 /*
  * Fastpath with forced inlining to produce a kfree and kmem_cache_free that
@@ -4677,7 +5595,10 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 	memcg_slab_free_hook(s, slab, &object, 1);
 	alloc_tagging_slab_free_hook(s, slab, &object, 1);
 
-	if (likely(slab_free_hook(s, object, slab_want_init_on_free(s), false)))
+	if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
+		return;
+
+	if (!s->cpu_sheaves || !free_to_pcs(s, object))
 		do_slab_free(s, slab, object, object, 1, addr);
 }
 
@@ -5273,6 +6194,15 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)
 	if (!size)
 		return;
 
+	/*
+	 * freeing to sheaves is so incompatible with the detached freelist so
+	 * once we go that way, we have to do everything differently
+	 */
+	if (s && s->cpu_sheaves) {
+		free_to_pcs_bulk(s, size, p);
+		return;
+	}
+
 	do {
 		struct detached_freelist df;
 
@@ -5391,7 +6321,7 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
 int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 				 void **p)
 {
-	int i;
+	unsigned int i = 0;
 
 	if (!size)
 		return 0;
@@ -5400,9 +6330,20 @@ int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 	if (unlikely(!s))
 		return 0;
 
-	i = __kmem_cache_alloc_bulk(s, flags, size, p);
-	if (unlikely(i == 0))
-		return 0;
+	if (s->cpu_sheaves)
+		i = alloc_from_pcs_bulk(s, size, p);
+
+	if (i < size) {
+		/*
+		 * If we ran out of memory, don't bother with freeing back to
+		 * the percpu sheaves, we have bigger problems.
+		 */
+		if (unlikely(__kmem_cache_alloc_bulk(s, flags, size - i, p + i) == 0)) {
+			if (i > 0)
+				__kmem_cache_free_bulk(s, i, p);
+			return 0;
+		}
+	}
 
 	/*
 	 * memcg and kmem_cache debug support and memory initialization.
@@ -5412,11 +6353,11 @@ int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 		    slab_want_init_on_alloc(flags, s), s->object_size))) {
 		return 0;
 	}
-	return i;
+
+	return size;
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof);
 
-
 /*
  * Object placement in a slab is made very easy because we always start at
  * offset 0. If we tune the size of the object to the alignment then we can
@@ -5550,7 +6491,7 @@ static inline int calculate_order(unsigned int size)
 }
 
 static void
-init_kmem_cache_node(struct kmem_cache_node *n)
+init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn)
 {
 	n->nr_partial = 0;
 	spin_lock_init(&n->list_lock);
@@ -5560,6 +6501,9 @@ init_kmem_cache_node(struct kmem_cache_node *n)
 	atomic_long_set(&n->total_objects, 0);
 	INIT_LIST_HEAD(&n->full);
 #endif
+	n->barn = barn;
+	if (barn)
+		barn_init(barn);
 }
 
 #ifndef CONFIG_SLUB_TINY
@@ -5590,6 +6534,26 @@ static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
 }
 #endif /* CONFIG_SLUB_TINY */
 
+static int init_percpu_sheaves(struct kmem_cache *s)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct slub_percpu_sheaves *pcs;
+
+		pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+		local_trylock_init(&pcs->lock);
+
+		pcs->main = alloc_empty_sheaf(s, GFP_KERNEL);
+
+		if (!pcs->main)
+			return -ENOMEM;
+	}
+
+	return 0;
+}
+
 static struct kmem_cache *kmem_cache_node;
 
 /*
@@ -5625,7 +6589,7 @@ static void early_kmem_cache_node_alloc(int node)
 	slab->freelist = get_freepointer(kmem_cache_node, n);
 	slab->inuse = 1;
 	kmem_cache_node->node[node] = n;
-	init_kmem_cache_node(n);
+	init_kmem_cache_node(n, NULL);
 	inc_slabs_node(kmem_cache_node, node, slab->objects);
 
 	/*
@@ -5641,6 +6605,13 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
 	struct kmem_cache_node *n;
 
 	for_each_kmem_cache_node(s, node, n) {
+		if (n->barn) {
+			WARN_ON(n->barn->nr_full);
+			WARN_ON(n->barn->nr_empty);
+			kfree(n->barn);
+			n->barn = NULL;
+		}
+
 		s->node[node] = NULL;
 		kmem_cache_free(kmem_cache_node, n);
 	}
@@ -5649,6 +6620,8 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
 void __kmem_cache_release(struct kmem_cache *s)
 {
 	cache_random_seq_destroy(s);
+	if (s->cpu_sheaves)
+		pcs_destroy(s);
 #ifndef CONFIG_SLUB_TINY
 	free_percpu(s->cpu_slab);
 #endif
@@ -5661,18 +6634,29 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
 
 	for_each_node_mask(node, slab_nodes) {
 		struct kmem_cache_node *n;
+		struct node_barn *barn = NULL;
 
 		if (slab_state == DOWN) {
 			early_kmem_cache_node_alloc(node);
 			continue;
 		}
+
+		if (s->cpu_sheaves) {
+			barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node);
+
+			if (!barn)
+				return 0;
+		}
+
 		n = kmem_cache_alloc_node(kmem_cache_node,
 						GFP_KERNEL, node);
-
-		if (!n)
+		if (!n) {
+			kfree(barn);
 			return 0;
+		}
+
+		init_kmem_cache_node(n, barn);
 
-		init_kmem_cache_node(n);
 		s->node[node] = n;
 	}
 	return 1;
@@ -5929,6 +6913,8 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
 	flush_all_cpus_locked(s);
 	/* Attempt to free all objects */
 	for_each_kmem_cache_node(s, node, n) {
+		if (n->barn)
+			barn_shrink(s, n->barn);
 		free_partial(s, n);
 		if (n->nr_partial || node_nr_slabs(n))
 			return 1;
@@ -6132,6 +7118,9 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)
 		for (i = 0; i < SHRINK_PROMOTE_MAX; i++)
 			INIT_LIST_HEAD(promote + i);
 
+		if (n->barn)
+			barn_shrink(s, n->barn);
+
 		spin_lock_irqsave(&n->list_lock, flags);
 
 		/*
@@ -6211,12 +7200,24 @@ static int slab_mem_going_online_callback(int nid)
 	 */
 	mutex_lock(&slab_mutex);
 	list_for_each_entry(s, &slab_caches, list) {
+		struct node_barn *barn = NULL;
+
 		/*
 		 * The structure may already exist if the node was previously
 		 * onlined and offlined.
 		 */
 		if (get_node(s, nid))
 			continue;
+
+		if (s->cpu_sheaves) {
+			barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, nid);
+
+			if (!barn) {
+				ret = -ENOMEM;
+				goto out;
+			}
+		}
+
 		/*
 		 * XXX: kmem_cache_alloc_node will fallback to other nodes
 		 *      since memory is not yet available from the node that
@@ -6224,10 +7225,13 @@ static int slab_mem_going_online_callback(int nid)
 		 */
 		n = kmem_cache_alloc(kmem_cache_node, GFP_KERNEL);
 		if (!n) {
+			kfree(barn);
 			ret = -ENOMEM;
 			goto out;
 		}
-		init_kmem_cache_node(n);
+
+		init_kmem_cache_node(n, barn);
+
 		s->node[nid] = n;
 	}
 	/*
@@ -6440,6 +7444,17 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,
 
 	set_cpu_partial(s);
 
+	if (args->sheaf_capacity && !IS_ENABLED(CONFIG_SLUB_TINY)
+					&& !(s->flags & SLAB_DEBUG_FLAGS)) {
+		s->cpu_sheaves = alloc_percpu(struct slub_percpu_sheaves);
+		if (!s->cpu_sheaves) {
+			err = -ENOMEM;
+			goto out;
+		}
+		// TODO: increase capacity to grow slab_sheaf up to next kmalloc size?
+		s->sheaf_capacity = args->sheaf_capacity;
+	}
+
 #ifdef CONFIG_NUMA
 	s->remote_node_defrag_ratio = 1000;
 #endif
@@ -6456,6 +7471,12 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,
 	if (!alloc_kmem_cache_cpus(s))
 		goto out;
 
+	if (s->cpu_sheaves) {
+		err = init_percpu_sheaves(s);
+		if (err)
+			goto out;
+	}
+
 	err = 0;
 
 	/* Mutex is not taken during early boot */
@@ -6908,6 +7929,12 @@ static ssize_t order_show(struct kmem_cache *s, char *buf)
 }
 SLAB_ATTR_RO(order);
 
+static ssize_t sheaf_capacity_show(struct kmem_cache *s, char *buf)
+{
+	return sysfs_emit(buf, "%u\n", s->sheaf_capacity);
+}
+SLAB_ATTR_RO(sheaf_capacity);
+
 static ssize_t min_partial_show(struct kmem_cache *s, char *buf)
 {
 	return sysfs_emit(buf, "%lu\n", s->min_partial);
@@ -7255,8 +8282,10 @@ static ssize_t text##_store(struct kmem_cache *s,		\
 }								\
 SLAB_ATTR(text);						\
 
+STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
 STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
 STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
+STAT_ATTR(FREE_PCS, free_cpu_sheaf);
 STAT_ATTR(FREE_FASTPATH, free_fastpath);
 STAT_ATTR(FREE_SLOWPATH, free_slowpath);
 STAT_ATTR(FREE_FROZEN, free_frozen);
@@ -7281,6 +8310,14 @@ STAT_ATTR(CPU_PARTIAL_ALLOC, cpu_partial_alloc);
 STAT_ATTR(CPU_PARTIAL_FREE, cpu_partial_free);
 STAT_ATTR(CPU_PARTIAL_NODE, cpu_partial_node);
 STAT_ATTR(CPU_PARTIAL_DRAIN, cpu_partial_drain);
+STAT_ATTR(SHEAF_FLUSH, sheaf_flush);
+STAT_ATTR(SHEAF_REFILL, sheaf_refill);
+STAT_ATTR(SHEAF_ALLOC, sheaf_alloc);
+STAT_ATTR(SHEAF_FREE, sheaf_free);
+STAT_ATTR(BARN_GET, barn_get);
+STAT_ATTR(BARN_GET_FAIL, barn_get_fail);
+STAT_ATTR(BARN_PUT, barn_put);
+STAT_ATTR(BARN_PUT_FAIL, barn_put_fail);
 #endif	/* CONFIG_SLUB_STATS */
 
 #ifdef CONFIG_KFENCE
@@ -7311,6 +8348,7 @@ static struct attribute *slab_attrs[] = {
 	&object_size_attr.attr,
 	&objs_per_slab_attr.attr,
 	&order_attr.attr,
+	&sheaf_capacity_attr.attr,
 	&min_partial_attr.attr,
 	&cpu_partial_attr.attr,
 	&objects_partial_attr.attr,
@@ -7342,8 +8380,10 @@ static struct attribute *slab_attrs[] = {
 	&remote_node_defrag_ratio_attr.attr,
 #endif
 #ifdef CONFIG_SLUB_STATS
+	&alloc_cpu_sheaf_attr.attr,
 	&alloc_fastpath_attr.attr,
 	&alloc_slowpath_attr.attr,
+	&free_cpu_sheaf_attr.attr,
 	&free_fastpath_attr.attr,
 	&free_slowpath_attr.attr,
 	&free_frozen_attr.attr,
@@ -7368,6 +8408,14 @@ static struct attribute *slab_attrs[] = {
 	&cpu_partial_free_attr.attr,
 	&cpu_partial_node_attr.attr,
 	&cpu_partial_drain_attr.attr,
+	&sheaf_flush_attr.attr,
+	&sheaf_refill_attr.attr,
+	&sheaf_alloc_attr.attr,
+	&sheaf_free_attr.attr,
+	&barn_get_attr.attr,
+	&barn_get_fail_attr.attr,
+	&barn_put_attr.attr,
+	&barn_put_fail_attr.attr,
 #endif
 #ifdef CONFIG_FAILSLAB
 	&failslab_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:49 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:49 +0200
Subject: [PATCH v7 07/21] slab: skip percpu sheaves for remote object
 freeing
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-7-71c114cdefef@suse.cz>

Since we don't control the NUMA locality of objects in percpu sheaves,
allocations with node restrictions bypass them. Allocations without
restrictions may however still expect to get local objects with high
probability, and the introduction of sheaves can decrease it due to
freed object from a remote node ending up in percpu sheaves.

The fraction of such remote frees seems low (5% on an 8-node machine)
but it can be expected that some cache or workload specific corner cases
exist. We can either conclude that this is not a problem due to the low
fraction, or we can make remote frees bypass percpu sheaves and go
directly to their slabs. This will make the remote frees more expensive,
but if if's only a small fraction, most frees will still benefit from
the lower overhead of percpu sheaves.

This patch thus makes remote object freeing bypass percpu sheaves,
including bulk freeing, and kfree_rcu() via the rcu_free sheaf. However
it's not intended to be 100% guarantee that percpu sheaves will only
contain local objects. The refill from slabs does not provide that
guarantee in the first place, and there might be cpu migrations
happening when we need to unlock the local_lock. Avoiding all that could
be possible but complicated so we can leave it for later investigation
whether it would be worth it. It can be expected that the more selective
freeing will itself prevent accumulation of remote objects in percpu
sheaves so any such violations would have only short-term effects.

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab_common.c |  7 +++++--
 mm/slub.c        | 42 ++++++++++++++++++++++++++++++++++++------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 2d806e02568532a1000fd3912db6978e945dcfa8..08f5baee1309e5b5f10a22b8b3b0a09dfb314419 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1623,8 +1623,11 @@ static bool kfree_rcu_sheaf(void *obj)
 
 	slab = folio_slab(folio);
 	s = slab->slab_cache;
-	if (s->cpu_sheaves)
-		return __kfree_rcu_sheaf(s, obj);
+	if (s->cpu_sheaves) {
+		if (likely(!IS_ENABLED(CONFIG_NUMA) ||
+			   slab_nid(slab) == numa_mem_id()))
+			return __kfree_rcu_sheaf(s, obj);
+	}
 
 	return false;
 }
diff --git a/mm/slub.c b/mm/slub.c
index d078630f7211b617f390a16fa080da8a1df45355..8e34d6afccf41efad0c6620126e09f874bdcb663 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -472,6 +472,7 @@ struct slab_sheaf {
 	};
 	struct kmem_cache *cache;
 	unsigned int size;
+	int node; /* only used for rcu_sheaf */
 	void *objects[];
 };
 
@@ -5756,7 +5757,7 @@ static void rcu_free_sheaf(struct rcu_head *head)
 	 */
 	__rcu_free_sheaf_prepare(s, sheaf);
 
-	barn = get_node(s, numa_mem_id())->barn;
+	barn = get_node(s, sheaf->node)->barn;
 
 	/* due to slab_free_hook() */
 	if (unlikely(sheaf->size == 0))
@@ -5842,10 +5843,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 
 	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
 
-	if (likely(rcu_sheaf->size < s->sheaf_capacity))
+	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
 		rcu_sheaf = NULL;
-	else
+	} else {
 		pcs->rcu_free = NULL;
+		rcu_sheaf->node = numa_mem_id();
+	}
 
 	local_unlock(&s->cpu_sheaves->lock);
 
@@ -5872,7 +5875,11 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	bool init = slab_want_init_on_free(s);
 	unsigned int batch, i = 0;
 	struct node_barn *barn;
+	void *remote_objects[PCS_BATCH_MAX];
+	unsigned int remote_nr = 0;
+	int node = numa_mem_id();
 
+next_remote_batch:
 	while (i < size) {
 		struct slab *slab = virt_to_slab(p[i]);
 
@@ -5882,7 +5889,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 		if (unlikely(!slab_free_hook(s, p[i], init, false))) {
 			p[i] = p[--size];
 			if (!size)
-				return;
+				goto flush_remote;
+			continue;
+		}
+
+		if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != node)) {
+			remote_objects[remote_nr] = p[i];
+			p[i] = p[--size];
+			if (++remote_nr >= PCS_BATCH_MAX)
+				goto flush_remote;
 			continue;
 		}
 
@@ -5952,6 +5967,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	 */
 fallback:
 	__kmem_cache_free_bulk(s, size, p);
+
+flush_remote:
+	if (remote_nr) {
+		__kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]);
+		if (i < size) {
+			remote_nr = 0;
+			goto next_remote_batch;
+		}
+	}
 }
 
 #ifndef CONFIG_SLUB_TINY
@@ -6043,8 +6067,14 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 	if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
 		return;
 
-	if (!s->cpu_sheaves || !free_to_pcs(s, object))
-		do_slab_free(s, slab, object, object, 1, addr);
+	if (s->cpu_sheaves && likely(!IS_ENABLED(CONFIG_NUMA) ||
+				     slab_nid(slab) == numa_mem_id())) {
+		if (likely(free_to_pcs(s, object))) {
+			return;
+		}
+	}
+
+	do_slab_free(s, slab, object, object, 1, addr);
 }
 
 #ifdef CONFIG_MEMCG

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:50 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:50 +0200
Subject: [PATCH v7 08/21] slab: allow NUMA restricted allocations to use
 percpu sheaves
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-8-71c114cdefef@suse.cz>

Currently allocations asking for a specific node explicitly or via
mempolicy in strict_numa node bypass percpu sheaves. Since sheaves
contain mostly local objects, we can try allocating from them if the
local node happens to be the requested node or allowed by the mempolicy.
If we find the object from percpu sheaves is not from the expected node,
we skip the sheaves - this should be rare.

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 46 insertions(+), 7 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 8e34d6afccf41efad0c6620126e09f874bdcb663..d3d054579a8af710a0cbf68fe9688565c84d3609 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4816,18 +4816,43 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 }
 
 static __fastpath_inline
-void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
 {
 	struct slub_percpu_sheaves *pcs;
+	bool node_requested;
 	void *object;
 
 #ifdef CONFIG_NUMA
-	if (static_branch_unlikely(&strict_numa)) {
-		if (current->mempolicy)
-			return NULL;
+	if (static_branch_unlikely(&strict_numa) &&
+			 node == NUMA_NO_NODE) {
+
+		struct mempolicy *mpol = current->mempolicy;
+
+		if (mpol) {
+			/*
+			 * Special BIND rule support. If the local node
+			 * is in permitted set then do not redirect
+			 * to a particular node.
+			 * Otherwise we apply the memory policy to get
+			 * the node we need to allocate on.
+			 */
+			if (mpol->mode != MPOL_BIND ||
+					!node_isset(numa_mem_id(), mpol->nodes))
+
+				node = mempolicy_slab_node();
+		}
 	}
 #endif
 
+	node_requested = IS_ENABLED(CONFIG_NUMA) && node != NUMA_NO_NODE;
+
+	/*
+	 * We assume the percpu sheaves contain only local objects although it's
+	 * not completely guaranteed, so we verify later.
+	 */
+	if (unlikely(node_requested && node != numa_mem_id()))
+		return NULL;
+
 	if (!local_trylock(&s->cpu_sheaves->lock))
 		return NULL;
 
@@ -4839,7 +4864,21 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
 			return NULL;
 	}
 
-	object = pcs->main->objects[--pcs->main->size];
+	object = pcs->main->objects[pcs->main->size - 1];
+
+	if (unlikely(node_requested)) {
+		/*
+		 * Verify that the object was from the node we want. This could
+		 * be false because of cpu migration during an unlocked part of
+		 * the current allocation or previous freeing process.
+		 */
+		if (folio_nid(virt_to_folio(object)) != node) {
+			local_unlock(&s->cpu_sheaves->lock);
+			return NULL;
+		}
+	}
+
+	pcs->main->size--;
 
 	local_unlock(&s->cpu_sheaves->lock);
 
@@ -4939,8 +4978,8 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 	if (unlikely(object))
 		goto out;
 
-	if (s->cpu_sheaves && node == NUMA_NO_NODE)
-		object = alloc_from_pcs(s, gfpflags);
+	if (s->cpu_sheaves)
+		object = alloc_from_pcs(s, gfpflags, node);
 
 	if (!object)
 		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:52 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:52 +0200
Subject: [PATCH v7 10/21] tools/testing/vma: Implement vm_refcnt reset
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-10-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the reset of the ref count in vma_lock_init().  This is needed if
the vma memory is not zeroed on allocation.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/vma/vma_internal.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 3639aa8dd2b06ebe5b9cfcfe6669994fd38c482d..2f1c586400cfd65413d2ff82199590b43796b18c 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -1418,8 +1418,8 @@ static inline void ksm_exit(struct mm_struct *mm)
 
 static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)
 {
-	(void)vma;
-	(void)reset_refcnt;
+	if (reset_refcnt)
+		refcount_set(&vma->vm_refcnt, 0);
 }
 
 static inline void vma_numab_state_init(struct vm_area_struct *vma)

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:48 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:48 +0200
Subject: [PATCH v7 06/21] slab: determine barn status racily outside of
 lock
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-6-71c114cdefef@suse.cz>

The possibility of many barn operations is determined by the current
number of full or empty sheaves. Taking the barn->lock just to find out
that e.g. there are no empty sheaves results in unnecessary overhead and
lock contention. Thus perform these checks outside of the lock with a
data_race() annotated variable read and fail quickly without taking the
lock.

Checks for sheaf availability that racily succeed have to be obviously
repeated under the lock for correctness, but we can skip repeating
checks if there are too many sheaves on the given list as the limits
don't need to be strict.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index c6ca9b60acd15520410ac08d252bb09e111db6f1..d078630f7211b617f390a16fa080da8a1df45355 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2801,9 +2801,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn)
 	struct slab_sheaf *empty = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_empty))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_empty) {
+	if (likely(barn->nr_empty)) {
 		empty = list_first_entry(&barn->sheaves_empty,
 					 struct slab_sheaf, barn_list);
 		list_del(&empty->barn_list);
@@ -2850,6 +2853,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn)
 	struct slab_sheaf *sheaf = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_full) && !data_race(barn->nr_empty))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
 	if (barn->nr_full) {
@@ -2880,9 +2886,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty)
 	struct slab_sheaf *full = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_full))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_full) {
+	if (likely(barn->nr_full)) {
 		full = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
 					barn_list);
 		list_del(&full->barn_list);
@@ -2906,19 +2915,23 @@ barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full)
 	struct slab_sheaf *empty;
 	unsigned long flags;
 
+	/* we don't repeat this check under barn->lock as it's not critical */
+	if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES)
+		return ERR_PTR(-E2BIG);
+	if (!data_race(barn->nr_empty))
+		return ERR_PTR(-ENOMEM);
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_full >= MAX_FULL_SHEAVES) {
-		empty = ERR_PTR(-E2BIG);
-	} else if (!barn->nr_empty) {
-		empty = ERR_PTR(-ENOMEM);
-	} else {
+	if (likely(barn->nr_empty)) {
 		empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf,
 					 barn_list);
 		list_del(&empty->barn_list);
 		list_add(&full->barn_list, &barn->sheaves_full);
 		barn->nr_empty--;
 		barn->nr_full++;
+	} else {
+		empty = ERR_PTR(-ENOMEM);
 	}
 
 	spin_unlock_irqrestore(&barn->lock, flags);

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:47 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:47 +0200
Subject: [PATCH v7 05/21] slab: sheaf prefilling for guaranteed allocations
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-5-71c114cdefef@suse.cz>

Add functions for efficient guaranteed allocations e.g. in a critical
section that cannot sleep, when the exact number of allocations is not
known beforehand, but an upper limit can be calculated.

kmem_cache_prefill_sheaf() returns a sheaf containing at least given
number of objects.

kmem_cache_alloc_from_sheaf() will allocate an object from the sheaf
and is guaranteed not to fail until depleted.

kmem_cache_return_sheaf() is for giving the sheaf back to the slab
allocator after the critical section. This will also attempt to refill
it to cache's sheaf capacity for better efficiency of sheaves handling,
but it's not stricly necessary to succeed.

kmem_cache_refill_sheaf() can be used to refill a previously obtained
sheaf to requested size. If the current size is sufficient, it does
nothing. If the requested size exceeds cache's sheaf_capacity and the
sheaf's current capacity, the sheaf will be replaced with a new one,
hence the indirect pointer parameter.

kmem_cache_sheaf_size() can be used to query the current size.

The implementation supports requesting sizes that exceed cache's
sheaf_capacity, but it is not efficient - such "oversize" sheaves are
allocated fresh in kmem_cache_prefill_sheaf() and flushed and freed
immediately by kmem_cache_return_sheaf(). kmem_cache_refill_sheaf()
might be especially ineffective when replacing a sheaf with a new one of
a larger capacity. It is therefore better to size cache's
sheaf_capacity accordingly to make oversize sheaves exceptional.

CONFIG_SLUB_STATS counters are added for sheaf prefill and return
operations. A prefill or return is considered _fast when it is able to
grab or return a percpu spare sheaf (even if the sheaf needs a refill to
satisfy the request, as those should amortize over time), and _slow
otherwise (when the barn or even sheaf allocation/freeing has to be
involved). sheaf_prefill_oversize is provided to determine how many
prefills were oversize (counter for oversize returns is not necessary as
all oversize refills result in oversize returns).

When slub_debug is enabled for a cache with sheaves, no percpu sheaves
exist for it, but the prefill functionality is still provided simply by
all prefilled sheaves becoming oversize. If percpu sheaves are not
created for a cache due to not passing the sheaf_capacity argument on
cache creation, the prefills also work through oversize sheaves, but
there's a WARN_ON_ONCE() to indicate the omission.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/slab.h |  16 ++++
 mm/slub.c            | 263 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 279 insertions(+)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 49acbcdc6696fd120c402adf757b3f41660ad50a..680193356ac7a22f9df5cd9b71ff8b81e26404ad 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -829,6 +829,22 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags,
 				   int node) __assume_slab_alignment __malloc;
 #define kmem_cache_alloc_node(...)	alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__))
 
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
+
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf **sheafp, unsigned int size);
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+				       struct slab_sheaf *sheaf);
+
+void *kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *cachep, gfp_t gfp,
+			struct slab_sheaf *sheaf) __assume_slab_alignment __malloc;
+#define kmem_cache_alloc_from_sheaf(...)	\
+			alloc_hooks(kmem_cache_alloc_from_sheaf_noprof(__VA_ARGS__))
+
+unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
+
 /*
  * These macros allow declaring a kmem_buckets * parameter alongside size, which
  * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call
diff --git a/mm/slub.c b/mm/slub.c
index 6a64478befdebdb44cd7896d673bd20a7a6e2889..c6ca9b60acd15520410ac08d252bb09e111db6f1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -401,6 +401,11 @@ enum stat_item {
 	BARN_GET_FAIL,		/* Failed to get full sheaf from barn */
 	BARN_PUT,		/* Put full sheaf to barn */
 	BARN_PUT_FAIL,		/* Failed to put full sheaf to barn */
+	SHEAF_PREFILL_FAST,	/* Sheaf prefill grabbed the spare sheaf */
+	SHEAF_PREFILL_SLOW,	/* Sheaf prefill found no spare sheaf */
+	SHEAF_PREFILL_OVERSIZE,	/* Allocation of oversize sheaf for prefill */
+	SHEAF_RETURN_FAST,	/* Sheaf return reattached spare sheaf */
+	SHEAF_RETURN_SLOW,	/* Sheaf return could not reattach spare */
 	NR_SLUB_STAT_ITEMS
 };
 
@@ -462,6 +467,8 @@ struct slab_sheaf {
 	union {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
+		/* only used for prefilled sheafs */
+		unsigned int capacity;
 	};
 	struct kmem_cache *cache;
 	unsigned int size;
@@ -2838,6 +2845,30 @@ static void barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf
 	spin_unlock_irqrestore(&barn->lock, flags);
 }
 
+static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn)
+{
+	struct slab_sheaf *sheaf = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full) {
+		sheaf = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
+					barn_list);
+		list_del(&sheaf->barn_list);
+		barn->nr_full--;
+	} else if (barn->nr_empty) {
+		sheaf = list_first_entry(&barn->sheaves_empty,
+					 struct slab_sheaf, barn_list);
+		list_del(&sheaf->barn_list);
+		barn->nr_empty--;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return sheaf;
+}
+
 /*
  * If a full sheaf is available, return it and put the supplied empty one to
  * barn. We ignore the limit on empty sheaves as the number of sheaves doesn't
@@ -4970,6 +5001,228 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int nod
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node_noprof);
 
+/*
+ * returns a sheaf that has at least the requested size
+ * when prefilling is needed, do so with given gfp flags
+ *
+ * return NULL if sheaf allocation or prefilling failed
+ */
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *sheaf = NULL;
+
+	if (unlikely(size > s->sheaf_capacity)) {
+
+		/*
+		 * slab_debug disables cpu sheaves intentionally so all
+		 * prefilled sheaves become "oversize" and we give up on
+		 * performance for the debugging. Same with SLUB_TINY.
+		 * Creating a cache without sheaves and then requesting a
+		 * prefilled sheaf is however not expected, so warn.
+		 */
+		WARN_ON_ONCE(s->sheaf_capacity == 0 &&
+			     !IS_ENABLED(CONFIG_SLUB_TINY) &&
+			     !(s->flags & SLAB_DEBUG_FLAGS));
+
+		sheaf = kzalloc(struct_size(sheaf, objects, size), gfp);
+		if (!sheaf)
+			return NULL;
+
+		stat(s, SHEAF_PREFILL_OVERSIZE);
+		sheaf->cache = s;
+		sheaf->capacity = size;
+
+		if (!__kmem_cache_alloc_bulk(s, gfp, size,
+					     &sheaf->objects[0])) {
+			kfree(sheaf);
+			return NULL;
+		}
+
+		sheaf->size = size;
+
+		return sheaf;
+	}
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (pcs->spare) {
+		sheaf = pcs->spare;
+		pcs->spare = NULL;
+		stat(s, SHEAF_PREFILL_FAST);
+	} else {
+		stat(s, SHEAF_PREFILL_SLOW);
+		sheaf = barn_get_full_or_empty_sheaf(get_barn(s));
+		if (sheaf && sheaf->size)
+			stat(s, BARN_GET);
+		else
+			stat(s, BARN_GET_FAIL);
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+
+	if (!sheaf)
+		sheaf = alloc_empty_sheaf(s, gfp);
+
+	if (sheaf && sheaf->size < size) {
+		if (refill_sheaf(s, sheaf, gfp)) {
+			sheaf_flush_unused(s, sheaf);
+			free_empty_sheaf(s, sheaf);
+			sheaf = NULL;
+		}
+	}
+
+	if (sheaf)
+		sheaf->capacity = s->sheaf_capacity;
+
+	return sheaf;
+}
+
+/*
+ * Use this to return a sheaf obtained by kmem_cache_prefill_sheaf()
+ *
+ * If the sheaf cannot simply become the percpu spare sheaf, but there's space
+ * for a full sheaf in the barn, we try to refill the sheaf back to the cache's
+ * sheaf_capacity to avoid handling partially full sheaves.
+ *
+ * If the refill fails because gfp is e.g. GFP_NOWAIT, or the barn is full, the
+ * sheaf is instead flushed and freed.
+ */
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+			     struct slab_sheaf *sheaf)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct node_barn *barn;
+
+	if (unlikely(sheaf->capacity != s->sheaf_capacity)) {
+		sheaf_flush_unused(s, sheaf);
+		kfree(sheaf);
+		return;
+	}
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	barn = get_barn(s);
+
+	if (!pcs->spare) {
+		pcs->spare = sheaf;
+		sheaf = NULL;
+		stat(s, SHEAF_RETURN_FAST);
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (!sheaf)
+		return;
+
+	stat(s, SHEAF_RETURN_SLOW);
+
+	/*
+	 * If the barn has too many full sheaves or we fail to refill the sheaf,
+	 * simply flush and free it.
+	 */
+	if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES ||
+	    refill_sheaf(s, sheaf, gfp)) {
+		sheaf_flush_unused(s, sheaf);
+		free_empty_sheaf(s, sheaf);
+		return;
+	}
+
+	barn_put_full_sheaf(barn, sheaf);
+	stat(s, BARN_PUT);
+}
+
+/*
+ * refill a sheaf previously returned by kmem_cache_prefill_sheaf to at least
+ * the given size
+ *
+ * the sheaf might be replaced by a new one when requesting more than
+ * s->sheaf_capacity objects if such replacement is necessary, but the refill
+ * fails (returning -ENOMEM), the existing sheaf is left intact
+ *
+ * In practice we always refill to full sheaf's capacity.
+ */
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+			    struct slab_sheaf **sheafp, unsigned int size)
+{
+	struct slab_sheaf *sheaf;
+
+	/*
+	 * TODO: do we want to support *sheaf == NULL to be equivalent of
+	 * kmem_cache_prefill_sheaf() ?
+	 */
+	if (!sheafp || !(*sheafp))
+		return -EINVAL;
+
+	sheaf = *sheafp;
+	if (sheaf->size >= size)
+		return 0;
+
+	if (likely(sheaf->capacity >= size)) {
+		if (likely(sheaf->capacity == s->sheaf_capacity))
+			return refill_sheaf(s, sheaf, gfp);
+
+		if (!__kmem_cache_alloc_bulk(s, gfp, sheaf->capacity - sheaf->size,
+					     &sheaf->objects[sheaf->size])) {
+			return -ENOMEM;
+		}
+		sheaf->size = sheaf->capacity;
+
+		return 0;
+	}
+
+	/*
+	 * We had a regular sized sheaf and need an oversize one, or we had an
+	 * oversize one already but need a larger one now.
+	 * This should be a very rare path so let's not complicate it.
+	 */
+	sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
+	if (!sheaf)
+		return -ENOMEM;
+
+	kmem_cache_return_sheaf(s, gfp, *sheafp);
+	*sheafp = sheaf;
+	return 0;
+}
+
+/*
+ * Allocate from a sheaf obtained by kmem_cache_prefill_sheaf()
+ *
+ * Guaranteed not to fail as many allocations as was the requested size.
+ * After the sheaf is emptied, it fails - no fallback to the slab cache itself.
+ *
+ * The gfp parameter is meant only to specify __GFP_ZERO or __GFP_ACCOUNT
+ * memcg charging is forced over limit if necessary, to avoid failure.
+ */
+void *
+kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
+				   struct slab_sheaf *sheaf)
+{
+	void *ret = NULL;
+	bool init;
+
+	if (sheaf->size == 0)
+		goto out;
+
+	ret = sheaf->objects[--sheaf->size];
+
+	init = slab_want_init_on_alloc(gfp, s);
+
+	/* add __GFP_NOFAIL to force successful memcg charging */
+	slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, init, s->object_size);
+out:
+	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE);
+
+	return ret;
+}
+
+unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
+{
+	return sheaf->size;
+}
 /*
  * To avoid unnecessary overhead, we pass through large allocation requests
  * directly to the page allocator. We use __GFP_COMP, because we will need to
@@ -8504,6 +8757,11 @@ STAT_ATTR(BARN_GET, barn_get);
 STAT_ATTR(BARN_GET_FAIL, barn_get_fail);
 STAT_ATTR(BARN_PUT, barn_put);
 STAT_ATTR(BARN_PUT_FAIL, barn_put_fail);
+STAT_ATTR(SHEAF_PREFILL_FAST, sheaf_prefill_fast);
+STAT_ATTR(SHEAF_PREFILL_SLOW, sheaf_prefill_slow);
+STAT_ATTR(SHEAF_PREFILL_OVERSIZE, sheaf_prefill_oversize);
+STAT_ATTR(SHEAF_RETURN_FAST, sheaf_return_fast);
+STAT_ATTR(SHEAF_RETURN_SLOW, sheaf_return_slow);
 #endif	/* CONFIG_SLUB_STATS */
 
 #ifdef CONFIG_KFENCE
@@ -8604,6 +8862,11 @@ static struct attribute *slab_attrs[] = {
 	&barn_get_fail_attr.attr,
 	&barn_put_attr.attr,
 	&barn_put_fail_attr.attr,
+	&sheaf_prefill_fast_attr.attr,
+	&sheaf_prefill_slow_attr.attr,
+	&sheaf_prefill_oversize_attr.attr,
+	&sheaf_return_fast_attr.attr,
+	&sheaf_return_slow_attr.attr,
 #endif
 #ifdef CONFIG_FAILSLAB
 	&failslab_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:53 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:53 +0200
Subject: [PATCH v7 11/21] tools/testing: Add support for changes to slab
 for sheaves
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-11-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The slab changes for sheaves requires more effort in the testing code.
Unite all the kmem_cache work into the tools/include slab header for
both the vma and maple tree testing.

The vma test code also requires importing more #defines to allow for
seamless use of the shared kmem_cache code.

This adds the pthread header to the slab header in the tools directory
to allow for the pthread_mutex in linux.c.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h        | 137 ++++++++++++++++++++++++++++++++++++--
 tools/testing/shared/linux.c      |  26 ++------
 tools/testing/shared/maple-shim.c |   1 +
 tools/testing/vma/vma_internal.h  |  94 +-------------------------
 4 files changed, 142 insertions(+), 116 deletions(-)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c87051e2b26f5a7fee0362697fae067076b8e84d..c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -4,11 +4,31 @@
 
 #include <linux/types.h>
 #include <linux/gfp.h>
+#include <pthread.h>
 
-#define SLAB_PANIC 2
 #define SLAB_RECLAIM_ACCOUNT    0x00020000UL            /* Objects are reclaimable */
 
 #define kzalloc_node(size, flags, node) kmalloc(size, flags)
+enum _slab_flag_bits {
+	_SLAB_KMALLOC,
+	_SLAB_HWCACHE_ALIGN,
+	_SLAB_PANIC,
+	_SLAB_TYPESAFE_BY_RCU,
+	_SLAB_ACCOUNT,
+	_SLAB_FLAGS_LAST_BIT
+};
+
+#define __SLAB_FLAG_BIT(nr)	((unsigned int __force)(1U << (nr)))
+#define __SLAB_FLAG_UNUSED	((unsigned int __force)(0U))
+
+#define SLAB_HWCACHE_ALIGN	__SLAB_FLAG_BIT(_SLAB_HWCACHE_ALIGN)
+#define SLAB_PANIC		__SLAB_FLAG_BIT(_SLAB_PANIC)
+#define SLAB_TYPESAFE_BY_RCU	__SLAB_FLAG_BIT(_SLAB_TYPESAFE_BY_RCU)
+#ifdef CONFIG_MEMCG
+# define SLAB_ACCOUNT		__SLAB_FLAG_BIT(_SLAB_ACCOUNT)
+#else
+# define SLAB_ACCOUNT		__SLAB_FLAG_UNUSED
+#endif
 
 void *kmalloc(size_t size, gfp_t gfp);
 void kfree(void *p);
@@ -23,6 +43,86 @@ enum slab_state {
 	FULL
 };
 
+struct kmem_cache {
+	pthread_mutex_t lock;
+	unsigned int size;
+	unsigned int align;
+	unsigned int sheaf_capacity;
+	int nr_objs;
+	void *objs;
+	void (*ctor)(void *);
+	bool non_kernel_enabled;
+	unsigned int non_kernel;
+	unsigned long nr_allocated;
+	unsigned long nr_tallocated;
+	bool exec_callback;
+	void (*callback)(void *);
+	void *private;
+};
+
+struct kmem_cache_args {
+	/**
+	 * @align: The required alignment for the objects.
+	 *
+	 * %0 means no specific alignment is requested.
+	 */
+	unsigned int align;
+	/**
+	 * @sheaf_capacity: The maximum size of the sheaf.
+	 */
+	unsigned int sheaf_capacity;
+	/**
+	 * @useroffset: Usercopy region offset.
+	 *
+	 * %0 is a valid offset, when @usersize is non-%0
+	 */
+	unsigned int useroffset;
+	/**
+	 * @usersize: Usercopy region size.
+	 *
+	 * %0 means no usercopy region is specified.
+	 */
+	unsigned int usersize;
+	/**
+	 * @freeptr_offset: Custom offset for the free pointer
+	 * in &SLAB_TYPESAFE_BY_RCU caches
+	 *
+	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
+	 * outside of the object. This might cause the object to grow in size.
+	 * Cache creators that have a reason to avoid this can specify a custom
+	 * free pointer offset in their struct where the free pointer will be
+	 * placed.
+	 *
+	 * Note that placing the free pointer inside the object requires the
+	 * caller to ensure that no fields are invalidated that are required to
+	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
+	 * details).
+	 *
+	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
+	 * is specified, %use_freeptr_offset must be set %true.
+	 *
+	 * Note that @ctor currently isn't supported with custom free pointers
+	 * as a @ctor requires an external free pointer.
+	 */
+	unsigned int freeptr_offset;
+	/**
+	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
+	 */
+	bool use_freeptr_offset;
+	/**
+	 * @ctor: A constructor for the objects.
+	 *
+	 * The constructor is invoked for each object in a newly allocated slab
+	 * page. It is the cache user's responsibility to free object in the
+	 * same state as after calling the constructor, or deal appropriately
+	 * with any differences between a freshly constructed and a reallocated
+	 * object.
+	 *
+	 * %NULL means no constructor.
+	 */
+	void (*ctor)(void *);
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -37,9 +137,38 @@ static inline void *kmem_cache_alloc(struct kmem_cache *cachep, int flags)
 }
 void kmem_cache_free(struct kmem_cache *cachep, void *objp);
 
-struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
-			unsigned int align, unsigned int flags,
-			void (*ctor)(void *));
+
+struct kmem_cache *
+__kmem_cache_create_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags);
+
+/* If NULL is passed for @args, use this variant with default arguments. */
+static inline struct kmem_cache *
+__kmem_cache_default_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags)
+{
+	struct kmem_cache_args kmem_default_args = {};
+
+	return __kmem_cache_create_args(name, size, &kmem_default_args, flags);
+}
+
+static inline struct kmem_cache *
+__kmem_cache_create(const char *name, unsigned int size, unsigned int align,
+		unsigned int flags, void (*ctor)(void *))
+{
+	struct kmem_cache_args kmem_args = {
+		.align	= align,
+		.ctor	= ctor,
+	};
+
+	return __kmem_cache_create_args(name, size, &kmem_args, flags);
+}
+
+#define kmem_cache_create(__name, __object_size, __args, ...)           \
+	_Generic((__args),                                              \
+		struct kmem_cache_args *: __kmem_cache_create_args,	\
+		void *: __kmem_cache_default_args,			\
+		default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__)
 
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 0f97fb0d19e19c327aa4843a35b45cc086f4f366..97b8412ccbb6d222604c7b397c53c65618d8d51b 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -16,21 +16,6 @@ int nr_allocated;
 int preempt_count;
 int test_verbose;
 
-struct kmem_cache {
-	pthread_mutex_t lock;
-	unsigned int size;
-	unsigned int align;
-	int nr_objs;
-	void *objs;
-	void (*ctor)(void *);
-	unsigned int non_kernel;
-	unsigned long nr_allocated;
-	unsigned long nr_tallocated;
-	bool exec_callback;
-	void (*callback)(void *);
-	void *private;
-};
-
 void kmem_cache_set_callback(struct kmem_cache *cachep, void (*callback)(void *))
 {
 	cachep->callback = callback;
@@ -234,23 +219,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 }
 
 struct kmem_cache *
-kmem_cache_create(const char *name, unsigned int size, unsigned int align,
-		unsigned int flags, void (*ctor)(void *))
+__kmem_cache_create_args(const char *name, unsigned int size,
+			  struct kmem_cache_args *args,
+			  unsigned int flags)
 {
 	struct kmem_cache *ret = malloc(sizeof(*ret));
 
 	pthread_mutex_init(&ret->lock, NULL);
 	ret->size = size;
-	ret->align = align;
+	ret->align = args->align;
+	ret->sheaf_capacity = args->sheaf_capacity;
 	ret->nr_objs = 0;
 	ret->nr_allocated = 0;
 	ret->nr_tallocated = 0;
 	ret->objs = NULL;
-	ret->ctor = ctor;
+	ret->ctor = args->ctor;
 	ret->non_kernel = 0;
 	ret->exec_callback = false;
 	ret->callback = NULL;
 	ret->private = NULL;
+
 	return ret;
 }
 
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 640df76f483e09f3b6f85612786060dd273e2362..9d7b743415660305416e972fa75b56824211b0eb 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -3,5 +3,6 @@
 /* Very simple shim around the maple tree. */
 
 #include "maple-shared.h"
+#include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 2f1c586400cfd65413d2ff82199590b43796b18c..972ab2686e0a3654cef611ce9f3409bc0c38dc80 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -26,6 +26,7 @@
 #include <linux/mm.h>
 #include <linux/rbtree.h>
 #include <linux/refcount.h>
+#include <linux/slab.h>
 
 extern unsigned long stack_guard_gap;
 #ifdef CONFIG_MMU
@@ -509,65 +510,6 @@ struct pagetable_move_control {
 		.len_in = len_,						\
 	}
 
-struct kmem_cache_args {
-	/**
-	 * @align: The required alignment for the objects.
-	 *
-	 * %0 means no specific alignment is requested.
-	 */
-	unsigned int align;
-	/**
-	 * @useroffset: Usercopy region offset.
-	 *
-	 * %0 is a valid offset, when @usersize is non-%0
-	 */
-	unsigned int useroffset;
-	/**
-	 * @usersize: Usercopy region size.
-	 *
-	 * %0 means no usercopy region is specified.
-	 */
-	unsigned int usersize;
-	/**
-	 * @freeptr_offset: Custom offset for the free pointer
-	 * in &SLAB_TYPESAFE_BY_RCU caches
-	 *
-	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
-	 * outside of the object. This might cause the object to grow in size.
-	 * Cache creators that have a reason to avoid this can specify a custom
-	 * free pointer offset in their struct where the free pointer will be
-	 * placed.
-	 *
-	 * Note that placing the free pointer inside the object requires the
-	 * caller to ensure that no fields are invalidated that are required to
-	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
-	 * details).
-	 *
-	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
-	 * is specified, %use_freeptr_offset must be set %true.
-	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
-	 */
-	unsigned int freeptr_offset;
-	/**
-	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
-	 */
-	bool use_freeptr_offset;
-	/**
-	 * @ctor: A constructor for the objects.
-	 *
-	 * The constructor is invoked for each object in a newly allocated slab
-	 * page. It is the cache user's responsibility to free object in the
-	 * same state as after calling the constructor, or deal appropriately
-	 * with any differences between a freshly constructed and a reallocated
-	 * object.
-	 *
-	 * %NULL means no constructor.
-	 */
-	void (*ctor)(void *);
-};
-
 static inline void vma_iter_invalidate(struct vma_iterator *vmi)
 {
 	mas_pause(&vmi->mas);
@@ -652,40 +594,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_lock_seq = UINT_MAX;
 }
 
-struct kmem_cache {
-	const char *name;
-	size_t object_size;
-	struct kmem_cache_args *args;
-};
-
-static inline struct kmem_cache *__kmem_cache_create(const char *name,
-						     size_t object_size,
-						     struct kmem_cache_args *args)
-{
-	struct kmem_cache *ret = malloc(sizeof(struct kmem_cache));
-
-	ret->name = name;
-	ret->object_size = object_size;
-	ret->args = args;
-
-	return ret;
-}
-
-#define kmem_cache_create(__name, __object_size, __args, ...)           \
-	__kmem_cache_create((__name), (__object_size), (__args))
-
-static inline void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
-{
-	(void)gfpflags;
-
-	return calloc(s->object_size, 1);
-}
-
-static inline void kmem_cache_free(struct kmem_cache *s, void *x)
-{
-	free(x);
-}
-
 /*
  * These are defined in vma.h, but sadly vm_stat_account() is referenced by
  * kernel/fork.c, so we have to these broadly available there, and temporarily

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:51 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:51 +0200
Subject: [PATCH v7 09/21] tools/testing/maple_tree: Fix
 check_bulk_rebalance() locks
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-9-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

The check_bulk_rebalance() test was not correctly locking the tree which
caused issues with the sheaves testing in later patches.  Adding the
missing locks fixed the issue.

Fixes: a6e0ceb7bf48 ("maple_tree: check for MA_STATE_BULK on setting wr_rebalance")
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 172700fb7784d29f9403003b4484a5ebd7aa316b..159d5307b30a4b37e6cf2941848b8718e1b891d9 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -36465,6 +36465,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
 
 	build_full_tree(mt, 0, 2);
 
+	mas_lock(&mas);
 	/* erase every entry in the tree */
 	do {
 		/* set up bulk store mode */
@@ -36474,6 +36475,7 @@ static inline void check_bulk_rebalance(struct maple_tree *mt)
 	} while (mas_prev(&mas, 0) != NULL);
 
 	mas_destroy(&mas);
+	mas_unlock(&mas);
 }
 
 void farmer_tests(void)

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:46 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:46 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>

Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
For caches with sheaves, on each cpu maintain a rcu_free sheaf in
addition to main and spare sheaves.

kfree_rcu() operations will try to put objects on this sheaf. Once full,
the sheaf is detached and submitted to call_rcu() with a handler that
will try to put it in the barn, or flush to slab pages using bulk free,
when the barn is full. Then a new empty sheaf must be obtained to put
more objects there.

It's possible that no free sheaves are available to use for a new
rcu_free sheaf, and the allocation in kfree_rcu() context can only use
GFP_NOWAIT and thus may fail. In that case, fall back to the existing
kfree_rcu() implementation.

Expected advantages:
- batching the kfree_rcu() operations, that could eventually replace the
  existing batching
- sheaves can be reused for allocations via barn instead of being
  flushed to slabs, which is more efficient
  - this includes cases where only some cpus are allowed to process rcu
    callbacks (Android)

Possible disadvantage:
- objects might be waiting for more than their grace period (it is
  determined by the last object freed into the sheaf), increasing memory
  usage - but the existing batching does that too.

Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
implementation favors smaller memory footprint over performance.

Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
count how many kfree_rcu() used the rcu_free sheaf successfully and how
many had to fall back to the existing implementation.

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab.h        |   2 +
 mm/slab_common.c |  24 +++++++
 mm/slub.c        | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 216 insertions(+), 2 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..f1866f2d9b211bb0d7f24644b80ef4b50a7c3d24 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -435,6 +435,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
 }
 
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
 			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
diff --git a/mm/slab_common.c b/mm/slab_common.c
index e2b197e47866c30acdbd1fee4159f262a751c5a7..2d806e02568532a1000fd3912db6978e945dcfa8 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
 		kvfree_rcu_list(head);
 }
 
+static bool kfree_rcu_sheaf(void *obj)
+{
+	struct kmem_cache *s;
+	struct folio *folio;
+	struct slab *slab;
+
+	if (is_vmalloc_addr(obj))
+		return false;
+
+	folio = virt_to_folio(obj);
+	if (unlikely(!folio_test_slab(folio)))
+		return false;
+
+	slab = folio_slab(folio);
+	s = slab->slab_cache;
+	if (s->cpu_sheaves)
+		return __kfree_rcu_sheaf(s, obj);
+
+	return false;
+}
+
 static bool
 need_offload_krc(struct kfree_rcu_cpu *krcp)
 {
@@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
+	if (kfree_rcu_sheaf(ptr))
+		return;
+
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
diff --git a/mm/slub.c b/mm/slub.c
index 42cb5848f1cecb17174967ff8b102b20a50110e3..6a64478befdebdb44cd7896d673bd20a7a6e2889 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -367,6 +367,8 @@ enum stat_item {
 	ALLOC_FASTPATH,		/* Allocation from cpu slab */
 	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
 	FREE_PCS,		/* Free to percpu sheaf */
+	FREE_RCU_SHEAF,		/* Free to rcu_free sheaf */
+	FREE_RCU_SHEAF_FAIL,	/* Failed to free to a rcu_free sheaf */
 	FREE_FASTPATH,		/* Free to cpu slab */
 	FREE_SLOWPATH,		/* Freeing not to cpu slab */
 	FREE_FROZEN,		/* Freeing to frozen slab */
@@ -461,6 +463,7 @@ struct slab_sheaf {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
 	};
+	struct kmem_cache *cache;
 	unsigned int size;
 	void *objects[];
 };
@@ -469,6 +472,7 @@ struct slub_percpu_sheaves {
 	local_trylock_t lock;
 	struct slab_sheaf *main; /* never NULL when unlocked */
 	struct slab_sheaf *spare; /* empty or full, may be NULL */
+	struct slab_sheaf *rcu_free; /* for batching kfree_rcu() */
 };
 
 /*
@@ -2531,6 +2535,8 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
 	if (unlikely(!sheaf))
 		return NULL;
 
+	sheaf->cache = s;
+
 	stat(s, SHEAF_ALLOC);
 
 	return sheaf;
@@ -2655,6 +2661,43 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	sheaf->size = 0;
 }
 
+static void __rcu_free_sheaf_prepare(struct kmem_cache *s,
+				     struct slab_sheaf *sheaf)
+{
+	bool init = slab_want_init_on_free(s);
+	void **p = &sheaf->objects[0];
+	unsigned int i = 0;
+
+	while (i < sheaf->size) {
+		struct slab *slab = virt_to_slab(p[i]);
+
+		memcg_slab_free_hook(s, slab, p + i, 1);
+		alloc_tagging_slab_free_hook(s, slab, p + i, 1);
+
+		if (unlikely(!slab_free_hook(s, p[i], init, true))) {
+			p[i] = p[--sheaf->size];
+			continue;
+		}
+
+		i++;
+	}
+}
+
+static void rcu_free_sheaf_nobarn(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+	s = sheaf->cache;
+
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	sheaf_flush_unused(s, sheaf);
+
+	free_empty_sheaf(s, sheaf);
+}
+
 /*
  * Caller needs to make sure migration is disabled in order to fully flush
  * single cpu's sheaves
@@ -2667,7 +2710,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 static void pcs_flush_all(struct kmem_cache *s)
 {
 	struct slub_percpu_sheaves *pcs;
-	struct slab_sheaf *spare;
+	struct slab_sheaf *spare, *rcu_free;
 
 	local_lock(&s->cpu_sheaves->lock);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
@@ -2675,6 +2718,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 	spare = pcs->spare;
 	pcs->spare = NULL;
 
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
 	local_unlock(&s->cpu_sheaves->lock);
 
 	if (spare) {
@@ -2682,6 +2728,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 		free_empty_sheaf(s, spare);
 	}
 
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+
 	sheaf_flush_main(s);
 }
 
@@ -2698,6 +2747,11 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
 		free_empty_sheaf(s, pcs->spare);
 		pcs->spare = NULL;
 	}
+
+	if (pcs->rcu_free) {
+		call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+		pcs->rcu_free = NULL;
+	}
 }
 
 static void pcs_destroy(struct kmem_cache *s)
@@ -2723,6 +2777,7 @@ static void pcs_destroy(struct kmem_cache *s)
 		 */
 
 		WARN_ON(pcs->spare);
+		WARN_ON(pcs->rcu_free);
 
 		if (!WARN_ON(pcs->main->size)) {
 			free_empty_sheaf(s, pcs->main);
@@ -3780,7 +3835,7 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s)
 
 	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
-	return (pcs->spare || pcs->main->size);
+	return (pcs->spare || pcs->rcu_free || pcs->main->size);
 }
 
 static void pcs_flush_all(struct kmem_cache *s);
@@ -5415,6 +5470,130 @@ bool free_to_pcs(struct kmem_cache *s, void *object)
 	return true;
 }
 
+static void rcu_free_sheaf(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct node_barn *barn;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+
+	s = sheaf->cache;
+
+	/*
+	 * This may remove some objects due to slab_free_hook() returning false,
+	 * so that the sheaf might no longer be completely full. But it's easier
+	 * to handle it as full (unless it became completely empty), as the code
+	 * handles it fine. The only downside is that sheaf will serve fewer
+	 * allocations when reused. It only happens due to debugging, which is a
+	 * performance hit anyway.
+	 */
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	barn = get_node(s, numa_mem_id())->barn;
+
+	/* due to slab_free_hook() */
+	if (unlikely(sheaf->size == 0))
+		goto empty;
+
+	/*
+	 * Checking nr_full/nr_empty outside lock avoids contention in case the
+	 * barn is at the respective limit. Due to the race we might go over the
+	 * limit but that should be rare and harmless.
+	 */
+
+	if (data_race(barn->nr_full) < MAX_FULL_SHEAVES) {
+		stat(s, BARN_PUT);
+		barn_put_full_sheaf(barn, sheaf);
+		return;
+	}
+
+	stat(s, BARN_PUT_FAIL);
+	sheaf_flush_unused(s, sheaf);
+
+empty:
+	if (data_race(barn->nr_empty) < MAX_EMPTY_SHEAVES) {
+		barn_put_empty_sheaf(barn, sheaf);
+		return;
+	}
+
+	free_empty_sheaf(s, sheaf);
+}
+
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_sheaf;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto fail;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(!pcs->rcu_free)) {
+
+		struct slab_sheaf *empty;
+		struct node_barn *barn;
+
+		if (pcs->spare && pcs->spare->size == 0) {
+			pcs->rcu_free = pcs->spare;
+			pcs->spare = NULL;
+			goto do_free;
+		}
+
+		barn = get_barn(s);
+
+		empty = barn_get_empty_sheaf(barn);
+
+		if (empty) {
+			pcs->rcu_free = empty;
+			goto do_free;
+		}
+
+		local_unlock(&s->cpu_sheaves->lock);
+
+		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+
+		if (!empty)
+			goto fail;
+
+		if (!local_trylock(&s->cpu_sheaves->lock)) {
+			barn_put_empty_sheaf(barn, empty);
+			goto fail;
+		}
+
+		pcs = this_cpu_ptr(s->cpu_sheaves);
+
+		if (unlikely(pcs->rcu_free))
+			barn_put_empty_sheaf(barn, empty);
+		else
+			pcs->rcu_free = empty;
+	}
+
+do_free:
+
+	rcu_sheaf = pcs->rcu_free;
+
+	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
+
+	if (likely(rcu_sheaf->size < s->sheaf_capacity))
+		rcu_sheaf = NULL;
+	else
+		pcs->rcu_free = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (rcu_sheaf)
+		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+
+	stat(s, FREE_RCU_SHEAF);
+	return true;
+
+fail:
+	stat(s, FREE_RCU_SHEAF_FAIL);
+	return false;
+}
+
 /*
  * Bulk free objects to the percpu sheaves.
  * Unlike free_to_pcs() this includes the calls to all necessary hooks
@@ -6911,6 +7090,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
 	struct kmem_cache_node *n;
 
 	flush_all_cpus_locked(s);
+
+	/* we might have rcu sheaves in flight */
+	if (s->cpu_sheaves)
+		rcu_barrier();
+
 	/* Attempt to free all objects */
 	for_each_kmem_cache_node(s, node, n) {
 		if (n->barn)
@@ -8286,6 +8470,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
 STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
 STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
 STAT_ATTR(FREE_PCS, free_cpu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail);
 STAT_ATTR(FREE_FASTPATH, free_fastpath);
 STAT_ATTR(FREE_SLOWPATH, free_slowpath);
 STAT_ATTR(FREE_FROZEN, free_frozen);
@@ -8384,6 +8570,8 @@ static struct attribute *slab_attrs[] = {
 	&alloc_fastpath_attr.attr,
 	&alloc_slowpath_attr.attr,
 	&free_cpu_sheaf_attr.attr,
+	&free_rcu_sheaf_attr.attr,
+	&free_rcu_sheaf_fail_attr.attr,
 	&free_fastpath_attr.attr,
 	&free_slowpath_attr.attr,
 	&free_frozen_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:55 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:55 +0200
Subject: [PATCH v7 13/21] maple_tree: use percpu sheaves for
 maple_node_cache
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-13-71c114cdefef@suse.cz>

Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
improve its performance. Note this will not immediately take advantage
of sheaf batching of kfree_rcu() operations due to the maple tree using
call_rcu with custom callbacks. The followup changes to maple tree will
change that and also make use of the prefilled sheaves functionality.

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
---
 lib/maple_tree.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a962ca374467d0533185f2db3d35ff..a0db6bdc63793b8bbd544e246391d99e880dede3 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -6302,9 +6302,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 
 void __init maple_tree_init(void)
 {
+	struct kmem_cache_args args = {
+		.align  = sizeof(struct maple_node),
+		.sheaf_capacity = 32,
+	};
+
 	maple_node_cache = kmem_cache_create("maple_node",
-			sizeof(struct maple_node), sizeof(struct maple_node),
-			SLAB_PANIC, NULL);
+			sizeof(struct maple_node), &args,
+			SLAB_PANIC);
 }
 
 /**

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:54 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:54 +0200
Subject: [PATCH v7 12/21] mm, vma: use percpu sheaves for vm_area_struct
 cache
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-12-71c114cdefef@suse.cz>

Create the vm_area_struct cache with percpu sheaves of size 32 to
improve its performance.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/vma_init.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/vma_init.c b/mm/vma_init.c
index 8e53c7943561e7324e7992946b4065dec1149b82..52c6b55fac4519e0da39ca75ad018e14449d1d95 100644
--- a/mm/vma_init.c
+++ b/mm/vma_init.c
@@ -16,6 +16,7 @@ void __init vma_state_init(void)
 	struct kmem_cache_args args = {
 		.use_freeptr_offset = true,
 		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
+		.sheaf_capacity = 32,
 	};
 
 	vm_area_cachep = kmem_cache_create("vm_area_struct",

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 06:00:02 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 15:00:02 +0200
Subject: [PATCH v7 20/21] maple_tree: Add single node allocation support to
 maple state
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-20-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The fast path through a write will require replacing a single node in
the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
special case the node store operation by just allocating one node in the
maple state.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |  4 +++-
 lib/maple_tree.c                 | 47 +++++++++++++++++++++++++++++++++++-----
 tools/testing/radix-tree/maple.c |  9 ++++++--
 3 files changed, 51 insertions(+), 9 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index 166fd67e00d882b1e6de1f80c1b590bba7497cd3..562a1e9e5132b5b1fa8f8402a7cadd8abb65e323 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -443,6 +443,7 @@ struct ma_state {
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
 	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	struct maple_node *alloc;	/* allocated nodes */
 	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
@@ -491,8 +492,9 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.node_request= 0,					\
 		.sheaf = NULL,						\
+		.alloc = NULL,						\
+		.node_request= 0,					\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 3f14bfa7fe1c20aac3e127f0defd268c3dbca6aa..f2b81a8d840b219819dd10104cd25120ba913ec7 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1095,16 +1095,23 @@ static int mas_ascend(struct ma_state *mas)
  *
  * Return: A pointer to a maple node.
  */
-static inline struct maple_node *mas_pop_node(struct ma_state *mas)
+static __always_inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
 	struct maple_node *ret;
 
+	if (mas->alloc) {
+		ret = mas->alloc;
+		mas->alloc = NULL;
+		goto out;
+	}
+
 	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
 	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
-	memset(ret, 0, sizeof(*ret));
 
+out:
+	memset(ret, 0, sizeof(*ret));
 	return ret;
 }
 
@@ -1115,9 +1122,34 @@ static inline struct maple_node *mas_pop_node(struct ma_state *mas)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	if (unlikely(mas->sheaf)) {
-		unsigned long refill = mas->node_request;
+	if (!mas->node_request)
+		return;
+
+	if (mas->node_request == 1) {
+		if (mas->sheaf)
+			goto use_sheaf;
+
+		if (mas->alloc)
+			return;
 
+		mas->alloc = mt_alloc_one(gfp);
+		if (!mas->alloc)
+			goto error;
+
+		mas->node_request = 0;
+		return;
+	}
+
+use_sheaf:
+	if (unlikely(mas->alloc)) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+
+	if (mas->sheaf) {
+		unsigned long refill;
+
+		refill = mas->node_request;
 		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
 			mas->node_request = 0;
 			return;
@@ -5380,8 +5412,11 @@ void mas_destroy(struct ma_state *mas)
 	mas->node_request = 0;
 	if (mas->sheaf)
 		mt_return_sheaf(mas->sheaf);
-
 	mas->sheaf = NULL;
+
+	if (mas->alloc)
+		kfree(mas->alloc);
+	mas->alloc = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6079,7 +6114,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas->sheaf)
+	if (!mas->sheaf && !mas->alloc)
 		return false;
 
 	mas->status = ma_start;
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index da3e03d73b52162dab6fa5c368ad7b71b9e58521..89da991e12cd97e44971757ddc105ef46c68ea4c 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -35095,10 +35095,15 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 
 static int mas_allocated(struct ma_state *mas)
 {
+	int total = 0;
+
+	if (mas->alloc)
+		total++;
+
 	if (mas->sheaf)
-		return kmem_cache_sheaf_size(mas->sheaf);
+		total += kmem_cache_sheaf_size(mas->sheaf);
 
-	return 0;
+	return total;
 }
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:56 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:56 +0200
Subject: [PATCH v7 14/21] tools/testing: include maple-shim.c in maple.c
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-14-71c114cdefef@suse.cz>

There's some duplicated code and we are about to add more functionality
in maple-shared.h that we will need in the userspace maple test to be
available, so include it via maple-shim.c

Co-developed-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 159d5307b30a4b37e6cf2941848b8718e1b891d9..7fe91f24849b35723ec6aadbe45ec7d2abedcc11 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -8,14 +8,6 @@
  * difficult to handle in kernel tests.
  */
 
-#define CONFIG_DEBUG_MAPLE_TREE
-#define CONFIG_MAPLE_SEARCH
-#define MAPLE_32BIT (MAPLE_NODE_SLOTS > 31)
-#include "test.h"
-#include <stdlib.h>
-#include <time.h>
-#include <linux/init.h>
-
 #define module_init(x)
 #define module_exit(x)
 #define MODULE_AUTHOR(x)
@@ -23,7 +15,9 @@
 #define MODULE_LICENSE(x)
 #define dump_stack()	assert(0)
 
-#include "../../../lib/maple_tree.c"
+#include "test.h"
+
+#include "../shared/maple-shim.c"
 #include "../../../lib/test_maple_tree.c"
 
 #define RCU_RANGE_COUNT 1000

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 06:00:00 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 15:00:00 +0200
Subject: [PATCH v7 18/21] tools/testing: Add support for prefilled slab
 sheafs
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-18-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the prefilled sheaf structs to the slab header and the associated
functions to the testing/shared/linux.c file.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h   | 28 ++++++++++++++
 tools/testing/shared/linux.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08..94937a699402bd1f31887dfb52b6fd0a3c986f43 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -123,6 +123,18 @@ struct kmem_cache_args {
 	void (*ctor)(void *);
 };
 
+struct slab_sheaf {
+	union {
+		struct list_head barn_list;
+		/* only used for prefilled sheafs */
+		unsigned int capacity;
+	};
+	struct kmem_cache *cache;
+	unsigned int size;
+	int node; /* only used for rcu_sheaf */
+	void *objects[];
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -173,5 +185,21 @@ __kmem_cache_create(const char *name, unsigned int size, unsigned int align,
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 			  void **list);
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf **sheafp, unsigned int size);
+
+static inline unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
+{
+	return sheaf->size;
+}
 
 #endif		/* _TOOLS_SLAB_H */
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 97b8412ccbb6d222604c7b397c53c65618d8d51b..4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -137,6 +137,12 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
 	if (kmalloc_verbose)
 		pr_debug("Bulk free %p[0-%zu]\n", list, size - 1);
 
+	if (cachep->exec_callback) {
+		if (cachep->callback)
+			cachep->callback(cachep->private);
+		cachep->exec_callback = false;
+	}
+
 	pthread_mutex_lock(&cachep->lock);
 	for (int i = 0; i < size; i++)
 		kmem_cache_free_locked(cachep, list[i]);
@@ -242,6 +248,89 @@ __kmem_cache_create_args(const char *name, unsigned int size,
 	return ret;
 }
 
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
+{
+	struct slab_sheaf *sheaf;
+	unsigned int capacity;
+
+	if (s->exec_callback) {
+		if (s->callback)
+			s->callback(s->private);
+		s->exec_callback = false;
+	}
+
+	capacity = max(size, s->sheaf_capacity);
+
+	sheaf = calloc(1, sizeof(*sheaf) + sizeof(void *) * capacity);
+	if (!sheaf)
+		return NULL;
+
+	sheaf->cache = s;
+	sheaf->capacity = capacity;
+	sheaf->size = kmem_cache_alloc_bulk(s, gfp, size, sheaf->objects);
+	if (!sheaf->size) {
+		free(sheaf);
+		return NULL;
+	}
+
+	return sheaf;
+}
+
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf **sheafp, unsigned int size)
+{
+	struct slab_sheaf *sheaf = *sheafp;
+	int refill;
+
+	if (sheaf->size >= size)
+		return 0;
+
+	if (size > sheaf->capacity) {
+		sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
+		if (!sheaf)
+			return -ENOMEM;
+
+		kmem_cache_return_sheaf(s, gfp, *sheafp);
+		*sheafp = sheaf;
+		return 0;
+	}
+
+	refill = kmem_cache_alloc_bulk(s, gfp, size - sheaf->size,
+				       &sheaf->objects[sheaf->size]);
+	if (!refill)
+		return -ENOMEM;
+
+	sheaf->size += refill;
+	return 0;
+}
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf *sheaf)
+{
+	if (sheaf->size)
+		kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
+
+	free(sheaf);
+}
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf)
+{
+	void *obj;
+
+	if (sheaf->size == 0) {
+		printf("Nothing left in sheaf!\n");
+		return NULL;
+	}
+
+	obj = sheaf->objects[--sheaf->size];
+	sheaf->objects[sheaf->size] = NULL;
+
+	return obj;
+}
+
 /*
  * Test the test infrastructure for kem_cache_alloc/free and bulk counterparts.
  */

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:57 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:57 +0200
Subject: [PATCH v7 15/21] testing/radix-tree/maple: Hack around kfree_rcu
 not existing
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-15-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

liburcu doesn't have kfree_rcu (or anything similar). Despite that, we
can hack around it in a trivial fashion, by adding a wrapper.

The wrapper only works for maple_nodes because we cannot get the
kmem_cache pointer any other way in the test code.

Link: https://lore.kernel.org/all/20250812162124.59417-1-pfalcato at suse.de/
Suggested-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/shared/maple-shared.h | 11 +++++++++++
 tools/testing/shared/maple-shim.c   |  6 ++++++
 2 files changed, 17 insertions(+)

diff --git a/tools/testing/shared/maple-shared.h b/tools/testing/shared/maple-shared.h
index dc4d30f3860b9bd23b4177c7d7926ac686887815..2a1e9a8594a2834326cd9374738b2a2c7c3f9f7c 100644
--- a/tools/testing/shared/maple-shared.h
+++ b/tools/testing/shared/maple-shared.h
@@ -10,4 +10,15 @@
 #include <time.h>
 #include "linux/init.h"
 
+void maple_rcu_cb(struct rcu_head *head);
+#define rcu_cb		maple_rcu_cb
+
+#define kfree_rcu(_struct, _memb)		\
+do {                                            \
+    typeof(_struct) _p_struct = (_struct);      \
+                                                \
+    call_rcu(&((_p_struct)->_memb), rcu_cb);    \
+} while(0);
+
+
 #endif /* __MAPLE_SHARED_H__ */
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 9d7b743415660305416e972fa75b56824211b0eb..16252ee616c0489c80490ff25b8d255427bf9fdc 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -6,3 +6,9 @@
 #include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
+
+void maple_rcu_cb(struct rcu_head *head) {
+	struct maple_node *node = container_of(head, struct maple_node, rcu);
+
+	kmem_cache_free(maple_node_cache, node);
+}

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:58 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:58 +0200
Subject: [PATCH v7 16/21] maple_tree: Use kfree_rcu in ma_free_rcu
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-16-71c114cdefef@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
possible to call it on non-kmalloc objects, but this restriction was
lifted ever since SLOB was dropped from the kernel, and since commit
6c6c47b063b5 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").

Thus, replace call_rcu + mt_free_rcu with kfree_rcu.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index a0db6bdc63793b8bbd544e246391d99e880dede3..d77e82362f03905040ac61630f92fe9af1e59f98 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -191,13 +191,6 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
-static void mt_free_rcu(struct rcu_head *head)
-{
-	struct maple_node *node = container_of(head, struct maple_node, rcu);
-
-	kmem_cache_free(maple_node_cache, node);
-}
-
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -208,7 +201,7 @@ static void mt_free_rcu(struct rcu_head *head)
 static void ma_free_rcu(struct maple_node *node)
 {
 	WARN_ON(node->parent != ma_parent_ptr(node));
-	call_rcu(&node->rcu, mt_free_rcu);
+	kfree_rcu(node, rcu);
 }
 
 static void mt_set_height(struct maple_tree *mt, unsigned char height)
@@ -5281,7 +5274,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_rcu(&node->rcu);
+	mt_free_one(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5365,7 +5358,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_rcu(&node->rcu);
+		mt_free_one(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 06:00:03 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 15:00:03 +0200
Subject: [PATCH v7 21/21] maple_tree: Convert forking to use the sheaf
 interface
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-21-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

Use the generic interface which should result in less bulk allocations
during a forking.

A part of this is to abstract the freeing of the sheaf or maple state
allocations into its own function so mas_destroy() and the tree
duplication code can use the same functionality to return any unused
resources.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 42 +++++++++++++++++++++++-------------------
 1 file changed, 23 insertions(+), 19 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index f2b81a8d840b219819dd10104cd25120ba913ec7..c185dbc90007a0046a37f194e73e261caa1848c0 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1172,6 +1172,19 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 	mas_set_err(mas, -ENOMEM);
 }
 
+static inline void mas_empty_nodes(struct ma_state *mas)
+{
+	mas->node_request = 0;
+	if (mas->sheaf) {
+		mt_return_sheaf(mas->sheaf);
+		mas->sheaf = NULL;
+	}
+
+	if (mas->alloc) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+}
 
 /*
  * mas_free() - Free an encoded maple node
@@ -5408,15 +5421,7 @@ void mas_destroy(struct ma_state *mas)
 		mas->mas_flags &= ~MA_STATE_REBALANCE;
 	}
 	mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
-
-	mas->node_request = 0;
-	if (mas->sheaf)
-		mt_return_sheaf(mas->sheaf);
-	mas->sheaf = NULL;
-
-	if (mas->alloc)
-		kfree(mas->alloc);
-	mas->alloc = NULL;
+	mas_empty_nodes(mas);
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6504,7 +6509,7 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	struct maple_node *node = mte_to_node(mas->node);
 	struct maple_node *new_node = mte_to_node(new_mas->node);
 	enum maple_type type;
-	unsigned char request, count, i;
+	unsigned char count, i;
 	void __rcu **slots;
 	void __rcu **new_slots;
 	unsigned long val;
@@ -6512,20 +6517,17 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	/* Allocate memory for child nodes. */
 	type = mte_node_type(mas->node);
 	new_slots = ma_slots(new_node, type);
-	request = mas_data_end(mas) + 1;
-	count = mt_alloc_bulk(gfp, request, (void **)new_slots);
-	if (unlikely(count < request)) {
-		memset(new_slots, 0, request * sizeof(void *));
-		mas_set_err(mas, -ENOMEM);
+	count = mas->node_request = mas_data_end(mas) + 1;
+	mas_alloc_nodes(mas, gfp);
+	if (unlikely(mas_is_err(mas)))
 		return;
-	}
 
-	/* Restore node type information in slots. */
 	slots = ma_slots(node, type);
 	for (i = 0; i < count; i++) {
 		val = (unsigned long)mt_slot_locked(mas->tree, slots, i);
 		val &= MAPLE_NODE_MASK;
-		((unsigned long *)new_slots)[i] |= val;
+		new_slots[i] = ma_mnode_ptr((unsigned long)mas_pop_node(mas) |
+					    val);
 	}
 }
 
@@ -6579,7 +6581,7 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 			/* Only allocate child nodes for non-leaf nodes. */
 			mas_dup_alloc(mas, new_mas, gfp);
 			if (unlikely(mas_is_err(mas)))
-				return;
+				goto empty_mas;
 		} else {
 			/*
 			 * This is the last leaf node and duplication is
@@ -6612,6 +6614,8 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 	/* Make them the same height */
 	new_mas->tree->ma_flags = mas->tree->ma_flags;
 	rcu_assign_pointer(new_mas->tree->ma_root, root);
+empty_mas:
+	mas_empty_nodes(mas);
 }
 
 /**

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 05:59:59 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 14:59:59 +0200
Subject: [PATCH v7 17/21] maple_tree: Replace mt_free_one() with kfree()
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-17-71c114cdefef@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree() is a little shorter and works with kmem_cache_alloc'd pointers
too. Also lets us remove one more helper.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index d77e82362f03905040ac61630f92fe9af1e59f98..b361b484cfcaacd99472dd4c2b8de9260b307425 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -181,11 +181,6 @@ static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
 	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
 }
 
-static inline void mt_free_one(struct maple_node *node)
-{
-	kmem_cache_free(maple_node_cache, node);
-}
-
 static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 {
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
@@ -5274,7 +5269,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_one(node);
+	kfree(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5358,7 +5353,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_one(node);
+		kfree(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }
@@ -5585,7 +5580,7 @@ void mas_destroy(struct ma_state *mas)
 			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
 			total -= count;
 		}
-		mt_free_one(ma_mnode_ptr(node));
+		kfree(ma_mnode_ptr(node));
 		total--;
 	}
 
@@ -6635,7 +6630,7 @@ static void mas_dup_free(struct ma_state *mas)
 	}
 
 	node = mte_to_node(mas->node);
-	mt_free_one(node);
+	kfree(node);
 }
 
 /*

-- 
2.51.0



From vbabka at suse.cz  Wed Sep  3 06:00:01 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 03 Sep 2025 15:00:01 +0200
Subject: [PATCH v7 19/21] maple_tree: Prefilled sheaf conversion and
 testing
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <20250903-slub-percpu-caches-v7-19-71c114cdefef@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Use prefilled sheaves instead of bulk allocations. This should speed up
the allocations and the return path of unused allocations.

Remove the push and pop of nodes from the maple state as this is now
handled by the slab layer with sheaves.

Testing has been removed as necessary since the features of the tree
have been reduced.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |   6 +-
 lib/maple_tree.c                 | 329 ++++++---------------------
 lib/test_maple_tree.c            |   8 +
 tools/testing/radix-tree/maple.c | 464 ++-------------------------------------
 tools/testing/shared/linux.c     |   5 +-
 5 files changed, 99 insertions(+), 713 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -442,7 +442,8 @@ struct ma_state {
 	struct maple_enode *node;	/* The node containing this entry */
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
-	struct maple_alloc *alloc;	/* Allocated nodes for this operation */
+	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
 	unsigned char offset;
@@ -490,7 +491,8 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.alloc = NULL,						\
+		.node_request= 0,					\
+		.sheaf = NULL,						\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b361b484cfcaacd99472dd4c2b8de9260b307425..3f14bfa7fe1c20aac3e127f0defd268c3dbca6aa 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -186,6 +186,22 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
+static void mt_return_sheaf(struct slab_sheaf *sheaf)
+{
+	kmem_cache_return_sheaf(maple_node_cache, GFP_KERNEL, sheaf);
+}
+
+static struct slab_sheaf *mt_get_sheaf(gfp_t gfp, int count)
+{
+	return kmem_cache_prefill_sheaf(maple_node_cache, gfp, count);
+}
+
+static int mt_refill_sheaf(gfp_t gfp, struct slab_sheaf **sheaf,
+		unsigned int size)
+{
+	return kmem_cache_refill_sheaf(maple_node_cache, gfp, sheaf, size);
+}
+
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -578,67 +594,6 @@ static __always_inline bool mte_dead_node(const struct maple_enode *enode)
 	return ma_dead_node(node);
 }
 
-/*
- * mas_allocated() - Get the number of nodes allocated in a maple state.
- * @mas: The maple state
- *
- * The ma_state alloc member is overloaded to hold a pointer to the first
- * allocated node or to the number of requested nodes to allocate.  If bit 0 is
- * set, then the alloc contains the number of requested nodes.  If there is an
- * allocated node, then the total allocated nodes is in that node.
- *
- * Return: The total number of nodes allocated
- */
-static inline unsigned long mas_allocated(const struct ma_state *mas)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1))
-		return 0;
-
-	return mas->alloc->total;
-}
-
-/*
- * mas_set_alloc_req() - Set the requested number of allocations.
- * @mas: the maple state
- * @count: the number of allocations.
- *
- * The requested number of allocations is either in the first allocated node,
- * located in @mas->alloc->request_count, or directly in @mas->alloc if there is
- * no allocated node.  Set the request either in the node or do the necessary
- * encoding to store in @mas->alloc directly.
- */
-static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
-		if (!count)
-			mas->alloc = NULL;
-		else
-			mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
-		return;
-	}
-
-	mas->alloc->request_count = count;
-}
-
-/*
- * mas_alloc_req() - get the requested number of allocations.
- * @mas: The maple state
- *
- * The alloc count is either stored directly in @mas, or in
- * @mas->alloc->request_count if there is at least one node allocated.  Decode
- * the request count if it's stored directly in @mas->alloc.
- *
- * Return: The allocation request count.
- */
-static inline unsigned int mas_alloc_req(const struct ma_state *mas)
-{
-	if ((unsigned long)mas->alloc & 0x1)
-		return (unsigned long)(mas->alloc) >> 1;
-	else if (mas->alloc)
-		return mas->alloc->request_count;
-	return 0;
-}
-
 /*
  * ma_pivots() - Get a pointer to the maple node pivots.
  * @node: the maple node
@@ -1142,77 +1097,15 @@ static int mas_ascend(struct ma_state *mas)
  */
 static inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
-	struct maple_alloc *ret, *node = mas->alloc;
-	unsigned long total = mas_allocated(mas);
-	unsigned int req = mas_alloc_req(mas);
+	struct maple_node *ret;
 
-	/* nothing or a request pending. */
-	if (WARN_ON(!total))
+	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
-	if (total == 1) {
-		/* single allocation in this ma_state */
-		mas->alloc = NULL;
-		ret = node;
-		goto single_node;
-	}
-
-	if (node->node_count == 1) {
-		/* Single allocation in this node. */
-		mas->alloc = node->slot[0];
-		mas->alloc->total = node->total - 1;
-		ret = node;
-		goto new_head;
-	}
-	node->total--;
-	ret = node->slot[--node->node_count];
-	node->slot[node->node_count] = NULL;
-
-single_node:
-new_head:
-	if (req) {
-		req++;
-		mas_set_alloc_req(mas, req);
-	}
-
+	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
 	memset(ret, 0, sizeof(*ret));
-	return (struct maple_node *)ret;
-}
-
-/*
- * mas_push_node() - Push a node back on the maple state allocation.
- * @mas: The maple state
- * @used: The used maple node
- *
- * Stores the maple node back into @mas->alloc for reuse.  Updates allocated and
- * requested node count as necessary.
- */
-static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
-{
-	struct maple_alloc *reuse = (struct maple_alloc *)used;
-	struct maple_alloc *head = mas->alloc;
-	unsigned long count;
-	unsigned int requested = mas_alloc_req(mas);
 
-	count = mas_allocated(mas);
-
-	reuse->request_count = 0;
-	reuse->node_count = 0;
-	if (count) {
-		if (head->node_count < MAPLE_ALLOC_SLOTS) {
-			head->slot[head->node_count++] = reuse;
-			head->total++;
-			goto done;
-		}
-		reuse->slot[0] = head;
-		reuse->node_count = 1;
-	}
-
-	reuse->total = count + 1;
-	mas->alloc = reuse;
-done:
-	if (requested > 1)
-		mas_set_alloc_req(mas, requested - 1);
+	return ret;
 }
 
 /*
@@ -1222,75 +1115,32 @@ static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	struct maple_alloc *node;
-	unsigned long allocated = mas_allocated(mas);
-	unsigned int requested = mas_alloc_req(mas);
-	unsigned int count;
-	void **slots = NULL;
-	unsigned int max_req = 0;
-
-	if (!requested)
-		return;
+	if (unlikely(mas->sheaf)) {
+		unsigned long refill = mas->node_request;
 
-	mas_set_alloc_req(mas, 0);
-	if (mas->mas_flags & MA_STATE_PREALLOC) {
-		if (allocated)
+		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
+			mas->node_request = 0;
 			return;
-		WARN_ON(!allocated);
-	}
-
-	if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) {
-		node = (struct maple_alloc *)mt_alloc_one(gfp);
-		if (!node)
-			goto nomem_one;
-
-		if (allocated) {
-			node->slot[0] = mas->alloc;
-			node->node_count = 1;
-		} else {
-			node->node_count = 0;
 		}
 
-		mas->alloc = node;
-		node->total = ++allocated;
-		node->request_count = 0;
-		requested--;
-	}
+		if (mt_refill_sheaf(gfp, &mas->sheaf, refill))
+			goto error;
 
-	node = mas->alloc;
-	while (requested) {
-		max_req = MAPLE_ALLOC_SLOTS - node->node_count;
-		slots = (void **)&node->slot[node->node_count];
-		max_req = min(requested, max_req);
-		count = mt_alloc_bulk(gfp, max_req, slots);
-		if (!count)
-			goto nomem_bulk;
-
-		if (node->node_count == 0) {
-			node->slot[0]->node_count = 0;
-			node->slot[0]->request_count = 0;
-		}
+		mas->node_request = 0;
+		return;
+	}
 
-		node->node_count += count;
-		allocated += count;
-		/* find a non-full node*/
-		do {
-			node = node->slot[0];
-		} while (unlikely(node->node_count == MAPLE_ALLOC_SLOTS));
-		requested -= count;
+	mas->sheaf = mt_get_sheaf(gfp, mas->node_request);
+	if (likely(mas->sheaf)) {
+		mas->node_request = 0;
+		return;
 	}
-	mas->alloc->total = allocated;
-	return;
 
-nomem_bulk:
-	/* Clean up potential freed allocations on bulk failure */
-	memset(slots, 0, max_req * sizeof(unsigned long));
-	mas->alloc->total = allocated;
-nomem_one:
-	mas_set_alloc_req(mas, requested);
+error:
 	mas_set_err(mas, -ENOMEM);
 }
 
+
 /*
  * mas_free() - Free an encoded maple node
  * @mas: The maple state
@@ -1301,42 +1151,7 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
  */
 static inline void mas_free(struct ma_state *mas, struct maple_enode *used)
 {
-	struct maple_node *tmp = mte_to_node(used);
-
-	if (mt_in_rcu(mas->tree))
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
-}
-
-/*
- * mas_node_count_gfp() - Check if enough nodes are allocated and request more
- * if there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- * @gfp: the gfp flags
- */
-static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
-{
-	unsigned long allocated = mas_allocated(mas);
-
-	if (allocated < count) {
-		mas_set_alloc_req(mas, count - allocated);
-		mas_alloc_nodes(mas, gfp);
-	}
-}
-
-/*
- * mas_node_count() - Check if enough nodes are allocated and request more if
- * there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- *
- * Note: Uses GFP_NOWAIT | __GFP_NOWARN for gfp flags.
- */
-static void mas_node_count(struct ma_state *mas, int count)
-{
-	return mas_node_count_gfp(mas, count, GFP_NOWAIT | __GFP_NOWARN);
+	ma_free_rcu(mte_to_node(used));
 }
 
 /*
@@ -2511,10 +2326,7 @@ static inline void mas_topiary_node(struct ma_state *mas,
 	enode = tmp_mas->node;
 	tmp = mte_to_node(enode);
 	mte_set_node_dead(enode);
-	if (in_rcu)
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
+	ma_free_rcu(tmp);
 }
 
 /*
@@ -4162,7 +3974,7 @@ static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas)
  *
  * Return: Number of nodes required for preallocation.
  */
-static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
+static inline void mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 {
 	struct ma_state *mas = wr_mas->mas;
 	unsigned char height = mas_mt_height(mas);
@@ -4208,7 +4020,7 @@ static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 		WARN_ON_ONCE(1);
 	}
 
-	return ret;
+	mas->node_request = ret;
 }
 
 /*
@@ -4269,15 +4081,15 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
  */
 static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry)
 {
-	int request;
+	struct ma_state *mas = wr_mas->mas;
 
 	mas_wr_prealloc_setup(wr_mas);
-	wr_mas->mas->store_type = mas_wr_store_type(wr_mas);
-	request = mas_prealloc_calc(wr_mas, entry);
-	if (!request)
+	mas->store_type = mas_wr_store_type(wr_mas);
+	mas_prealloc_calc(wr_mas, entry);
+	if (!mas->node_request)
 		return;
 
-	mas_node_count(wr_mas->mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT | __GFP_NOWARN);
 }
 
 /**
@@ -5390,7 +5202,6 @@ static inline void mte_destroy_walk(struct maple_enode *enode,
  */
 void *mas_store(struct ma_state *mas, void *entry)
 {
-	int request;
 	MA_WR_STATE(wr_mas, mas, entry);
 
 	trace_ma_write(__func__, mas, 0, entry);
@@ -5420,11 +5231,11 @@ void *mas_store(struct ma_state *mas, void *entry)
 		return wr_mas.content;
 	}
 
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto store;
 
-	mas_node_count(mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT | __GFP_NOWARN);
 	if (mas_is_err(mas))
 		return NULL;
 
@@ -5512,20 +5323,19 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc);
 int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 {
 	MA_WR_STATE(wr_mas, mas, entry);
-	int ret = 0;
-	int request;
 
 	mas_wr_prealloc_setup(&wr_mas);
 	mas->store_type = mas_wr_store_type(&wr_mas);
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto set_flag;
 
 	mas->mas_flags &= ~MA_STATE_PREALLOC;
-	mas_node_count_gfp(mas, request, gfp);
+	mas_alloc_nodes(mas, gfp);
 	if (mas_is_err(mas)) {
-		mas_set_alloc_req(mas, 0);
-		ret = xa_err(mas->node);
+		int ret = xa_err(mas->node);
+
+		mas->node_request = 0;
 		mas_destroy(mas);
 		mas_reset(mas);
 		return ret;
@@ -5533,7 +5343,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 
 set_flag:
 	mas->mas_flags |= MA_STATE_PREALLOC;
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(mas_preallocate);
 
@@ -5547,9 +5357,6 @@ EXPORT_SYMBOL_GPL(mas_preallocate);
  */
 void mas_destroy(struct ma_state *mas)
 {
-	struct maple_alloc *node;
-	unsigned long total;
-
 	/*
 	 * When using mas_for_each() to insert an expected number of elements,
 	 * it is possible that the number inserted is less than the expected
@@ -5570,21 +5377,11 @@ void mas_destroy(struct ma_state *mas)
 	}
 	mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
 
-	total = mas_allocated(mas);
-	while (total) {
-		node = mas->alloc;
-		mas->alloc = node->slot[0];
-		if (node->node_count > 1) {
-			size_t count = node->node_count - 1;
-
-			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
-			total -= count;
-		}
-		kfree(ma_mnode_ptr(node));
-		total--;
-	}
+	mas->node_request = 0;
+	if (mas->sheaf)
+		mt_return_sheaf(mas->sheaf);
 
-	mas->alloc = NULL;
+	mas->sheaf = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -5634,7 +5431,8 @@ int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries)
 	/* Internal nodes */
 	nr_nodes += DIV_ROUND_UP(nr_nodes, nonleaf_cap);
 	/* Add working room for split (2 nodes) + new parents */
-	mas_node_count_gfp(mas, nr_nodes + 3, GFP_KERNEL);
+	mas->node_request = nr_nodes + 3;
+	mas_alloc_nodes(mas, GFP_KERNEL);
 
 	/* Detect if allocations run out */
 	mas->mas_flags |= MA_STATE_PREALLOC;
@@ -6281,7 +6079,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas_allocated(mas))
+	if (!mas->sheaf)
 		return false;
 
 	mas->status = ma_start;
@@ -7676,8 +7474,9 @@ void mas_dump(const struct ma_state *mas)
 
 	pr_err("[%u/%u] index=%lx last=%lx\n", mas->offset, mas->end,
 	       mas->index, mas->last);
-	pr_err("     min=%lx max=%lx alloc=" PTR_FMT ", depth=%u, flags=%x\n",
-	       mas->min, mas->max, mas->alloc, mas->depth, mas->mas_flags);
+	pr_err("     min=%lx max=%lx sheaf=" PTR_FMT ", request %lu depth=%u, flags=%x\n",
+	       mas->min, mas->max, mas->sheaf, mas->node_request, mas->depth,
+	       mas->mas_flags);
 	if (mas->index > mas->last)
 		pr_err("Check index & last\n");
 }
diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c
index cb3936595b0d56a9682ff100eba54693a1427829..1848d127eb50650e7cc2b9dfbb15ed93aa889f01 100644
--- a/lib/test_maple_tree.c
+++ b/lib/test_maple_tree.c
@@ -2746,6 +2746,7 @@ static noinline void __init check_fuzzer(struct maple_tree *mt)
 	mtree_test_erase(mt, ULONG_MAX - 10);
 }
 
+#if 0
 /* duplicate the tree with a specific gap */
 static noinline void __init check_dup_gaps(struct maple_tree *mt,
 				    unsigned long nr_entries, bool zero_start,
@@ -2770,6 +2771,7 @@ static noinline void __init check_dup_gaps(struct maple_tree *mt,
 		mtree_store_range(mt, i*10, (i+1)*10 - gap,
 				  xa_mk_value(i), GFP_KERNEL);
 
+	mt_dump(mt, mt_dump_dec);
 	mt_init_flags(&newmt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN);
 	mt_set_non_kernel(99999);
 	down_write(&newmt_lock);
@@ -2779,9 +2781,12 @@ static noinline void __init check_dup_gaps(struct maple_tree *mt,
 
 	rcu_read_lock();
 	mas_for_each(&mas, tmp, ULONG_MAX) {
+		printk("%lu nodes %lu\n", mas.index,
+		       kmem_cache_sheaf_count(newmas.sheaf));
 		newmas.index = mas.index;
 		newmas.last = mas.last;
 		mas_store(&newmas, tmp);
+		mt_dump(&newmt, mt_dump_dec);
 	}
 	rcu_read_unlock();
 	mas_destroy(&newmas);
@@ -2878,6 +2883,7 @@ static noinline void __init check_dup(struct maple_tree *mt)
 		cond_resched();
 	}
 }
+#endif
 
 static noinline void __init check_bnode_min_spanning(struct maple_tree *mt)
 {
@@ -4077,9 +4083,11 @@ static int __init maple_tree_seed(void)
 	check_fuzzer(&tree);
 	mtree_destroy(&tree);
 
+#if 0
 	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 	check_dup(&tree);
 	mtree_destroy(&tree);
+#endif
 
 	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 	check_bnode_min_spanning(&tree);
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 7fe91f24849b35723ec6aadbe45ec7d2abedcc11..da3e03d73b52162dab6fa5c368ad7b71b9e58521 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -57,430 +57,6 @@ struct rcu_reader_struct {
 	struct rcu_test_struct2 *test;
 };
 
-static int get_alloc_node_count(struct ma_state *mas)
-{
-	int count = 1;
-	struct maple_alloc *node = mas->alloc;
-
-	if (!node || ((unsigned long)node & 0x1))
-		return 0;
-	while (node->node_count) {
-		count += node->node_count;
-		node = node->slot[0];
-	}
-	return count;
-}
-
-static void check_mas_alloc_node_count(struct ma_state *mas)
-{
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 1, GFP_KERNEL);
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 3, GFP_KERNEL);
-	MT_BUG_ON(mas->tree, get_alloc_node_count(mas) != mas->alloc->total);
-	mas_destroy(mas);
-}
-
-/*
- * check_new_node() - Check the creation of new nodes and error path
- * verification.
- */
-static noinline void __init check_new_node(struct maple_tree *mt)
-{
-
-	struct maple_node *mn, *mn2, *mn3;
-	struct maple_alloc *smn;
-	struct maple_node *nodes[100];
-	int i, j, total;
-
-	MA_STATE(mas, mt, 0, 0);
-
-	check_mas_alloc_node_count(&mas);
-
-	/* Try allocating 3 nodes */
-	mtree_lock(mt);
-	mt_set_non_kernel(0);
-	/* request 3 nodes to be allocated. */
-	mas_node_count(&mas, 3);
-	/* Allocation request of 3. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 3);
-	/* Allocate failed. */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	mas_push_node(&mas, mn);
-	mas_reset(&mas);
-	mas_destroy(&mas);
-	mtree_unlock(mt);
-
-
-	/* Try allocating 1 node, then 2 more */
-	mtree_lock(mt);
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	/* Check Allocation request of 1. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	/* Eat the requested node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	mas.status = ma_start;
-	mas_destroy(&mas);
-	/* Allocate 3 nodes, will fail. */
-	mas_node_count(&mas, 3);
-	/* Drop the lock and allocate 3 nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	/* Ensure 3 are allocated. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Allocation request of 0. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 0);
-
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	/* Ensure we counted 3. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Free. */
-	mas_reset(&mas);
-	mas_destroy(&mas);
-
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	/* Check the node is only one node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	MT_BUG_ON(mt, mas.alloc->node_count);
-
-	mas_set_alloc_req(&mas, 2); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 2);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	for (i = 2; i >= 0; i--) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		MT_BUG_ON(mt, !mn);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 64;
-	mas_set_alloc_req(&mas, total); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != total);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	for (i = total; i > 0; i--) {
-		unsigned int e = 0; /* expected node_count */
-
-		if (!MAPLE_32BIT) {
-			if (i >= 35)
-				e = i - 34;
-			else if (i >= 5)
-				e = i - 4;
-			else if (i >= 2)
-				e = i - 1;
-		} else {
-			if (i >= 4)
-				e = i - 3;
-			else if (i >= 1)
-				e = i - 1;
-			else
-				e = 0;
-		}
-
-		MT_BUG_ON(mt, mas.alloc->node_count != e);
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		MT_BUG_ON(mt, mas_allocated(&mas) != i - 1);
-		MT_BUG_ON(mt, !mn);
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 100;
-	for (i = 1; i < total; i++) {
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = i; j > 0; j--) {
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			MT_BUG_ON(mt, !mn);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			nodes[j] = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-
-		while (j) {
-			j--;
-			mas_push_node(&mas, nodes[j]);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-		mas_reset(&mas);
-		MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL));
-		mas_destroy(&mas);
-
-	}
-
-	/* Set allocation request. */
-	total = 500;
-	mas_node_count(&mas, total);
-	/* Drop the lock and allocate the nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	MT_BUG_ON(mt, !mas.alloc);
-	i = 1;
-	smn = mas.alloc;
-	while (i < total) {
-		for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) {
-			i++;
-			MT_BUG_ON(mt, !smn->slot[j]);
-			if (i == total)
-				break;
-		}
-		smn = smn->slot[0]; /* next. */
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != total);
-	mas_reset(&mas);
-	mas_destroy(&mas); /* Free. */
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	for (i = 1; i < 128; i++) {
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = i; j > 0; j--) { /*Free the requests */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	}
-
-	for (i = 1; i < MAPLE_NODE_MASK + 1; i++) {
-		MA_STATE(mas2, mt, 0, 0);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas2, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-		MT_BUG_ON(mt, mas_allocated(&mas2) != i);
-
-		for (j = i; j > 0; j--) { /*Free the requests */
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-			mn = mas_pop_node(&mas2); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas2) != 0);
-	}
-
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	mn = mas_pop_node(&mas); /* get the next node. */
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
-
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	/* Check the limit of pop/push/pop */
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_alloc_req(&mas));
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count  != MAPLE_ALLOC_SLOTS);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-
-	for (i = 3; i < MAPLE_NODE_MASK * 3; i++) {
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put it back */
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mn3 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_push_node(&mas, mn3);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-	}
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 5); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 5);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 2);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 3 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 3 + 2);
-	mas_destroy(&mas);
-
-	mtree_unlock(mt);
-}
-
 /*
  * Check erasing including RCU.
  */
@@ -35452,8 +35028,7 @@ static void check_dfs_preorder(struct maple_tree *mt)
 	mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
 	mas_reset(&mas);
 	mt_zero_nr_tallocated();
-	mt_set_non_kernel(200);
-	mas_expected_entries(&mas, max);
+	mt_set_non_kernel(1000);
 	for (count = 0; count <= max; count++) {
 		mas.index = mas.last = count;
 		mas_store(&mas, xa_mk_value(count));
@@ -35518,6 +35093,13 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 	return vacant_height;
 }
 
+static int mas_allocated(struct ma_state *mas)
+{
+	if (mas->sheaf)
+		return kmem_cache_sheaf_size(mas->sheaf);
+
+	return 0;
+}
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)
 {
@@ -35536,7 +35118,10 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 
 	/* Spanning store */
 	mas_set_range(&mas, 470, 500);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
+
+	mas_wr_preallocate(&wr_mas, ptr);
+	MT_BUG_ON(mt, mas.store_type != wr_spanning_store);
+	MT_BUG_ON(mt, mas_is_err(&mas));
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
 	vacant_height = get_vacant_height(&wr_mas, ptr);
@@ -35546,6 +35131,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	allocated = mas_allocated(&mas);
 	MT_BUG_ON(mt, allocated != 0);
 
+	mas_wr_preallocate(&wr_mas, ptr);
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -35586,20 +35172,6 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	mn->parent = ma_parent_ptr(mn);
 	ma_free_rcu(mn);
 
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	allocated = mas_allocated(&mas);
-	height = mas_mt_height(&mas);
-	vacant_height = get_vacant_height(&wr_mas, ptr);
-	MT_BUG_ON(mt, allocated != 1 + (height - vacant_height) * 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	mas_destroy(&mas);
-	allocated = mas_allocated(&mas);
-	MT_BUG_ON(mt, allocated != 0);
-
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -36400,11 +35972,17 @@ static void check_nomem_writer_race(struct maple_tree *mt)
 	check_load(mt, 6, xa_mk_value(0xC));
 	mtree_unlock(mt);
 
+	mt_set_non_kernel(0);
 	/* test for the same race but with mas_store_gfp() */
 	mtree_store_range(mt, 0, 5, xa_mk_value(0xA), GFP_KERNEL);
 	mtree_store_range(mt, 6, 10, NULL, GFP_KERNEL);
 
 	mas_set_range(&mas, 0, 5);
+
+	/* setup writer 2 that will trigger the race condition */
+	mt_set_private(mt);
+	mt_set_callback(writer2);
+
 	mtree_lock(mt);
 	mas_store_gfp(&mas, NULL, GFP_KERNEL);
 
@@ -36546,10 +36124,6 @@ void farmer_tests(void)
 	check_erase_testset(&tree);
 	mtree_destroy(&tree);
 
-	mt_init_flags(&tree, 0);
-	check_new_node(&tree);
-	mtree_destroy(&tree);
-
 	if (!MAPLE_32BIT) {
 		mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 		check_rcu_simulated(&tree);
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1..8c72571559583759456c2b469a2abc2611117c13 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -64,7 +64,8 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
 
 	if (!(gfp & __GFP_DIRECT_RECLAIM)) {
 		if (!cachep->non_kernel) {
-			cachep->exec_callback = true;
+			if (cachep->callback)
+				cachep->exec_callback = true;
 			return NULL;
 		}
 
@@ -210,6 +211,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 		for (i = 0; i < size; i++)
 			__kmem_cache_free_locked(cachep, p[i]);
 		pthread_mutex_unlock(&cachep->lock);
+		if (cachep->callback)
+			cachep->exec_callback = true;
 		return 0;
 	}
 

-- 
2.51.0



From lorenzo.stoakes at oracle.com  Wed Sep  3 06:31:21 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Wed, 3 Sep 2025 14:31:21 +0100
Subject: [PATCH v6 08/10] mm, vma: use percpu sheaves for vm_area_struct
 cache
In-Reply-To: <814c7054-3a43-4190-893f-22e074dc93cd@suse.cz>
References: <20250827-slub-percpu-caches-v6-0-f0f775a3f73f@suse.cz>
 <20250827-slub-percpu-caches-v6-8-f0f775a3f73f@suse.cz>
 <195d00ab-5429-48ff-b5dd-a45b26cd9ea9@lucifer.local>
 <814c7054-3a43-4190-893f-22e074dc93cd@suse.cz>
Message-ID: <d396be64-3a95-499f-a074-68297fa7bb48@lucifer.local>

On Wed, Sep 03, 2025 at 02:47:28PM +0200, Vlastimil Babka wrote:
> On 9/2/25 13:13, Lorenzo Stoakes wrote:
> > On Wed, Aug 27, 2025 at 10:26:40AM +0200, Vlastimil Babka wrote:
> >> Create the vm_area_struct cache with percpu sheaves of size 32 to
> >> improve its performance.
> >>
> >> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> >> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> >> ---
> >>  mm/vma_init.c | 1 +
> >>  1 file changed, 1 insertion(+)
> >>
> >> diff --git a/mm/vma_init.c b/mm/vma_init.c
> >> index 8e53c7943561e7324e7992946b4065dec1149b82..52c6b55fac4519e0da39ca75ad018e14449d1d95 100644
> >> --- a/mm/vma_init.c
> >> +++ b/mm/vma_init.c
> >> @@ -16,6 +16,7 @@ void __init vma_state_init(void)
> >>  	struct kmem_cache_args args = {
> >>  		.use_freeptr_offset = true,
> >>  		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
> >> +		.sheaf_capacity = 32,
> >
> > This breaks the VMA tests.
> >
> > Please also update tools/testing/vma/vma_internal.h to add this field
> > (probably actually at the commit that adds the field in mainline :P)
>
> Thanks, one of Liam's patches to support maple tree tests with sheaves does
> that, we somehow missed that the vma test also needs it so will reorder
> accordingly.
> > I do wish we could have these tests be run by a bot :/ seems today I am
> > that bot :)
> Sorry :)

Thanks :) Just happy to find the problem and help it get sorted! :>)


From oliver.sang at intel.com  Wed Sep  3 07:53:57 2025
From: oliver.sang at intel.com (kernel test robot)
Date: Wed, 3 Sep 2025 22:53:57 +0800
Subject: [vbabka:slub-percpu-sheaves-v6r1] [maple_tree]  46bfd29ca2:
 stress-ng.pagemove.page_remaps_per_sec 7.8% regression
Message-ID: <202509032258.411b604d-lkp@intel.com>



Hello,

kernel test robot noticed a 7.8% regression of stress-ng.pagemove.page_remaps_per_sec on:


commit: 46bfd29ca2ac25defc473db5682ea0dd8215814a ("maple_tree: use percpu sheaves for maple_node_cache")
https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-percpu-sheaves-v6r1

testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-12
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

	nr_threads: 100%
	testtime: 60s
	test: pagemove
	cpufreq_governor: performance




If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang at intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202509032258.411b604d-lkp at intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250903/202509032258.411b604d-lkp at intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  gcc-12/performance/x86_64-rhel-9.4/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp7/pagemove/stress-ng/60s

commit: 
  ac43de991d ("tools: Add sheaves support to testing infrastructure")
  46bfd29ca2 ("maple_tree: use percpu sheaves for maple_node_cache")

ac43de991d9a4a42 46bfd29ca2ac25defc473db5682 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
      0.15 ? 31%      +8.7        8.90 ? 29%  mpstat.cpu.all.soft%
     26992 ? 31%     -86.5%       3649 ? 18%  vmstat.system.cs
    304348 ? 17%    +124.5%     683375 ? 23%  meminfo.SUnreclaim
    400437 ? 12%     +94.6%     779201 ? 20%  meminfo.Slab
     38862 ? 20%    +116.9%      84299 ? 24%  numa-vmstat.node0.nr_slab_unreclaimable
     36829 ? 15%    +113.9%      78790 ? 25%  numa-vmstat.node1.nr_slab_unreclaimable
      8305 ? 22%     +64.2%      13635 ? 20%  perf-c2c.DRAM.local
      5195 ? 13%    +304.2%      20998 ?  5%  perf-c2c.HITM.local
      6550 ? 12%    +238.2%      22155 ?  4%  perf-c2c.HITM.total
    157267 ? 20%    +129.0%     360199 ? 25%  numa-meminfo.node0.SUnreclaim
    199417 ? 24%     +97.7%     394339 ? 23%  numa-meminfo.node0.Slab
    148061 ? 15%    +127.0%     336065 ? 26%  numa-meminfo.node1.SUnreclaim
    202003 ? 13%     +96.9%     397753 ? 22%  numa-meminfo.node1.Slab
     76366 ? 17%    +121.9%     169482 ? 23%  proc-vmstat.nr_slab_unreclaimable
  31732425            -6.4%   29713727        proc-vmstat.numa_hit
  31663453            -6.4%   29647558        proc-vmstat.numa_local
  60845467            -1.9%   59703569        proc-vmstat.pgalloc_normal
  60410743            -1.7%   59357556        proc-vmstat.pgfree
  20530240            -8.2%   18847954        stress-ng.pagemove.ops
    342222            -8.2%     314179        stress-ng.pagemove.ops_per_sec
    242134            -7.8%     223297        stress-ng.pagemove.page_remaps_per_sec
    951112           -89.1%     103990 ?  7%  stress-ng.time.involuntary_context_switches
      5975            -8.1%       5493        stress-ng.time.percent_of_cpu_this_job_got
      3499            -8.1%       3217        stress-ng.time.system_time
     91.57 ?  2%      -7.3%      84.85        stress-ng.time.user_time
     28017 ? 32%     -87.2%       3575 ? 18%  perf-stat.i.context-switches
      1969 ?  3%      -7.4%       1823 ?  2%  perf-stat.i.cycles-between-cache-misses
      0.50 ?  2%     +18.4%       0.59 ?  2%  perf-stat.overall.MPKI
      0.24 ?  4%      +0.0        0.28 ?  4%  perf-stat.overall.branch-miss-rate%
     73.76 ?  2%      +6.1       79.90        perf-stat.overall.cache-miss-rate%
      1.00            +8.6%       1.09        perf-stat.overall.cpi
      2003 ?  3%      -8.3%       1837 ?  3%  perf-stat.overall.cycles-between-cache-misses
      1.00            -7.9%       0.92        perf-stat.overall.ipc
     27584 ? 31%     -87.2%       3524 ? 18%  perf-stat.ps.context-switches
 1.181e+13            -7.7%   1.09e+13        perf-stat.total.instructions
     82208 ? 35%     -83.3%      13717 ?223%  sched_debug.cfs_rq:/.left_deadline.avg
   1731609 ? 24%     -82.9%     295452 ?223%  sched_debug.cfs_rq:/.left_deadline.max
    360700 ? 27%     -82.8%      61870 ?223%  sched_debug.cfs_rq:/.left_deadline.stddev
     82206 ? 35%     -83.3%      13717 ?223%  sched_debug.cfs_rq:/.left_vruntime.avg
   1731555 ? 24%     -82.9%     295431 ?223%  sched_debug.cfs_rq:/.left_vruntime.max
    360691 ? 27%     -82.8%      61868 ?223%  sched_debug.cfs_rq:/.left_vruntime.stddev
    316185 ?101%     -94.7%      16803 ? 72%  sched_debug.cfs_rq:/.load.avg
  18547721 ?108%     -98.6%     255633 ?149%  sched_debug.cfs_rq:/.load.max
   2319717 ?106%     -98.3%      38982 ?145%  sched_debug.cfs_rq:/.load.stddev
    885.40 ? 42%     -79.9%     178.37 ? 45%  sched_debug.cfs_rq:/.load_avg.stddev
     82206 ? 35%     -83.3%      13717 ?223%  sched_debug.cfs_rq:/.right_vruntime.avg
   1731559 ? 24%     -82.9%     295431 ?223%  sched_debug.cfs_rq:/.right_vruntime.max
    360691 ? 27%     -82.8%      61869 ?223%  sched_debug.cfs_rq:/.right_vruntime.stddev
     18775 ? 22%     -78.2%       4087 ? 19%  sched_debug.cpu.nr_switches.avg
     29179 ? 23%     -44.8%      16108 ? 21%  sched_debug.cpu.nr_switches.max
     14138 ? 23%     -89.8%       1448 ? 23%  sched_debug.cpu.nr_switches.min
      0.01 ? 20%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.get_free_pages_noprof
      0.09 ? 15%    +190.1%       0.27 ? 27%  perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.34 ? 10%   +1689.9%       6.10 ? 27%  perf-sched.sch_delay.avg.ms.__cond_resched.change_pmd_range.isra.0.change_pud_range
      0.29 ? 13%   +1932.3%       5.86 ? 19%  perf-sched.sch_delay.avg.ms.__cond_resched.change_pud_range.isra.0.change_protection_range
      0.25 ?  9%   +2276.9%       5.97 ?  5%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      0.34 ?  7%   +1657.8%       6.00 ? 16%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.anon_vma_clone.copy_vma.copy_vma_and_data
      0.35 ?  5%   +1974.1%       7.24 ? 12%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.move_ptes.move_page_tables.copy_vma_and_data
      0.39 ? 17%   +1522.1%       6.25 ? 11%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.unlink_anon_vmas.free_pgtables.vms_clear_ptes
      0.29 ?  7%   +1834.4%       5.57 ? 18%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_prepare.__split_vma.vms_gather_munmap_vmas
      0.30 ? 11%   +1905.2%       6.02 ? 21%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
      0.49 ? 24%    +863.8%       4.77 ? 63%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      0.32 ? 11%   +1739.1%       5.97 ? 20%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
      0.05 ? 25%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.kfree_rcu_work.process_one_work.worker_thread.kthread
      0.26 ? 10%   +2559.4%       6.90 ? 22%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      0.30 ?  8%   +2062.2%       6.39 ? 13%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      0.34 ? 32%   +1754.3%       6.27 ? 43%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.vma_link
      0.32 ? 11%   +1913.0%       6.49 ? 18%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      0.31 ? 10%   +1967.2%       6.46 ? 19%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      0.33 ? 12%   +1852.2%       6.49 ? 12%  perf-sched.sch_delay.avg.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
      0.02 ? 25%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
      0.39 ?  8%   +1385.0%       5.86 ? 13%  perf-sched.sch_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      0.04 ?119%    +276.3%       0.13 ? 23%  perf-sched.sch_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.53 ? 29%    +105.4%       1.10 ? 54%  perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      0.24 ?129%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.29 ? 14%   +1898.2%       5.87 ? 22%  perf-sched.sch_delay.avg.ms.__cond_resched.unmap_page_range.unmap_vmas.vms_clear_ptes.part
      0.27 ?  8%   +2310.4%       6.41 ?  9%  perf-sched.sch_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      0.32 ? 19%   +1824.1%       6.18 ? 48%  perf-sched.sch_delay.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      0.21 ?  7%   +3360.3%       7.24 ? 16%  perf-sched.sch_delay.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      0.23 ?  8%   +1503.0%       3.64 ? 12%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.52 ? 30%    +560.2%       3.45 ? 56%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.47 ? 53%    +662.5%       3.59 ? 46%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.02 ? 20%    +191.8%       0.05 ? 29%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.03 ? 19%   +1119.2%       0.37 ? 25%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.03 ?  2%    +262.7%       0.12 ? 92%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.02 ? 25%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.get_free_pages_noprof
      3.10 ? 28%   +1019.8%      34.72 ? 30%  perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      3.79 ? 22%   +1135.3%      46.77 ? 14%  perf-sched.sch_delay.max.ms.__cond_resched.change_pmd_range.isra.0.change_pud_range
      4.36 ? 21%   +1064.9%      50.74 ? 10%  perf-sched.sch_delay.max.ms.__cond_resched.change_pud_range.isra.0.change_protection_range
      5.47 ? 17%    +870.9%      53.14 ?  4%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      4.69 ? 15%    +925.3%      48.13 ?  6%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.anon_vma_clone.copy_vma.copy_vma_and_data
      4.64 ? 21%   +1020.2%      51.94 ?  4%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.move_ptes.move_page_tables.copy_vma_and_data
      4.03 ? 13%   +1051.4%      46.45 ? 19%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.unlink_anon_vmas.free_pgtables.vms_clear_ptes
      5.06 ? 26%    +876.5%      49.38 ? 10%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_prepare.__split_vma.vms_gather_munmap_vmas
      4.59 ? 33%   +1037.7%      52.22 ?  4%  perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
      3.24 ? 14%    +769.1%      28.18 ? 70%  perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      5.24 ? 32%    +898.2%      52.33 ?  5%  perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
      2.33 ?  2%     +40.3%       3.26 ? 37%  perf-sched.sch_delay.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64
      1.54 ? 56%     -99.3%       0.01 ?115%  perf-sched.sch_delay.max.ms.__cond_resched.kfree_rcu_monitor.process_one_work.worker_thread.kthread
      3.01 ? 67%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.kfree_rcu_work.process_one_work.worker_thread.kthread
      5.97 ? 18%    +793.3%      53.30 ?  5%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      6.14 ? 32%    +821.3%      56.60 ?  3%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      2.27 ? 44%   +1109.8%      27.49 ? 48%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.vma_link
      6.04 ? 22%    +795.0%      54.04        perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      5.43 ? 20%    +874.7%      52.94 ?  3%  perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      8.25 ? 60%    +562.2%      54.63        perf-sched.sch_delay.max.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
      0.97 ?101%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
      5.50 ? 17%    +830.6%      51.18 ?  6%  perf-sched.sch_delay.max.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      0.21 ?142%   +4630.1%      10.00 ? 44%  perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.50 ?121%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      3.33 ? 10%    +994.1%      36.47 ? 27%  perf-sched.sch_delay.max.ms.__cond_resched.unmap_page_range.unmap_vmas.vms_clear_ptes.part
      6.23 ? 12%    +789.7%      55.47        perf-sched.sch_delay.max.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      3.97 ? 24%    +812.2%      36.25 ? 42%  perf-sched.sch_delay.max.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      5.95 ? 25%    +791.9%      53.08 ?  7%  perf-sched.sch_delay.max.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      1.97 ? 58%    +263.6%       7.15 ? 52%  perf-sched.sch_delay.max.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
      7.12 ? 27%    +668.8%      54.75        perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.89 ? 44%    +720.4%      48.36 ? 10%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      5.76 ? 41%    +794.7%      51.50 ?  7%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.92 ?117%   +1009.3%      10.23 ? 49%  perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
      2.97 ? 16%    +155.7%       7.60 ? 84%  perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
      1.78 ? 69%    +245.5%       6.16 ? 57%  perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3.92 ? 27%    +228.4%      12.89 ? 13%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     27.62 ? 71%     -82.7%       4.77 ? 39%  perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      0.16 ?  8%   +1850.6%       3.12 ?  9%  perf-sched.total_sch_delay.average.ms
      9.73          +582.2%      66.40 ?  4%  perf-sched.total_wait_and_delay.average.ms
    140704           -88.3%      16412 ?  4%  perf-sched.total_wait_and_delay.count.ms
      3638 ?  5%     -19.8%       2917 ?  5%  perf-sched.total_wait_and_delay.max.ms
      9.57 ?  2%    +561.0%      63.28 ?  4%  perf-sched.total_wait_time.average.ms
      3638 ?  5%     -19.8%       2917 ?  5%  perf-sched.total_wait_time.max.ms
     39.58 ?154%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.get_free_pages_noprof
     10.91 ?  2%     +25.1%      13.64 ?  9%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      0.50 ?  9%   +2276.3%      11.95 ?  5%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      0.28 ?101%   +4248.3%      12.04 ? 21%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
      0.65 ? 11%   +1739.6%      11.95 ? 20%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
      0.52 ? 10%   +2560.3%      13.80 ? 22%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      0.59 ?  8%   +2062.1%      12.78 ? 13%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      0.64 ? 11%   +1912.0%      12.97 ? 18%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      0.62 ? 10%   +1967.2%      12.91 ? 19%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      0.66 ? 12%   +1854.0%      12.99 ? 12%  perf-sched.wait_and_delay.avg.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
      0.79 ?  8%   +1384.4%      11.72 ? 13%  perf-sched.wait_and_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      0.53 ?  8%   +2309.6%      12.82 ?  9%  perf-sched.wait_and_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      0.42 ?  7%   +3361.8%      14.48 ? 16%  perf-sched.wait_and_delay.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      0.90 ?  5%   +1294.4%      12.51 ? 11%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1.21 ? 31%    +495.3%       7.19 ? 46%  perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
    105.52 ?  8%     +68.9%     178.18 ?  7%  perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
    522.24 ?  3%     -67.5%     169.60 ? 15%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     12.74         +4477.0%     583.16 ?  5%  perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      3.83 ? 50%    -100.0%       0.00        perf-sched.wait_and_delay.count.__cond_resched.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.get_free_pages_noprof
      4875 ?  7%     -92.9%     345.83 ? 11%  perf-sched.wait_and_delay.count.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      1491 ?  5%     -82.5%     261.67 ? 12%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
     93.83 ? 43%     -75.7%      22.83 ? 79%  perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write
    215.00 ? 13%     -99.8%       0.33 ?141%  perf-sched.wait_and_delay.count.__cond_resched.kfree_rcu_monitor.process_one_work.worker_thread.kthread
      3319 ? 14%     -94.6%     178.33 ? 11%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      6164 ?  4%     -83.2%       1033 ? 10%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      2792 ?  5%     -79.8%     563.83 ?  7%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      2914 ?  5%     -82.5%     509.67 ?  9%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      4177 ?  6%     -78.4%     900.83 ? 10%  perf-sched.wait_and_delay.count.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
      1261 ?  6%     -76.2%     300.33 ? 11%  perf-sched.wait_and_delay.count.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      6.33 ? 98%  +11578.9%     739.67 ? 12%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     15156 ?  2%     -90.6%       1421 ?  5%  perf-sched.wait_and_delay.count.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      4087 ?  5%     -94.5%     226.83 ?  8%  perf-sched.wait_and_delay.count.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
     16960           -92.5%       1273 ? 19%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1756 ? 19%     -54.4%     801.33 ? 82%  perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
    116.67 ?  5%     -39.7%      70.33 ?  7%  perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
    689.33 ?  9%    +338.6%       3023 ? 14%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     65400           -99.3%     476.83 ?  7%  perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
    151.10 ?202%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.get_free_pages_noprof
      1001           +22.7%       1227 ? 16%  perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
     10.94 ? 17%    +870.9%     106.27 ?  4%  perf-sched.wait_and_delay.max.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      3.94 ?113%   +2553.3%     104.43 ?  4%  perf-sched.wait_and_delay.max.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
     10.48 ? 32%    +898.2%     104.65 ?  5%  perf-sched.wait_and_delay.max.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
     11.93 ? 18%    +793.3%     106.60 ?  5%  perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
     12.29 ? 32%    +821.3%     113.20 ?  3%  perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
     12.07 ? 22%    +795.0%     108.07        perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
     10.86 ? 20%    +874.7%     105.88 ?  3%  perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
     16.50 ? 60%    +562.2%     109.26        perf-sched.wait_and_delay.max.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
     11.00 ? 17%    +830.6%     102.37 ?  6%  perf-sched.wait_and_delay.max.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     12.47 ? 12%    +789.7%     110.94        perf-sched.wait_and_delay.max.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
     11.90 ? 25%    +791.9%     106.17 ?  7%  perf-sched.wait_and_delay.max.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      6.56 ? 26%    +159.9%      17.04 ? 46%  perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3545 ?  9%     -69.1%       1094 ?  7%  perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.34 ? 10%   +1689.9%       6.10 ? 27%  perf-sched.wait_time.avg.ms.__cond_resched.change_pmd_range.isra.0.change_pud_range
      0.29 ? 13%   +1932.2%       5.86 ? 19%  perf-sched.wait_time.avg.ms.__cond_resched.change_pud_range.isra.0.change_protection_range
      0.25 ?  9%   +2276.9%       5.97 ?  5%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      0.34 ?  7%   +1657.8%       6.00 ? 16%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.anon_vma_clone.copy_vma.copy_vma_and_data
      0.35 ?  5%   +1974.1%       7.24 ? 12%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.move_ptes.move_page_tables.copy_vma_and_data
      0.39 ? 17%   +1522.1%       6.25 ? 11%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.unlink_anon_vmas.free_pgtables.vms_clear_ptes
      0.29 ?  7%   +1834.4%       5.57 ? 18%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.vma_prepare.__split_vma.vms_gather_munmap_vmas
      0.30 ? 11%   +1905.2%       6.02 ? 21%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
      0.49 ? 24%    +863.8%       4.77 ? 63%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      0.32 ? 11%   +1739.1%       5.97 ? 20%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
     11.93 ? 30%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.kfree_rcu_work.process_one_work.worker_thread.kthread
      0.26 ? 10%   +2559.4%       6.90 ? 22%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      0.30 ?  8%   +2062.2%       6.39 ? 13%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      0.34 ? 32%   +1754.3%       6.27 ? 43%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.vma_link
      0.32 ? 11%   +1913.0%       6.49 ? 18%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      0.31 ? 10%   +1967.2%       6.46 ? 19%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      0.33 ? 12%   +1852.2%       6.49 ? 12%  perf-sched.wait_time.avg.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
      8.76 ? 42%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
      0.39 ?  8%   +1385.0%       5.86 ? 13%  perf-sched.wait_time.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      0.24 ?129%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.29 ? 14%   +1898.2%       5.87 ? 22%  perf-sched.wait_time.avg.ms.__cond_resched.unmap_page_range.unmap_vmas.vms_clear_ptes.part
      0.27 ?  8%   +2310.4%       6.41 ?  9%  perf-sched.wait_time.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      0.32 ? 19%   +1824.1%       6.18 ? 48%  perf-sched.wait_time.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      0.21 ?  7%   +3360.2%       7.24 ? 16%  perf-sched.wait_time.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      0.67 ?  7%   +1223.9%       8.88 ? 13%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      0.52 ? 30%    +559.6%       3.45 ? 56%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.74 ? 32%    +389.0%       3.61 ? 46%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
    104.99 ?  8%     +68.9%     177.29 ?  7%  perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
    522.21 ?  3%     -67.6%     169.23 ? 15%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     12.71         +4488.3%     583.04 ?  5%  perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
      1000           +22.0%       1220 ? 17%  perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      3.79 ? 22%   +1135.3%      46.77 ? 14%  perf-sched.wait_time.max.ms.__cond_resched.change_pmd_range.isra.0.change_pud_range
      4.36 ? 21%   +1064.9%      50.74 ? 10%  perf-sched.wait_time.max.ms.__cond_resched.change_pud_range.isra.0.change_protection_range
      5.47 ? 17%    +870.9%      53.14 ?  4%  perf-sched.wait_time.max.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.vms_gather_munmap_vmas
      4.69 ? 15%    +925.3%      48.13 ?  6%  perf-sched.wait_time.max.ms.__cond_resched.down_write.anon_vma_clone.copy_vma.copy_vma_and_data
      4.64 ? 21%   +1020.2%      51.94 ?  4%  perf-sched.wait_time.max.ms.__cond_resched.down_write.move_ptes.move_page_tables.copy_vma_and_data
      4.03 ? 13%   +1051.4%      46.45 ? 19%  perf-sched.wait_time.max.ms.__cond_resched.down_write.unlink_anon_vmas.free_pgtables.vms_clear_ptes
      5.06 ? 26%    +876.5%      49.38 ? 10%  perf-sched.wait_time.max.ms.__cond_resched.down_write.vma_prepare.__split_vma.vms_gather_munmap_vmas
      4.59 ? 33%   +1037.7%      52.22 ?  4%  perf-sched.wait_time.max.ms.__cond_resched.down_write.vma_prepare.commit_merge.vma_expand
      3.24 ? 14%    +769.1%      28.18 ? 70%  perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64
      5.24 ? 32%    +898.2%      52.33 ?  5%  perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.do_mremap.__do_sys_mremap.do_syscall_64
      2.33 ?  2%     +40.3%       3.26 ? 37%  perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64
    920.04 ?115%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.kfree_rcu_work.process_one_work.worker_thread.kthread
      5.97 ? 18%    +793.3%      53.30 ?  5%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      6.14 ? 32%    +821.3%      56.60 ?  3%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.commit_merge
      2.27 ? 44%   +1109.8%      27.49 ? 48%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.vma_link
      6.04 ? 22%    +795.0%      54.04        perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      5.43 ? 20%    +874.7%      52.94 ?  3%  perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.copy_vma.copy_vma_and_data
      8.25 ? 60%    +562.2%      54.63        perf-sched.wait_time.max.ms.__cond_resched.move_page_tables.copy_vma_and_data.move_vma.remap_move
    681.18 ? 74%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork
      5.50 ? 17%    +830.6%      51.18 ?  6%  perf-sched.wait_time.max.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      0.50 ?121%    -100.0%       0.00        perf-sched.wait_time.max.ms.__cond_resched.task_work_run.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      3.33 ? 10%    +994.1%      36.47 ? 27%  perf-sched.wait_time.max.ms.__cond_resched.unmap_page_range.unmap_vmas.vms_clear_ptes.part
      6.23 ? 12%    +789.7%      55.47        perf-sched.wait_time.max.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      3.97 ? 24%    +812.2%      36.25 ? 42%  perf-sched.wait_time.max.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      5.95 ? 25%    +791.9%      53.08 ?  7%  perf-sched.wait_time.max.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      6.12 ? 40%    +689.8%      48.36 ? 10%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.92 ?117%   +1009.3%      10.23 ? 49%  perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
      5.73 ? 15%    +117.2%      12.44 ? 45%  perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3545 ?  9%     -69.1%       1094 ?  7%  perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm




Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



From Liam.Howlett at oracle.com  Wed Sep  3 11:00:54 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Wed, 3 Sep 2025 14:00:54 -0400
Subject: [PATCH] maple_tree: remove lockdep_map_p typedef
In-Reply-To: <20250902-maple-lockdep-p-v1-1-3ae5a398a379@google.com>
References: <20250902-maple-lockdep-p-v1-1-3ae5a398a379@google.com>
Message-ID: <o2vlyvq53kciyc5vhm7jgrvormdw6oyqmsnzbgagiq4niuurop@n3xa7p7ti7xq>

* Alice Ryhl <aliceryhl at google.com> [250902 04:36]:
> Having the ma_external_lock field exist when CONFIG_LOCKDEP=n isn't used
> anywhere, so just get rid of it. This also avoids generating a typedef
> called lockdep_map_p that could overlap with typedefs in other header
> files.
> 
> Reviewed-by: Danilo Krummrich <dakr at kernel.org>
> Signed-off-by: Alice Ryhl <aliceryhl at google.com>

If it helps you then let's do it this way.  I don't think there's much
risk leaving it the way it is, but there's not much risk changing it
either.

Reviewed-by: Liam R. Howlett <Liam.Howlett at oracle.com>

> ---
> Originally sent as
> https://lore.kernel.org/all/20250819-maple-tree-v2-1-229b48657bab at google.com/
> 
> But I'm moving it out of that series as it did not have the intended
> effect on the Rust bindgen output. However, I still think it makes sense
> as a pure cleanup patch.
> ---
>  include/linux/maple_tree.h | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
> index bafe143b1f783202e27b32567fffee4149e8e266..8244679ba1758235e049acbaedee62aae5c0e226 100644
> --- a/include/linux/maple_tree.h
> +++ b/include/linux/maple_tree.h
> @@ -194,7 +194,6 @@ enum store_type {
>  #define MAPLE_RESERVED_RANGE	4096
>  
>  #ifdef CONFIG_LOCKDEP
> -typedef struct lockdep_map *lockdep_map_p;
>  #define mt_lock_is_held(mt)                                             \
>  	(!(mt)->ma_external_lock || lock_is_held((mt)->ma_external_lock))
>  
> @@ -207,7 +206,6 @@ typedef struct lockdep_map *lockdep_map_p;
>  
>  #define mt_on_stack(mt)			(mt).ma_external_lock = NULL
>  #else
> -typedef struct { /* nothing */ } lockdep_map_p;
>  #define mt_lock_is_held(mt)		1
>  #define mt_write_lock_is_held(mt)	1
>  #define mt_set_external_lock(mt, lock)	do { } while (0)
> @@ -230,8 +228,10 @@ typedef struct { /* nothing */ } lockdep_map_p;
>   */
>  struct maple_tree {
>  	union {
> -		spinlock_t	ma_lock;
> -		lockdep_map_p	ma_external_lock;
> +		spinlock_t		ma_lock;
> +#ifdef CONFIG_LOCKDEP
> +		struct lockdep_map	*ma_external_lock;
> +#endif
>  	};
>  	unsigned int	ma_flags;
>  	void __rcu      *ma_root;
> 
> ---
> base-commit: 1b237f190eb3d36f52dffe07a40b5eb210280e00
> change-id: 20250902-maple-lockdep-p-a930e865c76b
> 
> Best regards,
> -- 
> Alice Ryhl <aliceryhl at google.com>
> 


From Liam.Howlett at oracle.com  Wed Sep  3 12:56:03 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Wed, 3 Sep 2025 15:56:03 -0400
Subject: [RFC PATCH 2/6] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <bdb27f25-d979-4dce-889e-a2fd17fc06a5@lucifer.local>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-3-Liam.Howlett@oracle.com>
 <bdb27f25-d979-4dce-889e-a2fd17fc06a5@lucifer.local>
Message-ID: <ft4brzgr6jitmyfs4jplbokpqoz2z2ejsudfrezuke2oqygbs5@b5i3zaxgxrta>

* Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 14:38]:
> On Fri, Aug 15, 2025 at 03:10:27PM -0400, Liam R. Howlett wrote:
> > Create the new function tear_down_vmas() to remove a range of vmas.
> > exit_mmap() will be removing all the vmas.
> >
> > This is necessary for future patches.
> >
> > No functional changes intended.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> 
> This function is pure and complete insanity, but this change looks
> good. Couple nits below.
> 
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> 
> > ---
> >  mm/mmap.c | 37 ++++++++++++++++++++++++-------------
> >  1 file changed, 24 insertions(+), 13 deletions(-)
> >
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index c4c315b480af7..0995a48b46d59 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
> >  }
> >  EXPORT_SYMBOL(vm_brk_flags);
> >
> > +static inline
> > +unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
> > +		struct vm_area_struct *vma, unsigned long max)
> > +{
> > +	unsigned long nr_accounted = 0;
> > +	int count = 0;
> > +
> > +	mmap_assert_write_locked(mm);
> > +	vma_iter_set(vmi, vma->vm_end);
> > +	do {
> > +		if (vma->vm_flags & VM_ACCOUNT)
> > +			nr_accounted += vma_pages(vma);
> > +		vma_mark_detached(vma);
> > +		remove_vma(vma);
> > +		count++;
> > +		cond_resched();
> > +		vma = vma_next(vmi);
> > +	} while (vma && vma->vm_end <= max);
> > +
> > +	BUG_ON(count != mm->map_count);
> 
> Can we make this a WARN_ON() or WARN_ON_ONCE() while we're at it?

Sure!

> 
> > +	return nr_accounted;
> > +}
> > +
> >  /* Release all mmaps. */
> >  void exit_mmap(struct mm_struct *mm)
> >  {
> > @@ -1257,7 +1280,6 @@ void exit_mmap(struct mm_struct *mm)
> >  	struct vm_area_struct *vma;
> >  	unsigned long nr_accounted = 0;
> 
> No need to initialise this to 0 any more.

There is a goto label below that skips calling the tear down, so this is
still needed.

Thanks,
Liam


From Liam.Howlett at oracle.com  Wed Sep  3 12:57:37 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Wed, 3 Sep 2025 15:57:37 -0400
Subject: [RFC PATCH 3/6] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <a32ea375-f093-4f8f-9270-fbe9f7977cc6@lucifer.local>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-4-Liam.Howlett@oracle.com>
 <a32ea375-f093-4f8f-9270-fbe9f7977cc6@lucifer.local>
Message-ID: <w43jgvlabgvxpq2jb3zv7zo64r4xbvmad7cf75l3keests7djf@lmqslvejyyk4>

* Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 14:48]:
> On Fri, Aug 15, 2025 at 03:10:28PM -0400, Liam R. Howlett wrote:
> > Add a limit to the vma search instead of using the start and end of the
> > one passed in.
> >
> > No functional changes intended.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > ---
> >  mm/vma.c | 6 ++++--
> >  mm/vma.h | 1 +
> >  2 files changed, 5 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/vma.c b/mm/vma.c
> > index 3b12c7579831b..fd270345c25d3 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
> >   * Called with the mm semaphore held.
> >   */
> >  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> > +		unsigned long vma_min, unsigned long vma_max,
> >  		struct vm_area_struct *prev, struct vm_area_struct *next)
> 
> Luckily we will be getting rid of this as soon as I complete the mmap_prepare +
> friends series (_actively_ working on this right now).
> 
> Anyway, this is getting quickly horrid, since this is only in vma.c, we could
> make this static and pass through map to reduce some of the horror?
> 
> We actually can hand it vmi (and thus &vmi->mas via map, so we could implement
> it as:
> 
> void unmap_region(struct mmap_state *map, struct vm_area_struct *vma,
> 		  unsigned long min, unsigned long max);
> 

Ah... I could do this on top of the last patch.. that might be more
clear?

> >  {
> >  	struct mm_struct *mm = vma->vm_mm;
> > @@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> >
> >  	tlb_gather_mmu(&tlb, mm);
> >  	update_hiwater_rss(mm);
> > -	unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
> > +	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
> >  		   /* mm_wr_locked = */ true);
> >  	mas_set(mas, vma->vm_end);
> >  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> > @@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
> >
> >  		vma_iter_set(vmi, vma->vm_end);
> >  		/* Undo any partial mapping done by a device driver. */
> > -		unmap_region(&vmi->mas, vma, map->prev, map->next);
> > +		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> > +			     map->prev, map->next);
> >
> >  		return error;
> >  	}
> > diff --git a/mm/vma.h b/mm/vma.h
> > index b123a9cdedb0d..336dae295853e 100644
> > --- a/mm/vma.h
> > +++ b/mm/vma.h
> > @@ -281,6 +281,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
> >  void remove_vma(struct vm_area_struct *vma);
> >
> >  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> > +		unsigned long min, unsigned long max,
> >  		struct vm_area_struct *prev, struct vm_area_struct *next);
> >
> >  /* We are about to modify the VMA's flags. */
> > --
> > 2.47.2
> >


From Liam.Howlett at oracle.com  Wed Sep  3 13:19:04 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Wed, 3 Sep 2025 16:19:04 -0400
Subject: [RFC PATCH 4/6] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-5-Liam.Howlett@oracle.com>
 <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
Message-ID: <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>

* Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 15:14]:
> On Fri, Aug 15, 2025 at 03:10:29PM -0400, Liam R. Howlett wrote:
> > The ceiling and tree search limit need to be different arguments for the
> > future change in the failed fork attempt.
> >
> > No functional changes intended.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> 
> (Obv. in addition to comment about broken VMA tests :P)
> 
> I guess intent is that if we discover any page tables beyond tree_max then
> we ought to just wipe them all out so, in effect, we don't consider
> mappings at or past tree_max to be valid?

Actually... there are some archs that map outside the vma and they are
valid.. I think mips? and I think lower, but yeah.. it's needed.  This
is why prev->vm_end and next->vm_start are used as page table limits,
afaik.  This is a serious annoyance because it frequently adds walks
that are infrequently necessary to the vma tree.

> 
> I feel like we need a comment to this effect as this is confusing as it is.
> 
> Could we add a kerneldoc comment for free_pgtables() spelling this out?

I'll add a note here, but confusion will probably increase.  I'll add a
note about the tree max as well.

> 
> > ---
> >  mm/internal.h | 4 +++-
> >  mm/memory.c   | 7 ++++---
> >  mm/mmap.c     | 2 +-
> >  mm/vma.c      | 3 ++-
> >  4 files changed, 10 insertions(+), 6 deletions(-)
> >
> > diff --git a/mm/internal.h b/mm/internal.h
> > index 45b725c3dc030..f9a278ac76d83 100644
> > --- a/mm/internal.h
> > +++ b/mm/internal.h
> > @@ -444,7 +444,9 @@ void folio_activate(struct folio *folio);
> >
> >  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		   struct vm_area_struct *start_vma, unsigned long floor,
> > -		   unsigned long ceiling, bool mm_wr_locked);
> > +		   unsigned long ceiling, unsigned long tree_max,
> > +		   bool mm_wr_locked);
> > +
> >  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
> >
> >  struct zap_details;
> > diff --git a/mm/memory.c b/mm/memory.c
> > index 0ba4f6b718471..3346514562bba 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -371,7 +371,8 @@ void free_pgd_range(struct mmu_gather *tlb,
> >
> >  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		   struct vm_area_struct *vma, unsigned long floor,
> > -		   unsigned long ceiling, bool mm_wr_locked)
> > +		   unsigned long ceiling, unsigned long tree_max,
> > +		   bool mm_wr_locked)
> >  {
> >  	struct unlink_vma_file_batch vb;
> >
> > @@ -385,7 +386,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
> >  		 * be 0.  This will underflow and is okay.
> >  		 */
> > -		next = mas_find(mas, ceiling - 1);
> > +		next = mas_find(mas, tree_max - 1);
> 
> Do we need to put some sort of sanity checks in to make sure tree_max <= ceiling
> (though this 0 case is a pain... so I guess tree_max - 1 <= ceiling - 1?)

Sure!

> 
> >  		if (unlikely(xa_is_zero(next)))
> >  			next = NULL;
> >
> > @@ -405,7 +406,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		 */
> >  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
> >  			vma = next;
> > -			next = mas_find(mas, ceiling - 1);
> > +			next = mas_find(mas, tree_max - 1);
> >  			if (unlikely(xa_is_zero(next)))
> >  				next = NULL;
> >  			if (mm_wr_locked)
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index 0995a48b46d59..eba2bc81bc749 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -1311,7 +1311,7 @@ void exit_mmap(struct mm_struct *mm)
> >  	mt_clear_in_rcu(&mm->mm_mt);
> >  	vma_iter_set(&vmi, vma->vm_end);
> >  	free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
> > -		      USER_PGTABLES_CEILING, true);
> > +		      USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
> >  	tlb_finish_mmu(&tlb);
> >
> >  	/*
> > diff --git a/mm/vma.c b/mm/vma.c
> > index fd270345c25d3..aa75ca8618609 100644
> > --- a/mm/vma.c
> > +++ b/mm/vma.c
> > @@ -486,6 +486,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> >  		   /* mm_wr_locked = */ true);
> >  	mas_set(mas, vma->vm_end);
> >  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> > +		      next ? next->vm_start : USER_PGTABLES_CEILING,
> >  		      next ? next->vm_start : USER_PGTABLES_CEILING,
> >  		      /* mm_wr_locked = */ true);
> >  	tlb_finish_mmu(&tlb);
> > @@ -1232,7 +1233,7 @@ static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
> >  	mas_set(mas_detach, 1);
> >  	/* start and end may be different if there is no prev or next vma. */
> >  	free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
> > -		      vms->unmap_end, mm_wr_locked);
> > +		      vms->unmap_end, vms->unmap_end, mm_wr_locked);
> >  	tlb_finish_mmu(&tlb);
> >  	vms->clear_ptes = false;
> >  }
> > --
> > 2.47.2
> >


From Liam.Howlett at oracle.com  Wed Sep  3 17:13:45 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Wed, 3 Sep 2025 20:13:45 -0400
Subject: [RFC PATCH 6/6] mm: Change dup_mmap() recovery
In-Reply-To: <de385044-e9a2-45a1-b74f-68ba80bba146@lucifer.local>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-7-Liam.Howlett@oracle.com>
 <de385044-e9a2-45a1-b74f-68ba80bba146@lucifer.local>
Message-ID: <jr57bivqyzarzicmitzuz4unk5g6gytfnh5olmradn7nmei7yt@s35yr2zhknz5>

* Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 16:34]:
> On Fri, Aug 15, 2025 at 03:10:31PM -0400, Liam R. Howlett wrote:
> > When the dup_mmap() fails during the vma duplication or setup, don't
> > write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> > free the new resources, leaving an empty vma tree.
> 
> Yesss like it!
> 
> >
> > Using XA_ZERO introduced races where the vma could be found between
> > dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> > race can occur because the mm can be reached through the other trees
> > via successfully copied vmas and other methods such as the swapoff code.
> 
> Yeah god.
> 
> >
> > XA_ZERO was marking the location to stop vma removal and pagetable
> > freeing.  The newly created arguments to the unmap_vmas() and
> > free_pgtables() serve this function.
> 
> Nice.
> 
> >
> > Replacing the XA_ZERO entry use with the new argument list also means
> > the checks for xa_is_zero() are no longer necessary so these are also
> > removed.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> 
> Ah finally the 'action' patch :)
> 
> Obviously my review is on the basis that you fixup the rebase etc.
> 
> This broadly looks right but some various nits etc. below and will have
> another look through on new revision - this whole area is pretty crazy!
> 
> > ---
> >  mm/memory.c |  6 +-----
> >  mm/mmap.c   | 39 ++++++++++++++++++++++++++++-----------
> >  2 files changed, 29 insertions(+), 16 deletions(-)
> >
> > diff --git a/mm/memory.c b/mm/memory.c
> > index 3346514562bba..8cd58f171bae4 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -387,8 +387,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		 * be 0.  This will underflow and is okay.
> >  		 */
> >  		next = mas_find(mas, tree_max - 1);
> > -		if (unlikely(xa_is_zero(next)))
> > -			next = NULL;
> >
> >  		/*
> >  		 * Hide vma from rmap and truncate_pagecache before freeing
> > @@ -407,8 +405,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> >  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
> >  			vma = next;
> >  			next = mas_find(mas, tree_max - 1);
> > -			if (unlikely(xa_is_zero(next)))
> > -				next = NULL;
> >  			if (mm_wr_locked)
> >  				vma_start_write(vma);
> >  			unlink_anon_vmas(vma);
> > @@ -1978,7 +1974,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> >  				 mm_wr_locked);
> >  		hugetlb_zap_end(vma, &details);
> >  		vma = mas_find(mas, tree_end - 1);
> > -	} while (vma && likely(!xa_is_zero(vma)));
> > +	} while (vma);
> >  	mmu_notifier_invalidate_range_end(&range);
> >  }
> >
> > diff --git a/mm/mmap.c b/mm/mmap.c
> > index eba2bc81bc749..5acc0b5f8c14a 100644
> > --- a/mm/mmap.c
> > +++ b/mm/mmap.c
> > @@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
> >  	arch_exit_mmap(mm);
> >
> >  	vma = vma_next(&vmi);
> > -	if (!vma || unlikely(xa_is_zero(vma))) {
> > +	if (!vma) {
> >  		/* Can happen if dup_mmap() received an OOM */
> >  		mmap_read_unlock(mm);
> >  		mmap_write_lock(mm);
> > @@ -1858,20 +1858,37 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
> >  		ksm_fork(mm, oldmm);
> >  		khugepaged_fork(mm, oldmm);
> >  	} else {
> > +		unsigned long max;
> >
> >  		/*
> > -		 * The entire maple tree has already been duplicated. If the
> > -		 * mmap duplication fails, mark the failure point with
> > -		 * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
> > -		 * stop releasing VMAs that have not been duplicated after this
> > -		 * point.
> > +		 * The entire maple tree has already been duplicated, but
> > +		 * replacing the vmas failed at mpnt (which could be NULL if
> > +		 * all were allocated but the last vma was not fully set up. Use
> 
> Missing ')'.

Thanks.

> 
> > +		 * the start address of the failure point to clean up the half
> > +		 * initialized tree.
> 
> NIT: Is 'half' correct? Maybe 'partially'?

Thanks.

> 
> >  		 */
> > -		if (mpnt) {
> > -			mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
> > -			mas_store(&vmi.mas, XA_ZERO_ENTRY);
> > -			/* Avoid OOM iterating a broken tree */
> > -			set_bit(MMF_OOM_SKIP, &mm->flags);
> > +		if (!mm->map_count) {
> > +			/* zero vmas were written to the new tree. */
> > +			max = 0;
> 
> OK I guess this then will result in the intentional underflow thing, maybe
> worth mentioning?

No, nothing will happen as the if (max) will evaluate as false.

> 
> > +		} else if (mpnt) {
> > +			/* mid-tree failure */
> 
> Partial?

Right, thanks.

> 
> > +			max = mpnt->vm_start;
> > +		} else {
> > +			/* All vmas were written to the new tree */
> > +			max = ULONG_MAX;
> >  		}
> > +
> > +		/* Hide mm from oom killer because the memory is being freed */
> > +		set_bit(MMF_OOM_SKIP, &mm->flags);
> 
> Obv. update to mm_flags_set(MMF_OOM_SKIP, mm);

Right, happened on rebase.

> 
> > +		if (max) {
> > +			vma_iter_set(&vmi, 0);
> > +			tmp = vma_next(&vmi);
> > +			flush_cache_mm(mm);
> > +			unmap_region(&vmi.mas, tmp, 0, max, max, NULL, NULL);
> 
> (An aside for the unmap_region() impl, maybe let's name the pg_max as
> tree_max to make it consistent across both?)
> 
> Hm we could still use the mmap struct here if we put in vma.h. Just have to
> set vmi, obv prev, next NULL.
> 
> So:
> 
> 	struct mmap_state map {
> 		.vmi = &vmi,
> 	}
> 
> 	unmap_region(&map, tmp, 0, max, max);
> 
> Possibly overkill and hopefully stack ok but makes other callers less
> horrid.
> 
> Maybe also good to add a comment spelling out what each of these params do.

So I've tried to do this cleanly but it hasn't worked out.  I think I'll
add another struct to clean this up in another patch on this series.

Thanks,
Liam


From harry.yoo at oracle.com  Wed Sep  3 18:38:34 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Thu, 4 Sep 2025 10:38:34 +0900
Subject: [PATCH v7 01/21] locking/local_lock: Expose dep_map in
 local_trylock_t.
In-Reply-To: <20250903-slub-percpu-caches-v7-1-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-1-71c114cdefef@suse.cz>
Message-ID: <aLjtmm70QBr2rdoS@hyeyoo>

On Wed, Sep 03, 2025 at 02:59:43PM +0200, Vlastimil Babka wrote:
> From: Alexei Starovoitov <ast at kernel.org>
> 
> lockdep_is_held() macro assumes that "struct lockdep_map dep_map;"
> is a top level field of any lock that participates in LOCKDEP.
> Make it so for local_trylock_t.
> 
> Reviewed-by: Sebastian Andrzej Siewior <bigeasy at linutronix.de>
> Signed-off-by: Alexei Starovoitov <ast at kernel.org>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---

Looks good to me,
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

>  include/linux/local_lock_internal.h | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h
> index d80b5306a2c0ccf95a3405b6b947b5f1f9a3bd38..949de37700dbc10feafc06d0b52382cf2e00c694 100644
> --- a/include/linux/local_lock_internal.h
> +++ b/include/linux/local_lock_internal.h
> @@ -17,7 +17,10 @@ typedef struct {
>  
>  /* local_trylock() and local_trylock_irqsave() only work with local_trylock_t */
>  typedef struct {
> -	local_lock_t	llock;
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +	struct lockdep_map	dep_map;
> +	struct task_struct	*owner;
> +#endif
>  	u8		acquired;
>  } local_trylock_t;
>  
> @@ -31,7 +34,7 @@ typedef struct {
>  	.owner = NULL,
>  
>  # define LOCAL_TRYLOCK_DEBUG_INIT(lockname)		\
> -	.llock = { LOCAL_LOCK_DEBUG_INIT((lockname).llock) },
> +	LOCAL_LOCK_DEBUG_INIT(lockname)
>  
>  static inline void local_lock_acquire(local_lock_t *l)
>  {
> @@ -81,7 +84,7 @@ do {								\
>  	local_lock_debug_init(lock);				\
>  } while (0)
>  
> -#define __local_trylock_init(lock) __local_lock_init(lock.llock)
> +#define __local_trylock_init(lock) __local_lock_init((local_lock_t *)lock)
>  
>  #define __spinlock_nested_bh_init(lock)				\
>  do {								\
> 
> -- 
> 2.51.0
> 

-- 
Cheers,
Harry / Hyeonggon


From harry.yoo at oracle.com  Wed Sep  3 18:41:01 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Thu, 4 Sep 2025 10:41:01 +0900
Subject: [PATCH v7 02/21] slab: simplify init_kmem_cache_nodes() error
 handling
In-Reply-To: <20250903-slub-percpu-caches-v7-2-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-2-71c114cdefef@suse.cz>
Message-ID: <aLjuLRHHfEmJ2tRf@hyeyoo>

On Wed, Sep 03, 2025 at 02:59:44PM +0200, Vlastimil Babka wrote:
> We don't need to call free_kmem_cache_nodes() immediately when failing
> to allocate a kmem_cache_node, because when we return 0,
> do_kmem_cache_create() calls __kmem_cache_release() which also performs
> free_kmem_cache_nodes().
> 
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Looks good to me,
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

> ---
>  mm/slub.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/mm/slub.c b/mm/slub.c
> index 30003763d224c2704a4b93082b8b47af12dcffc5..9f671ec76131c4b0b28d5d568aa45842b5efb6d4 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -5669,10 +5669,8 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
>  		n = kmem_cache_alloc_node(kmem_cache_node,
>  						GFP_KERNEL, node);
>  
> -		if (!n) {
> -			free_kmem_cache_nodes(s);
> +		if (!n)
>  			return 0;
> -		}
>  
>  		init_kmem_cache_node(n);
>  		s->node[node] = n;
> 
> -- 
> 2.51.0
> 
> 

-- 
Cheers,
Harry / Hyeonggon


From oliver.sang at intel.com  Wed Sep  3 22:20:30 2025
From: oliver.sang at intel.com (kernel test robot)
Date: Thu, 4 Sep 2025 13:20:30 +0800
Subject: [vbabka:slub-percpu-sheaves-v6r2] [maple_tree] 99848157f7:
 WARNING:possible_circular_locking_dependency_detected
Message-ID: <202509041248.b7ece14a-lkp@intel.com>



Hello,

kernel test robot noticed "WARNING:possible_circular_locking_dependency_detected" on:

commit: 99848157f71c03ca5e975b0e686def1d66e8b19f ("maple_tree: Sheaf conversion")
https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-percpu-sheaves-v6r2

in testcase: rcuscale
version: 
with following parameters:

	runtime: 300s
	scale_type: rcu



config: x86_64-randconfig-075-20250831
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang at intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202509041248.b7ece14a-lkp at intel.com


[   17.413963][   T74] WARNING: possible circular locking dependency detected
[   17.414491][   T74] 6.17.0-rc3-00017-g99848157f71c #1 Not tainted
[   17.414936][   T74] ------------------------------------------------------
[   17.415483][   T74] systemd-cryptse/74 is trying to acquire lock:
[ 17.415965][ T74] ffffffffb02789c0 (fs_reclaim){+.+.}-{0:0}, at: prepare_alloc_pages+0xd5/0x23d 
[   17.416782][   T74]
[   17.416782][   T74] but task is already holding lock:
[ 17.417298][ T74] ffff88812925dbf8 (&mapping->i_mmap_rwsem){++++}-{4:4}, at: vma_prepare (mm/vma.c:296) 
[   17.418021][   T74]
[   17.418021][   T74] which lock already depends on the new lock.
[   17.418021][   T74]
[   17.418779][   T74]
[   17.418779][   T74] the existing dependency chain (in reverse order) is:
[   17.419530][   T74]
[   17.419530][   T74] -> #1 (&mapping->i_mmap_rwsem){++++}-{4:4}:
[ 17.420152][ T74] validate_chain (kernel/locking/lockdep.c:3285 kernel/locking/lockdep.c:3908) 
[ 17.420562][ T74] __lock_acquire (kernel/locking/lockdep.c:5237) 
[ 17.420968][ T74] lock_acquire (kernel/locking/lockdep.c:470 kernel/locking/lockdep.c:5870) 
[ 17.421364][ T74] down_write (kernel/locking/rwsem.c:1315 kernel/locking/rwsem.c:1326 kernel/locking/rwsem.c:1591) 
[ 17.421735][ T74] dma_resv_lockdep (include/linux/fs.h:557 drivers/dma-buf/dma-resv.c:798) 
[ 17.422159][ T74] do_one_initcall (init/main.c:1269) 
[ 17.422573][ T74] do_initcalls (init/main.c:1330 init/main.c:1347) 
[ 17.423031][ T74] kernel_init_freeable (init/main.c:1583) 
[ 17.423481][ T74] kernel_init (init/main.c:1471) 
[ 17.423840][ T74] ret_from_fork (arch/x86/kernel/process.c:154) 
[ 17.424211][ T74] ret_from_fork_asm (arch/x86/entry/entry_64.S:258) 
[   17.424612][   T74]
[   17.424612][   T74] -> #0 (fs_reclaim){+.+.}-{0:0}:
[ 17.425162][ T74] check_noncircular (kernel/locking/lockdep.c:2179) 
[ 17.425561][ T74] check_prev_add (kernel/locking/lockdep.c:3166) 
[ 17.425954][ T74] validate_chain (kernel/locking/lockdep.c:3285 kernel/locking/lockdep.c:3908) 
[ 17.426681][ T74] __lock_acquire (kernel/locking/lockdep.c:5237) 
[ 17.427085][ T74] lock_acquire (kernel/locking/lockdep.c:470 kernel/locking/lockdep.c:5870) 
[ 17.427458][ T74] __fs_reclaim_acquire (mm/page_alloc.c:4235) 
[ 17.427840][ T74] prepare_alloc_pages+0xd5/0x23d 
[ 17.428255][ T74] __alloc_frozen_pages_noprof (mm/page_alloc.c:5137) 
[ 17.428658][ T74] alloc_slab_page (mm/slub.c:2990) 
[ 17.428987][ T74] allocate_slab (mm/slub.c:3155) 
[ 17.429316][ T74] ___slab_alloc (mm/slub.c:4406 (discriminator 3)) 
[ 17.429757][ T74] __kmem_cache_alloc_bulk (mm/slub.c:6757) 
[ 17.430188][ T74] refill_sheaf (mm/slub.c:2571) 
[ 17.430554][ T74] kmem_cache_return_sheaf (mm/slub.c:5174 (discriminator 1)) 
[ 17.430998][ T74] mas_destroy (lib/maple_tree.c:5384) 
[ 17.431366][ T74] mas_store_prealloc (lib/maple_tree.c:5292 (discriminator 4)) 
[ 17.431789][ T74] commit_merge (mm/vma.c:767) 
[ 17.432172][ T74] vma_expand (mm/vma.c:1158) 
[ 17.432530][ T74] vma_merge_new_range (mm/vma.c:1095 (discriminator 1)) 
[ 17.433237][ T74] __mmap_region (mm/vma.c:2664) 
[ 17.433621][ T74] mmap_region (mm/vma.c:2739) 
[ 17.433987][ T74] do_mmap (mm/mmap.c:559) 
[ 17.434342][ T74] vm_mmap_pgoff (mm/util.c:582) 
[ 17.434736][ T74] ksys_mmap_pgoff (mm/mmap.c:606) 
[ 17.435128][ T74] do_int80_emulation (arch/x86/entry/syscall_32.c:83 arch/x86/entry/syscall_32.c:172) 
[ 17.435557][ T74] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) 
[   17.436031][   T74]
[   17.436031][   T74] other info that might help us debug this:
[   17.436031][   T74]
[   17.436803][   T74]  Possible unsafe locking scenario:
[   17.436803][   T74]
[   17.437381][   T74]        CPU0                    CPU1
[   17.437800][   T74]        ----                    ----
[   17.438213][   T74]   lock(&mapping->i_mmap_rwsem);
[   17.438614][   T74]                                lock(fs_reclaim);
[   17.439118][   T74]                                lock(&mapping->i_mmap_rwsem);
[   17.439737][   T74]   lock(fs_reclaim);
[   17.440045][   T74]
[   17.440045][   T74]  *** DEADLOCK ***
[   17.440045][   T74]
[   17.440668][   T74] 2 locks held by systemd-cryptse/74:
[ 17.441085][ T74] #0: ffff88813b6a9d48 (&mm->mmap_lock){++++}-{4:4}, at: vm_mmap_pgoff (include/linux/mmap_lock.h:415 mm/util.c:578) 
[ 17.441777][ T74] #1: ffff88812925dbf8 (&mapping->i_mmap_rwsem){++++}-{4:4}, at: vma_prepare (mm/vma.c:296) 
[   17.442517][   T74]
[   17.442517][   T74] stack backtrace:
[   17.443019][   T74] CPU: 0 UID: 0 PID: 74 Comm: systemd-cryptse Not tainted 6.17.0-rc3-00017-g99848157f71c #1 PREEMPT(full)
[   17.443875][   T74] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
[   17.444619][   T74] Call Trace:
[   17.444877][   T74]  <TASK>
[ 17.445101][ T74] print_circular_bug (kernel/locking/lockdep.c:2046) 
[ 17.445469][ T74] check_noncircular (kernel/locking/lockdep.c:2179) 
[ 17.445823][ T74] ? add_chain_block (kernel/locking/lockdep.c:3442) 
[ 17.446245][ T74] check_prev_add (kernel/locking/lockdep.c:3166) 
[ 17.446608][ T74] validate_chain (kernel/locking/lockdep.c:3285 kernel/locking/lockdep.c:3908) 
[ 17.446960][ T74] __lock_acquire (kernel/locking/lockdep.c:5237) 
[ 17.447332][ T74] ? prepare_alloc_pages+0xd5/0x23d 
[ 17.447831][ T74] lock_acquire (kernel/locking/lockdep.c:470 kernel/locking/lockdep.c:5870) 
[ 17.448188][ T74] ? prepare_alloc_pages+0xd5/0x23d 
[ 17.448649][ T74] ? lookup_chain_cache (kernel/locking/lockdep.c:3801 (discriminator 1)) 
[ 17.449018][ T74] __fs_reclaim_acquire (mm/page_alloc.c:4235) 
[ 17.449456][ T74] ? prepare_alloc_pages+0xd5/0x23d 
[ 17.449923][ T74] prepare_alloc_pages+0xd5/0x23d 
[ 17.450400][ T74] __alloc_frozen_pages_noprof (mm/page_alloc.c:5137) 
[ 17.450855][ T74] ? __alloc_pages_slowpath+0xa4d/0xa4d 
[ 17.451376][ T74] ? lock_is_held_type (kernel/locking/lockdep.c:5601 kernel/locking/lockdep.c:5940) 
[ 17.451778][ T74] ? find_held_lock (kernel/locking/lockdep.c:5350) 
[ 17.452142][ T74] ? mark_lock (kernel/locking/lockdep.c:4731 (discriminator 3)) 
[ 17.452492][ T74] alloc_slab_page (mm/slub.c:2990) 
[ 17.452912][ T74] allocate_slab (mm/slub.c:3155) 
[ 17.453271][ T74] ___slab_alloc (mm/slub.c:4406 (discriminator 3)) 
[ 17.453601][ T74] ? lock_acquire (kernel/locking/lockdep.c:470 kernel/locking/lockdep.c:5870) 
[ 17.453949][ T74] ? refill_sheaf (mm/slub.c:2571) 
[ 17.454278][ T74] ? mark_lock (kernel/locking/lockdep.c:4731 (discriminator 3)) 
[ 17.454611][ T74] ? mark_held_locks (kernel/locking/lockdep.c:4325) 
[ 17.454987][ T74] __kmem_cache_alloc_bulk (mm/slub.c:6757) 
[ 17.455387][ T74] refill_sheaf (mm/slub.c:2571) 
[ 17.455731][ T74] kmem_cache_return_sheaf (mm/slub.c:5174 (discriminator 1)) 
[ 17.456221][ T74] mas_destroy (lib/maple_tree.c:5384) 
[ 17.456575][ T74] mas_store_prealloc (lib/maple_tree.c:5292 (discriminator 4)) 
[ 17.456968][ T74] ? mas_store (lib/maple_tree.c:5292) 
[ 17.457303][ T74] ? __mas_set_range (include/linux/maple_tree.h:755 (discriminator 20)) 
[ 17.457685][ T74] commit_merge (mm/vma.c:767) 
[ 17.458038][ T74] ? vma_complete (mm/vma.c:731) 
[ 17.458384][ T74] ? lock_is_held_type (kernel/locking/lockdep.c:5601 kernel/locking/lockdep.c:5940) 
[ 17.458757][ T74] vma_expand (mm/vma.c:1158) 
[ 17.459090][ T74] ? unmap_region (mm/vma.c:1121) 
[ 17.459519][ T74] ? can_vma_merge_right (mm/vma.c:197 mm/vma.c:440) 
[ 17.459890][ T74] vma_merge_new_range (mm/vma.c:1095 (discriminator 1)) 
[ 17.460253][ T74] __mmap_region (mm/vma.c:2664) 
[ 17.460604][ T74] ? __mmap_new_vma (mm/vma.c:2641) 
[ 17.460977][ T74] ? inc_stack_record_count (include/linux/refcount.h:290 include/linux/refcount.h:312 mm/page_owner.c:216) 
[ 17.461414][ T74] ? lock_is_held_type (kernel/locking/lockdep.c:5601 kernel/locking/lockdep.c:5940) 
[ 17.461808][ T74] ? mas_prev_slot (lib/maple_tree.c:4369) 
[ 17.462190][ T74] ? mas_prev_slot (lib/maple_tree.c:5664) 
[ 17.462586][ T74] ? lock_is_held_type (kernel/locking/lockdep.c:5601 kernel/locking/lockdep.c:5940) 
[ 17.463001][ T74] mmap_region (mm/vma.c:2739) 
[ 17.463353][ T74] ? generic_file_readonly_mmap (mm/filemap.c:3852) 
[ 17.463779][ T74] do_mmap (mm/mmap.c:559) 
[ 17.464082][ T74] ? __get_unmapped_area (mm/mmap.c:339) 
[ 17.464468][ T74] ? tracer_preempt_on (kernel/trace/trace_irqsoff.c:689) 
[ 17.464878][ T74] vm_mmap_pgoff (mm/util.c:582) 
[ 17.465244][ T74] ? randomize_page (mm/util.c:568) 
[ 17.465615][ T74] ksys_mmap_pgoff (mm/mmap.c:606) 
[ 17.466035][ T74] do_int80_emulation (arch/x86/entry/syscall_32.c:83 arch/x86/entry/syscall_32.c:172) 
[ 17.466429][ T74] asm_int80_emulation (arch/x86/include/asm/idtentry.h:626) 
[   17.466827][   T74] RIP: 0023:0xf7f95cb0
[ 17.467133][ T74] Code: 4c 24 34 89 44 24 0c 8b 44 24 44 8b 54 24 38 8b 74 24 3c 8b 7c 24 40 a9 ff 0f 00 00 75 1c c1 e8 0c 89 c5 b8 c0 00 00 00 cd 80 <3d> 00 f0 ff ff 77 21 83 c4 1c 5b 5e 5f 5d c3 90 83 c4 1c b8 ea ff
All code
========
   0:	4c 24 34             	rex.WR and $0x34,%al
   3:	89 44 24 0c          	mov    %eax,0xc(%rsp)
   7:	8b 44 24 44          	mov    0x44(%rsp),%eax
   b:	8b 54 24 38          	mov    0x38(%rsp),%edx
   f:	8b 74 24 3c          	mov    0x3c(%rsp),%esi
  13:	8b 7c 24 40          	mov    0x40(%rsp),%edi
  17:	a9 ff 0f 00 00       	test   $0xfff,%eax
  1c:	75 1c                	jne    0x3a
  1e:	c1 e8 0c             	shr    $0xc,%eax
  21:	89 c5                	mov    %eax,%ebp
  23:	b8 c0 00 00 00       	mov    $0xc0,%eax
  28:	cd 80                	int    $0x80
  2a:*	3d 00 f0 ff ff       	cmp    $0xfffff000,%eax		<-- trapping instruction
  2f:	77 21                	ja     0x52


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250904/202509041248.b7ece14a-lkp at intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



From andriy.shevchenko at linux.intel.com  Thu Sep  4 02:04:23 2025
From: andriy.shevchenko at linux.intel.com (Andy Shevchenko)
Date: Thu,  4 Sep 2025 11:04:23 +0200
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
Message-ID: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>

clang complains about unused function:

lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]

Fix this by removing unused code.

Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>
---
 lib/maple_tree.c | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index fce329d6c1b2..ca8b3c2e0362 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -176,11 +176,6 @@ static inline struct maple_node *mt_alloc_one(gfp_t gfp)
 	return kmem_cache_alloc(maple_node_cache, gfp);
 }
 
-static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
-{
-	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
-}
-
 static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 {
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
-- 
2.50.1



From david at redhat.com  Thu Sep  4 03:20:55 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 4 Sep 2025 12:20:55 +0200
Subject: [RFC PATCH 4/6] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-5-Liam.Howlett@oracle.com>
 <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
 <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
Message-ID: <a3777992-21fc-4e55-9a5e-72b17dc86135@redhat.com>

On 03.09.25 22:19, Liam R. Howlett wrote:
> * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 15:14]:
>> On Fri, Aug 15, 2025 at 03:10:29PM -0400, Liam R. Howlett wrote:
>>> The ceiling and tree search limit need to be different arguments for the
>>> future change in the failed fork attempt.
>>>
>>> No functional changes intended.
>>>
>>> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
>>
>> (Obv. in addition to comment about broken VMA tests :P)
>>
>> I guess intent is that if we discover any page tables beyond tree_max then
>> we ought to just wipe them all out so, in effect, we don't consider
>> mappings at or past tree_max to be valid?
> 
> Actually... there are some archs that map outside the vma and they are
> valid.. I think mips? and I think lower, but yeah.. it's needed.  This
> is why prev->vm_end and next->vm_start are used as page table limits,
> afaik.  This is a serious annoyance because it frequently adds walks
> that are infrequently necessary to the vma tree.

Hm, does that still exist?

I recall something odd ... was it that gate area thingy (in_gate_area) 
we also have to handle in GUP code? The is x86/arm though, not mips.

-- 
Cheers

David / dhildenb



From Liam.Howlett at oracle.com  Thu Sep  4 06:32:52 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 4 Sep 2025 09:32:52 -0400
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
In-Reply-To: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
References: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
Message-ID: <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>

* Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 05:04]:
> clang complains about unused function:
> 
> lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]
> 
> Fix this by removing unused code.
> 
> Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
> Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>

Thanks.  This should be squashed into the patch listed in the fixes tag
since it's not a stable commit yet.

> ---
>  lib/maple_tree.c | 5 -----
>  1 file changed, 5 deletions(-)
> 
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index fce329d6c1b2..ca8b3c2e0362 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -176,11 +176,6 @@ static inline struct maple_node *mt_alloc_one(gfp_t gfp)
>  	return kmem_cache_alloc(maple_node_cache, gfp);
>  }
>  
> -static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
> -{
> -	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
> -}
> -
>  static inline void mt_free_bulk(size_t size, void __rcu **nodes)
>  {
>  	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
> -- 
> 2.50.1
> 


From andriy.shevchenko at linux.intel.com  Thu Sep  4 06:37:27 2025
From: andriy.shevchenko at linux.intel.com (Andy Shevchenko)
Date: Thu, 4 Sep 2025 16:37:27 +0300
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
In-Reply-To: <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>
References: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
 <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>
Message-ID: <aLmWF3V8ILOkL8W2@smile.fi.intel.com>

On Thu, Sep 04, 2025 at 09:32:52AM -0400, Liam R. Howlett wrote:
> * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 05:04]:
> > clang complains about unused function:
> > 
> > lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]
> > 
> > Fix this by removing unused code.
> > 
> > Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
> > Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>
> 
> Thanks.  This should be squashed into the patch listed in the fixes tag
> since it's not a stable commit yet.

I fine with any solution, I just want this to be fixes rather sooner. So, please go ahead!

-- 
With Best Regards,
Andy Shevchenko




From Liam.Howlett at oracle.com  Thu Sep  4 06:42:53 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 4 Sep 2025 09:42:53 -0400
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
In-Reply-To: <aLmWF3V8ILOkL8W2@smile.fi.intel.com>
References: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
 <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>
 <aLmWF3V8ILOkL8W2@smile.fi.intel.com>
Message-ID: <2ts445ufkl7mhtdd3ucs2c6lsp7fxbx6b5tqmu2tdecxprok55@fdr2dwppnoks>

* Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 09:37]:
> On Thu, Sep 04, 2025 at 09:32:52AM -0400, Liam R. Howlett wrote:
> > * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 05:04]:
> > > clang complains about unused function:
> > > 
> > > lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]
> > > 
> > > Fix this by removing unused code.
> > > 
> > > Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
> > > Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>
> > 
> > Thanks.  This should be squashed into the patch listed in the fixes tag
> > since it's not a stable commit yet.
> 
> I fine with any solution, I just want this to be fixes rather sooner. So, please go ahead!

Sounds good.  I've added Vlastimil to the email as it went through his
branch.

Can you please Cc the SoB people next time?

Thanks,
Liam


From andriy.shevchenko at linux.intel.com  Thu Sep  4 06:54:17 2025
From: andriy.shevchenko at linux.intel.com (Andy Shevchenko)
Date: Thu, 4 Sep 2025 16:54:17 +0300
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
In-Reply-To: <2ts445ufkl7mhtdd3ucs2c6lsp7fxbx6b5tqmu2tdecxprok55@fdr2dwppnoks>
References: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
 <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>
 <aLmWF3V8ILOkL8W2@smile.fi.intel.com>
 <2ts445ufkl7mhtdd3ucs2c6lsp7fxbx6b5tqmu2tdecxprok55@fdr2dwppnoks>
Message-ID: <aLmaCY9TAbLSb45o@smile.fi.intel.com>

On Thu, Sep 04, 2025 at 09:42:53AM -0400, Liam R. Howlett wrote:
> * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 09:37]:
> > On Thu, Sep 04, 2025 at 09:32:52AM -0400, Liam R. Howlett wrote:
> > > * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 05:04]:
> > > > clang complains about unused function:
> > > > 
> > > > lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]
> > > > 
> > > > Fix this by removing unused code.
> > > > 
> > > > Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
> > > > Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>
> > > 
> > > Thanks.  This should be squashed into the patch listed in the fixes tag
> > > since it's not a stable commit yet.
> > 
> > I fine with any solution, I just want this to be fixes rather sooner. So, please go ahead!
> 
> Sounds good.  I've added Vlastimil to the email as it went through his
> branch.
> 
> Can you please Cc the SoB people next time?

I will try to remember. I used get_maintainers.pl. If it's not enough,
can we got this documented somehow? Or if it's already documented,
can you point out to it?

-- 
With Best Regards,
Andy Shevchenko




From vbabka at suse.cz  Thu Sep  4 08:12:08 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Thu, 4 Sep 2025 17:12:08 +0200
Subject: [PATCH v1 1/1] mapple_tree: Fix build error (`make W=1`)
In-Reply-To: <2ts445ufkl7mhtdd3ucs2c6lsp7fxbx6b5tqmu2tdecxprok55@fdr2dwppnoks>
References: <20250904090423.2293933-1-andriy.shevchenko@linux.intel.com>
 <6nk7wvp2vbsc5myxnx4mfohpq4yk4ixazkvif4hntilpuz7jtw@a2q2if4nc7kn>
 <aLmWF3V8ILOkL8W2@smile.fi.intel.com>
 <2ts445ufkl7mhtdd3ucs2c6lsp7fxbx6b5tqmu2tdecxprok55@fdr2dwppnoks>
Message-ID: <6614bc25-f514-4a05-b45a-037d951adc77@suse.cz>

On 9/4/25 15:42, Liam R. Howlett wrote:
> * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 09:37]:
>> On Thu, Sep 04, 2025 at 09:32:52AM -0400, Liam R. Howlett wrote:
>> > * Andy Shevchenko <andriy.shevchenko at linux.intel.com> [250904 05:04]:
>> > > clang complains about unused function:
>> > > 
>> > > lib/maple_tree.c:179:19: error: unused function 'mt_alloc_bulk' [-Werror,-Wunused-function]
>> > > 
>> > > Fix this by removing unused code.
>> > > 
>> > > Fixes: a48d52b2d21b ("maple_tree: Convert forking to use the sheaf interface")
>> > > Signed-off-by: Andy Shevchenko <andriy.shevchenko at linux.intel.com>
>> > 
>> > Thanks.  This should be squashed into the patch listed in the fixes tag
>> > since it's not a stable commit yet.
>> 
>> I fine with any solution, I just want this to be fixes rather sooner. So, please go ahead!
> 
> Sounds good.  I've added Vlastimil to the email as it went through his
> branch.

Squashing into the tomorrow's linux-next release, thanks.

> Can you please Cc the SoB people next time?
> 
> Thanks,
> Liam



From vbabka at suse.cz  Thu Sep  4 08:13:07 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Thu, 4 Sep 2025 17:13:07 +0200
Subject: [vbabka:slub-percpu-sheaves-v6r2] [maple_tree] 99848157f7:
 WARNING:possible_circular_locking_dependency_detected
In-Reply-To: <202509041248.b7ece14a-lkp@intel.com>
References: <202509041248.b7ece14a-lkp@intel.com>
Message-ID: <12b43696-d684-4502-aa48-6179702c8b11@suse.cz>

On 9/4/25 07:20, kernel test robot wrote:
> 
> 
> Hello,
> 
> kernel test robot noticed "WARNING:possible_circular_locking_dependency_detected" on:
> 
> commit: 99848157f71c03ca5e975b0e686def1d66e8b19f ("maple_tree: Sheaf conversion")
> https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-percpu-sheaves-v6r2
> 
> in testcase: rcuscale
> version: 
> with following parameters:
> 
> 	runtime: 300s
> 	scale_type: rcu
> 
> 
> 
> config: x86_64-randconfig-075-20250831
> compiler: gcc-12
> test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G
> 
> (please refer to attached dmesg/kmsg for entire log/backtrace)
> 
> 
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <oliver.sang at intel.com>
> | Closes: https://lore.kernel.org/oe-lkp/202509041248.b7ece14a-lkp at intel.com

Thanks, will be fixed in the next -next snapshot.



From lorenzo.stoakes at oracle.com  Thu Sep  4 08:21:28 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 4 Sep 2025 16:21:28 +0100
Subject: [RFC PATCH 2/6] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <ft4brzgr6jitmyfs4jplbokpqoz2z2ejsudfrezuke2oqygbs5@b5i3zaxgxrta>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-3-Liam.Howlett@oracle.com>
 <bdb27f25-d979-4dce-889e-a2fd17fc06a5@lucifer.local>
 <ft4brzgr6jitmyfs4jplbokpqoz2z2ejsudfrezuke2oqygbs5@b5i3zaxgxrta>
Message-ID: <2460fe1c-3e9a-465a-b164-12f92aabde00@lucifer.local>

On Wed, Sep 03, 2025 at 03:56:03PM -0400, Liam R. Howlett wrote:
> * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 14:38]:
> > On Fri, Aug 15, 2025 at 03:10:27PM -0400, Liam R. Howlett wrote:
> > > Create the new function tear_down_vmas() to remove a range of vmas.
> > > exit_mmap() will be removing all the vmas.
> > >
> > > This is necessary for future patches.
> > >
> > > No functional changes intended.
> > >
> > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> >
> > This function is pure and complete insanity, but this change looks
> > good. Couple nits below.
> >
> > Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> >
> > > ---
> > >  mm/mmap.c | 37 ++++++++++++++++++++++++-------------
> > >  1 file changed, 24 insertions(+), 13 deletions(-)
> > >
> > > diff --git a/mm/mmap.c b/mm/mmap.c
> > > index c4c315b480af7..0995a48b46d59 100644
> > > --- a/mm/mmap.c
> > > +++ b/mm/mmap.c
> > > @@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
> > >  }
> > >  EXPORT_SYMBOL(vm_brk_flags);
> > >
> > > +static inline
> > > +unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
> > > +		struct vm_area_struct *vma, unsigned long max)
> > > +{
> > > +	unsigned long nr_accounted = 0;
> > > +	int count = 0;
> > > +
> > > +	mmap_assert_write_locked(mm);
> > > +	vma_iter_set(vmi, vma->vm_end);
> > > +	do {
> > > +		if (vma->vm_flags & VM_ACCOUNT)
> > > +			nr_accounted += vma_pages(vma);
> > > +		vma_mark_detached(vma);
> > > +		remove_vma(vma);
> > > +		count++;
> > > +		cond_resched();
> > > +		vma = vma_next(vmi);
> > > +	} while (vma && vma->vm_end <= max);
> > > +
> > > +	BUG_ON(count != mm->map_count);
> >
> > Can we make this a WARN_ON() or WARN_ON_ONCE() while we're at it?
>
> Sure!

Thanks :)

>
> >
> > > +	return nr_accounted;
> > > +}
> > > +
> > >  /* Release all mmaps. */
> > >  void exit_mmap(struct mm_struct *mm)
> > >  {
> > > @@ -1257,7 +1280,6 @@ void exit_mmap(struct mm_struct *mm)
> > >  	struct vm_area_struct *vma;
> > >  	unsigned long nr_accounted = 0;
> >
> > No need to initialise this to 0 any more.
>
> There is a goto label below that skips calling the tear down, so this is
> still needed.

Ah yeah, sorry missed the goto destroy there. No worries then :)

>
> Thanks,
> Liam


From lorenzo.stoakes at oracle.com  Thu Sep  4 08:23:48 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 4 Sep 2025 16:23:48 +0100
Subject: [RFC PATCH 3/6] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <w43jgvlabgvxpq2jb3zv7zo64r4xbvmad7cf75l3keests7djf@lmqslvejyyk4>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-4-Liam.Howlett@oracle.com>
 <a32ea375-f093-4f8f-9270-fbe9f7977cc6@lucifer.local>
 <w43jgvlabgvxpq2jb3zv7zo64r4xbvmad7cf75l3keests7djf@lmqslvejyyk4>
Message-ID: <ed75cb72-82d1-4ade-99ca-abb55cc6ef35@lucifer.local>

On Wed, Sep 03, 2025 at 03:57:37PM -0400, Liam R. Howlett wrote:
> * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 14:48]:
> > On Fri, Aug 15, 2025 at 03:10:28PM -0400, Liam R. Howlett wrote:
> > > Add a limit to the vma search instead of using the start and end of the
> > > one passed in.
> > >
> > > No functional changes intended.
> > >
> > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > > ---
> > >  mm/vma.c | 6 ++++--
> > >  mm/vma.h | 1 +
> > >  2 files changed, 5 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/mm/vma.c b/mm/vma.c
> > > index 3b12c7579831b..fd270345c25d3 100644
> > > --- a/mm/vma.c
> > > +++ b/mm/vma.c
> > > @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
> > >   * Called with the mm semaphore held.
> > >   */
> > >  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> > > +		unsigned long vma_min, unsigned long vma_max,
> > >  		struct vm_area_struct *prev, struct vm_area_struct *next)
> >
> > Luckily we will be getting rid of this as soon as I complete the mmap_prepare +
> > friends series (_actively_ working on this right now).
> >
> > Anyway, this is getting quickly horrid, since this is only in vma.c, we could
> > make this static and pass through map to reduce some of the horror?
> >
> > We actually can hand it vmi (and thus &vmi->mas via map, so we could implement
> > it as:
> >
> > void unmap_region(struct mmap_state *map, struct vm_area_struct *vma,
> > 		  unsigned long min, unsigned long max);
> >
>
> Ah... I could do this on top of the last patch.. that might be more
> clear?

Ack, yeah I think so! Thanks! :)

>
> > >  {
> > >  	struct mm_struct *mm = vma->vm_mm;
> > > @@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> > >
> > >  	tlb_gather_mmu(&tlb, mm);
> > >  	update_hiwater_rss(mm);
> > > -	unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
> > > +	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
> > >  		   /* mm_wr_locked = */ true);
> > >  	mas_set(mas, vma->vm_end);
> > >  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> > > @@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
> > >
> > >  		vma_iter_set(vmi, vma->vm_end);
> > >  		/* Undo any partial mapping done by a device driver. */
> > > -		unmap_region(&vmi->mas, vma, map->prev, map->next);
> > > +		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> > > +			     map->prev, map->next);
> > >
> > >  		return error;
> > >  	}
> > > diff --git a/mm/vma.h b/mm/vma.h
> > > index b123a9cdedb0d..336dae295853e 100644
> > > --- a/mm/vma.h
> > > +++ b/mm/vma.h
> > > @@ -281,6 +281,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
> > >  void remove_vma(struct vm_area_struct *vma);
> > >
> > >  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> > > +		unsigned long min, unsigned long max,
> > >  		struct vm_area_struct *prev, struct vm_area_struct *next);
> > >
> > >  /* We are about to modify the VMA's flags. */
> > > --
> > > 2.47.2
> > >


From lorenzo.stoakes at oracle.com  Thu Sep  4 08:33:53 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 4 Sep 2025 16:33:53 +0100
Subject: [RFC PATCH 4/6] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-5-Liam.Howlett@oracle.com>
 <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
 <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
Message-ID: <a558f789-8c21-4e10-acc6-fdcea110563d@lucifer.local>

On Wed, Sep 03, 2025 at 04:19:04PM -0400, Liam R. Howlett wrote:
> * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 15:14]:
> > On Fri, Aug 15, 2025 at 03:10:29PM -0400, Liam R. Howlett wrote:
> > > The ceiling and tree search limit need to be different arguments for the
> > > future change in the failed fork attempt.
> > >
> > > No functional changes intended.
> > >
> > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> >
> > (Obv. in addition to comment about broken VMA tests :P)
> >
> > I guess intent is that if we discover any page tables beyond tree_max then
> > we ought to just wipe them all out so, in effect, we don't consider
> > mappings at or past tree_max to be valid?
>
> Actually... there are some archs that map outside the vma and they are
> valid.. I think mips? and I think lower, but yeah.. it's needed.  This
> is why prev->vm_end and next->vm_start are used as page table limits,
> afaik.  This is a serious annoyance because it frequently adds walks
> that are infrequently necessary to the vma tree.

ugh god. I was vaguely aware of this but that's gross. Very gross.

I need to document all the VMA weirdnesses smoewhere.

What do they do this for? Guard pages or something?

>
> >
> > I feel like we need a comment to this effect as this is confusing as it is.
> >
> > Could we add a kerneldoc comment for free_pgtables() spelling this out?
>
> I'll add a note here, but confusion will probably increase.  I'll add a
> note about the tree max as well.

Thanks!

>
> >
> > > ---
> > >  mm/internal.h | 4 +++-
> > >  mm/memory.c   | 7 ++++---
> > >  mm/mmap.c     | 2 +-
> > >  mm/vma.c      | 3 ++-
> > >  4 files changed, 10 insertions(+), 6 deletions(-)
> > >
> > > diff --git a/mm/internal.h b/mm/internal.h
> > > index 45b725c3dc030..f9a278ac76d83 100644
> > > --- a/mm/internal.h
> > > +++ b/mm/internal.h
> > > @@ -444,7 +444,9 @@ void folio_activate(struct folio *folio);
> > >
> > >  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> > >  		   struct vm_area_struct *start_vma, unsigned long floor,
> > > -		   unsigned long ceiling, bool mm_wr_locked);
> > > +		   unsigned long ceiling, unsigned long tree_max,
> > > +		   bool mm_wr_locked);
> > > +
> > >  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
> > >
> > >  struct zap_details;
> > > diff --git a/mm/memory.c b/mm/memory.c
> > > index 0ba4f6b718471..3346514562bba 100644
> > > --- a/mm/memory.c
> > > +++ b/mm/memory.c
> > > @@ -371,7 +371,8 @@ void free_pgd_range(struct mmu_gather *tlb,
> > >
> > >  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> > >  		   struct vm_area_struct *vma, unsigned long floor,
> > > -		   unsigned long ceiling, bool mm_wr_locked)
> > > +		   unsigned long ceiling, unsigned long tree_max,
> > > +		   bool mm_wr_locked)
> > >  {
> > >  	struct unlink_vma_file_batch vb;
> > >
> > > @@ -385,7 +386,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> > >  		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
> > >  		 * be 0.  This will underflow and is okay.
> > >  		 */
> > > -		next = mas_find(mas, ceiling - 1);
> > > +		next = mas_find(mas, tree_max - 1);
> >
> > Do we need to put some sort of sanity checks in to make sure tree_max <= ceiling
> > (though this 0 case is a pain... so I guess tree_max - 1 <= ceiling - 1?)
>
> Sure!

Thanks!


From lorenzo.stoakes at oracle.com  Thu Sep  4 08:40:59 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 4 Sep 2025 16:40:59 +0100
Subject: [RFC PATCH 6/6] mm: Change dup_mmap() recovery
In-Reply-To: <jr57bivqyzarzicmitzuz4unk5g6gytfnh5olmradn7nmei7yt@s35yr2zhknz5>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-7-Liam.Howlett@oracle.com>
 <de385044-e9a2-45a1-b74f-68ba80bba146@lucifer.local>
 <jr57bivqyzarzicmitzuz4unk5g6gytfnh5olmradn7nmei7yt@s35yr2zhknz5>
Message-ID: <6d59d785-9fe0-4ed0-b9a1-b9419d2f7ceb@lucifer.local>

On Wed, Sep 03, 2025 at 08:13:45PM -0400, Liam R. Howlett wrote:
> * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 16:34]:
> > On Fri, Aug 15, 2025 at 03:10:31PM -0400, Liam R. Howlett wrote:
> > > When the dup_mmap() fails during the vma duplication or setup, don't
> > > write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> > > free the new resources, leaving an empty vma tree.
> >
> > Yesss like it!
> >
> > >
> > > Using XA_ZERO introduced races where the vma could be found between
> > > dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> > > race can occur because the mm can be reached through the other trees
> > > via successfully copied vmas and other methods such as the swapoff code.
> >
> > Yeah god.
> >
> > >
> > > XA_ZERO was marking the location to stop vma removal and pagetable
> > > freeing.  The newly created arguments to the unmap_vmas() and
> > > free_pgtables() serve this function.
> >
> > Nice.
> >
> > >
> > > Replacing the XA_ZERO entry use with the new argument list also means
> > > the checks for xa_is_zero() are no longer necessary so these are also
> > > removed.
> > >
> > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> >
> > Ah finally the 'action' patch :)
> >
> > Obviously my review is on the basis that you fixup the rebase etc.
> >
> > This broadly looks right but some various nits etc. below and will have
> > another look through on new revision - this whole area is pretty crazy!
> >
> > > ---
> > >  mm/memory.c |  6 +-----
> > >  mm/mmap.c   | 39 ++++++++++++++++++++++++++++-----------
> > >  2 files changed, 29 insertions(+), 16 deletions(-)
> > >
> > > diff --git a/mm/memory.c b/mm/memory.c
> > > index 3346514562bba..8cd58f171bae4 100644
> > > --- a/mm/memory.c
> > > +++ b/mm/memory.c
> > > @@ -387,8 +387,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> > >  		 * be 0.  This will underflow and is okay.
> > >  		 */
> > >  		next = mas_find(mas, tree_max - 1);
> > > -		if (unlikely(xa_is_zero(next)))
> > > -			next = NULL;
> > >
> > >  		/*
> > >  		 * Hide vma from rmap and truncate_pagecache before freeing
> > > @@ -407,8 +405,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> > >  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
> > >  			vma = next;
> > >  			next = mas_find(mas, tree_max - 1);
> > > -			if (unlikely(xa_is_zero(next)))
> > > -				next = NULL;
> > >  			if (mm_wr_locked)
> > >  				vma_start_write(vma);
> > >  			unlink_anon_vmas(vma);
> > > @@ -1978,7 +1974,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> > >  				 mm_wr_locked);
> > >  		hugetlb_zap_end(vma, &details);
> > >  		vma = mas_find(mas, tree_end - 1);
> > > -	} while (vma && likely(!xa_is_zero(vma)));
> > > +	} while (vma);
> > >  	mmu_notifier_invalidate_range_end(&range);
> > >  }
> > >
> > > diff --git a/mm/mmap.c b/mm/mmap.c
> > > index eba2bc81bc749..5acc0b5f8c14a 100644
> > > --- a/mm/mmap.c
> > > +++ b/mm/mmap.c
> > > @@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
> > >  	arch_exit_mmap(mm);
> > >
> > >  	vma = vma_next(&vmi);
> > > -	if (!vma || unlikely(xa_is_zero(vma))) {
> > > +	if (!vma) {
> > >  		/* Can happen if dup_mmap() received an OOM */
> > >  		mmap_read_unlock(mm);
> > >  		mmap_write_lock(mm);
> > > @@ -1858,20 +1858,37 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
> > >  		ksm_fork(mm, oldmm);
> > >  		khugepaged_fork(mm, oldmm);
> > >  	} else {
> > > +		unsigned long max;
> > >
> > >  		/*
> > > -		 * The entire maple tree has already been duplicated. If the
> > > -		 * mmap duplication fails, mark the failure point with
> > > -		 * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
> > > -		 * stop releasing VMAs that have not been duplicated after this
> > > -		 * point.
> > > +		 * The entire maple tree has already been duplicated, but
> > > +		 * replacing the vmas failed at mpnt (which could be NULL if
> > > +		 * all were allocated but the last vma was not fully set up. Use
> >
> > Missing ')'.
>
> Thanks.
>
> >
> > > +		 * the start address of the failure point to clean up the half
> > > +		 * initialized tree.
> >
> > NIT: Is 'half' correct? Maybe 'partially'?
>
> Thanks.
>
> >
> > >  		 */
> > > -		if (mpnt) {
> > > -			mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
> > > -			mas_store(&vmi.mas, XA_ZERO_ENTRY);
> > > -			/* Avoid OOM iterating a broken tree */
> > > -			set_bit(MMF_OOM_SKIP, &mm->flags);
> > > +		if (!mm->map_count) {
> > > +			/* zero vmas were written to the new tree. */
> > > +			max = 0;
> >
> > OK I guess this then will result in the intentional underflow thing, maybe
> > worth mentioning?
>
> No, nothing will happen as the if (max) will evaluate as false.

Ahhh ok. This makes more sense :)

>
> >
> > > +		} else if (mpnt) {
> > > +			/* mid-tree failure */
> >
> > Partial?
>
> Right, thanks.
>
> >
> > > +			max = mpnt->vm_start;
> > > +		} else {
> > > +			/* All vmas were written to the new tree */
> > > +			max = ULONG_MAX;
> > >  		}
> > > +
> > > +		/* Hide mm from oom killer because the memory is being freed */
> > > +		set_bit(MMF_OOM_SKIP, &mm->flags);
> >
> > Obv. update to mm_flags_set(MMF_OOM_SKIP, mm);
>
> Right, happened on rebase.
>
> >
> > > +		if (max) {
> > > +			vma_iter_set(&vmi, 0);
> > > +			tmp = vma_next(&vmi);
> > > +			flush_cache_mm(mm);
> > > +			unmap_region(&vmi.mas, tmp, 0, max, max, NULL, NULL);
> >
> > (An aside for the unmap_region() impl, maybe let's name the pg_max as
> > tree_max to make it consistent across both?)
> >
> > Hm we could still use the mmap struct here if we put in vma.h. Just have to
> > set vmi, obv prev, next NULL.
> >
> > So:
> >
> > 	struct mmap_state map {
> > 		.vmi = &vmi,
> > 	}
> >
> > 	unmap_region(&map, tmp, 0, max, max);
> >
> > Possibly overkill and hopefully stack ok but makes other callers less
> > horrid.
> >
> > Maybe also good to add a comment spelling out what each of these params do.
>
> So I've tried to do this cleanly but it hasn't worked out.  I think I'll
> add another struct to clean this up in another patch on this series.

Moar helper structs! Yes yes yes! ;)

>
> Thanks,
> Liam

Cheers, Lorenzo


From lorenzo.stoakes at oracle.com  Thu Sep  4 08:36:17 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 4 Sep 2025 16:36:17 +0100
Subject: [RFC PATCH 4/6] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <a3777992-21fc-4e55-9a5e-72b17dc86135@redhat.com>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-5-Liam.Howlett@oracle.com>
 <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
 <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
 <a3777992-21fc-4e55-9a5e-72b17dc86135@redhat.com>
Message-ID: <6b94ec3b-5f03-4201-8e5b-bbf2aec242ff@lucifer.local>

On Thu, Sep 04, 2025 at 12:20:55PM +0200, David Hildenbrand wrote:
> On 03.09.25 22:19, Liam R. Howlett wrote:
> > * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 15:14]:
> > > On Fri, Aug 15, 2025 at 03:10:29PM -0400, Liam R. Howlett wrote:
> > > > The ceiling and tree search limit need to be different arguments for the
> > > > future change in the failed fork attempt.
> > > >
> > > > No functional changes intended.
> > > >
> > > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > >
> > > (Obv. in addition to comment about broken VMA tests :P)
> > >
> > > I guess intent is that if we discover any page tables beyond tree_max then
> > > we ought to just wipe them all out so, in effect, we don't consider
> > > mappings at or past tree_max to be valid?
> >
> > Actually... there are some archs that map outside the vma and they are
> > valid.. I think mips? and I think lower, but yeah.. it's needed.  This
> > is why prev->vm_end and next->vm_start are used as page table limits,
> > afaik.  This is a serious annoyance because it frequently adds walks
> > that are infrequently necessary to the vma tree.
>
> Hm, does that still exist?
>
> I recall something odd ... was it that gate area thingy (in_gate_area) we
> also have to handle in GUP code? The is x86/arm though, not mips.

Isn't gate area the VSYSCALL area so that's basically a kernel mapping address
that we allow userland to access?

>
> --
> Cheers
>
> David / dhildenb
>


From vbabka at suse.cz  Mon Sep  8 00:55:45 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 8 Sep 2025 09:55:45 +0200
Subject: [PATCH v7 00/21] SLUB percpu sheaves
In-Reply-To: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
Message-ID: <fbfc21ab-c947-422d-94c6-0db897ee25f3@suse.cz>

On 9/3/25 14:59, Vlastimil Babka wrote:
> Hi,
> 
> I'm sending v7 as there were some non-trivial fixes needed since v6. I'm
> including back the maple tree part that I sent separately the last time,
> as some reordering between both parts was necessary to avoid
> intermittently breaking the userspace vma tests. See the changelog at
> the end of the cover letter for details and links.
> The v7 will replace+extend the v6 in slab/for-next.

Liam made further changes/cleanups to the maple tree side, mainly the commit

76fcafc3894f ("maple_tree: Drop bulk insert support")

but it affects some later patches too due to some conflicts

for now the result is here: (and in slab/for-next)

https://web.git.kernel.org/pub/scm/linux/kernel/git/vbabka/linux.git/log/?h=slub-percpu-sheaves-v8r1

I don't want to spam with new vX too often, so will send in 2 days but can
can do earlier if there's interest.


From harry.yoo at oracle.com  Mon Sep  8 04:19:52 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Mon, 8 Sep 2025 20:19:52 +0900
Subject: [PATCH v7 03/21] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <20250903-slub-percpu-caches-v7-3-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-3-71c114cdefef@suse.cz>
Message-ID: <aL672Jeqi99atefN@hyeyoo>

On Wed, Sep 03, 2025 at 02:59:45PM +0200, Vlastimil Babka wrote:
> Specifying a non-zero value for a new struct kmem_cache_args field
> sheaf_capacity will setup a caching layer of percpu arrays called
> sheaves of given capacity for the created cache.
> 
> Allocations from the cache will allocate via the percpu sheaves (main or
> spare) as long as they have no NUMA node preference. Frees will also
> put the object back into one of the sheaves.
> 
> When both percpu sheaves are found empty during an allocation, an empty
> sheaf may be replaced with a full one from the per-node barn. If none
> are available and the allocation is allowed to block, an empty sheaf is
> refilled from slab(s) by an internal bulk alloc operation. When both
> percpu sheaves are full during freeing, the barn can replace a full one
> with an empty one, unless over a full sheaves limit. In that case a
> sheaf is flushed to slab(s) by an internal bulk free operation. Flushing
> sheaves and barns is also wired to the existing cpu flushing and cache
> shrinking operations.
> 
> The sheaves do not distinguish NUMA locality of the cached objects. If
> an allocation is requested with kmem_cache_alloc_node() (or a mempolicy
> with strict_numa mode enabled) with a specific node (not NUMA_NO_NODE),
> the sheaves are bypassed.
> 
> The bulk operations exposed to slab users also try to utilize the
> sheaves as long as the necessary (full or empty) sheaves are available
> on the cpu or in the barn. Once depleted, they will fallback to bulk
> alloc/free to slabs directly to avoid double copying.
> 
> The sheaf_capacity value is exported in sysfs for observability.
> 
> Sysfs CONFIG_SLUB_STATS counters alloc_cpu_sheaf and free_cpu_sheaf
> count objects allocated or freed using the sheaves (and thus not
> counting towards the other alloc/free path counters). Counters
> sheaf_refill and sheaf_flush count objects filled or flushed from or to
> slab pages, and can be used to assess how effective the caching is. The
> refill and flush operations will also count towards the usual
> alloc_fastpath/slowpath, free_fastpath/slowpath and other counters for
> the backing slabs.  For barn operations, barn_get and barn_put count how
> many full sheaves were get from or put to the barn, the _fail variants
> count how many such requests could not be satisfied mainly  because the
> barn was either empty or full. While the barn also holds empty sheaves
> to make some operations easier, these are not as critical to mandate own
> counters.  Finally, there are sheaf_alloc/sheaf_free counters.
> 
> Access to the percpu sheaves is protected by local_trylock() when
> potential callers include irq context, and local_lock() otherwise (such
> as when we already know the gfp flags allow blocking). The trylock
> failures should be rare and we can easily fallback. Each per-NUMA-node
> barn has a spin_lock.
> 
> When slub_debug is enabled for a cache with sheaf_capacity also
> specified, the latter is ignored so that allocations and frees reach the
> slow path where debugging hooks are processed. Similarly, we ignore it
> with CONFIG_SLUB_TINY which prefers low memory usage to performance.
> 
> [boot failure: https://lore.kernel.org/all/583eacf5-c971-451a-9f76-fed0e341b815 at linux.ibm.com/ ]
> Reported-and-tested-by: Venkat Rao Bagalkote <venkat88 at linux.ibm.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  include/linux/slab.h |   31 ++
>  mm/slab.h            |    2 +
>  mm/slab_common.c     |    5 +-
>  mm/slub.c            | 1164 +++++++++++++++++++++++++++++++++++++++++++++++---
>  4 files changed, 1143 insertions(+), 59 deletions(-)
> 
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index bfe7c40eeee1a01c175766935c1e3c0304434a53..e2b197e47866c30acdbd1fee4159f262a751c5a7 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -163,6 +163,9 @@ int slab_unmergeable(struct kmem_cache *s)
>  		return 1;
>  #endif
>  
> +	if (s->cpu_sheaves)
> +		return 1;
> +
>  	/*
>  	 * We may have set a slab to be unmergeable during bootstrap.
>  	 */
> @@ -321,7 +324,7 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
>  		    object_size - args->usersize < args->useroffset))
>  		args->usersize = args->useroffset = 0;
>  
> -	if (!args->usersize)
> +	if (!args->usersize && !args->sheaf_capacity)
>  		s = __kmem_cache_alias(name, object_size, args->align, flags,
>  				       args->ctor);

Can we merge caches that use sheaves in the future if the capacity
is the same, or are there any restrictions for merging that I overlooked?

>  /*
>   * Slab allocation and freeing
>   */
> @@ -3344,11 +3748,42 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu)
>  	put_partials_cpu(s, c);
>  }
>  
> -struct slub_flush_work {
> -	struct work_struct work;
> -	struct kmem_cache *s;
> -	bool skip;
> -};
> +static inline void flush_this_cpu_slab(struct kmem_cache *s)
> +{
> +	struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab);
> +
> +	if (c->slab)
> +		flush_slab(s, c);
> +
> +	put_partials(s);
> +}
> +
> +static bool has_cpu_slab(int cpu, struct kmem_cache *s)
> +{
> +	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
> +
> +	return c->slab || slub_percpu_partial(c);
> +}
> +
> +#else /* CONFIG_SLUB_TINY */
> +static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
> +static inline bool has_cpu_slab(int cpu, struct kmem_cache *s) { return false; }
> +static inline void flush_this_cpu_slab(struct kmem_cache *s) { }
> +#endif /* CONFIG_SLUB_TINY */
> +
> +static bool has_pcs_used(int cpu, struct kmem_cache *s)
> +{
> +	struct slub_percpu_sheaves *pcs;
> +
> +	if (!s->cpu_sheaves)
> +		return false;
> +
> +	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
> +
> +	return (pcs->spare || pcs->main->size);
> +}
> +
> +static void pcs_flush_all(struct kmem_cache *s);

nit: we don't need these functions to flush sheaves if SLUB_TINY=y
as we don't create sheaves for SLUB_TINY anymore?

>  /*
>   * Flush cpu slab.
> @@ -3358,30 +3793,18 @@ struct slub_flush_work {
>  static void flush_cpu_slab(struct work_struct *w)
>  {
>  	struct kmem_cache *s;
> -	struct kmem_cache_cpu *c;
>  	struct slub_flush_work *sfw;
>  
>  	sfw = container_of(w, struct slub_flush_work, work);
>  
>  	s = sfw->s;
> -	c = this_cpu_ptr(s->cpu_slab);
> -
> -	if (c->slab)
> -		flush_slab(s, c);
> -
> -	put_partials(s);
> -}
>  
> -static bool has_cpu_slab(int cpu, struct kmem_cache *s)
> -{
> -	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
> +	if (s->cpu_sheaves)
> +		pcs_flush_all(s);
>  
> -	return c->slab || slub_percpu_partial(c);
> +	flush_this_cpu_slab(s);
>  } 
> -#else /* CONFIG_SLUB_TINY */
> -static inline void flush_all_cpus_locked(struct kmem_cache *s) { }
> -static inline void flush_all(struct kmem_cache *s) { }
> -static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
> -static inline int slub_cpu_dead(unsigned int cpu) { return 0; }
> -#endif /* CONFIG_SLUB_TINY */
> -
>  /*
>   * Check if the objects in a per cpu structure fit numa
>   * locality expectations.
> @@ -4191,30 +4610,240 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>  }
>  
>  /*
> - * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
> - * have the fastpath folded into their functions. So no function call
> - * overhead for requests that can be satisfied on the fastpath.
> - *
> - * The fastpath works by first checking if the lockless freelist can be used.
> - * If not then __slab_alloc is called for slow processing.
> + * Replace the empty main sheaf with a (at least partially) full sheaf.
>   *
> - * Otherwise we can simply pick the next object from the lockless free list.
> + * Must be called with the cpu_sheaves local lock locked. If successful, returns
> + * the pcs pointer and the local lock locked (possibly on a different cpu than
> + * initially called). If not successful, returns NULL and the local lock
> + * unlocked.
>   */
> -static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
> -		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
> +static struct slub_percpu_sheaves *
> +__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
>  {
> -	void *object;
> -	bool init = false;
> +	struct slab_sheaf *empty = NULL;
> +	struct slab_sheaf *full;
> +	struct node_barn *barn;
> +	bool can_alloc;
>  
> -	s = slab_pre_alloc_hook(s, gfpflags);
> -	if (unlikely(!s))
> +	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
> +
> +	if (pcs->spare && pcs->spare->size > 0) {
> +		swap(pcs->main, pcs->spare);
> +		return pcs;
> +	}
> +
> +	barn = get_barn(s);
> +
> +	full = barn_replace_empty_sheaf(barn, pcs->main);
> +
> +	if (full) {
> +		stat(s, BARN_GET);
> +		pcs->main = full;
> +		return pcs;
> +	}
> +
> +	stat(s, BARN_GET_FAIL);
> +
> +	can_alloc = gfpflags_allow_blocking(gfp);
> +
> +	if (can_alloc) {
> +		if (pcs->spare) {
> +			empty = pcs->spare;
> +			pcs->spare = NULL;
> +		} else {
> +			empty = barn_get_empty_sheaf(barn);
> +		}
> +	}
> +
> +	local_unlock(&s->cpu_sheaves->lock);
> +
> +	if (!can_alloc)
> +		return NULL;
> +
> +	if (empty) {
> +		if (!refill_sheaf(s, empty, gfp)) {
> +			full = empty;
> +		} else {
> +			/*
> +			 * we must be very low on memory so don't bother
> +			 * with the barn
> +			 */
> +			free_empty_sheaf(s, empty);
> +		}
> +	} else {
> +		full = alloc_full_sheaf(s, gfp);
> +	}
> +
> +	if (!full)
> +		return NULL;
> +
> +	/*
> +	 * we can reach here only when gfpflags_allow_blocking
> +	 * so this must not be an irq
> +	 */
> +	local_lock(&s->cpu_sheaves->lock);
> +	pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +	/*
> +	 * If we are returning empty sheaf, we either got it from the
> +	 * barn or had to allocate one. If we are returning a full
> +	 * sheaf, it's due to racing or being migrated to a different
> +	 * cpu. Breaching the barn's sheaf limits should be thus rare
> +	 * enough so just ignore them to simplify the recovery.
> +	 */
> +
> +	if (pcs->main->size == 0) {
> +		barn_put_empty_sheaf(barn, pcs->main);

It should be very rare but it should do
barn = get_barn(s); again after taking s->cpu_sheaves->lock?

> +		pcs->main = full;
> +		return pcs;
> +	}
> +
> +	if (!pcs->spare) {
> +		pcs->spare = full;
> +		return pcs;
> +	}
> +
> +	if (pcs->spare->size == 0) {
> +		barn_put_empty_sheaf(barn, pcs->spare);
> +		pcs->spare = full;
> +		return pcs;
> +	}
> +
> +	barn_put_full_sheaf(barn, full);
> +	stat(s, BARN_PUT);
> +
> +	return pcs;
> +}
> @@ -4591,6 +5220,295 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab,
>  	discard_slab(s, slab);
>  }
>  
> +/*
> + * Replace the full main sheaf with a (at least partially) empty sheaf.
> + *
> + * Must be called with the cpu_sheaves local lock locked. If successful, returns
> + * the pcs pointer and the local lock locked (possibly on a different cpu than
> + * initially called). If not successful, returns NULL and the local lock
> + * unlocked.
> + */
> +static struct slub_percpu_sheaves *
> +__pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs)
> +{
> +	struct slab_sheaf *empty;
> +	struct node_barn *barn;
> +	bool put_fail;
> +
> +restart:
> +	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
> +
> +	barn = get_barn(s);
> +	put_fail = false;
> +
> +	if (!pcs->spare) {
> +		empty = barn_get_empty_sheaf(barn);
> +		if (empty) {
> +			pcs->spare = pcs->main;
> +			pcs->main = empty;
> +			return pcs;
> +		}
> +		goto alloc_empty;
> +	}
> +
> +	if (pcs->spare->size < s->sheaf_capacity) {
> +		swap(pcs->main, pcs->spare);
> +		return pcs;
> +	}
> +
> +	empty = barn_replace_full_sheaf(barn, pcs->main);
> +
> +	if (!IS_ERR(empty)) {
> +		stat(s, BARN_PUT);
> +		pcs->main = empty;
> +		return pcs;
> +	}
> +
> +	if (PTR_ERR(empty) == -E2BIG) {
> +		/* Since we got here, spare exists and is full */
> +		struct slab_sheaf *to_flush = pcs->spare;
> +
> +		stat(s, BARN_PUT_FAIL);
> +
> +		pcs->spare = NULL;
> +		local_unlock(&s->cpu_sheaves->lock);
> +
> +		sheaf_flush_unused(s, to_flush);
> +		empty = to_flush;
> +		goto got_empty;
> +	}
> +
> +	/*
> +	 * We could not replace full sheaf because barn had no empty
> +	 * sheaves. We can still allocate it and put the full sheaf in
> +	 * __pcs_install_empty_sheaf(), but if we fail to allocate it,
> +	 * make sure to count the fail.
> +	 */
> +	put_fail = true;
> +
> +alloc_empty:
> +	local_unlock(&s->cpu_sheaves->lock);
> +
> +	empty = alloc_empty_sheaf(s, GFP_NOWAIT);
> +	if (empty)
> +		goto got_empty;
> +
> +	if (put_fail)
> +		 stat(s, BARN_PUT_FAIL);
> +
> +	if (!sheaf_flush_main(s))
> +		return NULL;
> +
> +	if (!local_trylock(&s->cpu_sheaves->lock))
> +		return NULL;
> +
> +	pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +	/*
> +	 * we flushed the main sheaf so it should be empty now,
> +	 * but in case we got preempted or migrated, we need to
> +	 * check again
> +	 */
> +	if (pcs->main->size == s->sheaf_capacity)
> +		goto restart;
> +
> +	return pcs;
> +
> +got_empty:
> +	if (!local_trylock(&s->cpu_sheaves->lock)) {
> +		barn_put_empty_sheaf(barn, empty);

Same here, we might have gotten migrated to a different node.

> +		return NULL;
> +	}
> +
> +	pcs = this_cpu_ptr(s->cpu_sheaves);
> +	__pcs_install_empty_sheaf(s, pcs, empty);
> +
> +	return pcs;
> +}

Otherwise looks good to me!

-- 
Cheers,
Harry / Hyeonggon


From urezki at gmail.com  Mon Sep  8 04:59:18 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Mon, 8 Sep 2025 13:59:18 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
Message-ID: <aL7FFpIMmXtzzSL1@pc638.lan>

On Wed, Sep 03, 2025 at 02:59:46PM +0200, Vlastimil Babka wrote:
> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> addition to main and spare sheaves.
> 
> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> the sheaf is detached and submitted to call_rcu() with a handler that
> will try to put it in the barn, or flush to slab pages using bulk free,
> when the barn is full. Then a new empty sheaf must be obtained to put
> more objects there.
> 
> It's possible that no free sheaves are available to use for a new
> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> kfree_rcu() implementation.
> 
> Expected advantages:
> - batching the kfree_rcu() operations, that could eventually replace the
>   existing batching
> - sheaves can be reused for allocations via barn instead of being
>   flushed to slabs, which is more efficient
>   - this includes cases where only some cpus are allowed to process rcu
>     callbacks (Android)
> 
> Possible disadvantage:
> - objects might be waiting for more than their grace period (it is
>   determined by the last object freed into the sheaf), increasing memory
>   usage - but the existing batching does that too.
> 
> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> implementation favors smaller memory footprint over performance.
> 
> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> many had to fall back to the existing implementation.
> 
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  mm/slab.h        |   2 +
>  mm/slab_common.c |  24 +++++++
>  mm/slub.c        | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 216 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/slab.h b/mm/slab.h
> index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..f1866f2d9b211bb0d7f24644b80ef4b50a7c3d24 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -435,6 +435,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
>  	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
>  }
>  
> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> +
>  #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
>  			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
>  			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index e2b197e47866c30acdbd1fee4159f262a751c5a7..2d806e02568532a1000fd3912db6978e945dcfa8 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
>  		kvfree_rcu_list(head);
>  }
>  
> +static bool kfree_rcu_sheaf(void *obj)
> +{
> +	struct kmem_cache *s;
> +	struct folio *folio;
> +	struct slab *slab;
> +
> +	if (is_vmalloc_addr(obj))
> +		return false;
> +
> +	folio = virt_to_folio(obj);
> +	if (unlikely(!folio_test_slab(folio)))
> +		return false;
> +
> +	slab = folio_slab(folio);
> +	s = slab->slab_cache;
> +	if (s->cpu_sheaves)
> +		return __kfree_rcu_sheaf(s, obj);
> +
> +	return false;
> +}
> +
>  static bool
>  need_offload_krc(struct kfree_rcu_cpu *krcp)
>  {
> @@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
>  	if (!head)
>  		might_sleep();
>  
> +	if (kfree_rcu_sheaf(ptr))
> +		return;
> +
Uh.. I have some concerns about this.

This patch introduces a new path which is a collision to the
existing kvfree_rcu() logic. It implements some batching which
we already have.

- kvfree_rcu_barrier() does not know about "sheaf" path. Am i missing
  something? How do you guarantee that kvfree_rcu_barrier() flushes
  sheafs? If it is part of kvfree_rcu() it has to care about this.

- we do not allocate in kvfree_rcu() path because of PREEMMPT_RT, i.e.
  kvfree_rcu() is supposed it can be called from the non-sleeping contexts.
- call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().

IMO, it is worth to reuse existing logic in the kvfree_rcu(). I can help
with it when i have more cycles as part of my RCU work.

--
Uladzislau Rezki


From vbabka at suse.cz  Mon Sep  8 05:26:11 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 8 Sep 2025 14:26:11 +0200
Subject: [PATCH v7 03/21] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <aL672Jeqi99atefN@hyeyoo>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-3-71c114cdefef@suse.cz>
 <aL672Jeqi99atefN@hyeyoo>
Message-ID: <be0ba18e-f752-4df3-a224-fc978431236c@suse.cz>

On 9/8/25 13:19, Harry Yoo wrote:
> On Wed, Sep 03, 2025 at 02:59:45PM +0200, Vlastimil Babka wrote:
>> diff --git a/mm/slab_common.c b/mm/slab_common.c
>> index bfe7c40eeee1a01c175766935c1e3c0304434a53..e2b197e47866c30acdbd1fee4159f262a751c5a7 100644
>> --- a/mm/slab_common.c
>> +++ b/mm/slab_common.c
>> @@ -163,6 +163,9 @@ int slab_unmergeable(struct kmem_cache *s)
>>  		return 1;
>>  #endif
>>  
>> +	if (s->cpu_sheaves)
>> +		return 1;
>> +
>>  	/*
>>  	 * We may have set a slab to be unmergeable during bootstrap.
>>  	 */
>> @@ -321,7 +324,7 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
>>  		    object_size - args->usersize < args->useroffset))
>>  		args->usersize = args->useroffset = 0;
>>  
>> -	if (!args->usersize)
>> +	if (!args->usersize && !args->sheaf_capacity)
>>  		s = __kmem_cache_alias(name, object_size, args->align, flags,
>>  				       args->ctor);
> 
> Can we merge caches that use sheaves in the future if the capacity
> is the same, or are there any restrictions for merging that I overlooked?

I think we will be able to merge. It will make more sense if we get to
enabling sheaves for everything and set capacity automatically, with the
args->sheaf_capacity serving only as a lower bound. Right now we can leave
it as is I think.
>> +static bool has_pcs_used(int cpu, struct kmem_cache *s)
>> +{
>> +	struct slub_percpu_sheaves *pcs;
>> +
>> +	if (!s->cpu_sheaves)
>> +		return false;
>> +
>> +	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
>> +
>> +	return (pcs->spare || pcs->main->size);
>> +}
>> +
>> +static void pcs_flush_all(struct kmem_cache *s);
> 
> nit: we don't need these functions to flush sheaves if SLUB_TINY=y
> as we don't create sheaves for SLUB_TINY anymore?

I'll check but also if this is true I would expect to have a clang warning
report already. Maybe they are called but don't need to?
TBH we should get rid of SLUB_TINY at this point, maybe have only a boot
option that disables percpu caching and keep the code compiled.
>>  /*
>>   * Flush cpu slab.
>> @@ -3358,30 +3793,18 @@ struct slub_flush_work {
>>  static void flush_cpu_slab(struct work_struct *w)
>>  {
>>  	struct kmem_cache *s;
>> -	struct kmem_cache_cpu *c;
>>  	struct slub_flush_work *sfw;
>>  
>>  	sfw = container_of(w, struct slub_flush_work, work);
>>  
>>  	s = sfw->s;
>> -	c = this_cpu_ptr(s->cpu_slab);
>> -
>> -	if (c->slab)
>> -		flush_slab(s, c);
>> -
>> -	put_partials(s);
>> -}
>>  
>> -static bool has_cpu_slab(int cpu, struct kmem_cache *s)
>> -{
>> -	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
>> +	if (s->cpu_sheaves)
>> +		pcs_flush_all(s);
>>  
>> -	return c->slab || slub_percpu_partial(c);
>> +	flush_this_cpu_slab(s);
>>  } 
>> -#else /* CONFIG_SLUB_TINY */
>> -static inline void flush_all_cpus_locked(struct kmem_cache *s) { }
>> -static inline void flush_all(struct kmem_cache *s) { }
>> -static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
>> -static inline int slub_cpu_dead(unsigned int cpu) { return 0; }
>> -#endif /* CONFIG_SLUB_TINY */
>> -
>>  /*
>>   * Check if the objects in a per cpu structure fit numa
>>   * locality expectations.
>> @@ -4191,30 +4610,240 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
>>  }
>>  
>>  /*
>> - * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
>> - * have the fastpath folded into their functions. So no function call
>> - * overhead for requests that can be satisfied on the fastpath.
>> - *
>> - * The fastpath works by first checking if the lockless freelist can be used.
>> - * If not then __slab_alloc is called for slow processing.
>> + * Replace the empty main sheaf with a (at least partially) full sheaf.
>>   *
>> - * Otherwise we can simply pick the next object from the lockless free list.
>> + * Must be called with the cpu_sheaves local lock locked. If successful, returns
>> + * the pcs pointer and the local lock locked (possibly on a different cpu than
>> + * initially called). If not successful, returns NULL and the local lock
>> + * unlocked.
>>   */
>> -static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
>> -		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
>> +static struct slub_percpu_sheaves *
>> +__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
>>  {
>> -	void *object;
>> -	bool init = false;
>> +	struct slab_sheaf *empty = NULL;
>> +	struct slab_sheaf *full;
>> +	struct node_barn *barn;
>> +	bool can_alloc;
>>  
>> -	s = slab_pre_alloc_hook(s, gfpflags);
>> -	if (unlikely(!s))
>> +	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
>> +
>> +	if (pcs->spare && pcs->spare->size > 0) {
>> +		swap(pcs->main, pcs->spare);
>> +		return pcs;
>> +	}
>> +
>> +	barn = get_barn(s);
>> +
>> +	full = barn_replace_empty_sheaf(barn, pcs->main);
>> +
>> +	if (full) {
>> +		stat(s, BARN_GET);
>> +		pcs->main = full;
>> +		return pcs;
>> +	}
>> +
>> +	stat(s, BARN_GET_FAIL);
>> +
>> +	can_alloc = gfpflags_allow_blocking(gfp);
>> +
>> +	if (can_alloc) {
>> +		if (pcs->spare) {
>> +			empty = pcs->spare;
>> +			pcs->spare = NULL;
>> +		} else {
>> +			empty = barn_get_empty_sheaf(barn);
>> +		}
>> +	}
>> +
>> +	local_unlock(&s->cpu_sheaves->lock);
>> +
>> +	if (!can_alloc)
>> +		return NULL;
>> +
>> +	if (empty) {
>> +		if (!refill_sheaf(s, empty, gfp)) {
>> +			full = empty;
>> +		} else {
>> +			/*
>> +			 * we must be very low on memory so don't bother
>> +			 * with the barn
>> +			 */
>> +			free_empty_sheaf(s, empty);
>> +		}
>> +	} else {
>> +		full = alloc_full_sheaf(s, gfp);
>> +	}
>> +
>> +	if (!full)
>> +		return NULL;
>> +
>> +	/*
>> +	 * we can reach here only when gfpflags_allow_blocking
>> +	 * so this must not be an irq
>> +	 */
>> +	local_lock(&s->cpu_sheaves->lock);
>> +	pcs = this_cpu_ptr(s->cpu_sheaves);
>> +
>> +	/*
>> +	 * If we are returning empty sheaf, we either got it from the
>> +	 * barn or had to allocate one. If we are returning a full
>> +	 * sheaf, it's due to racing or being migrated to a different
>> +	 * cpu. Breaching the barn's sheaf limits should be thus rare
>> +	 * enough so just ignore them to simplify the recovery.
>> +	 */
>> +
>> +	if (pcs->main->size == 0) {
>> +		barn_put_empty_sheaf(barn, pcs->main);
> 
> It should be very rare but it should do
> barn = get_barn(s); again after taking s->cpu_sheaves->lock?

Yeah but it shouldn't really matter which node we use for the empty sheaf
return. And in case of the full sheaf below it's actually better for
preserving numa locality to use the original barn.


From vbabka at suse.cz  Mon Sep  8 05:45:11 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 8 Sep 2025 14:45:11 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aL7FFpIMmXtzzSL1@pc638.lan>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan>
Message-ID: <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>

On 9/8/25 13:59, Uladzislau Rezki wrote:
> On Wed, Sep 03, 2025 at 02:59:46PM +0200, Vlastimil Babka wrote:
>> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
>> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
>> addition to main and spare sheaves.
>> 
>> kfree_rcu() operations will try to put objects on this sheaf. Once full,
>> the sheaf is detached and submitted to call_rcu() with a handler that
>> will try to put it in the barn, or flush to slab pages using bulk free,
>> when the barn is full. Then a new empty sheaf must be obtained to put
>> more objects there.
>> 
>> It's possible that no free sheaves are available to use for a new
>> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
>> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
>> kfree_rcu() implementation.
>> 
>> Expected advantages:
>> - batching the kfree_rcu() operations, that could eventually replace the
>>   existing batching
>> - sheaves can be reused for allocations via barn instead of being
>>   flushed to slabs, which is more efficient
>>   - this includes cases where only some cpus are allowed to process rcu
>>     callbacks (Android)
>> 
>> Possible disadvantage:
>> - objects might be waiting for more than their grace period (it is
>>   determined by the last object freed into the sheaf), increasing memory
>>   usage - but the existing batching does that too.
>> 
>> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
>> implementation favors smaller memory footprint over performance.
>> 
>> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
>> count how many kfree_rcu() used the rcu_free sheaf successfully and how
>> many had to fall back to the existing implementation.
>> 
>> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
>> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
>> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
>> ---
>>  mm/slab.h        |   2 +
>>  mm/slab_common.c |  24 +++++++
>>  mm/slub.c        | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>  3 files changed, 216 insertions(+), 2 deletions(-)
>> 
>> diff --git a/mm/slab.h b/mm/slab.h
>> index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..f1866f2d9b211bb0d7f24644b80ef4b50a7c3d24 100644
>> --- a/mm/slab.h
>> +++ b/mm/slab.h
>> @@ -435,6 +435,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
>>  	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
>>  }
>>  
>> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
>> +
>>  #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
>>  			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
>>  			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
>> diff --git a/mm/slab_common.c b/mm/slab_common.c
>> index e2b197e47866c30acdbd1fee4159f262a751c5a7..2d806e02568532a1000fd3912db6978e945dcfa8 100644
>> --- a/mm/slab_common.c
>> +++ b/mm/slab_common.c
>> @@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
>>  		kvfree_rcu_list(head);
>>  }
>>  
>> +static bool kfree_rcu_sheaf(void *obj)
>> +{
>> +	struct kmem_cache *s;
>> +	struct folio *folio;
>> +	struct slab *slab;
>> +
>> +	if (is_vmalloc_addr(obj))
>> +		return false;
>> +
>> +	folio = virt_to_folio(obj);
>> +	if (unlikely(!folio_test_slab(folio)))
>> +		return false;
>> +
>> +	slab = folio_slab(folio);
>> +	s = slab->slab_cache;
>> +	if (s->cpu_sheaves)
>> +		return __kfree_rcu_sheaf(s, obj);
>> +
>> +	return false;
>> +}
>> +
>>  static bool
>>  need_offload_krc(struct kfree_rcu_cpu *krcp)
>>  {
>> @@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
>>  	if (!head)
>>  		might_sleep();
>>  
>> +	if (kfree_rcu_sheaf(ptr))
>> +		return;
>> +
> Uh.. I have some concerns about this.
> 
> This patch introduces a new path which is a collision to the
> existing kvfree_rcu() logic. It implements some batching which
> we already have.

Yes but for caches with sheaves it's better to recycle the whole sheaf (as
described), which is so different from the existing batching scheme that I'm
not sure if there's a sensible way to combine them.

> - kvfree_rcu_barrier() does not know about "sheaf" path. Am i missing
>   something? How do you guarantee that kvfree_rcu_barrier() flushes
>   sheafs? If it is part of kvfree_rcu() it has to care about this.

Hm good point, thanks. I've taken care of handling flushing related to
kfree_rcu() sheaves in kmem_cache_destroy(), but forgot that
kvfree_rcu_barrier() can be also used outside of that - we have one user in
codetag_unload_module() currently.

> - we do not allocate in kvfree_rcu() path because of PREEMMPT_RT, i.e.
>   kvfree_rcu() is supposed it can be called from the non-sleeping contexts.

Hm I could not find where that distinction is in the code, can you give a
hint please. In __kfree_rcu_sheaf() I do only have a GFP_NOWAIT attempt.

> - call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().

If call_rcu() is called once per 32 kfree_rcu() filling up the rcu sheaf, is
it still too slow?
> IMO, it is worth to reuse existing logic in the kvfree_rcu(). I can help
> with it when i have more cycles as part of my RCU work.
It would be most welcome! I'd suggest we currently proceed with this after I
fix up kvfree_rcu_barrier(), and can attempt to consolidate later.

> --
> Uladzislau Rezki



From urezki at gmail.com  Tue Sep  9 02:08:20 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Tue, 9 Sep 2025 11:08:20 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan>
 <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
Message-ID: <aL_uhPtztx7Ef0T2@pc636>

On Mon, Sep 08, 2025 at 02:45:11PM +0200, Vlastimil Babka wrote:
> On 9/8/25 13:59, Uladzislau Rezki wrote:
> > On Wed, Sep 03, 2025 at 02:59:46PM +0200, Vlastimil Babka wrote:
> >> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> >> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> >> addition to main and spare sheaves.
> >> 
> >> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> >> the sheaf is detached and submitted to call_rcu() with a handler that
> >> will try to put it in the barn, or flush to slab pages using bulk free,
> >> when the barn is full. Then a new empty sheaf must be obtained to put
> >> more objects there.
> >> 
> >> It's possible that no free sheaves are available to use for a new
> >> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> >> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> >> kfree_rcu() implementation.
> >> 
> >> Expected advantages:
> >> - batching the kfree_rcu() operations, that could eventually replace the
> >>   existing batching
> >> - sheaves can be reused for allocations via barn instead of being
> >>   flushed to slabs, which is more efficient
> >>   - this includes cases where only some cpus are allowed to process rcu
> >>     callbacks (Android)
> >> 
> >> Possible disadvantage:
> >> - objects might be waiting for more than their grace period (it is
> >>   determined by the last object freed into the sheaf), increasing memory
> >>   usage - but the existing batching does that too.
> >> 
> >> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> >> implementation favors smaller memory footprint over performance.
> >> 
> >> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> >> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> >> many had to fall back to the existing implementation.
> >> 
> >> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> >> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> >> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> >> ---
> >>  mm/slab.h        |   2 +
> >>  mm/slab_common.c |  24 +++++++
> >>  mm/slub.c        | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> >>  3 files changed, 216 insertions(+), 2 deletions(-)
> >> 
> >> diff --git a/mm/slab.h b/mm/slab.h
> >> index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..f1866f2d9b211bb0d7f24644b80ef4b50a7c3d24 100644
> >> --- a/mm/slab.h
> >> +++ b/mm/slab.h
> >> @@ -435,6 +435,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
> >>  	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
> >>  }
> >>  
> >> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> >> +
> >>  #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
> >>  			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
> >>  			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
> >> diff --git a/mm/slab_common.c b/mm/slab_common.c
> >> index e2b197e47866c30acdbd1fee4159f262a751c5a7..2d806e02568532a1000fd3912db6978e945dcfa8 100644
> >> --- a/mm/slab_common.c
> >> +++ b/mm/slab_common.c
> >> @@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
> >>  		kvfree_rcu_list(head);
> >>  }
> >>  
> >> +static bool kfree_rcu_sheaf(void *obj)
> >> +{
> >> +	struct kmem_cache *s;
> >> +	struct folio *folio;
> >> +	struct slab *slab;
> >> +
> >> +	if (is_vmalloc_addr(obj))
> >> +		return false;
> >> +
> >> +	folio = virt_to_folio(obj);
> >> +	if (unlikely(!folio_test_slab(folio)))
> >> +		return false;
> >> +
> >> +	slab = folio_slab(folio);
> >> +	s = slab->slab_cache;
> >> +	if (s->cpu_sheaves)
> >> +		return __kfree_rcu_sheaf(s, obj);
> >> +
> >> +	return false;
> >> +}
> >> +
> >>  static bool
> >>  need_offload_krc(struct kfree_rcu_cpu *krcp)
> >>  {
> >> @@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> >>  	if (!head)
> >>  		might_sleep();
> >>  
> >> +	if (kfree_rcu_sheaf(ptr))
> >> +		return;
> >> +
> > Uh.. I have some concerns about this.
> > 
> > This patch introduces a new path which is a collision to the
> > existing kvfree_rcu() logic. It implements some batching which
> > we already have.
> 
> Yes but for caches with sheaves it's better to recycle the whole sheaf (as
> described), which is so different from the existing batching scheme that I'm
> not sure if there's a sensible way to combine them.
> 
> > - kvfree_rcu_barrier() does not know about "sheaf" path. Am i missing
> >   something? How do you guarantee that kvfree_rcu_barrier() flushes
> >   sheafs? If it is part of kvfree_rcu() it has to care about this.
> 
> Hm good point, thanks. I've taken care of handling flushing related to
> kfree_rcu() sheaves in kmem_cache_destroy(), but forgot that
> kvfree_rcu_barrier() can be also used outside of that - we have one user in
> codetag_unload_module() currently.
> 
> > - we do not allocate in kvfree_rcu() path because of PREEMMPT_RT, i.e.
> >   kvfree_rcu() is supposed it can be called from the non-sleeping contexts.
> 
> Hm I could not find where that distinction is in the code, can you give a
> hint please. In __kfree_rcu_sheaf() I do only have a GFP_NOWAIT attempt.
> 
For PREEMPT_RT a regular spin-lock is an rt-mutex which can sleep. We
made kvfree_rcu() to make it possible to invoke it from non-sleep contexts:

CONFIG_PREEMPT_RT

preempt_disable() or something similar;
 kvfree_rcu();
  GFP_NOWAIT - lock rt-mutex

If GFP_NOWAIT semantic does not access any spin-locks then we are safe
or if it uses raw_spin_locks.

> > - call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().
> 
> If call_rcu() is called once per 32 kfree_rcu() filling up the rcu sheaf, is
> it still too slow?
>
You do not know where in a queue this callback lands, in the beginning,
in the end, etc. It is part of generic list which is processed one by
one. It can contain thousands of callbacks.

If performance is not needed then it is not an issue. But in
kvfree_rcu() we do not use it, because of we want to offload
fast.

--
Uladzislau Rezki


From urezki at gmail.com  Tue Sep  9 02:14:43 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Tue, 9 Sep 2025 11:14:43 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <aL_uhPtztx7Ef0T2@pc636>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan>
 <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
 <aL_uhPtztx7Ef0T2@pc636>
Message-ID: <aL_wAxNv5ifF2IRD@pc636>

On Tue, Sep 09, 2025 at 11:08:20AM +0200, Uladzislau Rezki wrote:
> On Mon, Sep 08, 2025 at 02:45:11PM +0200, Vlastimil Babka wrote:
> > On 9/8/25 13:59, Uladzislau Rezki wrote:
> > > On Wed, Sep 03, 2025 at 02:59:46PM +0200, Vlastimil Babka wrote:
> > >> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> > >> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> > >> addition to main and spare sheaves.
> > >> 
> > >> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> > >> the sheaf is detached and submitted to call_rcu() with a handler that
> > >> will try to put it in the barn, or flush to slab pages using bulk free,
> > >> when the barn is full. Then a new empty sheaf must be obtained to put
> > >> more objects there.
> > >> 
> > >> It's possible that no free sheaves are available to use for a new
> > >> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> > >> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> > >> kfree_rcu() implementation.
> > >> 
> > >> Expected advantages:
> > >> - batching the kfree_rcu() operations, that could eventually replace the
> > >>   existing batching
> > >> - sheaves can be reused for allocations via barn instead of being
> > >>   flushed to slabs, which is more efficient
> > >>   - this includes cases where only some cpus are allowed to process rcu
> > >>     callbacks (Android)
> > >> 
> > >> Possible disadvantage:
> > >> - objects might be waiting for more than their grace period (it is
> > >>   determined by the last object freed into the sheaf), increasing memory
> > >>   usage - but the existing batching does that too.
> > >> 
> > >> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> > >> implementation favors smaller memory footprint over performance.
> > >> 
> > >> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> > >> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> > >> many had to fall back to the existing implementation.
> > >> 
> > >> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> > >> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> > >> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> > >> ---
> > >>  mm/slab.h        |   2 +
> > >>  mm/slab_common.c |  24 +++++++
> > >>  mm/slub.c        | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> > >>  3 files changed, 216 insertions(+), 2 deletions(-)
> > >> 
> > >> diff --git a/mm/slab.h b/mm/slab.h
> > >> index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..f1866f2d9b211bb0d7f24644b80ef4b50a7c3d24 100644
> > >> --- a/mm/slab.h
> > >> +++ b/mm/slab.h
> > >> @@ -435,6 +435,8 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
> > >>  	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
> > >>  }
> > >>  
> > >> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> > >> +
> > >>  #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
> > >>  			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
> > >>  			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
> > >> diff --git a/mm/slab_common.c b/mm/slab_common.c
> > >> index e2b197e47866c30acdbd1fee4159f262a751c5a7..2d806e02568532a1000fd3912db6978e945dcfa8 100644
> > >> --- a/mm/slab_common.c
> > >> +++ b/mm/slab_common.c
> > >> @@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
> > >>  		kvfree_rcu_list(head);
> > >>  }
> > >>  
> > >> +static bool kfree_rcu_sheaf(void *obj)
> > >> +{
> > >> +	struct kmem_cache *s;
> > >> +	struct folio *folio;
> > >> +	struct slab *slab;
> > >> +
> > >> +	if (is_vmalloc_addr(obj))
> > >> +		return false;
> > >> +
> > >> +	folio = virt_to_folio(obj);
> > >> +	if (unlikely(!folio_test_slab(folio)))
> > >> +		return false;
> > >> +
> > >> +	slab = folio_slab(folio);
> > >> +	s = slab->slab_cache;
> > >> +	if (s->cpu_sheaves)
> > >> +		return __kfree_rcu_sheaf(s, obj);
> > >> +
> > >> +	return false;
> > >> +}
> > >> +
> > >>  static bool
> > >>  need_offload_krc(struct kfree_rcu_cpu *krcp)
> > >>  {
> > >> @@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
> > >>  	if (!head)
> > >>  		might_sleep();
> > >>  
> > >> +	if (kfree_rcu_sheaf(ptr))
> > >> +		return;
> > >> +
> > > Uh.. I have some concerns about this.
> > > 
> > > This patch introduces a new path which is a collision to the
> > > existing kvfree_rcu() logic. It implements some batching which
> > > we already have.
> > 
> > Yes but for caches with sheaves it's better to recycle the whole sheaf (as
> > described), which is so different from the existing batching scheme that I'm
> > not sure if there's a sensible way to combine them.
> > 
> > > - kvfree_rcu_barrier() does not know about "sheaf" path. Am i missing
> > >   something? How do you guarantee that kvfree_rcu_barrier() flushes
> > >   sheafs? If it is part of kvfree_rcu() it has to care about this.
> > 
> > Hm good point, thanks. I've taken care of handling flushing related to
> > kfree_rcu() sheaves in kmem_cache_destroy(), but forgot that
> > kvfree_rcu_barrier() can be also used outside of that - we have one user in
> > codetag_unload_module() currently.
> > 
> > > - we do not allocate in kvfree_rcu() path because of PREEMMPT_RT, i.e.
> > >   kvfree_rcu() is supposed it can be called from the non-sleeping contexts.
> > 
> > Hm I could not find where that distinction is in the code, can you give a
> > hint please. In __kfree_rcu_sheaf() I do only have a GFP_NOWAIT attempt.
> > 
> For PREEMPT_RT a regular spin-lock is an rt-mutex which can sleep. We
> made kvfree_rcu() to make it possible to invoke it from non-sleep contexts:
> 
> CONFIG_PREEMPT_RT
> 
> preempt_disable() or something similar;
>  kvfree_rcu();
>   GFP_NOWAIT - lock rt-mutex
> 
> If GFP_NOWAIT semantic does not access any spin-locks then we are safe
> or if it uses raw_spin_locks.
> 
And this is valid only for double argument, single argument you can
invoke from sleeping context only, then you can allocate.

--
Uladzislau Rezki


From vbabka at suse.cz  Tue Sep  9 03:20:36 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Tue, 9 Sep 2025 12:20:36 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aL_uhPtztx7Ef0T2@pc636>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan> <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
 <aL_uhPtztx7Ef0T2@pc636>
Message-ID: <838888f2-7a0f-49f0-9e33-592f13b14ad3@suse.cz>

On 9/9/25 11:08, Uladzislau Rezki wrote:
> On Mon, Sep 08, 2025 at 02:45:11PM +0200, Vlastimil Babka wrote:
>> 
>> Hm I could not find where that distinction is in the code, can you give a
>> hint please. In __kfree_rcu_sheaf() I do only have a GFP_NOWAIT attempt.
>> 
> For PREEMPT_RT a regular spin-lock is an rt-mutex which can sleep. We
> made kvfree_rcu() to make it possible to invoke it from non-sleep contexts:

Oh you mean it's not allocating even on !RT so there's no RT-specific code.

> CONFIG_PREEMPT_RT
> 
> preempt_disable() or something similar;
>  kvfree_rcu();
>   GFP_NOWAIT - lock rt-mutex
> 
> If GFP_NOWAIT semantic does not access any spin-locks then we are safe
> or if it uses raw_spin_locks.

It does access spinlocks so it's not safe. Thanks, I didn't realize that
aspect of kfree_rcu(). We'll need to solve this before making sheaves
enabled everywhere. I don't think the vma or maple tree code would
kfree_rcu() vma or maple_node in such a restricted context. But to be safe
I'll just disable the kfree rcu sheaves for PREEMPT_RT for now.
>> > - call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().
>> 
>> If call_rcu() is called once per 32 kfree_rcu() filling up the rcu sheaf, is
>> it still too slow?
>>
> You do not know where in a queue this callback lands, in the beginning,
> in the end, etc. It is part of generic list which is processed one by
> one. It can contain thousands of callbacks.
> 
> If performance is not needed then it is not an issue. But in
> kvfree_rcu() we do not use it, because of we want to offload
> fast.

So IIUC one could call_rcu(obj, some_callback_accessing_obj) and then do a
kfree_rcu(obj) and the latter can be processed first?
> --
> Uladzislau Rezki



From Liam.Howlett at oracle.com  Tue Sep  9 07:35:15 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue, 9 Sep 2025 10:35:15 -0400
Subject: [PATCH v7 04/21] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <aL_uhPtztx7Ef0T2@pc636>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan>
 <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
 <aL_uhPtztx7Ef0T2@pc636>
Message-ID: <772oyaa3j27tlklhpo6oqxsdtlvsg5goh2opzuig6xvgztgum4@scsoxrgtqm2f>

* Uladzislau Rezki <urezki at gmail.com> [250909 05:08]:

...

> 
> > > - call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().
> > 
> > If call_rcu() is called once per 32 kfree_rcu() filling up the rcu sheaf, is
> > it still too slow?
> >
> You do not know where in a queue this callback lands, in the beginning,
> in the end, etc. It is part of generic list which is processed one by
> one. It can contain thousands of callbacks.

How does this differ from kvfree_rcu()?

Surely if you have enough calls to kvfree_rcu(), you will end up with a
large list of frees before the end of a grace period?  Our placement in
the freeing order would still be dependent on what else is using the
infrastructure in the same grace period, right?

How is kvfree_rcu() affected by rcu callback offloading to a specific
cpu and rcu expedite?  Often these two features come into play for
certain workloads which are of concern to us.

> 
> If performance is not needed then it is not an issue. But in
> kvfree_rcu() we do not use it, because of we want to offload
> fast.

Today, I free things using call_rcu() and a custom callback so I would
think stacking 32 together would make the list shorter, but latency
would increase by waiting until there are 32.

If we wanted to flush the kvfree_rcu() list, is it done in the same way
as the call_rcu() list, or is there a better way?

Thanks,
Liam



From vbabka at suse.cz  Tue Sep  9 07:55:55 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Tue, 9 Sep 2025 16:55:55 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <838888f2-7a0f-49f0-9e33-592f13b14ad3@suse.cz>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan> <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
 <aL_uhPtztx7Ef0T2@pc636> <838888f2-7a0f-49f0-9e33-592f13b14ad3@suse.cz>
Message-ID: <64e0795d-c285-47d4-8282-a31d10fe1c47@suse.cz>

On 9/9/25 12:20, Vlastimil Babka wrote:
> On 9/9/25 11:08, Uladzislau Rezki wrote:
>> On Mon, Sep 08, 2025 at 02:45:11PM +0200, Vlastimil Babka wrote:
>>> 
>>> Hm I could not find where that distinction is in the code, can you give a
>>> hint please. In __kfree_rcu_sheaf() I do only have a GFP_NOWAIT attempt.
>>> 
>> For PREEMPT_RT a regular spin-lock is an rt-mutex which can sleep. We
>> made kvfree_rcu() to make it possible to invoke it from non-sleep contexts:
> 
> Oh you mean it's not allocating even on !RT so there's no RT-specific code.
> 
>> CONFIG_PREEMPT_RT
>> 
>> preempt_disable() or something similar;
>>  kvfree_rcu();
>>   GFP_NOWAIT - lock rt-mutex
>> 
>> If GFP_NOWAIT semantic does not access any spin-locks then we are safe
>> or if it uses raw_spin_locks.
> 
> It does access spinlocks so it's not safe. Thanks, I didn't realize that
> aspect of kfree_rcu(). We'll need to solve this before making sheaves
> enabled everywhere. I don't think the vma or maple tree code would
> kfree_rcu() vma or maple_node in such a restricted context. But to be safe
> I'll just disable the kfree rcu sheaves for PREEMPT_RT for now.

So I came up with this fixup to avoid PREEMPT_RT troubles and make
kvfree_rcu_barrier() work.

----8<----
>From 15a8db2ef716b5db547f2d86ab30d8774333fb04 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka at suse.cz>
Date: Tue, 9 Sep 2025 16:18:52 +0200
Subject: [PATCH] slub: fix issues with kfree_rcu sheaf handling

Fix two issues reported by Ulad:

- on PREEMPT_RT if kfree_rcu() comes from an atomic context, taking a
  spinlock on the barn or doing a GFP_NOWAIT allocation of a new sheaf
  might not be possible. For now just limit the usage of
  kfree_rcu_sheaf() to !PREEMPT_RT

- kvfree_rcu_barrier() must flush all rcu_free sheaves to deliver on its
  promise. The usage isn't limited to destroying of a single cache. Add
  flush_all_rcu_sheaves() to do that.

Reported-by: Uladzislau Rezki <urezki at gmail.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab.h        |  1 +
 mm/slab_common.c |  4 ++-
 mm/slub.c        | 74 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 78 insertions(+), 1 deletion(-)

diff --git a/mm/slab.h b/mm/slab.h
index f1866f2d9b21..e82e51c44bd0 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -436,6 +436,7 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
 }
 
 bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+void flush_all_rcu_sheaves(void);
 
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 2d806e025685..005a4319c06a 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1973,7 +1973,7 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
-	if (kfree_rcu_sheaf(ptr))
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
 		return;
 
 	// Queue the object but don't yet schedule the batch.
@@ -2050,6 +2050,8 @@ void kvfree_rcu_barrier(void)
 	bool queued;
 	int i, cpu;
 
+	flush_all_rcu_sheaves();
+
 	/*
 	 * Firstly we detach objects and queue them over an RCU-batch
 	 * for all CPUs. Finally queued works are flushed for each CPU.
diff --git a/mm/slub.c b/mm/slub.c
index 9f9b7e1fa356..19cd8444ae5d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3895,6 +3895,80 @@ static void flush_all(struct kmem_cache *s)
 	cpus_read_unlock();
 }
 
+static void flush_rcu_sheaf(struct work_struct *w)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_free;
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+
+	sfw = container_of(w, struct slub_flush_work, work);
+	s = sfw->s;
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+}
+
+
+/* needed for kvfree_rcu_barrier() */
+void flush_all_rcu_sheaves()
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+	bool flushed = false;
+	unsigned int cpu;
+
+	cpus_read_lock();
+	mutex_lock(&slab_mutex);
+
+	list_for_each_entry(s, &slab_caches, list) {
+		if (!s->cpu_sheaves)
+			continue;
+
+		mutex_lock(&flush_lock);
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+			if (!pcs->rcu_free || !pcs->rcu_free->size) {
+				sfw->skip = true;
+				continue;
+			}
+
+			INIT_WORK(&sfw->work, flush_rcu_sheaf);
+			sfw->skip = false;
+			sfw->s = s;
+			queue_work_on(cpu, flushwq, &sfw->work);
+			flushed = true;
+		}
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+			if (sfw->skip)
+				continue;
+			flush_work(&sfw->work);
+		}
+
+		mutex_unlock(&flush_lock);
+	}
+
+	mutex_unlock(&slab_mutex);
+	cpus_read_unlock();
+
+	if (flushed)
+		rcu_barrier();
+}
+
 /*
  * Use the cpu notifier to insure that the cpu slabs are flushed when
  * necessary.
-- 
2.51.0




From Liam.Howlett at oracle.com  Tue Sep  9 10:19:24 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue, 9 Sep 2025 13:19:24 -0400
Subject: [RFC PATCH 4/6] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <a3777992-21fc-4e55-9a5e-72b17dc86135@redhat.com>
References: <20250815191031.3769540-1-Liam.Howlett@oracle.com>
 <20250815191031.3769540-5-Liam.Howlett@oracle.com>
 <39c46a52-f63a-4f3d-a276-6f59c55680ba@lucifer.local>
 <hs5nuhqzqyt5mp3fzznuaiyyck6b6ovuxoxayvm5ofhoa5324m@gtmezuqareiq>
 <a3777992-21fc-4e55-9a5e-72b17dc86135@redhat.com>
Message-ID: <gtt62i4wiflnhetv2w4kanbs64z2bdmxhth3hqdz5bcflfamkk@ojhlmqy4r3po>

* David Hildenbrand <david at redhat.com> [250904 06:21]:
> On 03.09.25 22:19, Liam R. Howlett wrote:
> > * Lorenzo Stoakes <lorenzo.stoakes at oracle.com> [250819 15:14]:
> > > On Fri, Aug 15, 2025 at 03:10:29PM -0400, Liam R. Howlett wrote:
> > > > The ceiling and tree search limit need to be different arguments for the
> > > > future change in the failed fork attempt.
> > > > 
> > > > No functional changes intended.
> > > > 
> > > > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > > 
> > > (Obv. in addition to comment about broken VMA tests :P)
> > > 
> > > I guess intent is that if we discover any page tables beyond tree_max then
> > > we ought to just wipe them all out so, in effect, we don't consider
> > > mappings at or past tree_max to be valid?
> > 
> > Actually... there are some archs that map outside the vma and they are
> > valid.. I think mips? and I think lower, but yeah.. it's needed.  This
> > is why prev->vm_end and next->vm_start are used as page table limits,
> > afaik.  This is a serious annoyance because it frequently adds walks
> > that are infrequently necessary to the vma tree.
> 
> Hm, does that still exist?

I think it does?

arch/mips/mm/fault.c still checks for addresses between VMALLOC_START
and VMALLOC_END, as well as MODULES_VADDR and MODULES_END and
(potentially, depending on CONFIG) jumps to vmalloc_fault.

I tried to find the statement of the start/end going to the next/prev
vma that came across before, but I cannot.  It may have been in a git
log for something else entirely.

> I recall something odd ... was it that gate area thingy (in_gate_area) we
> also have to handle in GUP code? The is x86/arm though, not mips.

IIRC, gate area is to do with vdso/vvars and so going to the maximum
allowed pte would unmap that even when it's not in the vma tree - which
it is not.  This is true for most platforms.

But if that's the case, then unmapping the last vma in the tree would
cause the vdso to no longer work - which doesn't make sense to me?

I'm not sure if any platform maps them at a low value so that
"prev->vm_start or 0" makes sense, but I would not be surprised.

Thanks,
Liam


From Liam.Howlett at oracle.com  Tue Sep  9 12:09:37 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:37 -0400
Subject: [PATCH v1 1/9] mm/mmap: Move exit_mmap() trace point
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-2-Liam.Howlett@oracle.com>

Move the trace point later in the function so that it is not skipped in
the event of a failed fork.

Acked-by: Chris Li <chrisl at kernel.org>
Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/mmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index 5fd3b80fda1d5..b07b3ec5e28f5 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1310,9 +1310,9 @@ void exit_mmap(struct mm_struct *mm)
 
 	BUG_ON(count != mm->map_count);
 
-	trace_exit_mmap(mm);
 destroy:
 	__mt_destroy(&mm->mm_mt);
+	trace_exit_mmap(mm);
 	mmap_write_unlock(mm);
 	vm_unacct_memory(nr_accounted);
 }
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:36 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:36 -0400
Subject: [PATCH v1 0/9] Remove XA_ZERO from error recovery of dup_mmap()
Message-ID: <20250909190945.1030905-1-Liam.Howlett@oracle.com>

It is possible that the dup_mmap() call fails on allocating or setting
up a vma after the maple tree of the oldmm is copied.  Today, that
failure point is marked by inserting an XA_ZERO entry over the failure
point so that the exact location does not need to be communicated
through to exit_mmap().

However, a race exists in the tear down process because the dup_mmap()
drops the mmap lock before exit_mmap() can remove the partially set up
vma tree.  This means that other tasks may get to the mm tree and find
the invalid vma pointer (since it's an XA_ZERO entry), even though the
mm is marked as MMF_OOM_SKIP and MMF_UNSTABLE.

To remove the race fully, the tree must be cleaned up before dropping
the lock.  This is accomplished by extracting the vma cleanup in
exit_mmap() and changing the required functions to pass through the vma
search limit.  Any other tree modifications would require extra cycles
which should be spent on freeing memory.

This does run the risk of increasing the possibility of finding no vmas
(which is already possible!) in code this isn't careful.

The final three patches are to address the excessive argument lists
being passed between the functions.  Using the struct unmap_desc also
allows some special-case code to be removed in favour of the struct
setup differences.

RFC: https://lore.kernel.org/linux-mm/20250815191031.3769540-1-Liam.Howlett at oracle.com/
Changes since RFC:
- Change BUG_ON to WARN_ON_ONCE in tear_down_vmas() - Thanks Lorenzo
- Added sanity WARN_ON_ONCE() to free_pgtables() for ceiling and
  tree_max - Thanks Lorenzo
- Added comment to free_pgtables() in a vain attempt to inform users
  what's going on - Thanks Lorenzo
- Fixed free_pgtables() in testing code
- Added struct unmap_desc to describe the unmap event to reduce the
  argument list  - Thanks Lorenzo
- Rebased against mm-new

Liam R. Howlett (9):
  mm/mmap: Move exit_mmap() trace point
  mm/mmap: Abstract vma clean up from exit_mmap()
  mm/vma: Add limits to unmap_region() for vmas
  mm/memory: Add tree limit to free_pgtables()
  mm/vma: Add page table limit to unmap_region()
  mm: Change dup_mmap() recovery
  mm: Introduce unmap_desc struct to reduce function arguments
  mm/vma: Use unmap_desc in vms_clear_ptes() and exit_mmap()
  mm: Use unmap_desc struct for freeing page tables.

 include/linux/mm.h               |  3 -
 mm/internal.h                    |  8 ++-
 mm/memory.c                      | 69 +++++++++++++----------
 mm/mmap.c                        | 96 ++++++++++++++++++++++----------
 mm/vma.c                         | 55 +++++++++---------
 mm/vma.h                         | 48 +++++++++++++++-
 tools/testing/vma/vma_internal.h | 24 ++------
 7 files changed, 193 insertions(+), 110 deletions(-)

-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:39 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:39 -0400
Subject: [PATCH v1 3/9] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-4-Liam.Howlett@oracle.com>

Add a limit to the vma search instead of using the start and end of the
one passed in.

No functional changes intended.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/vma.c | 6 ++++--
 mm/vma.h | 1 +
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/mm/vma.c b/mm/vma.c
index abe0da33c8446..a648e0555c873 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
  * Called with the mm semaphore held.
  */
 void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
+		unsigned long vma_min, unsigned long vma_max,
 		struct vm_area_struct *prev, struct vm_area_struct *next)
 {
 	struct mm_struct *mm = vma->vm_mm;
@@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
 
 	tlb_gather_mmu(&tlb, mm);
 	update_hiwater_rss(mm);
-	unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
+	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
 		   /* mm_wr_locked = */ true);
 	mas_set(mas, vma->vm_end);
 	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
@@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
 
 		vma_iter_set(vmi, vma->vm_end);
 		/* Undo any partial mapping done by a device driver. */
-		unmap_region(&vmi->mas, vma, map->prev, map->next);
+		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
+			     map->prev, map->next);
 
 		return error;
 	}
diff --git a/mm/vma.h b/mm/vma.h
index 9183fe5490090..a9d0cef684ddb 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -261,6 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
 void remove_vma(struct vm_area_struct *vma);
 
 void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
+		unsigned long min, unsigned long max,
 		struct vm_area_struct *prev, struct vm_area_struct *next);
 
 /* We are about to modify the VMA's flags. */
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:38 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:38 -0400
Subject: [PATCH v1 2/9] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-3-Liam.Howlett@oracle.com>

Create the new function tear_down_vmas() to remove a range of vmas.
exit_mmap() will be removing all the vmas.

This is necessary for future patches.

No functional changes intended.

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/mmap.c | 37 ++++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 13 deletions(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index b07b3ec5e28f5..a290448a53bb2 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
 }
 EXPORT_SYMBOL(vm_brk_flags);
 
+static inline
+unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
+		struct vm_area_struct *vma, unsigned long max)
+{
+	unsigned long nr_accounted = 0;
+	int count = 0;
+
+	mmap_assert_write_locked(mm);
+	vma_iter_set(vmi, vma->vm_end);
+	do {
+		if (vma->vm_flags & VM_ACCOUNT)
+			nr_accounted += vma_pages(vma);
+		vma_mark_detached(vma);
+		remove_vma(vma);
+		count++;
+		cond_resched();
+		vma = vma_next(vmi);
+	} while (vma && vma->vm_end <= max);
+
+	WARN_ON_ONCE(count != mm->map_count);
+	return nr_accounted;
+}
+
 /* Release all mmaps. */
 void exit_mmap(struct mm_struct *mm)
 {
@@ -1257,7 +1280,6 @@ void exit_mmap(struct mm_struct *mm)
 	struct vm_area_struct *vma;
 	unsigned long nr_accounted = 0;
 	VMA_ITERATOR(vmi, mm, 0);
-	int count = 0;
 
 	/* mm's last user has gone, and its about to be pulled down */
 	mmu_notifier_release(mm);
@@ -1297,18 +1319,7 @@ void exit_mmap(struct mm_struct *mm)
 	 * enabled, without holding any MM locks besides the unreachable
 	 * mmap_write_lock.
 	 */
-	vma_iter_set(&vmi, vma->vm_end);
-	do {
-		if (vma->vm_flags & VM_ACCOUNT)
-			nr_accounted += vma_pages(vma);
-		vma_mark_detached(vma);
-		remove_vma(vma);
-		count++;
-		cond_resched();
-		vma = vma_next(&vmi);
-	} while (vma && likely(!xa_is_zero(vma)));
-
-	BUG_ON(count != mm->map_count);
+	nr_accounted = tear_down_vmas(mm, &vmi, vma, ULONG_MAX);
 
 destroy:
 	__mt_destroy(&mm->mm_mt);
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:41 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:41 -0400
Subject: [PATCH v1 5/9] mm/vma: Add page table limit to unmap_region()
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-6-Liam.Howlett@oracle.com>

The unmap_region() calls need to pass through the page table limit for a
future patch.

No functional changes intended.

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/vma.c | 5 +++--
 mm/vma.h | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)

diff --git a/mm/vma.c b/mm/vma.c
index 1bae142bbc0f1..4c850ffd83a4b 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -474,7 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
  * Called with the mm semaphore held.
  */
 void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
-		unsigned long vma_min, unsigned long vma_max,
+		unsigned long vma_min, unsigned long vma_max, unsigned long pg_max,
 		struct vm_area_struct *prev, struct vm_area_struct *next)
 {
 	struct mm_struct *mm = vma->vm_mm;
@@ -487,7 +487,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
 	mas_set(mas, vma->vm_end);
 	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
 		      next ? next->vm_start : USER_PGTABLES_CEILING,
-		      next ? next->vm_start : USER_PGTABLES_CEILING,
+		      pg_max,
 		      /* mm_wr_locked = */ true);
 	tlb_finish_mmu(&tlb);
 }
@@ -2420,6 +2420,7 @@ static int __mmap_new_file_vma(struct mmap_state *map,
 		vma_iter_set(vmi, vma->vm_end);
 		/* Undo any partial mapping done by a device driver. */
 		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
+			     map->next ? map->next->vm_start : USER_PGTABLES_CEILING,
 			     map->prev, map->next);
 
 		return error;
diff --git a/mm/vma.h b/mm/vma.h
index a9d0cef684ddb..b0ebc81d5862e 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -261,7 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
 void remove_vma(struct vm_area_struct *vma);
 
 void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
-		unsigned long min, unsigned long max,
+		unsigned long min, unsigned long max, unsigned long pg_max,
 		struct vm_area_struct *prev, struct vm_area_struct *next);
 
 /* We are about to modify the VMA's flags. */
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:40 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:40 -0400
Subject: [PATCH v1 4/9] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-5-Liam.Howlett@oracle.com>

The ceiling and tree search limit need to be different arguments for the
future change in the failed fork attempt.

Add some documentation around free_pgtables() and the limits in an
attempt to clarify the floor and ceiling use as well as the new
tree_max.

Test code also updated.

No functional changes intended.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/internal.h                    |  4 +++-
 mm/memory.c                      | 28 +++++++++++++++++++++++++---
 mm/mmap.c                        |  2 +-
 mm/vma.c                         |  3 ++-
 tools/testing/vma/vma_internal.h |  3 ++-
 5 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 63e3ec8d63be7..d295252407fee 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -444,7 +444,9 @@ void folio_activate(struct folio *folio);
 
 void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *start_vma, unsigned long floor,
-		   unsigned long ceiling, bool mm_wr_locked);
+		   unsigned long ceiling, unsigned long tree_max,
+		   bool mm_wr_locked);
+
 void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
 
 struct zap_details;
diff --git a/mm/memory.c b/mm/memory.c
index 3e0404bd57a02..24716b3713f66 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -369,12 +369,34 @@ void free_pgd_range(struct mmu_gather *tlb,
 	} while (pgd++, addr = next, addr != end);
 }
 
+/*
+ * free_pgtables() - Free a range of page tables
+ * @tlb: The mmu gather
+ * @mas: The maple state
+ * @vma: The first vma
+ * @floor: The lowest page table address
+ * @ceiling: The highest page table address
+ * @tree_max: The highest tree search address
+ * @mm_wr_locked: boolean indicating if the mm is write locked
+ *
+ * Note: Floor and ceiling are provided to indicate the absolute range of the
+ * page tables that should be removed.  This can differ from the vma mappings on
+ * some archs that may have mappings that need to be removed outside the vmas.
+ * Note that the prev->vm_end and next->vm_start are often used.
+ *
+ * The tree_max differs from the ceiling when a dup_mmap() failed and the tree
+ * has unrelated data to the mm_struct being torn down.
+ */
 void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *vma, unsigned long floor,
-		   unsigned long ceiling, bool mm_wr_locked)
+		   unsigned long ceiling, unsigned long tree_max,
+		   bool mm_wr_locked)
 {
 	struct unlink_vma_file_batch vb;
 
+	/* underflow can happen and is fine */
+	WARN_ON_ONCE(tree_max - 1 > ceiling - 1);
+
 	tlb_free_vmas(tlb);
 
 	do {
@@ -385,7 +407,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
 		 * be 0.  This will underflow and is okay.
 		 */
-		next = mas_find(mas, ceiling - 1);
+		next = mas_find(mas, tree_max - 1);
 		if (unlikely(xa_is_zero(next)))
 			next = NULL;
 
@@ -405,7 +427,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		 */
 		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
 			vma = next;
-			next = mas_find(mas, ceiling - 1);
+			next = mas_find(mas, tree_max - 1);
 			if (unlikely(xa_is_zero(next)))
 				next = NULL;
 			if (mm_wr_locked)
diff --git a/mm/mmap.c b/mm/mmap.c
index a290448a53bb2..0f4808f135fe6 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1311,7 +1311,7 @@ void exit_mmap(struct mm_struct *mm)
 	mt_clear_in_rcu(&mm->mm_mt);
 	vma_iter_set(&vmi, vma->vm_end);
 	free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
-		      USER_PGTABLES_CEILING, true);
+		      USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
 	tlb_finish_mmu(&tlb);
 
 	/*
diff --git a/mm/vma.c b/mm/vma.c
index a648e0555c873..1bae142bbc0f1 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -486,6 +486,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
 		   /* mm_wr_locked = */ true);
 	mas_set(mas, vma->vm_end);
 	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
+		      next ? next->vm_start : USER_PGTABLES_CEILING,
 		      next ? next->vm_start : USER_PGTABLES_CEILING,
 		      /* mm_wr_locked = */ true);
 	tlb_finish_mmu(&tlb);
@@ -1232,7 +1233,7 @@ static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
 	mas_set(mas_detach, 1);
 	/* start and end may be different if there is no prev or next vma. */
 	free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
-		      vms->unmap_end, mm_wr_locked);
+		      vms->unmap_end, vms->unmap_end, mm_wr_locked);
 	tlb_finish_mmu(&tlb);
 	vms->clear_ptes = false;
 }
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 07167446dcf42..823d379e1fac2 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -900,7 +900,8 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
 
 static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *vma, unsigned long floor,
-		   unsigned long ceiling, bool mm_wr_locked)
+		   unsigned long ceiling, unsigned long tree_max,
+		   bool mm_wr_locked)
 {
 	(void)tlb;
 	(void)mas;
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:44 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:44 -0400
Subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and exit_mmap()
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-9-Liam.Howlett@oracle.com>

vms_clear_ptes() is slightly different than other callers to
unmap_region() and so had the unmapping open-coded.  Using the new
structure it is now possible to special-case the struct setup instead of
having the open-coded function.

exit_mmap() also calls unmap_vmas() with many arguemnts.  Using the
unmap_all_init() function to set the unmap descriptor for all vmas makes
this a bit easier to read.

Update to the vma test code is necessary to ensure testing continues to
function.

No functional changes intended.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 include/linux/mm.h               |  3 ---
 mm/internal.h                    |  3 +++
 mm/memory.c                      | 24 ++++++++------------
 mm/mmap.c                        |  5 +++-
 mm/vma.c                         | 39 ++++++++++++++++++--------------
 mm/vma.h                         | 14 ++++++++++++
 tools/testing/vma/vma_internal.h | 14 ++++--------
 7 files changed, 56 insertions(+), 46 deletions(-)

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 892fe5dbf9de0..23eb59d543390 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2450,9 +2450,6 @@ static inline void zap_vma_pages(struct vm_area_struct *vma)
 	zap_page_range_single(vma, vma->vm_start,
 			      vma->vm_end - vma->vm_start, NULL);
 }
-void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
-		struct vm_area_struct *start_vma, unsigned long start,
-		unsigned long end, unsigned long tree_end, bool mm_wr_locked);
 
 struct mmu_notifier_range;
 
diff --git a/mm/internal.h b/mm/internal.h
index d295252407fee..1239944f2830a 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -197,6 +197,9 @@ static inline void vma_close(struct vm_area_struct *vma)
 	}
 }
 
+/* unmap_vmas is in mm/memory.c */
+void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);
+
 #ifdef CONFIG_MMU
 
 /* Flags for folio_pte_batch(). */
diff --git a/mm/memory.c b/mm/memory.c
index 829cd94950182..8d4d976311037 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2084,12 +2084,7 @@ static void unmap_single_vma(struct mmu_gather *tlb,
 /**
  * unmap_vmas - unmap a range of memory covered by a list of vma's
  * @tlb: address of the caller's struct mmu_gather
- * @mas: the maple state
- * @vma: the starting vma
- * @start_addr: virtual address at which to start unmapping
- * @end_addr: virtual address at which to end unmapping
- * @tree_end: The maximum index to check
- * @mm_wr_locked: lock flag
+ * @unmap: The unmap_desc
  *
  * Unmap all pages in the vma list.
  *
@@ -2102,11 +2097,9 @@ static void unmap_single_vma(struct mmu_gather *tlb,
  * ensure that any thus-far unmapped pages are flushed before unmap_vmas()
  * drops the lock and schedules.
  */
-void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
-		struct vm_area_struct *vma, unsigned long start_addr,
-		unsigned long end_addr, unsigned long tree_end,
-		bool mm_wr_locked)
+void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
 {
+	struct vm_area_struct *vma;
 	struct mmu_notifier_range range;
 	struct zap_details details = {
 		.zap_flags = ZAP_FLAG_DROP_MARKER | ZAP_FLAG_UNMAP,
@@ -2114,17 +2107,18 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
 		.even_cows = true,
 	};
 
+	vma = unmap->first;
 	mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma->vm_mm,
-				start_addr, end_addr);
+				unmap->vma_min, unmap->vma_max);
 	mmu_notifier_invalidate_range_start(&range);
 	do {
-		unsigned long start = start_addr;
-		unsigned long end = end_addr;
+		unsigned long start = unmap->vma_min;
+		unsigned long end = unmap->vma_max;
 		hugetlb_zap_begin(vma, &start, &end);
 		unmap_single_vma(tlb, vma, start, end, &details,
-				 mm_wr_locked);
+				 unmap->mm_wr_locked);
 		hugetlb_zap_end(vma, &details);
-		vma = mas_find(mas, tree_end - 1);
+		vma = mas_find(unmap->mas, unmap->tree_max - 1);
 	} while (vma);
 	mmu_notifier_invalidate_range_end(&range);
 }
diff --git a/mm/mmap.c b/mm/mmap.c
index 5c9bd3f20e53f..6011f62b0a294 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1280,10 +1280,12 @@ void exit_mmap(struct mm_struct *mm)
 	struct vm_area_struct *vma;
 	unsigned long nr_accounted = 0;
 	VMA_ITERATOR(vmi, mm, 0);
+	struct unmap_desc unmap;
 
 	/* mm's last user has gone, and its about to be pulled down */
 	mmu_notifier_release(mm);
 
+	unmap.mm_wr_locked = false;
 	mmap_read_lock(mm);
 	arch_exit_mmap(mm);
 
@@ -1295,11 +1297,12 @@ void exit_mmap(struct mm_struct *mm)
 		goto destroy;
 	}
 
+	unmap_all_init(&unmap, &vmi, vma);
 	flush_cache_mm(mm);
 	tlb_gather_mmu_fullmm(&tlb, mm);
 	/* update_hiwater_rss(mm) here? but nobody should be looking */
 	/* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */
-	unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false);
+	unmap_vmas(&tlb, &unmap);
 	mmap_read_unlock(mm);
 
 	/*
diff --git a/mm/vma.c b/mm/vma.c
index c92384975cbb2..ad64cd9795ef3 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -481,8 +481,7 @@ void unmap_region(struct unmap_desc *desc)
 
 	tlb_gather_mmu(&tlb, mm);
 	update_hiwater_rss(mm);
-	unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
-		   desc->vma_max, desc->mm_wr_locked);
+	unmap_vmas(&tlb, desc);
 	mas_set(mas, desc->tree_reset);
 	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
 		      desc->last_pgaddr, desc->tree_max,
@@ -1213,26 +1212,32 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
 static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
 		    struct ma_state *mas_detach, bool mm_wr_locked)
 {
-	struct mmu_gather tlb;
+	struct unmap_desc unmap = {
+		.mas = mas_detach,
+		.first = vms->vma,
+		/* start and end may be different if there is no prev or next vma. */
+		.first_pgaddr = vms->unmap_start,
+		.last_pgaddr = vms->unmap_end,
+		.vma_min = vms->start,
+		.vma_max = vms->end,
+		/*
+		 * The tree limits and reset differ from the normal case since it's a
+		 * side-tree
+		 */
+		.tree_reset = 1,
+		.tree_max = vms->vma_count,
+		/*
+		 * We can free page tables without write-locking mmap_lock because VMAs
+		 * were isolated before we downgraded mmap_lock.
+		 */
+		.mm_wr_locked = mm_wr_locked,
+	};
 
 	if (!vms->clear_ptes) /* Nothing to do */
 		return;
 
-	/*
-	 * We can free page tables without write-locking mmap_lock because VMAs
-	 * were isolated before we downgraded mmap_lock.
-	 */
 	mas_set(mas_detach, 1);
-	tlb_gather_mmu(&tlb, vms->vma->vm_mm);
-	update_hiwater_rss(vms->vma->vm_mm);
-	unmap_vmas(&tlb, mas_detach, vms->vma, vms->start, vms->end,
-		   vms->vma_count, mm_wr_locked);
-
-	mas_set(mas_detach, 1);
-	/* start and end may be different if there is no prev or next vma. */
-	free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
-		      vms->unmap_end, vms->unmap_end, mm_wr_locked);
-	tlb_finish_mmu(&tlb);
+	unmap_region(&unmap);
 	vms->clear_ptes = false;
 }
 
diff --git a/mm/vma.h b/mm/vma.h
index 4edd5d26ffcfc..8b55a0c73d097 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -164,6 +164,20 @@ struct unmap_desc {
 	bool mm_wr_locked;            /* If the mmap write lock is held */
 };
 
+static inline void unmap_all_init(struct unmap_desc *desc,
+		struct vma_iterator *vmi, struct vm_area_struct *vma)
+{
+	desc->mas = &vmi->mas;
+	desc->first = vma;
+	desc->first_pgaddr = FIRST_USER_ADDRESS;
+	desc->last_pgaddr = USER_PGTABLES_CEILING;
+	desc->vma_min = 0;
+	desc->vma_max = ULONG_MAX;
+	desc->tree_max = ULONG_MAX;
+	desc->tree_reset = vma->vm_end;
+	desc->mm_wr_locked = false;
+}
+
 #define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
 	struct unmap_desc name = {                                          \
 		.mas = &(_vmi)->mas,                                          \
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 823d379e1fac2..d73ad4747d40a 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -884,18 +884,12 @@ static inline void update_hiwater_vm(struct mm_struct *)
 {
 }
 
-static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
-		      struct vm_area_struct *vma, unsigned long start_addr,
-		      unsigned long end_addr, unsigned long tree_end,
-		      bool mm_wr_locked)
+struct unmap_desc;
+
+static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
 {
 	(void)tlb;
-	(void)mas;
-	(void)vma;
-	(void)start_addr;
-	(void)end_addr;
-	(void)tree_end;
-	(void)mm_wr_locked;
+	(void)unmap;
 }
 
 static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:45 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:45 -0400
Subject: [PATCH v1 9/9] mm: Use unmap_desc struct for freeing page tables.
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-10-Liam.Howlett@oracle.com>

Pass through the unmap_desc to free_pgtables() because it almost has
everything necessary and is already on the stack.

Updates testing code as necessary.

No functional changes intended.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/internal.h                    |  5 +----
 mm/memory.c                      | 21 ++++++++++-----------
 mm/mmap.c                        |  6 +++---
 mm/vma.c                         |  7 ++-----
 tools/testing/vma/vma_internal.h | 11 ++---------
 5 files changed, 18 insertions(+), 32 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 1239944f2830a..f22329967e908 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -445,10 +445,7 @@ bool __folio_end_writeback(struct folio *folio);
 void deactivate_file_folio(struct folio *folio);
 void folio_activate(struct folio *folio);
 
-void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
-		   struct vm_area_struct *start_vma, unsigned long floor,
-		   unsigned long ceiling, unsigned long tree_max,
-		   bool mm_wr_locked);
+void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc);
 
 void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
 
diff --git a/mm/memory.c b/mm/memory.c
index 8d4d976311037..98c5ffd28a109 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -387,15 +387,14 @@ void free_pgd_range(struct mmu_gather *tlb,
  * The tree_max differs from the ceiling when a dup_mmap() failed and the tree
  * has unrelated data to the mm_struct being torn down.
  */
-void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
-		   struct vm_area_struct *vma, unsigned long floor,
-		   unsigned long ceiling, unsigned long tree_max,
-		   bool mm_wr_locked)
+void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)
 {
 	struct unlink_vma_file_batch vb;
+	struct ma_state *mas = desc->mas;
+	struct vm_area_struct *vma = desc->first;
 
 	/* underflow can happen and is fine */
-	WARN_ON_ONCE(tree_max - 1 > ceiling - 1);
+	WARN_ON_ONCE(desc->tree_max - 1 > desc->last_pgaddr - 1);
 
 	tlb_free_vmas(tlb);
 
@@ -407,13 +406,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
 		 * be 0.  This will underflow and is okay.
 		 */
-		next = mas_find(mas, tree_max - 1);
+		next = mas_find(mas, desc->tree_max - 1);
 
 		/*
 		 * Hide vma from rmap and truncate_pagecache before freeing
 		 * pgtables
 		 */
-		if (mm_wr_locked)
+		if (desc->mm_wr_locked)
 			vma_start_write(vma);
 		unlink_anon_vmas(vma);
 
@@ -425,16 +424,16 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		 */
 		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
 			vma = next;
-			next = mas_find(mas, tree_max - 1);
-			if (mm_wr_locked)
+			next = mas_find(mas, desc->tree_max - 1);
+			if (desc->mm_wr_locked)
 				vma_start_write(vma);
 			unlink_anon_vmas(vma);
 			unlink_file_vma_batch_add(&vb, vma);
 		}
 		unlink_file_vma_batch_final(&vb);
 
-		free_pgd_range(tlb, addr, vma->vm_end,
-			floor, next ? next->vm_start : ceiling);
+		free_pgd_range(tlb, addr, vma->vm_end, desc->first_pgaddr,
+			       next ? next->vm_start : desc->last_pgaddr);
 		vma = next;
 	} while (vma);
 }
diff --git a/mm/mmap.c b/mm/mmap.c
index 6011f62b0a294..9908481452780 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1311,10 +1311,10 @@ void exit_mmap(struct mm_struct *mm)
 	 */
 	mm_flags_set(MMF_OOM_SKIP, mm);
 	mmap_write_lock(mm);
+	unmap.mm_wr_locked = true;
 	mt_clear_in_rcu(&mm->mm_mt);
-	vma_iter_set(&vmi, vma->vm_end);
-	free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
-		      USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
+	vma_iter_set(&vmi, unmap.tree_reset);
+	free_pgtables(&tlb, &unmap);
 	tlb_finish_mmu(&tlb);
 
 	/*
diff --git a/mm/vma.c b/mm/vma.c
index ad64cd9795ef3..ba155a539d160 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -476,16 +476,13 @@ void remove_vma(struct vm_area_struct *vma)
 void unmap_region(struct unmap_desc *desc)
 {
 	struct mm_struct *mm = desc->first->vm_mm;
-	struct ma_state *mas = desc->mas;
 	struct mmu_gather tlb;
 
 	tlb_gather_mmu(&tlb, mm);
 	update_hiwater_rss(mm);
 	unmap_vmas(&tlb, desc);
-	mas_set(mas, desc->tree_reset);
-	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
-		      desc->last_pgaddr, desc->tree_max,
-		      desc->mm_wr_locked);
+	mas_set(desc->mas, desc->tree_reset);
+	free_pgtables(&tlb, desc);
 	tlb_finish_mmu(&tlb);
 }
 
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index d73ad4747d40a..435c5a24480bc 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -892,17 +892,10 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
 	(void)unmap;
 }
 
-static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
-		   struct vm_area_struct *vma, unsigned long floor,
-		   unsigned long ceiling, unsigned long tree_max,
-		   bool mm_wr_locked)
+static inline void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)
 {
 	(void)tlb;
-	(void)mas;
-	(void)vma;
-	(void)floor;
-	(void)ceiling;
-	(void)mm_wr_locked;
+	(void)desc;
 }
 
 static inline void mapping_unmap_writable(struct address_space *)
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:42 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:42 -0400
Subject: [PATCH v1 6/9] mm: Change dup_mmap() recovery
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-7-Liam.Howlett@oracle.com>

When the dup_mmap() fails during the vma duplication or setup, don't
write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
free the new resources, leaving an empty vma tree.

Using XA_ZERO introduced races where the vma could be found between
dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
race can occur because the mm can be reached through the other trees
via successfully copied vmas and other methods such as the swapoff code.

XA_ZERO was marking the location to stop vma removal and pagetable
freeing.  The newly created arguments to the unmap_vmas() and
free_pgtables() serve this function.

Replacing the XA_ZERO entry use with the new argument list also means
the checks for xa_is_zero() are no longer necessary so these are also
removed.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/memory.c |  6 +-----
 mm/mmap.c   | 42 +++++++++++++++++++++++++++++++-----------
 2 files changed, 32 insertions(+), 16 deletions(-)

diff --git a/mm/memory.c b/mm/memory.c
index 24716b3713f66..829cd94950182 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -408,8 +408,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		 * be 0.  This will underflow and is okay.
 		 */
 		next = mas_find(mas, tree_max - 1);
-		if (unlikely(xa_is_zero(next)))
-			next = NULL;
 
 		/*
 		 * Hide vma from rmap and truncate_pagecache before freeing
@@ -428,8 +426,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
 			vma = next;
 			next = mas_find(mas, tree_max - 1);
-			if (unlikely(xa_is_zero(next)))
-				next = NULL;
 			if (mm_wr_locked)
 				vma_start_write(vma);
 			unlink_anon_vmas(vma);
@@ -2129,7 +2125,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
 				 mm_wr_locked);
 		hugetlb_zap_end(vma, &details);
 		vma = mas_find(mas, tree_end - 1);
-	} while (vma && likely(!xa_is_zero(vma)));
+	} while (vma);
 	mmu_notifier_invalidate_range_end(&range);
 }
 
diff --git a/mm/mmap.c b/mm/mmap.c
index 0f4808f135fe6..aa4770b8d7f1e 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
 	arch_exit_mmap(mm);
 
 	vma = vma_next(&vmi);
-	if (!vma || unlikely(xa_is_zero(vma))) {
+	if (!vma) {
 		/* Can happen if dup_mmap() received an OOM */
 		mmap_read_unlock(mm);
 		mmap_write_lock(mm);
@@ -1858,20 +1858,40 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 		ksm_fork(mm, oldmm);
 		khugepaged_fork(mm, oldmm);
 	} else {
+		unsigned long max;
 
 		/*
-		 * The entire maple tree has already been duplicated. If the
-		 * mmap duplication fails, mark the failure point with
-		 * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
-		 * stop releasing VMAs that have not been duplicated after this
-		 * point.
+		 * The entire maple tree has already been duplicated, but
+		 * replacing the vmas failed at mpnt (which could be NULL if
+		 * all were allocated but the last vma was not fully set up).
+		 * Use the start address of the failure point to clean up the
+		 * partially initialized tree.
 		 */
-		if (mpnt) {
-			mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
-			mas_store(&vmi.mas, XA_ZERO_ENTRY);
-			/* Avoid OOM iterating a broken tree */
-			mm_flags_set(MMF_OOM_SKIP, mm);
+		if (!mm->map_count) {
+			/* zero vmas were written to the new tree. */
+			max = 0;
+		} else if (mpnt) {
+			/* partial tree failure */
+			max = mpnt->vm_start;
+		} else {
+			/* All vmas were written to the new tree */
+			max = ULONG_MAX;
 		}
+
+		/* Hide mm from oom killer because the memory is being freed */
+		mm_flags_set(MMF_OOM_SKIP, mm);
+		if (max) {
+			vma_iter_set(&vmi, 0);
+			tmp = vma_next(&vmi);
+			flush_cache_mm(mm);
+			unmap_region(&vmi.mas, /* vma = */ tmp,
+				     /*vma_min = */ 0, /* vma_max = */ max,
+				     /* pg_max = */ max, /* prev = */ NULL,
+				     /* next = */ NULL);
+			charge = tear_down_vmas(mm, &vmi, tmp, max);
+			vm_unacct_memory(charge);
+		}
+		__mt_destroy(&mm->mm_mt);
 		/*
 		 * The mm_struct is going to exit, but the locks will be dropped
 		 * first.  Set the mm_struct as unstable is advisable as it is
-- 
2.47.2



From Liam.Howlett at oracle.com  Tue Sep  9 12:09:43 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Tue,  9 Sep 2025 15:09:43 -0400
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce function arguments
In-Reply-To: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
Message-ID: <20250909190945.1030905-8-Liam.Howlett@oracle.com>

The unmap_region code uses a number of arguments that could use better
documentation.  With the addition of a descriptor for unmap (called
unmap_desc), the arguments can be more self-documenting and increase the
descriptions within the declaration.

No functional change intended

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
---
 mm/mmap.c | 12 ++++++++----
 mm/vma.c  | 27 ++++++++++++---------------
 mm/vma.h  | 35 ++++++++++++++++++++++++++++++++---
 3 files changed, 52 insertions(+), 22 deletions(-)

diff --git a/mm/mmap.c b/mm/mmap.c
index aa4770b8d7f1e..5c9bd3f20e53f 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1883,11 +1883,15 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 		if (max) {
 			vma_iter_set(&vmi, 0);
 			tmp = vma_next(&vmi);
+			UNMAP_REGION(unmap, &vmi, /* first vma = */ tmp,
+				     /* min vma addr = */ 0,
+				     /* max vma addr = */ max,
+				     /* prev = */ NULL, /* next = */ NULL);
+
+			/* Don't free the pgtables higher than the failure */
+			unmap.tree_max = max;
 			flush_cache_mm(mm);
-			unmap_region(&vmi.mas, /* vma = */ tmp,
-				     /*vma_min = */ 0, /* vma_max = */ max,
-				     /* pg_max = */ max, /* prev = */ NULL,
-				     /* next = */ NULL);
+			unmap_region(&unmap);
 			charge = tear_down_vmas(mm, &vmi, tmp, max);
 			vm_unacct_memory(charge);
 		}
diff --git a/mm/vma.c b/mm/vma.c
index 4c850ffd83a4b..c92384975cbb2 100644
--- a/mm/vma.c
+++ b/mm/vma.c
@@ -473,22 +473,20 @@ void remove_vma(struct vm_area_struct *vma)
  *
  * Called with the mm semaphore held.
  */
-void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
-		unsigned long vma_min, unsigned long vma_max, unsigned long pg_max,
-		struct vm_area_struct *prev, struct vm_area_struct *next)
+void unmap_region(struct unmap_desc *desc)
 {
-	struct mm_struct *mm = vma->vm_mm;
+	struct mm_struct *mm = desc->first->vm_mm;
+	struct ma_state *mas = desc->mas;
 	struct mmu_gather tlb;
 
 	tlb_gather_mmu(&tlb, mm);
 	update_hiwater_rss(mm);
-	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
-		   /* mm_wr_locked = */ true);
-	mas_set(mas, vma->vm_end);
-	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
-		      next ? next->vm_start : USER_PGTABLES_CEILING,
-		      pg_max,
-		      /* mm_wr_locked = */ true);
+	unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
+		   desc->vma_max, desc->mm_wr_locked);
+	mas_set(mas, desc->tree_reset);
+	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
+		      desc->last_pgaddr, desc->tree_max,
+		      desc->mm_wr_locked);
 	tlb_finish_mmu(&tlb);
 }
 
@@ -2414,15 +2412,14 @@ static int __mmap_new_file_vma(struct mmap_state *map,
 
 	error = mmap_file(vma->vm_file, vma);
 	if (error) {
+		UNMAP_REGION(unmap, vmi, vma, vma->vm_start, vma->vm_end,
+			     map->prev, map->next);
 		fput(vma->vm_file);
 		vma->vm_file = NULL;
 
 		vma_iter_set(vmi, vma->vm_end);
 		/* Undo any partial mapping done by a device driver. */
-		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
-			     map->next ? map->next->vm_start : USER_PGTABLES_CEILING,
-			     map->prev, map->next);
-
+		unmap_region(&unmap);
 		return error;
 	}
 
diff --git a/mm/vma.h b/mm/vma.h
index b0ebc81d5862e..4edd5d26ffcfc 100644
--- a/mm/vma.h
+++ b/mm/vma.h
@@ -152,6 +152,37 @@ struct vma_merge_struct {
 
 };
 
+struct unmap_desc {
+	struct  ma_state *mas;        /* the maple state point to the first vma */
+	struct vm_area_struct *first; /* The first vma */
+	unsigned long first_pgaddr;   /* The first pagetable address to free */
+	unsigned long last_pgaddr;    /* The last pagetable address to free */
+	unsigned long vma_min;        /* The min vma address */
+	unsigned long vma_max;        /* The max vma address */
+	unsigned long tree_max;       /* Maximum for the vma tree search */
+	unsigned long tree_reset;     /* Where to reset the vma tree walk */
+	bool mm_wr_locked;            /* If the mmap write lock is held */
+};
+
+#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
+	struct unmap_desc name = {                                          \
+		.mas = &(_vmi)->mas,                                          \
+		.first = _vma,                                                \
+		.first_pgaddr = _prev ?                                       \
+			((struct vm_area_struct *)_prev)->vm_end :            \
+			FIRST_USER_ADDRESS,                                   \
+		.last_pgaddr = _next ?                                        \
+			((struct vm_area_struct *)_next)->vm_start :          \
+			USER_PGTABLES_CEILING,                                \
+		.vma_min = _vma_min,                                          \
+		.vma_max = _vma_max,                                          \
+		.tree_max = _next ?                                           \
+			((struct vm_area_struct *)_next)->vm_start :          \
+			USER_PGTABLES_CEILING,                                \
+		.tree_reset = _vma->vm_end,                                   \
+		.mm_wr_locked = true,                                         \
+	}
+
 static inline bool vmg_nomem(struct vma_merge_struct *vmg)
 {
 	return vmg->state == VMA_MERGE_ERROR_NOMEM;
@@ -260,9 +291,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
 
 void remove_vma(struct vm_area_struct *vma);
 
-void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
-		unsigned long min, unsigned long max, unsigned long pg_max,
-		struct vm_area_struct *prev, struct vm_area_struct *next);
+void unmap_region(struct unmap_desc *desc);
 
 /* We are about to modify the VMA's flags. */
 __must_check struct vm_area_struct
-- 
2.47.2



From surenb at google.com  Tue Sep  9 13:07:57 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 13:07:57 -0700
Subject: [PATCH v1 1/9] mm/mmap: Move exit_mmap() trace point
In-Reply-To: <20250909190945.1030905-2-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-2-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpGsGvYfCQmx1ix86zUM-u8n2EbSATS5bcHMs9=_yEtRzg@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:09?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> Move the trace point later in the function so that it is not skipped in
> the event of a failed fork.
>
> Acked-by: Chris Li <chrisl at kernel.org>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/mmap.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 5fd3b80fda1d5..b07b3ec5e28f5 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1310,9 +1310,9 @@ void exit_mmap(struct mm_struct *mm)
>
>         BUG_ON(count != mm->map_count);
>
> -       trace_exit_mmap(mm);
>  destroy:
>         __mt_destroy(&mm->mm_mt);
> +       trace_exit_mmap(mm);
>         mmap_write_unlock(mm);
>         vm_unacct_memory(nr_accounted);
>  }
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 13:09:24 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 13:09:24 -0700
Subject: [PATCH v1 2/9] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <20250909190945.1030905-3-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-3-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpFOt3_W+YQNy+xPzY_R6JUM3EOsbaZh+uznSNpF1RKX0Q@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:09?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> Create the new function tear_down_vmas() to remove a range of vmas.
> exit_mmap() will be removing all the vmas.
>
> This is necessary for future patches.
>
> No functional changes intended.
>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/mmap.c | 37 ++++++++++++++++++++++++-------------
>  1 file changed, 24 insertions(+), 13 deletions(-)
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index b07b3ec5e28f5..a290448a53bb2 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
>  }
>  EXPORT_SYMBOL(vm_brk_flags);
>
> +static inline

nit: Maybe let the compiler decide whether to inline this one?

> +unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
> +               struct vm_area_struct *vma, unsigned long max)
> +{
> +       unsigned long nr_accounted = 0;
> +       int count = 0;
> +
> +       mmap_assert_write_locked(mm);
> +       vma_iter_set(vmi, vma->vm_end);
> +       do {
> +               if (vma->vm_flags & VM_ACCOUNT)
> +                       nr_accounted += vma_pages(vma);
> +               vma_mark_detached(vma);
> +               remove_vma(vma);
> +               count++;
> +               cond_resched();
> +               vma = vma_next(vmi);
> +       } while (vma && vma->vm_end <= max);
> +
> +       WARN_ON_ONCE(count != mm->map_count);
> +       return nr_accounted;
> +}
> +
>  /* Release all mmaps. */
>  void exit_mmap(struct mm_struct *mm)
>  {
> @@ -1257,7 +1280,6 @@ void exit_mmap(struct mm_struct *mm)
>         struct vm_area_struct *vma;
>         unsigned long nr_accounted = 0;
>         VMA_ITERATOR(vmi, mm, 0);
> -       int count = 0;
>
>         /* mm's last user has gone, and its about to be pulled down */
>         mmu_notifier_release(mm);
> @@ -1297,18 +1319,7 @@ void exit_mmap(struct mm_struct *mm)
>          * enabled, without holding any MM locks besides the unreachable
>          * mmap_write_lock.
>          */
> -       vma_iter_set(&vmi, vma->vm_end);
> -       do {
> -               if (vma->vm_flags & VM_ACCOUNT)
> -                       nr_accounted += vma_pages(vma);
> -               vma_mark_detached(vma);
> -               remove_vma(vma);
> -               count++;
> -               cond_resched();
> -               vma = vma_next(&vmi);
> -       } while (vma && likely(!xa_is_zero(vma)));
> -
> -       BUG_ON(count != mm->map_count);
> +       nr_accounted = tear_down_vmas(mm, &vmi, vma, ULONG_MAX);
>
>  destroy:
>         __mt_destroy(&mm->mm_mt);
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 13:09:50 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 13:09:50 -0700
Subject: [PATCH v1 3/9] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <20250909190945.1030905-4-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-4-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpGSX3VMnFH3u67mKZTe1pXOYXoFc92XiU+TVjPs==5mcA@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> Add a limit to the vma search instead of using the start and end of the
> one passed in.

Would be good to explain why this is needed. I'm sure I'll find the
answer in the later patches though.


>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/vma.c | 6 ++++--
>  mm/vma.h | 1 +
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vma.c b/mm/vma.c
> index abe0da33c8446..a648e0555c873 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
>   * Called with the mm semaphore held.
>   */
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +               unsigned long vma_min, unsigned long vma_max,
>                 struct vm_area_struct *prev, struct vm_area_struct *next)
>  {
>         struct mm_struct *mm = vma->vm_mm;
> @@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>
>         tlb_gather_mmu(&tlb, mm);
>         update_hiwater_rss(mm);
> -       unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
> +       unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
>                    /* mm_wr_locked = */ true);
>         mas_set(mas, vma->vm_end);
>         free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> @@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>
>                 vma_iter_set(vmi, vma->vm_end);
>                 /* Undo any partial mapping done by a device driver. */
> -               unmap_region(&vmi->mas, vma, map->prev, map->next);
> +               unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> +                            map->prev, map->next);
>
>                 return error;
>         }
> diff --git a/mm/vma.h b/mm/vma.h
> index 9183fe5490090..a9d0cef684ddb 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -261,6 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>  void remove_vma(struct vm_area_struct *vma);
>
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +               unsigned long min, unsigned long max,
>                 struct vm_area_struct *prev, struct vm_area_struct *next);
>
>  /* We are about to modify the VMA's flags. */
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 14:05:47 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 14:05:47 -0700
Subject: [PATCH v1 4/9] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <20250909190945.1030905-5-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-5-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpEqVNyYtEabhDc4DqHbgJdYXcMvryvfxmXU8uJGe0FdRA@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> The ceiling and tree search limit need to be different arguments for the
> future change in the failed fork attempt.
>
> Add some documentation around free_pgtables() and the limits in an
> attempt to clarify the floor and ceiling use as well as the new
> tree_max.
>
> Test code also updated.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  mm/internal.h                    |  4 +++-
>  mm/memory.c                      | 28 +++++++++++++++++++++++++---
>  mm/mmap.c                        |  2 +-
>  mm/vma.c                         |  3 ++-
>  tools/testing/vma/vma_internal.h |  3 ++-
>  5 files changed, 33 insertions(+), 7 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 63e3ec8d63be7..d295252407fee 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -444,7 +444,9 @@ void folio_activate(struct folio *folio);
>
>  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                    struct vm_area_struct *start_vma, unsigned long floor,
> -                  unsigned long ceiling, bool mm_wr_locked);
> +                  unsigned long ceiling, unsigned long tree_max,
> +                  bool mm_wr_locked);
> +
>  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
>
>  struct zap_details;
> diff --git a/mm/memory.c b/mm/memory.c
> index 3e0404bd57a02..24716b3713f66 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -369,12 +369,34 @@ void free_pgd_range(struct mmu_gather *tlb,
>         } while (pgd++, addr = next, addr != end);
>  }
>
> +/*
> + * free_pgtables() - Free a range of page tables
> + * @tlb: The mmu gather
> + * @mas: The maple state
> + * @vma: The first vma
> + * @floor: The lowest page table address
> + * @ceiling: The highest page table address
> + * @tree_max: The highest tree search address
> + * @mm_wr_locked: boolean indicating if the mm is write locked
> + *
> + * Note: Floor and ceiling are provided to indicate the absolute range of the
> + * page tables that should be removed.  This can differ from the vma mappings on
> + * some archs that may have mappings that need to be removed outside the vmas.
> + * Note that the prev->vm_end and next->vm_start are often used.
> + *
> + * The tree_max differs from the ceiling when a dup_mmap() failed and the tree
> + * has unrelated data to the mm_struct being torn down.
> + */
>  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                    struct vm_area_struct *vma, unsigned long floor,
> -                  unsigned long ceiling, bool mm_wr_locked)
> +                  unsigned long ceiling, unsigned long tree_max,
> +                  bool mm_wr_locked)
>  {
>         struct unlink_vma_file_batch vb;
>
> +       /* underflow can happen and is fine */

This comment is a bit confusing... I think the below check covers 2 cases:
1. if ceiling == 0 then tree_max can be anything;
2. if ceiling > 0 then tree_max <= ceiling;
Is that what you intended? If so, maybe amend the comments for the
free_pgtables() function explaining this more explicitly?

> +       WARN_ON_ONCE(tree_max - 1 > ceiling - 1);

I would prefer WARN_ON_ONCE(ceiling && tree_max > ceiling); as more
descriptive but that might be just me.

> +
>         tlb_free_vmas(tlb);
>
>         do {
> @@ -385,7 +407,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                  * Note: USER_PGTABLES_CEILING may be passed as ceiling and may

You replaced ceiling with tree_max, so the above comment needs to be
updated as well.

>                  * be 0.  This will underflow and is okay.
>                  */
> -               next = mas_find(mas, ceiling - 1);
> +               next = mas_find(mas, tree_max - 1);
>                 if (unlikely(xa_is_zero(next)))
>                         next = NULL;
>
> @@ -405,7 +427,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                  */
>                 while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>                         vma = next;
> -                       next = mas_find(mas, ceiling - 1);
> +                       next = mas_find(mas, tree_max - 1);
>                         if (unlikely(xa_is_zero(next)))
>                                 next = NULL;
>                         if (mm_wr_locked)
> diff --git a/mm/mmap.c b/mm/mmap.c
> index a290448a53bb2..0f4808f135fe6 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1311,7 +1311,7 @@ void exit_mmap(struct mm_struct *mm)
>         mt_clear_in_rcu(&mm->mm_mt);
>         vma_iter_set(&vmi, vma->vm_end);
>         free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
> -                     USER_PGTABLES_CEILING, true);
> +                     USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
>         tlb_finish_mmu(&tlb);
>
>         /*
> diff --git a/mm/vma.c b/mm/vma.c
> index a648e0555c873..1bae142bbc0f1 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -486,6 +486,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>                    /* mm_wr_locked = */ true);
>         mas_set(mas, vma->vm_end);
>         free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> +                     next ? next->vm_start : USER_PGTABLES_CEILING,
>                       next ? next->vm_start : USER_PGTABLES_CEILING,
>                       /* mm_wr_locked = */ true);
>         tlb_finish_mmu(&tlb);
> @@ -1232,7 +1233,7 @@ static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
>         mas_set(mas_detach, 1);
>         /* start and end may be different if there is no prev or next vma. */
>         free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
> -                     vms->unmap_end, mm_wr_locked);
> +                     vms->unmap_end, vms->unmap_end, mm_wr_locked);
>         tlb_finish_mmu(&tlb);
>         vms->clear_ptes = false;
>  }
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index 07167446dcf42..823d379e1fac2 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -900,7 +900,8 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>
>  static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                    struct vm_area_struct *vma, unsigned long floor,
> -                  unsigned long ceiling, bool mm_wr_locked)
> +                  unsigned long ceiling, unsigned long tree_max,
> +                  bool mm_wr_locked)
>  {
>         (void)tlb;
>         (void)mas;
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 14:29:23 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 14:29:23 -0700
Subject: [PATCH v1 5/9] mm/vma: Add page table limit to unmap_region()
In-Reply-To: <20250909190945.1030905-6-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-6-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpG6US+wPA7V5-0Dkho+BRs6uNUCYCV1GyRrbvpzC1N=sg@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> The unmap_region() calls need to pass through the page table limit for a
> future patch.
>
> No functional changes intended.
>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  mm/vma.c | 5 +++--
>  mm/vma.h | 2 +-
>  2 files changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/mm/vma.c b/mm/vma.c
> index 1bae142bbc0f1..4c850ffd83a4b 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -474,7 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
>   * Called with the mm semaphore held.
>   */
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -               unsigned long vma_min, unsigned long vma_max,
> +               unsigned long vma_min, unsigned long vma_max, unsigned long pg_max,
>                 struct vm_area_struct *prev, struct vm_area_struct *next)
>  {
>         struct mm_struct *mm = vma->vm_mm;
> @@ -487,7 +487,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>         mas_set(mas, vma->vm_end);
>         free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
>                       next ? next->vm_start : USER_PGTABLES_CEILING,
> -                     next ? next->vm_start : USER_PGTABLES_CEILING,
> +                     pg_max,

Hmm. The free_pgtables() parameters were advertised as:

@floor: The lowest page table address
@ceiling: The highest page table address
@tree_max: The highest tree search address

but here tree_max=pg_max. I would expect pg_max to mean "The highest
page table address", IOW we should have ceiling=pg_max. Either the
order of the parameters is wrong here or the names are misleading.
I also think in the previous patch we should have renamed
free_pgtables() parameters to vma_min, vma_max and pg_max like here
for consistency and to avoid confusion.

>                       /* mm_wr_locked = */ true);
>         tlb_finish_mmu(&tlb);
>  }
> @@ -2420,6 +2420,7 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>                 vma_iter_set(vmi, vma->vm_end);
>                 /* Undo any partial mapping done by a device driver. */
>                 unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> +                            map->next ? map->next->vm_start : USER_PGTABLES_CEILING,
>                              map->prev, map->next);
>
>                 return error;
> diff --git a/mm/vma.h b/mm/vma.h
> index a9d0cef684ddb..b0ebc81d5862e 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -261,7 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>  void remove_vma(struct vm_area_struct *vma);
>
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -               unsigned long min, unsigned long max,
> +               unsigned long min, unsigned long max, unsigned long pg_max,
>                 struct vm_area_struct *prev, struct vm_area_struct *next);
>
>  /* We are about to modify the VMA's flags. */
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 14:44:05 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 14:44:05 -0700
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce function arguments
In-Reply-To: <20250909190945.1030905-8-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-8-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> The unmap_region code uses a number of arguments that could use better
> documentation.  With the addition of a descriptor for unmap (called
> unmap_desc), the arguments can be more self-documenting and increase the
> descriptions within the declaration.
>
> No functional change intended
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  mm/mmap.c | 12 ++++++++----
>  mm/vma.c  | 27 ++++++++++++---------------
>  mm/vma.h  | 35 ++++++++++++++++++++++++++++++++---
>  3 files changed, 52 insertions(+), 22 deletions(-)
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index aa4770b8d7f1e..5c9bd3f20e53f 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1883,11 +1883,15 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
>                 if (max) {
>                         vma_iter_set(&vmi, 0);
>                         tmp = vma_next(&vmi);
> +                       UNMAP_REGION(unmap, &vmi, /* first vma = */ tmp,
> +                                    /* min vma addr = */ 0,
> +                                    /* max vma addr = */ max,
> +                                    /* prev = */ NULL, /* next = */ NULL);
> +
> +                       /* Don't free the pgtables higher than the failure */
> +                       unmap.tree_max = max;

Sorry, the naming still feels confusing... tree_max is described as /*
Maximum for the vma tree search */ and here we set it to /* Don't free
the pgtables higher than the failure */.

>                         flush_cache_mm(mm);
> -                       unmap_region(&vmi.mas, /* vma = */ tmp,
> -                                    /*vma_min = */ 0, /* vma_max = */ max,
> -                                    /* pg_max = */ max, /* prev = */ NULL,
> -                                    /* next = */ NULL);
> +                       unmap_region(&unmap);
>                         charge = tear_down_vmas(mm, &vmi, tmp, max);
>                         vm_unacct_memory(charge);
>                 }
> diff --git a/mm/vma.c b/mm/vma.c
> index 4c850ffd83a4b..c92384975cbb2 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -473,22 +473,20 @@ void remove_vma(struct vm_area_struct *vma)
>   *
>   * Called with the mm semaphore held.
>   */
> -void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -               unsigned long vma_min, unsigned long vma_max, unsigned long pg_max,
> -               struct vm_area_struct *prev, struct vm_area_struct *next)
> +void unmap_region(struct unmap_desc *desc)
>  {
> -       struct mm_struct *mm = vma->vm_mm;
> +       struct mm_struct *mm = desc->first->vm_mm;
> +       struct ma_state *mas = desc->mas;
>         struct mmu_gather tlb;
>
>         tlb_gather_mmu(&tlb, mm);
>         update_hiwater_rss(mm);
> -       unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
> -                  /* mm_wr_locked = */ true);
> -       mas_set(mas, vma->vm_end);
> -       free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> -                     next ? next->vm_start : USER_PGTABLES_CEILING,
> -                     pg_max,
> -                     /* mm_wr_locked = */ true);
> +       unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
> +                  desc->vma_max, desc->mm_wr_locked);
> +       mas_set(mas, desc->tree_reset);
> +       free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
> +                     desc->last_pgaddr, desc->tree_max,
> +                     desc->mm_wr_locked);
>         tlb_finish_mmu(&tlb);
>  }
>
> @@ -2414,15 +2412,14 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>
>         error = mmap_file(vma->vm_file, vma);
>         if (error) {
> +               UNMAP_REGION(unmap, vmi, vma, vma->vm_start, vma->vm_end,
> +                            map->prev, map->next);
>                 fput(vma->vm_file);
>                 vma->vm_file = NULL;
>
>                 vma_iter_set(vmi, vma->vm_end);
>                 /* Undo any partial mapping done by a device driver. */
> -               unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> -                            map->next ? map->next->vm_start : USER_PGTABLES_CEILING,
> -                            map->prev, map->next);
> -
> +               unmap_region(&unmap);
>                 return error;
>         }
>
> diff --git a/mm/vma.h b/mm/vma.h
> index b0ebc81d5862e..4edd5d26ffcfc 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -152,6 +152,37 @@ struct vma_merge_struct {
>
>  };
>
> +struct unmap_desc {
> +       struct  ma_state *mas;        /* the maple state point to the first vma */
> +       struct vm_area_struct *first; /* The first vma */
> +       unsigned long first_pgaddr;   /* The first pagetable address to free */
> +       unsigned long last_pgaddr;    /* The last pagetable address to free */
> +       unsigned long vma_min;        /* The min vma address */
> +       unsigned long vma_max;        /* The max vma address */
> +       unsigned long tree_max;       /* Maximum for the vma tree search */
> +       unsigned long tree_reset;     /* Where to reset the vma tree walk */
> +       bool mm_wr_locked;            /* If the mmap write lock is held */
> +};
> +
> +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \

Maybe DEFINE_UNMAP_REGION() similar to DEFINE_PER_CPU() or DEFINE_SPINLOCK()?

> +       struct unmap_desc name = {                                          \
> +               .mas = &(_vmi)->mas,                                          \
> +               .first = _vma,                                                \
> +               .first_pgaddr = _prev ?                                       \
> +                       ((struct vm_area_struct *)_prev)->vm_end :            \
> +                       FIRST_USER_ADDRESS,                                   \
> +               .last_pgaddr = _next ?                                        \
> +                       ((struct vm_area_struct *)_next)->vm_start :          \
> +                       USER_PGTABLES_CEILING,                                \
> +               .vma_min = _vma_min,                                          \
> +               .vma_max = _vma_max,                                          \
> +               .tree_max = _next ?                                           \
> +                       ((struct vm_area_struct *)_next)->vm_start :          \
> +                       USER_PGTABLES_CEILING,                                \
> +               .tree_reset = _vma->vm_end,                                   \
> +               .mm_wr_locked = true,                                         \
> +       }
> +
>  static inline bool vmg_nomem(struct vma_merge_struct *vmg)
>  {
>         return vmg->state == VMA_MERGE_ERROR_NOMEM;
> @@ -260,9 +291,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>
>  void remove_vma(struct vm_area_struct *vma);
>
> -void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -               unsigned long min, unsigned long max, unsigned long pg_max,
> -               struct vm_area_struct *prev, struct vm_area_struct *next);
> +void unmap_region(struct unmap_desc *desc);
>
>  /* We are about to modify the VMA's flags. */
>  __must_check struct vm_area_struct
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 15:03:38 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 15:03:38 -0700
Subject: [PATCH v1 6/9] mm: Change dup_mmap() recovery
In-Reply-To: <20250909190945.1030905-7-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-7-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpHz9Q-GMSZQ0xY8_oAuoWTA6DUHfCses7e7_FL8mi_1bQ@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> When the dup_mmap() fails during the vma duplication or setup, don't
> write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> free the new resources, leaving an empty vma tree.
>
> Using XA_ZERO introduced races where the vma could be found between
> dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> race can occur because the mm can be reached through the other trees
> via successfully copied vmas and other methods such as the swapoff code.
>
> XA_ZERO was marking the location to stop vma removal and pagetable
> freeing.  The newly created arguments to the unmap_vmas() and
> free_pgtables() serve this function.
>
> Replacing the XA_ZERO entry use with the new argument list also means
> the checks for xa_is_zero() are no longer necessary so these are also
> removed.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/memory.c |  6 +-----
>  mm/mmap.c   | 42 +++++++++++++++++++++++++++++++-----------
>  2 files changed, 32 insertions(+), 16 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 24716b3713f66..829cd94950182 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -408,8 +408,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                  * be 0.  This will underflow and is okay.
>                  */
>                 next = mas_find(mas, tree_max - 1);
> -               if (unlikely(xa_is_zero(next)))
> -                       next = NULL;
>
>                 /*
>                  * Hide vma from rmap and truncate_pagecache before freeing
> @@ -428,8 +426,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                 while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>                         vma = next;
>                         next = mas_find(mas, tree_max - 1);
> -                       if (unlikely(xa_is_zero(next)))
> -                               next = NULL;
>                         if (mm_wr_locked)
>                                 vma_start_write(vma);
>                         unlink_anon_vmas(vma);
> @@ -2129,7 +2125,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>                                  mm_wr_locked);
>                 hugetlb_zap_end(vma, &details);
>                 vma = mas_find(mas, tree_end - 1);
> -       } while (vma && likely(!xa_is_zero(vma)));
> +       } while (vma);
>         mmu_notifier_invalidate_range_end(&range);
>  }
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 0f4808f135fe6..aa4770b8d7f1e 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
>         arch_exit_mmap(mm);
>
>         vma = vma_next(&vmi);
> -       if (!vma || unlikely(xa_is_zero(vma))) {
> +       if (!vma) {
>                 /* Can happen if dup_mmap() received an OOM */
>                 mmap_read_unlock(mm);
>                 mmap_write_lock(mm);
> @@ -1858,20 +1858,40 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
>                 ksm_fork(mm, oldmm);
>                 khugepaged_fork(mm, oldmm);
>         } else {
> +               unsigned long max;
>
>                 /*
> -                * The entire maple tree has already been duplicated. If the
> -                * mmap duplication fails, mark the failure point with
> -                * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
> -                * stop releasing VMAs that have not been duplicated after this
> -                * point.
> +                * The entire maple tree has already been duplicated, but
> +                * replacing the vmas failed at mpnt (which could be NULL if
> +                * all were allocated but the last vma was not fully set up).
> +                * Use the start address of the failure point to clean up the
> +                * partially initialized tree.
>                  */
> -               if (mpnt) {
> -                       mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
> -                       mas_store(&vmi.mas, XA_ZERO_ENTRY);
> -                       /* Avoid OOM iterating a broken tree */
> -                       mm_flags_set(MMF_OOM_SKIP, mm);
> +               if (!mm->map_count) {
> +                       /* zero vmas were written to the new tree. */
> +                       max = 0;
> +               } else if (mpnt) {
> +                       /* partial tree failure */
> +                       max = mpnt->vm_start;
> +               } else {
> +                       /* All vmas were written to the new tree */

So, the cleanup for this case used to be handled by exit_mmap(). I
think it's ok to do it here but the changelog should mention this
change as well IMHO.

> +                       max = ULONG_MAX;
>                 }
> +
> +               /* Hide mm from oom killer because the memory is being freed */
> +               mm_flags_set(MMF_OOM_SKIP, mm);
> +               if (max) {
> +                       vma_iter_set(&vmi, 0);
> +                       tmp = vma_next(&vmi);
> +                       flush_cache_mm(mm);
> +                       unmap_region(&vmi.mas, /* vma = */ tmp,
> +                                    /*vma_min = */ 0, /* vma_max = */ max,
> +                                    /* pg_max = */ max, /* prev = */ NULL,
> +                                    /* next = */ NULL);
> +                       charge = tear_down_vmas(mm, &vmi, tmp, max);
> +                       vm_unacct_memory(charge);
> +               }
> +               __mt_destroy(&mm->mm_mt);
>                 /*
>                  * The mm_struct is going to exit, but the locks will be dropped
>                  * first.  Set the mm_struct as unstable is advisable as it is
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 15:16:16 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 15:16:16 -0700
Subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and exit_mmap()
In-Reply-To: <20250909190945.1030905-9-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-9-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpFGo4JXQQGECFdCgNNVaQc5MDLghdmnVXEK8Zfag7KHiw@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> vms_clear_ptes() is slightly different than other callers to
> unmap_region() and so had the unmapping open-coded.  Using the new
> structure it is now possible to special-case the struct setup instead of
> having the open-coded function.
>
> exit_mmap() also calls unmap_vmas() with many arguemnts.  Using the
> unmap_all_init() function to set the unmap descriptor for all vmas makes
> this a bit easier to read.
>
> Update to the vma test code is necessary to ensure testing continues to
> function.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  include/linux/mm.h               |  3 ---
>  mm/internal.h                    |  3 +++
>  mm/memory.c                      | 24 ++++++++------------
>  mm/mmap.c                        |  5 +++-
>  mm/vma.c                         | 39 ++++++++++++++++++--------------
>  mm/vma.h                         | 14 ++++++++++++
>  tools/testing/vma/vma_internal.h | 14 ++++--------
>  7 files changed, 56 insertions(+), 46 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 892fe5dbf9de0..23eb59d543390 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2450,9 +2450,6 @@ static inline void zap_vma_pages(struct vm_area_struct *vma)
>         zap_page_range_single(vma, vma->vm_start,
>                               vma->vm_end - vma->vm_start, NULL);
>  }
> -void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -               struct vm_area_struct *start_vma, unsigned long start,
> -               unsigned long end, unsigned long tree_end, bool mm_wr_locked);
>
>  struct mmu_notifier_range;
>
> diff --git a/mm/internal.h b/mm/internal.h
> index d295252407fee..1239944f2830a 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -197,6 +197,9 @@ static inline void vma_close(struct vm_area_struct *vma)
>         }
>  }
>
> +/* unmap_vmas is in mm/memory.c */
> +void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);
> +
>  #ifdef CONFIG_MMU
>
>  /* Flags for folio_pte_batch(). */
> diff --git a/mm/memory.c b/mm/memory.c
> index 829cd94950182..8d4d976311037 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2084,12 +2084,7 @@ static void unmap_single_vma(struct mmu_gather *tlb,
>  /**
>   * unmap_vmas - unmap a range of memory covered by a list of vma's
>   * @tlb: address of the caller's struct mmu_gather
> - * @mas: the maple state
> - * @vma: the starting vma
> - * @start_addr: virtual address at which to start unmapping
> - * @end_addr: virtual address at which to end unmapping
> - * @tree_end: The maximum index to check
> - * @mm_wr_locked: lock flag
> + * @unmap: The unmap_desc
>   *
>   * Unmap all pages in the vma list.
>   *
> @@ -2102,11 +2097,9 @@ static void unmap_single_vma(struct mmu_gather *tlb,
>   * ensure that any thus-far unmapped pages are flushed before unmap_vmas()
>   * drops the lock and schedules.
>   */
> -void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -               struct vm_area_struct *vma, unsigned long start_addr,
> -               unsigned long end_addr, unsigned long tree_end,
> -               bool mm_wr_locked)
> +void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>  {
> +       struct vm_area_struct *vma;
>         struct mmu_notifier_range range;
>         struct zap_details details = {
>                 .zap_flags = ZAP_FLAG_DROP_MARKER | ZAP_FLAG_UNMAP,
> @@ -2114,17 +2107,18 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>                 .even_cows = true,
>         };
>
> +       vma = unmap->first;
>         mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma->vm_mm,
> -                               start_addr, end_addr);
> +                               unmap->vma_min, unmap->vma_max);
>         mmu_notifier_invalidate_range_start(&range);
>         do {
> -               unsigned long start = start_addr;
> -               unsigned long end = end_addr;
> +               unsigned long start = unmap->vma_min;
> +               unsigned long end = unmap->vma_max;
>                 hugetlb_zap_begin(vma, &start, &end);
>                 unmap_single_vma(tlb, vma, start, end, &details,
> -                                mm_wr_locked);
> +                                unmap->mm_wr_locked);
>                 hugetlb_zap_end(vma, &details);
> -               vma = mas_find(mas, tree_end - 1);
> +               vma = mas_find(unmap->mas, unmap->tree_max - 1);
>         } while (vma);
>         mmu_notifier_invalidate_range_end(&range);
>  }
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 5c9bd3f20e53f..6011f62b0a294 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1280,10 +1280,12 @@ void exit_mmap(struct mm_struct *mm)
>         struct vm_area_struct *vma;
>         unsigned long nr_accounted = 0;
>         VMA_ITERATOR(vmi, mm, 0);
> +       struct unmap_desc unmap;
>
>         /* mm's last user has gone, and its about to be pulled down */
>         mmu_notifier_release(mm);
>
> +       unmap.mm_wr_locked = false;

This will be reset by unmap_all_init() anyway, right?

>         mmap_read_lock(mm);
>         arch_exit_mmap(mm);
>
> @@ -1295,11 +1297,12 @@ void exit_mmap(struct mm_struct *mm)
>                 goto destroy;
>         }
>
> +       unmap_all_init(&unmap, &vmi, vma);

Can we use a macro, something like DEFINE_UNMAP_ALL_REGIONS() instead
of unmap_all_init()?

>         flush_cache_mm(mm);
>         tlb_gather_mmu_fullmm(&tlb, mm);
>         /* update_hiwater_rss(mm) here? but nobody should be looking */
>         /* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */
> -       unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false);
> +       unmap_vmas(&tlb, &unmap);
>         mmap_read_unlock(mm);
>
>         /*
> diff --git a/mm/vma.c b/mm/vma.c
> index c92384975cbb2..ad64cd9795ef3 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -481,8 +481,7 @@ void unmap_region(struct unmap_desc *desc)
>
>         tlb_gather_mmu(&tlb, mm);
>         update_hiwater_rss(mm);
> -       unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
> -                  desc->vma_max, desc->mm_wr_locked);
> +       unmap_vmas(&tlb, desc);
>         mas_set(mas, desc->tree_reset);
>         free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
>                       desc->last_pgaddr, desc->tree_max,
> @@ -1213,26 +1212,32 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
>  static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
>                     struct ma_state *mas_detach, bool mm_wr_locked)
>  {
> -       struct mmu_gather tlb;
> +       struct unmap_desc unmap = {
> +               .mas = mas_detach,
> +               .first = vms->vma,
> +               /* start and end may be different if there is no prev or next vma. */
> +               .first_pgaddr = vms->unmap_start,
> +               .last_pgaddr = vms->unmap_end,
> +               .vma_min = vms->start,
> +               .vma_max = vms->end,
> +               /*
> +                * The tree limits and reset differ from the normal case since it's a
> +                * side-tree
> +                */
> +               .tree_reset = 1,
> +               .tree_max = vms->vma_count,
> +               /*
> +                * We can free page tables without write-locking mmap_lock because VMAs
> +                * were isolated before we downgraded mmap_lock.
> +                */
> +               .mm_wr_locked = mm_wr_locked,
> +       };
>
>         if (!vms->clear_ptes) /* Nothing to do */
>                 return;
>
> -       /*
> -        * We can free page tables without write-locking mmap_lock because VMAs
> -        * were isolated before we downgraded mmap_lock.
> -        */
>         mas_set(mas_detach, 1);
> -       tlb_gather_mmu(&tlb, vms->vma->vm_mm);
> -       update_hiwater_rss(vms->vma->vm_mm);
> -       unmap_vmas(&tlb, mas_detach, vms->vma, vms->start, vms->end,
> -                  vms->vma_count, mm_wr_locked);
> -
> -       mas_set(mas_detach, 1);
> -       /* start and end may be different if there is no prev or next vma. */
> -       free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
> -                     vms->unmap_end, vms->unmap_end, mm_wr_locked);
> -       tlb_finish_mmu(&tlb);
> +       unmap_region(&unmap);
>         vms->clear_ptes = false;
>  }
>
> diff --git a/mm/vma.h b/mm/vma.h
> index 4edd5d26ffcfc..8b55a0c73d097 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -164,6 +164,20 @@ struct unmap_desc {
>         bool mm_wr_locked;            /* If the mmap write lock is held */
>  };
>
> +static inline void unmap_all_init(struct unmap_desc *desc,
> +               struct vma_iterator *vmi, struct vm_area_struct *vma)
> +{
> +       desc->mas = &vmi->mas;
> +       desc->first = vma;
> +       desc->first_pgaddr = FIRST_USER_ADDRESS;
> +       desc->last_pgaddr = USER_PGTABLES_CEILING;
> +       desc->vma_min = 0;
> +       desc->vma_max = ULONG_MAX;
> +       desc->tree_max = ULONG_MAX;
> +       desc->tree_reset = vma->vm_end;
> +       desc->mm_wr_locked = false;
> +}
> +
>  #define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
>         struct unmap_desc name = {                                          \
>                 .mas = &(_vmi)->mas,                                          \
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index 823d379e1fac2..d73ad4747d40a 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -884,18 +884,12 @@ static inline void update_hiwater_vm(struct mm_struct *)
>  {
>  }
>
> -static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -                     struct vm_area_struct *vma, unsigned long start_addr,
> -                     unsigned long end_addr, unsigned long tree_end,
> -                     bool mm_wr_locked)
> +struct unmap_desc;
> +
> +static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>  {
>         (void)tlb;
> -       (void)mas;
> -       (void)vma;
> -       (void)start_addr;
> -       (void)end_addr;
> -       (void)tree_end;
> -       (void)mm_wr_locked;
> +       (void)unmap;
>  }
>
>  static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> --
> 2.47.2
>


From surenb at google.com  Tue Sep  9 15:27:15 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 9 Sep 2025 15:27:15 -0700
Subject: [PATCH v1 9/9] mm: Use unmap_desc struct for freeing page tables.
In-Reply-To: <20250909190945.1030905-10-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com> <20250909190945.1030905-10-Liam.Howlett@oracle.com>
Message-ID: <CAJuCfpHp2sgKBqrfyn+xfCJTA-AFGFUTdhSz4dxNff=SNYLJgg@mail.gmail.com>

On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> Pass through the unmap_desc to free_pgtables() because it almost has
> everything necessary and is already on the stack.
>
> Updates testing code as necessary.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/internal.h                    |  5 +----
>  mm/memory.c                      | 21 ++++++++++-----------
>  mm/mmap.c                        |  6 +++---
>  mm/vma.c                         |  7 ++-----
>  tools/testing/vma/vma_internal.h | 11 ++---------
>  5 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 1239944f2830a..f22329967e908 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -445,10 +445,7 @@ bool __folio_end_writeback(struct folio *folio);
>  void deactivate_file_folio(struct folio *folio);
>  void folio_activate(struct folio *folio);
>
> -void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -                  struct vm_area_struct *start_vma, unsigned long floor,
> -                  unsigned long ceiling, unsigned long tree_max,
> -                  bool mm_wr_locked);
> +void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc);
>
>  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 8d4d976311037..98c5ffd28a109 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -387,15 +387,14 @@ void free_pgd_range(struct mmu_gather *tlb,
>   * The tree_max differs from the ceiling when a dup_mmap() failed and the tree

"ceiling" is not a parameter anymore, so the comments should also change.

>   * has unrelated data to the mm_struct being torn down.
>   */
> -void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -                  struct vm_area_struct *vma, unsigned long floor,
> -                  unsigned long ceiling, unsigned long tree_max,
> -                  bool mm_wr_locked)
> +void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)
>  {
>         struct unlink_vma_file_batch vb;
> +       struct ma_state *mas = desc->mas;
> +       struct vm_area_struct *vma = desc->first;
>
>         /* underflow can happen and is fine */
> -       WARN_ON_ONCE(tree_max - 1 > ceiling - 1);
> +       WARN_ON_ONCE(desc->tree_max - 1 > desc->last_pgaddr - 1);
>
>         tlb_free_vmas(tlb);
>
> @@ -407,13 +406,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                  * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
>                  * be 0.  This will underflow and is okay.
>                  */
> -               next = mas_find(mas, tree_max - 1);
> +               next = mas_find(mas, desc->tree_max - 1);
>
>                 /*
>                  * Hide vma from rmap and truncate_pagecache before freeing
>                  * pgtables
>                  */
> -               if (mm_wr_locked)
> +               if (desc->mm_wr_locked)
>                         vma_start_write(vma);
>                 unlink_anon_vmas(vma);
>
> @@ -425,16 +424,16 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>                  */
>                 while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>                         vma = next;
> -                       next = mas_find(mas, tree_max - 1);
> -                       if (mm_wr_locked)
> +                       next = mas_find(mas, desc->tree_max - 1);
> +                       if (desc->mm_wr_locked)
>                                 vma_start_write(vma);
>                         unlink_anon_vmas(vma);
>                         unlink_file_vma_batch_add(&vb, vma);
>                 }
>                 unlink_file_vma_batch_final(&vb);
>
> -               free_pgd_range(tlb, addr, vma->vm_end,
> -                       floor, next ? next->vm_start : ceiling);
> +               free_pgd_range(tlb, addr, vma->vm_end, desc->first_pgaddr,
> +                              next ? next->vm_start : desc->last_pgaddr);

Much better names IMO. Thank you!

>                 vma = next;
>         } while (vma);
>  }
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 6011f62b0a294..9908481452780 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1311,10 +1311,10 @@ void exit_mmap(struct mm_struct *mm)
>          */
>         mm_flags_set(MMF_OOM_SKIP, mm);
>         mmap_write_lock(mm);
> +       unmap.mm_wr_locked = true;
>         mt_clear_in_rcu(&mm->mm_mt);
> -       vma_iter_set(&vmi, vma->vm_end);
> -       free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
> -                     USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
> +       vma_iter_set(&vmi, unmap.tree_reset);
> +       free_pgtables(&tlb, &unmap);
>         tlb_finish_mmu(&tlb);
>
>         /*
> diff --git a/mm/vma.c b/mm/vma.c
> index ad64cd9795ef3..ba155a539d160 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -476,16 +476,13 @@ void remove_vma(struct vm_area_struct *vma)
>  void unmap_region(struct unmap_desc *desc)
>  {
>         struct mm_struct *mm = desc->first->vm_mm;
> -       struct ma_state *mas = desc->mas;
>         struct mmu_gather tlb;
>
>         tlb_gather_mmu(&tlb, mm);
>         update_hiwater_rss(mm);
>         unmap_vmas(&tlb, desc);
> -       mas_set(mas, desc->tree_reset);
> -       free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
> -                     desc->last_pgaddr, desc->tree_max,
> -                     desc->mm_wr_locked);
> +       mas_set(desc->mas, desc->tree_reset);
> +       free_pgtables(&tlb, desc);
>         tlb_finish_mmu(&tlb);
>  }
>
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index d73ad4747d40a..435c5a24480bc 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -892,17 +892,10 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>         (void)unmap;
>  }
>
> -static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -                  struct vm_area_struct *vma, unsigned long floor,
> -                  unsigned long ceiling, unsigned long tree_max,
> -                  bool mm_wr_locked)
> +static inline void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)
>  {
>         (void)tlb;
> -       (void)mas;
> -       (void)vma;
> -       (void)floor;
> -       (void)ceiling;
> -       (void)mm_wr_locked;
> +       (void)desc;
>  }
>
>  static inline void mapping_unmap_writable(struct address_space *)
> --
> 2.47.2
>


From urezki at gmail.com  Wed Sep 10 00:31:05 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Wed, 10 Sep 2025 09:31:05 +0200
Subject: [PATCH v7 04/21] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <772oyaa3j27tlklhpo6oqxsdtlvsg5goh2opzuig6xvgztgum4@scsoxrgtqm2f>
References: <20250903-slub-percpu-caches-v7-0-71c114cdefef@suse.cz>
 <20250903-slub-percpu-caches-v7-4-71c114cdefef@suse.cz>
 <aL7FFpIMmXtzzSL1@pc638.lan>
 <6f8274da-a010-4bb3-b3d6-690481b5ace0@suse.cz>
 <aL_uhPtztx7Ef0T2@pc636>
 <772oyaa3j27tlklhpo6oqxsdtlvsg5goh2opzuig6xvgztgum4@scsoxrgtqm2f>
Message-ID: <aMEpOTLh8EX6DXYq@pc636>

On Tue, Sep 09, 2025 at 10:35:15AM -0400, Liam R. Howlett wrote:
> * Uladzislau Rezki <urezki at gmail.com> [250909 05:08]:
> 
> ...
> 
> > 
> > > > - call_rcu() can be slow, therefore we do not use it in the kvfree_rcu().
> > > 
> > > If call_rcu() is called once per 32 kfree_rcu() filling up the rcu sheaf, is
> > > it still too slow?
> > >
> > You do not know where in a queue this callback lands, in the beginning,
> > in the end, etc. It is part of generic list which is processed one by
> > one. It can contain thousands of callbacks.
> 
> How does this differ from kvfree_rcu()?
> 
> Surely if you have enough calls to kvfree_rcu(), you will end up with a
> large list of frees before the end of a grace period?  Our placement in
> the freeing order would still be dependent on what else is using the
> infrastructure in the same grace period, right?
> 
In kfree_rcu() we use page blocks to carry pointers. Lists can be used
if there is a low memory condition so a page can not be allocated or
cache is empty. But this is not part of carr_rcu() track in any way.

Right regular call_rcu() puts callback into its own internal lists and
they are processed one by one during list iteration. In such lists you
can have hundred of thousand callback.

>
> How is kvfree_rcu() affected by rcu callback offloading to a specific
> cpu and rcu expedite?  Often these two features come into play for
> certain workloads which are of concern to us.
> 
We maintain a separate path. Offload is done after a grace period is
over. It is classic way. Historically all deferred freeing was one
call_rcu() per ptr.

> > 
> > If performance is not needed then it is not an issue. But in
> > kvfree_rcu() we do not use it, because of we want to offload
> > fast.
> 
> Today, I free things using call_rcu() and a custom callback so I would
> think stacking 32 together would make the list shorter, but latency
> would increase by waiting until there are 32.
> 
> If we wanted to flush the kvfree_rcu() list, is it done in the same way
> as the call_rcu() list, or is there a better way?
> 
For this case we have kvfree_rcu_barrier(). It is not same as call_rcu()
flushing.

--
Uladzislau Rezki


From vbabka at suse.cz  Wed Sep 10 01:01:03 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:03 +0200
Subject: [PATCH v8 01/23] locking/local_lock: Expose dep_map in
 local_trylock_t.
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-1-ca3099d8352c@suse.cz>

From: Alexei Starovoitov <ast at kernel.org>

lockdep_is_held() macro assumes that "struct lockdep_map dep_map;"
is a top level field of any lock that participates in LOCKDEP.
Make it so for local_trylock_t.

Reviewed-by: Sebastian Andrzej Siewior <bigeasy at linutronix.de>
Signed-off-by: Alexei Starovoitov <ast at kernel.org>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/local_lock_internal.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h
index d80b5306a2c0ccf95a3405b6b947b5f1f9a3bd38..949de37700dbc10feafc06d0b52382cf2e00c694 100644
--- a/include/linux/local_lock_internal.h
+++ b/include/linux/local_lock_internal.h
@@ -17,7 +17,10 @@ typedef struct {
 
 /* local_trylock() and local_trylock_irqsave() only work with local_trylock_t */
 typedef struct {
-	local_lock_t	llock;
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+	struct lockdep_map	dep_map;
+	struct task_struct	*owner;
+#endif
 	u8		acquired;
 } local_trylock_t;
 
@@ -31,7 +34,7 @@ typedef struct {
 	.owner = NULL,
 
 # define LOCAL_TRYLOCK_DEBUG_INIT(lockname)		\
-	.llock = { LOCAL_LOCK_DEBUG_INIT((lockname).llock) },
+	LOCAL_LOCK_DEBUG_INIT(lockname)
 
 static inline void local_lock_acquire(local_lock_t *l)
 {
@@ -81,7 +84,7 @@ do {								\
 	local_lock_debug_init(lock);				\
 } while (0)
 
-#define __local_trylock_init(lock) __local_lock_init(lock.llock)
+#define __local_trylock_init(lock) __local_lock_init((local_lock_t *)lock)
 
 #define __spinlock_nested_bh_init(lock)				\
 do {								\

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:04 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:04 +0200
Subject: [PATCH v8 02/23] slab: simplify init_kmem_cache_nodes() error
 handling
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-2-ca3099d8352c@suse.cz>

We don't need to call free_kmem_cache_nodes() immediately when failing
to allocate a kmem_cache_node, because when we return 0,
do_kmem_cache_create() calls __kmem_cache_release() which also performs
free_kmem_cache_nodes().

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 30003763d224c2704a4b93082b8b47af12dcffc5..9f671ec76131c4b0b28d5d568aa45842b5efb6d4 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -5669,10 +5669,8 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
 		n = kmem_cache_alloc_node(kmem_cache_node,
 						GFP_KERNEL, node);
 
-		if (!n) {
-			free_kmem_cache_nodes(s);
+		if (!n)
 			return 0;
-		}
 
 		init_kmem_cache_node(n);
 		s->node[node] = n;

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:14 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:14 +0200
Subject: [PATCH v8 12/23] tools/testing/vma: Implement vm_refcnt reset
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-12-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the reset of the ref count in vma_lock_init().  This is needed if
the vma memory is not zeroed on allocation.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/vma/vma_internal.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index f8cf5b184d5b51dd627ff440943a7af3c549f482..6b6e2b05918c9f95b537f26e20a943b34082825a 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -1373,6 +1373,8 @@ static inline void ksm_exit(struct mm_struct *mm)
 
 static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)
 {
+	if (reset_refcnt)
+		refcount_set(&vma->vm_refcnt, 0);
 }
 
 static inline void vma_numab_state_init(struct vm_area_struct *vma)

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:07 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:07 +0200
Subject: [PATCH v8 05/23] slab: sheaf prefilling for guaranteed allocations
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-5-ca3099d8352c@suse.cz>

Add functions for efficient guaranteed allocations e.g. in a critical
section that cannot sleep, when the exact number of allocations is not
known beforehand, but an upper limit can be calculated.

kmem_cache_prefill_sheaf() returns a sheaf containing at least given
number of objects.

kmem_cache_alloc_from_sheaf() will allocate an object from the sheaf
and is guaranteed not to fail until depleted.

kmem_cache_return_sheaf() is for giving the sheaf back to the slab
allocator after the critical section. This will also attempt to refill
it to cache's sheaf capacity for better efficiency of sheaves handling,
but it's not stricly necessary to succeed.

kmem_cache_refill_sheaf() can be used to refill a previously obtained
sheaf to requested size. If the current size is sufficient, it does
nothing. If the requested size exceeds cache's sheaf_capacity and the
sheaf's current capacity, the sheaf will be replaced with a new one,
hence the indirect pointer parameter.

kmem_cache_sheaf_size() can be used to query the current size.

The implementation supports requesting sizes that exceed cache's
sheaf_capacity, but it is not efficient - such "oversize" sheaves are
allocated fresh in kmem_cache_prefill_sheaf() and flushed and freed
immediately by kmem_cache_return_sheaf(). kmem_cache_refill_sheaf()
might be especially ineffective when replacing a sheaf with a new one of
a larger capacity. It is therefore better to size cache's
sheaf_capacity accordingly to make oversize sheaves exceptional.

CONFIG_SLUB_STATS counters are added for sheaf prefill and return
operations. A prefill or return is considered _fast when it is able to
grab or return a percpu spare sheaf (even if the sheaf needs a refill to
satisfy the request, as those should amortize over time), and _slow
otherwise (when the barn or even sheaf allocation/freeing has to be
involved). sheaf_prefill_oversize is provided to determine how many
prefills were oversize (counter for oversize returns is not necessary as
all oversize refills result in oversize returns).

When slub_debug is enabled for a cache with sheaves, no percpu sheaves
exist for it, but the prefill functionality is still provided simply by
all prefilled sheaves becoming oversize. If percpu sheaves are not
created for a cache due to not passing the sheaf_capacity argument on
cache creation, the prefills also work through oversize sheaves, but
there's a WARN_ON_ONCE() to indicate the omission.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/slab.h |  16 ++++
 mm/slub.c            | 263 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 279 insertions(+)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 49acbcdc6696fd120c402adf757b3f41660ad50a..680193356ac7a22f9df5cd9b71ff8b81e26404ad 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -829,6 +829,22 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t flags,
 				   int node) __assume_slab_alignment __malloc;
 #define kmem_cache_alloc_node(...)	alloc_hooks(kmem_cache_alloc_node_noprof(__VA_ARGS__))
 
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
+
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf **sheafp, unsigned int size);
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+				       struct slab_sheaf *sheaf);
+
+void *kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *cachep, gfp_t gfp,
+			struct slab_sheaf *sheaf) __assume_slab_alignment __malloc;
+#define kmem_cache_alloc_from_sheaf(...)	\
+			alloc_hooks(kmem_cache_alloc_from_sheaf_noprof(__VA_ARGS__))
+
+unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf);
+
 /*
  * These macros allow declaring a kmem_buckets * parameter alongside size, which
  * can be compiled out with CONFIG_SLAB_BUCKETS=n so that a large number of call
diff --git a/mm/slub.c b/mm/slub.c
index 19cd8444ae5d210c77ae767912ca1ff3fc69c2a8..38f5b865d3093556171e0f6530d395718b438099 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -401,6 +401,11 @@ enum stat_item {
 	BARN_GET_FAIL,		/* Failed to get full sheaf from barn */
 	BARN_PUT,		/* Put full sheaf to barn */
 	BARN_PUT_FAIL,		/* Failed to put full sheaf to barn */
+	SHEAF_PREFILL_FAST,	/* Sheaf prefill grabbed the spare sheaf */
+	SHEAF_PREFILL_SLOW,	/* Sheaf prefill found no spare sheaf */
+	SHEAF_PREFILL_OVERSIZE,	/* Allocation of oversize sheaf for prefill */
+	SHEAF_RETURN_FAST,	/* Sheaf return reattached spare sheaf */
+	SHEAF_RETURN_SLOW,	/* Sheaf return could not reattach spare */
 	NR_SLUB_STAT_ITEMS
 };
 
@@ -462,6 +467,8 @@ struct slab_sheaf {
 	union {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
+		/* only used for prefilled sheafs */
+		unsigned int capacity;
 	};
 	struct kmem_cache *cache;
 	unsigned int size;
@@ -2838,6 +2845,30 @@ static void barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf
 	spin_unlock_irqrestore(&barn->lock, flags);
 }
 
+static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn)
+{
+	struct slab_sheaf *sheaf = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full) {
+		sheaf = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
+					barn_list);
+		list_del(&sheaf->barn_list);
+		barn->nr_full--;
+	} else if (barn->nr_empty) {
+		sheaf = list_first_entry(&barn->sheaves_empty,
+					 struct slab_sheaf, barn_list);
+		list_del(&sheaf->barn_list);
+		barn->nr_empty--;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return sheaf;
+}
+
 /*
  * If a full sheaf is available, return it and put the supplied empty one to
  * barn. We ignore the limit on empty sheaves as the number of sheaves doesn't
@@ -5042,6 +5073,228 @@ void *kmem_cache_alloc_node_noprof(struct kmem_cache *s, gfp_t gfpflags, int nod
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node_noprof);
 
+/*
+ * returns a sheaf that has at least the requested size
+ * when prefilling is needed, do so with given gfp flags
+ *
+ * return NULL if sheaf allocation or prefilling failed
+ */
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *sheaf = NULL;
+
+	if (unlikely(size > s->sheaf_capacity)) {
+
+		/*
+		 * slab_debug disables cpu sheaves intentionally so all
+		 * prefilled sheaves become "oversize" and we give up on
+		 * performance for the debugging. Same with SLUB_TINY.
+		 * Creating a cache without sheaves and then requesting a
+		 * prefilled sheaf is however not expected, so warn.
+		 */
+		WARN_ON_ONCE(s->sheaf_capacity == 0 &&
+			     !IS_ENABLED(CONFIG_SLUB_TINY) &&
+			     !(s->flags & SLAB_DEBUG_FLAGS));
+
+		sheaf = kzalloc(struct_size(sheaf, objects, size), gfp);
+		if (!sheaf)
+			return NULL;
+
+		stat(s, SHEAF_PREFILL_OVERSIZE);
+		sheaf->cache = s;
+		sheaf->capacity = size;
+
+		if (!__kmem_cache_alloc_bulk(s, gfp, size,
+					     &sheaf->objects[0])) {
+			kfree(sheaf);
+			return NULL;
+		}
+
+		sheaf->size = size;
+
+		return sheaf;
+	}
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (pcs->spare) {
+		sheaf = pcs->spare;
+		pcs->spare = NULL;
+		stat(s, SHEAF_PREFILL_FAST);
+	} else {
+		stat(s, SHEAF_PREFILL_SLOW);
+		sheaf = barn_get_full_or_empty_sheaf(get_barn(s));
+		if (sheaf && sheaf->size)
+			stat(s, BARN_GET);
+		else
+			stat(s, BARN_GET_FAIL);
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+
+	if (!sheaf)
+		sheaf = alloc_empty_sheaf(s, gfp);
+
+	if (sheaf && sheaf->size < size) {
+		if (refill_sheaf(s, sheaf, gfp)) {
+			sheaf_flush_unused(s, sheaf);
+			free_empty_sheaf(s, sheaf);
+			sheaf = NULL;
+		}
+	}
+
+	if (sheaf)
+		sheaf->capacity = s->sheaf_capacity;
+
+	return sheaf;
+}
+
+/*
+ * Use this to return a sheaf obtained by kmem_cache_prefill_sheaf()
+ *
+ * If the sheaf cannot simply become the percpu spare sheaf, but there's space
+ * for a full sheaf in the barn, we try to refill the sheaf back to the cache's
+ * sheaf_capacity to avoid handling partially full sheaves.
+ *
+ * If the refill fails because gfp is e.g. GFP_NOWAIT, or the barn is full, the
+ * sheaf is instead flushed and freed.
+ */
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+			     struct slab_sheaf *sheaf)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct node_barn *barn;
+
+	if (unlikely(sheaf->capacity != s->sheaf_capacity)) {
+		sheaf_flush_unused(s, sheaf);
+		kfree(sheaf);
+		return;
+	}
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	barn = get_barn(s);
+
+	if (!pcs->spare) {
+		pcs->spare = sheaf;
+		sheaf = NULL;
+		stat(s, SHEAF_RETURN_FAST);
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (!sheaf)
+		return;
+
+	stat(s, SHEAF_RETURN_SLOW);
+
+	/*
+	 * If the barn has too many full sheaves or we fail to refill the sheaf,
+	 * simply flush and free it.
+	 */
+	if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES ||
+	    refill_sheaf(s, sheaf, gfp)) {
+		sheaf_flush_unused(s, sheaf);
+		free_empty_sheaf(s, sheaf);
+		return;
+	}
+
+	barn_put_full_sheaf(barn, sheaf);
+	stat(s, BARN_PUT);
+}
+
+/*
+ * refill a sheaf previously returned by kmem_cache_prefill_sheaf to at least
+ * the given size
+ *
+ * the sheaf might be replaced by a new one when requesting more than
+ * s->sheaf_capacity objects if such replacement is necessary, but the refill
+ * fails (returning -ENOMEM), the existing sheaf is left intact
+ *
+ * In practice we always refill to full sheaf's capacity.
+ */
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+			    struct slab_sheaf **sheafp, unsigned int size)
+{
+	struct slab_sheaf *sheaf;
+
+	/*
+	 * TODO: do we want to support *sheaf == NULL to be equivalent of
+	 * kmem_cache_prefill_sheaf() ?
+	 */
+	if (!sheafp || !(*sheafp))
+		return -EINVAL;
+
+	sheaf = *sheafp;
+	if (sheaf->size >= size)
+		return 0;
+
+	if (likely(sheaf->capacity >= size)) {
+		if (likely(sheaf->capacity == s->sheaf_capacity))
+			return refill_sheaf(s, sheaf, gfp);
+
+		if (!__kmem_cache_alloc_bulk(s, gfp, sheaf->capacity - sheaf->size,
+					     &sheaf->objects[sheaf->size])) {
+			return -ENOMEM;
+		}
+		sheaf->size = sheaf->capacity;
+
+		return 0;
+	}
+
+	/*
+	 * We had a regular sized sheaf and need an oversize one, or we had an
+	 * oversize one already but need a larger one now.
+	 * This should be a very rare path so let's not complicate it.
+	 */
+	sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
+	if (!sheaf)
+		return -ENOMEM;
+
+	kmem_cache_return_sheaf(s, gfp, *sheafp);
+	*sheafp = sheaf;
+	return 0;
+}
+
+/*
+ * Allocate from a sheaf obtained by kmem_cache_prefill_sheaf()
+ *
+ * Guaranteed not to fail as many allocations as was the requested size.
+ * After the sheaf is emptied, it fails - no fallback to the slab cache itself.
+ *
+ * The gfp parameter is meant only to specify __GFP_ZERO or __GFP_ACCOUNT
+ * memcg charging is forced over limit if necessary, to avoid failure.
+ */
+void *
+kmem_cache_alloc_from_sheaf_noprof(struct kmem_cache *s, gfp_t gfp,
+				   struct slab_sheaf *sheaf)
+{
+	void *ret = NULL;
+	bool init;
+
+	if (sheaf->size == 0)
+		goto out;
+
+	ret = sheaf->objects[--sheaf->size];
+
+	init = slab_want_init_on_alloc(gfp, s);
+
+	/* add __GFP_NOFAIL to force successful memcg charging */
+	slab_post_alloc_hook(s, NULL, gfp | __GFP_NOFAIL, 1, &ret, init, s->object_size);
+out:
+	trace_kmem_cache_alloc(_RET_IP_, ret, s, gfp, NUMA_NO_NODE);
+
+	return ret;
+}
+
+unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
+{
+	return sheaf->size;
+}
 /*
  * To avoid unnecessary overhead, we pass through large allocation requests
  * directly to the page allocator. We use __GFP_COMP, because we will need to
@@ -8576,6 +8829,11 @@ STAT_ATTR(BARN_GET, barn_get);
 STAT_ATTR(BARN_GET_FAIL, barn_get_fail);
 STAT_ATTR(BARN_PUT, barn_put);
 STAT_ATTR(BARN_PUT_FAIL, barn_put_fail);
+STAT_ATTR(SHEAF_PREFILL_FAST, sheaf_prefill_fast);
+STAT_ATTR(SHEAF_PREFILL_SLOW, sheaf_prefill_slow);
+STAT_ATTR(SHEAF_PREFILL_OVERSIZE, sheaf_prefill_oversize);
+STAT_ATTR(SHEAF_RETURN_FAST, sheaf_return_fast);
+STAT_ATTR(SHEAF_RETURN_SLOW, sheaf_return_slow);
 #endif	/* CONFIG_SLUB_STATS */
 
 #ifdef CONFIG_KFENCE
@@ -8676,6 +8934,11 @@ static struct attribute *slab_attrs[] = {
 	&barn_get_fail_attr.attr,
 	&barn_put_attr.attr,
 	&barn_put_fail_attr.attr,
+	&sheaf_prefill_fast_attr.attr,
+	&sheaf_prefill_slow_attr.attr,
+	&sheaf_prefill_oversize_attr.attr,
+	&sheaf_return_fast_attr.attr,
+	&sheaf_return_slow_attr.attr,
 #endif
 #ifdef CONFIG_FAILSLAB
 	&failslab_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:02 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:02 +0200
Subject: [PATCH v8 00/23] SLUB percpu sheaves
Message-ID: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>

Hi,

I'm sending full v8 due to more changes in the middle of the series that
resulted in later patches being fixed up due to conflicts (details in
the changelog below).
The v8 will replace+extend the v7 in slab/for-next.

===

This series adds an opt-in percpu array-based caching layer to SLUB.
It has evolved to a state where kmem caches with sheaves are compatible
with all SLUB features (slub_debug, SLUB_TINY, NUMA locality
considerations). My hope is therefore that it can eventually be enabled
for all kmem caches and replace the cpu (partial) slabs.

Note the name "sheaf" was invented by Matthew Wilcox so we don't call
the arrays magazines like the original Bonwick paper. The per-NUMA-node
cache of sheaves is thus called "barn".

This caching may seem similar to the arrays in SLAB, but there are some
important differences:

- does not distinguish NUMA locality, thus there are no per-node
  "shared" arrays (with possible lock contention) and no "alien" arrays
  that would need periodical flushing
  - NUMA restricted allocations and strict_numa mode is still honoured,
    the percpu sheaves are bypassed for those allocations
  - a later patch (for separate evaluation) makes freeing remote objects
    bypass sheaves so sheaves contain mostly (not strictly) local objects
- improves kfree_rcu() handling by reusing whole sheaves
- there is an API for obtaining a preallocated sheaf that can be used
  for guaranteed and efficient allocations in a restricted context, when
  the upper bound for needed objects is known but rarely reached
- opt-in, not used for every cache (for now)

The motivation comes mainly from the ongoing work related to VMA locking
scalability and the related maple tree operations. This is why VMA and
maple nodes caches are sheaf-enabled in the patchset. In v5 I include
Liam's patches for full maple tree conversion that uses the improved
preallocation API.

A sheaf-enabled cache has the following expected advantages:

- Cheaper fast paths. For allocations, instead of local double cmpxchg,
  thanks to local_trylock() it becomes a preempt_disable() and no atomic
  operations. Same for freeing, which is otherwise a local double cmpxchg
  only for short term allocations (so the same slab is still active on the
  same cpu when freeing the object) and a more costly locked double
  cmpxchg otherwise.

- kfree_rcu() batching and recycling. kfree_rcu() will put objects to a
  separate percpu sheaf and only submit the whole sheaf to call_rcu()
  when full. After the grace period, the sheaf can be used for
  allocations, which is more efficient than freeing and reallocating
  individual slab objects (even with the batching done by kfree_rcu()
  implementation itself). In case only some cpus are allowed to handle rcu
  callbacks, the sheaf can still be made available to other cpus on the
  same node via the shared barn. The maple_node cache uses kfree_rcu() and
  thus can benefit from this.
  Note: this path is currently limited to !PREEMPT_RT

- Preallocation support. A prefilled sheaf can be privately borrowed to
  perform a short term operation that is not allowed to block in the
  middle and may need to allocate some objects. If an upper bound (worst
  case) for the number of allocations is known, but only much fewer
  allocations actually needed on average, borrowing and returning a sheaf
  is much more efficient then a bulk allocation for the worst case
  followed by a bulk free of the many unused objects. Maple tree write
  operations should benefit from this.

- Compatibility with slub_debug. When slub_debug is enabled for a cache,
  we simply don't create the percpu sheaves so that the debugging hooks
  (at the node partial list slowpaths) are reached as before. The same
  thing is done for CONFIG_SLUB_TINY. Sheaf preallocation still works by
  reusing the (ineffective) paths for requests exceeding the cache's
  sheaf_capacity. This is in line with the existing approach where
  debugging bypasses the fast paths and SLUB_TINY preferes memory
  savings over performance.

GIT TREES:

this series: https://git.kernel.org/vbabka/l/slub-percpu-sheaves-v8r2
It is based on v6.17-rc3.

this series plus a microbenchmark hacked into slub_kunit:
https://git.kernel.org/vbabka/l/slub-percpu-sheaves-v8-benchmarking

It allows evaluating overhead of the added sheaves code, and benefits
for single-threaded allocation/frees of varying batch size. I plan to
look into adding multi-threaded scenarios too.

The last commit there also adds sheaves to every cache to allow
measuring effects on other caches than vma and maple node. Note these
measurements should be compared to slab_nomerge boots without sheaves,
as adding sheaves makes caches unmergeable.

RESULTS:

In order to get some numbers that should be only due to differences in
implementation and no cache layout side-effects in users of the slab
objects etc, I have started with a in-kernel microbenchmark that does
allocating and freeing from a slab cache with or without sheaves and/or
memcg. It's either alternating single object alloc and free, or
allocates 10 objects and frees them, then 100, then 1000
- in order to see the effects of exhausting percpu sheaves or barn, or
(without sheaves) the percpu slabs. The order of objects to free can
be also shuffled instead of FIFO - to stress the non-sheaf freeing
slowpath more.

Measurements done on Ryzen 7 5700, bare metal.

The first question was how just having the sheaves implementation affects
existing no-sheaf caches due to the extra (unused) code. I have experimented
with changing inlining and adding unlikely() to the sheaves case. The
optimum seems is what's currently in the implementation - fast-path sheaves
usage is inlined, any handling of main sheaf empty on alloc/full on free is
a separate function, and the if (s->sheaf_capacity) has neither likely() nor
unlikely(). When I added unlikely() it destroyed the performance of sheaves
completely.

So the result is that with batch size 10, there's 2.4% overhead, and the
other cases are all impacted less than this. Hopefully acceptable with the
plan that eventually there would be sheaves everywhere and the current
cpu (partial) slabs scheme removed.

As for benefits of enabling sheaves (capacity=32) see the results below,
looks all good here. Of course this microbenchmark is not a complete
story though for at least these reasons:

- no kfree_rcu() evaluation
- doesn't show barn spinlock contention effects. In theory shouldn't be
worse than without sheaves because after exhausting cpu (partial) slabs, the
list_lock has to be taken. Sheaf capacity vs capacity of partial slabs is a
matter of tuning.

---------------------------------
 BATCH SIZE: 1 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115660272
 bench: no memcg, sheaves
 average (excl. iter 0): 95734972
 sheaves better by 17.2%
 bench: memcg, no sheaves
 average (excl. iter 0): 163682964
 bench: memcg, sheaves
 average (excl. iter 0): 144792803
 sheaves better by 11.5%

 ---------------------------------
 BATCH SIZE: 10 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115496906
 bench: no memcg, sheaves
 average (excl. iter 0): 97781102
 sheaves better by 15.3%
 bench: memcg, no sheaves
 average (excl. iter 0): 162771491
 bench: memcg, sheaves
 average (excl. iter 0): 144746490
 sheaves better by 11.0%

 ---------------------------------
 BATCH SIZE: 100 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 151796052
 bench: no memcg, sheaves
 average (excl. iter 0): 104641753
 sheaves better by 31.0%
 bench: memcg, no sheaves
 average (excl. iter 0): 200733436
 bench: memcg, sheaves
 average (excl. iter 0): 151340989
 sheaves better by 24.6%

 ---------------------------------
 BATCH SIZE: 1000 SHUFFLED: NO
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 187623118
 bench: no memcg, sheaves
 average (excl. iter 0): 130914624
 sheaves better by 30.2%
 bench: memcg, no sheaves
 average (excl. iter 0): 240239575
 bench: memcg, sheaves
 average (excl. iter 0): 181474462
 sheaves better by 24.4%

 ---------------------------------
 BATCH SIZE: 10 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 115110219
 bench: no memcg, sheaves
 average (excl. iter 0): 100597405
 sheaves better by 12.6%
 bench: memcg, no sheaves
 average (excl. iter 0): 163573377
 bench: memcg, sheaves
 average (excl. iter 0): 144535545
 sheaves better by 11.6%

 ---------------------------------
 BATCH SIZE: 100 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 152457970
 bench: no memcg, sheaves
 average (excl. iter 0): 108720274
 sheaves better by 28.6%
 bench: memcg, no sheaves
 average (excl. iter 0): 203478732
 bench: memcg, sheaves
 average (excl. iter 0): 151241821
 sheaves better by 25.6%

 ---------------------------------
 BATCH SIZE: 1000 SHUFFLED: YES
 ---------------------------------

 bench: no memcg, no sheaves
 average (excl. iter 0): 189950559
 bench: no memcg, sheaves
 average (excl. iter 0): 177934450
 sheaves better by 6.3%
 bench: memcg, no sheaves
 average (excl. iter 0): 242988187
 bench: memcg, sheaves
 average (excl. iter 0): 221609979
 sheaves better by 8.7%

Vlastimil

---
Changes in v8:
- Liam provided a new patch "maple_tree: Drop bulk insert support" as
  the bulk insert was broken in v7 (not used anywhere but broke its
  internal tests) This has changed the context for later patches somewhat.
- Fix issues with "slab: add sheaf support for batching kfree_rcu()
  operations" reported by Ulad:
  - obtaining empty sheaf from kfree_rcu() is potentially dangerous on
    PREEMPT_RT - disable using rcu_free sheaves there for now
  - kvfree_rcu_barrier() must flush all rcu_free sheaves
  - dropped R-b's from Suren and Harry due to these changes being
    nontrivial
  - Added some R-b's
- Incorporated two patches originally in mm tree to avoid conflicts:
  - "maple_tree: remove redundant __GFP_NOWARN"
  - "tools/testing/vma: clean up stubs in vma_internal.h"
- Link to v7: https://patch.msgid.link/20250903-slub-percpu-caches-v7-0-71c114cdefef at suse.cz

Changes in v7:
- Incorporate Alexei's patch for local_trylock_t to fix
  lockdep_assert_held() on RT, reported by Thorsten.
  Patch: https://lore.kernel.org/all/20250718021646.73353-2-alexei.starovoitov at gmail.com/
- Remove pcs->barn pointer to fix boot failures reported by Venkat:
  https://lore.kernel.org/all/866d7f30-7cde-4c88-87ba-bdad16075433 at linux.ibm.com/
  This was because initializing it for all possible cpus can get bogus
  nid values from cpu_to_mem() for non-online cpus.
  Instead introduce get_barn() to obtain it from node, the fast paths
  are not affected anyway. Otherwise cpu hotplug online callback would
  be necessary and I haven't found a cpuhp state that would satisfy all
  the constraints (including being safe to take slab_mutex).
  The draft fix patch from the bug report thread is effectively squashed
  into several of the sheaves patches.
- Incorporate maple tree patches posted separately here:
  https://lore.kernel.org/all/20250901-maple-sheaves-v1-0-d6a1166b53f2 at suse.cz/
  Everything is reordered so that vma and maple userspace tests compile
  and pass at every step (fixes a broken vma test report from Lorenzo).
  Add tags from Sid for two of the patches there.
- Link to v6: https://patch.msgid.link/20250827-slub-percpu-caches-v6-0-f0f775a3f73f at suse.cz

Changes in v6:
- Applied feedback and review tags from Suren and Harry (thanks!)
- Separate patch for init_kmem_cache_nodes() error handling change.
- Removed the more involved maple tree conversion to be posted as a
  separate followup series.
- Link to v5: https://patch.msgid.link/20250723-slub-percpu-caches-v5-0-b792cd830f5d at suse.cz

Changes in v5:
- Apply review tags (Harry, Suren) except where changed too much (first
  patch).
- Handle CONFIG_SLUB_TINY by not creating percpu sheaves (Harry)
- Apply review feedback (typos, comments).
- Extract handling sheaf slow paths to separate non-inline functions
  __pcs_handle_empty() and __pcs_handle_full().
- Fix empty sheaf leak in rcu_free_sheaf() (Suren)
- Add "allow NUMA restricted allocations to use percpu sheaves".
- Add Liam's maple tree full sheaf conversion patches for easier
  evaluation.
- Rebase to v6.16-rc1.
- Link to v4: https://patch.msgid.link/20250425-slub-percpu-caches-v4-0-8a636982b4a4 at suse.cz

Changes in v4:
- slub_debug disables sheaves for the cache in order to work properly
- strict_numa mode works as intended
- added a separate patch to make freeing remote objects skip sheaves
- various code refactoring suggested by Suren and Harry
- removed less useful stat counters and added missing ones for barn
  and prefilled sheaf events
- Link to v3: https://lore.kernel.org/r/20250317-slub-percpu-caches-v3-0-9d9884d8b643 at suse.cz

Changes in v3:
- Squash localtry_lock conversion so it's used immediately.
- Incorporate feedback and add tags from Suren and Harry - thanks!
  - Mostly adding comments and some refactoring.
  - Fixes for kfree_rcu_sheaf() vmalloc handling, cpu hotremove
    flushing.
  - Fix wrong condition in kmem_cache_return_sheaf() that may have
    affected performance negatively.
  - Refactoring of free_to_pcs()
- Link to v2: https://lore.kernel.org/r/20250214-slub-percpu-caches-v2-0-88592ee0966a at suse.cz

Changes in v2:
- Removed kfree_rcu() destructors support as VMAs will not need it
  anymore after [3] is merged.
- Changed to localtry_lock_t borrowed from [2] instead of an own
  implementation of the same idea.
- Many fixes and improvements thanks to Liam's adoption for maple tree
  nodes.
- Userspace Testing stubs by Liam.
- Reduced limitations/todos - hooking to kfree_rcu() is complete,
  prefilled sheaves can exceed cache's sheaf_capacity.
- Link to v1: https://lore.kernel.org/r/20241112-slub-percpu-caches-v1-0-ddc0bdc27e05 at suse.cz

---
Alexei Starovoitov (1):
      locking/local_lock: Expose dep_map in local_trylock_t.

Liam R. Howlett (8):
      maple_tree: Drop bulk insert support
      tools/testing/vma: Implement vm_refcnt reset
      tools/testing: Add support for changes to slab for sheaves
      testing/radix-tree/maple: Hack around kfree_rcu not existing
      tools/testing: Add support for prefilled slab sheafs
      maple_tree: Prefilled sheaf conversion and testing
      maple_tree: Add single node allocation support to maple state
      maple_tree: Convert forking to use the sheaf interface

Lorenzo Stoakes (1):
      tools/testing/vma: clean up stubs in vma_internal.h

Pedro Falcato (2):
      maple_tree: Use kfree_rcu in ma_free_rcu
      maple_tree: Replace mt_free_one() with kfree()

Qianfeng Rong (1):
      maple_tree: remove redundant __GFP_NOWARN

Vlastimil Babka (10):
      slab: simplify init_kmem_cache_nodes() error handling
      slab: add opt-in caching layer of percpu sheaves
      slab: add sheaf support for batching kfree_rcu() operations
      slab: sheaf prefilling for guaranteed allocations
      slab: determine barn status racily outside of lock
      slab: skip percpu sheaves for remote object freeing
      slab: allow NUMA restricted allocations to use percpu sheaves
      mm, vma: use percpu sheaves for vm_area_struct cache
      maple_tree: use percpu sheaves for maple_node_cache
      tools/testing: include maple-shim.c in maple.c

 include/linux/local_lock_internal.h |    9 +-
 include/linux/maple_tree.h          |    6 +-
 include/linux/slab.h                |   47 +
 lib/maple_tree.c                    |  667 +++----------
 lib/test_maple_tree.c               |  137 ---
 mm/slab.h                           |    5 +
 mm/slab_common.c                    |   34 +-
 mm/slub.c                           | 1761 +++++++++++++++++++++++++++++++++--
 mm/vma_init.c                       |    1 +
 tools/include/linux/slab.h          |  165 +++-
 tools/testing/radix-tree/maple.c    |  514 +---------
 tools/testing/shared/linux.c        |  120 ++-
 tools/testing/shared/maple-shared.h |   11 +
 tools/testing/shared/maple-shim.c   |    7 +
 tools/testing/vma/vma_internal.h    |  259 ++----
 15 files changed, 2282 insertions(+), 1461 deletions(-)
---
base-commit: 82efd569a8909f2b13140c1b3de88535aea0b051
change-id: 20231128-slub-percpu-caches-9441892011d7

Best regards,
-- 
Vlastimil Babka <vbabka at suse.cz>



From vbabka at suse.cz  Wed Sep 10 01:01:05 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:05 +0200
Subject: [PATCH v8 03/23] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-3-ca3099d8352c@suse.cz>

Specifying a non-zero value for a new struct kmem_cache_args field
sheaf_capacity will setup a caching layer of percpu arrays called
sheaves of given capacity for the created cache.

Allocations from the cache will allocate via the percpu sheaves (main or
spare) as long as they have no NUMA node preference. Frees will also
put the object back into one of the sheaves.

When both percpu sheaves are found empty during an allocation, an empty
sheaf may be replaced with a full one from the per-node barn. If none
are available and the allocation is allowed to block, an empty sheaf is
refilled from slab(s) by an internal bulk alloc operation. When both
percpu sheaves are full during freeing, the barn can replace a full one
with an empty one, unless over a full sheaves limit. In that case a
sheaf is flushed to slab(s) by an internal bulk free operation. Flushing
sheaves and barns is also wired to the existing cpu flushing and cache
shrinking operations.

The sheaves do not distinguish NUMA locality of the cached objects. If
an allocation is requested with kmem_cache_alloc_node() (or a mempolicy
with strict_numa mode enabled) with a specific node (not NUMA_NO_NODE),
the sheaves are bypassed.

The bulk operations exposed to slab users also try to utilize the
sheaves as long as the necessary (full or empty) sheaves are available
on the cpu or in the barn. Once depleted, they will fallback to bulk
alloc/free to slabs directly to avoid double copying.

The sheaf_capacity value is exported in sysfs for observability.

Sysfs CONFIG_SLUB_STATS counters alloc_cpu_sheaf and free_cpu_sheaf
count objects allocated or freed using the sheaves (and thus not
counting towards the other alloc/free path counters). Counters
sheaf_refill and sheaf_flush count objects filled or flushed from or to
slab pages, and can be used to assess how effective the caching is. The
refill and flush operations will also count towards the usual
alloc_fastpath/slowpath, free_fastpath/slowpath and other counters for
the backing slabs.  For barn operations, barn_get and barn_put count how
many full sheaves were get from or put to the barn, the _fail variants
count how many such requests could not be satisfied mainly  because the
barn was either empty or full. While the barn also holds empty sheaves
to make some operations easier, these are not as critical to mandate own
counters.  Finally, there are sheaf_alloc/sheaf_free counters.

Access to the percpu sheaves is protected by local_trylock() when
potential callers include irq context, and local_lock() otherwise (such
as when we already know the gfp flags allow blocking). The trylock
failures should be rare and we can easily fallback. Each per-NUMA-node
barn has a spin_lock.

When slub_debug is enabled for a cache with sheaf_capacity also
specified, the latter is ignored so that allocations and frees reach the
slow path where debugging hooks are processed. Similarly, we ignore it
with CONFIG_SLUB_TINY which prefers low memory usage to performance.

[boot failure: https://lore.kernel.org/all/583eacf5-c971-451a-9f76-fed0e341b815 at linux.ibm.com/ ]

Reported-and-tested-by: Venkat Rao Bagalkote <venkat88 at linux.ibm.com>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/slab.h |   31 ++
 mm/slab.h            |    2 +
 mm/slab_common.c     |    5 +-
 mm/slub.c            | 1164 +++++++++++++++++++++++++++++++++++++++++++++++---
 4 files changed, 1142 insertions(+), 60 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index d5a8ab98035cf3e3d9043e3b038e1bebeff05b52..49acbcdc6696fd120c402adf757b3f41660ad50a 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -335,6 +335,37 @@ struct kmem_cache_args {
 	 * %NULL means no constructor.
 	 */
 	void (*ctor)(void *);
+	/**
+	 * @sheaf_capacity: Enable sheaves of given capacity for the cache.
+	 *
+	 * With a non-zero value, allocations from the cache go through caching
+	 * arrays called sheaves. Each cpu has a main sheaf that's always
+	 * present, and a spare sheaf that may be not present. When both become
+	 * empty, there's an attempt to replace an empty sheaf with a full sheaf
+	 * from the per-node barn.
+	 *
+	 * When no full sheaf is available, and gfp flags allow blocking, a
+	 * sheaf is allocated and filled from slab(s) using bulk allocation.
+	 * Otherwise the allocation falls back to the normal operation
+	 * allocating a single object from a slab.
+	 *
+	 * Analogically when freeing and both percpu sheaves are full, the barn
+	 * may replace it with an empty sheaf, unless it's over capacity. In
+	 * that case a sheaf is bulk freed to slab pages.
+	 *
+	 * The sheaves do not enforce NUMA placement of objects, so allocations
+	 * via kmem_cache_alloc_node() with a node specified other than
+	 * NUMA_NO_NODE will bypass them.
+	 *
+	 * Bulk allocation and free operations also try to use the cpu sheaves
+	 * and barn, but fallback to using slab pages directly.
+	 *
+	 * When slub_debug is enabled for the cache, the sheaf_capacity argument
+	 * is ignored.
+	 *
+	 * %0 means no sheaves will be created.
+	 */
+	unsigned int sheaf_capacity;
 };
 
 struct kmem_cache *__kmem_cache_create_args(const char *name,
diff --git a/mm/slab.h b/mm/slab.h
index 248b34c839b7ca39cf14e139c62d116efb97d30f..206987ce44a4d053ebe3b5e50784d2dd23822cd1 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -235,6 +235,7 @@ struct kmem_cache {
 #ifndef CONFIG_SLUB_TINY
 	struct kmem_cache_cpu __percpu *cpu_slab;
 #endif
+	struct slub_percpu_sheaves __percpu *cpu_sheaves;
 	/* Used for retrieving partial slabs, etc. */
 	slab_flags_t flags;
 	unsigned long min_partial;
@@ -248,6 +249,7 @@ struct kmem_cache {
 	/* Number of per cpu partial slabs to keep around */
 	unsigned int cpu_partial_slabs;
 #endif
+	unsigned int sheaf_capacity;
 	struct kmem_cache_order_objects oo;
 
 	/* Allocation and freeing of slabs */
diff --git a/mm/slab_common.c b/mm/slab_common.c
index bfe7c40eeee1a01c175766935c1e3c0304434a53..e2b197e47866c30acdbd1fee4159f262a751c5a7 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -163,6 +163,9 @@ int slab_unmergeable(struct kmem_cache *s)
 		return 1;
 #endif
 
+	if (s->cpu_sheaves)
+		return 1;
+
 	/*
 	 * We may have set a slab to be unmergeable during bootstrap.
 	 */
@@ -321,7 +324,7 @@ struct kmem_cache *__kmem_cache_create_args(const char *name,
 		    object_size - args->usersize < args->useroffset))
 		args->usersize = args->useroffset = 0;
 
-	if (!args->usersize)
+	if (!args->usersize && !args->sheaf_capacity)
 		s = __kmem_cache_alias(name, object_size, args->align, flags,
 				       args->ctor);
 	if (s)
diff --git a/mm/slub.c b/mm/slub.c
index 9f671ec76131c4b0b28d5d568aa45842b5efb6d4..cba188b7e04ddf86debf9bc27a2f725db1b2056e 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -363,8 +363,10 @@ static inline void debugfs_slab_add(struct kmem_cache *s) { }
 #endif
 
 enum stat_item {
+	ALLOC_PCS,		/* Allocation from percpu sheaf */
 	ALLOC_FASTPATH,		/* Allocation from cpu slab */
 	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
+	FREE_PCS,		/* Free to percpu sheaf */
 	FREE_FASTPATH,		/* Free to cpu slab */
 	FREE_SLOWPATH,		/* Freeing not to cpu slab */
 	FREE_FROZEN,		/* Freeing to frozen slab */
@@ -389,6 +391,14 @@ enum stat_item {
 	CPU_PARTIAL_FREE,	/* Refill cpu partial on free */
 	CPU_PARTIAL_NODE,	/* Refill cpu partial from node partial */
 	CPU_PARTIAL_DRAIN,	/* Drain cpu partial to node partial */
+	SHEAF_FLUSH,		/* Objects flushed from a sheaf */
+	SHEAF_REFILL,		/* Objects refilled to a sheaf */
+	SHEAF_ALLOC,		/* Allocation of an empty sheaf */
+	SHEAF_FREE,		/* Freeing of an empty sheaf */
+	BARN_GET,		/* Got full sheaf from barn */
+	BARN_GET_FAIL,		/* Failed to get full sheaf from barn */
+	BARN_PUT,		/* Put full sheaf to barn */
+	BARN_PUT_FAIL,		/* Failed to put full sheaf to barn */
 	NR_SLUB_STAT_ITEMS
 };
 
@@ -435,6 +445,32 @@ void stat_add(const struct kmem_cache *s, enum stat_item si, int v)
 #endif
 }
 
+#define MAX_FULL_SHEAVES	10
+#define MAX_EMPTY_SHEAVES	10
+
+struct node_barn {
+	spinlock_t lock;
+	struct list_head sheaves_full;
+	struct list_head sheaves_empty;
+	unsigned int nr_full;
+	unsigned int nr_empty;
+};
+
+struct slab_sheaf {
+	union {
+		struct rcu_head rcu_head;
+		struct list_head barn_list;
+	};
+	unsigned int size;
+	void *objects[];
+};
+
+struct slub_percpu_sheaves {
+	local_trylock_t lock;
+	struct slab_sheaf *main; /* never NULL when unlocked */
+	struct slab_sheaf *spare; /* empty or full, may be NULL */
+};
+
 /*
  * The slab lists for all objects.
  */
@@ -447,6 +483,7 @@ struct kmem_cache_node {
 	atomic_long_t total_objects;
 	struct list_head full;
 #endif
+	struct node_barn *barn;
 };
 
 static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
@@ -454,6 +491,12 @@ static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
 	return s->node[node];
 }
 
+/* Get the barn of the current cpu's memory node */
+static inline struct node_barn *get_barn(struct kmem_cache *s)
+{
+	return get_node(s, numa_mem_id())->barn;
+}
+
 /*
  * Iterator over all nodes. The body will be executed for each node that has
  * a kmem_cache_node structure allocated (which is true for all online nodes)
@@ -470,12 +513,19 @@ static inline struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
  */
 static nodemask_t slab_nodes;
 
-#ifndef CONFIG_SLUB_TINY
 /*
  * Workqueue used for flush_cpu_slab().
  */
 static struct workqueue_struct *flushwq;
-#endif
+
+struct slub_flush_work {
+	struct work_struct work;
+	struct kmem_cache *s;
+	bool skip;
+};
+
+static DEFINE_MUTEX(flush_lock);
+static DEFINE_PER_CPU(struct slub_flush_work, slub_flush);
 
 /********************************************************************
  * 			Core slab cache functions
@@ -2473,6 +2523,360 @@ static void *setup_object(struct kmem_cache *s, void *object)
 	return object;
 }
 
+static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slab_sheaf *sheaf = kzalloc(struct_size(sheaf, objects,
+					s->sheaf_capacity), gfp);
+
+	if (unlikely(!sheaf))
+		return NULL;
+
+	stat(s, SHEAF_ALLOC);
+
+	return sheaf;
+}
+
+static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+	kfree(sheaf);
+
+	stat(s, SHEAF_FREE);
+}
+
+static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
+				   size_t size, void **p);
+
+
+static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
+			 gfp_t gfp)
+{
+	int to_fill = s->sheaf_capacity - sheaf->size;
+	int filled;
+
+	if (!to_fill)
+		return 0;
+
+	filled = __kmem_cache_alloc_bulk(s, gfp, to_fill,
+					 &sheaf->objects[sheaf->size]);
+
+	sheaf->size += filled;
+
+	stat_add(s, SHEAF_REFILL, filled);
+
+	if (filled < to_fill)
+		return -ENOMEM;
+
+	return 0;
+}
+
+
+static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slab_sheaf *sheaf = alloc_empty_sheaf(s, gfp);
+
+	if (!sheaf)
+		return NULL;
+
+	if (refill_sheaf(s, sheaf, gfp)) {
+		free_empty_sheaf(s, sheaf);
+		return NULL;
+	}
+
+	return sheaf;
+}
+
+/*
+ * Maximum number of objects freed during a single flush of main pcs sheaf.
+ * Translates directly to an on-stack array size.
+ */
+#define PCS_BATCH_MAX	32U
+
+static void __kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p);
+
+/*
+ * Free all objects from the main sheaf. In order to perform
+ * __kmem_cache_free_bulk() outside of cpu_sheaves->lock, work in batches where
+ * object pointers are moved to a on-stack array under the lock. To bound the
+ * stack usage, limit each batch to PCS_BATCH_MAX.
+ *
+ * returns true if at least partially flushed
+ */
+static bool sheaf_flush_main(struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+	unsigned int batch, remaining;
+	void *objects[PCS_BATCH_MAX];
+	struct slab_sheaf *sheaf;
+	bool ret = false;
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return ret;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	sheaf = pcs->main;
+
+	batch = min(PCS_BATCH_MAX, sheaf->size);
+
+	sheaf->size -= batch;
+	memcpy(objects, sheaf->objects + sheaf->size, batch * sizeof(void *));
+
+	remaining = sheaf->size;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	__kmem_cache_free_bulk(s, batch, &objects[0]);
+
+	stat_add(s, SHEAF_FLUSH, batch);
+
+	ret = true;
+
+	if (remaining)
+		goto next_batch;
+
+	return ret;
+}
+
+/*
+ * Free all objects from a sheaf that's unused, i.e. not linked to any
+ * cpu_sheaves, so we need no locking and batching. The locking is also not
+ * necessary when flushing cpu's sheaves (both spare and main) during cpu
+ * hotremove as the cpu is not executing anymore.
+ */
+static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
+{
+	if (!sheaf->size)
+		return;
+
+	stat_add(s, SHEAF_FLUSH, sheaf->size);
+
+	__kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
+
+	sheaf->size = 0;
+}
+
+/*
+ * Caller needs to make sure migration is disabled in order to fully flush
+ * single cpu's sheaves
+ *
+ * must not be called from an irq
+ *
+ * flushing operations are rare so let's keep it simple and flush to slabs
+ * directly, skipping the barn
+ */
+static void pcs_flush_all(struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *spare;
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	spare = pcs->spare;
+	pcs->spare = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (spare) {
+		sheaf_flush_unused(s, spare);
+		free_empty_sheaf(s, spare);
+	}
+
+	sheaf_flush_main(s);
+}
+
+static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+	/* The cpu is not executing anymore so we don't need pcs->lock */
+	sheaf_flush_unused(s, pcs->main);
+	if (pcs->spare) {
+		sheaf_flush_unused(s, pcs->spare);
+		free_empty_sheaf(s, pcs->spare);
+		pcs->spare = NULL;
+	}
+}
+
+static void pcs_destroy(struct kmem_cache *s)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct slub_percpu_sheaves *pcs;
+
+		pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+		/* can happen when unwinding failed create */
+		if (!pcs->main)
+			continue;
+
+		/*
+		 * We have already passed __kmem_cache_shutdown() so everything
+		 * was flushed and there should be no objects allocated from
+		 * slabs, otherwise kmem_cache_destroy() would have aborted.
+		 * Therefore something would have to be really wrong if the
+		 * warnings here trigger, and we should rather leave objects and
+		 * sheaves to leak in that case.
+		 */
+
+		WARN_ON(pcs->spare);
+
+		if (!WARN_ON(pcs->main->size)) {
+			free_empty_sheaf(s, pcs->main);
+			pcs->main = NULL;
+		}
+	}
+
+	free_percpu(s->cpu_sheaves);
+	s->cpu_sheaves = NULL;
+}
+
+static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn)
+{
+	struct slab_sheaf *empty = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_empty) {
+		empty = list_first_entry(&barn->sheaves_empty,
+					 struct slab_sheaf, barn_list);
+		list_del(&empty->barn_list);
+		barn->nr_empty--;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return empty;
+}
+
+/*
+ * The following two functions are used mainly in cases where we have to undo an
+ * intended action due to a race or cpu migration. Thus they do not check the
+ * empty or full sheaf limits for simplicity.
+ */
+
+static void barn_put_empty_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_add(&sheaf->barn_list, &barn->sheaves_empty);
+	barn->nr_empty++;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+}
+
+static void barn_put_full_sheaf(struct node_barn *barn, struct slab_sheaf *sheaf)
+{
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_add(&sheaf->barn_list, &barn->sheaves_full);
+	barn->nr_full++;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+}
+
+/*
+ * If a full sheaf is available, return it and put the supplied empty one to
+ * barn. We ignore the limit on empty sheaves as the number of sheaves doesn't
+ * change.
+ */
+static struct slab_sheaf *
+barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty)
+{
+	struct slab_sheaf *full = NULL;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full) {
+		full = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
+					barn_list);
+		list_del(&full->barn_list);
+		list_add(&empty->barn_list, &barn->sheaves_empty);
+		barn->nr_full--;
+		barn->nr_empty++;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return full;
+}
+
+/*
+ * If an empty sheaf is available, return it and put the supplied full one to
+ * barn. But if there are too many full sheaves, reject this with -E2BIG.
+ */
+static struct slab_sheaf *
+barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full)
+{
+	struct slab_sheaf *empty;
+	unsigned long flags;
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	if (barn->nr_full >= MAX_FULL_SHEAVES) {
+		empty = ERR_PTR(-E2BIG);
+	} else if (!barn->nr_empty) {
+		empty = ERR_PTR(-ENOMEM);
+	} else {
+		empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf,
+					 barn_list);
+		list_del(&empty->barn_list);
+		list_add(&full->barn_list, &barn->sheaves_full);
+		barn->nr_empty--;
+		barn->nr_full++;
+	}
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	return empty;
+}
+
+static void barn_init(struct node_barn *barn)
+{
+	spin_lock_init(&barn->lock);
+	INIT_LIST_HEAD(&barn->sheaves_full);
+	INIT_LIST_HEAD(&barn->sheaves_empty);
+	barn->nr_full = 0;
+	barn->nr_empty = 0;
+}
+
+static void barn_shrink(struct kmem_cache *s, struct node_barn *barn)
+{
+	struct list_head empty_list;
+	struct list_head full_list;
+	struct slab_sheaf *sheaf, *sheaf2;
+	unsigned long flags;
+
+	INIT_LIST_HEAD(&empty_list);
+	INIT_LIST_HEAD(&full_list);
+
+	spin_lock_irqsave(&barn->lock, flags);
+
+	list_splice_init(&barn->sheaves_full, &full_list);
+	barn->nr_full = 0;
+	list_splice_init(&barn->sheaves_empty, &empty_list);
+	barn->nr_empty = 0;
+
+	spin_unlock_irqrestore(&barn->lock, flags);
+
+	list_for_each_entry_safe(sheaf, sheaf2, &full_list, barn_list) {
+		sheaf_flush_unused(s, sheaf);
+		free_empty_sheaf(s, sheaf);
+	}
+
+	list_for_each_entry_safe(sheaf, sheaf2, &empty_list, barn_list)
+		free_empty_sheaf(s, sheaf);
+}
+
 /*
  * Slab allocation and freeing
  */
@@ -3344,11 +3748,40 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu)
 	put_partials_cpu(s, c);
 }
 
-struct slub_flush_work {
-	struct work_struct work;
-	struct kmem_cache *s;
-	bool skip;
-};
+static inline void flush_this_cpu_slab(struct kmem_cache *s)
+{
+	struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab);
+
+	if (c->slab)
+		flush_slab(s, c);
+
+	put_partials(s);
+}
+
+static bool has_cpu_slab(int cpu, struct kmem_cache *s)
+{
+	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
+
+	return c->slab || slub_percpu_partial(c);
+}
+
+#else /* CONFIG_SLUB_TINY */
+static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
+static inline bool has_cpu_slab(int cpu, struct kmem_cache *s) { return false; }
+static inline void flush_this_cpu_slab(struct kmem_cache *s) { }
+#endif /* CONFIG_SLUB_TINY */
+
+static bool has_pcs_used(int cpu, struct kmem_cache *s)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	if (!s->cpu_sheaves)
+		return false;
+
+	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+	return (pcs->spare || pcs->main->size);
+}
 
 /*
  * Flush cpu slab.
@@ -3358,30 +3791,18 @@ struct slub_flush_work {
 static void flush_cpu_slab(struct work_struct *w)
 {
 	struct kmem_cache *s;
-	struct kmem_cache_cpu *c;
 	struct slub_flush_work *sfw;
 
 	sfw = container_of(w, struct slub_flush_work, work);
 
 	s = sfw->s;
-	c = this_cpu_ptr(s->cpu_slab);
-
-	if (c->slab)
-		flush_slab(s, c);
-
-	put_partials(s);
-}
 
-static bool has_cpu_slab(int cpu, struct kmem_cache *s)
-{
-	struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu);
+	if (s->cpu_sheaves)
+		pcs_flush_all(s);
 
-	return c->slab || slub_percpu_partial(c);
+	flush_this_cpu_slab(s);
 }
 
-static DEFINE_MUTEX(flush_lock);
-static DEFINE_PER_CPU(struct slub_flush_work, slub_flush);
-
 static void flush_all_cpus_locked(struct kmem_cache *s)
 {
 	struct slub_flush_work *sfw;
@@ -3392,7 +3813,7 @@ static void flush_all_cpus_locked(struct kmem_cache *s)
 
 	for_each_online_cpu(cpu) {
 		sfw = &per_cpu(slub_flush, cpu);
-		if (!has_cpu_slab(cpu, s)) {
+		if (!has_cpu_slab(cpu, s) && !has_pcs_used(cpu, s)) {
 			sfw->skip = true;
 			continue;
 		}
@@ -3428,19 +3849,15 @@ static int slub_cpu_dead(unsigned int cpu)
 	struct kmem_cache *s;
 
 	mutex_lock(&slab_mutex);
-	list_for_each_entry(s, &slab_caches, list)
+	list_for_each_entry(s, &slab_caches, list) {
 		__flush_cpu_slab(s, cpu);
+		if (s->cpu_sheaves)
+			__pcs_flush_all_cpu(s, cpu);
+	}
 	mutex_unlock(&slab_mutex);
 	return 0;
 }
 
-#else /* CONFIG_SLUB_TINY */
-static inline void flush_all_cpus_locked(struct kmem_cache *s) { }
-static inline void flush_all(struct kmem_cache *s) { }
-static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { }
-static inline int slub_cpu_dead(unsigned int cpu) { return 0; }
-#endif /* CONFIG_SLUB_TINY */
-
 /*
  * Check if the objects in a per cpu structure fit numa
  * locality expectations.
@@ -4191,30 +4608,240 @@ bool slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 }
 
 /*
- * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
- * have the fastpath folded into their functions. So no function call
- * overhead for requests that can be satisfied on the fastpath.
- *
- * The fastpath works by first checking if the lockless freelist can be used.
- * If not then __slab_alloc is called for slow processing.
+ * Replace the empty main sheaf with a (at least partially) full sheaf.
  *
- * Otherwise we can simply pick the next object from the lockless free list.
+ * Must be called with the cpu_sheaves local lock locked. If successful, returns
+ * the pcs pointer and the local lock locked (possibly on a different cpu than
+ * initially called). If not successful, returns NULL and the local lock
+ * unlocked.
  */
-static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
-		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+static struct slub_percpu_sheaves *
+__pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, gfp_t gfp)
 {
-	void *object;
-	bool init = false;
+	struct slab_sheaf *empty = NULL;
+	struct slab_sheaf *full;
+	struct node_barn *barn;
+	bool can_alloc;
 
-	s = slab_pre_alloc_hook(s, gfpflags);
-	if (unlikely(!s))
-		return NULL;
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	if (pcs->spare && pcs->spare->size > 0) {
+		swap(pcs->main, pcs->spare);
+		return pcs;
+	}
+
+	barn = get_barn(s);
+
+	full = barn_replace_empty_sheaf(barn, pcs->main);
+
+	if (full) {
+		stat(s, BARN_GET);
+		pcs->main = full;
+		return pcs;
+	}
+
+	stat(s, BARN_GET_FAIL);
+
+	can_alloc = gfpflags_allow_blocking(gfp);
+
+	if (can_alloc) {
+		if (pcs->spare) {
+			empty = pcs->spare;
+			pcs->spare = NULL;
+		} else {
+			empty = barn_get_empty_sheaf(barn);
+		}
+	}
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (!can_alloc)
+		return NULL;
+
+	if (empty) {
+		if (!refill_sheaf(s, empty, gfp)) {
+			full = empty;
+		} else {
+			/*
+			 * we must be very low on memory so don't bother
+			 * with the barn
+			 */
+			free_empty_sheaf(s, empty);
+		}
+	} else {
+		full = alloc_full_sheaf(s, gfp);
+	}
+
+	if (!full)
+		return NULL;
+
+	/*
+	 * we can reach here only when gfpflags_allow_blocking
+	 * so this must not be an irq
+	 */
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	/*
+	 * If we are returning empty sheaf, we either got it from the
+	 * barn or had to allocate one. If we are returning a full
+	 * sheaf, it's due to racing or being migrated to a different
+	 * cpu. Breaching the barn's sheaf limits should be thus rare
+	 * enough so just ignore them to simplify the recovery.
+	 */
+
+	if (pcs->main->size == 0) {
+		barn_put_empty_sheaf(barn, pcs->main);
+		pcs->main = full;
+		return pcs;
+	}
+
+	if (!pcs->spare) {
+		pcs->spare = full;
+		return pcs;
+	}
+
+	if (pcs->spare->size == 0) {
+		barn_put_empty_sheaf(barn, pcs->spare);
+		pcs->spare = full;
+		return pcs;
+	}
+
+	barn_put_full_sheaf(barn, full);
+	stat(s, BARN_PUT);
+
+	return pcs;
+}
+
+static __fastpath_inline
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
+{
+	struct slub_percpu_sheaves *pcs;
+	void *object;
+
+#ifdef CONFIG_NUMA
+	if (static_branch_unlikely(&strict_numa)) {
+		if (current->mempolicy)
+			return NULL;
+	}
+#endif
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return NULL;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == 0)) {
+		pcs = __pcs_replace_empty_main(s, pcs, gfp);
+		if (unlikely(!pcs))
+			return NULL;
+	}
+
+	object = pcs->main->objects[--pcs->main->size];
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat(s, ALLOC_PCS);
+
+	return object;
+}
+
+static __fastpath_inline
+unsigned int alloc_from_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *main;
+	unsigned int allocated = 0;
+	unsigned int batch;
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return allocated;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == 0)) {
+
+		struct slab_sheaf *full;
+
+		if (pcs->spare && pcs->spare->size > 0) {
+			swap(pcs->main, pcs->spare);
+			goto do_alloc;
+		}
+
+		full = barn_replace_empty_sheaf(get_barn(s), pcs->main);
+
+		if (full) {
+			stat(s, BARN_GET);
+			pcs->main = full;
+			goto do_alloc;
+		}
+
+		stat(s, BARN_GET_FAIL);
+
+		local_unlock(&s->cpu_sheaves->lock);
+
+		/*
+		 * Once full sheaves in barn are depleted, let the bulk
+		 * allocation continue from slab pages, otherwise we would just
+		 * be copying arrays of pointers twice.
+		 */
+		return allocated;
+	}
+
+do_alloc:
+
+	main = pcs->main;
+	batch = min(size, main->size);
+
+	main->size -= batch;
+	memcpy(p, main->objects + main->size, batch * sizeof(void *));
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat_add(s, ALLOC_PCS, batch);
+
+	allocated += batch;
+
+	if (batch < size) {
+		p += batch;
+		size -= batch;
+		goto next_batch;
+	}
+
+	return allocated;
+}
+
+
+/*
+ * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc)
+ * have the fastpath folded into their functions. So no function call
+ * overhead for requests that can be satisfied on the fastpath.
+ *
+ * The fastpath works by first checking if the lockless freelist can be used.
+ * If not then __slab_alloc is called for slow processing.
+ *
+ * Otherwise we can simply pick the next object from the lockless free list.
+ */
+static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list_lru *lru,
+		gfp_t gfpflags, int node, unsigned long addr, size_t orig_size)
+{
+	void *object;
+	bool init = false;
+
+	s = slab_pre_alloc_hook(s, gfpflags);
+	if (unlikely(!s))
+		return NULL;
 
 	object = kfence_alloc(s, orig_size, gfpflags);
 	if (unlikely(object))
 		goto out;
 
-	object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
+	if (s->cpu_sheaves && node == NUMA_NO_NODE)
+		object = alloc_from_pcs(s, gfpflags);
+
+	if (!object)
+		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
 
 	maybe_wipe_obj_freeptr(s, object);
 	init = slab_want_init_on_alloc(gfpflags, s);
@@ -4591,6 +5218,295 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab,
 	discard_slab(s, slab);
 }
 
+/*
+ * pcs is locked. We should have get rid of the spare sheaf and obtained an
+ * empty sheaf, while the main sheaf is full. We want to install the empty sheaf
+ * as a main sheaf, and make the current main sheaf a spare sheaf.
+ *
+ * However due to having relinquished the cpu_sheaves lock when obtaining
+ * the empty sheaf, we need to handle some unlikely but possible cases.
+ *
+ * If we put any sheaf to barn here, it's because we were interrupted or have
+ * been migrated to a different cpu, which should be rare enough so just ignore
+ * the barn's limits to simplify the handling.
+ *
+ * An alternative scenario that gets us here is when we fail
+ * barn_replace_full_sheaf(), because there's no empty sheaf available in the
+ * barn, so we had to allocate it by alloc_empty_sheaf(). But because we saw the
+ * limit on full sheaves was not exceeded, we assume it didn't change and just
+ * put the full sheaf there.
+ */
+static void __pcs_install_empty_sheaf(struct kmem_cache *s,
+		struct slub_percpu_sheaves *pcs, struct slab_sheaf *empty)
+{
+	struct node_barn *barn;
+
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	/* This is what we expect to find if nobody interrupted us. */
+	if (likely(!pcs->spare)) {
+		pcs->spare = pcs->main;
+		pcs->main = empty;
+		return;
+	}
+
+	barn = get_barn(s);
+
+	/*
+	 * Unlikely because if the main sheaf had space, we would have just
+	 * freed to it. Get rid of our empty sheaf.
+	 */
+	if (pcs->main->size < s->sheaf_capacity) {
+		barn_put_empty_sheaf(barn, empty);
+		return;
+	}
+
+	/* Also unlikely for the same reason */
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		barn_put_empty_sheaf(barn, empty);
+		return;
+	}
+
+	/*
+	 * We probably failed barn_replace_full_sheaf() due to no empty sheaf
+	 * available there, but we allocated one, so finish the job.
+	 */
+	barn_put_full_sheaf(barn, pcs->main);
+	stat(s, BARN_PUT);
+	pcs->main = empty;
+}
+
+/*
+ * Replace the full main sheaf with a (at least partially) empty sheaf.
+ *
+ * Must be called with the cpu_sheaves local lock locked. If successful, returns
+ * the pcs pointer and the local lock locked (possibly on a different cpu than
+ * initially called). If not successful, returns NULL and the local lock
+ * unlocked.
+ */
+static struct slub_percpu_sheaves *
+__pcs_replace_full_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs)
+{
+	struct slab_sheaf *empty;
+	struct node_barn *barn;
+	bool put_fail;
+
+restart:
+	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
+
+	barn = get_barn(s);
+	put_fail = false;
+
+	if (!pcs->spare) {
+		empty = barn_get_empty_sheaf(barn);
+		if (empty) {
+			pcs->spare = pcs->main;
+			pcs->main = empty;
+			return pcs;
+		}
+		goto alloc_empty;
+	}
+
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		return pcs;
+	}
+
+	empty = barn_replace_full_sheaf(barn, pcs->main);
+
+	if (!IS_ERR(empty)) {
+		stat(s, BARN_PUT);
+		pcs->main = empty;
+		return pcs;
+	}
+
+	if (PTR_ERR(empty) == -E2BIG) {
+		/* Since we got here, spare exists and is full */
+		struct slab_sheaf *to_flush = pcs->spare;
+
+		stat(s, BARN_PUT_FAIL);
+
+		pcs->spare = NULL;
+		local_unlock(&s->cpu_sheaves->lock);
+
+		sheaf_flush_unused(s, to_flush);
+		empty = to_flush;
+		goto got_empty;
+	}
+
+	/*
+	 * We could not replace full sheaf because barn had no empty
+	 * sheaves. We can still allocate it and put the full sheaf in
+	 * __pcs_install_empty_sheaf(), but if we fail to allocate it,
+	 * make sure to count the fail.
+	 */
+	put_fail = true;
+
+alloc_empty:
+	local_unlock(&s->cpu_sheaves->lock);
+
+	empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+	if (empty)
+		goto got_empty;
+
+	if (put_fail)
+		 stat(s, BARN_PUT_FAIL);
+
+	if (!sheaf_flush_main(s))
+		return NULL;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return NULL;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	/*
+	 * we flushed the main sheaf so it should be empty now,
+	 * but in case we got preempted or migrated, we need to
+	 * check again
+	 */
+	if (pcs->main->size == s->sheaf_capacity)
+		goto restart;
+
+	return pcs;
+
+got_empty:
+	if (!local_trylock(&s->cpu_sheaves->lock)) {
+		barn_put_empty_sheaf(barn, empty);
+		return NULL;
+	}
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+	__pcs_install_empty_sheaf(s, pcs, empty);
+
+	return pcs;
+}
+
+/*
+ * Free an object to the percpu sheaves.
+ * The object is expected to have passed slab_free_hook() already.
+ */
+static __fastpath_inline
+bool free_to_pcs(struct kmem_cache *s, void *object)
+{
+	struct slub_percpu_sheaves *pcs;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		return false;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(pcs->main->size == s->sheaf_capacity)) {
+
+		pcs = __pcs_replace_full_main(s, pcs);
+		if (unlikely(!pcs))
+			return false;
+	}
+
+	pcs->main->objects[pcs->main->size++] = object;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat(s, FREE_PCS);
+
+	return true;
+}
+
+/*
+ * Bulk free objects to the percpu sheaves.
+ * Unlike free_to_pcs() this includes the calls to all necessary hooks
+ * and the fallback to freeing to slab pages.
+ */
+static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *main, *empty;
+	bool init = slab_want_init_on_free(s);
+	unsigned int batch, i = 0;
+	struct node_barn *barn;
+
+	while (i < size) {
+		struct slab *slab = virt_to_slab(p[i]);
+
+		memcg_slab_free_hook(s, slab, p + i, 1);
+		alloc_tagging_slab_free_hook(s, slab, p + i, 1);
+
+		if (unlikely(!slab_free_hook(s, p[i], init, false))) {
+			p[i] = p[--size];
+			if (!size)
+				return;
+			continue;
+		}
+
+		i++;
+	}
+
+next_batch:
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto fallback;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (likely(pcs->main->size < s->sheaf_capacity))
+		goto do_free;
+
+	barn = get_barn(s);
+
+	if (!pcs->spare) {
+		empty = barn_get_empty_sheaf(barn);
+		if (!empty)
+			goto no_empty;
+
+		pcs->spare = pcs->main;
+		pcs->main = empty;
+		goto do_free;
+	}
+
+	if (pcs->spare->size < s->sheaf_capacity) {
+		swap(pcs->main, pcs->spare);
+		goto do_free;
+	}
+
+	empty = barn_replace_full_sheaf(barn, pcs->main);
+	if (IS_ERR(empty)) {
+		stat(s, BARN_PUT_FAIL);
+		goto no_empty;
+	}
+
+	stat(s, BARN_PUT);
+	pcs->main = empty;
+
+do_free:
+	main = pcs->main;
+	batch = min(size, s->sheaf_capacity - main->size);
+
+	memcpy(main->objects + main->size, p, batch * sizeof(void *));
+	main->size += batch;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat_add(s, FREE_PCS, batch);
+
+	if (batch < size) {
+		p += batch;
+		size -= batch;
+		goto next_batch;
+	}
+
+	return;
+
+no_empty:
+	local_unlock(&s->cpu_sheaves->lock);
+
+	/*
+	 * if we depleted all empty sheaves in the barn or there are too
+	 * many full sheaves, free the rest to slab pages
+	 */
+fallback:
+	__kmem_cache_free_bulk(s, size, p);
+}
+
 #ifndef CONFIG_SLUB_TINY
 /*
  * Fastpath with forced inlining to produce a kfree and kmem_cache_free that
@@ -4677,7 +5593,10 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 	memcg_slab_free_hook(s, slab, &object, 1);
 	alloc_tagging_slab_free_hook(s, slab, &object, 1);
 
-	if (likely(slab_free_hook(s, object, slab_want_init_on_free(s), false)))
+	if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
+		return;
+
+	if (!s->cpu_sheaves || !free_to_pcs(s, object))
 		do_slab_free(s, slab, object, object, 1, addr);
 }
 
@@ -5273,6 +6192,15 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)
 	if (!size)
 		return;
 
+	/*
+	 * freeing to sheaves is so incompatible with the detached freelist so
+	 * once we go that way, we have to do everything differently
+	 */
+	if (s && s->cpu_sheaves) {
+		free_to_pcs_bulk(s, size, p);
+		return;
+	}
+
 	do {
 		struct detached_freelist df;
 
@@ -5391,7 +6319,7 @@ static int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags,
 int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 				 void **p)
 {
-	int i;
+	unsigned int i = 0;
 
 	if (!size)
 		return 0;
@@ -5400,9 +6328,20 @@ int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 	if (unlikely(!s))
 		return 0;
 
-	i = __kmem_cache_alloc_bulk(s, flags, size, p);
-	if (unlikely(i == 0))
-		return 0;
+	if (s->cpu_sheaves)
+		i = alloc_from_pcs_bulk(s, size, p);
+
+	if (i < size) {
+		/*
+		 * If we ran out of memory, don't bother with freeing back to
+		 * the percpu sheaves, we have bigger problems.
+		 */
+		if (unlikely(__kmem_cache_alloc_bulk(s, flags, size - i, p + i) == 0)) {
+			if (i > 0)
+				__kmem_cache_free_bulk(s, i, p);
+			return 0;
+		}
+	}
 
 	/*
 	 * memcg and kmem_cache debug support and memory initialization.
@@ -5412,11 +6351,11 @@ int kmem_cache_alloc_bulk_noprof(struct kmem_cache *s, gfp_t flags, size_t size,
 		    slab_want_init_on_alloc(flags, s), s->object_size))) {
 		return 0;
 	}
-	return i;
+
+	return size;
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk_noprof);
 
-
 /*
  * Object placement in a slab is made very easy because we always start at
  * offset 0. If we tune the size of the object to the alignment then we can
@@ -5550,7 +6489,7 @@ static inline int calculate_order(unsigned int size)
 }
 
 static void
-init_kmem_cache_node(struct kmem_cache_node *n)
+init_kmem_cache_node(struct kmem_cache_node *n, struct node_barn *barn)
 {
 	n->nr_partial = 0;
 	spin_lock_init(&n->list_lock);
@@ -5560,6 +6499,9 @@ init_kmem_cache_node(struct kmem_cache_node *n)
 	atomic_long_set(&n->total_objects, 0);
 	INIT_LIST_HEAD(&n->full);
 #endif
+	n->barn = barn;
+	if (barn)
+		barn_init(barn);
 }
 
 #ifndef CONFIG_SLUB_TINY
@@ -5590,6 +6532,26 @@ static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
 }
 #endif /* CONFIG_SLUB_TINY */
 
+static int init_percpu_sheaves(struct kmem_cache *s)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu) {
+		struct slub_percpu_sheaves *pcs;
+
+		pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+		local_trylock_init(&pcs->lock);
+
+		pcs->main = alloc_empty_sheaf(s, GFP_KERNEL);
+
+		if (!pcs->main)
+			return -ENOMEM;
+	}
+
+	return 0;
+}
+
 static struct kmem_cache *kmem_cache_node;
 
 /*
@@ -5625,7 +6587,7 @@ static void early_kmem_cache_node_alloc(int node)
 	slab->freelist = get_freepointer(kmem_cache_node, n);
 	slab->inuse = 1;
 	kmem_cache_node->node[node] = n;
-	init_kmem_cache_node(n);
+	init_kmem_cache_node(n, NULL);
 	inc_slabs_node(kmem_cache_node, node, slab->objects);
 
 	/*
@@ -5641,6 +6603,13 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
 	struct kmem_cache_node *n;
 
 	for_each_kmem_cache_node(s, node, n) {
+		if (n->barn) {
+			WARN_ON(n->barn->nr_full);
+			WARN_ON(n->barn->nr_empty);
+			kfree(n->barn);
+			n->barn = NULL;
+		}
+
 		s->node[node] = NULL;
 		kmem_cache_free(kmem_cache_node, n);
 	}
@@ -5649,6 +6618,8 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
 void __kmem_cache_release(struct kmem_cache *s)
 {
 	cache_random_seq_destroy(s);
+	if (s->cpu_sheaves)
+		pcs_destroy(s);
 #ifndef CONFIG_SLUB_TINY
 	free_percpu(s->cpu_slab);
 #endif
@@ -5661,18 +6632,29 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
 
 	for_each_node_mask(node, slab_nodes) {
 		struct kmem_cache_node *n;
+		struct node_barn *barn = NULL;
 
 		if (slab_state == DOWN) {
 			early_kmem_cache_node_alloc(node);
 			continue;
 		}
+
+		if (s->cpu_sheaves) {
+			barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, node);
+
+			if (!barn)
+				return 0;
+		}
+
 		n = kmem_cache_alloc_node(kmem_cache_node,
 						GFP_KERNEL, node);
-
-		if (!n)
+		if (!n) {
+			kfree(barn);
 			return 0;
+		}
+
+		init_kmem_cache_node(n, barn);
 
-		init_kmem_cache_node(n);
 		s->node[node] = n;
 	}
 	return 1;
@@ -5929,6 +6911,8 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
 	flush_all_cpus_locked(s);
 	/* Attempt to free all objects */
 	for_each_kmem_cache_node(s, node, n) {
+		if (n->barn)
+			barn_shrink(s, n->barn);
 		free_partial(s, n);
 		if (n->nr_partial || node_nr_slabs(n))
 			return 1;
@@ -6132,6 +7116,9 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)
 		for (i = 0; i < SHRINK_PROMOTE_MAX; i++)
 			INIT_LIST_HEAD(promote + i);
 
+		if (n->barn)
+			barn_shrink(s, n->barn);
+
 		spin_lock_irqsave(&n->list_lock, flags);
 
 		/*
@@ -6211,12 +7198,24 @@ static int slab_mem_going_online_callback(int nid)
 	 */
 	mutex_lock(&slab_mutex);
 	list_for_each_entry(s, &slab_caches, list) {
+		struct node_barn *barn = NULL;
+
 		/*
 		 * The structure may already exist if the node was previously
 		 * onlined and offlined.
 		 */
 		if (get_node(s, nid))
 			continue;
+
+		if (s->cpu_sheaves) {
+			barn = kmalloc_node(sizeof(*barn), GFP_KERNEL, nid);
+
+			if (!barn) {
+				ret = -ENOMEM;
+				goto out;
+			}
+		}
+
 		/*
 		 * XXX: kmem_cache_alloc_node will fallback to other nodes
 		 *      since memory is not yet available from the node that
@@ -6224,10 +7223,13 @@ static int slab_mem_going_online_callback(int nid)
 		 */
 		n = kmem_cache_alloc(kmem_cache_node, GFP_KERNEL);
 		if (!n) {
+			kfree(barn);
 			ret = -ENOMEM;
 			goto out;
 		}
-		init_kmem_cache_node(n);
+
+		init_kmem_cache_node(n, barn);
+
 		s->node[nid] = n;
 	}
 	/*
@@ -6440,6 +7442,17 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,
 
 	set_cpu_partial(s);
 
+	if (args->sheaf_capacity && !IS_ENABLED(CONFIG_SLUB_TINY)
+					&& !(s->flags & SLAB_DEBUG_FLAGS)) {
+		s->cpu_sheaves = alloc_percpu(struct slub_percpu_sheaves);
+		if (!s->cpu_sheaves) {
+			err = -ENOMEM;
+			goto out;
+		}
+		// TODO: increase capacity to grow slab_sheaf up to next kmalloc size?
+		s->sheaf_capacity = args->sheaf_capacity;
+	}
+
 #ifdef CONFIG_NUMA
 	s->remote_node_defrag_ratio = 1000;
 #endif
@@ -6456,6 +7469,12 @@ int do_kmem_cache_create(struct kmem_cache *s, const char *name,
 	if (!alloc_kmem_cache_cpus(s))
 		goto out;
 
+	if (s->cpu_sheaves) {
+		err = init_percpu_sheaves(s);
+		if (err)
+			goto out;
+	}
+
 	err = 0;
 
 	/* Mutex is not taken during early boot */
@@ -6908,6 +7927,12 @@ static ssize_t order_show(struct kmem_cache *s, char *buf)
 }
 SLAB_ATTR_RO(order);
 
+static ssize_t sheaf_capacity_show(struct kmem_cache *s, char *buf)
+{
+	return sysfs_emit(buf, "%u\n", s->sheaf_capacity);
+}
+SLAB_ATTR_RO(sheaf_capacity);
+
 static ssize_t min_partial_show(struct kmem_cache *s, char *buf)
 {
 	return sysfs_emit(buf, "%lu\n", s->min_partial);
@@ -7255,8 +8280,10 @@ static ssize_t text##_store(struct kmem_cache *s,		\
 }								\
 SLAB_ATTR(text);						\
 
+STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
 STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
 STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
+STAT_ATTR(FREE_PCS, free_cpu_sheaf);
 STAT_ATTR(FREE_FASTPATH, free_fastpath);
 STAT_ATTR(FREE_SLOWPATH, free_slowpath);
 STAT_ATTR(FREE_FROZEN, free_frozen);
@@ -7281,6 +8308,14 @@ STAT_ATTR(CPU_PARTIAL_ALLOC, cpu_partial_alloc);
 STAT_ATTR(CPU_PARTIAL_FREE, cpu_partial_free);
 STAT_ATTR(CPU_PARTIAL_NODE, cpu_partial_node);
 STAT_ATTR(CPU_PARTIAL_DRAIN, cpu_partial_drain);
+STAT_ATTR(SHEAF_FLUSH, sheaf_flush);
+STAT_ATTR(SHEAF_REFILL, sheaf_refill);
+STAT_ATTR(SHEAF_ALLOC, sheaf_alloc);
+STAT_ATTR(SHEAF_FREE, sheaf_free);
+STAT_ATTR(BARN_GET, barn_get);
+STAT_ATTR(BARN_GET_FAIL, barn_get_fail);
+STAT_ATTR(BARN_PUT, barn_put);
+STAT_ATTR(BARN_PUT_FAIL, barn_put_fail);
 #endif	/* CONFIG_SLUB_STATS */
 
 #ifdef CONFIG_KFENCE
@@ -7311,6 +8346,7 @@ static struct attribute *slab_attrs[] = {
 	&object_size_attr.attr,
 	&objs_per_slab_attr.attr,
 	&order_attr.attr,
+	&sheaf_capacity_attr.attr,
 	&min_partial_attr.attr,
 	&cpu_partial_attr.attr,
 	&objects_partial_attr.attr,
@@ -7342,8 +8378,10 @@ static struct attribute *slab_attrs[] = {
 	&remote_node_defrag_ratio_attr.attr,
 #endif
 #ifdef CONFIG_SLUB_STATS
+	&alloc_cpu_sheaf_attr.attr,
 	&alloc_fastpath_attr.attr,
 	&alloc_slowpath_attr.attr,
+	&free_cpu_sheaf_attr.attr,
 	&free_fastpath_attr.attr,
 	&free_slowpath_attr.attr,
 	&free_frozen_attr.attr,
@@ -7368,6 +8406,14 @@ static struct attribute *slab_attrs[] = {
 	&cpu_partial_free_attr.attr,
 	&cpu_partial_node_attr.attr,
 	&cpu_partial_drain_attr.attr,
+	&sheaf_flush_attr.attr,
+	&sheaf_refill_attr.attr,
+	&sheaf_alloc_attr.attr,
+	&sheaf_free_attr.attr,
+	&barn_get_attr.attr,
+	&barn_get_fail_attr.attr,
+	&barn_put_attr.attr,
+	&barn_put_fail_attr.attr,
 #endif
 #ifdef CONFIG_FAILSLAB
 	&failslab_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:08 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:08 +0200
Subject: [PATCH v8 06/23] slab: determine barn status racily outside of
 lock
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-6-ca3099d8352c@suse.cz>

The possibility of many barn operations is determined by the current
number of full or empty sheaves. Taking the barn->lock just to find out
that e.g. there are no empty sheaves results in unnecessary overhead and
lock contention. Thus perform these checks outside of the lock with a
data_race() annotated variable read and fail quickly without taking the
lock.

Checks for sheaf availability that racily succeed have to be obviously
repeated under the lock for correctness, but we can skip repeating
checks if there are too many sheaves on the given list as the limits
don't need to be strict.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 38f5b865d3093556171e0f6530d395718b438099..35274ce4e709c9da7ac8f9006c824f28709e923d 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -2801,9 +2801,12 @@ static struct slab_sheaf *barn_get_empty_sheaf(struct node_barn *barn)
 	struct slab_sheaf *empty = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_empty))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_empty) {
+	if (likely(barn->nr_empty)) {
 		empty = list_first_entry(&barn->sheaves_empty,
 					 struct slab_sheaf, barn_list);
 		list_del(&empty->barn_list);
@@ -2850,6 +2853,9 @@ static struct slab_sheaf *barn_get_full_or_empty_sheaf(struct node_barn *barn)
 	struct slab_sheaf *sheaf = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_full) && !data_race(barn->nr_empty))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
 	if (barn->nr_full) {
@@ -2880,9 +2886,12 @@ barn_replace_empty_sheaf(struct node_barn *barn, struct slab_sheaf *empty)
 	struct slab_sheaf *full = NULL;
 	unsigned long flags;
 
+	if (!data_race(barn->nr_full))
+		return NULL;
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_full) {
+	if (likely(barn->nr_full)) {
 		full = list_first_entry(&barn->sheaves_full, struct slab_sheaf,
 					barn_list);
 		list_del(&full->barn_list);
@@ -2906,19 +2915,23 @@ barn_replace_full_sheaf(struct node_barn *barn, struct slab_sheaf *full)
 	struct slab_sheaf *empty;
 	unsigned long flags;
 
+	/* we don't repeat this check under barn->lock as it's not critical */
+	if (data_race(barn->nr_full) >= MAX_FULL_SHEAVES)
+		return ERR_PTR(-E2BIG);
+	if (!data_race(barn->nr_empty))
+		return ERR_PTR(-ENOMEM);
+
 	spin_lock_irqsave(&barn->lock, flags);
 
-	if (barn->nr_full >= MAX_FULL_SHEAVES) {
-		empty = ERR_PTR(-E2BIG);
-	} else if (!barn->nr_empty) {
-		empty = ERR_PTR(-ENOMEM);
-	} else {
+	if (likely(barn->nr_empty)) {
 		empty = list_first_entry(&barn->sheaves_empty, struct slab_sheaf,
 					 barn_list);
 		list_del(&empty->barn_list);
 		list_add(&full->barn_list, &barn->sheaves_full);
 		barn->nr_empty--;
 		barn->nr_full++;
+	} else {
+		empty = ERR_PTR(-ENOMEM);
 	}
 
 	spin_unlock_irqrestore(&barn->lock, flags);

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:13 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:13 +0200
Subject: [PATCH v8 11/23] maple_tree: Drop bulk insert support
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-11-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Bulk insert mode was added to facilitate forking faster, but forking now
uses __mt_dup() to duplicate the tree.

The addition of sheaves has made the bulk allocations difficult to
maintain - since the expected entries would preallocate into the maple
state.  A big part of the maple state node allocation was the ability to
push nodes back onto the state for later use, which was essential to the
bulk insert algorithm.

Remove mas_expected_entries() and mas_destroy_rebalance() functions as
well as the MA_STATE_BULK and MA_STATE_REBALANCE maple state flags since
there are no users anymore.  Drop the associated testing as well.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c                 | 270 +--------------------------------------
 lib/test_maple_tree.c            | 137 --------------------
 tools/testing/radix-tree/maple.c |  36 ------
 3 files changed, 4 insertions(+), 439 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 38fb68c082915211c80f473d313159599fe97e2c..4f0e30b57b0cef9e5cf791f3f64f5898752db402 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -83,13 +83,9 @@
 
 /*
  * Maple state flags
- * * MA_STATE_BULK		- Bulk insert mode
- * * MA_STATE_REBALANCE		- Indicate a rebalance during bulk insert
  * * MA_STATE_PREALLOC		- Preallocated nodes, WARN_ON allocation
  */
-#define MA_STATE_BULK		1
-#define MA_STATE_REBALANCE	2
-#define MA_STATE_PREALLOC	4
+#define MA_STATE_PREALLOC	1
 
 #define ma_parent_ptr(x) ((struct maple_pnode *)(x))
 #define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT)
@@ -1031,24 +1027,6 @@ static inline void mas_descend(struct ma_state *mas)
 	mas->node = mas_slot(mas, slots, mas->offset);
 }
 
-/*
- * mte_set_gap() - Set a maple node gap.
- * @mn: The encoded maple node
- * @gap: The offset of the gap to set
- * @val: The gap value
- */
-static inline void mte_set_gap(const struct maple_enode *mn,
-				 unsigned char gap, unsigned long val)
-{
-	switch (mte_node_type(mn)) {
-	default:
-		break;
-	case maple_arange_64:
-		mte_to_node(mn)->ma64.gap[gap] = val;
-		break;
-	}
-}
-
 /*
  * mas_ascend() - Walk up a level of the tree.
  * @mas: The maple state
@@ -1878,21 +1856,7 @@ static inline int mab_calc_split(struct ma_state *mas,
 	 * end on a NULL entry, with the exception of the left-most leaf.  The
 	 * limitation means that the split of a node must be checked for this condition
 	 * and be able to put more data in one direction or the other.
-	 */
-	if (unlikely((mas->mas_flags & MA_STATE_BULK))) {
-		*mid_split = 0;
-		split = b_end - mt_min_slots[bn->type];
-
-		if (!ma_is_leaf(bn->type))
-			return split;
-
-		mas->mas_flags |= MA_STATE_REBALANCE;
-		if (!bn->slot[split])
-			split--;
-		return split;
-	}
-
-	/*
+	 *
 	 * Although extremely rare, it is possible to enter what is known as the 3-way
 	 * split scenario.  The 3-way split comes about by means of a store of a range
 	 * that overwrites the end and beginning of two full nodes.  The result is a set
@@ -2039,27 +2003,6 @@ static inline void mab_mas_cp(struct maple_big_node *b_node,
 	}
 }
 
-/*
- * mas_bulk_rebalance() - Rebalance the end of a tree after a bulk insert.
- * @mas: The maple state
- * @end: The maple node end
- * @mt: The maple node type
- */
-static inline void mas_bulk_rebalance(struct ma_state *mas, unsigned char end,
-				      enum maple_type mt)
-{
-	if (!(mas->mas_flags & MA_STATE_BULK))
-		return;
-
-	if (mte_is_root(mas->node))
-		return;
-
-	if (end > mt_min_slots[mt]) {
-		mas->mas_flags &= ~MA_STATE_REBALANCE;
-		return;
-	}
-}
-
 /*
  * mas_store_b_node() - Store an @entry into the b_node while also copying the
  * data from a maple encoded node.
@@ -2109,9 +2052,6 @@ static noinline_for_kasan void mas_store_b_node(struct ma_wr_state *wr_mas,
 	/* Handle new range ending before old range ends */
 	piv = mas_safe_pivot(mas, wr_mas->pivots, offset_end, wr_mas->type);
 	if (piv > mas->last) {
-		if (piv == ULONG_MAX)
-			mas_bulk_rebalance(mas, b_node->b_end, wr_mas->type);
-
 		if (offset_end != slot)
 			wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
 							  offset_end);
@@ -3011,126 +2951,6 @@ static inline void mas_rebalance(struct ma_state *mas,
 	return mas_spanning_rebalance(mas, &mast, empty_count);
 }
 
-/*
- * mas_destroy_rebalance() - Rebalance left-most node while destroying the maple
- * state.
- * @mas: The maple state
- * @end: The end of the left-most node.
- *
- * During a mass-insert event (such as forking), it may be necessary to
- * rebalance the left-most node when it is not sufficient.
- */
-static inline void mas_destroy_rebalance(struct ma_state *mas, unsigned char end)
-{
-	enum maple_type mt = mte_node_type(mas->node);
-	struct maple_node reuse, *newnode, *parent, *new_left, *left, *node;
-	struct maple_enode *eparent, *old_eparent;
-	unsigned char offset, tmp, split = mt_slots[mt] / 2;
-	void __rcu **l_slots, **slots;
-	unsigned long *l_pivs, *pivs, gap;
-	bool in_rcu = mt_in_rcu(mas->tree);
-	unsigned char new_height = mas_mt_height(mas);
-
-	MA_STATE(l_mas, mas->tree, mas->index, mas->last);
-
-	l_mas = *mas;
-	mas_prev_sibling(&l_mas);
-
-	/* set up node. */
-	if (in_rcu) {
-		newnode = mas_pop_node(mas);
-	} else {
-		newnode = &reuse;
-	}
-
-	node = mas_mn(mas);
-	newnode->parent = node->parent;
-	slots = ma_slots(newnode, mt);
-	pivs = ma_pivots(newnode, mt);
-	left = mas_mn(&l_mas);
-	l_slots = ma_slots(left, mt);
-	l_pivs = ma_pivots(left, mt);
-	if (!l_slots[split])
-		split++;
-	tmp = mas_data_end(&l_mas) - split;
-
-	memcpy(slots, l_slots + split + 1, sizeof(void *) * tmp);
-	memcpy(pivs, l_pivs + split + 1, sizeof(unsigned long) * tmp);
-	pivs[tmp] = l_mas.max;
-	memcpy(slots + tmp, ma_slots(node, mt), sizeof(void *) * end);
-	memcpy(pivs + tmp, ma_pivots(node, mt), sizeof(unsigned long) * end);
-
-	l_mas.max = l_pivs[split];
-	mas->min = l_mas.max + 1;
-	old_eparent = mt_mk_node(mte_parent(l_mas.node),
-			     mas_parent_type(&l_mas, l_mas.node));
-	tmp += end;
-	if (!in_rcu) {
-		unsigned char max_p = mt_pivots[mt];
-		unsigned char max_s = mt_slots[mt];
-
-		if (tmp < max_p)
-			memset(pivs + tmp, 0,
-			       sizeof(unsigned long) * (max_p - tmp));
-
-		if (tmp < mt_slots[mt])
-			memset(slots + tmp, 0, sizeof(void *) * (max_s - tmp));
-
-		memcpy(node, newnode, sizeof(struct maple_node));
-		ma_set_meta(node, mt, 0, tmp - 1);
-		mte_set_pivot(old_eparent, mte_parent_slot(l_mas.node),
-			      l_pivs[split]);
-
-		/* Remove data from l_pivs. */
-		tmp = split + 1;
-		memset(l_pivs + tmp, 0, sizeof(unsigned long) * (max_p - tmp));
-		memset(l_slots + tmp, 0, sizeof(void *) * (max_s - tmp));
-		ma_set_meta(left, mt, 0, split);
-		eparent = old_eparent;
-
-		goto done;
-	}
-
-	/* RCU requires replacing both l_mas, mas, and parent. */
-	mas->node = mt_mk_node(newnode, mt);
-	ma_set_meta(newnode, mt, 0, tmp);
-
-	new_left = mas_pop_node(mas);
-	new_left->parent = left->parent;
-	mt = mte_node_type(l_mas.node);
-	slots = ma_slots(new_left, mt);
-	pivs = ma_pivots(new_left, mt);
-	memcpy(slots, l_slots, sizeof(void *) * split);
-	memcpy(pivs, l_pivs, sizeof(unsigned long) * split);
-	ma_set_meta(new_left, mt, 0, split);
-	l_mas.node = mt_mk_node(new_left, mt);
-
-	/* replace parent. */
-	offset = mte_parent_slot(mas->node);
-	mt = mas_parent_type(&l_mas, l_mas.node);
-	parent = mas_pop_node(mas);
-	slots = ma_slots(parent, mt);
-	pivs = ma_pivots(parent, mt);
-	memcpy(parent, mte_to_node(old_eparent), sizeof(struct maple_node));
-	rcu_assign_pointer(slots[offset], mas->node);
-	rcu_assign_pointer(slots[offset - 1], l_mas.node);
-	pivs[offset - 1] = l_mas.max;
-	eparent = mt_mk_node(parent, mt);
-done:
-	gap = mas_leaf_max_gap(mas);
-	mte_set_gap(eparent, mte_parent_slot(mas->node), gap);
-	gap = mas_leaf_max_gap(&l_mas);
-	mte_set_gap(eparent, mte_parent_slot(l_mas.node), gap);
-	mas_ascend(mas);
-
-	if (in_rcu) {
-		mas_replace_node(mas, old_eparent, new_height);
-		mas_adopt_children(mas, mas->node);
-	}
-
-	mas_update_gap(mas);
-}
-
 /*
  * mas_split_final_node() - Split the final node in a subtree operation.
  * @mast: the maple subtree state
@@ -3837,8 +3657,6 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas,
 
 	if (mas->last == wr_mas->end_piv)
 		offset_end++; /* don't copy this offset */
-	else if (unlikely(wr_mas->r_max == ULONG_MAX))
-		mas_bulk_rebalance(mas, mas->end, wr_mas->type);
 
 	/* set up node. */
 	if (in_rcu) {
@@ -4255,7 +4073,7 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
 	new_end = mas_wr_new_end(wr_mas);
 	/* Potential spanning rebalance collapsing a node */
 	if (new_end < mt_min_slots[wr_mas->type]) {
-		if (!mte_is_root(mas->node) && !(mas->mas_flags & MA_STATE_BULK))
+		if (!mte_is_root(mas->node))
 			return  wr_rebalance;
 		return wr_node_store;
 	}
@@ -5562,25 +5380,7 @@ void mas_destroy(struct ma_state *mas)
 	struct maple_alloc *node;
 	unsigned long total;
 
-	/*
-	 * When using mas_for_each() to insert an expected number of elements,
-	 * it is possible that the number inserted is less than the expected
-	 * number.  To fix an invalid final node, a check is performed here to
-	 * rebalance the previous node with the final node.
-	 */
-	if (mas->mas_flags & MA_STATE_REBALANCE) {
-		unsigned char end;
-		if (mas_is_err(mas))
-			mas_reset(mas);
-		mas_start(mas);
-		mtree_range_walk(mas);
-		end = mas->end + 1;
-		if (end < mt_min_slot_count(mas->node) - 1)
-			mas_destroy_rebalance(mas, end);
-
-		mas->mas_flags &= ~MA_STATE_REBALANCE;
-	}
-	mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
+	mas->mas_flags &= ~MA_STATE_PREALLOC;
 
 	total = mas_allocated(mas);
 	while (total) {
@@ -5600,68 +5400,6 @@ void mas_destroy(struct ma_state *mas)
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
-/*
- * mas_expected_entries() - Set the expected number of entries that will be inserted.
- * @mas: The maple state
- * @nr_entries: The number of expected entries.
- *
- * This will attempt to pre-allocate enough nodes to store the expected number
- * of entries.  The allocations will occur using the bulk allocator interface
- * for speed.  Please call mas_destroy() on the @mas after inserting the entries
- * to ensure any unused nodes are freed.
- *
- * Return: 0 on success, -ENOMEM if memory could not be allocated.
- */
-int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries)
-{
-	int nonleaf_cap = MAPLE_ARANGE64_SLOTS - 2;
-	struct maple_enode *enode = mas->node;
-	int nr_nodes;
-	int ret;
-
-	/*
-	 * Sometimes it is necessary to duplicate a tree to a new tree, such as
-	 * forking a process and duplicating the VMAs from one tree to a new
-	 * tree.  When such a situation arises, it is known that the new tree is
-	 * not going to be used until the entire tree is populated.  For
-	 * performance reasons, it is best to use a bulk load with RCU disabled.
-	 * This allows for optimistic splitting that favours the left and reuse
-	 * of nodes during the operation.
-	 */
-
-	/* Optimize splitting for bulk insert in-order */
-	mas->mas_flags |= MA_STATE_BULK;
-
-	/*
-	 * Avoid overflow, assume a gap between each entry and a trailing null.
-	 * If this is wrong, it just means allocation can happen during
-	 * insertion of entries.
-	 */
-	nr_nodes = max(nr_entries, nr_entries * 2 + 1);
-	if (!mt_is_alloc(mas->tree))
-		nonleaf_cap = MAPLE_RANGE64_SLOTS - 2;
-
-	/* Leaves; reduce slots to keep space for expansion */
-	nr_nodes = DIV_ROUND_UP(nr_nodes, MAPLE_RANGE64_SLOTS - 2);
-	/* Internal nodes */
-	nr_nodes += DIV_ROUND_UP(nr_nodes, nonleaf_cap);
-	/* Add working room for split (2 nodes) + new parents */
-	mas_node_count_gfp(mas, nr_nodes + 3, GFP_KERNEL);
-
-	/* Detect if allocations run out */
-	mas->mas_flags |= MA_STATE_PREALLOC;
-
-	if (!mas_is_err(mas))
-		return 0;
-
-	ret = xa_err(mas->node);
-	mas->node = enode;
-	mas_destroy(mas);
-	return ret;
-
-}
-EXPORT_SYMBOL_GPL(mas_expected_entries);
-
 static void mas_may_activate(struct ma_state *mas)
 {
 	if (!mas->node) {
diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c
index cb3936595b0d56a9682ff100eba54693a1427829..14fbbee32046a13d54d60dcac2b45be2bd190ac4 100644
--- a/lib/test_maple_tree.c
+++ b/lib/test_maple_tree.c
@@ -2746,139 +2746,6 @@ static noinline void __init check_fuzzer(struct maple_tree *mt)
 	mtree_test_erase(mt, ULONG_MAX - 10);
 }
 
-/* duplicate the tree with a specific gap */
-static noinline void __init check_dup_gaps(struct maple_tree *mt,
-				    unsigned long nr_entries, bool zero_start,
-				    unsigned long gap)
-{
-	unsigned long i = 0;
-	struct maple_tree newmt;
-	int ret;
-	void *tmp;
-	MA_STATE(mas, mt, 0, 0);
-	MA_STATE(newmas, &newmt, 0, 0);
-	struct rw_semaphore newmt_lock;
-
-	init_rwsem(&newmt_lock);
-	mt_set_external_lock(&newmt, &newmt_lock);
-
-	if (!zero_start)
-		i = 1;
-
-	mt_zero_nr_tallocated();
-	for (; i <= nr_entries; i++)
-		mtree_store_range(mt, i*10, (i+1)*10 - gap,
-				  xa_mk_value(i), GFP_KERNEL);
-
-	mt_init_flags(&newmt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN);
-	mt_set_non_kernel(99999);
-	down_write(&newmt_lock);
-	ret = mas_expected_entries(&newmas, nr_entries);
-	mt_set_non_kernel(0);
-	MT_BUG_ON(mt, ret != 0);
-
-	rcu_read_lock();
-	mas_for_each(&mas, tmp, ULONG_MAX) {
-		newmas.index = mas.index;
-		newmas.last = mas.last;
-		mas_store(&newmas, tmp);
-	}
-	rcu_read_unlock();
-	mas_destroy(&newmas);
-
-	__mt_destroy(&newmt);
-	up_write(&newmt_lock);
-}
-
-/* Duplicate many sizes of trees.  Mainly to test expected entry values */
-static noinline void __init check_dup(struct maple_tree *mt)
-{
-	int i;
-	int big_start = 100010;
-
-	/* Check with a value at zero */
-	for (i = 10; i < 1000; i++) {
-		mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-		check_dup_gaps(mt, i, true, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-	}
-
-	cond_resched();
-	mt_cache_shrink();
-	/* Check with a value at zero, no gap */
-	for (i = 1000; i < 2000; i++) {
-		mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-		check_dup_gaps(mt, i, true, 0);
-		mtree_destroy(mt);
-		rcu_barrier();
-	}
-
-	cond_resched();
-	mt_cache_shrink();
-	/* Check with a value at zero and unreasonably large */
-	for (i = big_start; i < big_start + 10; i++) {
-		mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-		check_dup_gaps(mt, i, true, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-	}
-
-	cond_resched();
-	mt_cache_shrink();
-	/* Small to medium size not starting at zero*/
-	for (i = 200; i < 1000; i++) {
-		mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-		check_dup_gaps(mt, i, false, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-	}
-
-	cond_resched();
-	mt_cache_shrink();
-	/* Unreasonably large not starting at zero*/
-	for (i = big_start; i < big_start + 10; i++) {
-		mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-		check_dup_gaps(mt, i, false, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-		cond_resched();
-		mt_cache_shrink();
-	}
-
-	/* Check non-allocation tree not starting at zero */
-	for (i = 1500; i < 3000; i++) {
-		mt_init_flags(mt, 0);
-		check_dup_gaps(mt, i, false, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-		cond_resched();
-		if (i % 2 == 0)
-			mt_cache_shrink();
-	}
-
-	mt_cache_shrink();
-	/* Check non-allocation tree starting at zero */
-	for (i = 200; i < 1000; i++) {
-		mt_init_flags(mt, 0);
-		check_dup_gaps(mt, i, true, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-		cond_resched();
-	}
-
-	mt_cache_shrink();
-	/* Unreasonably large */
-	for (i = big_start + 5; i < big_start + 10; i++) {
-		mt_init_flags(mt, 0);
-		check_dup_gaps(mt, i, true, 5);
-		mtree_destroy(mt);
-		rcu_barrier();
-		mt_cache_shrink();
-		cond_resched();
-	}
-}
-
 static noinline void __init check_bnode_min_spanning(struct maple_tree *mt)
 {
 	int i = 50;
@@ -4077,10 +3944,6 @@ static int __init maple_tree_seed(void)
 	check_fuzzer(&tree);
 	mtree_destroy(&tree);
 
-	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
-	check_dup(&tree);
-	mtree_destroy(&tree);
-
 	mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 	check_bnode_min_spanning(&tree);
 	mtree_destroy(&tree);
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 172700fb7784d29f9403003b4484a5ebd7aa316b..c0543060dae2510477963331fb0ccdffd78ea965 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -35455,17 +35455,6 @@ static void check_dfs_preorder(struct maple_tree *mt)
 	MT_BUG_ON(mt, count != e);
 	mtree_destroy(mt);
 
-	mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
-	mas_reset(&mas);
-	mt_zero_nr_tallocated();
-	mt_set_non_kernel(200);
-	mas_expected_entries(&mas, max);
-	for (count = 0; count <= max; count++) {
-		mas.index = mas.last = count;
-		mas_store(&mas, xa_mk_value(count));
-		MT_BUG_ON(mt, mas_is_err(&mas));
-	}
-	mas_destroy(&mas);
 	rcu_barrier();
 	/*
 	 * pr_info(" ->seq test of 0-%lu %luK in %d active (%d total)\n",
@@ -36454,27 +36443,6 @@ static inline int check_vma_modification(struct maple_tree *mt)
 	return 0;
 }
 
-/*
- * test to check that bulk stores do not use wr_rebalance as the store
- * type.
- */
-static inline void check_bulk_rebalance(struct maple_tree *mt)
-{
-	MA_STATE(mas, mt, ULONG_MAX, ULONG_MAX);
-	int max = 10;
-
-	build_full_tree(mt, 0, 2);
-
-	/* erase every entry in the tree */
-	do {
-		/* set up bulk store mode */
-		mas_expected_entries(&mas, max);
-		mas_erase(&mas);
-		MT_BUG_ON(mt, mas.store_type == wr_rebalance);
-	} while (mas_prev(&mas, 0) != NULL);
-
-	mas_destroy(&mas);
-}
 
 void farmer_tests(void)
 {
@@ -36487,10 +36455,6 @@ void farmer_tests(void)
 	check_vma_modification(&tree);
 	mtree_destroy(&tree);
 
-	mt_init(&tree);
-	check_bulk_rebalance(&tree);
-	mtree_destroy(&tree);
-
 	tree.ma_root = xa_mk_value(0);
 	mt_dump(&tree, mt_dump_dec);
 

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:10 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:10 +0200
Subject: [PATCH v8 08/23] slab: allow NUMA restricted allocations to use
 percpu sheaves
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-8-ca3099d8352c@suse.cz>

Currently allocations asking for a specific node explicitly or via
mempolicy in strict_numa node bypass percpu sheaves. Since sheaves
contain mostly local objects, we can try allocating from them if the
local node happens to be the requested node or allowed by the mempolicy.
If we find the object from percpu sheaves is not from the expected node,
we skip the sheaves - this should be rare.

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slub.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 46 insertions(+), 7 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
index 9699d048b2cd08ee75c4cc3d1e460868704520b1..3746c0229cc2f9658a589416c63c21fbf2850c44 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4888,18 +4888,43 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 }
 
 static __fastpath_inline
-void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
+void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
 {
 	struct slub_percpu_sheaves *pcs;
+	bool node_requested;
 	void *object;
 
 #ifdef CONFIG_NUMA
-	if (static_branch_unlikely(&strict_numa)) {
-		if (current->mempolicy)
-			return NULL;
+	if (static_branch_unlikely(&strict_numa) &&
+			 node == NUMA_NO_NODE) {
+
+		struct mempolicy *mpol = current->mempolicy;
+
+		if (mpol) {
+			/*
+			 * Special BIND rule support. If the local node
+			 * is in permitted set then do not redirect
+			 * to a particular node.
+			 * Otherwise we apply the memory policy to get
+			 * the node we need to allocate on.
+			 */
+			if (mpol->mode != MPOL_BIND ||
+					!node_isset(numa_mem_id(), mpol->nodes))
+
+				node = mempolicy_slab_node();
+		}
 	}
 #endif
 
+	node_requested = IS_ENABLED(CONFIG_NUMA) && node != NUMA_NO_NODE;
+
+	/*
+	 * We assume the percpu sheaves contain only local objects although it's
+	 * not completely guaranteed, so we verify later.
+	 */
+	if (unlikely(node_requested && node != numa_mem_id()))
+		return NULL;
+
 	if (!local_trylock(&s->cpu_sheaves->lock))
 		return NULL;
 
@@ -4911,7 +4936,21 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
 			return NULL;
 	}
 
-	object = pcs->main->objects[--pcs->main->size];
+	object = pcs->main->objects[pcs->main->size - 1];
+
+	if (unlikely(node_requested)) {
+		/*
+		 * Verify that the object was from the node we want. This could
+		 * be false because of cpu migration during an unlocked part of
+		 * the current allocation or previous freeing process.
+		 */
+		if (folio_nid(virt_to_folio(object)) != node) {
+			local_unlock(&s->cpu_sheaves->lock);
+			return NULL;
+		}
+	}
+
+	pcs->main->size--;
 
 	local_unlock(&s->cpu_sheaves->lock);
 
@@ -5011,8 +5050,8 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
 	if (unlikely(object))
 		goto out;
 
-	if (s->cpu_sheaves && node == NUMA_NO_NODE)
-		object = alloc_from_pcs(s, gfpflags);
+	if (s->cpu_sheaves)
+		object = alloc_from_pcs(s, gfpflags, node);
 
 	if (!object)
 		object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:09 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:09 +0200
Subject: [PATCH v8 07/23] slab: skip percpu sheaves for remote object
 freeing
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-7-ca3099d8352c@suse.cz>

Since we don't control the NUMA locality of objects in percpu sheaves,
allocations with node restrictions bypass them. Allocations without
restrictions may however still expect to get local objects with high
probability, and the introduction of sheaves can decrease it due to
freed object from a remote node ending up in percpu sheaves.

The fraction of such remote frees seems low (5% on an 8-node machine)
but it can be expected that some cache or workload specific corner cases
exist. We can either conclude that this is not a problem due to the low
fraction, or we can make remote frees bypass percpu sheaves and go
directly to their slabs. This will make the remote frees more expensive,
but if if's only a small fraction, most frees will still benefit from
the lower overhead of percpu sheaves.

This patch thus makes remote object freeing bypass percpu sheaves,
including bulk freeing, and kfree_rcu() via the rcu_free sheaf. However
it's not intended to be 100% guarantee that percpu sheaves will only
contain local objects. The refill from slabs does not provide that
guarantee in the first place, and there might be cpu migrations
happening when we need to unlock the local_lock. Avoiding all that could
be possible but complicated so we can leave it for later investigation
whether it would be worth it. It can be expected that the more selective
freeing will itself prevent accumulation of remote objects in percpu
sheaves so any such violations would have only short-term effects.

Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab_common.c |  7 +++++--
 mm/slub.c        | 42 ++++++++++++++++++++++++++++++++++++------
 2 files changed, 41 insertions(+), 8 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 005a4319c06a01d2b616a75396fcc43766a62ddb..b6601e0fe598e24bd8d456dce4fc82c65b342bfd 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1623,8 +1623,11 @@ static bool kfree_rcu_sheaf(void *obj)
 
 	slab = folio_slab(folio);
 	s = slab->slab_cache;
-	if (s->cpu_sheaves)
-		return __kfree_rcu_sheaf(s, obj);
+	if (s->cpu_sheaves) {
+		if (likely(!IS_ENABLED(CONFIG_NUMA) ||
+			   slab_nid(slab) == numa_mem_id()))
+			return __kfree_rcu_sheaf(s, obj);
+	}
 
 	return false;
 }
diff --git a/mm/slub.c b/mm/slub.c
index 35274ce4e709c9da7ac8f9006c824f28709e923d..9699d048b2cd08ee75c4cc3d1e460868704520b1 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -472,6 +472,7 @@ struct slab_sheaf {
 	};
 	struct kmem_cache *cache;
 	unsigned int size;
+	int node; /* only used for rcu_sheaf */
 	void *objects[];
 };
 
@@ -5828,7 +5829,7 @@ static void rcu_free_sheaf(struct rcu_head *head)
 	 */
 	__rcu_free_sheaf_prepare(s, sheaf);
 
-	barn = get_node(s, numa_mem_id())->barn;
+	barn = get_node(s, sheaf->node)->barn;
 
 	/* due to slab_free_hook() */
 	if (unlikely(sheaf->size == 0))
@@ -5914,10 +5915,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 
 	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
 
-	if (likely(rcu_sheaf->size < s->sheaf_capacity))
+	if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
 		rcu_sheaf = NULL;
-	else
+	} else {
 		pcs->rcu_free = NULL;
+		rcu_sheaf->node = numa_mem_id();
+	}
 
 	local_unlock(&s->cpu_sheaves->lock);
 
@@ -5944,7 +5947,11 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	bool init = slab_want_init_on_free(s);
 	unsigned int batch, i = 0;
 	struct node_barn *barn;
+	void *remote_objects[PCS_BATCH_MAX];
+	unsigned int remote_nr = 0;
+	int node = numa_mem_id();
 
+next_remote_batch:
 	while (i < size) {
 		struct slab *slab = virt_to_slab(p[i]);
 
@@ -5954,7 +5961,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 		if (unlikely(!slab_free_hook(s, p[i], init, false))) {
 			p[i] = p[--size];
 			if (!size)
-				return;
+				goto flush_remote;
+			continue;
+		}
+
+		if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != node)) {
+			remote_objects[remote_nr] = p[i];
+			p[i] = p[--size];
+			if (++remote_nr >= PCS_BATCH_MAX)
+				goto flush_remote;
 			continue;
 		}
 
@@ -6024,6 +6039,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
 	 */
 fallback:
 	__kmem_cache_free_bulk(s, size, p);
+
+flush_remote:
+	if (remote_nr) {
+		__kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]);
+		if (i < size) {
+			remote_nr = 0;
+			goto next_remote_batch;
+		}
+	}
 }
 
 #ifndef CONFIG_SLUB_TINY
@@ -6115,8 +6139,14 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
 	if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
 		return;
 
-	if (!s->cpu_sheaves || !free_to_pcs(s, object))
-		do_slab_free(s, slab, object, object, 1, addr);
+	if (s->cpu_sheaves && likely(!IS_ENABLED(CONFIG_NUMA) ||
+				     slab_nid(slab) == numa_mem_id())) {
+		if (likely(free_to_pcs(s, object))) {
+			return;
+		}
+	}
+
+	do_slab_free(s, slab, object, object, 1, addr);
 }
 
 #ifdef CONFIG_MEMCG

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:19 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:19 +0200
Subject: [PATCH v8 17/23] testing/radix-tree/maple: Hack around kfree_rcu
 not existing
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-17-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

liburcu doesn't have kfree_rcu (or anything similar). Despite that, we
can hack around it in a trivial fashion, by adding a wrapper.

The wrapper only works for maple_nodes because we cannot get the
kmem_cache pointer any other way in the test code.

Link: https://lore.kernel.org/all/20250812162124.59417-1-pfalcato at suse.de/
Suggested-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/shared/maple-shared.h | 11 +++++++++++
 tools/testing/shared/maple-shim.c   |  6 ++++++
 2 files changed, 17 insertions(+)

diff --git a/tools/testing/shared/maple-shared.h b/tools/testing/shared/maple-shared.h
index dc4d30f3860b9bd23b4177c7d7926ac686887815..2a1e9a8594a2834326cd9374738b2a2c7c3f9f7c 100644
--- a/tools/testing/shared/maple-shared.h
+++ b/tools/testing/shared/maple-shared.h
@@ -10,4 +10,15 @@
 #include <time.h>
 #include "linux/init.h"
 
+void maple_rcu_cb(struct rcu_head *head);
+#define rcu_cb		maple_rcu_cb
+
+#define kfree_rcu(_struct, _memb)		\
+do {                                            \
+    typeof(_struct) _p_struct = (_struct);      \
+                                                \
+    call_rcu(&((_p_struct)->_memb), rcu_cb);    \
+} while(0);
+
+
 #endif /* __MAPLE_SHARED_H__ */
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 9d7b743415660305416e972fa75b56824211b0eb..16252ee616c0489c80490ff25b8d255427bf9fdc 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -6,3 +6,9 @@
 #include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
+
+void maple_rcu_cb(struct rcu_head *head) {
+	struct maple_node *node = container_of(head, struct maple_node, rcu);
+
+	kmem_cache_free(maple_node_cache, node);
+}

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:18 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:18 +0200
Subject: [PATCH v8 16/23] tools/testing: include maple-shim.c in maple.c
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-16-ca3099d8352c@suse.cz>

There's some duplicated code and we are about to add more functionality
in maple-shared.h that we will need in the userspace maple test to be
available, so include it via maple-shim.c

Co-developed-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/radix-tree/maple.c | 12 +++---------
 1 file changed, 3 insertions(+), 9 deletions(-)

diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index c0543060dae2510477963331fb0ccdffd78ea965..4a35e1e7c64b7ce347cbd1693beeaacb0c4c330e 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -8,14 +8,6 @@
  * difficult to handle in kernel tests.
  */
 
-#define CONFIG_DEBUG_MAPLE_TREE
-#define CONFIG_MAPLE_SEARCH
-#define MAPLE_32BIT (MAPLE_NODE_SLOTS > 31)
-#include "test.h"
-#include <stdlib.h>
-#include <time.h>
-#include <linux/init.h>
-
 #define module_init(x)
 #define module_exit(x)
 #define MODULE_AUTHOR(x)
@@ -23,7 +15,9 @@
 #define MODULE_LICENSE(x)
 #define dump_stack()	assert(0)
 
-#include "../../../lib/maple_tree.c"
+#include "test.h"
+
+#include "../shared/maple-shim.c"
 #include "../../../lib/test_maple_tree.c"
 
 #define RCU_RANGE_COUNT 1000

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:16 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:16 +0200
Subject: [PATCH v8 14/23] mm, vma: use percpu sheaves for vm_area_struct
 cache
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-14-ca3099d8352c@suse.cz>

Create the vm_area_struct cache with percpu sheaves of size 32 to
improve its performance.

Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/vma_init.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/vma_init.c b/mm/vma_init.c
index 8e53c7943561e7324e7992946b4065dec1149b82..52c6b55fac4519e0da39ca75ad018e14449d1d95 100644
--- a/mm/vma_init.c
+++ b/mm/vma_init.c
@@ -16,6 +16,7 @@ void __init vma_state_init(void)
 	struct kmem_cache_args args = {
 		.use_freeptr_offset = true,
 		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
+		.sheaf_capacity = 32,
 	};
 
 	vm_area_cachep = kmem_cache_create("vm_area_struct",

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:15 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:15 +0200
Subject: [PATCH v8 13/23] tools/testing: Add support for changes to slab
 for sheaves
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-13-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The slab changes for sheaves requires more effort in the testing code.
Unite all the kmem_cache work into the tools/include slab header for
both the vma and maple tree testing.

The vma test code also requires importing more #defines to allow for
seamless use of the shared kmem_cache code.

This adds the pthread header to the slab header in the tools directory
to allow for the pthread_mutex in linux.c.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h        | 137 ++++++++++++++++++++++++++++++++++++--
 tools/testing/shared/linux.c      |  26 ++------
 tools/testing/shared/maple-shim.c |   1 +
 tools/testing/vma/vma_internal.h  |  92 +------------------------
 4 files changed, 142 insertions(+), 114 deletions(-)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c87051e2b26f5a7fee0362697fae067076b8e84d..c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -4,11 +4,31 @@
 
 #include <linux/types.h>
 #include <linux/gfp.h>
+#include <pthread.h>
 
-#define SLAB_PANIC 2
 #define SLAB_RECLAIM_ACCOUNT    0x00020000UL            /* Objects are reclaimable */
 
 #define kzalloc_node(size, flags, node) kmalloc(size, flags)
+enum _slab_flag_bits {
+	_SLAB_KMALLOC,
+	_SLAB_HWCACHE_ALIGN,
+	_SLAB_PANIC,
+	_SLAB_TYPESAFE_BY_RCU,
+	_SLAB_ACCOUNT,
+	_SLAB_FLAGS_LAST_BIT
+};
+
+#define __SLAB_FLAG_BIT(nr)	((unsigned int __force)(1U << (nr)))
+#define __SLAB_FLAG_UNUSED	((unsigned int __force)(0U))
+
+#define SLAB_HWCACHE_ALIGN	__SLAB_FLAG_BIT(_SLAB_HWCACHE_ALIGN)
+#define SLAB_PANIC		__SLAB_FLAG_BIT(_SLAB_PANIC)
+#define SLAB_TYPESAFE_BY_RCU	__SLAB_FLAG_BIT(_SLAB_TYPESAFE_BY_RCU)
+#ifdef CONFIG_MEMCG
+# define SLAB_ACCOUNT		__SLAB_FLAG_BIT(_SLAB_ACCOUNT)
+#else
+# define SLAB_ACCOUNT		__SLAB_FLAG_UNUSED
+#endif
 
 void *kmalloc(size_t size, gfp_t gfp);
 void kfree(void *p);
@@ -23,6 +43,86 @@ enum slab_state {
 	FULL
 };
 
+struct kmem_cache {
+	pthread_mutex_t lock;
+	unsigned int size;
+	unsigned int align;
+	unsigned int sheaf_capacity;
+	int nr_objs;
+	void *objs;
+	void (*ctor)(void *);
+	bool non_kernel_enabled;
+	unsigned int non_kernel;
+	unsigned long nr_allocated;
+	unsigned long nr_tallocated;
+	bool exec_callback;
+	void (*callback)(void *);
+	void *private;
+};
+
+struct kmem_cache_args {
+	/**
+	 * @align: The required alignment for the objects.
+	 *
+	 * %0 means no specific alignment is requested.
+	 */
+	unsigned int align;
+	/**
+	 * @sheaf_capacity: The maximum size of the sheaf.
+	 */
+	unsigned int sheaf_capacity;
+	/**
+	 * @useroffset: Usercopy region offset.
+	 *
+	 * %0 is a valid offset, when @usersize is non-%0
+	 */
+	unsigned int useroffset;
+	/**
+	 * @usersize: Usercopy region size.
+	 *
+	 * %0 means no usercopy region is specified.
+	 */
+	unsigned int usersize;
+	/**
+	 * @freeptr_offset: Custom offset for the free pointer
+	 * in &SLAB_TYPESAFE_BY_RCU caches
+	 *
+	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
+	 * outside of the object. This might cause the object to grow in size.
+	 * Cache creators that have a reason to avoid this can specify a custom
+	 * free pointer offset in their struct where the free pointer will be
+	 * placed.
+	 *
+	 * Note that placing the free pointer inside the object requires the
+	 * caller to ensure that no fields are invalidated that are required to
+	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
+	 * details).
+	 *
+	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
+	 * is specified, %use_freeptr_offset must be set %true.
+	 *
+	 * Note that @ctor currently isn't supported with custom free pointers
+	 * as a @ctor requires an external free pointer.
+	 */
+	unsigned int freeptr_offset;
+	/**
+	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
+	 */
+	bool use_freeptr_offset;
+	/**
+	 * @ctor: A constructor for the objects.
+	 *
+	 * The constructor is invoked for each object in a newly allocated slab
+	 * page. It is the cache user's responsibility to free object in the
+	 * same state as after calling the constructor, or deal appropriately
+	 * with any differences between a freshly constructed and a reallocated
+	 * object.
+	 *
+	 * %NULL means no constructor.
+	 */
+	void (*ctor)(void *);
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -37,9 +137,38 @@ static inline void *kmem_cache_alloc(struct kmem_cache *cachep, int flags)
 }
 void kmem_cache_free(struct kmem_cache *cachep, void *objp);
 
-struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
-			unsigned int align, unsigned int flags,
-			void (*ctor)(void *));
+
+struct kmem_cache *
+__kmem_cache_create_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags);
+
+/* If NULL is passed for @args, use this variant with default arguments. */
+static inline struct kmem_cache *
+__kmem_cache_default_args(const char *name, unsigned int size,
+		struct kmem_cache_args *args, unsigned int flags)
+{
+	struct kmem_cache_args kmem_default_args = {};
+
+	return __kmem_cache_create_args(name, size, &kmem_default_args, flags);
+}
+
+static inline struct kmem_cache *
+__kmem_cache_create(const char *name, unsigned int size, unsigned int align,
+		unsigned int flags, void (*ctor)(void *))
+{
+	struct kmem_cache_args kmem_args = {
+		.align	= align,
+		.ctor	= ctor,
+	};
+
+	return __kmem_cache_create_args(name, size, &kmem_args, flags);
+}
+
+#define kmem_cache_create(__name, __object_size, __args, ...)           \
+	_Generic((__args),                                              \
+		struct kmem_cache_args *: __kmem_cache_create_args,	\
+		void *: __kmem_cache_default_args,			\
+		default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__)
 
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 0f97fb0d19e19c327aa4843a35b45cc086f4f366..97b8412ccbb6d222604c7b397c53c65618d8d51b 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -16,21 +16,6 @@ int nr_allocated;
 int preempt_count;
 int test_verbose;
 
-struct kmem_cache {
-	pthread_mutex_t lock;
-	unsigned int size;
-	unsigned int align;
-	int nr_objs;
-	void *objs;
-	void (*ctor)(void *);
-	unsigned int non_kernel;
-	unsigned long nr_allocated;
-	unsigned long nr_tallocated;
-	bool exec_callback;
-	void (*callback)(void *);
-	void *private;
-};
-
 void kmem_cache_set_callback(struct kmem_cache *cachep, void (*callback)(void *))
 {
 	cachep->callback = callback;
@@ -234,23 +219,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 }
 
 struct kmem_cache *
-kmem_cache_create(const char *name, unsigned int size, unsigned int align,
-		unsigned int flags, void (*ctor)(void *))
+__kmem_cache_create_args(const char *name, unsigned int size,
+			  struct kmem_cache_args *args,
+			  unsigned int flags)
 {
 	struct kmem_cache *ret = malloc(sizeof(*ret));
 
 	pthread_mutex_init(&ret->lock, NULL);
 	ret->size = size;
-	ret->align = align;
+	ret->align = args->align;
+	ret->sheaf_capacity = args->sheaf_capacity;
 	ret->nr_objs = 0;
 	ret->nr_allocated = 0;
 	ret->nr_tallocated = 0;
 	ret->objs = NULL;
-	ret->ctor = ctor;
+	ret->ctor = args->ctor;
 	ret->non_kernel = 0;
 	ret->exec_callback = false;
 	ret->callback = NULL;
 	ret->private = NULL;
+
 	return ret;
 }
 
diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
index 640df76f483e09f3b6f85612786060dd273e2362..9d7b743415660305416e972fa75b56824211b0eb 100644
--- a/tools/testing/shared/maple-shim.c
+++ b/tools/testing/shared/maple-shim.c
@@ -3,5 +3,6 @@
 /* Very simple shim around the maple tree. */
 
 #include "maple-shared.h"
+#include <linux/slab.h>
 
 #include "../../../lib/maple_tree.c"
diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 6b6e2b05918c9f95b537f26e20a943b34082825a..d5b87fa6a133f6d676488de2538c509e0f0e1d54 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -26,6 +26,7 @@
 #include <linux/mm.h>
 #include <linux/rbtree.h>
 #include <linux/refcount.h>
+#include <linux/slab.h>
 
 extern unsigned long stack_guard_gap;
 #ifdef CONFIG_MMU
@@ -509,65 +510,6 @@ struct pagetable_move_control {
 		.len_in = len_,						\
 	}
 
-struct kmem_cache_args {
-	/**
-	 * @align: The required alignment for the objects.
-	 *
-	 * %0 means no specific alignment is requested.
-	 */
-	unsigned int align;
-	/**
-	 * @useroffset: Usercopy region offset.
-	 *
-	 * %0 is a valid offset, when @usersize is non-%0
-	 */
-	unsigned int useroffset;
-	/**
-	 * @usersize: Usercopy region size.
-	 *
-	 * %0 means no usercopy region is specified.
-	 */
-	unsigned int usersize;
-	/**
-	 * @freeptr_offset: Custom offset for the free pointer
-	 * in &SLAB_TYPESAFE_BY_RCU caches
-	 *
-	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
-	 * outside of the object. This might cause the object to grow in size.
-	 * Cache creators that have a reason to avoid this can specify a custom
-	 * free pointer offset in their struct where the free pointer will be
-	 * placed.
-	 *
-	 * Note that placing the free pointer inside the object requires the
-	 * caller to ensure that no fields are invalidated that are required to
-	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
-	 * details).
-	 *
-	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
-	 * is specified, %use_freeptr_offset must be set %true.
-	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
-	 */
-	unsigned int freeptr_offset;
-	/**
-	 * @use_freeptr_offset: Whether a @freeptr_offset is used.
-	 */
-	bool use_freeptr_offset;
-	/**
-	 * @ctor: A constructor for the objects.
-	 *
-	 * The constructor is invoked for each object in a newly allocated slab
-	 * page. It is the cache user's responsibility to free object in the
-	 * same state as after calling the constructor, or deal appropriately
-	 * with any differences between a freshly constructed and a reallocated
-	 * object.
-	 *
-	 * %NULL means no constructor.
-	 */
-	void (*ctor)(void *);
-};
-
 static inline void vma_iter_invalidate(struct vma_iterator *vmi)
 {
 	mas_pause(&vmi->mas);
@@ -652,38 +594,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_lock_seq = UINT_MAX;
 }
 
-struct kmem_cache {
-	const char *name;
-	size_t object_size;
-	struct kmem_cache_args *args;
-};
-
-static inline struct kmem_cache *__kmem_cache_create(const char *name,
-						     size_t object_size,
-						     struct kmem_cache_args *args)
-{
-	struct kmem_cache *ret = malloc(sizeof(struct kmem_cache));
-
-	ret->name = name;
-	ret->object_size = object_size;
-	ret->args = args;
-
-	return ret;
-}
-
-#define kmem_cache_create(__name, __object_size, __args, ...)           \
-	__kmem_cache_create((__name), (__object_size), (__args))
-
-static inline void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
-{
-	return calloc(1, s->object_size);
-}
-
-static inline void kmem_cache_free(struct kmem_cache *s, void *x)
-{
-	free(x);
-}
-
 /*
  * These are defined in vma.h, but sadly vm_stat_account() is referenced by
  * kernel/fork.c, so we have to these broadly available there, and temporarily

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:12 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:12 +0200
Subject: [PATCH v8 10/23] tools/testing/vma: clean up stubs in
 vma_internal.h
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-10-ca3099d8352c@suse.cz>

From: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

We do not need to references arguments just to avoid compiler warnings,
the warning in question does not arise here, so remove all of the
instances of '(void)xxx' introduced purely to avoid this warning.

As reported by WagYuli in the referenced mail, GCC 8.3 and before will
have issues compiling this file if parameter names are not provided, so
ensure these are always provided.

Finally, perform a trivial fix up of kmem_cache_alloc() which technically
has parameters in the incorrect order (as reported by Vlastimil Babka
off-list).

Link: https://lkml.kernel.org/r/20250826102824.22730-1-lorenzo.stoakes at oracle.com
Signed-off-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
Reported-by: WangYuli <wangyuli at uniontech.com>
Closes: https://lore.kernel.org/linux-mm/EFCEBE7E301589DE+20250729084700.208767-1-wangyuli at uniontech.com/
Reported-by: Vlastimil Babka <vbabka at suse.cz>
Acked-by: Vlastimil Babka <vbabka at suse.cz>
Reviewed-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Cc: Jann Horn <jannh at google.com>
Cc: WangYuli <wangyuli at uniontech.com>
Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/testing/vma/vma_internal.h | 167 +++++++++++++--------------------------
 1 file changed, 57 insertions(+), 110 deletions(-)

diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
index 3639aa8dd2b06ebe5b9cfcfe6669994fd38c482d..f8cf5b184d5b51dd627ff440943a7af3c549f482 100644
--- a/tools/testing/vma/vma_internal.h
+++ b/tools/testing/vma/vma_internal.h
@@ -676,9 +676,7 @@ static inline struct kmem_cache *__kmem_cache_create(const char *name,
 
 static inline void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
 {
-	(void)gfpflags;
-
-	return calloc(s->object_size, 1);
+	return calloc(1, s->object_size);
 }
 
 static inline void kmem_cache_free(struct kmem_cache *s, void *x)
@@ -842,11 +840,11 @@ static inline unsigned long vma_pages(struct vm_area_struct *vma)
 	return (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
 }
 
-static inline void fput(struct file *)
+static inline void fput(struct file *file)
 {
 }
 
-static inline void mpol_put(struct mempolicy *)
+static inline void mpol_put(struct mempolicy *pol)
 {
 }
 
@@ -854,15 +852,15 @@ static inline void lru_add_drain(void)
 {
 }
 
-static inline void tlb_gather_mmu(struct mmu_gather *, struct mm_struct *)
+static inline void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm)
 {
 }
 
-static inline void update_hiwater_rss(struct mm_struct *)
+static inline void update_hiwater_rss(struct mm_struct *mm)
 {
 }
 
-static inline void update_hiwater_vm(struct mm_struct *)
+static inline void update_hiwater_vm(struct mm_struct *mm)
 {
 }
 
@@ -871,36 +869,23 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
 		      unsigned long end_addr, unsigned long tree_end,
 		      bool mm_wr_locked)
 {
-	(void)tlb;
-	(void)mas;
-	(void)vma;
-	(void)start_addr;
-	(void)end_addr;
-	(void)tree_end;
-	(void)mm_wr_locked;
 }
 
 static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
 		   struct vm_area_struct *vma, unsigned long floor,
 		   unsigned long ceiling, bool mm_wr_locked)
 {
-	(void)tlb;
-	(void)mas;
-	(void)vma;
-	(void)floor;
-	(void)ceiling;
-	(void)mm_wr_locked;
 }
 
-static inline void mapping_unmap_writable(struct address_space *)
+static inline void mapping_unmap_writable(struct address_space *mapping)
 {
 }
 
-static inline void flush_dcache_mmap_lock(struct address_space *)
+static inline void flush_dcache_mmap_lock(struct address_space *mapping)
 {
 }
 
-static inline void tlb_finish_mmu(struct mmu_gather *)
+static inline void tlb_finish_mmu(struct mmu_gather *tlb)
 {
 }
 
@@ -909,7 +894,7 @@ static inline struct file *get_file(struct file *f)
 	return f;
 }
 
-static inline int vma_dup_policy(struct vm_area_struct *, struct vm_area_struct *)
+static inline int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst)
 {
 	return 0;
 }
@@ -936,10 +921,6 @@ static inline void vma_adjust_trans_huge(struct vm_area_struct *vma,
 					 unsigned long end,
 					 struct vm_area_struct *next)
 {
-	(void)vma;
-	(void)start;
-	(void)end;
-	(void)next;
 }
 
 static inline void hugetlb_split(struct vm_area_struct *, unsigned long) {}
@@ -959,51 +940,48 @@ static inline void vm_acct_memory(long pages)
 {
 }
 
-static inline void vma_interval_tree_insert(struct vm_area_struct *,
-					    struct rb_root_cached *)
+static inline void vma_interval_tree_insert(struct vm_area_struct *vma,
+					    struct rb_root_cached *rb)
 {
 }
 
-static inline void vma_interval_tree_remove(struct vm_area_struct *,
-					    struct rb_root_cached *)
+static inline void vma_interval_tree_remove(struct vm_area_struct *vma,
+					    struct rb_root_cached *rb)
 {
 }
 
-static inline void flush_dcache_mmap_unlock(struct address_space *)
+static inline void flush_dcache_mmap_unlock(struct address_space *mapping)
 {
 }
 
-static inline void anon_vma_interval_tree_insert(struct anon_vma_chain*,
-						 struct rb_root_cached *)
+static inline void anon_vma_interval_tree_insert(struct anon_vma_chain *avc,
+						 struct rb_root_cached *rb)
 {
 }
 
-static inline void anon_vma_interval_tree_remove(struct anon_vma_chain*,
-						 struct rb_root_cached *)
+static inline void anon_vma_interval_tree_remove(struct anon_vma_chain *avc,
+						 struct rb_root_cached *rb)
 {
 }
 
-static inline void uprobe_mmap(struct vm_area_struct *)
+static inline void uprobe_mmap(struct vm_area_struct *vma)
 {
 }
 
 static inline void uprobe_munmap(struct vm_area_struct *vma,
 				 unsigned long start, unsigned long end)
 {
-	(void)vma;
-	(void)start;
-	(void)end;
 }
 
-static inline void i_mmap_lock_write(struct address_space *)
+static inline void i_mmap_lock_write(struct address_space *mapping)
 {
 }
 
-static inline void anon_vma_lock_write(struct anon_vma *)
+static inline void anon_vma_lock_write(struct anon_vma *anon_vma)
 {
 }
 
-static inline void vma_assert_write_locked(struct vm_area_struct *)
+static inline void vma_assert_write_locked(struct vm_area_struct *vma)
 {
 }
 
@@ -1013,16 +991,16 @@ static inline void unlink_anon_vmas(struct vm_area_struct *vma)
 	vma->anon_vma->was_unlinked = true;
 }
 
-static inline void anon_vma_unlock_write(struct anon_vma *)
+static inline void anon_vma_unlock_write(struct anon_vma *anon_vma)
 {
 }
 
-static inline void i_mmap_unlock_write(struct address_space *)
+static inline void i_mmap_unlock_write(struct address_space *mapping)
 {
 }
 
-static inline void anon_vma_merge(struct vm_area_struct *,
-				  struct vm_area_struct *)
+static inline void anon_vma_merge(struct vm_area_struct *vma,
+				  struct vm_area_struct *next)
 {
 }
 
@@ -1031,27 +1009,22 @@ static inline int userfaultfd_unmap_prep(struct vm_area_struct *vma,
 					 unsigned long end,
 					 struct list_head *unmaps)
 {
-	(void)vma;
-	(void)start;
-	(void)end;
-	(void)unmaps;
-
 	return 0;
 }
 
-static inline void mmap_write_downgrade(struct mm_struct *)
+static inline void mmap_write_downgrade(struct mm_struct *mm)
 {
 }
 
-static inline void mmap_read_unlock(struct mm_struct *)
+static inline void mmap_read_unlock(struct mm_struct *mm)
 {
 }
 
-static inline void mmap_write_unlock(struct mm_struct *)
+static inline void mmap_write_unlock(struct mm_struct *mm)
 {
 }
 
-static inline int mmap_write_lock_killable(struct mm_struct *)
+static inline int mmap_write_lock_killable(struct mm_struct *mm)
 {
 	return 0;
 }
@@ -1060,10 +1033,6 @@ static inline bool can_modify_mm(struct mm_struct *mm,
 				 unsigned long start,
 				 unsigned long end)
 {
-	(void)mm;
-	(void)start;
-	(void)end;
-
 	return true;
 }
 
@@ -1071,16 +1040,13 @@ static inline void arch_unmap(struct mm_struct *mm,
 				 unsigned long start,
 				 unsigned long end)
 {
-	(void)mm;
-	(void)start;
-	(void)end;
 }
 
-static inline void mmap_assert_locked(struct mm_struct *)
+static inline void mmap_assert_locked(struct mm_struct *mm)
 {
 }
 
-static inline bool mpol_equal(struct mempolicy *, struct mempolicy *)
+static inline bool mpol_equal(struct mempolicy *a, struct mempolicy *b)
 {
 	return true;
 }
@@ -1088,63 +1054,62 @@ static inline bool mpol_equal(struct mempolicy *, struct mempolicy *)
 static inline void khugepaged_enter_vma(struct vm_area_struct *vma,
 			  vm_flags_t vm_flags)
 {
-	(void)vma;
-	(void)vm_flags;
 }
 
-static inline bool mapping_can_writeback(struct address_space *)
+static inline bool mapping_can_writeback(struct address_space *mapping)
 {
 	return true;
 }
 
-static inline bool is_vm_hugetlb_page(struct vm_area_struct *)
+static inline bool is_vm_hugetlb_page(struct vm_area_struct *vma)
 {
 	return false;
 }
 
-static inline bool vma_soft_dirty_enabled(struct vm_area_struct *)
+static inline bool vma_soft_dirty_enabled(struct vm_area_struct *vma)
 {
 	return false;
 }
 
-static inline bool userfaultfd_wp(struct vm_area_struct *)
+static inline bool userfaultfd_wp(struct vm_area_struct *vma)
 {
 	return false;
 }
 
-static inline void mmap_assert_write_locked(struct mm_struct *)
+static inline void mmap_assert_write_locked(struct mm_struct *mm)
 {
 }
 
-static inline void mutex_lock(struct mutex *)
+static inline void mutex_lock(struct mutex *lock)
 {
 }
 
-static inline void mutex_unlock(struct mutex *)
+static inline void mutex_unlock(struct mutex *lock)
 {
 }
 
-static inline bool mutex_is_locked(struct mutex *)
+static inline bool mutex_is_locked(struct mutex *lock)
 {
 	return true;
 }
 
-static inline bool signal_pending(void *)
+static inline bool signal_pending(void *p)
 {
 	return false;
 }
 
-static inline bool is_file_hugepages(struct file *)
+static inline bool is_file_hugepages(struct file *file)
 {
 	return false;
 }
 
-static inline int security_vm_enough_memory_mm(struct mm_struct *, long)
+static inline int security_vm_enough_memory_mm(struct mm_struct *mm, long pages)
 {
 	return 0;
 }
 
-static inline bool may_expand_vm(struct mm_struct *, vm_flags_t, unsigned long)
+static inline bool may_expand_vm(struct mm_struct *mm, vm_flags_t flags,
+				 unsigned long npages)
 {
 	return true;
 }
@@ -1169,7 +1134,7 @@ static inline void vm_flags_clear(struct vm_area_struct *vma,
 	vma->__vm_flags &= ~flags;
 }
 
-static inline int shmem_zero_setup(struct vm_area_struct *)
+static inline int shmem_zero_setup(struct vm_area_struct *vma)
 {
 	return 0;
 }
@@ -1179,20 +1144,20 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma)
 	vma->vm_ops = NULL;
 }
 
-static inline void ksm_add_vma(struct vm_area_struct *)
+static inline void ksm_add_vma(struct vm_area_struct *vma)
 {
 }
 
-static inline void perf_event_mmap(struct vm_area_struct *)
+static inline void perf_event_mmap(struct vm_area_struct *vma)
 {
 }
 
-static inline bool vma_is_dax(struct vm_area_struct *)
+static inline bool vma_is_dax(struct vm_area_struct *vma)
 {
 	return false;
 }
 
-static inline struct vm_area_struct *get_gate_vma(struct mm_struct *)
+static inline struct vm_area_struct *get_gate_vma(struct mm_struct *mm)
 {
 	return NULL;
 }
@@ -1217,16 +1182,16 @@ static inline void vma_set_page_prot(struct vm_area_struct *vma)
 	WRITE_ONCE(vma->vm_page_prot, vm_page_prot);
 }
 
-static inline bool arch_validate_flags(vm_flags_t)
+static inline bool arch_validate_flags(vm_flags_t flags)
 {
 	return true;
 }
 
-static inline void vma_close(struct vm_area_struct *)
+static inline void vma_close(struct vm_area_struct *vma)
 {
 }
 
-static inline int mmap_file(struct file *, struct vm_area_struct *)
+static inline int mmap_file(struct file *file, struct vm_area_struct *vma)
 {
 	return 0;
 }
@@ -1388,8 +1353,6 @@ static inline int mapping_map_writable(struct address_space *mapping)
 
 static inline unsigned long move_page_tables(struct pagetable_move_control *pmc)
 {
-	(void)pmc;
-
 	return 0;
 }
 
@@ -1397,51 +1360,36 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
 			unsigned long addr, unsigned long end,
 			unsigned long floor, unsigned long ceiling)
 {
-	(void)tlb;
-	(void)addr;
-	(void)end;
-	(void)floor;
-	(void)ceiling;
 }
 
 static inline int ksm_execve(struct mm_struct *mm)
 {
-	(void)mm;
-
 	return 0;
 }
 
 static inline void ksm_exit(struct mm_struct *mm)
 {
-	(void)mm;
 }
 
 static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)
 {
-	(void)vma;
-	(void)reset_refcnt;
 }
 
 static inline void vma_numab_state_init(struct vm_area_struct *vma)
 {
-	(void)vma;
 }
 
 static inline void vma_numab_state_free(struct vm_area_struct *vma)
 {
-	(void)vma;
 }
 
 static inline void dup_anon_vma_name(struct vm_area_struct *orig_vma,
 				     struct vm_area_struct *new_vma)
 {
-	(void)orig_vma;
-	(void)new_vma;
 }
 
 static inline void free_anon_vma_name(struct vm_area_struct *vma)
 {
-	(void)vma;
 }
 
 /* Declared in vma.h. */
@@ -1495,7 +1443,6 @@ static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc)
 
 static inline void fixup_hugetlb_reservations(struct vm_area_struct *vma)
 {
-	(void)vma;
 }
 
 static inline void vma_set_file(struct vm_area_struct *vma, struct file *file)
@@ -1506,13 +1453,13 @@ static inline void vma_set_file(struct vm_area_struct *vma, struct file *file)
 	fput(file);
 }
 
-static inline bool shmem_file(struct file *)
+static inline bool shmem_file(struct file *file)
 {
 	return false;
 }
 
-static inline vm_flags_t ksm_vma_flags(const struct mm_struct *, const struct file *,
-			 vm_flags_t vm_flags)
+static inline vm_flags_t ksm_vma_flags(const struct mm_struct *mm,
+		const struct file *file, vm_flags_t vm_flags)
 {
 	return vm_flags;
 }

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:20 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:20 +0200
Subject: [PATCH v8 18/23] maple_tree: Use kfree_rcu in ma_free_rcu
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-18-ca3099d8352c@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
possible to call it on non-kmalloc objects, but this restriction was
lifted ever since SLOB was dropped from the kernel, and since commit
6c6c47b063b5 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").

Thus, replace call_rcu + mt_free_rcu with kfree_rcu.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 +++----------
 1 file changed, 3 insertions(+), 10 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index d034f170ac897341b40cfd050b6aee86b6d2cf60..c706e2e48f884fd156e25be2b17eb5e154774db7 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -187,13 +187,6 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
-static void mt_free_rcu(struct rcu_head *head)
-{
-	struct maple_node *node = container_of(head, struct maple_node, rcu);
-
-	kmem_cache_free(maple_node_cache, node);
-}
-
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -204,7 +197,7 @@ static void mt_free_rcu(struct rcu_head *head)
 static void ma_free_rcu(struct maple_node *node)
 {
 	WARN_ON(node->parent != ma_parent_ptr(node));
-	call_rcu(&node->rcu, mt_free_rcu);
+	kfree_rcu(node, rcu);
 }
 
 static void mt_set_height(struct maple_tree *mt, unsigned char height)
@@ -5099,7 +5092,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_rcu(&node->rcu);
+	mt_free_one(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5183,7 +5176,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_rcu(&node->rcu);
+		mt_free_one(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:06 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:06 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>

Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
For caches with sheaves, on each cpu maintain a rcu_free sheaf in
addition to main and spare sheaves.

kfree_rcu() operations will try to put objects on this sheaf. Once full,
the sheaf is detached and submitted to call_rcu() with a handler that
will try to put it in the barn, or flush to slab pages using bulk free,
when the barn is full. Then a new empty sheaf must be obtained to put
more objects there.

It's possible that no free sheaves are available to use for a new
rcu_free sheaf, and the allocation in kfree_rcu() context can only use
GFP_NOWAIT and thus may fail. In that case, fall back to the existing
kfree_rcu() implementation.

Expected advantages:
- batching the kfree_rcu() operations, that could eventually replace the
  existing batching
- sheaves can be reused for allocations via barn instead of being
  flushed to slabs, which is more efficient
  - this includes cases where only some cpus are allowed to process rcu
    callbacks (Android)

Possible disadvantage:
- objects might be waiting for more than their grace period (it is
  determined by the last object freed into the sheaf), increasing memory
  usage - but the existing batching does that too.

Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
implementation favors smaller memory footprint over performance.

Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
contexts where kfree_rcu() is called might not be compatible with taking
a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
spinlock - the current kfree_rcu() implementation avoids doing that.

Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
that have them. This is not a cheap operation, but the barrier usage is
rare - currently kmem_cache_destroy() or on module unload.

Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
count how many kfree_rcu() used the rcu_free sheaf successfully and how
many had to fall back to the existing implementation.

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab.h        |   3 +
 mm/slab_common.c |  26 ++++++
 mm/slub.c        | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 293 insertions(+), 2 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index 206987ce44a4d053ebe3b5e50784d2dd23822cd1..e82e51c44bd00042d433ac8b46c2b4bbbdded9b1 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -435,6 +435,9 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
 }
 
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+void flush_all_rcu_sheaves(void);
+
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
 			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
diff --git a/mm/slab_common.c b/mm/slab_common.c
index e2b197e47866c30acdbd1fee4159f262a751c5a7..005a4319c06a01d2b616a75396fcc43766a62ddb 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
 		kvfree_rcu_list(head);
 }
 
+static bool kfree_rcu_sheaf(void *obj)
+{
+	struct kmem_cache *s;
+	struct folio *folio;
+	struct slab *slab;
+
+	if (is_vmalloc_addr(obj))
+		return false;
+
+	folio = virt_to_folio(obj);
+	if (unlikely(!folio_test_slab(folio)))
+		return false;
+
+	slab = folio_slab(folio);
+	s = slab->slab_cache;
+	if (s->cpu_sheaves)
+		return __kfree_rcu_sheaf(s, obj);
+
+	return false;
+}
+
 static bool
 need_offload_krc(struct kfree_rcu_cpu *krcp)
 {
@@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
+		return;
+
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
@@ -2026,6 +2050,8 @@ void kvfree_rcu_barrier(void)
 	bool queued;
 	int i, cpu;
 
+	flush_all_rcu_sheaves();
+
 	/*
 	 * Firstly we detach objects and queue them over an RCU-batch
 	 * for all CPUs. Finally queued works are flushed for each CPU.
diff --git a/mm/slub.c b/mm/slub.c
index cba188b7e04ddf86debf9bc27a2f725db1b2056e..19cd8444ae5d210c77ae767912ca1ff3fc69c2a8 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -367,6 +367,8 @@ enum stat_item {
 	ALLOC_FASTPATH,		/* Allocation from cpu slab */
 	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
 	FREE_PCS,		/* Free to percpu sheaf */
+	FREE_RCU_SHEAF,		/* Free to rcu_free sheaf */
+	FREE_RCU_SHEAF_FAIL,	/* Failed to free to a rcu_free sheaf */
 	FREE_FASTPATH,		/* Free to cpu slab */
 	FREE_SLOWPATH,		/* Freeing not to cpu slab */
 	FREE_FROZEN,		/* Freeing to frozen slab */
@@ -461,6 +463,7 @@ struct slab_sheaf {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
 	};
+	struct kmem_cache *cache;
 	unsigned int size;
 	void *objects[];
 };
@@ -469,6 +472,7 @@ struct slub_percpu_sheaves {
 	local_trylock_t lock;
 	struct slab_sheaf *main; /* never NULL when unlocked */
 	struct slab_sheaf *spare; /* empty or full, may be NULL */
+	struct slab_sheaf *rcu_free; /* for batching kfree_rcu() */
 };
 
 /*
@@ -2531,6 +2535,8 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
 	if (unlikely(!sheaf))
 		return NULL;
 
+	sheaf->cache = s;
+
 	stat(s, SHEAF_ALLOC);
 
 	return sheaf;
@@ -2655,6 +2661,43 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	sheaf->size = 0;
 }
 
+static void __rcu_free_sheaf_prepare(struct kmem_cache *s,
+				     struct slab_sheaf *sheaf)
+{
+	bool init = slab_want_init_on_free(s);
+	void **p = &sheaf->objects[0];
+	unsigned int i = 0;
+
+	while (i < sheaf->size) {
+		struct slab *slab = virt_to_slab(p[i]);
+
+		memcg_slab_free_hook(s, slab, p + i, 1);
+		alloc_tagging_slab_free_hook(s, slab, p + i, 1);
+
+		if (unlikely(!slab_free_hook(s, p[i], init, true))) {
+			p[i] = p[--sheaf->size];
+			continue;
+		}
+
+		i++;
+	}
+}
+
+static void rcu_free_sheaf_nobarn(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+	s = sheaf->cache;
+
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	sheaf_flush_unused(s, sheaf);
+
+	free_empty_sheaf(s, sheaf);
+}
+
 /*
  * Caller needs to make sure migration is disabled in order to fully flush
  * single cpu's sheaves
@@ -2667,7 +2710,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 static void pcs_flush_all(struct kmem_cache *s)
 {
 	struct slub_percpu_sheaves *pcs;
-	struct slab_sheaf *spare;
+	struct slab_sheaf *spare, *rcu_free;
 
 	local_lock(&s->cpu_sheaves->lock);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
@@ -2675,6 +2718,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 	spare = pcs->spare;
 	pcs->spare = NULL;
 
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
 	local_unlock(&s->cpu_sheaves->lock);
 
 	if (spare) {
@@ -2682,6 +2728,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 		free_empty_sheaf(s, spare);
 	}
 
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+
 	sheaf_flush_main(s);
 }
 
@@ -2698,6 +2747,11 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
 		free_empty_sheaf(s, pcs->spare);
 		pcs->spare = NULL;
 	}
+
+	if (pcs->rcu_free) {
+		call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+		pcs->rcu_free = NULL;
+	}
 }
 
 static void pcs_destroy(struct kmem_cache *s)
@@ -2723,6 +2777,7 @@ static void pcs_destroy(struct kmem_cache *s)
 		 */
 
 		WARN_ON(pcs->spare);
+		WARN_ON(pcs->rcu_free);
 
 		if (!WARN_ON(pcs->main->size)) {
 			free_empty_sheaf(s, pcs->main);
@@ -3780,7 +3835,7 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s)
 
 	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
-	return (pcs->spare || pcs->main->size);
+	return (pcs->spare || pcs->rcu_free || pcs->main->size);
 }
 
 /*
@@ -3840,6 +3895,80 @@ static void flush_all(struct kmem_cache *s)
 	cpus_read_unlock();
 }
 
+static void flush_rcu_sheaf(struct work_struct *w)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_free;
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+
+	sfw = container_of(w, struct slub_flush_work, work);
+	s = sfw->s;
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+}
+
+
+/* needed for kvfree_rcu_barrier() */
+void flush_all_rcu_sheaves()
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+	bool flushed = false;
+	unsigned int cpu;
+
+	cpus_read_lock();
+	mutex_lock(&slab_mutex);
+
+	list_for_each_entry(s, &slab_caches, list) {
+		if (!s->cpu_sheaves)
+			continue;
+
+		mutex_lock(&flush_lock);
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
+
+			if (!pcs->rcu_free || !pcs->rcu_free->size) {
+				sfw->skip = true;
+				continue;
+			}
+
+			INIT_WORK(&sfw->work, flush_rcu_sheaf);
+			sfw->skip = false;
+			sfw->s = s;
+			queue_work_on(cpu, flushwq, &sfw->work);
+			flushed = true;
+		}
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+			if (sfw->skip)
+				continue;
+			flush_work(&sfw->work);
+		}
+
+		mutex_unlock(&flush_lock);
+	}
+
+	mutex_unlock(&slab_mutex);
+	cpus_read_unlock();
+
+	if (flushed)
+		rcu_barrier();
+}
+
 /*
  * Use the cpu notifier to insure that the cpu slabs are flushed when
  * necessary.
@@ -5413,6 +5542,130 @@ bool free_to_pcs(struct kmem_cache *s, void *object)
 	return true;
 }
 
+static void rcu_free_sheaf(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct node_barn *barn;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+
+	s = sheaf->cache;
+
+	/*
+	 * This may remove some objects due to slab_free_hook() returning false,
+	 * so that the sheaf might no longer be completely full. But it's easier
+	 * to handle it as full (unless it became completely empty), as the code
+	 * handles it fine. The only downside is that sheaf will serve fewer
+	 * allocations when reused. It only happens due to debugging, which is a
+	 * performance hit anyway.
+	 */
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	barn = get_node(s, numa_mem_id())->barn;
+
+	/* due to slab_free_hook() */
+	if (unlikely(sheaf->size == 0))
+		goto empty;
+
+	/*
+	 * Checking nr_full/nr_empty outside lock avoids contention in case the
+	 * barn is at the respective limit. Due to the race we might go over the
+	 * limit but that should be rare and harmless.
+	 */
+
+	if (data_race(barn->nr_full) < MAX_FULL_SHEAVES) {
+		stat(s, BARN_PUT);
+		barn_put_full_sheaf(barn, sheaf);
+		return;
+	}
+
+	stat(s, BARN_PUT_FAIL);
+	sheaf_flush_unused(s, sheaf);
+
+empty:
+	if (data_race(barn->nr_empty) < MAX_EMPTY_SHEAVES) {
+		barn_put_empty_sheaf(barn, sheaf);
+		return;
+	}
+
+	free_empty_sheaf(s, sheaf);
+}
+
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_sheaf;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto fail;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(!pcs->rcu_free)) {
+
+		struct slab_sheaf *empty;
+		struct node_barn *barn;
+
+		if (pcs->spare && pcs->spare->size == 0) {
+			pcs->rcu_free = pcs->spare;
+			pcs->spare = NULL;
+			goto do_free;
+		}
+
+		barn = get_barn(s);
+
+		empty = barn_get_empty_sheaf(barn);
+
+		if (empty) {
+			pcs->rcu_free = empty;
+			goto do_free;
+		}
+
+		local_unlock(&s->cpu_sheaves->lock);
+
+		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+
+		if (!empty)
+			goto fail;
+
+		if (!local_trylock(&s->cpu_sheaves->lock)) {
+			barn_put_empty_sheaf(barn, empty);
+			goto fail;
+		}
+
+		pcs = this_cpu_ptr(s->cpu_sheaves);
+
+		if (unlikely(pcs->rcu_free))
+			barn_put_empty_sheaf(barn, empty);
+		else
+			pcs->rcu_free = empty;
+	}
+
+do_free:
+
+	rcu_sheaf = pcs->rcu_free;
+
+	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
+
+	if (likely(rcu_sheaf->size < s->sheaf_capacity))
+		rcu_sheaf = NULL;
+	else
+		pcs->rcu_free = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (rcu_sheaf)
+		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+
+	stat(s, FREE_RCU_SHEAF);
+	return true;
+
+fail:
+	stat(s, FREE_RCU_SHEAF_FAIL);
+	return false;
+}
+
 /*
  * Bulk free objects to the percpu sheaves.
  * Unlike free_to_pcs() this includes the calls to all necessary hooks
@@ -6909,6 +7162,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
 	struct kmem_cache_node *n;
 
 	flush_all_cpus_locked(s);
+
+	/* we might have rcu sheaves in flight */
+	if (s->cpu_sheaves)
+		rcu_barrier();
+
 	/* Attempt to free all objects */
 	for_each_kmem_cache_node(s, node, n) {
 		if (n->barn)
@@ -8284,6 +8542,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
 STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
 STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
 STAT_ATTR(FREE_PCS, free_cpu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail);
 STAT_ATTR(FREE_FASTPATH, free_fastpath);
 STAT_ATTR(FREE_SLOWPATH, free_slowpath);
 STAT_ATTR(FREE_FROZEN, free_frozen);
@@ -8382,6 +8642,8 @@ static struct attribute *slab_attrs[] = {
 	&alloc_fastpath_attr.attr,
 	&alloc_slowpath_attr.attr,
 	&free_cpu_sheaf_attr.attr,
+	&free_rcu_sheaf_attr.attr,
+	&free_rcu_sheaf_fail_attr.attr,
 	&free_fastpath_attr.attr,
 	&free_slowpath_attr.attr,
 	&free_frozen_attr.attr,

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:17 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:17 +0200
Subject: [PATCH v8 15/23] maple_tree: use percpu sheaves for
 maple_node_cache
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-15-ca3099d8352c@suse.cz>

Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
improve its performance. Note this will not immediately take advantage
of sheaf batching of kfree_rcu() operations due to the maple tree using
call_rcu with custom callbacks. The followup changes to maple tree will
change that and also make use of the prefilled sheaves functionality.

Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 4f0e30b57b0cef9e5cf791f3f64f5898752db402..d034f170ac897341b40cfd050b6aee86b6d2cf60 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -6040,9 +6040,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 
 void __init maple_tree_init(void)
 {
+	struct kmem_cache_args args = {
+		.align  = sizeof(struct maple_node),
+		.sheaf_capacity = 32,
+	};
+
 	maple_node_cache = kmem_cache_create("maple_node",
-			sizeof(struct maple_node), sizeof(struct maple_node),
-			SLAB_PANIC, NULL);
+			sizeof(struct maple_node), &args,
+			SLAB_PANIC);
 }
 
 /**

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:11 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:11 +0200
Subject: [PATCH v8 09/23] maple_tree: remove redundant __GFP_NOWARN
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-9-ca3099d8352c@suse.cz>

From: Qianfeng Rong <rongqianfeng at vivo.com>

Commit 16f5dfbc851b ("gfp: include __GFP_NOWARN in GFP_NOWAIT") made
GFP_NOWAIT implicitly include __GFP_NOWARN.

Therefore, explicit __GFP_NOWARN combined with GFP_NOWAIT (e.g.,
`GFP_NOWAIT | __GFP_NOWARN`) is now redundant.  Let's clean up these
redundant flags across subsystems.

No functional changes.

Link: https://lkml.kernel.org/r/20250804125657.482109-1-rongqianfeng at vivo.com
Signed-off-by: Qianfeng Rong <rongqianfeng at vivo.com>
Reviewed-by: Wei Yang <richard.weiyang at gmail.com>
Reviewed-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Cc: Matthew Wilcox (Oracle) <willy at infradead.org>
Signed-off-by: Andrew Morton <akpm at linux-foundation.org>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a962ca374467d0533185f2db3d35ff..38fb68c082915211c80f473d313159599fe97e2c 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1344,11 +1344,11 @@ static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
  * @mas: The maple state
  * @count: The number of nodes needed
  *
- * Note: Uses GFP_NOWAIT | __GFP_NOWARN for gfp flags.
+ * Note: Uses GFP_NOWAIT for gfp flags.
  */
 static void mas_node_count(struct ma_state *mas, int count)
 {
-	return mas_node_count_gfp(mas, count, GFP_NOWAIT | __GFP_NOWARN);
+	return mas_node_count_gfp(mas, count, GFP_NOWAIT);
 }
 
 /*

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:23 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:23 +0200
Subject: [PATCH v8 21/23] maple_tree: Prefilled sheaf conversion and
 testing
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-21-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Use prefilled sheaves instead of bulk allocations. This should speed up
the allocations and the return path of unused allocations.

Remove the push and pop of nodes from the maple state as this is now
handled by the slab layer with sheaves.

Testing has been removed as necessary since the features of the tree
have been reduced.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |   6 +-
 lib/maple_tree.c                 | 326 ++++++---------------------
 tools/testing/radix-tree/maple.c | 461 ++-------------------------------------
 tools/testing/shared/linux.c     |   5 +-
 4 files changed, 88 insertions(+), 710 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -442,7 +442,8 @@ struct ma_state {
 	struct maple_enode *node;	/* The node containing this entry */
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
-	struct maple_alloc *alloc;	/* Allocated nodes for this operation */
+	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
 	unsigned char offset;
@@ -490,7 +491,8 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.alloc = NULL,						\
+		.node_request= 0,					\
+		.sheaf = NULL,						\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index 0439aaacf6cb1f39d0d23af2e2a5af1d27ab32be..a3fcb20227e506ed209554cc8c041a53f7ef4903 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -182,6 +182,22 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
 }
 
+static void mt_return_sheaf(struct slab_sheaf *sheaf)
+{
+	kmem_cache_return_sheaf(maple_node_cache, GFP_NOWAIT, sheaf);
+}
+
+static struct slab_sheaf *mt_get_sheaf(gfp_t gfp, int count)
+{
+	return kmem_cache_prefill_sheaf(maple_node_cache, gfp, count);
+}
+
+static int mt_refill_sheaf(gfp_t gfp, struct slab_sheaf **sheaf,
+		unsigned int size)
+{
+	return kmem_cache_refill_sheaf(maple_node_cache, gfp, sheaf, size);
+}
+
 /*
  * ma_free_rcu() - Use rcu callback to free a maple node
  * @node: The node to free
@@ -574,67 +590,6 @@ static __always_inline bool mte_dead_node(const struct maple_enode *enode)
 	return ma_dead_node(node);
 }
 
-/*
- * mas_allocated() - Get the number of nodes allocated in a maple state.
- * @mas: The maple state
- *
- * The ma_state alloc member is overloaded to hold a pointer to the first
- * allocated node or to the number of requested nodes to allocate.  If bit 0 is
- * set, then the alloc contains the number of requested nodes.  If there is an
- * allocated node, then the total allocated nodes is in that node.
- *
- * Return: The total number of nodes allocated
- */
-static inline unsigned long mas_allocated(const struct ma_state *mas)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1))
-		return 0;
-
-	return mas->alloc->total;
-}
-
-/*
- * mas_set_alloc_req() - Set the requested number of allocations.
- * @mas: the maple state
- * @count: the number of allocations.
- *
- * The requested number of allocations is either in the first allocated node,
- * located in @mas->alloc->request_count, or directly in @mas->alloc if there is
- * no allocated node.  Set the request either in the node or do the necessary
- * encoding to store in @mas->alloc directly.
- */
-static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
-{
-	if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
-		if (!count)
-			mas->alloc = NULL;
-		else
-			mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
-		return;
-	}
-
-	mas->alloc->request_count = count;
-}
-
-/*
- * mas_alloc_req() - get the requested number of allocations.
- * @mas: The maple state
- *
- * The alloc count is either stored directly in @mas, or in
- * @mas->alloc->request_count if there is at least one node allocated.  Decode
- * the request count if it's stored directly in @mas->alloc.
- *
- * Return: The allocation request count.
- */
-static inline unsigned int mas_alloc_req(const struct ma_state *mas)
-{
-	if ((unsigned long)mas->alloc & 0x1)
-		return (unsigned long)(mas->alloc) >> 1;
-	else if (mas->alloc)
-		return mas->alloc->request_count;
-	return 0;
-}
-
 /*
  * ma_pivots() - Get a pointer to the maple node pivots.
  * @node: the maple node
@@ -1120,77 +1075,15 @@ static int mas_ascend(struct ma_state *mas)
  */
 static inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
-	struct maple_alloc *ret, *node = mas->alloc;
-	unsigned long total = mas_allocated(mas);
-	unsigned int req = mas_alloc_req(mas);
+	struct maple_node *ret;
 
-	/* nothing or a request pending. */
-	if (WARN_ON(!total))
+	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
-	if (total == 1) {
-		/* single allocation in this ma_state */
-		mas->alloc = NULL;
-		ret = node;
-		goto single_node;
-	}
-
-	if (node->node_count == 1) {
-		/* Single allocation in this node. */
-		mas->alloc = node->slot[0];
-		mas->alloc->total = node->total - 1;
-		ret = node;
-		goto new_head;
-	}
-	node->total--;
-	ret = node->slot[--node->node_count];
-	node->slot[node->node_count] = NULL;
-
-single_node:
-new_head:
-	if (req) {
-		req++;
-		mas_set_alloc_req(mas, req);
-	}
-
+	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
 	memset(ret, 0, sizeof(*ret));
-	return (struct maple_node *)ret;
-}
-
-/*
- * mas_push_node() - Push a node back on the maple state allocation.
- * @mas: The maple state
- * @used: The used maple node
- *
- * Stores the maple node back into @mas->alloc for reuse.  Updates allocated and
- * requested node count as necessary.
- */
-static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
-{
-	struct maple_alloc *reuse = (struct maple_alloc *)used;
-	struct maple_alloc *head = mas->alloc;
-	unsigned long count;
-	unsigned int requested = mas_alloc_req(mas);
 
-	count = mas_allocated(mas);
-
-	reuse->request_count = 0;
-	reuse->node_count = 0;
-	if (count) {
-		if (head->node_count < MAPLE_ALLOC_SLOTS) {
-			head->slot[head->node_count++] = reuse;
-			head->total++;
-			goto done;
-		}
-		reuse->slot[0] = head;
-		reuse->node_count = 1;
-	}
-
-	reuse->total = count + 1;
-	mas->alloc = reuse;
-done:
-	if (requested > 1)
-		mas_set_alloc_req(mas, requested - 1);
+	return ret;
 }
 
 /*
@@ -1200,75 +1093,32 @@ static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	struct maple_alloc *node;
-	unsigned long allocated = mas_allocated(mas);
-	unsigned int requested = mas_alloc_req(mas);
-	unsigned int count;
-	void **slots = NULL;
-	unsigned int max_req = 0;
-
-	if (!requested)
-		return;
+	if (unlikely(mas->sheaf)) {
+		unsigned long refill = mas->node_request;
 
-	mas_set_alloc_req(mas, 0);
-	if (mas->mas_flags & MA_STATE_PREALLOC) {
-		if (allocated)
+		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
+			mas->node_request = 0;
 			return;
-		WARN_ON(!allocated);
-	}
-
-	if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) {
-		node = (struct maple_alloc *)mt_alloc_one(gfp);
-		if (!node)
-			goto nomem_one;
-
-		if (allocated) {
-			node->slot[0] = mas->alloc;
-			node->node_count = 1;
-		} else {
-			node->node_count = 0;
 		}
 
-		mas->alloc = node;
-		node->total = ++allocated;
-		node->request_count = 0;
-		requested--;
-	}
+		if (mt_refill_sheaf(gfp, &mas->sheaf, refill))
+			goto error;
 
-	node = mas->alloc;
-	while (requested) {
-		max_req = MAPLE_ALLOC_SLOTS - node->node_count;
-		slots = (void **)&node->slot[node->node_count];
-		max_req = min(requested, max_req);
-		count = mt_alloc_bulk(gfp, max_req, slots);
-		if (!count)
-			goto nomem_bulk;
-
-		if (node->node_count == 0) {
-			node->slot[0]->node_count = 0;
-			node->slot[0]->request_count = 0;
-		}
+		mas->node_request = 0;
+		return;
+	}
 
-		node->node_count += count;
-		allocated += count;
-		/* find a non-full node*/
-		do {
-			node = node->slot[0];
-		} while (unlikely(node->node_count == MAPLE_ALLOC_SLOTS));
-		requested -= count;
+	mas->sheaf = mt_get_sheaf(gfp, mas->node_request);
+	if (likely(mas->sheaf)) {
+		mas->node_request = 0;
+		return;
 	}
-	mas->alloc->total = allocated;
-	return;
 
-nomem_bulk:
-	/* Clean up potential freed allocations on bulk failure */
-	memset(slots, 0, max_req * sizeof(unsigned long));
-	mas->alloc->total = allocated;
-nomem_one:
-	mas_set_alloc_req(mas, requested);
+error:
 	mas_set_err(mas, -ENOMEM);
 }
 
+
 /*
  * mas_free() - Free an encoded maple node
  * @mas: The maple state
@@ -1279,42 +1129,7 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
  */
 static inline void mas_free(struct ma_state *mas, struct maple_enode *used)
 {
-	struct maple_node *tmp = mte_to_node(used);
-
-	if (mt_in_rcu(mas->tree))
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
-}
-
-/*
- * mas_node_count_gfp() - Check if enough nodes are allocated and request more
- * if there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- * @gfp: the gfp flags
- */
-static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
-{
-	unsigned long allocated = mas_allocated(mas);
-
-	if (allocated < count) {
-		mas_set_alloc_req(mas, count - allocated);
-		mas_alloc_nodes(mas, gfp);
-	}
-}
-
-/*
- * mas_node_count() - Check if enough nodes are allocated and request more if
- * there is not enough nodes.
- * @mas: The maple state
- * @count: The number of nodes needed
- *
- * Note: Uses GFP_NOWAIT for gfp flags.
- */
-static void mas_node_count(struct ma_state *mas, int count)
-{
-	return mas_node_count_gfp(mas, count, GFP_NOWAIT);
+	ma_free_rcu(mte_to_node(used));
 }
 
 /*
@@ -2451,10 +2266,7 @@ static inline void mas_topiary_node(struct ma_state *mas,
 	enode = tmp_mas->node;
 	tmp = mte_to_node(enode);
 	mte_set_node_dead(enode);
-	if (in_rcu)
-		ma_free_rcu(tmp);
-	else
-		mas_push_node(mas, tmp);
+	ma_free_rcu(tmp);
 }
 
 /*
@@ -3980,7 +3792,7 @@ static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas)
  *
  * Return: Number of nodes required for preallocation.
  */
-static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
+static inline void mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 {
 	struct ma_state *mas = wr_mas->mas;
 	unsigned char height = mas_mt_height(mas);
@@ -4026,7 +3838,7 @@ static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
 		WARN_ON_ONCE(1);
 	}
 
-	return ret;
+	mas->node_request = ret;
 }
 
 /*
@@ -4087,15 +3899,15 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
  */
 static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry)
 {
-	int request;
+	struct ma_state *mas = wr_mas->mas;
 
 	mas_wr_prealloc_setup(wr_mas);
-	wr_mas->mas->store_type = mas_wr_store_type(wr_mas);
-	request = mas_prealloc_calc(wr_mas, entry);
-	if (!request)
+	mas->store_type = mas_wr_store_type(wr_mas);
+	mas_prealloc_calc(wr_mas, entry);
+	if (!mas->node_request)
 		return;
 
-	mas_node_count(wr_mas->mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT);
 }
 
 /**
@@ -5208,7 +5020,6 @@ static inline void mte_destroy_walk(struct maple_enode *enode,
  */
 void *mas_store(struct ma_state *mas, void *entry)
 {
-	int request;
 	MA_WR_STATE(wr_mas, mas, entry);
 
 	trace_ma_write(__func__, mas, 0, entry);
@@ -5238,11 +5049,11 @@ void *mas_store(struct ma_state *mas, void *entry)
 		return wr_mas.content;
 	}
 
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto store;
 
-	mas_node_count(mas, request);
+	mas_alloc_nodes(mas, GFP_NOWAIT);
 	if (mas_is_err(mas))
 		return NULL;
 
@@ -5330,20 +5141,19 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc);
 int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 {
 	MA_WR_STATE(wr_mas, mas, entry);
-	int ret = 0;
-	int request;
 
 	mas_wr_prealloc_setup(&wr_mas);
 	mas->store_type = mas_wr_store_type(&wr_mas);
-	request = mas_prealloc_calc(&wr_mas, entry);
-	if (!request)
+	mas_prealloc_calc(&wr_mas, entry);
+	if (!mas->node_request)
 		goto set_flag;
 
 	mas->mas_flags &= ~MA_STATE_PREALLOC;
-	mas_node_count_gfp(mas, request, gfp);
+	mas_alloc_nodes(mas, gfp);
 	if (mas_is_err(mas)) {
-		mas_set_alloc_req(mas, 0);
-		ret = xa_err(mas->node);
+		int ret = xa_err(mas->node);
+
+		mas->node_request = 0;
 		mas_destroy(mas);
 		mas_reset(mas);
 		return ret;
@@ -5351,7 +5161,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
 
 set_flag:
 	mas->mas_flags |= MA_STATE_PREALLOC;
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(mas_preallocate);
 
@@ -5365,26 +5175,13 @@ EXPORT_SYMBOL_GPL(mas_preallocate);
  */
 void mas_destroy(struct ma_state *mas)
 {
-	struct maple_alloc *node;
-	unsigned long total;
-
 	mas->mas_flags &= ~MA_STATE_PREALLOC;
 
-	total = mas_allocated(mas);
-	while (total) {
-		node = mas->alloc;
-		mas->alloc = node->slot[0];
-		if (node->node_count > 1) {
-			size_t count = node->node_count - 1;
-
-			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
-			total -= count;
-		}
-		kfree(ma_mnode_ptr(node));
-		total--;
-	}
+	mas->node_request = 0;
+	if (mas->sheaf)
+		mt_return_sheaf(mas->sheaf);
 
-	mas->alloc = NULL;
+	mas->sheaf = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6019,7 +5816,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas_allocated(mas))
+	if (!mas->sheaf)
 		return false;
 
 	mas->status = ma_start;
@@ -7414,8 +7211,9 @@ void mas_dump(const struct ma_state *mas)
 
 	pr_err("[%u/%u] index=%lx last=%lx\n", mas->offset, mas->end,
 	       mas->index, mas->last);
-	pr_err("     min=%lx max=%lx alloc=" PTR_FMT ", depth=%u, flags=%x\n",
-	       mas->min, mas->max, mas->alloc, mas->depth, mas->mas_flags);
+	pr_err("     min=%lx max=%lx sheaf=" PTR_FMT ", request %lu depth=%u, flags=%x\n",
+	       mas->min, mas->max, mas->sheaf, mas->node_request, mas->depth,
+	       mas->mas_flags);
 	if (mas->index > mas->last)
 		pr_err("Check index & last\n");
 }
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 4a35e1e7c64b7ce347cbd1693beeaacb0c4c330e..72a8fe8e832a4150c6567b711768eba6a3fa6768 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -57,430 +57,6 @@ struct rcu_reader_struct {
 	struct rcu_test_struct2 *test;
 };
 
-static int get_alloc_node_count(struct ma_state *mas)
-{
-	int count = 1;
-	struct maple_alloc *node = mas->alloc;
-
-	if (!node || ((unsigned long)node & 0x1))
-		return 0;
-	while (node->node_count) {
-		count += node->node_count;
-		node = node->slot[0];
-	}
-	return count;
-}
-
-static void check_mas_alloc_node_count(struct ma_state *mas)
-{
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 1, GFP_KERNEL);
-	mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 3, GFP_KERNEL);
-	MT_BUG_ON(mas->tree, get_alloc_node_count(mas) != mas->alloc->total);
-	mas_destroy(mas);
-}
-
-/*
- * check_new_node() - Check the creation of new nodes and error path
- * verification.
- */
-static noinline void __init check_new_node(struct maple_tree *mt)
-{
-
-	struct maple_node *mn, *mn2, *mn3;
-	struct maple_alloc *smn;
-	struct maple_node *nodes[100];
-	int i, j, total;
-
-	MA_STATE(mas, mt, 0, 0);
-
-	check_mas_alloc_node_count(&mas);
-
-	/* Try allocating 3 nodes */
-	mtree_lock(mt);
-	mt_set_non_kernel(0);
-	/* request 3 nodes to be allocated. */
-	mas_node_count(&mas, 3);
-	/* Allocation request of 3. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 3);
-	/* Allocate failed. */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	mas_push_node(&mas, mn);
-	mas_reset(&mas);
-	mas_destroy(&mas);
-	mtree_unlock(mt);
-
-
-	/* Try allocating 1 node, then 2 more */
-	mtree_lock(mt);
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	/* Check Allocation request of 1. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	/* Eat the requested node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	mas.status = ma_start;
-	mas_destroy(&mas);
-	/* Allocate 3 nodes, will fail. */
-	mas_node_count(&mas, 3);
-	/* Drop the lock and allocate 3 nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	/* Ensure 3 are allocated. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Allocation request of 0. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 0);
-
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	/* Ensure we counted 3. */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	/* Free. */
-	mas_reset(&mas);
-	mas_destroy(&mas);
-
-	/* Set allocation request to 1. */
-	mas_set_alloc_req(&mas, 1);
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	mas_set_err(&mas, -ENOMEM);
-	/* Validate allocation request. */
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	/* Check the node is only one node. */
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, mn->slot[0] != NULL);
-	MT_BUG_ON(mt, mn->slot[1] != NULL);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != 1);
-	MT_BUG_ON(mt, mas.alloc->node_count);
-
-	mas_set_alloc_req(&mas, 2); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 2);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != 3);
-	MT_BUG_ON(mt, mas.alloc == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
-	MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
-	for (i = 2; i >= 0; i--) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		MT_BUG_ON(mt, !mn);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 64;
-	mas_set_alloc_req(&mas, total); /* request 2 more. */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != total);
-	mas_set_err(&mas, -ENOMEM);
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	for (i = total; i > 0; i--) {
-		unsigned int e = 0; /* expected node_count */
-
-		if (!MAPLE_32BIT) {
-			if (i >= 35)
-				e = i - 34;
-			else if (i >= 5)
-				e = i - 4;
-			else if (i >= 2)
-				e = i - 1;
-		} else {
-			if (i >= 4)
-				e = i - 3;
-			else if (i >= 1)
-				e = i - 1;
-			else
-				e = 0;
-		}
-
-		MT_BUG_ON(mt, mas.alloc->node_count != e);
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		MT_BUG_ON(mt, mas_allocated(&mas) != i - 1);
-		MT_BUG_ON(mt, !mn);
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-
-	total = 100;
-	for (i = 1; i < total; i++) {
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = i; j > 0; j--) {
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			MT_BUG_ON(mt, !mn);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-		mas_set_alloc_req(&mas, i);
-		mas_set_err(&mas, -ENOMEM);
-		MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			nodes[j] = mas_pop_node(&mas);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-
-		while (j) {
-			j--;
-			mas_push_node(&mas, nodes[j]);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != i);
-		for (j = 0; j <= i/2; j++) {
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
-			mn = mas_pop_node(&mas);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-			MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
-		}
-		mas_reset(&mas);
-		MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL));
-		mas_destroy(&mas);
-
-	}
-
-	/* Set allocation request. */
-	total = 500;
-	mas_node_count(&mas, total);
-	/* Drop the lock and allocate the nodes. */
-	mas_nomem(&mas, GFP_KERNEL);
-	MT_BUG_ON(mt, !mas.alloc);
-	i = 1;
-	smn = mas.alloc;
-	while (i < total) {
-		for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) {
-			i++;
-			MT_BUG_ON(mt, !smn->slot[j]);
-			if (i == total)
-				break;
-		}
-		smn = smn->slot[0]; /* next. */
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != total);
-	mas_reset(&mas);
-	mas_destroy(&mas); /* Free. */
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	for (i = 1; i < 128; i++) {
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = i; j > 0; j--) { /*Free the requests */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	}
-
-	for (i = 1; i < MAPLE_NODE_MASK + 1; i++) {
-		MA_STATE(mas2, mt, 0, 0);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
-		for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */
-			mn = mas_pop_node(&mas); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mas_push_node(&mas2, mn);
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-		MT_BUG_ON(mt, mas_allocated(&mas2) != i);
-
-		for (j = i; j > 0; j--) { /*Free the requests */
-			MT_BUG_ON(mt, mas_allocated(&mas2) != j);
-			mn = mas_pop_node(&mas2); /* get the next node. */
-			MT_BUG_ON(mt, mn == NULL);
-			MT_BUG_ON(mt, not_empty(mn));
-			mn->parent = ma_parent_ptr(mn);
-			ma_free_rcu(mn);
-		}
-		MT_BUG_ON(mt, mas_allocated(&mas2) != 0);
-	}
-
-
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	mn = mas_pop_node(&mas); /* get the next node. */
-	MT_BUG_ON(mt, mn == NULL);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
-
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
-
-	/* Check the limit of pop/push/pop */
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */
-	MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
-	MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
-	MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
-	MT_BUG_ON(mt, mas_alloc_req(&mas));
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	MT_BUG_ON(mt, mas.alloc->node_count  != MAPLE_ALLOC_SLOTS);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas.alloc->node_count != 1);
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, not_empty(mn));
-	mn->parent = ma_parent_ptr(mn);
-	ma_free_rcu(mn);
-	for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) {
-		mn = mas_pop_node(&mas);
-		MT_BUG_ON(mt, not_empty(mn));
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-	}
-	MT_BUG_ON(mt, mas_allocated(&mas) != 0);
-
-
-	for (i = 3; i < MAPLE_NODE_MASK * 3; i++) {
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put it back */
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn2 = mas_pop_node(&mas); /* get the next node. */
-		mn3 = mas_pop_node(&mas); /* get the next node. */
-		mas_push_node(&mas, mn); /* put them back */
-		mas_push_node(&mas, mn2);
-		mas_push_node(&mas, mn3);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-
-		mas.node = MA_ERROR(-ENOMEM);
-		mas_node_count(&mas, i); /* Request */
-		mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mn = mas_pop_node(&mas); /* get the next node. */
-		mn->parent = ma_parent_ptr(mn);
-		ma_free_rcu(mn);
-		mas_destroy(&mas);
-	}
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 5); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != 5);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 2);
-	mas_destroy(&mas);
-
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 1); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 1);
-	mas.node = MA_ERROR(-ENOMEM);
-	mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 3 + 2); /* Request */
-	mas_nomem(&mas, GFP_KERNEL); /* Fill request */
-	mas.status = ma_start;
-	MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 3 + 2);
-	mas_destroy(&mas);
-
-	mtree_unlock(mt);
-}
-
 /*
  * Check erasing including RCU.
  */
@@ -35507,6 +35083,13 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 	return vacant_height;
 }
 
+static int mas_allocated(struct ma_state *mas)
+{
+	if (mas->sheaf)
+		return kmem_cache_sheaf_size(mas->sheaf);
+
+	return 0;
+}
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)
 {
@@ -35525,7 +35108,10 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 
 	/* Spanning store */
 	mas_set_range(&mas, 470, 500);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
+
+	mas_wr_preallocate(&wr_mas, ptr);
+	MT_BUG_ON(mt, mas.store_type != wr_spanning_store);
+	MT_BUG_ON(mt, mas_is_err(&mas));
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
 	vacant_height = get_vacant_height(&wr_mas, ptr);
@@ -35535,6 +35121,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	allocated = mas_allocated(&mas);
 	MT_BUG_ON(mt, allocated != 0);
 
+	mas_wr_preallocate(&wr_mas, ptr);
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -35575,20 +35162,6 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
 	mn->parent = ma_parent_ptr(mn);
 	ma_free_rcu(mn);
 
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	allocated = mas_allocated(&mas);
-	height = mas_mt_height(&mas);
-	vacant_height = get_vacant_height(&wr_mas, ptr);
-	MT_BUG_ON(mt, allocated != 1 + (height - vacant_height) * 3);
-	mn = mas_pop_node(&mas);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1);
-	mas_push_node(&mas, mn);
-	MT_BUG_ON(mt, mas_allocated(&mas) != allocated);
-	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
-	mas_destroy(&mas);
-	allocated = mas_allocated(&mas);
-	MT_BUG_ON(mt, allocated != 0);
-
 	MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
 	allocated = mas_allocated(&mas);
 	height = mas_mt_height(&mas);
@@ -36389,11 +35962,17 @@ static void check_nomem_writer_race(struct maple_tree *mt)
 	check_load(mt, 6, xa_mk_value(0xC));
 	mtree_unlock(mt);
 
+	mt_set_non_kernel(0);
 	/* test for the same race but with mas_store_gfp() */
 	mtree_store_range(mt, 0, 5, xa_mk_value(0xA), GFP_KERNEL);
 	mtree_store_range(mt, 6, 10, NULL, GFP_KERNEL);
 
 	mas_set_range(&mas, 0, 5);
+
+	/* setup writer 2 that will trigger the race condition */
+	mt_set_private(mt);
+	mt_set_callback(writer2);
+
 	mtree_lock(mt);
 	mas_store_gfp(&mas, NULL, GFP_KERNEL);
 
@@ -36508,10 +36087,6 @@ void farmer_tests(void)
 	check_erase_testset(&tree);
 	mtree_destroy(&tree);
 
-	mt_init_flags(&tree, 0);
-	check_new_node(&tree);
-	mtree_destroy(&tree);
-
 	if (!MAPLE_32BIT) {
 		mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
 		check_rcu_simulated(&tree);
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1..8c72571559583759456c2b469a2abc2611117c13 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -64,7 +64,8 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
 
 	if (!(gfp & __GFP_DIRECT_RECLAIM)) {
 		if (!cachep->non_kernel) {
-			cachep->exec_callback = true;
+			if (cachep->callback)
+				cachep->exec_callback = true;
 			return NULL;
 		}
 
@@ -210,6 +211,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 		for (i = 0; i < size; i++)
 			__kmem_cache_free_locked(cachep, p[i]);
 		pthread_mutex_unlock(&cachep->lock);
+		if (cachep->callback)
+			cachep->exec_callback = true;
 		return 0;
 	}
 

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:21 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:21 +0200
Subject: [PATCH v8 19/23] maple_tree: Replace mt_free_one() with kfree()
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-19-ca3099d8352c@suse.cz>

From: Pedro Falcato <pfalcato at suse.de>

kfree() is a little shorter and works with kmem_cache_alloc'd pointers
too. Also lets us remove one more helper.

Signed-off-by: Pedro Falcato <pfalcato at suse.de>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index c706e2e48f884fd156e25be2b17eb5e154774db7..0439aaacf6cb1f39d0d23af2e2a5af1d27ab32be 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -177,11 +177,6 @@ static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
 	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
 }
 
-static inline void mt_free_one(struct maple_node *node)
-{
-	kmem_cache_free(maple_node_cache, node);
-}
-
 static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 {
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
@@ -5092,7 +5087,7 @@ static void mt_free_walk(struct rcu_head *head)
 	mt_free_bulk(node->slot_len, slots);
 
 free_leaf:
-	mt_free_one(node);
+	kfree(node);
 }
 
 static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
@@ -5176,7 +5171,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
 
 free_leaf:
 	if (free)
-		mt_free_one(node);
+		kfree(node);
 	else
 		mt_clear_meta(mt, node, node->type);
 }
@@ -5385,7 +5380,7 @@ void mas_destroy(struct ma_state *mas)
 			mt_free_bulk(count, (void __rcu **)&node->slot[1]);
 			total -= count;
 		}
-		mt_free_one(ma_mnode_ptr(node));
+		kfree(ma_mnode_ptr(node));
 		total--;
 	}
 
@@ -6373,7 +6368,7 @@ static void mas_dup_free(struct ma_state *mas)
 	}
 
 	node = mte_to_node(mas->node);
-	mt_free_one(node);
+	kfree(node);
 }
 
 /*

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:25 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:25 +0200
Subject: [PATCH v8 23/23] maple_tree: Convert forking to use the sheaf
 interface
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-23-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

Use the generic interface which should result in less bulk allocations
during a forking.

A part of this is to abstract the freeing of the sheaf or maple state
allocations into its own function so mas_destroy() and the tree
duplication code can use the same functionality to return any unused
resources.

[andriy.shevchenko at linux.intel.com: remove unused mt_alloc_bulk()]
Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Reviewed-by: Suren Baghdasaryan <surenb at google.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 lib/maple_tree.c | 47 +++++++++++++++++++++++------------------------
 1 file changed, 23 insertions(+), 24 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index a912e6a1d4378e72b967027b60f8f564476ad14e..bb51424053a5c4ceece7604877dfa3cd3780944a 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -172,11 +172,6 @@ static inline struct maple_node *mt_alloc_one(gfp_t gfp)
 	return kmem_cache_alloc(maple_node_cache, gfp);
 }
 
-static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
-{
-	return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
-}
-
 static inline void mt_free_bulk(size_t size, void __rcu **nodes)
 {
 	kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
@@ -1150,6 +1145,19 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 	mas_set_err(mas, -ENOMEM);
 }
 
+static inline void mas_empty_nodes(struct ma_state *mas)
+{
+	mas->node_request = 0;
+	if (mas->sheaf) {
+		mt_return_sheaf(mas->sheaf);
+		mas->sheaf = NULL;
+	}
+
+	if (mas->alloc) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+}
 
 /*
  * mas_free() - Free an encoded maple node
@@ -5208,15 +5216,7 @@ EXPORT_SYMBOL_GPL(mas_preallocate);
 void mas_destroy(struct ma_state *mas)
 {
 	mas->mas_flags &= ~MA_STATE_PREALLOC;
-
-	mas->node_request = 0;
-	if (mas->sheaf)
-		mt_return_sheaf(mas->sheaf);
-	mas->sheaf = NULL;
-
-	if (mas->alloc)
-		kfree(mas->alloc);
-	mas->alloc = NULL;
+	mas_empty_nodes(mas);
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -6241,7 +6241,7 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	struct maple_node *node = mte_to_node(mas->node);
 	struct maple_node *new_node = mte_to_node(new_mas->node);
 	enum maple_type type;
-	unsigned char request, count, i;
+	unsigned char count, i;
 	void __rcu **slots;
 	void __rcu **new_slots;
 	unsigned long val;
@@ -6249,20 +6249,17 @@ static inline void mas_dup_alloc(struct ma_state *mas, struct ma_state *new_mas,
 	/* Allocate memory for child nodes. */
 	type = mte_node_type(mas->node);
 	new_slots = ma_slots(new_node, type);
-	request = mas_data_end(mas) + 1;
-	count = mt_alloc_bulk(gfp, request, (void **)new_slots);
-	if (unlikely(count < request)) {
-		memset(new_slots, 0, request * sizeof(void *));
-		mas_set_err(mas, -ENOMEM);
+	count = mas->node_request = mas_data_end(mas) + 1;
+	mas_alloc_nodes(mas, gfp);
+	if (unlikely(mas_is_err(mas)))
 		return;
-	}
 
-	/* Restore node type information in slots. */
 	slots = ma_slots(node, type);
 	for (i = 0; i < count; i++) {
 		val = (unsigned long)mt_slot_locked(mas->tree, slots, i);
 		val &= MAPLE_NODE_MASK;
-		((unsigned long *)new_slots)[i] |= val;
+		new_slots[i] = ma_mnode_ptr((unsigned long)mas_pop_node(mas) |
+					    val);
 	}
 }
 
@@ -6316,7 +6313,7 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 			/* Only allocate child nodes for non-leaf nodes. */
 			mas_dup_alloc(mas, new_mas, gfp);
 			if (unlikely(mas_is_err(mas)))
-				return;
+				goto empty_mas;
 		} else {
 			/*
 			 * This is the last leaf node and duplication is
@@ -6349,6 +6346,8 @@ static inline void mas_dup_build(struct ma_state *mas, struct ma_state *new_mas,
 	/* Make them the same height */
 	new_mas->tree->ma_flags = mas->tree->ma_flags;
 	rcu_assign_pointer(new_mas->tree->ma_root, root);
+empty_mas:
+	mas_empty_nodes(mas);
 }
 
 /**

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:24 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:24 +0200
Subject: [PATCH v8 22/23] maple_tree: Add single node allocation support to
 maple state
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-22-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>

The fast path through a write will require replacing a single node in
the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
special case the node store operation by just allocating one node in the
maple state.

Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 include/linux/maple_tree.h       |  4 +++-
 lib/maple_tree.c                 | 47 +++++++++++++++++++++++++++++++++++-----
 tools/testing/radix-tree/maple.c |  9 ++++++--
 3 files changed, 51 insertions(+), 9 deletions(-)

diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
index 166fd67e00d882b1e6de1f80c1b590bba7497cd3..562a1e9e5132b5b1fa8f8402a7cadd8abb65e323 100644
--- a/include/linux/maple_tree.h
+++ b/include/linux/maple_tree.h
@@ -443,6 +443,7 @@ struct ma_state {
 	unsigned long min;		/* The minimum index of this node - implied pivot min */
 	unsigned long max;		/* The maximum index of this node - implied pivot max */
 	struct slab_sheaf *sheaf;	/* Allocated nodes for this operation */
+	struct maple_node *alloc;	/* allocated nodes */
 	unsigned long node_request;
 	enum maple_status status;	/* The status of the state (active, start, none, etc) */
 	unsigned char depth;		/* depth of tree descent during write */
@@ -491,8 +492,9 @@ struct ma_wr_state {
 		.status = ma_start,					\
 		.min = 0,						\
 		.max = ULONG_MAX,					\
-		.node_request= 0,					\
 		.sheaf = NULL,						\
+		.alloc = NULL,						\
+		.node_request= 0,					\
 		.mas_flags = 0,						\
 		.store_type = wr_invalid,				\
 	}
diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index a3fcb20227e506ed209554cc8c041a53f7ef4903..a912e6a1d4378e72b967027b60f8f564476ad14e 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -1073,16 +1073,23 @@ static int mas_ascend(struct ma_state *mas)
  *
  * Return: A pointer to a maple node.
  */
-static inline struct maple_node *mas_pop_node(struct ma_state *mas)
+static __always_inline struct maple_node *mas_pop_node(struct ma_state *mas)
 {
 	struct maple_node *ret;
 
+	if (mas->alloc) {
+		ret = mas->alloc;
+		mas->alloc = NULL;
+		goto out;
+	}
+
 	if (WARN_ON_ONCE(!mas->sheaf))
 		return NULL;
 
 	ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
-	memset(ret, 0, sizeof(*ret));
 
+out:
+	memset(ret, 0, sizeof(*ret));
 	return ret;
 }
 
@@ -1093,9 +1100,34 @@ static inline struct maple_node *mas_pop_node(struct ma_state *mas)
  */
 static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
 {
-	if (unlikely(mas->sheaf)) {
-		unsigned long refill = mas->node_request;
+	if (!mas->node_request)
+		return;
+
+	if (mas->node_request == 1) {
+		if (mas->sheaf)
+			goto use_sheaf;
+
+		if (mas->alloc)
+			return;
 
+		mas->alloc = mt_alloc_one(gfp);
+		if (!mas->alloc)
+			goto error;
+
+		mas->node_request = 0;
+		return;
+	}
+
+use_sheaf:
+	if (unlikely(mas->alloc)) {
+		kfree(mas->alloc);
+		mas->alloc = NULL;
+	}
+
+	if (mas->sheaf) {
+		unsigned long refill;
+
+		refill = mas->node_request;
 		if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
 			mas->node_request = 0;
 			return;
@@ -5180,8 +5212,11 @@ void mas_destroy(struct ma_state *mas)
 	mas->node_request = 0;
 	if (mas->sheaf)
 		mt_return_sheaf(mas->sheaf);
-
 	mas->sheaf = NULL;
+
+	if (mas->alloc)
+		kfree(mas->alloc);
+	mas->alloc = NULL;
 }
 EXPORT_SYMBOL_GPL(mas_destroy);
 
@@ -5816,7 +5851,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
 		mas_alloc_nodes(mas, gfp);
 	}
 
-	if (!mas->sheaf)
+	if (!mas->sheaf && !mas->alloc)
 		return false;
 
 	mas->status = ma_start;
diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
index 72a8fe8e832a4150c6567b711768eba6a3fa6768..83260f2efb1990b71093e456950069c24d75560e 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -35085,10 +35085,15 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
 
 static int mas_allocated(struct ma_state *mas)
 {
+	int total = 0;
+
+	if (mas->alloc)
+		total++;
+
 	if (mas->sheaf)
-		return kmem_cache_sheaf_size(mas->sheaf);
+		total += kmem_cache_sheaf_size(mas->sheaf);
 
-	return 0;
+	return total;
 }
 /* Preallocation testing */
 static noinline void __init check_prealloc(struct maple_tree *mt)

-- 
2.51.0



From vbabka at suse.cz  Wed Sep 10 01:01:22 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 10 Sep 2025 10:01:22 +0200
Subject: [PATCH v8 20/23] tools/testing: Add support for prefilled slab
 sheafs
In-Reply-To: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
Message-ID: <20250910-slub-percpu-caches-v8-20-ca3099d8352c@suse.cz>

From: "Liam R. Howlett" <Liam.Howlett at oracle.com>

Add the prefilled sheaf structs to the slab header and the associated
functions to the testing/shared/linux.c file.

Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 tools/include/linux/slab.h   | 28 ++++++++++++++
 tools/testing/shared/linux.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 117 insertions(+)

diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
index c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08..94937a699402bd1f31887dfb52b6fd0a3c986f43 100644
--- a/tools/include/linux/slab.h
+++ b/tools/include/linux/slab.h
@@ -123,6 +123,18 @@ struct kmem_cache_args {
 	void (*ctor)(void *);
 };
 
+struct slab_sheaf {
+	union {
+		struct list_head barn_list;
+		/* only used for prefilled sheafs */
+		unsigned int capacity;
+	};
+	struct kmem_cache *cache;
+	unsigned int size;
+	int node; /* only used for rcu_sheaf */
+	void *objects[];
+};
+
 static inline void *kzalloc(size_t size, gfp_t gfp)
 {
 	return kmalloc(size, gfp | __GFP_ZERO);
@@ -173,5 +185,21 @@ __kmem_cache_create(const char *name, unsigned int size, unsigned int align,
 void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
 int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
 			  void **list);
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf);
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf **sheafp, unsigned int size);
+
+static inline unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
+{
+	return sheaf->size;
+}
 
 #endif		/* _TOOLS_SLAB_H */
diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
index 97b8412ccbb6d222604c7b397c53c65618d8d51b..4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1 100644
--- a/tools/testing/shared/linux.c
+++ b/tools/testing/shared/linux.c
@@ -137,6 +137,12 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
 	if (kmalloc_verbose)
 		pr_debug("Bulk free %p[0-%zu]\n", list, size - 1);
 
+	if (cachep->exec_callback) {
+		if (cachep->callback)
+			cachep->callback(cachep->private);
+		cachep->exec_callback = false;
+	}
+
 	pthread_mutex_lock(&cachep->lock);
 	for (int i = 0; i < size; i++)
 		kmem_cache_free_locked(cachep, list[i]);
@@ -242,6 +248,89 @@ __kmem_cache_create_args(const char *name, unsigned int size,
 	return ret;
 }
 
+struct slab_sheaf *
+kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
+{
+	struct slab_sheaf *sheaf;
+	unsigned int capacity;
+
+	if (s->exec_callback) {
+		if (s->callback)
+			s->callback(s->private);
+		s->exec_callback = false;
+	}
+
+	capacity = max(size, s->sheaf_capacity);
+
+	sheaf = calloc(1, sizeof(*sheaf) + sizeof(void *) * capacity);
+	if (!sheaf)
+		return NULL;
+
+	sheaf->cache = s;
+	sheaf->capacity = capacity;
+	sheaf->size = kmem_cache_alloc_bulk(s, gfp, size, sheaf->objects);
+	if (!sheaf->size) {
+		free(sheaf);
+		return NULL;
+	}
+
+	return sheaf;
+}
+
+int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf **sheafp, unsigned int size)
+{
+	struct slab_sheaf *sheaf = *sheafp;
+	int refill;
+
+	if (sheaf->size >= size)
+		return 0;
+
+	if (size > sheaf->capacity) {
+		sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
+		if (!sheaf)
+			return -ENOMEM;
+
+		kmem_cache_return_sheaf(s, gfp, *sheafp);
+		*sheafp = sheaf;
+		return 0;
+	}
+
+	refill = kmem_cache_alloc_bulk(s, gfp, size - sheaf->size,
+				       &sheaf->objects[sheaf->size]);
+	if (!refill)
+		return -ENOMEM;
+
+	sheaf->size += refill;
+	return 0;
+}
+
+void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
+		 struct slab_sheaf *sheaf)
+{
+	if (sheaf->size)
+		kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
+
+	free(sheaf);
+}
+
+void *
+kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
+		struct slab_sheaf *sheaf)
+{
+	void *obj;
+
+	if (sheaf->size == 0) {
+		printf("Nothing left in sheaf!\n");
+		return NULL;
+	}
+
+	obj = sheaf->objects[--sheaf->size];
+	sheaf->objects[sheaf->size] = NULL;
+
+	return obj;
+}
+
 /*
  * Test the test infrastructure for kem_cache_alloc/free and bulk counterparts.
  */

-- 
2.51.0



From pfalcato at suse.de  Wed Sep 10 05:51:47 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 13:51:47 +0100
Subject: [PATCH v1 1/9] mm/mmap: Move exit_mmap() trace point
In-Reply-To: <20250909190945.1030905-2-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-2-Liam.Howlett@oracle.com>
Message-ID: <q6qlfs74tevbckjowf76y4fbfejr4ulyzrb4fkgq25qgebqnaf@colkcmdzt2bd>

On Tue, Sep 09, 2025 at 03:09:37PM -0400, Liam R. Howlett wrote:
> Move the trace point later in the function so that it is not skipped in
> the event of a failed fork.
> 
> Acked-by: Chris Li <chrisl at kernel.org>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 05:54:25 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 13:54:25 +0100
Subject: [PATCH v1 2/9] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <20250909190945.1030905-3-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-3-Liam.Howlett@oracle.com>
Message-ID: <lrkxxt3ywknfqudhf4u5kwgegrewqw6laxqonx5ysqmop6flxc@gwbqpadk5btv>

On Tue, Sep 09, 2025 at 03:09:38PM -0400, Liam R. Howlett wrote:
> Create the new function tear_down_vmas() to remove a range of vmas.
> exit_mmap() will be removing all the vmas.
> 
> This is necessary for future patches.
> 
> No functional changes intended.
> 
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  mm/mmap.c | 37 ++++++++++++++++++++++++-------------
>  1 file changed, 24 insertions(+), 13 deletions(-)
> 
> diff --git a/mm/mmap.c b/mm/mmap.c
> index b07b3ec5e28f5..a290448a53bb2 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
>  }
>  EXPORT_SYMBOL(vm_brk_flags);
>  
> +static inline
> +unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
> +		struct vm_area_struct *vma, unsigned long max)
> +{
> +	unsigned long nr_accounted = 0;
> +	int count = 0;
> +
> +	mmap_assert_write_locked(mm);
> +	vma_iter_set(vmi, vma->vm_end);
> +	do {
> +		if (vma->vm_flags & VM_ACCOUNT)
> +			nr_accounted += vma_pages(vma);
> +		vma_mark_detached(vma);
> +		remove_vma(vma);
> +		count++;
> +		cond_resched();
> +		vma = vma_next(vmi);
> +	} while (vma && vma->vm_end <= max);

By not checking for XA_ZERO_ENTRY, we're technically breaking bisectability
here.

In any case,

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 05:56:59 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 13:56:59 +0100
Subject: [PATCH v1 3/9] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <20250909190945.1030905-4-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-4-Liam.Howlett@oracle.com>
Message-ID: <k45ojp4yjaqp56azyuobcsxyqmm5uslo5gq3ddujwfi3ynvkqz@2oncyj3tbnhy>

On Tue, Sep 09, 2025 at 03:09:39PM -0400, Liam R. Howlett wrote:
> Add a limit to the vma search instead of using the start and end of the
> one passed in.
> 
> No functional changes intended.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  mm/vma.c | 6 ++++--
>  mm/vma.h | 1 +
>  2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vma.c b/mm/vma.c
> index abe0da33c8446..a648e0555c873 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
>   * Called with the mm semaphore held.
>   */
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +		unsigned long vma_min, unsigned long vma_max,
>  		struct vm_area_struct *prev, struct vm_area_struct *next)
>  {
>  	struct mm_struct *mm = vma->vm_mm;
> @@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>  
>  	tlb_gather_mmu(&tlb, mm);
>  	update_hiwater_rss(mm);
> -	unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
> +	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
>  		   /* mm_wr_locked = */ true);
>  	mas_set(mas, vma->vm_end);
>  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> @@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>  
>  		vma_iter_set(vmi, vma->vm_end);
>  		/* Undo any partial mapping done by a device driver. */
> -		unmap_region(&vmi->mas, vma, map->prev, map->next);
> +		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> +			     map->prev, map->next);
>  
>  		return error;
>  	}
> diff --git a/mm/vma.h b/mm/vma.h
> index 9183fe5490090..a9d0cef684ddb 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -261,6 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>  void remove_vma(struct vm_area_struct *vma);
>  
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +		unsigned long min, unsigned long max,
>  		struct vm_area_struct *prev, struct vm_area_struct *next);
>  

nit: min and max don't match the arg names in the actual definition of the function

Reviewed-by: Pedro Falcato <pfalcato at suse.de>


-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 06:08:01 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 14:08:01 +0100
Subject: [PATCH v1 4/9] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <20250909190945.1030905-5-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-5-Liam.Howlett@oracle.com>
Message-ID: <3pn4anukobmyhul2zw6cqwl4nvu47ecji4cldgsfer24p35k6s@km5dfxpzvzzf>

On Tue, Sep 09, 2025 at 03:09:40PM -0400, Liam R. Howlett wrote:
> The ceiling and tree search limit need to be different arguments for the
> future change in the failed fork attempt.
> 
> Add some documentation around free_pgtables() and the limits in an
> attempt to clarify the floor and ceiling use as well as the new
> tree_max.
> 
> Test code also updated.
> 
> No functional changes intended.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 06:08:34 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 14:08:34 +0100
Subject: [PATCH v1 5/9] mm/vma: Add page table limit to unmap_region()
In-Reply-To: <20250909190945.1030905-6-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-6-Liam.Howlett@oracle.com>
Message-ID: <5li7vx377obryqypk34l7aw4plsulyg7jqbdhfwkx3mawlzugd@yqqjkvy4uzou>

On Tue, Sep 09, 2025 at 03:09:41PM -0400, Liam R. Howlett wrote:
> The unmap_region() calls need to pass through the page table limit for a
> future patch.
> 
> No functional changes intended.
> 
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 06:10:17 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 14:10:17 +0100
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce
 function arguments
In-Reply-To: <20250909190945.1030905-8-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com>
Message-ID: <ss4haln3wwk4gv62hhhdmx7tyog4dyd6auogvupm26idludj4k@abasio3iebze>

On Tue, Sep 09, 2025 at 03:09:43PM -0400, Liam R. Howlett wrote:
> The unmap_region code uses a number of arguments that could use better
> documentation.  With the addition of a descriptor for unmap (called
> unmap_desc), the arguments can be more self-documenting and increase the
> descriptions within the declaration.
> 
> No functional change intended
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Neat, new helper struct just dropped!

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From pfalcato at suse.de  Wed Sep 10 06:23:48 2025
From: pfalcato at suse.de (Pedro Falcato)
Date: Wed, 10 Sep 2025 14:23:48 +0100
Subject: [PATCH v1 6/9] mm: Change dup_mmap() recovery
In-Reply-To: <20250909190945.1030905-7-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-7-Liam.Howlett@oracle.com>
Message-ID: <fkfi4zldsecyio6xnd6blou27me4nno3v3tnsaxz6xlt63yy2z@nxl7izptm6yx>

On Tue, Sep 09, 2025 at 03:09:42PM -0400, Liam R. Howlett wrote:
> When the dup_mmap() fails during the vma duplication or setup, don't
> write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> free the new resources, leaving an empty vma tree.
> 
> Using XA_ZERO introduced races where the vma could be found between
> dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> race can occur because the mm can be reached through the other trees
> via successfully copied vmas and other methods such as the swapoff code.
> 
> XA_ZERO was marking the location to stop vma removal and pagetable
> freeing.  The newly created arguments to the unmap_vmas() and
> free_pgtables() serve this function.
> 
> Replacing the XA_ZERO entry use with the new argument list also means
> the checks for xa_is_zero() are no longer necessary so these are also
> removed.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Reviewed-by: Pedro Falcato <pfalcato at suse.de>

-- 
Pedro


From ojeda at kernel.org  Wed Sep 10 07:02:12 2025
From: ojeda at kernel.org (Miguel Ojeda)
Date: Wed, 10 Sep 2025 16:02:12 +0200
Subject: [PATCH] rust: maple_tree: fix intra-doc link
Message-ID: <20250910140212.997771-1-ojeda@kernel.org>

A couple `Occupied` intra-doc links are broken:

    error: unresolved link to `Occupied`
      --> rust/kernel/maple_tree.rs:96:46
       |
    96 |     /// return an [`InsertError`] with the [`Occupied`] kind. It may also fail if memory
       |                                              ^^^^^^^^ no item named `Occupied` in scope
       |
       = help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
       = note: `-D rustdoc::broken-intra-doc-links` implied by `-D warnings`
       = help: to override `-D warnings` add `#[allow(rustdoc::broken_intra_doc_links)]`

    error: unresolved link to `Occupied`
       --> rust/kernel/maple_tree.rs:135:36
        |
    135 |     /// [`InsertError`] with the [`Occupied`] kind. It may also fail if memory allocation fails
        |                                    ^^^^^^^^ no item named `Occupied` in scope
        |
        = help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`

Thus fix them.

Fixes: 7f51d867d2ed ("rust: maple_tree: add MapleTree")
Signed-off-by: Miguel Ojeda <ojeda at kernel.org>
---
Andrew: as usual, please feel free to rebase if that is what you
normally do, of course. This was in next-20250910. Thanks!

 rust/kernel/maple_tree.rs | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/rust/kernel/maple_tree.rs b/rust/kernel/maple_tree.rs
index 31b6c13d08ef..e72eec56bf57 100644
--- a/rust/kernel/maple_tree.rs
+++ b/rust/kernel/maple_tree.rs
@@ -93,8 +93,7 @@ pub fn new() -> impl PinInit<Self> {
     /// # Errors
     ///
     /// If the maple tree already contains a range using the given index, then this call will
-    /// return an [`InsertError`] with the [`Occupied`] kind. It may also fail if memory
-    /// allocation fails.
+    /// return an [`InsertErrorKind::Occupied`]. It may also fail if memory allocation fails.
     ///
     /// # Examples
     ///
@@ -132,8 +131,8 @@ pub fn insert(&self, index: usize, value: T, gfp: Flags) -> Result<(), InsertErr
     /// # Errors
     ///
     /// If the maple tree already contains an overlapping range, then this call will return an
-    /// [`InsertError`] with the [`Occupied`] kind. It may also fail if memory allocation fails
-    /// or if the requested range is invalid (e.g. empty).
+    /// [`InsertErrorKind::Occupied`]. It may also fail if memory allocation fails or if the
+    /// requested range is invalid (e.g. empty).
     ///
     /// # Examples
     ///

base-commit: a3bff28df80b33386863823262550491155e896b
--
2.51.0


From lorenzo.stoakes at oracle.com  Thu Sep 11 01:56:45 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 09:56:45 +0100
Subject: [PATCH v1 3/9] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <20250909190945.1030905-4-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-4-Liam.Howlett@oracle.com>
Message-ID: <3744d50c-caac-4208-beb2-0cb3c6889b51@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:39PM -0400, Liam R. Howlett wrote:
> Add a limit to the vma search instead of using the start and end of the
> one passed in.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

OK I see you are splitting this up then addressing the args thing in a later
patch so that's all good :)

One nit below, but otherwise LGTM so:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

> ---
>  mm/vma.c | 6 ++++--
>  mm/vma.h | 1 +
>  2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/mm/vma.c b/mm/vma.c
> index abe0da33c8446..a648e0555c873 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
>   * Called with the mm semaphore held.
>   */
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +		unsigned long vma_min, unsigned long vma_max,
>  		struct vm_area_struct *prev, struct vm_area_struct *next)
>  {
>  	struct mm_struct *mm = vma->vm_mm;
> @@ -481,7 +482,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>
>  	tlb_gather_mmu(&tlb, mm);
>  	update_hiwater_rss(mm);
> -	unmap_vmas(&tlb, mas, vma, vma->vm_start, vma->vm_end, vma->vm_end,
> +	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
>  		   /* mm_wr_locked = */ true);
>  	mas_set(mas, vma->vm_end);
>  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> @@ -2417,7 +2418,8 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>
>  		vma_iter_set(vmi, vma->vm_end);
>  		/* Undo any partial mapping done by a device driver. */
> -		unmap_region(&vmi->mas, vma, map->prev, map->next);
> +		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> +			     map->prev, map->next);
>
>  		return error;
>  	}
> diff --git a/mm/vma.h b/mm/vma.h
> index 9183fe5490090..a9d0cef684ddb 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -261,6 +261,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>  void remove_vma(struct vm_area_struct *vma);
>
>  void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +		unsigned long min, unsigned long max,

NIT: you're changing the name of the parameter for min, max to vma_min, vma_max
between here and the implementation. Let's keep that consistent!

>  		struct vm_area_struct *prev, struct vm_area_struct *next);
>
>  /* We are about to modify the VMA's flags. */
> --
> 2.47.2
>


From lorenzo.stoakes at oracle.com  Thu Sep 11 02:08:08 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 10:08:08 +0100
Subject: [PATCH v1 4/9] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <20250909190945.1030905-5-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-5-Liam.Howlett@oracle.com>
Message-ID: <e44e1da5-7ee4-478a-bcc5-7726f3b52d9f@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:40PM -0400, Liam R. Howlett wrote:
> The ceiling and tree search limit need to be different arguments for the
> future change in the failed fork attempt.
>
> Add some documentation around free_pgtables() and the limits in an
> attempt to clarify the floor and ceiling use as well as the new
> tree_max.
>
> Test code also updated.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

LGTM other than the nits below, so with those addressed feel free to add:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

> ---
>  mm/internal.h                    |  4 +++-
>  mm/memory.c                      | 28 +++++++++++++++++++++++++---
>  mm/mmap.c                        |  2 +-
>  mm/vma.c                         |  3 ++-
>  tools/testing/vma/vma_internal.h |  3 ++-
>  5 files changed, 33 insertions(+), 7 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 63e3ec8d63be7..d295252407fee 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -444,7 +444,9 @@ void folio_activate(struct folio *folio);
>
>  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		   struct vm_area_struct *start_vma, unsigned long floor,
> -		   unsigned long ceiling, bool mm_wr_locked);
> +		   unsigned long ceiling, unsigned long tree_max,
> +		   bool mm_wr_locked);
> +
>  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
>
>  struct zap_details;
> diff --git a/mm/memory.c b/mm/memory.c
> index 3e0404bd57a02..24716b3713f66 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -369,12 +369,34 @@ void free_pgd_range(struct mmu_gather *tlb,
>  	} while (pgd++, addr = next, addr != end);
>  }
>
> +/*

NIT: /** right? This looks like a kernel-doc to me!

> + * free_pgtables() - Free a range of page tables
> + * @tlb: The mmu gather
> + * @mas: The maple state
> + * @vma: The first vma
> + * @floor: The lowest page table address
> + * @ceiling: The highest page table address
> + * @tree_max: The highest tree search address
> + * @mm_wr_locked: boolean indicating if the mm is write locked
> + *
> + * Note: Floor and ceiling are provided to indicate the absolute range of the
> + * page tables that should be removed.  This can differ from the vma mappings on
> + * some archs that may have mappings that need to be removed outside the vmas.
> + * Note that the prev->vm_end and next->vm_start are often used.

Great write-up though you are missing some horrified noises re: the arches doing
that, I guess the reader has to play them back in their head... ;)

> + *
> + * The tree_max differs from the ceiling when a dup_mmap() failed and the tree
> + * has unrelated data to the mm_struct being torn down.
> + */

Ohhh nice nice thanks for adding this comment!

>  void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		   struct vm_area_struct *vma, unsigned long floor,
> -		   unsigned long ceiling, bool mm_wr_locked)
> +		   unsigned long ceiling, unsigned long tree_max,
> +		   bool mm_wr_locked)
>  {
>  	struct unlink_vma_file_batch vb;
>
> +	/* underflow can happen and is fine */

I am taking this as a sign that underflow is in fact fine _everywhere_
including in internal plumbing! :P

> +	WARN_ON_ONCE(tree_max - 1 > ceiling - 1);

Hmm, so if tree_max == 1, and ceilling == 0, we're ok with that (would
resolve to tree_max = 0 and ceiling == ULONG_MAX), I guess relating to the
'interpret 0 as everything' semantics I think we have for ceiling?

I guess it's because these are exclusive.

So perhaps worth updating comment to:

/*
 * these values are exclusive bounds, with 0 being interpreted as the
 * entire range, so underflow is fine.
 */

or similar, just to really underline that...

> +
>  	tlb_free_vmas(tlb);
>
>  	do {
> @@ -385,7 +407,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
>  		 * be 0.  This will underflow and is okay.
>  		 */
> -		next = mas_find(mas, ceiling - 1);
> +		next = mas_find(mas, tree_max - 1);
>  		if (unlikely(xa_is_zero(next)))
>  			next = NULL;
>
> @@ -405,7 +427,7 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		 */
>  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>  			vma = next;
> -			next = mas_find(mas, ceiling - 1);
> +			next = mas_find(mas, tree_max - 1);
>  			if (unlikely(xa_is_zero(next)))
>  				next = NULL;
>  			if (mm_wr_locked)
> diff --git a/mm/mmap.c b/mm/mmap.c
> index a290448a53bb2..0f4808f135fe6 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1311,7 +1311,7 @@ void exit_mmap(struct mm_struct *mm)
>  	mt_clear_in_rcu(&mm->mm_mt);
>  	vma_iter_set(&vmi, vma->vm_end);
>  	free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
> -		      USER_PGTABLES_CEILING, true);
> +		      USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);

NIT: Might be nice, while we're here, to add your (very nice) convention of
prefacing boolean params with the param name, so here:

..., /* mm_wr_locked= */ true);

>  	tlb_finish_mmu(&tlb);
>
>  	/*
> diff --git a/mm/vma.c b/mm/vma.c
> index a648e0555c873..1bae142bbc0f1 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -486,6 +486,7 @@ void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
>  		   /* mm_wr_locked = */ true);
>  	mas_set(mas, vma->vm_end);
>  	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> +		      next ? next->vm_start : USER_PGTABLES_CEILING,
>  		      next ? next->vm_start : USER_PGTABLES_CEILING,
>  		      /* mm_wr_locked = */ true);
>  	tlb_finish_mmu(&tlb);
> @@ -1232,7 +1233,7 @@ static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
>  	mas_set(mas_detach, 1);
>  	/* start and end may be different if there is no prev or next vma. */
>  	free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
> -		      vms->unmap_end, mm_wr_locked);
> +		      vms->unmap_end, vms->unmap_end, mm_wr_locked);
>  	tlb_finish_mmu(&tlb);
>  	vms->clear_ptes = false;
>  }
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index 07167446dcf42..823d379e1fac2 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -900,7 +900,8 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>
>  static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		   struct vm_area_struct *vma, unsigned long floor,
> -		   unsigned long ceiling, bool mm_wr_locked)
> +		   unsigned long ceiling, unsigned long tree_max,
> +		   bool mm_wr_locked)
>  {
>  	(void)tlb;
>  	(void)mas;
> --
> 2.47.2
>


From lorenzo.stoakes at oracle.com  Thu Sep 11 02:13:06 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 10:13:06 +0100
Subject: [PATCH v1 6/9] mm: Change dup_mmap() recovery
In-Reply-To: <20250909190945.1030905-7-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-7-Liam.Howlett@oracle.com>
Message-ID: <c4cae5fa-9136-4e9a-b6ac-6073ebbabffb@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:42PM -0400, Liam R. Howlett wrote:
> When the dup_mmap() fails during the vma duplication or setup, don't
> write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> free the new resources, leaving an empty vma tree.
>
> Using XA_ZERO introduced races where the vma could be found between
> dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> race can occur because the mm can be reached through the other trees
> via successfully copied vmas and other methods such as the swapoff code.
>
> XA_ZERO was marking the location to stop vma removal and pagetable
> freeing.  The newly created arguments to the unmap_vmas() and
> free_pgtables() serve this function.
>
> Replacing the XA_ZERO entry use with the new argument list also means
> the checks for xa_is_zero() are no longer necessary so these are also
> removed.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

LGTM, so:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

> ---
>  mm/memory.c |  6 +-----
>  mm/mmap.c   | 42 +++++++++++++++++++++++++++++++-----------
>  2 files changed, 32 insertions(+), 16 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 24716b3713f66..829cd94950182 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -408,8 +408,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		 * be 0.  This will underflow and is okay.
>  		 */
>  		next = mas_find(mas, tree_max - 1);
> -		if (unlikely(xa_is_zero(next)))
> -			next = NULL;
>
>  		/*
>  		 * Hide vma from rmap and truncate_pagecache before freeing
> @@ -428,8 +426,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>  			vma = next;
>  			next = mas_find(mas, tree_max - 1);
> -			if (unlikely(xa_is_zero(next)))
> -				next = NULL;
>  			if (mm_wr_locked)
>  				vma_start_write(vma);
>  			unlink_anon_vmas(vma);
> @@ -2129,7 +2125,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>  				 mm_wr_locked);
>  		hugetlb_zap_end(vma, &details);
>  		vma = mas_find(mas, tree_end - 1);
> -	} while (vma && likely(!xa_is_zero(vma)));
> +	} while (vma);
>  	mmu_notifier_invalidate_range_end(&range);
>  }
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 0f4808f135fe6..aa4770b8d7f1e 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
>  	arch_exit_mmap(mm);
>
>  	vma = vma_next(&vmi);
> -	if (!vma || unlikely(xa_is_zero(vma))) {
> +	if (!vma) {
>  		/* Can happen if dup_mmap() received an OOM */
>  		mmap_read_unlock(mm);
>  		mmap_write_lock(mm);
> @@ -1858,20 +1858,40 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
>  		ksm_fork(mm, oldmm);
>  		khugepaged_fork(mm, oldmm);
>  	} else {
> +		unsigned long max;
>
>  		/*
> -		 * The entire maple tree has already been duplicated. If the
> -		 * mmap duplication fails, mark the failure point with
> -		 * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
> -		 * stop releasing VMAs that have not been duplicated after this
> -		 * point.
> +		 * The entire maple tree has already been duplicated, but
> +		 * replacing the vmas failed at mpnt (which could be NULL if
> +		 * all were allocated but the last vma was not fully set up).
> +		 * Use the start address of the failure point to clean up the
> +		 * partially initialized tree.
>  		 */

Thanks for this! Great comment.

> -		if (mpnt) {
> -			mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
> -			mas_store(&vmi.mas, XA_ZERO_ENTRY);
> -			/* Avoid OOM iterating a broken tree */
> -			mm_flags_set(MMF_OOM_SKIP, mm);
> +		if (!mm->map_count) {
> +			/* zero vmas were written to the new tree. */
> +			max = 0;
> +		} else if (mpnt) {
> +			/* partial tree failure */
> +			max = mpnt->vm_start;
> +		} else {
> +			/* All vmas were written to the new tree */
> +			max = ULONG_MAX;
>  		}
> +
> +		/* Hide mm from oom killer because the memory is being freed */
> +		mm_flags_set(MMF_OOM_SKIP, mm);
> +		if (max) {
> +			vma_iter_set(&vmi, 0);
> +			tmp = vma_next(&vmi);
> +			flush_cache_mm(mm);
> +			unmap_region(&vmi.mas, /* vma = */ tmp,
> +				     /*vma_min = */ 0, /* vma_max = */ max,
> +				     /* pg_max = */ max, /* prev = */ NULL,
> +				     /* next = */ NULL);

Thanks! This really helps.

> +			charge = tear_down_vmas(mm, &vmi, tmp, max);
> +			vm_unacct_memory(charge);
> +		}
> +		__mt_destroy(&mm->mm_mt);
>  		/*
>  		 * The mm_struct is going to exit, but the locks will be dropped
>  		 * first.  Set the mm_struct as unstable is advisable as it is
> --
> 2.47.2
>


From david at redhat.com  Thu Sep 11 02:19:46 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 11 Sep 2025 11:19:46 +0200
Subject: [PATCH v1 1/9] mm/mmap: Move exit_mmap() trace point
In-Reply-To: <20250909190945.1030905-2-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-2-Liam.Howlett@oracle.com>
Message-ID: <185efd78-56b6-49b8-9c8c-b5363ad70a6d@redhat.com>

On 09.09.25 21:09, Liam R. Howlett wrote:
> Move the trace point later in the function so that it is not skipped in
> the event of a failed fork.
> 
> Acked-by: Chris Li <chrisl at kernel.org>
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>   mm/mmap.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 5fd3b80fda1d5..b07b3ec5e28f5 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1310,9 +1310,9 @@ void exit_mmap(struct mm_struct *mm)


Reviewed-by: David Hildenbrand <david at redhat.com>

-- 
Cheers

David / dhildenb



From lorenzo.stoakes at oracle.com  Thu Sep 11 02:20:16 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 10:20:16 +0100
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce
 function arguments
In-Reply-To: <20250909190945.1030905-8-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com>
Message-ID: <8102e84f-a12b-4e8b-9e24-99e2e9bc112d@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:43PM -0400, Liam R. Howlett wrote:
> The unmap_region code uses a number of arguments that could use better
> documentation.  With the addition of a descriptor for unmap (called
> unmap_desc), the arguments can be more self-documenting and increase the
> descriptions within the declaration.
>
> No functional change intended
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

Apart from bikeshed below, LGTM so:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

> ---
>  mm/mmap.c | 12 ++++++++----
>  mm/vma.c  | 27 ++++++++++++---------------
>  mm/vma.h  | 35 ++++++++++++++++++++++++++++++++---
>  3 files changed, 52 insertions(+), 22 deletions(-)
>
> diff --git a/mm/mmap.c b/mm/mmap.c
> index aa4770b8d7f1e..5c9bd3f20e53f 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1883,11 +1883,15 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
>  		if (max) {
>  			vma_iter_set(&vmi, 0);
>  			tmp = vma_next(&vmi);
> +			UNMAP_REGION(unmap, &vmi, /* first vma = */ tmp,
> +				     /* min vma addr = */ 0,
> +				     /* max vma addr = */ max,
> +				     /* prev = */ NULL, /* next = */ NULL);
> +
> +			/* Don't free the pgtables higher than the failure */
> +			unmap.tree_max = max;
>  			flush_cache_mm(mm);
> -			unmap_region(&vmi.mas, /* vma = */ tmp,
> -				     /*vma_min = */ 0, /* vma_max = */ max,
> -				     /* pg_max = */ max, /* prev = */ NULL,
> -				     /* next = */ NULL);
> +			unmap_region(&unmap);
>  			charge = tear_down_vmas(mm, &vmi, tmp, max);
>  			vm_unacct_memory(charge);
>  		}
> diff --git a/mm/vma.c b/mm/vma.c
> index 4c850ffd83a4b..c92384975cbb2 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -473,22 +473,20 @@ void remove_vma(struct vm_area_struct *vma)
>   *
>   * Called with the mm semaphore held.
>   */
> -void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -		unsigned long vma_min, unsigned long vma_max, unsigned long pg_max,
> -		struct vm_area_struct *prev, struct vm_area_struct *next)
> +void unmap_region(struct unmap_desc *desc)
>  {
> -	struct mm_struct *mm = vma->vm_mm;
> +	struct mm_struct *mm = desc->first->vm_mm;
> +	struct ma_state *mas = desc->mas;
>  	struct mmu_gather tlb;
>
>  	tlb_gather_mmu(&tlb, mm);
>  	update_hiwater_rss(mm);
> -	unmap_vmas(&tlb, mas, vma, vma_min, vma_max, vma_max,
> -		   /* mm_wr_locked = */ true);
> -	mas_set(mas, vma->vm_end);
> -	free_pgtables(&tlb, mas, vma, prev ? prev->vm_end : FIRST_USER_ADDRESS,
> -		      next ? next->vm_start : USER_PGTABLES_CEILING,
> -		      pg_max,
> -		      /* mm_wr_locked = */ true);
> +	unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
> +		   desc->vma_max, desc->mm_wr_locked);
> +	mas_set(mas, desc->tree_reset);
> +	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
> +		      desc->last_pgaddr, desc->tree_max,
> +		      desc->mm_wr_locked);
>  	tlb_finish_mmu(&tlb);
>  }
>
> @@ -2414,15 +2412,14 @@ static int __mmap_new_file_vma(struct mmap_state *map,
>
>  	error = mmap_file(vma->vm_file, vma);
>  	if (error) {
> +		UNMAP_REGION(unmap, vmi, vma, vma->vm_start, vma->vm_end,
> +			     map->prev, map->next);
>  		fput(vma->vm_file);
>  		vma->vm_file = NULL;
>
>  		vma_iter_set(vmi, vma->vm_end);
>  		/* Undo any partial mapping done by a device driver. */
> -		unmap_region(&vmi->mas, vma, vma->vm_start, vma->vm_end,
> -			     map->next ? map->next->vm_start : USER_PGTABLES_CEILING,
> -			     map->prev, map->next);
> -
> +		unmap_region(&unmap);
>  		return error;
>  	}
>
> diff --git a/mm/vma.h b/mm/vma.h
> index b0ebc81d5862e..4edd5d26ffcfc 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -152,6 +152,37 @@ struct vma_merge_struct {
>
>  };
>
> +struct unmap_desc {
> +	struct  ma_state *mas;        /* the maple state point to the first vma */
> +	struct vm_area_struct *first; /* The first vma */
> +	unsigned long first_pgaddr;   /* The first pagetable address to free */
> +	unsigned long last_pgaddr;    /* The last pagetable address to free */
> +	unsigned long vma_min;        /* The min vma address */
> +	unsigned long vma_max;        /* The max vma address */
> +	unsigned long tree_max;       /* Maximum for the vma tree search */
> +	unsigned long tree_reset;     /* Where to reset the vma tree walk */
> +	bool mm_wr_locked;            /* If the mmap write lock is held */
> +};

Nice :) comments really help this be self-documenting also.

> +
> +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \

Not to bike shed, but I we already have MMAP_STATE() and VMG_MMAP_STATE() for
instance, so perhaps UNMAP_STATE() for consistency?

> +	struct unmap_desc name = {                                          \
> +		.mas = &(_vmi)->mas,                                          \
> +		.first = _vma,                                                \
> +		.first_pgaddr = _prev ?                                       \
> +			((struct vm_area_struct *)_prev)->vm_end :            \
> +			FIRST_USER_ADDRESS,                                   \
> +		.last_pgaddr = _next ?                                        \
> +			((struct vm_area_struct *)_next)->vm_start :          \
> +			USER_PGTABLES_CEILING,                                \
> +		.vma_min = _vma_min,                                          \
> +		.vma_max = _vma_max,                                          \
> +		.tree_max = _next ?                                           \
> +			((struct vm_area_struct *)_next)->vm_start :          \
> +			USER_PGTABLES_CEILING,                                \
> +		.tree_reset = _vma->vm_end,                                   \
> +		.mm_wr_locked = true,                                         \
> +	}
> +
>  static inline bool vmg_nomem(struct vma_merge_struct *vmg)
>  {
>  	return vmg->state == VMA_MERGE_ERROR_NOMEM;
> @@ -260,9 +291,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm,
>
>  void remove_vma(struct vm_area_struct *vma);
>
> -void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> -		unsigned long min, unsigned long max, unsigned long pg_max,
> -		struct vm_area_struct *prev, struct vm_area_struct *next);
> +void unmap_region(struct unmap_desc *desc);
>
>  /* We are about to modify the VMA's flags. */
>  __must_check struct vm_area_struct
> --
> 2.47.2
>


From david at redhat.com  Thu Sep 11 02:21:03 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 11 Sep 2025 11:21:03 +0200
Subject: [PATCH v1 2/9] mm/mmap: Abstract vma clean up from exit_mmap()
In-Reply-To: <20250909190945.1030905-3-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-3-Liam.Howlett@oracle.com>
Message-ID: <e58b16ea-3018-434e-9eff-eb6cdeef4429@redhat.com>

On 09.09.25 21:09, Liam R. Howlett wrote:
> Create the new function tear_down_vmas() to remove a range of vmas.
> exit_mmap() will be removing all the vmas.
> 
> This is necessary for future patches.
> 
> No functional changes intended.
> 
> Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>   mm/mmap.c | 37 ++++++++++++++++++++++++-------------
>   1 file changed, 24 insertions(+), 13 deletions(-)
> 
> diff --git a/mm/mmap.c b/mm/mmap.c
> index b07b3ec5e28f5..a290448a53bb2 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1250,6 +1250,29 @@ int vm_brk_flags(unsigned long addr, unsigned long request, vm_flags_t vm_flags)
>   }
>   EXPORT_SYMBOL(vm_brk_flags);
>   
> +static inline
> +unsigned long tear_down_vmas(struct mm_struct *mm, struct vma_iterator *vmi,
> +		struct vm_area_struct *vma, unsigned long max)
> +{
> +	unsigned long nr_accounted = 0;
> +	int count = 0;
> +
> +	mmap_assert_write_locked(mm);
> +	vma_iter_set(vmi, vma->vm_end);
> +	do {
> +		if (vma->vm_flags & VM_ACCOUNT)
> +			nr_accounted += vma_pages(vma);
> +		vma_mark_detached(vma);
> +		remove_vma(vma);
> +		count++;
> +		cond_resched();
> +		vma = vma_next(vmi);
> +	} while (vma && vma->vm_end <= max);
> +
> +	WARN_ON_ONCE(count != mm->map_count);

I would just do a VM_WARN_ON_ONCE() here.

Reviewed-by: David Hildenbrand <david at redhat.com>

-- 
Cheers

David / dhildenb



From david at redhat.com  Thu Sep 11 02:22:33 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 11 Sep 2025 11:22:33 +0200
Subject: [PATCH v1 3/9] mm/vma: Add limits to unmap_region() for vmas
In-Reply-To: <20250909190945.1030905-4-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-4-Liam.Howlett@oracle.com>
Message-ID: <59283f53-e249-411f-993b-1ed50f899c42@redhat.com>

On 09.09.25 21:09, Liam R. Howlett wrote:
> Add a limit to the vma search instead of using the start and end of the
> one passed in.
> 
> No functional changes intended.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>   mm/vma.c | 6 ++++--
>   mm/vma.h | 1 +
>   2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vma.c b/mm/vma.c
> index abe0da33c8446..a648e0555c873 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -474,6 +474,7 @@ void remove_vma(struct vm_area_struct *vma)
>    * Called with the mm semaphore held.
>    */
>   void unmap_region(struct ma_state *mas, struct vm_area_struct *vma,
> +		unsigned long vma_min, unsigned long vma_max,

Why not call it "start" and "end" ? :)

In any case

Reviewed-by: David Hildenbrand <david at redhat.com>

-- 
Cheers

David / dhildenb



From lorenzo.stoakes at oracle.com  Thu Sep 11 02:22:36 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 10:22:36 +0100
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce
 function arguments
In-Reply-To: <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com>
 <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>
Message-ID: <dfd8b8c6-d697-4032-be2a-569b9629c564@lucifer.local>

On Tue, Sep 09, 2025 at 02:44:05PM -0700, Suren Baghdasaryan wrote:
> On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
> > +struct unmap_desc {
> > +       struct  ma_state *mas;        /* the maple state point to the first vma */
> > +       struct vm_area_struct *first; /* The first vma */
> > +       unsigned long first_pgaddr;   /* The first pagetable address to free */
> > +       unsigned long last_pgaddr;    /* The last pagetable address to free */
> > +       unsigned long vma_min;        /* The min vma address */
> > +       unsigned long vma_max;        /* The max vma address */
> > +       unsigned long tree_max;       /* Maximum for the vma tree search */
> > +       unsigned long tree_reset;     /* Where to reset the vma tree walk */
> > +       bool mm_wr_locked;            /* If the mmap write lock is held */
> > +};
> > +
> > +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
>
> Maybe DEFINE_UNMAP_REGION() similar to DEFINE_PER_CPU() or DEFINE_SPINLOCK()?

Look at MMAP_STATE(), VMG_MMAP_STATE() for precedent in vma.c


From david at redhat.com  Thu Sep 11 02:24:38 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 11 Sep 2025 11:24:38 +0200
Subject: [PATCH v1 4/9] mm/memory: Add tree limit to free_pgtables()
In-Reply-To: <20250909190945.1030905-5-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-5-Liam.Howlett@oracle.com>
Message-ID: <11f35ae6-2891-44cd-a670-92ef2151b885@redhat.com>

> +/*
> + * free_pgtables() - Free a range of page tables
> + * @tlb: The mmu gather
> + * @mas: The maple state
> + * @vma: The first vma
> + * @floor: The lowest page table address
> + * @ceiling: The highest page table address
> + * @tree_max: The highest tree search address
> + * @mm_wr_locked: boolean indicating if the mm is write locked

I'm sure there is a good reason why we don't call this stuff here

floor -> start
ceiling -> end
tree_max -> tree_end


-- 
Cheers

David / dhildenb



From david at redhat.com  Thu Sep 11 02:31:03 2025
From: david at redhat.com (David Hildenbrand)
Date: Thu, 11 Sep 2025 11:31:03 +0200
Subject: [PATCH v1 6/9] mm: Change dup_mmap() recovery
In-Reply-To: <20250909190945.1030905-7-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-7-Liam.Howlett@oracle.com>
Message-ID: <a05ffc6c-33b3-4e14-92b6-852e8ff37d96@redhat.com>

On 09.09.25 21:09, Liam R. Howlett wrote:
> When the dup_mmap() fails during the vma duplication or setup, don't
> write the XA_ZERO entry in the vma tree.  Instead, destroy the tree and
> free the new resources, leaving an empty vma tree.
> 
> Using XA_ZERO introduced races where the vma could be found between
> dup_mmap() dropping all locks and exit_mmap() taking the locks.  The
> race can occur because the mm can be reached through the other trees
> via successfully copied vmas and other methods such as the swapoff code.
> 
> XA_ZERO was marking the location to stop vma removal and pagetable
> freeing.  The newly created arguments to the unmap_vmas() and
> free_pgtables() serve this function.
> 
> Replacing the XA_ZERO entry use with the new argument list also means
> the checks for xa_is_zero() are no longer necessary so these are also
> removed.
> 
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>   mm/memory.c |  6 +-----
>   mm/mmap.c   | 42 +++++++++++++++++++++++++++++++-----------
>   2 files changed, 32 insertions(+), 16 deletions(-)
> 
> diff --git a/mm/memory.c b/mm/memory.c
> index 24716b3713f66..829cd94950182 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -408,8 +408,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>   		 * be 0.  This will underflow and is okay.
>   		 */
>   		next = mas_find(mas, tree_max - 1);
> -		if (unlikely(xa_is_zero(next)))
> -			next = NULL;
>   
>   		/*
>   		 * Hide vma from rmap and truncate_pagecache before freeing
> @@ -428,8 +426,6 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>   		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>   			vma = next;
>   			next = mas_find(mas, tree_max - 1);
> -			if (unlikely(xa_is_zero(next)))
> -				next = NULL;
>   			if (mm_wr_locked)
>   				vma_start_write(vma);
>   			unlink_anon_vmas(vma);
> @@ -2129,7 +2125,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>   				 mm_wr_locked);
>   		hugetlb_zap_end(vma, &details);
>   		vma = mas_find(mas, tree_end - 1);
> -	} while (vma && likely(!xa_is_zero(vma)));
> +	} while (vma);
>   	mmu_notifier_invalidate_range_end(&range);
>   }
>   
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 0f4808f135fe6..aa4770b8d7f1e 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1288,7 +1288,7 @@ void exit_mmap(struct mm_struct *mm)
>   	arch_exit_mmap(mm);
>   
>   	vma = vma_next(&vmi);
> -	if (!vma || unlikely(xa_is_zero(vma))) {
> +	if (!vma) {
>   		/* Can happen if dup_mmap() received an OOM */
>   		mmap_read_unlock(mm);
>   		mmap_write_lock(mm);
> @@ -1858,20 +1858,40 @@ __latent_entropy int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
>   		ksm_fork(mm, oldmm);
>   		khugepaged_fork(mm, oldmm);
>   	} else {
> +		unsigned long max;
>   
>   		/*
> -		 * The entire maple tree has already been duplicated. If the
> -		 * mmap duplication fails, mark the failure point with
> -		 * XA_ZERO_ENTRY. In exit_mmap(), if this marker is encountered,
> -		 * stop releasing VMAs that have not been duplicated after this
> -		 * point.
> +		 * The entire maple tree has already been duplicated, but
> +		 * replacing the vmas failed at mpnt (which could be NULL if
> +		 * all were allocated but the last vma was not fully set up).
> +		 * Use the start address of the failure point to clean up the
> +		 * partially initialized tree.
>   		 */
> -		if (mpnt) {
> -			mas_set_range(&vmi.mas, mpnt->vm_start, mpnt->vm_end - 1);
> -			mas_store(&vmi.mas, XA_ZERO_ENTRY);
> -			/* Avoid OOM iterating a broken tree */
> -			mm_flags_set(MMF_OOM_SKIP, mm);
> +		if (!mm->map_count) {
> +			/* zero vmas were written to the new tree. */
> +			max = 0;
> +		} else if (mpnt) {
> +			/* partial tree failure */
> +			max = mpnt->vm_start;
> +		} else {
> +			/* All vmas were written to the new tree */
> +			max = ULONG_MAX;
>   		}
> +
> +		/* Hide mm from oom killer because the memory is being freed */
> +		mm_flags_set(MMF_OOM_SKIP, mm);
> +		if (max) {
> +			vma_iter_set(&vmi, 0);
> +			tmp = vma_next(&vmi);
> +			flush_cache_mm(mm);
> +			unmap_region(&vmi.mas, /* vma = */ tmp,
> +				     /*vma_min = */ 0, /* vma_max = */ max,
> +				     /* pg_max = */ max, /* prev = */ NULL,
> +				     /* next = */ NULL);
> +			charge = tear_down_vmas(mm, &vmi, tmp, max);
> +			vm_unacct_memory(charge);
> +		}
> +		__mt_destroy(&mm->mm_mt);

Usually comment about just calling things start/end, maybe with prefix 
if required.

Apart from that, LGTM.

-- 
Cheers

David / dhildenb



From lorenzo.stoakes at oracle.com  Thu Sep 11 02:53:13 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 10:53:13 +0100
Subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and
 exit_mmap()
In-Reply-To: <20250909190945.1030905-9-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-9-Liam.Howlett@oracle.com>
Message-ID: <35d63781-ea17-4472-87ec-541b6c8e0ecf@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:44PM -0400, Liam R. Howlett wrote:
> vms_clear_ptes() is slightly different than other callers to
> unmap_region() and so had the unmapping open-coded.  Using the new
> structure it is now possible to special-case the struct setup instead of
> having the open-coded function.
>
> exit_mmap() also calls unmap_vmas() with many arguemnts.  Using the
> unmap_all_init() function to set the unmap descriptor for all vmas makes
> this a bit easier to read.
>
> Update to the vma test code is necessary to ensure testing continues to
> function.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> ---
>  include/linux/mm.h               |  3 ---
>  mm/internal.h                    |  3 +++
>  mm/memory.c                      | 24 ++++++++------------
>  mm/mmap.c                        |  5 +++-
>  mm/vma.c                         | 39 ++++++++++++++++++--------------
>  mm/vma.h                         | 14 ++++++++++++
>  tools/testing/vma/vma_internal.h | 14 ++++--------
>  7 files changed, 56 insertions(+), 46 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index 892fe5dbf9de0..23eb59d543390 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -2450,9 +2450,6 @@ static inline void zap_vma_pages(struct vm_area_struct *vma)
>  	zap_page_range_single(vma, vma->vm_start,
>  			      vma->vm_end - vma->vm_start, NULL);
>  }
> -void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -		struct vm_area_struct *start_vma, unsigned long start,
> -		unsigned long end, unsigned long tree_end, bool mm_wr_locked);
>
>  struct mmu_notifier_range;
>
> diff --git a/mm/internal.h b/mm/internal.h
> index d295252407fee..1239944f2830a 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -197,6 +197,9 @@ static inline void vma_close(struct vm_area_struct *vma)
>  	}
>  }
>
> +/* unmap_vmas is in mm/memory.c */
> +void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);
> +
>  #ifdef CONFIG_MMU
>
>  /* Flags for folio_pte_batch(). */
> diff --git a/mm/memory.c b/mm/memory.c
> index 829cd94950182..8d4d976311037 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2084,12 +2084,7 @@ static void unmap_single_vma(struct mmu_gather *tlb,
>  /**
>   * unmap_vmas - unmap a range of memory covered by a list of vma's
>   * @tlb: address of the caller's struct mmu_gather
> - * @mas: the maple state
> - * @vma: the starting vma
> - * @start_addr: virtual address at which to start unmapping
> - * @end_addr: virtual address at which to end unmapping
> - * @tree_end: The maximum index to check
> - * @mm_wr_locked: lock flag
> + * @unmap: The unmap_desc
>   *
>   * Unmap all pages in the vma list.
>   *
> @@ -2102,11 +2097,9 @@ static void unmap_single_vma(struct mmu_gather *tlb,
>   * ensure that any thus-far unmapped pages are flushed before unmap_vmas()
>   * drops the lock and schedules.
>   */
> -void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -		struct vm_area_struct *vma, unsigned long start_addr,
> -		unsigned long end_addr, unsigned long tree_end,
> -		bool mm_wr_locked)
> +void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>  {
> +	struct vm_area_struct *vma;
>  	struct mmu_notifier_range range;
>  	struct zap_details details = {
>  		.zap_flags = ZAP_FLAG_DROP_MARKER | ZAP_FLAG_UNMAP,
> @@ -2114,17 +2107,18 @@ void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
>  		.even_cows = true,
>  	};
>
> +	vma = unmap->first;
>  	mmu_notifier_range_init(&range, MMU_NOTIFY_UNMAP, 0, vma->vm_mm,
> -				start_addr, end_addr);
> +				unmap->vma_min, unmap->vma_max);
>  	mmu_notifier_invalidate_range_start(&range);
>  	do {
> -		unsigned long start = start_addr;
> -		unsigned long end = end_addr;
> +		unsigned long start = unmap->vma_min;
> +		unsigned long end = unmap->vma_max;
>  		hugetlb_zap_begin(vma, &start, &end);
>  		unmap_single_vma(tlb, vma, start, end, &details,
> -				 mm_wr_locked);
> +				 unmap->mm_wr_locked);
>  		hugetlb_zap_end(vma, &details);
> -		vma = mas_find(mas, tree_end - 1);
> +		vma = mas_find(unmap->mas, unmap->tree_max - 1);

An aside, but do kinda see David's point about min, max & start, end & floor
ceililng, perhaps the prefix_ is enough to differentiate these and we could be
as consistent as we can be?

>  	} while (vma);
>  	mmu_notifier_invalidate_range_end(&range);
>  }
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 5c9bd3f20e53f..6011f62b0a294 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1280,10 +1280,12 @@ void exit_mmap(struct mm_struct *mm)
>  	struct vm_area_struct *vma;
>  	unsigned long nr_accounted = 0;
>  	VMA_ITERATOR(vmi, mm, 0);
> +	struct unmap_desc unmap;

Perhaps :

	struct unmap_desc unnmap = {
		.mm_wr_locked = false,
	};

To be clear? I mean it does mean we're initialising some fields twice
though... but at least avoids uninitialised state if we add new fields.

However I see you're already seeing mm_wr_locked to false in
unmap_all_init() anyway?

Ideally we'd have something like UNMAP_STATE_VMA() here, but ofc you don't
have the VMA yet.

So probably something like what you've done here is the way to go, but I'd
rather we initialise this var in case of adding fields later.

Then again... you would have to make sure the _init() function set all
fields anyway so maybe not necessary. Hmm.

>
>  	/* mm's last user has gone, and its about to be pulled down */
>  	mmu_notifier_release(mm);
>
> +	unmap.mm_wr_locked = false;
>  	mmap_read_lock(mm);
>  	arch_exit_mmap(mm);
>
> @@ -1295,11 +1297,12 @@ void exit_mmap(struct mm_struct *mm)
>  		goto destroy;
>  	}
>
> +	unmap_all_init(&unmap, &vmi, vma);
>  	flush_cache_mm(mm);
>  	tlb_gather_mmu_fullmm(&tlb, mm);
>  	/* update_hiwater_rss(mm) here? but nobody should be looking */
>  	/* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */
> -	unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false);
> +	unmap_vmas(&tlb, &unmap);
>  	mmap_read_unlock(mm);
>
>  	/*
> diff --git a/mm/vma.c b/mm/vma.c
> index c92384975cbb2..ad64cd9795ef3 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -481,8 +481,7 @@ void unmap_region(struct unmap_desc *desc)
>
>  	tlb_gather_mmu(&tlb, mm);
>  	update_hiwater_rss(mm);
> -	unmap_vmas(&tlb, mas, desc->first, desc->vma_min, desc->vma_max,
> -		   desc->vma_max, desc->mm_wr_locked);
> +	unmap_vmas(&tlb, desc);

Nit, perhaps didn't notice on previous commit actually, but you're being
inconsistent between naming this 'desc' and 'unmap'. Let's choose one and
stick with it.

>  	mas_set(mas, desc->tree_reset);
>  	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
>  		      desc->last_pgaddr, desc->tree_max,
> @@ -1213,26 +1212,32 @@ int vma_shrink(struct vma_iterator *vmi, struct vm_area_struct *vma,
>  static inline void vms_clear_ptes(struct vma_munmap_struct *vms,
>  		    struct ma_state *mas_detach, bool mm_wr_locked)
>  {
> -	struct mmu_gather tlb;
> +	struct unmap_desc unmap = {
> +		.mas = mas_detach,
> +		.first = vms->vma,
> +		/* start and end may be different if there is no prev or next vma. */
> +		.first_pgaddr = vms->unmap_start,
> +		.last_pgaddr = vms->unmap_end,
> +		.vma_min = vms->start,
> +		.vma_max = vms->end,
> +		/*
> +		 * The tree limits and reset differ from the normal case since it's a
> +		 * side-tree
> +		 */
> +		.tree_reset = 1,
> +		.tree_max = vms->vma_count,
> +		/*
> +		 * We can free page tables without write-locking mmap_lock because VMAs
> +		 * were isolated before we downgraded mmap_lock.
> +		 */
> +		.mm_wr_locked = mm_wr_locked,

These comments are great thanks! :)

> +	};
>
>  	if (!vms->clear_ptes) /* Nothing to do */
>  		return;
>
> -	/*
> -	 * We can free page tables without write-locking mmap_lock because VMAs
> -	 * were isolated before we downgraded mmap_lock.
> -	 */
>  	mas_set(mas_detach, 1);
> -	tlb_gather_mmu(&tlb, vms->vma->vm_mm);
> -	update_hiwater_rss(vms->vma->vm_mm);
> -	unmap_vmas(&tlb, mas_detach, vms->vma, vms->start, vms->end,
> -		   vms->vma_count, mm_wr_locked);
> -
> -	mas_set(mas_detach, 1);
> -	/* start and end may be different if there is no prev or next vma. */
> -	free_pgtables(&tlb, mas_detach, vms->vma, vms->unmap_start,
> -		      vms->unmap_end, vms->unmap_end, mm_wr_locked);
> -	tlb_finish_mmu(&tlb);
> +	unmap_region(&unmap);

HMmm why are we removing all these? I'm guessing this is a separate
refactoring, could we perhaps break this out, as it doesn't seem to quite
belong in this patch?

I mean this is really nice :) just I think belongs in another patch, unless
you feel it's really tied to this one/necessary here?

Sorry to be a pain... :)

>  	vms->clear_ptes = false;
>  }
>
> diff --git a/mm/vma.h b/mm/vma.h
> index 4edd5d26ffcfc..8b55a0c73d097 100644
> --- a/mm/vma.h
> +++ b/mm/vma.h
> @@ -164,6 +164,20 @@ struct unmap_desc {
>  	bool mm_wr_locked;            /* If the mmap write lock is held */
>  };
>
> +static inline void unmap_all_init(struct unmap_desc *desc,
> +		struct vma_iterator *vmi, struct vm_area_struct *vma)
> +{
> +	desc->mas = &vmi->mas;
> +	desc->first = vma;
> +	desc->first_pgaddr = FIRST_USER_ADDRESS;
> +	desc->last_pgaddr = USER_PGTABLES_CEILING;
> +	desc->vma_min = 0;
> +	desc->vma_max = ULONG_MAX;
> +	desc->tree_max = ULONG_MAX;
> +	desc->tree_reset = vma->vm_end;
> +	desc->mm_wr_locked = false;
> +}
> +
>  #define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
>  	struct unmap_desc name = {                                          \
>  		.mas = &(_vmi)->mas,                                          \
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index 823d379e1fac2..d73ad4747d40a 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -884,18 +884,12 @@ static inline void update_hiwater_vm(struct mm_struct *)
>  {
>  }
>
> -static inline void unmap_vmas(struct mmu_gather *tlb, struct ma_state *mas,
> -		      struct vm_area_struct *vma, unsigned long start_addr,
> -		      unsigned long end_addr, unsigned long tree_end,
> -		      bool mm_wr_locked)
> +struct unmap_desc;
> +
> +static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>  {
>  	(void)tlb;
> -	(void)mas;
> -	(void)vma;
> -	(void)start_addr;
> -	(void)end_addr;
> -	(void)tree_end;
> -	(void)mm_wr_locked;
> +	(void)unmap;
>  }

Hmm why is this still here, I thought I'd removed all these (void)'s... I think
actually that's what's in Vlasta's tree, so this will conflict unfortunately.

Anwyay, please just remove all these (void)'s, they're unnecessary.

(Also I'm sorry for making updating this file a thing, but I'm not sure there's
a better way)

>
>  static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> --
> 2.47.2
>


From lorenzo.stoakes at oracle.com  Thu Sep 11 03:06:29 2025
From: lorenzo.stoakes at oracle.com (Lorenzo Stoakes)
Date: Thu, 11 Sep 2025 11:06:29 +0100
Subject: [PATCH v1 9/9] mm: Use unmap_desc struct for freeing page tables.
In-Reply-To: <20250909190945.1030905-10-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-10-Liam.Howlett@oracle.com>
Message-ID: <2157a865-5481-43fd-8139-4b9662239eb4@lucifer.local>

On Tue, Sep 09, 2025 at 03:09:45PM -0400, Liam R. Howlett wrote:
> Pass through the unmap_desc to free_pgtables() because it almost has
> everything necessary and is already on the stack.
>
> Updates testing code as necessary.
>
> No functional changes intended.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>

To be clear, I _love_ the use of this helper struct. All commentary is more
so details. what you're doing here is great... :)

Other than nits below this looks fine so on basis of addressing them:

Reviewed-by: Lorenzo Stoakes <lorenzo.stoakes at oracle.com>

> ---
>  mm/internal.h                    |  5 +----
>  mm/memory.c                      | 21 ++++++++++-----------
>  mm/mmap.c                        |  6 +++---
>  mm/vma.c                         |  7 ++-----
>  tools/testing/vma/vma_internal.h | 11 ++---------
>  5 files changed, 18 insertions(+), 32 deletions(-)
>
> diff --git a/mm/internal.h b/mm/internal.h
> index 1239944f2830a..f22329967e908 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -445,10 +445,7 @@ bool __folio_end_writeback(struct folio *folio);
>  void deactivate_file_folio(struct folio *folio);
>  void folio_activate(struct folio *folio);
>
> -void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -		   struct vm_area_struct *start_vma, unsigned long floor,
> -		   unsigned long ceiling, unsigned long tree_max,
> -		   bool mm_wr_locked);
> +void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc);
>
>  void pmd_install(struct mm_struct *mm, pmd_t *pmd, pgtable_t *pte);
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 8d4d976311037..98c5ffd28a109 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -387,15 +387,14 @@ void free_pgd_range(struct mmu_gather *tlb,
>   * The tree_max differs from the ceiling when a dup_mmap() failed and the tree
>   * has unrelated data to the mm_struct being torn down.
>   */
> -void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -		   struct vm_area_struct *vma, unsigned long floor,
> -		   unsigned long ceiling, unsigned long tree_max,
> -		   bool mm_wr_locked)
> +void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)

God this is nice.

>  {
>  	struct unlink_vma_file_batch vb;
> +	struct ma_state *mas = desc->mas;
> +	struct vm_area_struct *vma = desc->first;
>
>  	/* underflow can happen and is fine */
> -	WARN_ON_ONCE(tree_max - 1 > ceiling - 1);
> +	WARN_ON_ONCE(desc->tree_max - 1 > desc->last_pgaddr - 1);
>
>  	tlb_free_vmas(tlb);
>
> @@ -407,13 +406,13 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		 * Note: USER_PGTABLES_CEILING may be passed as ceiling and may
>  		 * be 0.  This will underflow and is okay.
>  		 */
> -		next = mas_find(mas, tree_max - 1);
> +		next = mas_find(mas, desc->tree_max - 1);
>
>  		/*
>  		 * Hide vma from rmap and truncate_pagecache before freeing
>  		 * pgtables
>  		 */
> -		if (mm_wr_locked)
> +		if (desc->mm_wr_locked)
>  			vma_start_write(vma);
>  		unlink_anon_vmas(vma);
>
> @@ -425,16 +424,16 @@ void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
>  		 */
>  		while (next && next->vm_start <= vma->vm_end + PMD_SIZE) {
>  			vma = next;
> -			next = mas_find(mas, tree_max - 1);
> -			if (mm_wr_locked)
> +			next = mas_find(mas, desc->tree_max - 1);
> +			if (desc->mm_wr_locked)
>  				vma_start_write(vma);
>  			unlink_anon_vmas(vma);
>  			unlink_file_vma_batch_add(&vb, vma);
>  		}
>  		unlink_file_vma_batch_final(&vb);
>
> -		free_pgd_range(tlb, addr, vma->vm_end,
> -			floor, next ? next->vm_start : ceiling);
> +		free_pgd_range(tlb, addr, vma->vm_end, desc->first_pgaddr,
> +			       next ? next->vm_start : desc->last_pgaddr);

Hm, actually, isn't ceiling exclusive? So 'last_pgaddr' seems a bit
misleading?

>  		vma = next;
>  	} while (vma);
>  }
> diff --git a/mm/mmap.c b/mm/mmap.c
> index 6011f62b0a294..9908481452780 100644
> --- a/mm/mmap.c
> +++ b/mm/mmap.c
> @@ -1311,10 +1311,10 @@ void exit_mmap(struct mm_struct *mm)
>  	 */
>  	mm_flags_set(MMF_OOM_SKIP, mm);
>  	mmap_write_lock(mm);
> +	unmap.mm_wr_locked = true;
>  	mt_clear_in_rcu(&mm->mm_mt);
> -	vma_iter_set(&vmi, vma->vm_end);
> -	free_pgtables(&tlb, &vmi.mas, vma, FIRST_USER_ADDRESS,
> -		      USER_PGTABLES_CEILING, USER_PGTABLES_CEILING, true);
> +	vma_iter_set(&vmi, unmap.tree_reset);
> +	free_pgtables(&tlb, &unmap);

Lovely!

>  	tlb_finish_mmu(&tlb);
>
>  	/*
> diff --git a/mm/vma.c b/mm/vma.c
> index ad64cd9795ef3..ba155a539d160 100644
> --- a/mm/vma.c
> +++ b/mm/vma.c
> @@ -476,16 +476,13 @@ void remove_vma(struct vm_area_struct *vma)
>  void unmap_region(struct unmap_desc *desc)
>  {
>  	struct mm_struct *mm = desc->first->vm_mm;
> -	struct ma_state *mas = desc->mas;
>  	struct mmu_gather tlb;
>
>  	tlb_gather_mmu(&tlb, mm);
>  	update_hiwater_rss(mm);
>  	unmap_vmas(&tlb, desc);
> -	mas_set(mas, desc->tree_reset);
> -	free_pgtables(&tlb, mas, desc->first, desc->first_pgaddr,
> -		      desc->last_pgaddr, desc->tree_max,
> -		      desc->mm_wr_locked);
> +	mas_set(desc->mas, desc->tree_reset);
> +	free_pgtables(&tlb, desc);
>  	tlb_finish_mmu(&tlb);
>  }
>
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index d73ad4747d40a..435c5a24480bc 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -892,17 +892,10 @@ static inline void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap)
>  	(void)unmap;
>  }
>
> -static inline void free_pgtables(struct mmu_gather *tlb, struct ma_state *mas,
> -		   struct vm_area_struct *vma, unsigned long floor,
> -		   unsigned long ceiling, unsigned long tree_max,
> -		   bool mm_wr_locked)
> +static inline void free_pgtables(struct mmu_gather *tlb, struct unmap_desc *desc)
>  {
>  	(void)tlb;
> -	(void)mas;
> -	(void)vma;
> -	(void)floor;
> -	(void)ceiling;
> -	(void)mm_wr_locked;
> +	(void)desc;
>  }

Again, let's drop the (void)'s here. Will let Vlasta fix up the conflict I
guess ;)

>
>  static inline void mapping_unmap_writable(struct address_space *)
> --
> 2.47.2
>


From surenb at google.com  Thu Sep 11 09:51:33 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 11 Sep 2025 09:51:33 -0700
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce function arguments
In-Reply-To: <dfd8b8c6-d697-4032-be2a-569b9629c564@lucifer.local>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com> <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>
 <dfd8b8c6-d697-4032-be2a-569b9629c564@lucifer.local>
Message-ID: <CAJuCfpGFXbSFnqQtikNq0RoMfc93cLLnqrApq8211H+FpZpJ1Q@mail.gmail.com>

On Thu, Sep 11, 2025 at 2:22?AM Lorenzo Stoakes
<lorenzo.stoakes at oracle.com> wrote:
>
> On Tue, Sep 09, 2025 at 02:44:05PM -0700, Suren Baghdasaryan wrote:
> > On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
> > > +struct unmap_desc {
> > > +       struct  ma_state *mas;        /* the maple state point to the first vma */
> > > +       struct vm_area_struct *first; /* The first vma */
> > > +       unsigned long first_pgaddr;   /* The first pagetable address to free */
> > > +       unsigned long last_pgaddr;    /* The last pagetable address to free */
> > > +       unsigned long vma_min;        /* The min vma address */
> > > +       unsigned long vma_max;        /* The max vma address */
> > > +       unsigned long tree_max;       /* Maximum for the vma tree search */
> > > +       unsigned long tree_reset;     /* Where to reset the vma tree walk */
> > > +       bool mm_wr_locked;            /* If the mmap write lock is held */
> > > +};
> > > +
> > > +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
> >
> > Maybe DEFINE_UNMAP_REGION() similar to DEFINE_PER_CPU() or DEFINE_SPINLOCK()?
>
> Look at MMAP_STATE(), VMG_MMAP_STATE() for precedent in vma.c

Yeah but UNMAP_REGION() sounds like an action while MMAP_STATE(),
VMG_MMAP_STATE() do not. Anyway, whatever works I guess.


From Liam.Howlett at oracle.com  Thu Sep 11 09:56:29 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 11 Sep 2025 12:56:29 -0400
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce
 function arguments
In-Reply-To: <CAJuCfpGFXbSFnqQtikNq0RoMfc93cLLnqrApq8211H+FpZpJ1Q@mail.gmail.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com>
 <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>
 <dfd8b8c6-d697-4032-be2a-569b9629c564@lucifer.local>
 <CAJuCfpGFXbSFnqQtikNq0RoMfc93cLLnqrApq8211H+FpZpJ1Q@mail.gmail.com>
Message-ID: <aw6vbshqvskf6lwql354dc7xn4dh445uaadbqoydzai3rnbbc7@j2lm7ryywi4o>

* Suren Baghdasaryan <surenb at google.com> [250911 12:51]:
> On Thu, Sep 11, 2025 at 2:22?AM Lorenzo Stoakes
> <lorenzo.stoakes at oracle.com> wrote:
> >
> > On Tue, Sep 09, 2025 at 02:44:05PM -0700, Suren Baghdasaryan wrote:
> > > On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
> > > > +struct unmap_desc {
> > > > +       struct  ma_state *mas;        /* the maple state point to the first vma */
> > > > +       struct vm_area_struct *first; /* The first vma */
> > > > +       unsigned long first_pgaddr;   /* The first pagetable address to free */
> > > > +       unsigned long last_pgaddr;    /* The last pagetable address to free */
> > > > +       unsigned long vma_min;        /* The min vma address */
> > > > +       unsigned long vma_max;        /* The max vma address */
> > > > +       unsigned long tree_max;       /* Maximum for the vma tree search */
> > > > +       unsigned long tree_reset;     /* Where to reset the vma tree walk */
> > > > +       bool mm_wr_locked;            /* If the mmap write lock is held */
> > > > +};
> > > > +
> > > > +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
> > >
> > > Maybe DEFINE_UNMAP_REGION() similar to DEFINE_PER_CPU() or DEFINE_SPINLOCK()?
> >
> > Look at MMAP_STATE(), VMG_MMAP_STATE() for precedent in vma.c
> 
> Yeah but UNMAP_REGION() sounds like an action while MMAP_STATE(),
> VMG_MMAP_STATE() do not. Anyway, whatever works I guess.

Is UNMAP_STATE() okay?


From Liam.Howlett at oracle.com  Thu Sep 11 09:59:28 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 11 Sep 2025 12:59:28 -0400
Subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and
 exit_mmap()
In-Reply-To: <CAJuCfpFGo4JXQQGECFdCgNNVaQc5MDLghdmnVXEK8Zfag7KHiw@mail.gmail.com>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-9-Liam.Howlett@oracle.com>
 <CAJuCfpFGo4JXQQGECFdCgNNVaQc5MDLghdmnVXEK8Zfag7KHiw@mail.gmail.com>
Message-ID: <2slsnkd6abablb2zxvzti5w6e2sdacz2z7uzwvumodg4ccuykh@ul45isfoax3f>

* Suren Baghdasaryan <surenb at google.com> [250909 18:16]:
> On Tue, Sep 9, 2025 at 12:10?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
> >
> > vms_clear_ptes() is slightly different than other callers to
> > unmap_region() and so had the unmapping open-coded.  Using the new
> > structure it is now possible to special-case the struct setup instead of
> > having the open-coded function.
> >
> > exit_mmap() also calls unmap_vmas() with many arguemnts.  Using the
> > unmap_all_init() function to set the unmap descriptor for all vmas makes
> > this a bit easier to read.
> >
> > Update to the vma test code is necessary to ensure testing continues to
> > function.
> >
> > No functional changes intended.
> >
> > Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> > ---
> >  include/linux/mm.h               |  3 ---
> >  mm/internal.h                    |  3 +++
> >  mm/memory.c                      | 24 ++++++++------------
> >  mm/mmap.c                        |  5 +++-
> >  mm/vma.c                         | 39 ++++++++++++++++++--------------
> >  mm/vma.h                         | 14 ++++++++++++
> >  tools/testing/vma/vma_internal.h | 14 ++++--------
> >  7 files changed, 56 insertions(+), 46 deletions(-)

...

> >         struct vm_area_struct *vma;
> >         unsigned long nr_accounted = 0;
> >         VMA_ITERATOR(vmi, mm, 0);
> > +       struct unmap_desc unmap;
> >
> >         /* mm's last user has gone, and its about to be pulled down */
> >         mmu_notifier_release(mm);
> >
> > +       unmap.mm_wr_locked = false;
> 
> This will be reset by unmap_all_init() anyway, right?

Yes, I will drop that.  Thanks, I missed this when I rewrote to use a
different function.

> 
> >         mmap_read_lock(mm);
> >         arch_exit_mmap(mm);
> >
> > @@ -1295,11 +1297,12 @@ void exit_mmap(struct mm_struct *mm)
> >                 goto destroy;
> >         }
> >
> > +       unmap_all_init(&unmap, &vmi, vma);
> 
> Can we use a macro, something like DEFINE_UNMAP_ALL_REGIONS() instead
> of unmap_all_init()?

No, because the vma is unknown and we set up some of the unmap_desc from
the values in vma.

> 
> >         flush_cache_mm(mm);
> >         tlb_gather_mmu_fullmm(&tlb, mm);
> >         /* update_hiwater_rss(mm) here? but nobody should be looking */
> >         /* Use ULONG_MAX here to ensure all VMAs in the mm are unmapped */
> > -       unmap_vmas(&tlb, &vmi.mas, vma, 0, ULONG_MAX, ULONG_MAX, false);
> > +       unmap_vmas(&tlb, &unmap);
> >         mmap_read_unlock(mm);

...

Thanks,
Liam


From surenb at google.com  Thu Sep 11 10:03:50 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 11 Sep 2025 10:03:50 -0700
Subject: [PATCH v1 7/9] mm: Introduce unmap_desc struct to reduce function arguments
In-Reply-To: <aw6vbshqvskf6lwql354dc7xn4dh445uaadbqoydzai3rnbbc7@j2lm7ryywi4o>
References: <20250909190945.1030905-1-Liam.Howlett@oracle.com>
 <20250909190945.1030905-8-Liam.Howlett@oracle.com> <CAJuCfpG4D4ikZO1c8zN7HNgLTAco1ggk21rg9AUFwoztA95qSA@mail.gmail.com>
 <dfd8b8c6-d697-4032-be2a-569b9629c564@lucifer.local> <CAJuCfpGFXbSFnqQtikNq0RoMfc93cLLnqrApq8211H+FpZpJ1Q@mail.gmail.com>
 <aw6vbshqvskf6lwql354dc7xn4dh445uaadbqoydzai3rnbbc7@j2lm7ryywi4o>
Message-ID: <CAJuCfpEuuS+OnARDNQVcK4WnYkWJ-3t5woCN4QMOK86OSUSAog@mail.gmail.com>

On Thu, Sep 11, 2025 at 9:56?AM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
>
> * Suren Baghdasaryan <surenb at google.com> [250911 12:51]:
> > On Thu, Sep 11, 2025 at 2:22?AM Lorenzo Stoakes
> > <lorenzo.stoakes at oracle.com> wrote:
> > >
> > > On Tue, Sep 09, 2025 at 02:44:05PM -0700, Suren Baghdasaryan wrote:
> > > > On Tue, Sep 9, 2025 at 12:11?PM Liam R. Howlett <Liam.Howlett at oracle.com> wrote:
> > > > > +struct unmap_desc {
> > > > > +       struct  ma_state *mas;        /* the maple state point to the first vma */
> > > > > +       struct vm_area_struct *first; /* The first vma */
> > > > > +       unsigned long first_pgaddr;   /* The first pagetable address to free */
> > > > > +       unsigned long last_pgaddr;    /* The last pagetable address to free */
> > > > > +       unsigned long vma_min;        /* The min vma address */
> > > > > +       unsigned long vma_max;        /* The max vma address */
> > > > > +       unsigned long tree_max;       /* Maximum for the vma tree search */
> > > > > +       unsigned long tree_reset;     /* Where to reset the vma tree walk */
> > > > > +       bool mm_wr_locked;            /* If the mmap write lock is held */
> > > > > +};
> > > > > +
> > > > > +#define UNMAP_REGION(name, _vmi, _vma, _vma_min, _vma_max, _prev, _next)      \
> > > >
> > > > Maybe DEFINE_UNMAP_REGION() similar to DEFINE_PER_CPU() or DEFINE_SPINLOCK()?
> > >
> > > Look at MMAP_STATE(), VMG_MMAP_STATE() for precedent in vma.c
> >
> > Yeah but UNMAP_REGION() sounds like an action while MMAP_STATE(),
> > VMG_MMAP_STATE() do not. Anyway, whatever works I guess.
>
> Is UNMAP_STATE() okay?

I think so.


From senozhatsky at chromium.org  Thu Sep 11 17:38:58 2025
From: senozhatsky at chromium.org (Sergey Senozhatsky)
Date: Fri, 12 Sep 2025 09:38:58 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
Message-ID: <iycnfietmsgggxcdjuqngjuzbuygsxfpvkcz2fwm4frdxin42u@pxhmqcgejnov>

Hi Vlastimil,

On (25/09/10 10:01), Vlastimil Babka wrote:
[..]
> +
> +	if (rcu_free)
> +		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> +}
> +
> +
> +/* needed for kvfree_rcu_barrier() */
> +void flush_all_rcu_sheaves()
> +{

mm/slub.c:3960:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
 3960 | void flush_all_rcu_sheaves()
      |                           ^
      |                            void

---

diff --git a/mm/slub.c b/mm/slub.c
index 11ad4173e2f2..a1eae71a0f8c 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3955,9 +3955,8 @@ static void flush_rcu_sheaf(struct work_struct *w)
 		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
 }
 
-
 /* needed for kvfree_rcu_barrier() */
-void flush_all_rcu_sheaves()
+void flush_all_rcu_sheaves(void)
 {
 	struct slub_percpu_sheaves *pcs;
 	struct slub_flush_work *sfw;


From Liam.Howlett at oracle.com  Thu Sep 11 19:20:54 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 11 Sep 2025 22:20:54 -0400
Subject: [PATCH v8 15/23] maple_tree: use percpu sheaves for
 maple_node_cache
In-Reply-To: <20250910-slub-percpu-caches-v8-15-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-15-ca3099d8352c@suse.cz>
Message-ID: <hc3kari6hxfw6coyl24v56ynn4djvpzrln4eltctiwkafv47my@rbv4jxquriis>

* Vlastimil Babka <vbabka at suse.cz> [250910 04:02]:
> Setup the maple_node_cache with percpu sheaves of size 32 to hopefully
> improve its performance. Note this will not immediately take advantage
> of sheaf batching of kfree_rcu() operations due to the maple tree using
> call_rcu with custom callbacks. The followup changes to maple tree will
> change that and also make use of the prefilled sheaves functionality.
> 
> Reviewed-by: Sidhartha Kumar <sidhartha.kumar at oracle.com>
> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Liam R. Howlett <Liam.Howlett at oracle.com>

> ---
>  lib/maple_tree.c | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index 4f0e30b57b0cef9e5cf791f3f64f5898752db402..d034f170ac897341b40cfd050b6aee86b6d2cf60 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -6040,9 +6040,14 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
>  
>  void __init maple_tree_init(void)
>  {
> +	struct kmem_cache_args args = {
> +		.align  = sizeof(struct maple_node),
> +		.sheaf_capacity = 32,
> +	};
> +
>  	maple_node_cache = kmem_cache_create("maple_node",
> -			sizeof(struct maple_node), sizeof(struct maple_node),
> -			SLAB_PANIC, NULL);
> +			sizeof(struct maple_node), &args,
> +			SLAB_PANIC);
>  }
>  
>  /**
> 
> -- 
> 2.51.0
> 
> 
> -- 
> maple-tree mailing list
> maple-tree at lists.infradead.org
> https://lists.infradead.org/mailman/listinfo/maple-tree


From lkp at intel.com  Thu Sep 11 22:08:46 2025
From: lkp at intel.com (kernel test robot)
Date: Fri, 12 Sep 2025 13:08:46 +0800
Subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and
 exit_mmap()
In-Reply-To: <20250909190945.1030905-9-Liam.Howlett@oracle.com>
References: <20250909190945.1030905-9-Liam.Howlett@oracle.com>
Message-ID: <202509121212.sx5Tfe4e-lkp@intel.com>

Hi Liam,

kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]
[cannot apply to linus/master v6.17-rc5 next-20250911]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Liam-R-Howlett/mm-mmap-Move-exit_mmap-trace-point/20250910-031555
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20250909190945.1030905-9-Liam.Howlett%40oracle.com
patch subject: [PATCH v1 8/9] mm/vma: Use unmap_desc in vms_clear_ptes() and exit_mmap()
config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20250912/202509121212.sx5Tfe4e-lkp at intel.com/config)
compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 21857ae337e0892a5522b6e7337899caa61de2a6)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250912/202509121212.sx5Tfe4e-lkp at intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp at intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202509121212.sx5Tfe4e-lkp at intel.com/

All errors (new ones prefixed by >>):

   In file included from mm/filemap.c:53:
   In file included from mm/internal.h:23:
>> mm/vma.h:173:22: error: use of undeclared identifier 'USER_PGTABLES_CEILING'
     173 |         desc->last_pgaddr = USER_PGTABLES_CEILING;
         |                             ^~~~~~~~~~~~~~~~~~~~~
   In file included from mm/filemap.c:65:
   mm/swap.h:461:10: error: call to undeclared function 'swp_offset'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     461 |                 return swp_offset(folio->swap);
         |                        ^
   2 errors generated.
--
   In file included from mm/oom_kill.c:50:
   In file included from mm/internal.h:23:
>> mm/vma.h:173:22: error: use of undeclared identifier 'USER_PGTABLES_CEILING'
     173 |         desc->last_pgaddr = USER_PGTABLES_CEILING;
         |                             ^~~~~~~~~~~~~~~~~~~~~
   1 error generated.


vim +/USER_PGTABLES_CEILING +173 mm/vma.h

   166	
   167	static inline void unmap_all_init(struct unmap_desc *desc,
   168			struct vma_iterator *vmi, struct vm_area_struct *vma)
   169	{
   170		desc->mas = &vmi->mas;
   171		desc->first = vma;
   172		desc->first_pgaddr = FIRST_USER_ADDRESS;
 > 173		desc->last_pgaddr = USER_PGTABLES_CEILING;
   174		desc->vma_min = 0;
   175		desc->vma_max = ULONG_MAX;
   176		desc->tree_max = ULONG_MAX;
   177		desc->tree_reset = vma->vm_end;
   178		desc->mm_wr_locked = false;
   179	}
   180	

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


From vbabka at suse.cz  Fri Sep 12 00:03:55 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Fri, 12 Sep 2025 09:03:55 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <iycnfietmsgggxcdjuqngjuzbuygsxfpvkcz2fwm4frdxin42u@pxhmqcgejnov>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <iycnfietmsgggxcdjuqngjuzbuygsxfpvkcz2fwm4frdxin42u@pxhmqcgejnov>
Message-ID: <298f6a92-7bf0-4cf4-afbf-554ef3e8e7e7@suse.cz>

On 9/12/25 02:38, Sergey Senozhatsky wrote:
> Hi Vlastimil,
> 
> On (25/09/10 10:01), Vlastimil Babka wrote:
> [..]
>> +
>> +	if (rcu_free)
>> +		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
>> +}
>> +
>> +
>> +/* needed for kvfree_rcu_barrier() */
>> +void flush_all_rcu_sheaves()
>> +{
> 
> mm/slub.c:3960:27: error: a function declaration without a prototype is deprecated in all versions of C [-Werror,-Wstrict-prototypes]
>  3960 | void flush_all_rcu_sheaves()
>       |                           ^
>       |                            void
> 
> ---

Thanks, the bots told me too and it's fixed in -next

> diff --git a/mm/slub.c b/mm/slub.c
> index 11ad4173e2f2..a1eae71a0f8c 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3955,9 +3955,8 @@ static void flush_rcu_sheaf(struct work_struct *w)
>  		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
>  }
>  
> -
>  /* needed for kvfree_rcu_barrier() */
> -void flush_all_rcu_sheaves()
> +void flush_all_rcu_sheaves(void)
>  {
>  	struct slub_percpu_sheaves *pcs;
>  	struct slub_flush_work *sfw;



From aliceryhl at google.com  Fri Sep 12 01:07:33 2025
From: aliceryhl at google.com (Alice Ryhl)
Date: Fri, 12 Sep 2025 08:07:33 +0000
Subject: [PATCH] rust: maple_tree: fix intra-doc link
In-Reply-To: <20250910140212.997771-1-ojeda@kernel.org>
References: <20250910140212.997771-1-ojeda@kernel.org>
Message-ID: <aMPUxZoV9BCqDg0y@google.com>

On Wed, Sep 10, 2025 at 04:02:12PM +0200, Miguel Ojeda wrote:
> A couple `Occupied` intra-doc links are broken:
> 
>     error: unresolved link to `Occupied`
>       --> rust/kernel/maple_tree.rs:96:46
>        |
>     96 |     /// return an [`InsertError`] with the [`Occupied`] kind. It may also fail if memory
>        |                                              ^^^^^^^^ no item named `Occupied` in scope
>        |
>        = help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
>        = note: `-D rustdoc::broken-intra-doc-links` implied by `-D warnings`
>        = help: to override `-D warnings` add `#[allow(rustdoc::broken_intra_doc_links)]`
> 
>     error: unresolved link to `Occupied`
>        --> rust/kernel/maple_tree.rs:135:36
>         |
>     135 |     /// [`InsertError`] with the [`Occupied`] kind. It may also fail if memory allocation fails
>         |                                    ^^^^^^^^ no item named `Occupied` in scope
>         |
>         = help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
> 
> Thus fix them.
> 
> Fixes: 7f51d867d2ed ("rust: maple_tree: add MapleTree")
> Signed-off-by: Miguel Ojeda <ojeda at kernel.org>

Reviewed-by: Alice Ryhl <aliceryhl at google.com>



From foxido at foxido.dev  Fri Sep 12 02:53:30 2025
From: foxido at foxido.dev (Gladyshev Ilya)
Date: Fri, 12 Sep 2025 12:53:30 +0300
Subject: [PATCH] maple: fix incorrect rewinding in mas_empty_area_rev
Message-ID: <20250912095335.1225247-1-foxido@foxido.dev>

Previously, mas_empty_area_rev was rewinding to the previous node if
some offset was cached. This could lead to incorrect results because a
useful gap could be skipped. However, this was never triggered in the
kernel because mm subsystem calls mas_empty_area_rev on non cached mas.

This patch unifies the rewind logic between mas_empty_area_rev and
mas_empty_area, so they both rewind in their correct directions.

Signed-off-by: Gladyshev Ilya <foxido at foxido.dev>
---
 lib/maple_tree.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a9..c7790fff4825 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -5138,15 +5138,15 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
 
 	if (mas_is_start(mas))
 		mas_start(mas);
-	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
-		return -EBUSY;
 
 	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
 		return mas_sparse_area(mas, min, max, size, false);
-	else if (mas->offset >= 2)
-		mas->offset -= 2;
-	else
+	else if (!mas->offset)
 		mas->offset = mas_data_end(mas);
+	else if (mas->offset <= mas_data_end(mas) - 2)
+		mas->offset = mas->offset + 2;
+	else if (!mas_skip_node(mas))
+		return -EBUSY;
 
 
 	/* The start of the window can only be within these values. */
-- 
2.50.1



From sudarsanm at google.com  Fri Sep 12 17:09:35 2025
From: sudarsanm at google.com (Sudarsan Mahendran)
Date: Fri, 12 Sep 2025 17:09:35 -0700
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
Message-ID: <20250913000935.1021068-1-sudarsanm@google.com>

Hi Vlastimil,

I ported this patch series on top of v6.17 and ran
some benchmarks: will-it-scale, hackbench, redis,
unixbench and kernbench. I ran the benchmarks
on Intel Granite Rapids (480 cores), AMD Turin (512 cores)
and ARM (80 cores)

Summary of the results:

- Significant change (meaning >10% difference
  between base and experiment) on will-it-scale
  tests in AMD.
- No significant change on other benchmarks ran.

Summary of AMD will-it-scale test changes:

Number of runs : 15
Direction      : + is good

|            | MIN        | MAX        | MEAN       | MEDIAN     | STDDEV     |
|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
| brk1_8_processes
| BASE       | 7,667,220  | 7,705,767  | 7,682,782  | 7,676,211  | 12,733     |
| TEST       | 9,477,395  | 10,053,058 | 9,878,753  | 9,959,360  | 182,014    |
| %          | +23.61%    | +30.46%    | +28.58%    | +29.74%    | +1,329.46% |
| brk1_8_threads
| BASE       | 1,838,468  | 1,890,873  | 1,864,473  | 1,864,875  | 18,406     |
| TEST       | 1,464,917  | 1,668,207  | 1,608,345  | 1,654,578  | 70,558     |
| %          | -20.32%    | -11.78%    | -13.74%    | -11.28%    | +283.34%   |
| brk1_128_processes
| BASE       | 65,018,211 | 65,603,366 | 65,285,348 | 65,103,197 | 232,885    |
| TEST       | 51,287,801 | 52,161,228 | 51,647,807 | 51,509,505 | 277,326    |
| %          | -21.12%    | -20.49%    | -20.89%    | -20.88%    | +19.08%    |
| brk1_256_processes
| BASE       | 15,186,881 | 15,239,120 | 15,210,265 | 15,205,809 | 15,850     |
| TEST       | 20,478,924 | 23,936,204 | 22,754,698 | 22,771,320 | 1,255,974  |
| %          | +34.85%    | +57.07%    | +49.60%    | +49.75%    | +7,823.72% |
| brk1_384_processes
| BASE       | 11,587,076 | 11,851,775 | 11,765,869 | 11,806,007 | 73,736     |
| TEST       | 25,464,757 | 29,176,818 | 26,695,946 | 26,563,116 | 1,012,563  |
| %          | +119.77%   | +146.18%   | +126.89%   | +125.00%   | +1,273.23% |
| brk1_512_processes
| BASE       | 14,410,293 | 14,891,082 | 14,775,209 | 14,793,142 | 114,384    |
| TEST       | 22,918,195 | 29,648,177 | 25,204,321 | 25,128,471 | 1,604,037  |
| %          | +59.04%    | +99.10%    | +70.59%    | +69.87%    | +1,302.32% |
| mmap1_8_processes
| BASE       | 3,164,411  | 3,170,585  | 3,167,590  | 3,167,692  | 2,436      |
| TEST       | 3,516,242  | 3,756,209  | 3,684,585  | 3,698,731  | 68,659     |
| %          | +11.12%    | +18.47%    | +16.32%    | +16.76%    | +2,718.28% |
| mmap1_8_threads
| BASE       | 627,817    | 632,702    | 630,554    | 629,281    | 1,764      |
| TEST       | 541,202    | 554,097    | 549,104    | 549,896    | 4,257      |
| %          | -13.80%    | -12.42%    | -12.92%    | -12.62%    | +141.34%   |
| mmap1_128_processes
| BASE       | 30,303,429 | 30,736,686 | 30,466,107 | 30,343,821 | 174,985    |
| TEST       | 9,749,426  | 9,893,331  | 9,823,701  | 9,857,157  | 52,125     |
| %          | -67.83%    | -67.81%    | -67.76%    | -67.52%    | -70.21%    |
| mmap1_256_processes
| BASE       | 7,496,765  | 7,546,703  | 7,528,379  | 7,543,246  | 21,465     |
| TEST       | 10,868,119 | 16,947,857 | 12,695,418 | 11,608,083 | 2,157,787  |
| %          | +44.97%    | +124.57%   | +68.63%    | +53.89%    | +9,952.34% |
| mmap1_384_processes
| BASE       | 5,629,206  | 5,856,927  | 5,758,347  | 5,733,892  | 85,930     |
| TEST       | 12,053,514 | 13,635,555 | 12,966,975 | 13,283,450 | 606,325    |
| %          | +114.12%   | +135.49%   | +112.89%   | +104.26%   | +2,855.57% |
| mmap1_512_processes
| BASE       | 6,959,199  | 6,996,383  | 6,975,912  | 6,974,353  | 15,446     |
| TEST       | 10,197,814 | 12,029,690 | 11,458,180 | 11,381,726 | 534,690    |
| %          | +46.54%    | +71.94%    | +64.25%    | +63.19%    | +3,361.67% |
| tlb_flush2_384_threads
| BASE       | 2,953,477  | 3,021,464  | 3,003,512  | 3,014,264  | 25,525     |
| TEST       | 2,231,417  | 2,526,876  | 2,408,368  | 2,411,121  | 115,773    |
| %          | -24.45%    | -16.37%    | -19.81%    | -20.01%    | +353.55%   |
| tlb_flush2_512_threads
| BASE       | 2,499,486  | 2,542,966  | 2,520,278  | 2,530,049  | 17,161     |
| TEST       | 1,707,641  | 1,714,524  | 1,708,951  | 1,707,713  | 1,877      |
| %          | -31.68%    | -32.58%    | -32.19%    | -32.50%    | -89.06%    |
| mmap2_128_processes
| BASE       | 29,754,984 | 30,313,146 | 30,010,106 | 29,897,731 | 218,812    |
| TEST       | 9,688,640  | 9,750,688  | 9,710,137  | 9,696,830  | 23,428     |
| %          | -67.44%    | -67.83%    | -67.64%    | -67.57%    | -89.29%    |
| mmap2_256_processes
| BASE       | 7,483,929  | 7,532,461  | 7,491,876  | 7,489,398  | 11,134     |
| TEST       | 11,580,023 | 16,508,551 | 15,337,145 | 15,943,608 | 1,489,489  |
| %          | +54.73%    | +119.17%   | +104.72%   | +112.88%   | +13,276.75%|
| mmap2_384_processes
| BASE       | 5,725,503  | 5,826,364  | 5,763,341  | 5,765,247  | 29,674     |
| TEST       | 11,682,353 | 13,720,566 | 12,269,665 | 11,776,228 | 877,060    |
| %          | +104.04%   | +135.49%   | +112.89%   | +104.26%   | +2,855.57% |
| mmap2_512_processes
| BASE       | 6,959,199  | 6,996,383  | 6,975,912  | 6,974,353  | 15,446     |
| TEST       | 10,197,814 | 12,029,690 | 11,458,180 | 11,381,726 | 534,690    |
| %          | +46.54%    | +71.94%    | +64.25%    | +63.19%    | +3,361.67% |



From mjguzik at gmail.com  Sat Sep 13 07:35:03 2025
From: mjguzik at gmail.com (Mateusz Guzik)
Date: Sat, 13 Sep 2025 16:35:03 +0200
Subject: [PATCH v5 01/14] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <20250723-slub-percpu-caches-v5-1-b792cd830f5d@suse.cz>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250723-slub-percpu-caches-v5-1-b792cd830f5d@suse.cz>
Message-ID: <ja3gd6bdckiuanm3xu4hszpjm7euvffm3k7tmu7drh6mdel7m6@oyr635vikumk>

On Wed, Jul 23, 2025 at 03:34:34PM +0200, Vlastimil Babka wrote:
> The sheaves do not distinguish NUMA locality of the cached objects.

While currently sheaves are opt-in, to my understanding the plan is to
make this the default.

I would argue a hard requirement for a general purpose allocator in this
day and age is to provide node-local memory by default. Notably if you
have a workload which was careful to bind itself to one node, it should
not receive memory backed by other nodes unless there is no other
option. AFAIU this is satisifed with the stock allocator on the grounds
of running on a given domain, without having to explicitly demand memory
from it for everyting.

I expect the lack of NUMA-awareness to result in increased accumulation
of "mismatched" memory as uptime goes up, violating the above.

Some examples how I expect that to happen should this get expanded to
all allocations:
- wherever init happens to reap a zombie, task_struct and some more
  stuff may be "misplaced"
- even ignoring init, literally any fork/exec/exit heavy workload which
  runs on more than one node will be ripe with mismatched frees as the
  scheduler moves things around and the original parent reaps children
- a process passes a file descriptor to a process on another domain and
  the latter is the last to fput
- a container creates a bunch of dentries and whacks them
etc.

In all of these cases getting unlucky means you are using non-local
memory, which in turn will result in weird anomalies which suddenly
clear themselves up if you restart the program (or which show up out of
nowhere).

Arguably, the fork thing is a problem as is and *probably* could be
reduced by asking the scheduler upfront where it would run the child
domain-wise if it had to do it right now and making fork allocate memory
from that domain.

But even with this or some other mitigation in place there would be
plenty of potential to free non-local memory, so the general problem
statement stands.

I admit though I don't have a good solution as to how to handle the
"bad" frees. Someone (I think you?) stated that one of the previous
allocators was just freeing to per-domain lists or arrays and that was
causing trouble -- perhaps this would work if it came with small limits
in place for how big these can get?


From vbabka at suse.cz  Sat Sep 13 13:32:54 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Sat, 13 Sep 2025 22:32:54 +0200
Subject: [PATCH v5 01/14] slab: add opt-in caching layer of percpu sheaves
In-Reply-To: <ja3gd6bdckiuanm3xu4hszpjm7euvffm3k7tmu7drh6mdel7m6@oyr635vikumk>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250723-slub-percpu-caches-v5-1-b792cd830f5d@suse.cz>
 <ja3gd6bdckiuanm3xu4hszpjm7euvffm3k7tmu7drh6mdel7m6@oyr635vikumk>
Message-ID: <b923fd8a-4f07-4689-a0cf-4ac5b85c6fe5@suse.cz>

On 9/13/25 16:35, Mateusz Guzik wrote:
> On Wed, Jul 23, 2025 at 03:34:34PM +0200, Vlastimil Babka wrote:
>> The sheaves do not distinguish NUMA locality of the cached objects.
> 
> While currently sheaves are opt-in, to my understanding the plan is to
> make this the default.
> 
> I would argue a hard requirement for a general purpose allocator in this
> day and age is to provide node-local memory by default. Notably if you
> have a workload which was careful to bind itself to one node, it should
> not receive memory backed by other nodes unless there is no other
> option. AFAIU this is satisifed with the stock allocator on the grounds
> of running on a given domain, without having to explicitly demand memory
> from it for everyting.
> 
> I expect the lack of NUMA-awareness to result in increased accumulation
> of "mismatched" memory as uptime goes up, violating the above.

Yeah this is addressed in later patch in the series "mm, slub: skip percpu
sheaves for remote object freeing" and then "mm, slab: allow NUMA restricted
allocations to use percpu sheaves" further improves it. I should update
commit log in this patch too.

> I admit though I don't have a good solution as to how to handle the
> "bad" frees. Someone (I think you?) stated that one of the previous
> allocators was just freeing to per-domain lists or arrays and that was
> causing trouble -- perhaps this would work if it came with small limits
> in place for how big these can get?

Yes, the alien arrays of SLAB, with their dreaded periodical flushing.
To avoid that the later patch simply skips the percpu sheaf. Only about 5%
of frees seem to be remote, so 95% will still benefit.



From ej at inai.de  Mon Sep 15 00:51:25 2025
From: ej at inai.de (Jan Engelhardt)
Date: Mon, 15 Sep 2025 09:51:25 +0200 (CEST)
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <20250913000935.1021068-1-sudarsanm@google.com>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz> <20250913000935.1021068-1-sudarsanm@google.com>
Message-ID: <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>


On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
>
>Summary of the results:
>
>- Significant change (meaning >10% difference
>  between base and experiment) on will-it-scale
>  tests in AMD.
>
>Summary of AMD will-it-scale test changes:
>
>Number of runs : 15
>Direction      : + is good

If STDDEV grows more than mean, there is more jitter,
which is not "good".

>|            | MIN        | MAX        | MEAN       | MEDIAN     | STDDEV     |
>|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
>| brk1_8_processes
>| BASE       | 7,667,220  | 7,705,767  | 7,682,782  | 7,676,211  | 12,733     |
>| TEST       | 9,477,395  | 10,053,058 | 9,878,753  | 9,959,360  | 182,014    |
>| %          | +23.61%    | +30.46%    | +28.58%    | +29.74%    | +1,329.46% |
>
>| mmap2_256_processes
>| BASE       | 7,483,929  | 7,532,461  | 7,491,876  | 7,489,398  | 11,134     |
>| TEST       | 11,580,023 | 16,508,551 | 15,337,145 | 15,943,608 | 1,489,489  |
>| %          | +54.73%    | +119.17%   | +104.72%   | +112.88%   | +13,276.75%|


From paulmck at kernel.org  Mon Sep 15 05:13:23 2025
From: paulmck at kernel.org (Paul E. McKenney)
Date: Mon, 15 Sep 2025 05:13:23 -0700
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
Message-ID: <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>

On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> 
> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> >
> >Summary of the results:
> >
> >- Significant change (meaning >10% difference
> >  between base and experiment) on will-it-scale
> >  tests in AMD.
> >
> >Summary of AMD will-it-scale test changes:
> >
> >Number of runs : 15
> >Direction      : + is good
> 
> If STDDEV grows more than mean, there is more jitter,
> which is not "good".

This is true.  On the other hand, the mean grew way more in absolute
terms than did STDDEV.  So might this be a reasonable tradeoff?

Of course, if adjustments can be made to keep the increase in mean while
keeping STDDEV low, that would of course be even better.

							Thanx, Paul

> >|            | MIN        | MAX        | MEAN       | MEDIAN     | STDDEV     |
> >|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
> >| brk1_8_processes
> >| BASE       | 7,667,220  | 7,705,767  | 7,682,782  | 7,676,211  | 12,733     |
> >| TEST       | 9,477,395  | 10,053,058 | 9,878,753  | 9,959,360  | 182,014    |
> >| %          | +23.61%    | +30.46%    | +28.58%    | +29.74%    | +1,329.46% |
> >
> >| mmap2_256_processes
> >| BASE       | 7,483,929  | 7,532,461  | 7,491,876  | 7,489,398  | 11,134     |
> >| TEST       | 11,580,023 | 16,508,551 | 15,337,145 | 15,943,608 | 1,489,489  |
> >| %          | +54.73%    | +119.17%   | +104.72%   | +112.88%   | +13,276.75%|
> 


From vbabka at suse.cz  Mon Sep 15 08:22:14 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 15 Sep 2025 17:22:14 +0200
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
Message-ID: <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>

On 9/15/25 14:13, Paul E. McKenney wrote:
> On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
>> 
>> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
>> >
>> >Summary of the results:

In any case, thanks a lot for the results!

>> >- Significant change (meaning >10% difference
>> >  between base and experiment) on will-it-scale
>> >  tests in AMD.
>> >
>> >Summary of AMD will-it-scale test changes:
>> >
>> >Number of runs : 15
>> >Direction      : + is good
>> 
>> If STDDEV grows more than mean, there is more jitter,
>> which is not "good".
> 
> This is true.  On the other hand, the mean grew way more in absolute
> terms than did STDDEV.  So might this be a reasonable tradeoff?

Also I'd point out that MIN of TEST is better than MAX of BASE, which means
there's always an improvement for this config. So jitter here means it's
changing between better and more better :) and not between worse and (more)
better.

The annoying part of course is that for other configs it's consistently the
opposite.

> Of course, if adjustments can be made to keep the increase in mean while
> keeping STDDEV low, that would of course be even better.
> 
> 							Thanx, Paul
> 
>> >|            | MIN        | MAX        | MEAN       | MEDIAN     | STDDEV     |
>> >|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
>> >| brk1_8_processes
>> >| BASE       | 7,667,220  | 7,705,767  | 7,682,782  | 7,676,211  | 12,733     |
>> >| TEST       | 9,477,395  | 10,053,058 | 9,878,753  | 9,959,360  | 182,014    |
>> >| %          | +23.61%    | +30.46%    | +28.58%    | +29.74%    | +1,329.46% |
>> >
>> >| mmap2_256_processes
>> >| BASE       | 7,483,929  | 7,532,461  | 7,491,876  | 7,489,398  | 11,134     |
>> >| TEST       | 11,580,023 | 16,508,551 | 15,337,145 | 15,943,608 | 1,489,489  |
>> >| %          | +54.73%    | +119.17%   | +104.72%   | +112.88%   | +13,276.75%|
>> 



From surenb at google.com  Tue Sep 16 10:09:18 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Tue, 16 Sep 2025 10:09:18 -0700
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com> <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop> <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
Message-ID: <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>

On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> On 9/15/25 14:13, Paul E. McKenney wrote:
> > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> >>
> >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> >> >
> >> >Summary of the results:
>
> In any case, thanks a lot for the results!
>
> >> >- Significant change (meaning >10% difference
> >> >  between base and experiment) on will-it-scale
> >> >  tests in AMD.
> >> >
> >> >Summary of AMD will-it-scale test changes:
> >> >
> >> >Number of runs : 15
> >> >Direction      : + is good
> >>
> >> If STDDEV grows more than mean, there is more jitter,
> >> which is not "good".
> >
> > This is true.  On the other hand, the mean grew way more in absolute
> > terms than did STDDEV.  So might this be a reasonable tradeoff?
>
> Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> there's always an improvement for this config. So jitter here means it's
> changing between better and more better :) and not between worse and (more)
> better.
>
> The annoying part of course is that for other configs it's consistently the
> opposite.

Hi Vlastimil,
I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
faulting them in then unmapping and timing only mmap and munmap calls.
This is not a realistic scenario but works well for A/B comparison.

The numbers are below with sheaves showing a clear improvement:

Baseline
            avg             stdev
mmap        2.621073        0.2525161631
munmap      2.292965        0.008831973052
total       4.914038        0.2572620923

Sheaves
            avg            stdev           avg_diff        stdev_diff
mmap        1.561220667    0.07748897037   -40.44%        -69.31%
munmap      2.042071       0.03603083448   -10.94%        307.96%
total       3.603291667    0.113209047     -26.67%        -55.99%

Stdev for munmap went high but I see that there was only one run that
was very different from others, so that might have been just a noisy
run.

One thing I noticed is that with my stress testing mmap/munmap in a
loop we get lots of in-flight freed-by-RCU sheaves before the grace
period arrives and they get freed in bulk. Note that Android enables
lazy RCU config, so that affects the grace period and makes it longer
than normal. This results in sheaves being freed in bulk and when that
happens, the barn gets quickly full (we only have 10
(MAX_FULL_SHEAVES) free slots), the rest of the sheaves being freed
are destroyed instead of being reused.

I tried two modifications:
1. Use call_rcu_hurry() instead of call_rcu() when freeing the
sheaves. This should remove the effects of lazy RCU;
2. Keep a running count of in-flight RCU-freed sheaves and once it
reaches the number of free slots for full sheaves in the barn, I
schedule an rcu_barrier() to free all these in-flight sheaves. Note
that I added an additional condition to skip this RCU flush if the
number of free slots for full sheaves is less than MAX_FULL_SHEAVES/2.
That should prevent flushing to free only a small number of sheaves.

With these modifications the numbers get even better:

Sheaves with call_rcu_hurry
            avg                            avg_diff (vs Baseline)
mmap        1.279308                       -51.19%
munmap      1.983921                       -13.48%
total       3.263228                       -33.59%

Sheaves with rcu_barrier
            avg                            avg_diff (vs Baseline)
mmap        1.210455                       -53.82%
munmap      1.963739                       -14.36%
total       3.174194                       -35.41%

I didn't capture stdev because I did not run as many times as the
first two configurations.

Again, the tight loop in my test is not representative of a real
workloads and the numbers are definitely affected by the use of lazy
RCU mode in Android. While this information can be used for later
optimizations, I don't think these findings should block current
deployment of the sheaves.
Thanks,
Suren.


>
> > Of course, if adjustments can be made to keep the increase in mean while
> > keeping STDDEV low, that would of course be even better.
> >
> >                                                       Thanx, Paul
> >
> >> >|            | MIN        | MAX        | MEAN       | MEDIAN     | STDDEV     |
> >> >|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
> >> >| brk1_8_processes
> >> >| BASE       | 7,667,220  | 7,705,767  | 7,682,782  | 7,676,211  | 12,733     |
> >> >| TEST       | 9,477,395  | 10,053,058 | 9,878,753  | 9,959,360  | 182,014    |
> >> >| %          | +23.61%    | +30.46%    | +28.58%    | +29.74%    | +1,329.46% |
> >> >
> >> >| mmap2_256_processes
> >> >| BASE       | 7,483,929  | 7,532,461  | 7,491,876  | 7,489,398  | 11,134     |
> >> >| TEST       | 11,580,023 | 16,508,551 | 15,337,145 | 15,943,608 | 1,489,489  |
> >> >| %          | +54.73%    | +119.17%   | +104.72%   | +112.88%   | +13,276.75%|
> >>
>


From urezki at gmail.com  Tue Sep 16 22:19:54 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Wed, 17 Sep 2025 07:19:54 +0200
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
 <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
Message-ID: <aMpE-oSjtlDU4TSl@pc636>

On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> >
> > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > >>
> > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > >> >
> > >> >Summary of the results:
> >
> > In any case, thanks a lot for the results!
> >
> > >> >- Significant change (meaning >10% difference
> > >> >  between base and experiment) on will-it-scale
> > >> >  tests in AMD.
> > >> >
> > >> >Summary of AMD will-it-scale test changes:
> > >> >
> > >> >Number of runs : 15
> > >> >Direction      : + is good
> > >>
> > >> If STDDEV grows more than mean, there is more jitter,
> > >> which is not "good".
> > >
> > > This is true.  On the other hand, the mean grew way more in absolute
> > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> >
> > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > there's always an improvement for this config. So jitter here means it's
> > changing between better and more better :) and not between worse and (more)
> > better.
> >
> > The annoying part of course is that for other configs it's consistently the
> > opposite.
> 
> Hi Vlastimil,
> I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> faulting them in then unmapping and timing only mmap and munmap calls.
> This is not a realistic scenario but works well for A/B comparison.
> 
> The numbers are below with sheaves showing a clear improvement:
> 
> Baseline
>             avg             stdev
> mmap        2.621073        0.2525161631
> munmap      2.292965        0.008831973052
> total       4.914038        0.2572620923
> 
> Sheaves
>             avg            stdev           avg_diff        stdev_diff
> mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> munmap      2.042071       0.03603083448   -10.94%        307.96%
> total       3.603291667    0.113209047     -26.67%        -55.99%
> 
Could you run your test with dropping below patch?

[PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations

mmap()/munmap(), i assume it is a duration time in average, is the time
in microseconds?

Thank you. 

--
Uladzislau Rezki


From harry.yoo at oracle.com  Wed Sep 17 01:30:20 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Wed, 17 Sep 2025 17:30:20 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
Message-ID: <aMpxnACqmsQl-lp0@hyeyoo>

On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> addition to main and spare sheaves.
> 
> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> the sheaf is detached and submitted to call_rcu() with a handler that
> will try to put it in the barn, or flush to slab pages using bulk free,
> when the barn is full. Then a new empty sheaf must be obtained to put
> more objects there.
> 
> It's possible that no free sheaves are available to use for a new
> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> kfree_rcu() implementation.
> 
> Expected advantages:
> - batching the kfree_rcu() operations, that could eventually replace the
>   existing batching
> - sheaves can be reused for allocations via barn instead of being
>   flushed to slabs, which is more efficient
>   - this includes cases where only some cpus are allowed to process rcu
>     callbacks (Android)
> 
> Possible disadvantage:
> - objects might be waiting for more than their grace period (it is
>   determined by the last object freed into the sheaf), increasing memory
>   usage - but the existing batching does that too.
> 
> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> implementation favors smaller memory footprint over performance.
> 
> Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> contexts where kfree_rcu() is called might not be compatible with taking
> a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> spinlock - the current kfree_rcu() implementation avoids doing that.
> 
> Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> that have them. This is not a cheap operation, but the barrier usage is
> rare - currently kmem_cache_destroy() or on module unload.
> 
> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> many had to fall back to the existing implementation.
> 
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  mm/slab.h        |   3 +
>  mm/slab_common.c |  26 ++++++
>  mm/slub.c        | 266 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 293 insertions(+), 2 deletions(-)
> 
> @@ -3840,6 +3895,80 @@ static void flush_all(struct kmem_cache *s)
>  	cpus_read_unlock();
>  }
>  
> +/* needed for kvfree_rcu_barrier() */
> +void flush_all_rcu_sheaves()
> +{
> +	struct slub_percpu_sheaves *pcs;
> +	struct slub_flush_work *sfw;
> +	struct kmem_cache *s;
> +	bool flushed = false;
> +	unsigned int cpu;
> +
> +	cpus_read_lock();
> +	mutex_lock(&slab_mutex);
> +
> +	list_for_each_entry(s, &slab_caches, list) {
> +		if (!s->cpu_sheaves)
> +			continue;
> +
> +		mutex_lock(&flush_lock);
> +
> +		for_each_online_cpu(cpu) {
> +			sfw = &per_cpu(slub_flush, cpu);
> +			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
> +
> +			if (!pcs->rcu_free || !pcs->rcu_free->size) {

Is the compiler allowed to compile this to read pcs->rcu_free twice?
Something like:

flush_all_rcu_sheaves()			__kfree_rcu_sheaf()

pcs->rcu_free != NULL
					pcs->rcu_free = NULL
pcs->rcu_free == NULL
/* NULL-pointer-deref */
pcs->rcu_free->size

> +				sfw->skip = true;
> +				continue;
> +			}
>
> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> +			sfw->skip = false;
> +			sfw->s = s;
> +			queue_work_on(cpu, flushwq, &sfw->work);
> +			flushed = true;
> +		}
> +
> +		for_each_online_cpu(cpu) {
> +			sfw = &per_cpu(slub_flush, cpu);
> +			if (sfw->skip)
> +				continue;
> +			flush_work(&sfw->work);
> +		}
> +
> +		mutex_unlock(&flush_lock);
> +	}
> +
> +	mutex_unlock(&slab_mutex);
> +	cpus_read_unlock();
> +
> +	if (flushed)
> +		rcu_barrier();

I think we need to call rcu_barrier() even if flushed == false?

Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
be processed before flush_all_rcu_sheaves() is called, and
in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
so flushed == false but the rcu callback isn't processed yet
by the end of the function?

That sounds like a very unlikely to happen in a realistic scenario,
but still possible...

-- 
Cheers,
Harry / Hyeonggon


From vbabka at suse.cz  Wed Sep 17 02:55:10 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 17 Sep 2025 11:55:10 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aMpxnACqmsQl-lp0@hyeyoo>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
Message-ID: <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>

On 9/17/25 10:30, Harry Yoo wrote:
> On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> +/* needed for kvfree_rcu_barrier() */
>> +void flush_all_rcu_sheaves()
>> +{
>> +	struct slub_percpu_sheaves *pcs;
>> +	struct slub_flush_work *sfw;
>> +	struct kmem_cache *s;
>> +	bool flushed = false;
>> +	unsigned int cpu;
>> +
>> +	cpus_read_lock();
>> +	mutex_lock(&slab_mutex);
>> +
>> +	list_for_each_entry(s, &slab_caches, list) {
>> +		if (!s->cpu_sheaves)
>> +			continue;
>> +
>> +		mutex_lock(&flush_lock);
>> +
>> +		for_each_online_cpu(cpu) {
>> +			sfw = &per_cpu(slub_flush, cpu);
>> +			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
>> +
>> +			if (!pcs->rcu_free || !pcs->rcu_free->size) {
> 
> Is the compiler allowed to compile this to read pcs->rcu_free twice?
> Something like:
> 
> flush_all_rcu_sheaves()			__kfree_rcu_sheaf()
> 
> pcs->rcu_free != NULL
> 					pcs->rcu_free = NULL
> pcs->rcu_free == NULL
> /* NULL-pointer-deref */
> pcs->rcu_free->size

Good point, I'll remove the size check and simply pcs->rcu_free non-null
means we flush.

>> +				sfw->skip = true;
>> +				continue;
>> +			}
>>
>> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> +			sfw->skip = false;
>> +			sfw->s = s;
>> +			queue_work_on(cpu, flushwq, &sfw->work);
>> +			flushed = true;
>> +		}
>> +
>> +		for_each_online_cpu(cpu) {
>> +			sfw = &per_cpu(slub_flush, cpu);
>> +			if (sfw->skip)
>> +				continue;
>> +			flush_work(&sfw->work);
>> +		}
>> +
>> +		mutex_unlock(&flush_lock);
>> +	}
>> +
>> +	mutex_unlock(&slab_mutex);
>> +	cpus_read_unlock();
>> +
>> +	if (flushed)
>> +		rcu_barrier();
> 
> I think we need to call rcu_barrier() even if flushed == false?
> 
> Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> be processed before flush_all_rcu_sheaves() is called, and
> in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> so flushed == false but the rcu callback isn't processed yet
> by the end of the function?
> 
> That sounds like a very unlikely to happen in a realistic scenario,
> but still possible...

Yes also good point, will flush unconditionally.

Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
local_unlock(). So we don't end up seeing a NULL pcs->rcu_free in
flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
but didn't yet do the call_rcu() as it got preempted after local_unlock().

But then rcu_barrier() itself probably won't mean we make sure such cpus
finished the local_locked section, if we didn't queue work on them. So maybe
we need synchronize_rcu()?



From harry.yoo at oracle.com  Wed Sep 17 04:32:47 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Wed, 17 Sep 2025 20:32:47 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
Message-ID: <aMqcXyKRlZggLxu_@hyeyoo>

On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> On 9/17/25 10:30, Harry Yoo wrote:
> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >> +				sfw->skip = true;
> >> +				continue;
> >> +			}
> >>
> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >> +			sfw->skip = false;
> >> +			sfw->s = s;
> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >> +			flushed = true;
> >> +		}
> >> +
> >> +		for_each_online_cpu(cpu) {
> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> +			if (sfw->skip)
> >> +				continue;
> >> +			flush_work(&sfw->work);
> >> +		}
> >> +
> >> +		mutex_unlock(&flush_lock);
> >> +	}
> >> +
> >> +	mutex_unlock(&slab_mutex);
> >> +	cpus_read_unlock();
> >> +
> >> +	if (flushed)
> >> +		rcu_barrier();
> > 
> > I think we need to call rcu_barrier() even if flushed == false?
> > 
> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> > be processed before flush_all_rcu_sheaves() is called, and
> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> > so flushed == false but the rcu callback isn't processed yet
> > by the end of the function?
> > 
> > That sounds like a very unlikely to happen in a realistic scenario,
> > but still possible...
> 
> Yes also good point, will flush unconditionally.
> 
> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> local_unlock(). So we don't end up seeing a NULL pcs->rcu_free in
> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> but didn't yet do the call_rcu() as it got preempted after local_unlock().

Makes sense to me.

> But then rcu_barrier() itself probably won't mean we make sure such cpus
> finished the local_locked section, if we didn't queue work on them. So maybe
> we need synchronize_rcu()?

Ah, it works because preemption disabled section works as a RCU
read-side critical section?

But then are we allowed to do release the local_lock to allocate empty
sheaves in __kfree_rcu_sheaf()?

-- 
Cheers,
Harry / Hyeonggon


From paulmck at kernel.org  Wed Sep 17 04:36:12 2025
From: paulmck at kernel.org (Paul E. McKenney)
Date: Wed, 17 Sep 2025 04:36:12 -0700
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
Message-ID: <8e7055a9-f899-4aad-8ed7-6543077c05d1@paulmck-laptop>

On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> On 9/17/25 10:30, Harry Yoo wrote:
> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >> +/* needed for kvfree_rcu_barrier() */
> >> +void flush_all_rcu_sheaves()
> >> +{
> >> +	struct slub_percpu_sheaves *pcs;
> >> +	struct slub_flush_work *sfw;
> >> +	struct kmem_cache *s;
> >> +	bool flushed = false;
> >> +	unsigned int cpu;
> >> +
> >> +	cpus_read_lock();
> >> +	mutex_lock(&slab_mutex);
> >> +
> >> +	list_for_each_entry(s, &slab_caches, list) {
> >> +		if (!s->cpu_sheaves)
> >> +			continue;
> >> +
> >> +		mutex_lock(&flush_lock);
> >> +
> >> +		for_each_online_cpu(cpu) {
> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> +			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
> >> +
> >> +			if (!pcs->rcu_free || !pcs->rcu_free->size) {
> > 
> > Is the compiler allowed to compile this to read pcs->rcu_free twice?
> > Something like:
> > 
> > flush_all_rcu_sheaves()			__kfree_rcu_sheaf()
> > 
> > pcs->rcu_free != NULL
> > 					pcs->rcu_free = NULL
> > pcs->rcu_free == NULL
> > /* NULL-pointer-deref */
> > pcs->rcu_free->size
> 
> Good point, I'll remove the size check and simply pcs->rcu_free non-null
> means we flush.
> 
> >> +				sfw->skip = true;
> >> +				continue;
> >> +			}
> >>
> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >> +			sfw->skip = false;
> >> +			sfw->s = s;
> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >> +			flushed = true;
> >> +		}
> >> +
> >> +		for_each_online_cpu(cpu) {
> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> +			if (sfw->skip)
> >> +				continue;
> >> +			flush_work(&sfw->work);
> >> +		}
> >> +
> >> +		mutex_unlock(&flush_lock);
> >> +	}
> >> +
> >> +	mutex_unlock(&slab_mutex);
> >> +	cpus_read_unlock();
> >> +
> >> +	if (flushed)
> >> +		rcu_barrier();
> > 
> > I think we need to call rcu_barrier() even if flushed == false?
> > 
> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> > be processed before flush_all_rcu_sheaves() is called, and
> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> > so flushed == false but the rcu callback isn't processed yet
> > by the end of the function?
> > 
> > That sounds like a very unlikely to happen in a realistic scenario,
> > but still possible...
> 
> Yes also good point, will flush unconditionally.
> 
> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> local_unlock(). So we don't end up seeing a NULL pcs->rcu_free in
> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> 
> But then rcu_barrier() itself probably won't mean we make sure such cpus
> finished the local_locked section, if we didn't queue work on them. So maybe
> we need synchronize_rcu()?

Do you need both rcu_barrier() and synchronize_rcu(), maybe along with
kvfree_rcu_barrier() as well?  It would not be hard to make such a thing,
using workqueues or some such.  Not sure what the API should look like,
especially should people want other RCU flavors to get into the act
as well.

							Thanx, Paul


From harry.yoo at oracle.com  Wed Sep 17 04:46:35 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Wed, 17 Sep 2025 20:46:35 +0900
Subject: [PATCH v8 18/23] maple_tree: Use kfree_rcu in ma_free_rcu
In-Reply-To: <20250910-slub-percpu-caches-v8-18-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-18-ca3099d8352c@suse.cz>
Message-ID: <aMqfmzwCu1N_L1qp@hyeyoo>

On Wed, Sep 10, 2025 at 10:01:20AM +0200, Vlastimil Babka wrote:
> From: Pedro Falcato <pfalcato at suse.de>
> 
> kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
> possible to call it on non-kmalloc objects, but this restriction was
> lifted ever since SLOB was dropped from the kernel, and since commit
> 6c6c47b063b5 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").
> 
> Thus, replace call_rcu + mt_free_rcu with kfree_rcu.
> 
> Signed-off-by: Pedro Falcato <pfalcato at suse.de>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---

Looks good to me,
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

-- 
Cheers,
Harry / Hyeonggon


From vbabka at suse.cz  Wed Sep 17 05:05:49 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 17 Sep 2025 14:05:49 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aMqcXyKRlZggLxu_@hyeyoo>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
Message-ID: <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>

On 9/17/25 13:32, Harry Yoo wrote:
> On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>> On 9/17/25 10:30, Harry Yoo wrote:
>> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> >> +				sfw->skip = true;
>> >> +				continue;
>> >> +			}
>> >>
>> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> >> +			sfw->skip = false;
>> >> +			sfw->s = s;
>> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>> >> +			flushed = true;
>> >> +		}
>> >> +
>> >> +		for_each_online_cpu(cpu) {
>> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >> +			if (sfw->skip)
>> >> +				continue;
>> >> +			flush_work(&sfw->work);
>> >> +		}
>> >> +
>> >> +		mutex_unlock(&flush_lock);
>> >> +	}
>> >> +
>> >> +	mutex_unlock(&slab_mutex);
>> >> +	cpus_read_unlock();
>> >> +
>> >> +	if (flushed)
>> >> +		rcu_barrier();
>> > 
>> > I think we need to call rcu_barrier() even if flushed == false?
>> > 
>> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>> > be processed before flush_all_rcu_sheaves() is called, and
>> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>> > so flushed == false but the rcu callback isn't processed yet
>> > by the end of the function?
>> > 
>> > That sounds like a very unlikely to happen in a realistic scenario,
>> > but still possible...
>> 
>> Yes also good point, will flush unconditionally.
>> 
>> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>> local_unlock(). So we don't end up seeing a NULL pcs->rcu_free in
>> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> 
> Makes sense to me.
> 
>> But then rcu_barrier() itself probably won't mean we make sure such cpus
>> finished the local_locked section, if we didn't queue work on them. So maybe
>> we need synchronize_rcu()?
> 
> Ah, it works because preemption disabled section works as a RCU
> read-side critical section?

AFAIK yes? Or maybe not on RT where local_lock is taking a mutex? So we
should denote the RCU critical section explicitly too?

> But then are we allowed to do release the local_lock to allocate empty
> sheaves in __kfree_rcu_sheaf()?

I think so, as we do that when we found no rcu_sheaf in the first place.


From vbabka at suse.cz  Wed Sep 17 05:13:11 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 17 Sep 2025 14:13:11 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <8e7055a9-f899-4aad-8ed7-6543077c05d1@paulmck-laptop>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <8e7055a9-f899-4aad-8ed7-6543077c05d1@paulmck-laptop>
Message-ID: <98f10e2a-7a02-42e8-8042-984ed82d5ba7@suse.cz>

On 9/17/25 13:36, Paul E. McKenney wrote:
> On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>> On 9/17/25 10:30, Harry Yoo wrote:
>> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> >> +/* needed for kvfree_rcu_barrier() */
>> >> +void flush_all_rcu_sheaves()
>> >> +{
>> >> +	struct slub_percpu_sheaves *pcs;
>> >> +	struct slub_flush_work *sfw;
>> >> +	struct kmem_cache *s;
>> >> +	bool flushed = false;
>> >> +	unsigned int cpu;
>> >> +
>> >> +	cpus_read_lock();
>> >> +	mutex_lock(&slab_mutex);
>> >> +
>> >> +	list_for_each_entry(s, &slab_caches, list) {
>> >> +		if (!s->cpu_sheaves)
>> >> +			continue;
>> >> +
>> >> +		mutex_lock(&flush_lock);
>> >> +
>> >> +		for_each_online_cpu(cpu) {
>> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >> +			pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
>> >> +
>> >> +			if (!pcs->rcu_free || !pcs->rcu_free->size) {
>> > 
>> > Is the compiler allowed to compile this to read pcs->rcu_free twice?
>> > Something like:
>> > 
>> > flush_all_rcu_sheaves()			__kfree_rcu_sheaf()
>> > 
>> > pcs->rcu_free != NULL
>> > 					pcs->rcu_free = NULL
>> > pcs->rcu_free == NULL
>> > /* NULL-pointer-deref */
>> > pcs->rcu_free->size
>> 
>> Good point, I'll remove the size check and simply pcs->rcu_free non-null
>> means we flush.
>> 
>> >> +				sfw->skip = true;
>> >> +				continue;
>> >> +			}
>> >>
>> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> >> +			sfw->skip = false;
>> >> +			sfw->s = s;
>> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>> >> +			flushed = true;
>> >> +		}
>> >> +
>> >> +		for_each_online_cpu(cpu) {
>> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >> +			if (sfw->skip)
>> >> +				continue;
>> >> +			flush_work(&sfw->work);
>> >> +		}
>> >> +
>> >> +		mutex_unlock(&flush_lock);
>> >> +	}
>> >> +
>> >> +	mutex_unlock(&slab_mutex);
>> >> +	cpus_read_unlock();
>> >> +
>> >> +	if (flushed)
>> >> +		rcu_barrier();
>> > 
>> > I think we need to call rcu_barrier() even if flushed == false?
>> > 
>> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>> > be processed before flush_all_rcu_sheaves() is called, and
>> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>> > so flushed == false but the rcu callback isn't processed yet
>> > by the end of the function?
>> > 
>> > That sounds like a very unlikely to happen in a realistic scenario,
>> > but still possible...
>> 
>> Yes also good point, will flush unconditionally.
>> 
>> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>> local_unlock(). So we don't end up seeing a NULL pcs->rcu_free in
>> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>> but didn't yet do the call_rcu() as it got preempted after local_unlock().
>> 
>> But then rcu_barrier() itself probably won't mean we make sure such cpus
>> finished the local_locked section, if we didn't queue work on them. So maybe
>> we need synchronize_rcu()?
> 
> Do you need both rcu_barrier() and synchronize_rcu(), maybe along with

We need the local_lock protected sections of __kfree_rcu_sheaf() to be
finished (which might be doing call_rcu(rcu_sheaf) And then the pending
call_rcu(rcu_sheaf) callbacks to be executed.
I think that means both synchronize_rcu() and rcu_barrier(). Possibly a RCU
critical section in __kfree_rcu_sheaf() unless local_lock implies that even
on RT.

> kvfree_rcu_barrier() as well?

This (flush_all_rcu_sheaves()) is for the implementation of
kvfree_rcu_barrier() in a world with rcu_free sheaves. So it's called from
kvfree_rcu_barrier().

> It would not be hard to make such a thing,
> using workqueues or some such.  Not sure what the API should look like,

So there should be no one else calling such an API. There might be new users
of kvfree_rcu_barrier() doing this indirectly in the future.

> especially should people want other RCU flavors to get into the act
> as well.
> 
> 							Thanx, Paul



From harry.yoo at oracle.com  Wed Sep 17 06:07:43 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Wed, 17 Sep 2025 22:07:43 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
Message-ID: <aMqyn6nenR8V_p1y@hyeyoo>

On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> On 9/17/25 13:32, Harry Yoo wrote:
> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> >> On 9/17/25 10:30, Harry Yoo wrote:
> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >> >> +				sfw->skip = true;
> >> >> +				continue;
> >> >> +			}
> >> >>
> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >> >> +			sfw->skip = false;
> >> >> +			sfw->s = s;
> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >> >> +			flushed = true;
> >> >> +		}
> >> >> +
> >> >> +		for_each_online_cpu(cpu) {
> >> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> >> +			if (sfw->skip)
> >> >> +				continue;
> >> >> +			flush_work(&sfw->work);
> >> >> +		}
> >> >> +
> >> >> +		mutex_unlock(&flush_lock);
> >> >> +	}
> >> >> +
> >> >> +	mutex_unlock(&slab_mutex);
> >> >> +	cpus_read_unlock();
> >> >> +
> >> >> +	if (flushed)
> >> >> +		rcu_barrier();
> >> > 
> >> > I think we need to call rcu_barrier() even if flushed == false?
> >> > 
> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> >> > be processed before flush_all_rcu_sheaves() is called, and
> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> >> > so flushed == false but the rcu callback isn't processed yet
> >> > by the end of the function?
> >> > 
> >> > That sounds like a very unlikely to happen in a realistic scenario,
> >> > but still possible...
> >> 
> >> Yes also good point, will flush unconditionally.
> >> 
> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> >> local_unlock().
> >>
> >> So we don't end up seeing a NULL pcs->rcu_free in
> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> > 
> > Makes sense to me.

Wait, I'm confused.

I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
the object X to be freed before kvfree_rcu_barrier() returns?

IOW if flush_all_rcu_sheaves() is called while __kfree_rcu_sheaf(s, X) was
running on another CPU, we don't have to guarantee that
flush_all_rcu_sheaves() returns after the object X is freed?

> >> But then rcu_barrier() itself probably won't mean we make sure such cpus
> >> finished the local_locked section, if we didn't queue work on them. So maybe
> >> we need synchronize_rcu()?

So... we don't need a synchronize_rcu() then?

Or my brain started malfunctioning again :D

-- 
Cheers,
Harry / Hyeonggon


From vbabka at suse.cz  Wed Sep 17 06:21:31 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 17 Sep 2025 15:21:31 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aMqyn6nenR8V_p1y@hyeyoo>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo>
Message-ID: <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>

On 9/17/25 15:07, Harry Yoo wrote:
> On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
>> On 9/17/25 13:32, Harry Yoo wrote:
>> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>> >> On 9/17/25 10:30, Harry Yoo wrote:
>> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> >> >> +				sfw->skip = true;
>> >> >> +				continue;
>> >> >> +			}
>> >> >>
>> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> >> >> +			sfw->skip = false;
>> >> >> +			sfw->s = s;
>> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>> >> >> +			flushed = true;
>> >> >> +		}
>> >> >> +
>> >> >> +		for_each_online_cpu(cpu) {
>> >> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >> >> +			if (sfw->skip)
>> >> >> +				continue;
>> >> >> +			flush_work(&sfw->work);
>> >> >> +		}
>> >> >> +
>> >> >> +		mutex_unlock(&flush_lock);
>> >> >> +	}
>> >> >> +
>> >> >> +	mutex_unlock(&slab_mutex);
>> >> >> +	cpus_read_unlock();
>> >> >> +
>> >> >> +	if (flushed)
>> >> >> +		rcu_barrier();
>> >> > 
>> >> > I think we need to call rcu_barrier() even if flushed == false?
>> >> > 
>> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>> >> > be processed before flush_all_rcu_sheaves() is called, and
>> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>> >> > so flushed == false but the rcu callback isn't processed yet
>> >> > by the end of the function?
>> >> > 
>> >> > That sounds like a very unlikely to happen in a realistic scenario,
>> >> > but still possible...
>> >> 
>> >> Yes also good point, will flush unconditionally.
>> >> 
>> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>> >> local_unlock().
>> >>
>> >> So we don't end up seeing a NULL pcs->rcu_free in
>> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
>> > 
>> > Makes sense to me.
> 
> Wait, I'm confused.
> 
> I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> the object X to be freed before kvfree_rcu_barrier() returns?

Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
the rcu_sheaf fully and thus without submitting it to call_rcu(), then
migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
up the rcu_sheaf fully and is about to call_rcu() on it. And since that
sheaf also contains the object X, we should make sure that is flushed.

> IOW if flush_all_rcu_sheaves() is called while __kfree_rcu_sheaf(s, X) was
> running on another CPU, we don't have to guarantee that
> flush_all_rcu_sheaves() returns after the object X is freed?
> 
>> >> But then rcu_barrier() itself probably won't mean we make sure such cpus
>> >> finished the local_locked section, if we didn't queue work on them. So maybe
>> >> we need synchronize_rcu()?
> 
> So... we don't need a synchronize_rcu() then?
> 
> Or my brain started malfunctioning again :D
> 



From oliver.sang at intel.com  Tue Sep 16 19:40:05 2025
From: oliver.sang at intel.com (kernel test robot)
Date: Wed, 17 Sep 2025 10:40:05 +0800
Subject: [vbabka:slub-percpu-sheaves-v6r3] [maple_tree]  71a82407f1:
 will-it-scale.per_process_ops 18.7% regression
Message-ID: <202509170457.d1421a5-lkp@intel.com>



Hello,

kernel test robot noticed a 18.7% regression of will-it-scale.per_process_ops on:


commit: 71a82407f1c94903b3835b4c9351a0d4c2d5a58c ("maple_tree: use percpu sheaves for maple_node_cache")
https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-percpu-sheaves-v6r3

testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-12
test machine: 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory
parameters:

	nr_task: 100%
	mode: process
	test: mmap2
	cpufreq_governor: performance


In addition to that, the commit also has significant impact on the following tests:

+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops  21.9% regression                                         |
| test machine     | 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory |
| test parameters  | cpufreq_governor=performance                                                                           |
|                  | mode=process                                                                                           |
|                  | nr_task=100%                                                                                           |
|                  | test=mmap1                                                                                             |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops  28.9% regression                                         |
| test machine     | 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory |
| test parameters  | cpufreq_governor=performance                                                                           |
|                  | mode=process                                                                                           |
|                  | nr_task=100%                                                                                           |
|                  | test=brk2                                                                                              |
+------------------+--------------------------------------------------------------------------------------------------------+


If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang at intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202509170457.d1421a5-lkp at intel.com


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250917/202509170457.d1421a5-lkp at intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-9.4/process/100%/debian-12-x86_64-20240206.cgz/igk-spr-2sp2/mmap2/will-it-scale

commit: 
  7c1ca28747 ("tools/testing: Add testing support for slab caches with sheaves")
  71a82407f1 ("maple_tree: use percpu sheaves for maple_node_cache")

7c1ca28747078885 71a82407f1c94903b3835b4c935 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     11432           -17.7%       9408        vmstat.system.cs
  39366220           -18.7%   32021343        will-it-scale.192.processes
    205031           -18.7%     166777        will-it-scale.per_process_ops
  39366220           -18.7%   32021343        will-it-scale.workload
      0.70 ?  3%      -0.2        0.52 ?  2%  mpstat.cpu.all.irq%
     28.01            -9.4       18.61 ?  2%  mpstat.cpu.all.soft%
     68.05           +10.0       78.09        mpstat.cpu.all.sys%
      2.75            -0.5        2.29        mpstat.cpu.all.usr%
 1.431e+08           -40.9%   84522354 ?  2%  numa-numastat.node0.local_node
 1.432e+08           -40.9%   84641286 ?  2%  numa-numastat.node0.numa_hit
 1.428e+08           -40.2%   85421617 ?  2%  numa-numastat.node1.local_node
 1.429e+08           -40.2%   85491246 ?  2%  numa-numastat.node1.numa_hit
   3426536 ?  7%     +19.5%    4096012 ?  4%  meminfo.Active
   3426536 ?  7%     +19.5%    4096012 ?  4%  meminfo.Active(anon)
   6147555 ?  4%     +10.5%    6796087 ?  2%  meminfo.Cached
   1003875 ?  2%     +10.2%    1106188        meminfo.SUnreclaim
   2608589 ?  9%     +24.9%    3257121 ?  5%  meminfo.Shmem
     17.88 ? 34%    +478.3%     103.38 ? 17%  perf-c2c.DRAM.local
      5772 ? 12%    +130.2%      13287 ? 12%  perf-c2c.DRAM.remote
      2298 ? 15%    +353.2%      10416 ?  7%  perf-c2c.HITM.local
     83.50 ? 20%    +191.5%     243.38 ? 10%  perf-c2c.HITM.remote
      2381 ? 15%    +347.6%      10660 ?  7%  perf-c2c.HITM.total
   2792669 ?  7%     +29.4%    3615082 ?  7%  numa-meminfo.node1.Active
   2792669 ?  7%     +29.4%    3615082 ?  7%  numa-meminfo.node1.Active(anon)
    681909 ? 36%     +84.3%    1256571 ? 16%  numa-meminfo.node1.Mapped
    500148 ?  3%     +10.1%     550550        numa-meminfo.node1.SUnreclaim
   2473979 ?  9%     +25.8%    3113467 ?  4%  numa-meminfo.node1.Shmem
    596345 ?  6%      +9.6%     653761 ?  3%  numa-meminfo.node1.Slab
 1.433e+08           -40.9%   84639245 ?  2%  numa-vmstat.node0.numa_hit
 1.432e+08           -41.0%   84520313 ?  2%  numa-vmstat.node0.numa_local
    698292 ?  7%     +29.5%     904000 ?  7%  numa-vmstat.node1.nr_active_anon
    169921 ? 36%     +84.8%     313947 ? 16%  numa-vmstat.node1.nr_mapped
    618687 ?  9%     +25.9%     778630 ?  4%  numa-vmstat.node1.nr_shmem
    698291 ?  7%     +29.5%     904000 ?  7%  numa-vmstat.node1.nr_zone_active_anon
 1.429e+08           -40.2%   85488783 ?  2%  numa-vmstat.node1.numa_hit
 1.428e+08           -40.2%   85419154 ?  2%  numa-vmstat.node1.numa_local
    856589 ?  7%     +19.5%    1023830 ?  4%  proc-vmstat.nr_active_anon
   1536878 ?  4%     +10.5%    1698853 ?  2%  proc-vmstat.nr_file_pages
    652135 ?  9%     +24.8%     814111 ?  5%  proc-vmstat.nr_shmem
    255146 ?  3%      +8.4%     276482        proc-vmstat.nr_slab_unreclaimable
    856589 ?  7%     +19.5%    1023830 ?  4%  proc-vmstat.nr_zone_active_anon
 2.861e+08           -40.6%  1.701e+08 ?  2%  proc-vmstat.numa_hit
 2.859e+08           -40.6%  1.699e+08 ?  2%  proc-vmstat.numa_local
 1.138e+09           -40.8%  6.735e+08 ?  2%  proc-vmstat.pgalloc_normal
 1.137e+09           -40.9%  6.723e+08 ?  2%  proc-vmstat.pgfree
  1.27e+11           -15.8%  1.069e+11        perf-stat.i.branch-instructions
      0.09            +0.0        0.11        perf-stat.i.branch-miss-rate%
  99260313            +9.5%  1.087e+08        perf-stat.i.branch-misses
     67.33           -10.1       57.21        perf-stat.i.cache-miss-rate%
 5.165e+08           -14.9%  4.396e+08        perf-stat.i.cache-misses
     11452           -18.1%       9379        perf-stat.i.context-switches
      0.93           +19.5%       1.11        perf-stat.i.cpi
    497.86           -10.4%     446.33        perf-stat.i.cpu-migrations
      1066           +18.1%       1259        perf-stat.i.cycles-between-cache-misses
 5.916e+11           -15.9%  4.978e+11        perf-stat.i.instructions
      1.07           -16.2%       0.90        perf-stat.i.ipc
      0.08 ?  2%      +0.0        0.10        perf-stat.overall.branch-miss-rate%
     67.48           -10.2       57.26        perf-stat.overall.cache-miss-rate%
      0.93           +19.5%       1.11        perf-stat.overall.cpi
      1068           +18.0%       1260        perf-stat.overall.cycles-between-cache-misses
      1.07           -16.3%       0.90        perf-stat.overall.ipc
   4534964            +3.4%    4689735        perf-stat.overall.path-length
 1.265e+11           -15.8%  1.065e+11        perf-stat.ps.branch-instructions
  97925258            +9.9%  1.077e+08        perf-stat.ps.branch-misses
 5.149e+08           -14.8%  4.386e+08        perf-stat.ps.cache-misses
     11330           -17.6%       9332        perf-stat.ps.context-switches
    480.69            -9.3%     436.17        perf-stat.ps.cpu-migrations
 5.894e+11           -15.9%  4.959e+11        perf-stat.ps.instructions
 1.785e+14           -15.9%  1.502e+14        perf-stat.total.instructions
  21438331           +13.3%   24296162        sched_debug.cfs_rq:/.avg_vruntime.avg
  21903017           +12.4%   24610237        sched_debug.cfs_rq:/.avg_vruntime.max
  14885563 ?  3%     +27.3%   18943306 ?  2%  sched_debug.cfs_rq:/.avg_vruntime.min
    578444 ?  7%     -14.7%     493667 ?  9%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.37 ?  6%     -22.1%       0.29 ?  7%  sched_debug.cfs_rq:/.h_nr_queued.stddev
      0.37 ?  6%     -22.2%       0.28 ?  8%  sched_debug.cfs_rq:/.h_nr_runnable.stddev
   3275315 ? 10%     -49.9%    1642318 ? 31%  sched_debug.cfs_rq:/.left_deadline.avg
  21746330           +12.5%   24470456        sched_debug.cfs_rq:/.left_deadline.max
   7528183 ?  5%     -24.7%    5668652 ? 19%  sched_debug.cfs_rq:/.left_deadline.stddev
   3275271 ? 10%     -49.9%    1642299 ? 31%  sched_debug.cfs_rq:/.left_vruntime.avg
  21746042           +12.5%   24470158        sched_debug.cfs_rq:/.left_vruntime.max
   7528083 ?  5%     -24.7%    5668586 ? 19%  sched_debug.cfs_rq:/.left_vruntime.stddev
    136400 ? 13%     -52.3%      65063 ? 23%  sched_debug.cfs_rq:/.load.avg
    310764 ?  9%     -31.8%     212026 ? 14%  sched_debug.cfs_rq:/.load.stddev
    217.41 ?  6%     -43.5%     122.80 ? 10%  sched_debug.cfs_rq:/.load_avg.avg
     59.25 ? 17%     -49.3%      30.04 ? 33%  sched_debug.cfs_rq:/.load_avg.min
  21438356           +13.3%   24296174        sched_debug.cfs_rq:/.min_vruntime.avg
  21903027           +12.4%   24610242        sched_debug.cfs_rq:/.min_vruntime.max
  14885570 ?  3%     +27.3%   18943335 ?  2%  sched_debug.cfs_rq:/.min_vruntime.min
    578447 ?  7%     -14.7%     493665 ?  9%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.33 ?  7%     -28.0%       0.24 ? 13%  sched_debug.cfs_rq:/.nr_queued.stddev
   3275271 ? 10%     -49.9%    1642299 ? 31%  sched_debug.cfs_rq:/.right_vruntime.avg
  21746042           +12.5%   24470158        sched_debug.cfs_rq:/.right_vruntime.max
   7528083 ?  5%     -24.7%    5668586 ? 19%  sched_debug.cfs_rq:/.right_vruntime.stddev
      1094            -8.7%     998.91        sched_debug.cfs_rq:/.runnable_avg.avg
    170.03 ?  8%     -15.8%     143.13 ?  6%  sched_debug.cfs_rq:/.runnable_avg.stddev
     34.12 ?  7%     -16.9%      28.34 ? 10%  sched_debug.cpu.clock.stddev
    156.83 ? 38%    +104.5%     320.77 ? 31%  sched_debug.cpu.curr->pid.min
      1825 ?  6%     -27.7%       1320 ? 11%  sched_debug.cpu.curr->pid.stddev
      0.00 ? 34%     -37.1%       0.00 ? 12%  sched_debug.cpu.next_balance.stddev
      0.36 ?  6%     -21.0%       0.29 ?  9%  sched_debug.cpu.nr_running.stddev
     10049           -14.9%       8556        sched_debug.cpu.nr_switches.avg
      7617           -20.1%       6084        sched_debug.cpu.nr_switches.min
      7.21 ? 10%     -23.0%       5.55 ? 10%  perf-sched.sch_delay.avg.ms.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region
      7.89 ?  5%     -22.6%       6.10 ?  7%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__mmap_new_vma.__mmap_region.do_mmap
      7.40 ?  6%     -23.5%       5.66 ?  9%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes
      7.53 ?  9%     -21.8%       5.89 ?  8%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
      7.46 ?  6%     -22.4%       5.79 ? 12%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      7.45 ?  6%     -24.1%       5.66 ?  9%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      7.38 ?  6%     -22.9%       5.69 ?  8%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      7.51 ?  8%     -23.3%       5.76 ?  7%  perf-sched.sch_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      7.72 ?  7%     -27.0%       5.64 ?  5%  perf-sched.sch_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      7.51 ?  6%     -23.2%       5.77 ?  7%  perf-sched.sch_delay.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      7.73 ? 10%     -24.5%       5.84 ? 18%  perf-sched.sch_delay.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      5.60 ?  7%     -24.7%       4.22 ?  9%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      5.51 ?  9%     -29.7%       3.87 ?  9%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
      0.22 ?  9%     -22.9%       0.17 ? 16%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      0.14 ? 10%     -20.4%       0.11 ?  9%  perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      0.82 ?  9%     -37.8%       0.51 ? 16%  perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
     24.24 ? 38%     -43.4%      13.72 ? 49%  perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
     21.21 ? 34%     -34.5%      13.89 ? 17%  perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      3.56 ?  5%     -21.9%       2.79 ?  6%  perf-sched.total_sch_delay.average.ms
     51.64 ?  3%     +16.8%      60.30 ?  2%  perf-sched.total_wait_and_delay.average.ms
     57112 ?  3%     -16.6%      47640 ?  2%  perf-sched.total_wait_and_delay.count.ms
     48.07 ?  3%     +19.6%      57.51 ?  2%  perf-sched.total_wait_time.average.ms
     14.42 ? 10%     -23.0%      11.11 ? 10%  perf-sched.wait_and_delay.avg.ms.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region
     15.77 ?  5%     -22.6%      12.20 ?  7%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.__mmap_new_vma.__mmap_region.do_mmap
     14.81 ?  6%     -90.2%       1.45 ?264%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes
     15.06 ?  9%     -21.8%      11.78 ?  8%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
     14.93 ?  6%     -22.4%      11.59 ? 12%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
     14.90 ?  6%     -24.1%      11.31 ?  9%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
     14.77 ?  6%     -22.9%      11.38 ?  8%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
     15.02 ?  8%     -23.3%      11.52 ?  7%  perf-sched.wait_and_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     40.79 ?  3%     +36.8%      55.78 ?  4%  perf-sched.wait_and_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     15.44 ?  7%     -27.0%      11.27 ?  5%  perf-sched.wait_and_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
     15.01 ?  6%     -23.2%      11.53 ?  7%  perf-sched.wait_and_delay.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
     15.47 ? 10%     -24.5%      11.68 ? 18%  perf-sched.wait_and_delay.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
     12.78 ?  6%     -13.0%      11.12 ?  9%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     11.01 ?  9%     -29.7%       7.74 ?  9%  perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     74.48 ?  2%     +28.1%      95.43        perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    467.88 ? 10%     -40.1%     280.12 ?  5%  perf-sched.wait_and_delay.count.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region
      3471 ?  3%    +149.0%       8644 ?  3%  perf-sched.wait_and_delay.count.__cond_resched.down_write.__mmap_new_vma.__mmap_region.do_mmap
    373.88 ?  5%     -91.9%      30.25 ?264%  perf-sched.wait_and_delay.count.__cond_resched.down_write.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes
    459.38 ?  5%     -41.4%     269.25 ?  7%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
    663.88 ?  3%     -41.2%     390.50 ?  8%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
    917.62 ?  5%     -40.0%     550.25 ?  4%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      3824 ?  5%     -37.6%       2387 ?  3%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      1342 ?  5%     -43.6%     757.50 ?  2%  perf-sched.wait_and_delay.count.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      1156 ?  7%     -32.6%     779.12 ?  9%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      5403 ?  4%     -46.2%       2906 ?  3%  perf-sched.wait_and_delay.count.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      3395 ?  4%     -44.0%       1900 ?  4%  perf-sched.wait_and_delay.count.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
    569.38 ?  8%     -37.8%     354.25 ? 14%  perf-sched.wait_and_delay.count.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      5495 ?  5%     -39.5%       3324 ?  8%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     21359 ?  2%     -18.7%      17370        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     75.30 ? 14%     -89.2%       8.11 ?264%  perf-sched.wait_and_delay.max.ms.__cond_resched.down_write.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes
      7.21 ? 10%     -23.0%       5.55 ? 10%  perf-sched.wait_time.avg.ms.__cond_resched.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region
      7.89 ?  5%     -22.6%       6.10 ?  7%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.__mmap_new_vma.__mmap_region.do_mmap
      7.40 ?  6%     -23.5%       5.66 ?  9%  perf-sched.wait_time.avg.ms.__cond_resched.down_write.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes
      7.53 ?  9%     -21.8%       5.89 ?  8%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
      7.46 ?  6%     -22.4%       5.79 ? 12%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      7.45 ?  6%     -24.1%       5.66 ?  9%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      7.38 ?  6%     -22.9%       5.69 ?  8%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      7.51 ?  8%     -23.3%       5.76 ?  7%  perf-sched.wait_time.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     40.71 ?  3%     +36.9%      55.71 ?  4%  perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      7.72 ?  7%     -27.0%       5.64 ?  5%  perf-sched.wait_time.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      7.51 ?  6%     -23.2%       5.77 ?  7%  perf-sched.wait_time.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      7.73 ? 10%     -24.5%       5.84 ? 18%  perf-sched.wait_time.avg.ms.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
      5.51 ?  9%     -29.7%       3.87 ?  9%  perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     74.34 ?  2%     +28.2%      95.32        perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     40.52           -13.5       27.01        perf-profile.calltrace.cycles-pp.__munmap
     13.38           -13.4        0.00        perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
     39.07           -13.2       25.91        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
     38.95           -13.1       25.82        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     38.31           -12.9       25.37        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     38.25           -12.9       25.33        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     37.41           -12.7       24.72        perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.40           -12.4       24.00        perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     10.51           -10.5        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core
     19.56           -10.5        9.06 ?  6%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
     10.39           -10.4        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch
     21.92            -9.0       12.93 ?  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     21.92            -9.0       12.93 ?  5%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
     21.92            -9.0       12.93 ?  5%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
     21.91            -9.0       12.92 ?  5%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     21.90            -9.0       12.91 ?  5%  perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
     21.90            -9.0       12.91 ?  5%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     21.89            -9.0       12.91 ?  5%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
     21.85            -9.0       12.88 ?  5%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
     22.75            -7.2       15.60        perf-profile.calltrace.cycles-pp.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
     19.39            -6.1       13.26        perf-profile.calltrace.cycles-pp.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      5.79            -5.8        0.00        perf-profile.calltrace.cycles-pp.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
      5.55            -5.6        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core
      5.50 ?  2%      -5.5        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch
     12.39            -4.3        8.08        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas
     13.65            -4.3        9.37        perf-profile.calltrace.cycles-pp.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     13.07            -4.1        8.99        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      8.62            -3.8        4.84        perf-profile.calltrace.cycles-pp.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      3.42            -2.7        0.68 ?  2%  perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      3.22 ?  2%      -2.7        0.54 ?  2%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      5.53            -2.6        2.90        perf-profile.calltrace.cycles-pp.__cond_resched.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes
      5.88            -1.7        4.16        perf-profile.calltrace.cycles-pp.perf_event_mmap.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      5.49            -1.6        3.87        perf-profile.calltrace.cycles-pp.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap.vm_mmap_pgoff
      4.88            -1.5        3.34        perf-profile.calltrace.cycles-pp.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      4.96            -1.4        3.60        perf-profile.calltrace.cycles-pp.__get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      4.68            -1.3        3.39        perf-profile.calltrace.cycles-pp.shmem_get_unmapped_area.__get_unmapped_area.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      4.14            -1.2        2.92        perf-profile.calltrace.cycles-pp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      3.72            -1.1        2.58        perf-profile.calltrace.cycles-pp.rcu_all_qs.__cond_resched.zap_pmd_range.unmap_page_range.unmap_vmas
      3.89            -1.1        2.76        perf-profile.calltrace.cycles-pp.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      4.16            -1.1        3.05        perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area.do_mmap.vm_mmap_pgoff
      3.18            -1.0        2.16        perf-profile.calltrace.cycles-pp.free_pgd_range.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      3.14            -1.0        2.14        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes
      2.95            -0.9        2.00        perf-profile.calltrace.cycles-pp.free_p4d_range.free_pgd_range.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas
      3.42            -0.9        2.48        perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area.do_mmap
      3.35            -0.9        2.42        perf-profile.calltrace.cycles-pp.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area.__get_unmapped_area
      2.68            -0.9        1.82        perf-profile.calltrace.cycles-pp.free_pud_range.free_p4d_range.free_pgd_range.free_pgtables.vms_clear_ptes
      4.13            -0.8        3.31        perf-profile.calltrace.cycles-pp.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      1.23            -0.7        0.52        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__munmap
      1.22            -0.7        0.52        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__mmap
      1.97            -0.6        1.32        perf-profile.calltrace.cycles-pp.d_path.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap
      3.64            -0.6        3.09        perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.80 ?  5%      -0.5        0.26 ?100%  perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      3.27            -0.5        2.74        perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap
      1.68            -0.5        1.17        perf-profile.calltrace.cycles-pp.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      1.75            -0.4        1.31        perf-profile.calltrace.cycles-pp.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.shmem_get_unmapped_area
      1.33            -0.4        0.88        perf-profile.calltrace.cycles-pp.prepend_path.d_path.perf_event_mmap_event.perf_event_mmap.__mmap_region
      1.39            -0.4        0.95        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap
      1.34            -0.4        0.94        perf-profile.calltrace.cycles-pp.mas_find.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff
      1.22 ?  3%      -0.4        0.82 ?  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.72            -0.3        0.38 ? 57%  perf-profile.calltrace.cycles-pp.mas_walk.mas_find.__mmap_prepare.__mmap_region.do_mmap
      0.93            -0.3        0.64        perf-profile.calltrace.cycles-pp.unlink_file_vma_batch_process.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      1.19            -0.3        0.90        perf-profile.calltrace.cycles-pp.mas_rev_awalk.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown
      0.86            -0.3        0.59        perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      0.86            -0.3        0.60        perf-profile.calltrace.cycles-pp.shmem_mmap.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      0.81            -0.3        0.56        perf-profile.calltrace.cycles-pp.mas_find.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      1.02            -0.2        0.78        perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap
      0.76            -0.2        0.53        perf-profile.calltrace.cycles-pp.touch_atime.shmem_mmap.__mmap_new_vma.__mmap_region.do_mmap
      0.82            -0.2        0.59        perf-profile.calltrace.cycles-pp.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.75            -0.2        0.53        perf-profile.calltrace.cycles-pp.mas_store_gfp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.78            -0.2        0.56        perf-profile.calltrace.cycles-pp.__vma_start_write.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.72            -0.2        0.52        perf-profile.calltrace.cycles-pp.mas_walk.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      0.76            -0.2        0.58        perf-profile.calltrace.cycles-pp.__kmalloc_cache_noprof.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap
      0.68            -0.1        0.54        perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff
      0.62 ?  4%      -0.1        0.56 ?  2%  perf-profile.calltrace.cycles-pp.mas_put_in_tree.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      0.98            +0.1        1.05 ?  2%  perf-profile.calltrace.cycles-pp.__call_rcu_common.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      1.02            +0.1        1.11        perf-profile.calltrace.cycles-pp.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma.__mmap_region
      0.56 ?  2%      +0.1        0.68 ?  2%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma
      0.46 ? 37%      +0.2        0.62 ?  2%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap
      0.00            +0.6        0.63 ?  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk
      0.00            +0.6        0.63 ?  7%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.6        0.64 ?  7%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.6        0.64 ?  7%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf
      0.00            +0.6        0.64 ?  3%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.6        0.64 ?  3%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main
      0.00            +0.7        0.66 ?  2%  perf-profile.calltrace.cycles-pp.shuffle_freelist.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.7        0.69 ?  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +0.7        0.69 ?  7%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.7        0.69 ?  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +0.7        0.70 ?  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +0.7        0.72 ? 10%  perf-profile.calltrace.cycles-pp.get_partial_node.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.8        0.75 ?  9%  perf-profile.calltrace.cycles-pp.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.8        0.81 ?  6%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.8        0.81 ?  6%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf
      0.00            +0.8        0.82 ?  6%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.8        0.82 ?  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +0.9        0.86 ?  6%  perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.9        0.94 ?  3%  perf-profile.calltrace.cycles-pp.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +1.0        0.99 ? 52%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu
      0.00            +1.1        1.14 ?  7%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free
      0.00            +1.2        1.17 ?  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +1.2        1.20 ?  7%  perf-profile.calltrace.cycles-pp.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      2.19 ?  8%      +1.6        3.75 ?  3%  perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
      0.00            +2.0        2.03 ? 10%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +2.1        2.06 ? 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +2.1        2.08 ?  5%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt
      0.00            +2.1        2.09 ?  4%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +2.1        2.14 ?  9%  perf-profile.calltrace.cycles-pp.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +2.7        2.66 ?  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +2.7        2.68 ?  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +2.8        2.84 ?  3%  perf-profile.calltrace.cycles-pp.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +3.7        3.70 ?  6%  perf-profile.calltrace.cycles-pp.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +3.7        3.71 ? 11%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +3.8        3.83 ?  3%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +3.8        3.84 ? 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +4.9        4.85 ?  9%  perf-profile.calltrace.cycles-pp.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +7.3        7.34 ?  9%  perf-profile.calltrace.cycles-pp.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      0.00            +8.6        8.60 ?  6%  perf-profile.calltrace.cycles-pp.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
      0.00           +12.1       12.10        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +12.2       12.24        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +12.4       12.38        perf-profile.calltrace.cycles-pp.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00           +12.5       12.53        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +12.6       12.57        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +13.5       13.53        perf-profile.calltrace.cycles-pp.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
     36.77           +22.6       59.36        perf-profile.calltrace.cycles-pp.__mmap
     35.27           +22.9       58.22        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
     35.15           +23.0       58.13        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     34.37           +23.2       57.54        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     33.51           +23.5       56.96        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     32.28           +23.8       56.07        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     26.14           +25.3       51.48        perf-profile.calltrace.cycles-pp.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
     13.59           +28.9       42.44        perf-profile.calltrace.cycles-pp.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      4.76           +30.9       35.66        perf-profile.calltrace.cycles-pp.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      3.54           +31.2       34.77        perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap
      3.31           +31.3       34.59        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma.__mmap_region
      0.00           +34.2       34.16        perf-profile.calltrace.cycles-pp.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
     41.12           -13.6       27.48        perf-profile.children.cycles-pp.__munmap
     27.76           -13.3       14.50 ?  3%  perf-profile.children.cycles-pp.kmem_cache_free
     38.33           -12.9       25.38        perf-profile.children.cycles-pp.__x64_sys_munmap
     38.30           -12.9       25.36        perf-profile.children.cycles-pp.__vm_munmap
     37.45           -12.7       24.75        perf-profile.children.cycles-pp.do_vmi_munmap
     36.45           -12.4       24.03        perf-profile.children.cycles-pp.do_vmi_align_munmap
     17.84           -10.8        7.05 ?  6%  perf-profile.children.cycles-pp.__slab_free
     29.63           -10.0       19.58 ?  2%  perf-profile.children.cycles-pp.handle_softirqs
     29.61           -10.0       19.57 ?  2%  perf-profile.children.cycles-pp.rcu_core
     29.60           -10.0       19.56 ?  2%  perf-profile.children.cycles-pp.rcu_do_batch
     21.92            -9.0       12.93 ?  5%  perf-profile.children.cycles-pp.kthread
     21.92            -9.0       12.93 ?  5%  perf-profile.children.cycles-pp.ret_from_fork
     21.92            -9.0       12.93 ?  5%  perf-profile.children.cycles-pp.ret_from_fork_asm
     21.91            -9.0       12.92 ?  5%  perf-profile.children.cycles-pp.smpboot_thread_fn
     21.90            -9.0       12.91 ?  5%  perf-profile.children.cycles-pp.run_ksoftirqd
     22.94            -7.2       15.74        perf-profile.children.cycles-pp.vms_complete_munmap_vmas
     19.44            -6.1       13.30        perf-profile.children.cycles-pp.vms_clear_ptes
      8.10            -4.6        3.53 ?  6%  perf-profile.children.cycles-pp.__put_partials
     13.71            -4.3        9.40        perf-profile.children.cycles-pp.unmap_vmas
     13.11            -4.1        9.01        perf-profile.children.cycles-pp.unmap_page_range
      9.50            -4.0        5.46        perf-profile.children.cycles-pp.mas_store_gfp
     12.41            -3.9        8.52        perf-profile.children.cycles-pp.zap_pmd_range
      5.42 ?  2%      -2.6        2.84 ?  3%  perf-profile.children.cycles-pp.___slab_alloc
      7.14            -2.2        4.92        perf-profile.children.cycles-pp.__cond_resched
      5.91            -1.7        4.18        perf-profile.children.cycles-pp.perf_event_mmap
      5.56            -1.6        3.92        perf-profile.children.cycles-pp.perf_event_mmap_event
      4.98            -1.6        3.41        perf-profile.children.cycles-pp.free_pgtables
      5.00            -1.4        3.63        perf-profile.children.cycles-pp.__get_unmapped_area
      8.46            -1.3        7.15 ?  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      8.43            -1.3        7.12 ?  2%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      4.71            -1.3        3.41        perf-profile.children.cycles-pp.shmem_get_unmapped_area
      4.22            -1.3        2.95        perf-profile.children.cycles-pp.mas_find
      4.29            -1.3        3.02        perf-profile.children.cycles-pp.vms_gather_munmap_vmas
      2.80 ?  4%      -1.2        1.58 ?  7%  perf-profile.children.cycles-pp.get_partial_node
      3.78            -1.2        2.60        perf-profile.children.cycles-pp.rcu_all_qs
      3.99            -1.2        2.84        perf-profile.children.cycles-pp.__mmap_prepare
      7.14            -1.1        6.00        perf-profile.children.cycles-pp.mas_wr_node_store
      4.24            -1.1        3.10        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      7.73            -1.1        6.68 ?  3%  perf-profile.children.cycles-pp.__irq_exit_rcu
      3.20            -1.0        2.18        perf-profile.children.cycles-pp.free_pgd_range
      3.18            -1.0        2.17        perf-profile.children.cycles-pp.zap_pte_range
      3.46            -1.0        2.50        perf-profile.children.cycles-pp.vm_unmapped_area
      2.97            -1.0        2.02        perf-profile.children.cycles-pp.free_p4d_range
      3.41            -0.9        2.46        perf-profile.children.cycles-pp.unmapped_area_topdown
      1.85            -0.9        0.95 ?  2%  perf-profile.children.cycles-pp.allocate_slab
      2.70            -0.9        1.83        perf-profile.children.cycles-pp.free_pud_range
      4.17            -0.8        3.34        perf-profile.children.cycles-pp.mas_store_prealloc
      1.67            -0.8        0.86 ?  2%  perf-profile.children.cycles-pp.shuffle_freelist
      2.02            -0.7        1.36        perf-profile.children.cycles-pp.d_path
      1.71            -0.5        1.18        perf-profile.children.cycles-pp.vm_area_alloc
      1.76            -0.5        1.25        perf-profile.children.cycles-pp.mas_walk
      1.84            -0.5        1.36        perf-profile.children.cycles-pp.mas_empty_area_rev
      1.55            -0.5        1.07        perf-profile.children.cycles-pp.mas_next_slot
      1.36            -0.5        0.90        perf-profile.children.cycles-pp.prepend_path
      1.47            -0.4        1.03        perf-profile.children.cycles-pp.mas_prev_slot
      1.31            -0.4        0.91        perf-profile.children.cycles-pp.mas_wr_store_type
      1.13            -0.3        0.79        perf-profile.children.cycles-pp.__pi_memcpy
      1.44            -0.3        1.11        perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.61            -0.3        0.30 ?  2%  perf-profile.children.cycles-pp.setup_object
      0.83 ?  5%      -0.3        0.52 ?  4%  perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.98            -0.3        0.67        perf-profile.children.cycles-pp.unlink_file_vma_batch_process
      1.23            -0.3        0.93        perf-profile.children.cycles-pp.mas_rev_awalk
      0.89            -0.3        0.61        perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      0.88            -0.3        0.62        perf-profile.children.cycles-pp.prepend_copy
      0.88            -0.3        0.62        perf-profile.children.cycles-pp.shmem_mmap
      1.08            -0.3        0.82        perf-profile.children.cycles-pp.perf_iterate_sb
      1.04            -0.3        0.79        perf-profile.children.cycles-pp.mas_update_gap
      0.70 ?  3%      -0.3        0.45 ?  3%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.71 ?  3%      -0.2        0.46 ?  3%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.88            -0.2        0.63        perf-profile.children.cycles-pp.__vma_start_write
      0.78            -0.2        0.55        perf-profile.children.cycles-pp.touch_atime
      0.59 ?  3%      -0.2        0.36 ?  5%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.58 ?  3%      -0.2        0.35 ?  5%  perf-profile.children.cycles-pp.tick_nohz_handler
      0.74            -0.2        0.52        perf-profile.children.cycles-pp.copy_from_kernel_nofault
      0.74            -0.2        0.51        perf-profile.children.cycles-pp.down_write_killable
      0.55 ?  3%      -0.2        0.33 ?  5%  perf-profile.children.cycles-pp.update_process_times
      0.73            -0.2        0.51        perf-profile.children.cycles-pp.atime_needs_update
      0.67 ?  2%      -0.2        0.46 ?  3%  perf-profile.children.cycles-pp.kfree
      0.85 ?  2%      -0.2        0.64        perf-profile.children.cycles-pp.mas_pop_node
      0.68            -0.2        0.47        perf-profile.children.cycles-pp.up_write
      0.79            -0.2        0.60        perf-profile.children.cycles-pp.__kmalloc_cache_noprof
      0.60            -0.2        0.42        perf-profile.children.cycles-pp.__pte_offset_map_lock
      0.65            -0.2        0.47        perf-profile.children.cycles-pp.down_write
      0.65            -0.2        0.48        perf-profile.children.cycles-pp.mas_prev
      0.72            -0.2        0.56        perf-profile.children.cycles-pp.mas_leaf_max_gap
      0.79            -0.2        0.62        perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.51 ?  2%      -0.2        0.36 ?  2%  perf-profile.children.cycles-pp.strlen
      0.44 ?  2%      -0.1        0.30        perf-profile.children.cycles-pp.fget
      0.65            -0.1        0.50        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.47            -0.1        0.32        perf-profile.children.cycles-pp.remove_vma
      0.46            -0.1        0.32        perf-profile.children.cycles-pp.fput
      0.36 ?  3%      -0.1        0.24 ?  7%  perf-profile.children.cycles-pp.sched_tick
      0.83 ?  3%      -0.1        0.71        perf-profile.children.cycles-pp.mas_put_in_tree
      0.43            -0.1        0.31        perf-profile.children.cycles-pp.up_read
      0.35 ?  2%      -0.1        0.24 ?  2%  perf-profile.children.cycles-pp.mas_wr_store_entry
      0.40            -0.1        0.29        perf-profile.children.cycles-pp.mas_prev_setup
      0.39 ?  3%      -0.1        0.28 ?  2%  perf-profile.children.cycles-pp.current_time
      0.36 ?  2%      -0.1        0.26 ?  2%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.30            -0.1        0.20        perf-profile.children.cycles-pp.tlb_finish_mmu
      0.37            -0.1        0.26        perf-profile.children.cycles-pp.mas_data_end
      0.33            -0.1        0.23        perf-profile.children.cycles-pp.vma_merge_new_range
      0.26            -0.1        0.17        perf-profile.children.cycles-pp.tlb_gather_mmu
      0.35            -0.1        0.26 ?  2%  perf-profile.children.cycles-pp.vma_set_page_prot
      0.30            -0.1        0.21 ?  2%  perf-profile.children.cycles-pp.refill_obj_stock
      0.32 ?  2%      -0.1        0.23 ?  2%  perf-profile.children.cycles-pp.obj_cgroup_charge_account
      0.32            -0.1        0.23        perf-profile.children.cycles-pp.static_key_count
      0.16 ?  2%      -0.1        0.07 ?  4%  perf-profile.children.cycles-pp.alloc_pages_mpol
      0.15 ?  2%      -0.1        0.07 ?  7%  perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
      0.25            -0.1        0.17        perf-profile.children.cycles-pp.__vm_enough_memory
      0.14 ?  3%      -0.1        0.06 ?  6%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.26            -0.1        0.19 ?  2%  perf-profile.children.cycles-pp._raw_spin_lock
      0.25            -0.1        0.17        perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.29 ?  2%      -0.1        0.21 ?  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.26            -0.1        0.18        perf-profile.children.cycles-pp.copy_from_kernel_nofault_allowed
      0.20            -0.1        0.13        perf-profile.children.cycles-pp.unmap_single_vma
      0.23 ?  2%      -0.1        0.16        perf-profile.children.cycles-pp.vm_get_page_prot
      0.25            -0.1        0.18        perf-profile.children.cycles-pp.__account_obj_stock
      0.27            -0.1        0.20        perf-profile.children.cycles-pp.security_mmap_file
      0.21 ?  5%      -0.1        0.14 ?  4%  perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
      0.19 ?  2%      -0.1        0.12 ?  3%  perf-profile.children.cycles-pp.mas_prev_range
      0.23 ?  2%      -0.1        0.17        perf-profile.children.cycles-pp.mas_destroy
      0.20            -0.1        0.14 ?  2%  perf-profile.children.cycles-pp.mas_next_range
      0.15 ?  4%      -0.1        0.09 ?  6%  perf-profile.children.cycles-pp.__free_frozen_pages
      0.25            -0.1        0.19 ?  2%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.41            -0.1        0.35        perf-profile.children.cycles-pp.testcase
      0.19            -0.1        0.13 ?  3%  perf-profile.children.cycles-pp.userfaultfd_unmap_complete
      0.12 ?  5%      -0.1        0.06 ?  7%  perf-profile.children.cycles-pp.free_frozen_page_commit
      0.18            -0.1        0.12 ?  4%  perf-profile.children.cycles-pp.prepend
      0.19 ?  2%      -0.1        0.14 ?  2%  perf-profile.children.cycles-pp.vma_interval_tree_remove
      0.16 ?  2%      -0.1        0.10 ?  4%  perf-profile.children.cycles-pp.___pte_offset_map
      0.18            -0.1        0.13 ?  3%  perf-profile.children.cycles-pp.may_expand_vm
      0.22            -0.1        0.17 ?  2%  perf-profile.children.cycles-pp.vma_mark_detached
      0.21 ?  2%      -0.1        0.16 ?  3%  perf-profile.children.cycles-pp.x64_sys_call
      0.10 ?  4%      -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.18 ?  2%      -0.0        0.13        perf-profile.children.cycles-pp.vm_area_free
      0.14            -0.0        0.10 ?  5%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.13 ?  5%      -0.0        0.09 ?  9%  perf-profile.children.cycles-pp.task_tick_fair
      0.13            -0.0        0.09        perf-profile.children.cycles-pp.can_vma_merge_right
      0.14            -0.0        0.10        perf-profile.children.cycles-pp.ima_file_mmap
      0.12 ?  3%      -0.0        0.08 ?  8%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.13 ?  3%      -0.0        0.09        perf-profile.children.cycles-pp.uprobe_mmap
      0.28            -0.0        0.25        perf-profile.children.cycles-pp.downgrade_write
      0.12 ?  2%      -0.0        0.09 ?  3%  perf-profile.children.cycles-pp.get_align_mask
      0.08 ? 15%      -0.0        0.05 ?  8%  perf-profile.children.cycles-pp.perf_mmap__push
      0.09 ?  7%      -0.0        0.06 ? 11%  perf-profile.children.cycles-pp.update_curr
      0.09 ? 16%      -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.10 ?  3%      -0.0        0.07        perf-profile.children.cycles-pp.get_mmap_base
      0.10 ?  9%      -0.0        0.07 ?  6%  perf-profile.children.cycles-pp.ktime_get
      0.10            -0.0        0.07        perf-profile.children.cycles-pp.mas_node_count_gfp
      0.13 ?  3%      -0.0        0.10 ?  4%  perf-profile.children.cycles-pp.security_mmap_addr
      0.09            -0.0        0.06        perf-profile.children.cycles-pp.unlink_file_vma_batch_add
      0.09            -0.0        0.06        perf-profile.children.cycles-pp.userfaultfd_unmap_prep
      0.10 ? 10%      -0.0        0.07 ?  6%  perf-profile.children.cycles-pp.clockevents_program_event
      0.09 ? 14%      -0.0        0.06 ?  6%  perf-profile.children.cycles-pp.cmd_record
      0.10 ? 13%      -0.0        0.07 ?  7%  perf-profile.children.cycles-pp.handle_internal_command
      0.10 ? 13%      -0.0        0.07 ?  7%  perf-profile.children.cycles-pp.main
      0.13 ?  5%      -0.0        0.10 ?  6%  perf-profile.children.cycles-pp.memfd_check_seals_mmap
      0.10 ? 13%      -0.0        0.07 ?  7%  perf-profile.children.cycles-pp.run_builtin
      0.09 ?  4%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.shmem_mapping
      0.09            -0.0        0.06 ?  7%  perf-profile.children.cycles-pp.vm_stat_account
      0.08 ?  6%      -0.0        0.05        perf-profile.children.cycles-pp.mas_nomem
      0.09            -0.0        0.06 ?  7%  perf-profile.children.cycles-pp.mas_next_setup
      0.07            -0.0        0.05        perf-profile.children.cycles-pp.vma_interval_tree_insert
      0.06 ?  5%      -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.__mt_destroy
      0.06 ?  5%      -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.ksm_vma_flags
      0.07            -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.vma_wants_writenotify
      0.06            -0.0        0.05        perf-profile.children.cycles-pp.mt_free_rcu
      2.05            +0.2        2.21        perf-profile.children.cycles-pp.__call_rcu_common
      1.12            +0.2        1.33 ?  2%  perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.48 ?  7%      +0.3        0.75 ?  9%  perf-profile.children.cycles-pp.get_any_partial
      0.00            +0.4        0.36 ?  4%  perf-profile.children.cycles-pp.build_detached_freelist
      0.00            +2.0        1.97 ?  7%  perf-profile.children.cycles-pp.barn_replace_full_sheaf
      2.94 ?  7%      +2.8        5.77        perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +3.8        3.83 ?  3%  perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.00            +4.0        3.95 ?  6%  perf-profile.children.cycles-pp.barn_replace_empty_sheaf
      0.00            +9.2        9.23 ?  7%  perf-profile.children.cycles-pp.__kmem_cache_free_bulk
     74.44            +9.8       84.21        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     74.22            +9.8       84.04        perf-profile.children.cycles-pp.do_syscall_64
      0.00           +11.3       11.30 ?  4%  perf-profile.children.cycles-pp.__pcs_replace_full_main
      0.00           +12.4       12.38        perf-profile.children.cycles-pp.barn_put_empty_sheaf
      0.00           +13.6       13.56        perf-profile.children.cycles-pp.barn_get_empty_sheaf
     23.24           +15.5       38.70        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     23.51           +15.7       39.18        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     37.39           +22.4       59.83        perf-profile.children.cycles-pp.__mmap
     34.41           +23.2       57.58        perf-profile.children.cycles-pp.ksys_mmap_pgoff
     33.59           +23.4       57.02        perf-profile.children.cycles-pp.vm_mmap_pgoff
     32.35           +23.8       56.12        perf-profile.children.cycles-pp.do_mmap
     26.35           +25.3       51.63        perf-profile.children.cycles-pp.__mmap_region
      8.00           +28.2       36.15        perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
      7.00           +28.5       35.48        perf-profile.children.cycles-pp.mas_alloc_nodes
     13.70           +28.8       42.52        perf-profile.children.cycles-pp.__mmap_new_vma
      4.80           +30.9       35.69        perf-profile.children.cycles-pp.mas_preallocate
      0.00           +34.4       34.41        perf-profile.children.cycles-pp.__pcs_replace_empty_main
      3.92 ?  4%      -1.9        2.06        perf-profile.self.cycles-pp.__slab_free
      3.62            -1.1        2.56        perf-profile.self.cycles-pp.__cond_resched
      3.36            -1.0        2.40        perf-profile.self.cycles-pp.zap_pmd_range
      2.43            -0.7        1.69        perf-profile.self.cycles-pp.free_pud_range
      2.26            -0.7        1.57        perf-profile.self.cycles-pp.zap_pte_range
      2.20            -0.7        1.52        perf-profile.self.cycles-pp.rcu_all_qs
      1.36            -0.6        0.74 ?  2%  perf-profile.self.cycles-pp.shuffle_freelist
      1.54            -0.4        1.11        perf-profile.self.cycles-pp.mas_walk
      1.30            -0.4        0.88        perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      1.32            -0.4        0.94        perf-profile.self.cycles-pp.mas_next_slot
      1.28            -0.4        0.91        perf-profile.self.cycles-pp.mas_prev_slot
      1.24            -0.4        0.87        perf-profile.self.cycles-pp.mas_wr_node_store
      1.14            -0.3        0.81        perf-profile.self.cycles-pp.mas_wr_store_type
      0.60            -0.3        0.27        perf-profile.self.cycles-pp.___slab_alloc
      1.03            -0.3        0.72        perf-profile.self.cycles-pp.mas_find
      1.04            -0.3        0.73        perf-profile.self.cycles-pp.mas_store_gfp
      1.53            -0.3        1.24        perf-profile.self.cycles-pp.__mmap_region
      0.93            -0.3        0.67        perf-profile.self.cycles-pp.__pi_memcpy
      0.92            -0.2        0.68        perf-profile.self.cycles-pp.do_syscall_64
      1.34 ?  4%      -0.2        1.13        perf-profile.self.cycles-pp.__mmap_new_vma
      0.92            -0.2        0.72        perf-profile.self.cycles-pp.mas_rev_awalk
      0.59            -0.2        0.39 ?  2%  perf-profile.self.cycles-pp.prepend_path
      0.57            -0.2        0.38        perf-profile.self.cycles-pp.mas_store_prealloc
      0.47 ?  8%      -0.2        0.28 ?  8%  perf-profile.self.cycles-pp.__memcg_slab_free_hook
      0.74 ?  2%      -0.2        0.55        perf-profile.self.cycles-pp.mas_pop_node
      0.60 ?  3%      -0.2        0.41 ?  2%  perf-profile.self.cycles-pp.perf_event_mmap_event
      0.71            -0.2        0.53        perf-profile.self.cycles-pp.__vma_start_write
      0.75            -0.2        0.57        perf-profile.self.cycles-pp.__call_rcu_common
      0.55            -0.2        0.38        perf-profile.self.cycles-pp.unmapped_area_topdown
      0.57 ?  2%      -0.2        0.40 ?  3%  perf-profile.self.cycles-pp.kfree
      0.52            -0.2        0.35        perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.56            -0.2        0.40        perf-profile.self.cycles-pp.up_write
      0.54            -0.2        0.37        perf-profile.self.cycles-pp.vms_gather_munmap_vmas
      1.00            -0.2        0.84        perf-profile.self.cycles-pp.kmem_cache_free
      0.68            -0.2        0.52        perf-profile.self.cycles-pp.__mmap
      0.57            -0.2        0.41        perf-profile.self.cycles-pp.unmap_page_range
      0.65            -0.2        0.49        perf-profile.self.cycles-pp.__munmap
      0.55            -0.1        0.40        perf-profile.self.cycles-pp.do_vmi_align_munmap
      0.65            -0.1        0.51        perf-profile.self.cycles-pp.mas_leaf_max_gap
      0.64            -0.1        0.50        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.46 ?  3%      -0.1        0.33 ?  2%  perf-profile.self.cycles-pp.strlen
      0.58 ?  2%      -0.1        0.45        perf-profile.self.cycles-pp.mas_preallocate
      0.39 ?  3%      -0.1        0.25 ?  5%  perf-profile.self.cycles-pp.shmem_get_unmapped_area
      0.44            -0.1        0.31        perf-profile.self.cycles-pp.copy_from_kernel_nofault
      0.44            -0.1        0.31        perf-profile.self.cycles-pp.down_write_killable
      0.65 ?  2%      -0.1        0.52 ?  2%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.60            -0.1        0.48        perf-profile.self.cycles-pp.__kmalloc_cache_noprof
      0.38            -0.1        0.26        perf-profile.self.cycles-pp.fget
      0.44            -0.1        0.32        perf-profile.self.cycles-pp.mas_empty_area_rev
      0.20 ?  2%      -0.1        0.09 ?  3%  perf-profile.self.cycles-pp.setup_object
      0.39            -0.1        0.28        perf-profile.self.cycles-pp.fput
      0.39            -0.1        0.28        perf-profile.self.cycles-pp.vms_complete_munmap_vmas
      0.39            -0.1        0.28        perf-profile.self.cycles-pp.down_write
      0.36            -0.1        0.26        perf-profile.self.cycles-pp.__mmap_prepare
      0.31            -0.1        0.21 ?  2%  perf-profile.self.cycles-pp.d_path
      0.54            -0.1        0.44        perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
      0.37            -0.1        0.27        perf-profile.self.cycles-pp.up_read
      0.72 ?  4%      -0.1        0.62        perf-profile.self.cycles-pp.mas_put_in_tree
      0.33            -0.1        0.24        perf-profile.self.cycles-pp.security_vm_enough_memory_mm
      0.37            -0.1        0.28        perf-profile.self.cycles-pp.__vm_munmap
      0.38            -0.1        0.29        perf-profile.self.cycles-pp.mas_alloc_nodes
      0.32            -0.1        0.24 ?  2%  perf-profile.self.cycles-pp.mas_prev
      0.28            -0.1        0.20 ?  2%  perf-profile.self.cycles-pp.mas_wr_store_entry
      0.25            -0.1        0.17        perf-profile.self.cycles-pp.tlb_finish_mmu
      0.24 ?  2%      -0.1        0.16        perf-profile.self.cycles-pp.atime_needs_update
      0.34            -0.1        0.26        perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.30 ?  2%      -0.1        0.22 ?  2%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.29            -0.1        0.21 ?  2%  perf-profile.self.cycles-pp.mas_prev_setup
      0.25            -0.1        0.17        perf-profile.self.cycles-pp.free_pgtables
      0.23            -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.unmap_vmas
      0.29 ?  2%      -0.1        0.21 ?  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.30            -0.1        0.23 ?  2%  perf-profile.self.cycles-pp.perf_event_mmap
      0.22            -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.tlb_gather_mmu
      0.21            -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.vms_clear_ptes
      0.28 ?  2%      -0.1        0.21        perf-profile.self.cycles-pp.mas_data_end
      0.30            -0.1        0.23        perf-profile.self.cycles-pp.vm_mmap_pgoff
      0.23 ?  2%      -0.1        0.16 ?  2%  perf-profile.self.cycles-pp.free_p4d_range
      0.28            -0.1        0.21        perf-profile.self.cycles-pp.mas_update_gap
      0.26            -0.1        0.19        perf-profile.self.cycles-pp.vm_area_alloc
      0.20 ?  2%      -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.unlink_file_vma_batch_process
      0.21 ?  2%      -0.1        0.15 ?  2%  perf-profile.self.cycles-pp._raw_spin_lock
      0.21            -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.copy_from_kernel_nofault_allowed
      0.24            -0.1        0.18        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.19 ?  6%      -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
      0.22            -0.1        0.16        perf-profile.self.cycles-pp.free_pgd_range
      0.20 ?  2%      -0.1        0.14        perf-profile.self.cycles-pp.__pte_offset_map_lock
      0.58            -0.1        0.53 ?  2%  perf-profile.self.cycles-pp.do_mmap
      0.19            -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.18 ?  2%      -0.1        0.13        perf-profile.self.cycles-pp.vm_get_page_prot
      0.20            -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.__account_obj_stock
      0.20            -0.1        0.15        perf-profile.self.cycles-pp.static_key_count
      0.13            -0.0        0.08 ?  4%  perf-profile.self.cycles-pp.__put_partials
      0.16 ?  2%      -0.0        0.11 ?  2%  perf-profile.self.cycles-pp.refill_obj_stock
      0.16 ?  2%      -0.0        0.12 ?  4%  perf-profile.self.cycles-pp.do_vmi_munmap
      0.15 ?  2%      -0.0        0.11 ?  4%  perf-profile.self.cycles-pp.may_expand_vm
      0.16            -0.0        0.12 ?  2%  perf-profile.self.cycles-pp.ksys_mmap_pgoff
      0.17            -0.0        0.13 ?  2%  perf-profile.self.cycles-pp.mas_destroy
      0.16            -0.0        0.12        perf-profile.self.cycles-pp.__get_unmapped_area
      0.14 ?  2%      -0.0        0.10 ?  3%  perf-profile.self.cycles-pp.prepend
      0.13            -0.0        0.09        perf-profile.self.cycles-pp.unmap_single_vma
      0.17            -0.0        0.13        perf-profile.self.cycles-pp.x64_sys_call
      0.15 ?  2%      -0.0        0.11 ?  4%  perf-profile.self.cycles-pp.obj_cgroup_charge_account
      0.18            -0.0        0.14        perf-profile.self.cycles-pp.vma_mark_detached
      0.13 ?  2%      -0.0        0.09        perf-profile.self.cycles-pp.___pte_offset_map
      0.14            -0.0        0.10 ?  3%  perf-profile.self.cycles-pp.vma_merge_new_range
      0.11            -0.0        0.07 ?  4%  perf-profile.self.cycles-pp.unlink_anon_vmas
      0.15 ?  2%      -0.0        0.11        perf-profile.self.cycles-pp.vma_set_page_prot
      0.11            -0.0        0.07 ?  5%  perf-profile.self.cycles-pp.mas_next_range
      0.11 ?  4%      -0.0        0.07        perf-profile.self.cycles-pp.__vm_enough_memory
      0.12 ?  4%      -0.0        0.08 ?  9%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.14 ?  3%      -0.0        0.11        perf-profile.self.cycles-pp.userfaultfd_unmap_complete
      0.16 ?  3%      -0.0        0.13        perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.13            -0.0        0.10 ?  5%  perf-profile.self.cycles-pp.prepend_copy
      0.16 ?  3%      -0.0        0.13 ?  3%  perf-profile.self.cycles-pp.current_time
      0.11 ?  3%      -0.0        0.08 ?  5%  perf-profile.self.cycles-pp.can_vma_merge_right
      0.10 ?  3%      -0.0        0.07        perf-profile.self.cycles-pp.remove_vma
      0.13 ?  2%      -0.0        0.10        perf-profile.self.cycles-pp.vma_interval_tree_remove
      0.08            -0.0        0.05        perf-profile.self.cycles-pp.get_mmap_base
      0.12            -0.0        0.09        perf-profile.self.cycles-pp.security_mmap_file
      0.06            -0.0        0.03 ? 77%  perf-profile.self.cycles-pp.mas_next_setup
      0.09 ? 10%      -0.0        0.06 ?  7%  perf-profile.self.cycles-pp.ktime_get
      0.09 ?  5%      -0.0        0.07 ?  6%  perf-profile.self.cycles-pp.mas_prev_range
      0.24 ?  2%      -0.0        0.21        perf-profile.self.cycles-pp.downgrade_write
      0.07 ?  4%      -0.0        0.05        perf-profile.self.cycles-pp.get_partial_node
      0.08            -0.0        0.06        perf-profile.self.cycles-pp.ima_file_mmap
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.mas_node_count_gfp
      0.08            -0.0        0.06        perf-profile.self.cycles-pp.shmem_mmap
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.unlink_file_vma_batch_add
      0.09            -0.0        0.07        perf-profile.self.cycles-pp.uprobe_mmap
      0.08 ?  4%      -0.0        0.06        perf-profile.self.cycles-pp.security_mmap_addr
      0.07 ?  6%      -0.0        0.06 ?  8%  perf-profile.self.cycles-pp.get_align_mask
      0.07 ?  5%      -0.0        0.06 ?  7%  perf-profile.self.cycles-pp.memfd_check_seals_mmap
      0.06            -0.0        0.05        perf-profile.self.cycles-pp.vm_area_free
      0.00            +0.1        0.05 ?  9%  perf-profile.self.cycles-pp.munmap at plt
      0.00            +0.1        0.06 ?  9%  perf-profile.self.cycles-pp.barn_replace_full_sheaf
      0.00            +0.1        0.07 ?  5%  perf-profile.self.cycles-pp.__pcs_replace_full_main
      0.00            +0.1        0.14 ?  2%  perf-profile.self.cycles-pp.barn_put_empty_sheaf
      0.27 ?  2%      +0.2        0.48        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.22 ?  3%  perf-profile.self.cycles-pp.barn_replace_empty_sheaf
      1.08 ?  2%      +0.2        1.30 ?  2%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.00            +0.3        0.27        perf-profile.self.cycles-pp.barn_get_empty_sheaf
      0.00            +0.4        0.36 ?  3%  perf-profile.self.cycles-pp.build_detached_freelist
      0.00            +0.9        0.92 ?  3%  perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
      2.91 ?  7%      +2.8        5.73        perf-profile.self.cycles-pp.rcu_cblist_dequeue
     23.24           +15.5       38.70        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


***************************************************************************************************
igk-spr-2sp2: 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-9.4/process/100%/debian-12-x86_64-20240206.cgz/igk-spr-2sp2/mmap1/will-it-scale

commit: 
  7c1ca28747 ("tools/testing: Add testing support for slab caches with sheaves")
  71a82407f1 ("maple_tree: use percpu sheaves for maple_node_cache")

7c1ca28747078885 71a82407f1c94903b3835b4c935 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     12753 ?  2%     -24.2%       9672        vmstat.system.cs
     42.77            +1.3%      43.34        boot-time.boot
      7514            +1.5%       7624        boot-time.idle
    504934 ?  3%     +12.8%     569558 ?  2%  numa-meminfo.node0.SUnreclaim
    579032 ?  5%     +13.3%     656147 ?  6%  numa-meminfo.node0.Slab
  43196901           -21.9%   33720164        will-it-scale.192.processes
    224983           -21.9%     175625        will-it-scale.per_process_ops
  43196901           -21.9%   33720164        will-it-scale.workload
      0.66 ?  2%      -0.2        0.45 ?  2%  mpstat.cpu.all.irq%
     30.27           -10.8       19.47 ?  2%  mpstat.cpu.all.soft%
     65.69           +11.6       77.25        mpstat.cpu.all.sys%
      2.91            -0.6        2.26        mpstat.cpu.all.usr%
  1.54e+08           -41.5%   90063139 ?  2%  numa-numastat.node0.local_node
 1.541e+08           -41.5%   90167487 ?  2%  numa-numastat.node0.numa_hit
 1.505e+08 ?  2%     -39.6%   90890723 ?  2%  numa-numastat.node1.local_node
 1.507e+08 ?  2%     -39.6%   90989037 ?  2%  numa-numastat.node1.numa_hit
 1.541e+08           -41.5%   90154776 ?  2%  numa-vmstat.node0.numa_hit
  1.54e+08           -41.5%   90050433 ?  2%  numa-vmstat.node0.numa_local
 1.507e+08 ?  2%     -39.6%   90975360 ?  2%  numa-vmstat.node1.numa_hit
 1.506e+08 ?  2%     -39.6%   90877046 ?  2%  numa-vmstat.node1.numa_local
     27.38 ? 21%    +304.1%     110.62 ? 19%  perf-c2c.DRAM.local
      6940 ? 14%    +109.3%      14525 ? 11%  perf-c2c.DRAM.remote
      3065 ?  9%    +326.8%      13081 ? 12%  perf-c2c.HITM.local
    114.00 ?  9%    +149.9%     284.88 ? 16%  perf-c2c.HITM.remote
      3179 ?  9%    +320.5%      13366 ? 12%  perf-c2c.HITM.total
   3406339 ?  4%     +18.2%    4026112 ?  7%  meminfo.Active
   3406339 ?  4%     +18.2%    4026112 ?  7%  meminfo.Active(anon)
   6098973 ?  2%     +10.0%    6710127 ?  4%  meminfo.Cached
  30032595           +10.5%   33186524        meminfo.Committed_AS
   1011475 ?  2%     +12.2%    1134788 ?  3%  meminfo.SUnreclaim
   2560004 ?  6%     +23.9%    3171161 ?  8%  meminfo.Shmem
   1181332           +10.6%    1306100 ?  3%  meminfo.Slab
    851570 ?  4%     +18.2%    1006576 ?  7%  proc-vmstat.nr_active_anon
   1524757 ?  2%     +10.0%    1677652 ?  4%  proc-vmstat.nr_file_pages
    640014 ?  6%     +23.9%     792909 ?  8%  proc-vmstat.nr_shmem
    258590 ?  2%      +9.2%     282356        proc-vmstat.nr_slab_unreclaimable
    851570 ?  4%     +18.2%    1006576 ?  7%  proc-vmstat.nr_zone_active_anon
 3.048e+08           -40.6%  1.812e+08 ?  2%  proc-vmstat.numa_hit
 3.046e+08           -40.6%   1.81e+08 ?  2%  proc-vmstat.numa_local
 1.213e+09           -40.8%   7.18e+08 ?  2%  proc-vmstat.pgalloc_normal
 1.212e+09           -40.9%  7.169e+08 ?  2%  proc-vmstat.pgfree
      1.00            +2.1%       1.02        perf-stat.i.MPKI
 1.205e+11           -19.9%  9.651e+10        perf-stat.i.branch-instructions
      0.10            +0.0        0.13        perf-stat.i.branch-miss-rate%
 1.041e+08            +7.3%  1.117e+08        perf-stat.i.branch-misses
     67.15            -9.2       58.00        perf-stat.i.cache-miss-rate%
 5.616e+08           -18.5%  4.579e+08        perf-stat.i.cache-misses
 8.353e+08            -5.6%  7.889e+08        perf-stat.i.cache-references
     12735 ?  2%     -24.3%       9645        perf-stat.i.context-switches
      0.98           +25.7%       1.23        perf-stat.i.cpi
    568.41 ?  2%     -24.5%     429.39        perf-stat.i.cpu-migrations
    982.40           +23.1%       1209        perf-stat.i.cycles-between-cache-misses
  5.66e+11           -20.2%  4.518e+11        perf-stat.i.instructions
      1.03           -20.4%       0.82        perf-stat.i.ipc
      0.99            +2.1%       1.02        perf-stat.overall.MPKI
      0.09            +0.0        0.12        perf-stat.overall.branch-miss-rate%
     67.31            -9.3       58.05        perf-stat.overall.cache-miss-rate%
      0.98           +25.7%       1.23        perf-stat.overall.cpi
    982.46           +23.2%       1209        perf-stat.overall.cycles-between-cache-misses
      1.02           -20.5%       0.81        perf-stat.overall.ipc
   3952600            +2.2%    4040369        perf-stat.overall.path-length
 1.201e+11           -19.9%  9.614e+10        perf-stat.ps.branch-instructions
 1.029e+08            +7.6%  1.107e+08        perf-stat.ps.branch-misses
 5.608e+08           -18.5%   4.57e+08        perf-stat.ps.cache-misses
 8.332e+08            -5.5%  7.873e+08        perf-stat.ps.cache-references
     12643 ?  2%     -24.2%       9588        perf-stat.ps.context-switches
    554.75 ?  2%     -24.3%     419.88        perf-stat.ps.cpu-migrations
 5.637e+11           -20.2%  4.501e+11        perf-stat.ps.instructions
 1.707e+14           -20.2%  1.362e+14        perf-stat.total.instructions
  20670046           +16.4%   24055419        sched_debug.cfs_rq:/.avg_vruntime.avg
  21034792           +15.8%   24354538        sched_debug.cfs_rq:/.avg_vruntime.max
  14918360 ?  2%     +25.7%   18758523 ?  3%  sched_debug.cfs_rq:/.avg_vruntime.min
      0.37 ?  2%     -21.8%       0.29 ?  6%  sched_debug.cfs_rq:/.h_nr_queued.stddev
      0.37 ?  2%     -21.8%       0.29 ?  5%  sched_debug.cfs_rq:/.h_nr_runnable.stddev
   3188415 ? 13%     -54.4%    1453374 ? 28%  sched_debug.cfs_rq:/.left_deadline.avg
  20941830           +15.8%   24260743        sched_debug.cfs_rq:/.left_deadline.max
   7405402 ?  5%     -26.1%    5473882 ? 16%  sched_debug.cfs_rq:/.left_deadline.stddev
   3188370 ? 13%     -54.4%    1453356 ? 28%  sched_debug.cfs_rq:/.left_vruntime.avg
  20941509           +15.8%   24260483        sched_debug.cfs_rq:/.left_vruntime.max
   7405297 ?  5%     -26.1%    5473817 ? 16%  sched_debug.cfs_rq:/.left_vruntime.stddev
    138019 ?  9%     -56.1%      60630 ? 17%  sched_debug.cfs_rq:/.load.avg
    313107 ?  4%     -33.0%     209813 ? 12%  sched_debug.cfs_rq:/.load.stddev
    219.64 ?  5%     -40.9%     129.85 ? 17%  sched_debug.cfs_rq:/.load_avg.avg
     74.52 ? 22%     -61.8%      28.48 ? 11%  sched_debug.cfs_rq:/.load_avg.min
  20670074           +16.4%   24055432        sched_debug.cfs_rq:/.min_vruntime.avg
  21034799           +15.8%   24354550        sched_debug.cfs_rq:/.min_vruntime.max
  14918366 ?  2%     +25.7%   18758541 ?  3%  sched_debug.cfs_rq:/.min_vruntime.min
      0.34 ?  4%     -29.2%       0.24 ? 10%  sched_debug.cfs_rq:/.nr_queued.stddev
   3188371 ? 13%     -54.4%    1453357 ? 28%  sched_debug.cfs_rq:/.right_vruntime.avg
  20941509           +15.8%   24260486        sched_debug.cfs_rq:/.right_vruntime.max
   7405298 ?  5%     -26.1%    5473818 ? 16%  sched_debug.cfs_rq:/.right_vruntime.stddev
    103.28 ? 10%     -25.9%      76.56 ? 15%  sched_debug.cfs_rq:/.util_avg.stddev
    410551 ?  8%     +14.9%     471821 ?  7%  sched_debug.cpu.avg_idle.stddev
      1368 ?  6%     -21.8%       1070        sched_debug.cpu.clock_task.stddev
    169.54 ? 26%     +78.8%     303.17 ? 23%  sched_debug.cpu.curr->pid.min
      1871 ?  3%     -30.6%       1298 ?  8%  sched_debug.cpu.curr->pid.stddev
      0.37 ?  2%     -22.1%       0.29 ?  6%  sched_debug.cpu.nr_running.stddev
     11054 ?  2%     -21.2%       8711        sched_debug.cpu.nr_switches.avg
     65701 ?  6%     -23.2%      50488 ? 18%  sched_debug.cpu.nr_switches.max
      8399           -26.1%       6210        sched_debug.cpu.nr_switches.min
      5964 ?  8%     -26.2%       4402 ? 14%  sched_debug.cpu.nr_switches.stddev
      6.81 ?  6%     -16.1%       5.72 ?  8%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
      6.79 ?  6%     -18.4%       5.54 ?  9%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.69 ?  6%     -15.2%       5.67 ?  5%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      6.86 ?  7%     -19.0%       5.56 ?  5%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      6.90 ?  8%     -18.8%       5.60 ?  9%  perf-sched.sch_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      5.58 ? 29%     -51.7%       2.70 ? 28%  perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part
      7.14 ?  7%     -22.8%       5.52 ?  6%  perf-sched.sch_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      6.83 ?  6%     -16.8%       5.68 ?  7%  perf-sched.sch_delay.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      0.06 ? 42%     -64.1%       0.02 ? 43%  perf-sched.sch_delay.avg.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
      6.72 ?  7%     -17.5%       5.55 ?  7%  perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
      0.23 ? 17%     -37.0%       0.15 ? 22%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
     60.90 ? 36%     -42.8%      34.83 ? 38%  perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      3.32 ?  6%     -20.1%       2.66 ?  6%  perf-sched.total_sch_delay.average.ms
     46.22 ?  3%     +24.5%      57.55 ?  2%  perf-sched.total_wait_and_delay.average.ms
     64361 ?  3%     -22.5%      49858 ?  2%  perf-sched.total_wait_and_delay.count.ms
     42.90 ?  3%     +28.0%      54.89 ?  2%  perf-sched.total_wait_time.average.ms
     13.63 ?  6%     -16.1%      11.43 ?  8%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
     13.58 ?  6%     -18.4%      11.08 ?  9%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.39 ?  6%     -15.2%      11.35 ?  5%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
     13.73 ?  7%     -19.0%      11.12 ?  5%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
     13.80 ?  8%     -18.8%      11.21 ?  9%  perf-sched.wait_and_delay.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     35.08 ?  7%     +47.8%      51.83 ?  5%  perf-sched.wait_and_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     14.28 ?  7%     -22.8%      11.03 ?  6%  perf-sched.wait_and_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
     13.66 ?  6%     -16.8%      11.36 ?  7%  perf-sched.wait_and_delay.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
     64.08 ?  3%     +39.7%      89.53 ?  2%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    592.25 ?  4%     -48.5%     305.00 ?  5%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
    683.62 ?  6%     -48.1%     355.12 ?  6%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1123 ?  2%     -51.0%     550.75 ?  6%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      4737 ?  3%     -47.0%       2509 ?  2%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      2064 ?  4%     -52.2%     986.62 ?  2%  perf-sched.wait_and_delay.count.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      1323 ?  7%     -44.5%     734.50 ? 13%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      7292 ?  3%     -55.1%       3273 ?  3%  perf-sched.wait_and_delay.count.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      3604 ?  4%     -52.2%       1724 ?  4%  perf-sched.wait_and_delay.count.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      9502 ?  4%     +39.4%      13249 ?  4%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1129 ?  6%     -46.7%     601.50 ?  2%  perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
     24865 ?  3%     -25.6%      18503        perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
    121.79 ? 36%     -42.8%      69.65 ? 38%  perf-sched.wait_and_delay.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.81 ?  6%     -16.1%       5.72 ?  8%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
      6.79 ?  6%     -18.4%       5.54 ?  9%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.69 ?  6%     -15.2%       5.67 ?  5%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
      6.86 ?  7%     -19.0%       5.56 ?  5%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      6.90 ?  8%     -18.8%       5.60 ?  9%  perf-sched.wait_time.avg.ms.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
     35.01 ?  7%     +47.9%      51.79 ?  5%  perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      7.14 ?  7%     -22.8%       5.52 ?  6%  perf-sched.wait_time.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      6.83 ?  6%     -16.8%       5.68 ?  7%  perf-sched.wait_time.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range
      6.99 ?  6%     -16.9%       5.81 ?  8%  perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     63.95 ?  3%     +39.8%      89.43 ?  2%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     60.90 ? 36%     -42.8%      34.83 ? 38%  perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     41.87           -16.2       25.66        perf-profile.calltrace.cycles-pp.__munmap
     40.22           -15.7       24.49        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
     40.08           -15.7       24.39        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     39.34           -15.4       23.92        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     39.27           -15.4       23.88        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
     38.32           -15.1       23.25        perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
     37.00           -14.7       22.32        perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
     14.06           -13.9        0.13 ?173%  perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
     20.66           -11.7        8.93 ?  5%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
     10.94           -10.9        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core
     10.81           -10.8        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch
     23.30           -10.0       13.30 ?  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     23.30           -10.0       13.30 ?  5%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
     23.30           -10.0       13.30 ?  5%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
     23.29           -10.0       13.29 ?  5%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     23.28           -10.0       13.28 ?  5%  perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
     23.28           -10.0       13.28 ?  5%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     23.28           -10.0       13.28 ?  5%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
     23.23           -10.0       13.25 ?  5%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
     20.58            -7.4       13.20        perf-profile.calltrace.cycles-pp.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      6.16            -6.2        0.00        perf-profile.calltrace.cycles-pp.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
     17.09            -6.1       10.95        perf-profile.calltrace.cycles-pp.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      5.91            -5.9        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core
      5.86            -5.9        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch
     10.06            -5.1        4.99        perf-profile.calltrace.cycles-pp.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
     11.53            -4.1        7.40        perf-profile.calltrace.cycles-pp.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      9.99            -4.1        5.93        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas
     10.85            -3.9        6.94        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      4.22 ?  2%      -3.6        0.67 ?  3%  perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      4.02 ?  3%      -3.5        0.54 ?  4%  perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      6.10            -3.1        2.96        perf-profile.calltrace.cycles-pp.__cond_resched.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes
      5.33            -1.9        3.44        perf-profile.calltrace.cycles-pp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      5.29            -1.8        3.53        perf-profile.calltrace.cycles-pp.__get_unmapped_area.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      4.91            -1.6        3.27        perf-profile.calltrace.cycles-pp.thp_get_unmapped_area_vmflags.__get_unmapped_area.do_mmap.vm_mmap_pgoff.do_syscall_64
      4.60            -1.6        2.98        perf-profile.calltrace.cycles-pp.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      4.54            -1.5        3.03        perf-profile.calltrace.cycles-pp.arch_get_unmapped_area_topdown.thp_get_unmapped_area_vmflags.__get_unmapped_area.do_mmap.vm_mmap_pgoff
      4.48            -1.5        3.02        perf-profile.calltrace.cycles-pp.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      4.08            -1.5        2.63        perf-profile.calltrace.cycles-pp.rcu_all_qs.__cond_resched.zap_pmd_range.unmap_page_range.unmap_vmas
      3.72            -1.3        2.42        perf-profile.calltrace.cycles-pp.free_pgd_range.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      3.76            -1.3        2.46        perf-profile.calltrace.cycles-pp.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area_vmflags.__get_unmapped_area.do_mmap
      3.67            -1.3        2.40        perf-profile.calltrace.cycles-pp.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area_vmflags.__get_unmapped_area
      3.38            -1.2        2.20        perf-profile.calltrace.cycles-pp.free_p4d_range.free_pgd_range.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas
      5.26            -1.1        4.13 ?  2%  perf-profile.calltrace.cycles-pp.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      3.07            -1.1        2.00        perf-profile.calltrace.cycles-pp.free_pud_range.free_p4d_range.free_pgd_range.free_pgtables.vms_clear_ptes
      2.89            -1.1        1.82        perf-profile.calltrace.cycles-pp.perf_event_mmap.__mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      2.41            -0.9        1.52        perf-profile.calltrace.cycles-pp.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap.vm_mmap_pgoff
      4.03            -0.9        3.17 ?  2%  perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      1.36            -0.8        0.54        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__munmap
      1.35            -0.8        0.54        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__mmap
      4.18            -0.8        3.43 ?  2%  perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma.__mmap_region.do_mmap
      1.89            -0.7        1.22        perf-profile.calltrace.cycles-pp.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      1.92            -0.6        1.30        perf-profile.calltrace.cycles-pp.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown.thp_get_unmapped_area_vmflags
      1.54            -0.6        0.99        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region.do_mmap
      1.52            -0.5        1.00        perf-profile.calltrace.cycles-pp.mas_find.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff
      1.38 ?  2%      -0.5        0.89 ?  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      1.28            -0.5        0.83        perf-profile.calltrace.cycles-pp.mas_rev_awalk.mas_empty_area_rev.unmapped_area_topdown.vm_unmapped_area.arch_get_unmapped_area_topdown
      0.86 ?  4%      -0.4        0.47 ? 38%  perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap
      1.08            -0.4        0.70        perf-profile.calltrace.cycles-pp.mas_prev_slot.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      1.03            -0.4        0.65        perf-profile.calltrace.cycles-pp.mas_find.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.96            -0.4        0.60 ?  2%  perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.kmem_cache_alloc_noprof.vm_area_alloc.__mmap_new_vma.__mmap_region
      1.09            -0.4        0.74        perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.__mmap_region.do_mmap
      1.10            -0.3        0.79        perf-profile.calltrace.cycles-pp.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.85            -0.3        0.55        perf-profile.calltrace.cycles-pp.mas_store_gfp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.81            -0.3        0.52        perf-profile.calltrace.cycles-pp.mas_wr_store_type.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.84            -0.3        0.55        perf-profile.calltrace.cycles-pp.__vma_start_write.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap
      0.81            -0.3        0.53        perf-profile.calltrace.cycles-pp.mas_walk.mas_find.__mmap_prepare.__mmap_region.do_mmap
      0.99            -0.3        0.72        perf-profile.calltrace.cycles-pp.mas_walk.mas_find.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
      0.90            -0.2        0.68        perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.__mmap_prepare.__mmap_region.do_mmap.vm_mmap_pgoff
      1.06 ?  3%      -0.2        0.89 ?  3%  perf-profile.calltrace.cycles-pp.mas_pop_node.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma.__mmap_region
      0.69 ?  4%      -0.1        0.56 ?  3%  perf-profile.calltrace.cycles-pp.mas_put_in_tree.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.do_vmi_munmap
      0.55 ?  3%      +0.1        0.66 ?  5%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap
      1.04            +0.1        1.17 ?  3%  perf-profile.calltrace.cycles-pp.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma.__mmap_region
      0.52 ?  3%      +0.2        0.71 ?  4%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.__mmap_new_vma
      0.00            +0.7        0.66 ?  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.get_any_partial.___slab_alloc
      0.00            +0.7        0.66 ? 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk
      0.00            +0.7        0.68 ?  3%  perf-profile.calltrace.cycles-pp.shuffle_freelist.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.7        0.74 ?  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk
      0.00            +0.8        0.75 ? 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.8        0.79 ?  3%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main
      0.00            +0.8        0.80 ?  3%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.8        0.80 ?  4%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf
      0.00            +0.8        0.80 ?  4%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.8        0.84 ?  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +0.8        0.85 ?  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +0.9        0.85 ?  4%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.9        0.86 ?  4%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +0.9        0.86 ?  9%  perf-profile.calltrace.cycles-pp.get_partial_node.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +0.9        0.90 ?  9%  perf-profile.calltrace.cycles-pp.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.96 ? 38%      +0.9        1.88 ?  5%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu
      0.00            +0.9        0.93 ?  8%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.9        0.93 ?  8%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf
      0.00            +0.9        0.93 ?  8%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.9        0.94 ?  8%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +1.0        0.97 ?  3%  perf-profile.calltrace.cycles-pp.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +1.0        1.02 ?  9%  perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +1.2        1.22 ?  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free
      0.00            +1.2        1.24 ?  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +1.3        1.28 ?  8%  perf-profile.calltrace.cycles-pp.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      1.18 ?  5%      +1.3        2.51 ?  5%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt
      1.18 ?  5%      +1.3        2.52 ?  5%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      2.46 ?  6%      +1.8        4.25 ?  6%  perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
      0.00            +1.9        1.90 ?  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +1.9        1.93 ?  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +2.0        2.01 ?  9%  perf-profile.calltrace.cycles-pp.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +3.1        3.12 ?  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +3.1        3.14 ?  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +3.2        3.19 ?  4%  perf-profile.calltrace.cycles-pp.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +3.4        3.44 ? 11%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +3.6        3.59 ? 10%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +4.2        4.25 ?  4%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +4.3        4.28 ?  6%  perf-profile.calltrace.cycles-pp.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +4.7        4.71 ?  8%  perf-profile.calltrace.cycles-pp.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +7.1        7.10 ?  8%  perf-profile.calltrace.cycles-pp.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      0.00            +8.4        8.45 ?  5%  perf-profile.calltrace.cycles-pp.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
      0.00           +14.1       14.13        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +14.3       14.26        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +14.4       14.36 ?  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +14.4       14.39 ?  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +14.4       14.41        perf-profile.calltrace.cycles-pp.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00           +15.5       15.52 ?  2%  perf-profile.calltrace.cycles-pp.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
     34.15           +26.3       60.47        perf-profile.calltrace.cycles-pp.__mmap
     32.51           +26.8       59.34        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
     32.37           +26.9       59.24        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     31.31           +27.2       58.52        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     29.98           +27.6       57.63        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
     23.73           +29.7       53.40        perf-profile.calltrace.cycles-pp.__mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
     13.35           +33.1       46.47        perf-profile.calltrace.cycles-pp.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff.do_syscall_64
      5.65 ?  2%     +35.1       40.75        perf-profile.calltrace.cycles-pp.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap.vm_mmap_pgoff
      4.34 ?  2%     +35.6       39.91        perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_preallocate.__mmap_new_vma.__mmap_region.do_mmap
      4.11 ?  3%     +35.7       39.77        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma.__mmap_region
      0.00           +39.3       39.34        perf-profile.calltrace.cycles-pp.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__mmap_new_vma
     42.51           -16.4       26.11        perf-profile.children.cycles-pp.__munmap
     30.23           -15.6       14.63 ?  3%  perf-profile.children.cycles-pp.kmem_cache_free
     39.36           -15.4       23.93        perf-profile.children.cycles-pp.__x64_sys_munmap
     39.31           -15.4       23.91        perf-profile.children.cycles-pp.__vm_munmap
     38.36           -15.1       23.28        perf-profile.children.cycles-pp.do_vmi_munmap
     37.05           -14.7       22.35        perf-profile.children.cycles-pp.do_vmi_align_munmap
     19.26           -12.2        7.04 ?  5%  perf-profile.children.cycles-pp.__slab_free
     32.35           -12.0       20.33 ?  2%  perf-profile.children.cycles-pp.handle_softirqs
     32.33           -12.0       20.32 ?  2%  perf-profile.children.cycles-pp.rcu_core
     32.32           -12.0       20.31 ?  2%  perf-profile.children.cycles-pp.rcu_do_batch
     23.30           -10.0       13.30 ?  5%  perf-profile.children.cycles-pp.kthread
     23.30           -10.0       13.30 ?  5%  perf-profile.children.cycles-pp.ret_from_fork
     23.30           -10.0       13.30 ?  5%  perf-profile.children.cycles-pp.ret_from_fork_asm
     23.29           -10.0       13.29 ?  5%  perf-profile.children.cycles-pp.smpboot_thread_fn
     23.28           -10.0       13.28 ?  5%  perf-profile.children.cycles-pp.run_ksoftirqd
     20.80            -7.5       13.34        perf-profile.children.cycles-pp.vms_complete_munmap_vmas
     17.15            -6.2       10.98        perf-profile.children.cycles-pp.vms_clear_ptes
      8.92 ?  2%      -5.5        3.45 ?  6%  perf-profile.children.cycles-pp.__put_partials
     11.04            -5.4        5.64        perf-profile.children.cycles-pp.mas_store_gfp
     11.60            -4.2        7.44        perf-profile.children.cycles-pp.unmap_vmas
     10.90            -3.9        6.97        perf-profile.children.cycles-pp.unmap_page_range
      6.84 ?  3%      -3.6        3.19 ?  4%  perf-profile.children.cycles-pp.___slab_alloc
     10.02            -3.6        6.39        perf-profile.children.cycles-pp.zap_pmd_range
      7.52            -2.7        4.80        perf-profile.children.cycles-pp.__cond_resched
      9.72 ?  2%      -2.3        7.46 ?  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      9.68 ?  2%      -2.3        7.43 ?  3%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      3.94 ?  6%      -2.1        1.88 ?  9%  perf-profile.children.cycles-pp.get_partial_node
      9.06 ?  2%      -2.0        7.05 ?  3%  perf-profile.children.cycles-pp.__irq_exit_rcu
      5.51            -2.0        3.55        perf-profile.children.cycles-pp.vms_gather_munmap_vmas
      5.34            -1.8        3.56        perf-profile.children.cycles-pp.__get_unmapped_area
      5.11            -1.7        3.37        perf-profile.children.cycles-pp.mas_find
      8.47            -1.7        6.78 ?  2%  perf-profile.children.cycles-pp.mas_wr_node_store
      4.70            -1.7        3.04        perf-profile.children.cycles-pp.free_pgtables
      4.94            -1.7        3.28        perf-profile.children.cycles-pp.thp_get_unmapped_area_vmflags
      4.59            -1.5        3.07        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      4.59            -1.5        3.09        perf-profile.children.cycles-pp.__mmap_prepare
      3.95            -1.4        2.53        perf-profile.children.cycles-pp.rcu_all_qs
      3.76            -1.3        2.44        perf-profile.children.cycles-pp.free_pgd_range
      3.79            -1.3        2.49        perf-profile.children.cycles-pp.vm_unmapped_area
      3.73            -1.3        2.44        perf-profile.children.cycles-pp.unmapped_area_topdown
      3.40            -1.2        2.21        perf-profile.children.cycles-pp.free_p4d_range
      5.31            -1.1        4.16 ?  2%  perf-profile.children.cycles-pp.mas_store_prealloc
      2.92            -1.1        1.84        perf-profile.children.cycles-pp.perf_event_mmap
      3.08            -1.1        2.00        perf-profile.children.cycles-pp.free_pud_range
      2.02 ?  2%      -1.1        0.97 ?  3%  perf-profile.children.cycles-pp.allocate_slab
      1.83 ?  2%      -0.9        0.89 ?  3%  perf-profile.children.cycles-pp.shuffle_freelist
      2.53            -0.9        1.59        perf-profile.children.cycles-pp.perf_event_mmap_event
      1.99            -0.7        1.27        perf-profile.children.cycles-pp.mas_prev_slot
      2.17            -0.7        1.49        perf-profile.children.cycles-pp.mas_walk
      1.92            -0.7        1.24        perf-profile.children.cycles-pp.vm_area_alloc
      1.85            -0.7        1.18        perf-profile.children.cycles-pp.mas_next_slot
      2.02            -0.7        1.36        perf-profile.children.cycles-pp.mas_empty_area_rev
      1.47            -0.5        0.94        perf-profile.children.cycles-pp.mas_wr_store_type
      1.33            -0.5        0.86        perf-profile.children.cycles-pp.mas_rev_awalk
      1.59            -0.4        1.14        perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.25            -0.4        0.82        perf-profile.children.cycles-pp.__pi_memcpy
      1.19            -0.4        0.79        perf-profile.children.cycles-pp.mas_update_gap
      1.15            -0.4        0.77        perf-profile.children.cycles-pp.perf_iterate_sb
      0.99            -0.4        0.62 ?  2%  perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      0.66 ?  2%      -0.4        0.31 ?  3%  perf-profile.children.cycles-pp.setup_object
      0.90 ?  4%      -0.3        0.55 ?  5%  perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.96            -0.3        0.64        perf-profile.children.cycles-pp.__vma_start_write
      1.54 ?  3%      -0.3        1.22 ?  3%  perf-profile.children.cycles-pp.mas_pop_node
      0.82 ?  3%      -0.3        0.54 ?  2%  perf-profile.children.cycles-pp.down_write_killable
      0.84            -0.3        0.56        perf-profile.children.cycles-pp.mas_leaf_max_gap
      1.02            -0.3        0.76        perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.72            -0.2        0.47        perf-profile.children.cycles-pp.mas_prev
      0.64 ?  3%      -0.2        0.40 ?  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.63 ?  3%      -0.2        0.40 ?  5%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.75            -0.2        0.54        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.91 ?  3%      -0.2        0.70 ?  3%  perf-profile.children.cycles-pp.mas_put_in_tree
      0.53 ?  3%      -0.2        0.32 ?  7%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.52 ?  4%      -0.2        0.31 ?  6%  perf-profile.children.cycles-pp.tick_nohz_handler
      0.50 ?  3%      -0.2        0.29 ?  7%  perf-profile.children.cycles-pp.update_process_times
      0.54            -0.2        0.36        perf-profile.children.cycles-pp.mas_prev_setup
      0.42            -0.2        0.26        perf-profile.children.cycles-pp.mas_prev_range
      0.49            -0.2        0.33        perf-profile.children.cycles-pp.up_read
      0.43            -0.1        0.29 ?  2%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.41            -0.1        0.26        perf-profile.children.cycles-pp.mas_wr_store_entry
      0.41 ?  2%      -0.1        0.26        perf-profile.children.cycles-pp.mas_data_end
      0.34            -0.1        0.21        perf-profile.children.cycles-pp.tlb_finish_mmu
      0.36 ?  3%      -0.1        0.23 ?  4%  perf-profile.children.cycles-pp.obj_cgroup_charge_account
      0.31            -0.1        0.18 ?  2%  perf-profile.children.cycles-pp.sized_strscpy
      0.34            -0.1        0.22 ?  3%  perf-profile.children.cycles-pp.refill_obj_stock
      0.29            -0.1        0.16 ?  2%  perf-profile.children.cycles-pp.tlb_gather_mmu
      0.39            -0.1        0.26 ?  2%  perf-profile.children.cycles-pp.static_key_count
      0.47            -0.1        0.36        perf-profile.children.cycles-pp.vma_set_page_prot
      0.33            -0.1        0.21 ?  2%  perf-profile.children.cycles-pp.vma_merge_new_range
      0.32 ?  3%      -0.1        0.21 ?  8%  perf-profile.children.cycles-pp.sched_tick
      0.30            -0.1        0.19        perf-profile.children.cycles-pp.kfree
      0.28            -0.1        0.18 ?  2%  perf-profile.children.cycles-pp.__account_obj_stock
      0.28            -0.1        0.18        perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.28            -0.1        0.18 ?  2%  perf-profile.children.cycles-pp.up_write
      0.17            -0.1        0.07 ?  5%  perf-profile.children.cycles-pp.alloc_pages_mpol
      0.16 ?  2%      -0.1        0.07 ?  4%  perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
      0.15 ?  2%      -0.1        0.06 ?  6%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.26 ?  2%      -0.1        0.17 ?  3%  perf-profile.children.cycles-pp.vm_get_page_prot
      0.29            -0.1        0.20        perf-profile.children.cycles-pp.__vm_enough_memory
      0.29            -0.1        0.21 ?  2%  perf-profile.children.cycles-pp.downgrade_write
      0.28            -0.1        0.19        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.22 ?  2%      -0.1        0.14 ?  2%  perf-profile.children.cycles-pp.mas_next_range
      0.18            -0.1        0.10        perf-profile.children.cycles-pp.strnlen
      0.25            -0.1        0.17        perf-profile.children.cycles-pp.mas_destroy
      0.22            -0.1        0.14 ?  2%  perf-profile.children.cycles-pp.remove_vma
      0.23            -0.1        0.16 ?  3%  perf-profile.children.cycles-pp.x64_sys_call
      0.27            -0.1        0.19 ?  2%  perf-profile.children.cycles-pp.security_mmap_file
      0.25            -0.1        0.17        perf-profile.children.cycles-pp.vma_mark_detached
      0.22 ?  2%      -0.1        0.15        perf-profile.children.cycles-pp.vm_area_free
      0.27            -0.1        0.20        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.21            -0.1        0.14        perf-profile.children.cycles-pp.userfaultfd_unmap_complete
      0.15 ?  3%      -0.1        0.08 ?  5%  perf-profile.children.cycles-pp.__free_frozen_pages
      0.27            -0.1        0.20 ?  2%  perf-profile.children.cycles-pp.testcase
      0.20            -0.1        0.14 ?  4%  perf-profile.children.cycles-pp.may_expand_vm
      0.16 ?  2%      -0.1        0.10 ?  3%  perf-profile.children.cycles-pp.strlen
      0.17            -0.1        0.11        perf-profile.children.cycles-pp.security_mmap_addr
      0.16 ?  2%      -0.1        0.10 ?  4%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.12 ?  4%      -0.1        0.06 ?  5%  perf-profile.children.cycles-pp.free_frozen_page_commit
      0.13 ?  3%      -0.0        0.08 ?  5%  perf-profile.children.cycles-pp.mmap_region
      0.11 ?  9%      -0.0        0.06 ? 10%  perf-profile.children.cycles-pp.cmd_record
      0.10 ?  7%      -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.12 ?  5%      -0.0        0.08 ?  7%  perf-profile.children.cycles-pp.task_tick_fair
      0.10 ?  9%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.10 ?  4%      -0.0        0.06        perf-profile.children.cycles-pp.userfaultfd_unmap_prep
      0.15 ?  2%      -0.0        0.11 ?  3%  perf-profile.children.cycles-pp.get_mmap_base
      0.10 ? 10%      -0.0        0.06 ?  7%  perf-profile.children.cycles-pp.perf_mmap__push
      0.13            -0.0        0.09 ?  3%  perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.11 ?  9%      -0.0        0.07 ?  8%  perf-profile.children.cycles-pp.handle_internal_command
      0.11 ?  9%      -0.0        0.07 ?  8%  perf-profile.children.cycles-pp.main
      0.11 ?  9%      -0.0        0.07 ?  8%  perf-profile.children.cycles-pp.run_builtin
      0.10            -0.0        0.06        perf-profile.children.cycles-pp.can_vma_merge_right
      0.09 ? 11%      -0.0        0.05 ? 39%  perf-profile.children.cycles-pp.update_curr
      0.11            -0.0        0.07 ?  5%  perf-profile.children.cycles-pp.mas_node_count_gfp
      0.06 ?  5%      -0.0        0.02 ?100%  perf-profile.children.cycles-pp.__x64_sys_mmap
      0.14 ?  3%      -0.0        0.11 ?  4%  perf-profile.children.cycles-pp.cap_capable
      0.19 ?  2%      -0.0        0.15 ?  2%  perf-profile.children.cycles-pp.unmap_single_vma
      0.10            -0.0        0.07 ?  7%  perf-profile.children.cycles-pp.mas_next_setup
      0.10 ?  3%      -0.0        0.07        perf-profile.children.cycles-pp.ima_file_mmap
      0.11 ?  4%      -0.0        0.08 ? 11%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.08 ?  5%      -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.vma_wants_writenotify
      0.08 ?  5%      -0.0        0.06 ?  9%  perf-profile.children.cycles-pp.mas_nomem
      0.07 ?  8%      -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.ktime_get
      0.07            -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.cap_mmap_addr
      0.07            -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.finish_rcuwait
      0.13 ?  2%      -0.0        0.11 ?  4%  perf-profile.children.cycles-pp.vm_stat_account
      0.16 ?  2%      -0.0        0.14 ?  3%  perf-profile.children.cycles-pp.cap_vm_enough_memory
      0.16            -0.0        0.14 ?  4%  perf-profile.children.cycles-pp.munmap at plt
      0.07            -0.0        0.05        perf-profile.children.cycles-pp.__mt_destroy
      0.07            -0.0        0.05        perf-profile.children.cycles-pp.ksm_vma_flags
      0.06 ?  7%      -0.0        0.04 ? 37%  perf-profile.children.cycles-pp.clockevents_program_event
      0.06 ?  7%      -0.0        0.05        perf-profile.children.cycles-pp.mt_free_rcu
      0.71 ? 10%      +0.2        0.90 ?  9%  perf-profile.children.cycles-pp.get_any_partial
      1.12 ?  3%      +0.3        1.39 ?  4%  perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.00            +0.4        0.39 ?  5%  perf-profile.children.cycles-pp.build_detached_freelist
      0.00            +2.1        2.06 ?  8%  perf-profile.children.cycles-pp.barn_replace_full_sheaf
      3.33 ?  5%      +3.1        6.46 ?  4%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +4.3        4.25 ?  4%  perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.00            +4.5        4.54 ?  6%  perf-profile.children.cycles-pp.barn_replace_empty_sheaf
      0.00            +8.9        8.87 ?  7%  perf-profile.children.cycles-pp.__kmem_cache_free_bulk
      0.00           +11.0       11.03 ?  4%  perf-profile.children.cycles-pp.__pcs_replace_full_main
     72.83           +11.1       83.90        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     72.59           +11.1       83.73        perf-profile.children.cycles-pp.do_syscall_64
      0.00           +14.4       14.41        perf-profile.children.cycles-pp.barn_put_empty_sheaf
      0.00           +15.6       15.55 ?  2%  perf-profile.children.cycles-pp.barn_get_empty_sheaf
     25.77 ?  2%     +17.2       42.94        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     26.07           +17.4       43.44        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     34.78           +26.1       60.93        perf-profile.children.cycles-pp.__mmap
     31.40           +27.2       58.58        perf-profile.children.cycles-pp.vm_mmap_pgoff
     30.03           +27.6       57.66        perf-profile.children.cycles-pp.do_mmap
     23.91           +29.6       53.53        perf-profile.children.cycles-pp.__mmap_region
      9.76 ?  2%     +31.6       41.36        perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
      8.62 ?  2%     +32.0       40.62        perf-profile.children.cycles-pp.mas_alloc_nodes
     13.42           +33.1       46.52        perf-profile.children.cycles-pp.__mmap_new_vma
      5.70 ?  2%     +35.1       40.79        perf-profile.children.cycles-pp.mas_preallocate
      0.00           +39.6       39.60        perf-profile.children.cycles-pp.__pcs_replace_empty_main
      4.32 ?  5%      -2.0        2.33 ?  4%  perf-profile.self.cycles-pp.__slab_free
      3.76            -1.3        2.50        perf-profile.self.cycles-pp.__cond_resched
      3.53            -1.2        2.36        perf-profile.self.cycles-pp.zap_pmd_range
      2.73            -0.9        1.84        perf-profile.self.cycles-pp.free_pud_range
      2.26            -0.8        1.46        perf-profile.self.cycles-pp.rcu_all_qs
      1.47 ?  2%      -0.7        0.77 ?  3%  perf-profile.self.cycles-pp.shuffle_freelist
      1.71            -0.6        1.12        perf-profile.self.cycles-pp.mas_prev_slot
      1.47            -0.6        0.89        perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      1.55            -0.5        1.02        perf-profile.self.cycles-pp.mas_next_slot
      1.85            -0.5        1.33        perf-profile.self.cycles-pp.mas_walk
      1.66            -0.5        1.18        perf-profile.self.cycles-pp.__mmap_region
      1.36 ?  2%      -0.4        0.93 ?  2%  perf-profile.self.cycles-pp.mas_wr_node_store
      1.26            -0.4        0.83        perf-profile.self.cycles-pp.mas_wr_store_type
      1.17            -0.4        0.78        perf-profile.self.cycles-pp.mas_find
      1.16            -0.4        0.77        perf-profile.self.cycles-pp.mas_store_gfp
      0.67            -0.4        0.28 ?  3%  perf-profile.self.cycles-pp.___slab_alloc
      1.01            -0.3        0.68 ?  2%  perf-profile.self.cycles-pp.__pi_memcpy
      0.98            -0.3        0.66        perf-profile.self.cycles-pp.mas_rev_awalk
      1.05            -0.3        0.73        perf-profile.self.cycles-pp.do_syscall_64
      1.34 ?  3%      -0.3        1.08 ?  3%  perf-profile.self.cycles-pp.mas_pop_node
      1.17            -0.3        0.91        perf-profile.self.cycles-pp.kmem_cache_free
      0.83            -0.2        0.58        perf-profile.self.cycles-pp.__call_rcu_common
      0.72            -0.2        0.48        perf-profile.self.cycles-pp.mas_store_prealloc
      0.75            -0.2        0.52        perf-profile.self.cycles-pp.__vma_start_write
      0.75            -0.2        0.52        perf-profile.self.cycles-pp.mas_leaf_max_gap
      0.63            -0.2        0.41        perf-profile.self.cycles-pp.vms_gather_munmap_vmas
      0.56            -0.2        0.35        perf-profile.self.cycles-pp.perf_event_mmap_event
      0.74            -0.2        0.52        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.69            -0.2        0.48        perf-profile.self.cycles-pp.unmap_page_range
      0.59            -0.2        0.39        perf-profile.self.cycles-pp.unmapped_area_topdown
      0.69            -0.2        0.48        perf-profile.self.cycles-pp.__mmap
      0.63            -0.2        0.43        perf-profile.self.cycles-pp.do_vmi_align_munmap
      0.65            -0.2        0.45 ?  2%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.56            -0.2        0.36        perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.68            -0.2        0.48        perf-profile.self.cycles-pp.__munmap
      0.62 ?  2%      -0.2        0.42        perf-profile.self.cycles-pp.mas_preallocate
      0.48 ?  7%      -0.2        0.30 ?  5%  perf-profile.self.cycles-pp.__memcg_slab_free_hook
      0.78 ?  4%      -0.2        0.62 ?  3%  perf-profile.self.cycles-pp.mas_put_in_tree
      0.49 ?  4%      -0.2        0.33 ?  2%  perf-profile.self.cycles-pp.down_write_killable
      0.42            -0.1        0.28        perf-profile.self.cycles-pp.vms_complete_munmap_vmas
      0.40            -0.1        0.25        perf-profile.self.cycles-pp.mas_alloc_nodes
      0.58            -0.1        0.44        perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
      0.60            -0.1        0.46        perf-profile.self.cycles-pp.do_mmap
      0.22 ?  3%      -0.1        0.09 ?  4%  perf-profile.self.cycles-pp.setup_object
      0.39            -0.1        0.26 ?  2%  perf-profile.self.cycles-pp.mas_prev_setup
      0.40            -0.1        0.27 ?  2%  perf-profile.self.cycles-pp.__mmap_new_vma
      0.38            -0.1        0.26        perf-profile.self.cycles-pp.__mmap_prepare
      0.41            -0.1        0.29        perf-profile.self.cycles-pp.__vm_munmap
      0.41            -0.1        0.29        perf-profile.self.cycles-pp.up_read
      0.33            -0.1        0.22        perf-profile.self.cycles-pp.mas_wr_store_entry
      0.33            -0.1        0.22 ?  2%  perf-profile.self.cycles-pp.free_pgd_range
      0.35            -0.1        0.24 ?  2%  perf-profile.self.cycles-pp.mas_prev
      0.49            -0.1        0.38 ?  2%  perf-profile.self.cycles-pp.mas_empty_area_rev
      0.33            -0.1        0.22        perf-profile.self.cycles-pp.perf_event_mmap
      0.35            -0.1        0.24 ?  2%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.33            -0.1        0.22 ?  2%  perf-profile.self.cycles-pp.vm_mmap_pgoff
      0.37            -0.1        0.26        perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.26            -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.sized_strscpy
      0.28            -0.1        0.18 ?  2%  perf-profile.self.cycles-pp.tlb_finish_mmu
      0.31            -0.1        0.21 ?  2%  perf-profile.self.cycles-pp.mas_update_gap
      0.30            -0.1        0.20 ?  2%  perf-profile.self.cycles-pp.mas_data_end
      0.29            -0.1        0.19        perf-profile.self.cycles-pp.unmap_vmas
      0.29            -0.1        0.19 ?  2%  perf-profile.self.cycles-pp.free_pgtables
      0.23            -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.tlb_gather_mmu
      0.28            -0.1        0.18 ?  2%  perf-profile.self.cycles-pp.free_p4d_range
      0.29            -0.1        0.20        perf-profile.self.cycles-pp.vm_area_alloc
      0.39            -0.1        0.30        perf-profile.self.cycles-pp.security_vm_enough_memory_mm
      0.23 ?  2%      -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.vms_clear_ptes
      0.27            -0.1        0.19        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.24            -0.1        0.16        perf-profile.self.cycles-pp.thp_get_unmapped_area_vmflags
      0.23            -0.1        0.15        perf-profile.self.cycles-pp.__account_obj_stock
      0.23            -0.1        0.15 ?  3%  perf-profile.self.cycles-pp.up_write
      0.26            -0.1        0.19        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.21            -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.mas_prev_range
      0.22 ?  2%      -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.kfree
      0.21            -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.21 ?  2%      -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.vm_get_page_prot
      0.24            -0.1        0.18 ?  2%  perf-profile.self.cycles-pp.static_key_count
      0.24            -0.1        0.18 ?  2%  perf-profile.self.cycles-pp.downgrade_write
      0.14 ?  2%      -0.1        0.08        perf-profile.self.cycles-pp.strnlen
      0.19            -0.1        0.13 ?  2%  perf-profile.self.cycles-pp.x64_sys_call
      0.18 ?  2%      -0.1        0.12 ?  4%  perf-profile.self.cycles-pp.refill_obj_stock
      0.14 ?  3%      -0.1        0.09 ?  5%  perf-profile.self.cycles-pp.__put_partials
      0.16 ?  3%      -0.1        0.11 ?  3%  perf-profile.self.cycles-pp.obj_cgroup_charge_account
      0.22            -0.1        0.17 ?  2%  perf-profile.self.cycles-pp.__get_unmapped_area
      0.18 ?  2%      -0.1        0.12 ?  2%  perf-profile.self.cycles-pp.do_vmi_munmap
      0.16 ?  2%      -0.1        0.11        perf-profile.self.cycles-pp.vma_merge_new_range
      0.20 ?  2%      -0.1        0.15 ?  3%  perf-profile.self.cycles-pp.vma_mark_detached
      0.18            -0.1        0.13        perf-profile.self.cycles-pp.mas_destroy
      0.16 ?  3%      -0.1        0.12 ?  4%  perf-profile.self.cycles-pp.may_expand_vm
      0.13            -0.1        0.08        perf-profile.self.cycles-pp.strlen
      0.16            -0.1        0.11        perf-profile.self.cycles-pp.userfaultfd_unmap_complete
      0.18            -0.0        0.13        perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.12 ?  2%      -0.0        0.08        perf-profile.self.cycles-pp.unlink_anon_vmas
      0.15 ?  2%      -0.0        0.11 ?  2%  perf-profile.self.cycles-pp.security_mmap_file
      0.20 ?  5%      -0.0        0.16 ?  2%  perf-profile.self.cycles-pp.testcase
      0.12 ?  4%      -0.0        0.08 ?  5%  perf-profile.self.cycles-pp.mas_next_range
      0.11 ?  4%      -0.0        0.07        perf-profile.self.cycles-pp.security_mmap_addr
      0.09 ?  5%      -0.0        0.06 ?  9%  perf-profile.self.cycles-pp.get_partial_node
      0.11 ?  4%      -0.0        0.07 ?  4%  perf-profile.self.cycles-pp.ksys_mmap_pgoff
      0.10 ?  4%      -0.0        0.07 ? 13%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.10 ?  3%      -0.0        0.07 ?  4%  perf-profile.self.cycles-pp.mmap_region
      0.12            -0.0        0.09        perf-profile.self.cycles-pp.__vm_enough_memory
      0.11 ?  4%      -0.0        0.09 ?  5%  perf-profile.self.cycles-pp.cap_capable
      0.08            -0.0        0.05 ?  8%  perf-profile.self.cycles-pp.mas_node_count_gfp
      0.08 ?  5%      -0.0        0.05        perf-profile.self.cycles-pp.remove_vma
      0.15 ?  3%      -0.0        0.12        perf-profile.self.cycles-pp.unmap_single_vma
      0.22            -0.0        0.20 ?  3%  perf-profile.self.cycles-pp.vma_set_page_prot
      0.07 ?  5%      -0.0        0.05        perf-profile.self.cycles-pp.can_vma_merge_right
      0.11            -0.0        0.09 ?  4%  perf-profile.self.cycles-pp.get_mmap_base
      0.15 ?  2%      -0.0        0.13 ?  2%  perf-profile.self.cycles-pp.rcu_do_batch
      0.08            -0.0        0.06        perf-profile.self.cycles-pp.ima_file_mmap
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.mas_next_setup
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.vm_area_free
      0.07            -0.0        0.05        perf-profile.self.cycles-pp.vm_unmapped_area
      0.09            -0.0        0.08 ?  4%  perf-profile.self.cycles-pp.vm_stat_account
      0.00            +0.1        0.06 ?  9%  perf-profile.self.cycles-pp.barn_replace_full_sheaf
      0.00            +0.1        0.07 ?  5%  perf-profile.self.cycles-pp.__pcs_replace_full_main
      0.00            +0.1        0.14 ?  3%  perf-profile.self.cycles-pp.barn_put_empty_sheaf
      0.30 ?  2%      +0.2        0.49 ?  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.22 ?  3%  perf-profile.self.cycles-pp.barn_replace_empty_sheaf
      0.00            +0.3        0.27 ?  2%  perf-profile.self.cycles-pp.barn_get_empty_sheaf
      1.08 ?  3%      +0.3        1.36 ?  4%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.00            +0.4        0.39 ?  5%  perf-profile.self.cycles-pp.build_detached_freelist
      0.00            +1.0        0.99 ?  5%  perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
      3.30 ?  5%      +3.1        6.41 ?  4%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
     25.77 ?  2%     +17.2       42.94        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath



***************************************************************************************************
igk-spr-2sp1: 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
  gcc-12/performance/x86_64-rhel-9.4/process/100%/debian-12-x86_64-20240206.cgz/igk-spr-2sp1/brk2/will-it-scale

commit: 
  7c1ca28747 ("tools/testing: Add testing support for slab caches with sheaves")
  71a82407f1 ("maple_tree: use percpu sheaves for maple_node_cache")

7c1ca28747078885 71a82407f1c94903b3835b4c935 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     13420           -25.7%       9973        vmstat.system.cs
    253541            -1.3%     250326        vmstat.system.in
   3308095 ?  6%     +15.6%    3823642 ?  6%  meminfo.Active
   3308095 ?  6%     +15.6%    3823642 ?  6%  meminfo.Active(anon)
   2460051 ?  8%     +20.5%    2964917 ?  7%  meminfo.Shmem
   2801930 ? 11%     +22.2%    3423369 ? 11%  numa-meminfo.node1.Active
   2801930 ? 11%     +22.2%    3423369 ? 11%  numa-meminfo.node1.Active(anon)
   2353564 ?  9%     +24.1%    2921574 ?  8%  numa-meminfo.node1.Shmem
  91930871           -28.9%   65361342        will-it-scale.192.processes
    478806           -28.9%     340423        will-it-scale.per_process_ops
  91930871           -28.9%   65361342        will-it-scale.workload
      0.66 ?  2%      -0.2        0.46        mpstat.cpu.all.irq%
     36.37           -12.4       23.94        mpstat.cpu.all.soft%
     59.20           +13.7       72.89        mpstat.cpu.all.sys%
      3.25            -1.0        2.21        mpstat.cpu.all.usr%
 1.661e+08           -45.2%   90952422        numa-numastat.node0.local_node
 1.662e+08           -45.2%   91048208        numa-numastat.node0.numa_hit
 1.662e+08           -46.5%   88857357        numa-numastat.node1.local_node
 1.663e+08           -46.5%   88958022        numa-numastat.node1.numa_hit
     32.62 ? 27%    +431.0%     173.25 ? 31%  perf-c2c.DRAM.local
      6874 ? 12%     +92.5%      13233 ? 15%  perf-c2c.DRAM.remote
      3150 ? 14%    +249.4%      11008 ?  9%  perf-c2c.HITM.local
    141.25 ? 23%    +113.8%     302.00 ? 28%  perf-c2c.HITM.remote
      3291 ? 14%    +243.6%      11310 ? 10%  perf-c2c.HITM.total
 1.662e+08           -45.2%   91042741        numa-vmstat.node0.numa_hit
 1.661e+08           -45.2%   90946955        numa-vmstat.node0.numa_local
    700525 ? 11%     +22.2%     856047 ? 11%  numa-vmstat.node1.nr_active_anon
    588447 ?  9%     +24.2%     730599 ?  8%  numa-vmstat.node1.nr_shmem
    700525 ? 11%     +22.2%     856046 ? 11%  numa-vmstat.node1.nr_zone_active_anon
 1.663e+08           -46.5%   88953076        numa-vmstat.node1.numa_hit
 1.662e+08           -46.5%   88852412        numa-vmstat.node1.numa_local
    827002 ?  6%     +15.6%     955859 ?  6%  proc-vmstat.nr_active_anon
   1499751 ?  3%      +8.4%    1625915 ?  3%  proc-vmstat.nr_file_pages
    615013 ?  8%     +20.5%     741176 ?  7%  proc-vmstat.nr_shmem
    827002 ?  6%     +15.6%     955859 ?  6%  proc-vmstat.nr_zone_active_anon
 3.326e+08           -45.9%    1.8e+08        proc-vmstat.numa_hit
 3.324e+08           -45.9%  1.798e+08        proc-vmstat.numa_local
 1.324e+09           -46.1%  7.134e+08        proc-vmstat.pgalloc_normal
 1.323e+09           -46.2%  7.123e+08        proc-vmstat.pgfree
      1.56           +11.9%       1.75        perf-stat.i.MPKI
 8.852e+10           -25.3%  6.616e+10        perf-stat.i.branch-instructions
      0.10 ?  2%      +0.0        0.14        perf-stat.i.branch-miss-rate%
  80781336           +10.1%   88960612        perf-stat.i.branch-misses
     70.44            -9.5       60.99        perf-stat.i.cache-miss-rate%
 6.728e+08           -17.0%  5.585e+08        perf-stat.i.cache-misses
  9.54e+08            -4.1%  9.153e+08        perf-stat.i.cache-references
     13425           -26.0%       9936        perf-stat.i.context-switches
      1.28           +35.6%       1.73        perf-stat.i.cpi
    606.05           -21.9%     473.24        perf-stat.i.cpu-migrations
    818.41           +21.2%     991.61        perf-stat.i.cycles-between-cache-misses
 4.318e+11           -25.8%  3.202e+11        perf-stat.i.instructions
      0.78           -26.2%       0.58        perf-stat.i.ipc
      1.56           +11.8%       1.75        perf-stat.overall.MPKI
      0.09            +0.0        0.13        perf-stat.overall.branch-miss-rate%
     70.65            -9.6       61.03        perf-stat.overall.cache-miss-rate%
      1.28           +35.6%       1.73        perf-stat.overall.cpi
    817.89           +21.3%     992.40        perf-stat.overall.cycles-between-cache-misses
      0.78           -26.3%       0.58        perf-stat.overall.ipc
   1417447            +4.2%    1476964        perf-stat.overall.path-length
 8.811e+10           -25.3%  6.586e+10        perf-stat.ps.branch-instructions
  79203636           +11.0%   87942785        perf-stat.ps.branch-misses
 6.723e+08           -17.1%  5.573e+08        perf-stat.ps.cache-misses
 9.515e+08            -4.0%   9.13e+08        perf-stat.ps.cache-references
     13283           -25.6%       9881        perf-stat.ps.context-switches
    588.01           -21.3%     462.55        perf-stat.ps.cpu-migrations
 4.298e+11           -25.8%  3.187e+11        perf-stat.ps.instructions
 1.303e+14           -25.9%  9.654e+13        perf-stat.total.instructions
  18763712           +20.8%   22665007        sched_debug.cfs_rq:/.avg_vruntime.avg
  19177877           +20.6%   23132509        sched_debug.cfs_rq:/.avg_vruntime.max
  12561346 ?  4%     +38.8%   17432101 ?  4%  sched_debug.cfs_rq:/.avg_vruntime.min
      0.41           -23.4%       0.31 ?  9%  sched_debug.cfs_rq:/.h_nr_queued.stddev
      0.41           -23.5%       0.31 ?  9%  sched_debug.cfs_rq:/.h_nr_runnable.stddev
   3756224 ? 10%     -49.2%    1907836 ? 24%  sched_debug.cfs_rq:/.left_deadline.avg
  19102630           +20.2%   22961855        sched_debug.cfs_rq:/.left_deadline.max
   7394146 ?  4%     -19.1%    5981443 ? 13%  sched_debug.cfs_rq:/.left_deadline.stddev
   3756166 ? 10%     -49.2%    1907812 ? 24%  sched_debug.cfs_rq:/.left_vruntime.avg
  19102358           +20.2%   22961566        sched_debug.cfs_rq:/.left_vruntime.max
   7394032 ?  4%     -19.1%    5981369 ? 13%  sched_debug.cfs_rq:/.left_vruntime.stddev
    183320 ?  9%     -55.7%      81133 ? 22%  sched_debug.cfs_rq:/.load.avg
    349269 ?  4%     -31.7%     238535 ? 14%  sched_debug.cfs_rq:/.load.stddev
    285.14 ?  2%     -36.9%     179.94 ?  8%  sched_debug.cfs_rq:/.load_avg.avg
    735.94 ? 16%     -25.1%     551.17 ? 14%  sched_debug.cfs_rq:/.load_avg.max
     97.19 ? 20%     -49.5%      49.10 ? 29%  sched_debug.cfs_rq:/.load_avg.min
  18763745           +20.8%   22665021        sched_debug.cfs_rq:/.min_vruntime.avg
  19177908           +20.6%   23132531        sched_debug.cfs_rq:/.min_vruntime.max
  12561401 ?  4%     +38.8%   17432110 ?  4%  sched_debug.cfs_rq:/.min_vruntime.min
      1.01            -9.6%       0.92        sched_debug.cfs_rq:/.nr_queued.avg
      0.37 ?  4%     -28.2%       0.27 ? 11%  sched_debug.cfs_rq:/.nr_queued.stddev
   3756166 ? 10%     -49.2%    1907812 ? 24%  sched_debug.cfs_rq:/.right_vruntime.avg
  19102358           +20.2%   22961566        sched_debug.cfs_rq:/.right_vruntime.max
   7394033 ?  4%     -19.1%    5981369 ? 13%  sched_debug.cfs_rq:/.right_vruntime.stddev
      1161            -8.9%       1058        sched_debug.cfs_rq:/.runnable_avg.avg
    186.73 ?  6%     -15.0%     158.66 ?  7%  sched_debug.cfs_rq:/.runnable_avg.stddev
    106.71 ?  6%     -25.2%      79.79 ? 14%  sched_debug.cfs_rq:/.util_avg.stddev
     35.71 ?  8%     -21.6%      28.01 ? 14%  sched_debug.cpu.clock.stddev
      1284 ?  7%     -11.2%       1140        sched_debug.cpu.clock_task.stddev
      4431 ?  2%     +12.4%       4979        sched_debug.cpu.curr->pid.avg
    170.56 ? 36%     +88.2%     320.94 ? 21%  sched_debug.cpu.curr->pid.min
      2055 ?  4%     -28.7%       1465 ?  9%  sched_debug.cpu.curr->pid.stddev
      0.40 ?  2%     -23.5%       0.31 ?  9%  sched_debug.cpu.nr_running.stddev
     11614           -22.8%       8967        sched_debug.cpu.nr_switches.avg
      9169           -27.3%       6668        sched_debug.cpu.nr_switches.min
      0.00           -50.0%       0.00        sched_debug.rt_rq:.rt_nr_running.avg
      0.33           -50.0%       0.17        sched_debug.rt_rq:.rt_nr_running.max
      0.02           -50.0%       0.01        sched_debug.rt_rq:.rt_nr_running.stddev
      0.23 ? 18%     -32.0%       0.16 ? 18%  perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity
      8.07 ?  4%     -17.4%       6.67 ? 13%  perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.11 ?  6%     -17.7%       6.67 ? 12%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      8.20 ?  4%     -17.8%       6.74 ? 11%  perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      8.47 ?  5%     -20.1%       6.77 ? 11%  perf-sched.sch_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      7.23 ?  5%     -27.4%       5.25 ? 12%  perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1.18 ? 66%     -59.3%       0.48 ? 53%  perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork
      0.35 ? 12%     -17.4%       0.29 ?  8%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread
      3.97 ?  4%     -19.3%       3.21 ? 10%  perf-sched.total_sch_delay.average.ms
     44.41 ?  2%     +24.3%      55.21 ?  4%  perf-sched.total_wait_and_delay.average.ms
     68566 ?  2%     -21.4%      53890 ?  4%  perf-sched.total_wait_and_delay.count.ms
     40.44 ?  2%     +28.6%      52.00 ?  4%  perf-sched.total_wait_time.average.ms
     16.15 ?  4%     -17.4%      13.34 ? 13%  perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write_killable.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
     16.23 ?  6%     -17.7%      13.35 ? 12%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
     16.40 ?  4%     -17.8%      13.48 ? 11%  perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
     29.14 ?  6%     +50.5%      43.86 ? 10%  perf-sched.wait_and_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     16.95 ?  5%     -20.1%      13.54 ? 11%  perf-sched.wait_and_delay.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
     15.28 ?  5%     -22.0%      11.93 ? 11%  perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
     56.71 ?  3%     +46.8%      83.27 ?  3%  perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      1275 ?  4%     -50.3%     633.38 ?  6%  perf-sched.wait_and_delay.count.__cond_resched.down_write_killable.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1472 ?  5%     -45.4%     803.12 ?  6%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      1108 ?  5%     -53.8%     512.50 ?  3%  perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
    963.00 ?  6%     -56.6%     417.75 ?  2%  perf-sched.wait_and_delay.count.__cond_resched.remove_vma.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      1648 ?  9%     -36.0%       1054 ? 12%  perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     14703 ?  3%      +9.8%      16150 ?  4%  perf-sched.wait_and_delay.count.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
    981.62 ?  7%     -59.1%     401.75 ? 12%  perf-sched.wait_and_delay.count.__cond_resched.zap_pte_range.zap_pmd_range.isra.0
     10834 ?  3%     -43.1%       6168 ?  7%  perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
      1849 ?  9%     -44.2%       1031 ? 28%  perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
     27173 ?  3%     -26.2%      20048 ?  4%  perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      8.07 ?  4%     -17.4%       6.67 ? 13%  perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      8.11 ?  6%     -17.7%       6.67 ? 12%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
      8.20 ?  4%     -17.8%       6.74 ? 11%  perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
     29.08 ?  6%     +50.5%      43.77 ? 10%  perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      8.47 ?  5%     -20.1%       6.77 ? 11%  perf-sched.wait_time.avg.ms.__cond_resched.unmap_vmas.vms_clear_ptes.part.0
      0.20 ?  7%     +70.5%       0.34 ? 52%  perf-sched.wait_time.avg.ms.ipmi_thread.kthread.ret_from_fork.ret_from_fork_asm
      6.87 ? 64%     -43.0%       3.92 ? 15%  perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
     56.55 ?  3%     +47.0%      83.13 ?  3%  perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      2244 ? 68%     -90.2%     220.55 ?222%  perf-sched.wait_time.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64
      0.20          +128.0%       0.47 ? 74%  perf-sched.wait_time.max.ms.ipmi_thread.kthread.ret_from_fork.ret_from_fork_asm
     20.21 ?  2%     -18.7        1.50 ? 31%  perf-profile.calltrace.cycles-pp.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
     25.94           -14.7       11.25 ?  2%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
     28.93           -11.2       17.70 ?  3%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
     29.02           -11.2       17.82 ?  3%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
     29.02           -11.2       17.82 ?  3%  perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
     29.02           -11.2       17.82 ?  3%  perf-profile.calltrace.cycles-pp.ret_from_fork_asm
     29.00           -11.2       17.80 ?  3%  perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
     28.98           -11.2       17.79 ?  3%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
     28.98           -11.2       17.79 ?  3%  perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
     28.98           -11.2       17.79 ?  3%  perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
      9.79 ?  2%      -9.8        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch.rcu_core
      9.58 ?  2%      -9.6        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.kmem_cache_free.rcu_do_batch
     12.03            -7.3        4.71        perf-profile.calltrace.cycles-pp.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.76 ?  2%      -5.8        0.00        perf-profile.calltrace.cycles-pp.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
      5.60 ?  2%      -5.6        0.00        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk
      5.79 ?  2%      -5.4        0.38 ? 57%  perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      5.40 ?  2%      -5.4        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch.rcu_core
      5.33 ?  2%      -5.3        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.kmem_cache_free.rcu_do_batch
      5.00 ?  2%      -5.0        0.00        perf-profile.calltrace.cycles-pp.___slab_alloc.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_store_gfp.do_vmi_align_munmap
     12.46            -4.9        7.56        perf-profile.calltrace.cycles-pp.do_brk_flags.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      9.07            -4.3        4.80        perf-profile.calltrace.cycles-pp.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      6.59            -2.7        3.86        perf-profile.calltrace.cycles-pp.vma_merge_new_range.do_brk_flags.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.14            -2.5        2.65        perf-profile.calltrace.cycles-pp.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      5.90            -2.4        3.49        perf-profile.calltrace.cycles-pp.vma_expand.vma_merge_new_range.do_brk_flags.__do_sys_brk.do_syscall_64
      4.67            -1.9        2.80        perf-profile.calltrace.cycles-pp.commit_merge.vma_expand.vma_merge_new_range.do_brk_flags.__do_sys_brk
      3.03            -1.5        1.57        perf-profile.calltrace.cycles-pp.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      3.18            -1.4        1.80        perf-profile.calltrace.cycles-pp.perf_event_mmap.do_brk_flags.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.32            -1.4        3.97        perf-profile.calltrace.cycles-pp.vma_complete.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      5.03            -1.2        3.80        perf-profile.calltrace.cycles-pp.mas_store_prealloc.vma_complete.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap
      4.41            -1.2        3.21        perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      2.38            -1.2        1.23        perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap
      2.58            -1.1        1.47        perf-profile.calltrace.cycles-pp.perf_event_mmap_event.perf_event_mmap.do_brk_flags.__do_sys_brk.do_syscall_64
      2.84            -1.1        1.77        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.brk
      2.15            -1.0        1.14        perf-profile.calltrace.cycles-pp.mas_find.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      2.17            -0.9        1.23        perf-profile.calltrace.cycles-pp.vm_area_dup.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      2.24            -0.8        1.40        perf-profile.calltrace.cycles-pp.mas_store_prealloc.commit_merge.vma_expand.vma_merge_new_range.do_brk_flags
      1.88            -0.8        1.08        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap
      4.07            -0.8        3.28        perf-profile.calltrace.cycles-pp.mas_wr_node_store.mas_store_prealloc.vma_complete.__split_vma.vms_gather_munmap_vmas
      1.65            -0.8        0.87        perf-profile.calltrace.cycles-pp.mas_walk.mas_find.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.54            -0.8        0.78        perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes.vms_complete_munmap_vmas
      1.64 ?  4%      -0.7        0.90 ?  3%  perf-profile.calltrace.cycles-pp.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      1.87 ?  2%      -0.7        1.19        perf-profile.calltrace.cycles-pp.__vma_start_write.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      1.05 ?  7%      -0.6        0.48 ? 37%  perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kmem_cache_free.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      1.11            -0.6        0.56        perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.vms_clear_ptes
      1.20            -0.5        0.65        perf-profile.calltrace.cycles-pp.mas_preallocate.commit_merge.vma_expand.vma_merge_new_range.do_brk_flags
      1.24            -0.5        0.72        perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.kmem_cache_alloc_noprof.vm_area_dup.__split_vma.vms_gather_munmap_vmas
      1.29            -0.5        0.78        perf-profile.calltrace.cycles-pp.mas_store_gfp.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      1.05            -0.5        0.54        perf-profile.calltrace.cycles-pp.free_pgtables.vms_clear_ptes.vms_complete_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      1.10            -0.5        0.64        perf-profile.calltrace.cycles-pp.check_brk_limits.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      1.13            -0.5        0.67 ?  2%  perf-profile.calltrace.cycles-pp.perf_iterate_sb.perf_event_mmap_event.perf_event_mmap.do_brk_flags.__do_sys_brk
      0.98            -0.4        0.56        perf-profile.calltrace.cycles-pp.mas_prev_slot.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      0.96            -0.4        0.55        perf-profile.calltrace.cycles-pp.__get_unmapped_area.check_brk_limits.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.93            -0.4        0.52        perf-profile.calltrace.cycles-pp.__vma_start_write.vma_expand.vma_merge_new_range.do_brk_flags.__do_sys_brk
      0.97            -0.4        0.58 ?  2%  perf-profile.calltrace.cycles-pp.security_vm_enough_memory_mm.do_brk_flags.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.76 ?  4%      -0.3        0.46 ? 37%  perf-profile.calltrace.cycles-pp.mas_put_in_tree.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk
      0.79            -0.2        0.55        perf-profile.calltrace.cycles-pp.mas_update_gap.mas_store_prealloc.commit_merge.vma_expand.vma_merge_new_range
      1.19 ?  2%      +0.1        1.26 ?  2%  perf-profile.calltrace.cycles-pp.__call_rcu_common.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap.__do_sys_brk
      0.63 ?  3%      +0.2        0.84 ?  3%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_gfp.do_vmi_align_munmap
      1.19            +0.2        1.42 ?  2%  perf-profile.calltrace.cycles-pp.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.vma_complete.__split_vma
      0.58 ?  2%      +0.4        0.95 ?  3%  perf-profile.calltrace.cycles-pp.rcu_segcblist_enqueue.__call_rcu_common.mas_wr_node_store.mas_store_prealloc.vma_complete
      0.00            +0.6        0.60 ?  7%  perf-profile.calltrace.cycles-pp.build_detached_freelist.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +0.7        0.66 ?  3%  perf-profile.calltrace.cycles-pp.shuffle_freelist.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      1.05 ? 38%      +0.8        1.90 ? 10%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.get_any_partial.___slab_alloc
      0.00            +0.9        0.85 ?  7%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.9        0.85 ?  7%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf
      0.00            +0.9        0.86 ?  7%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.9        0.86 ?  7%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      1.48 ?  4%      +0.9        2.38 ?  5%  perf-profile.calltrace.cycles-pp.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu
      0.00            +0.9        0.94 ?  3%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main
      0.00            +0.9        0.94 ?  3%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +0.9        0.94 ?  4%  perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main
      0.00            +0.9        0.94 ?  4%  perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf
      0.00            +1.0        0.97 ?  3%  perf-profile.calltrace.cycles-pp.allocate_slab.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +1.0        0.99 ?  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +1.0        0.99 ?  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +1.0        0.99 ?  4%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +1.0        1.00 ?  4%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      1.56 ?  4%      +1.2        2.72 ?  4%  perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt
      1.56 ?  4%      +1.2        2.73 ?  4%  perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
      0.00            +1.3        1.26 ?  5%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free
      0.00            +1.3        1.32 ?  5%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +1.4        1.35 ?  5%  perf-profile.calltrace.cycles-pp.barn_replace_full_sheaf.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      0.00            +1.8        1.80 ?  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk
      0.00            +1.8        1.81 ?  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +1.9        1.91 ?  6%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +1.9        1.91 ? 11%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk
      0.00            +1.9        1.94 ?  6%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +2.0        2.04 ?  6%  perf-profile.calltrace.cycles-pp.__put_partials.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +2.2        2.19 ?  9%  perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +2.2        2.25 ? 11%  perf-profile.calltrace.cycles-pp.get_partial_node.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main
      0.00            +2.3        2.29 ? 11%  perf-profile.calltrace.cycles-pp.get_any_partial.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +2.9        2.86 ?  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main
      0.00            +3.0        3.05 ?  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free
      0.00            +3.2        3.18 ?  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00            +3.2        3.20 ?  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      2.85 ?  4%      +3.4        6.24 ?  7%  perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd
      0.00            +4.2        4.25 ?  4%  perf-profile.calltrace.cycles-pp.barn_replace_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +4.3        4.33 ?  4%  perf-profile.calltrace.cycles-pp.__slab_free.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch
      0.00            +5.7        5.74 ?  7%  perf-profile.calltrace.cycles-pp.___slab_alloc.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00            +6.6        6.61 ?  7%  perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_bulk.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00            +7.2        7.17 ?  3%  perf-profile.calltrace.cycles-pp.__kmem_cache_free_bulk.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core
      0.00            +8.6        8.59 ?  2%  perf-profile.calltrace.cycles-pp.__pcs_replace_full_main.kmem_cache_free.rcu_do_batch.rcu_core.handle_softirqs
     69.96           +11.0       80.97        perf-profile.calltrace.cycles-pp.brk
     66.66           +12.8       79.49        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.brk
     66.38           +12.9       79.32        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
     64.51           +13.8       78.28        perf-profile.calltrace.cycles-pp.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
      0.00           +15.7       15.74        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +15.9       15.86        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +16.0       15.98        perf-profile.calltrace.cycles-pp.barn_put_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
      0.00           +16.5       16.48        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof
      0.00           +16.5       16.52        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes
      0.00           +17.8       17.76        perf-profile.calltrace.cycles-pp.barn_get_empty_sheaf.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate
     45.44           +21.7       67.10        perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe.brk
     23.17           +33.8       56.96        perf-profile.calltrace.cycles-pp.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.02           +36.0       54.00        perf-profile.calltrace.cycles-pp.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk.do_syscall_64
      7.17           +39.7       46.84        perf-profile.calltrace.cycles-pp.mas_preallocate.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap.__do_sys_brk
      5.87 ?  2%     +40.3       46.15        perf-profile.calltrace.cycles-pp.mas_alloc_nodes.mas_preallocate.__split_vma.vms_gather_munmap_vmas.do_vmi_align_munmap
      5.63 ?  2%     +40.4       45.99        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma.vms_gather_munmap_vmas
      0.00           +45.6       45.62        perf-profile.calltrace.cycles-pp.__pcs_replace_empty_main.kmem_cache_alloc_noprof.mas_alloc_nodes.mas_preallocate.__split_vma
     36.48           -19.1       17.36        perf-profile.children.cycles-pp.kmem_cache_free
     25.78           -18.5        7.23        perf-profile.children.cycles-pp.__slab_free
     38.44           -13.7       24.73        perf-profile.children.cycles-pp.handle_softirqs
     38.42           -13.7       24.72        perf-profile.children.cycles-pp.rcu_core
     38.40           -13.7       24.71        perf-profile.children.cycles-pp.rcu_do_batch
     29.02           -11.2       17.82 ?  3%  perf-profile.children.cycles-pp.kthread
     29.02           -11.2       17.82 ?  3%  perf-profile.children.cycles-pp.ret_from_fork
     29.02           -11.2       17.82 ?  3%  perf-profile.children.cycles-pp.ret_from_fork_asm
     29.00           -11.2       17.80 ?  3%  perf-profile.children.cycles-pp.smpboot_thread_fn
     28.98           -11.2       17.79 ?  3%  perf-profile.children.cycles-pp.run_ksoftirqd
     13.47            -7.9        5.58        perf-profile.children.cycles-pp.mas_store_gfp
     12.59            -5.0        7.61        perf-profile.children.cycles-pp.do_brk_flags
      8.38 ?  2%      -4.7        3.63 ?  8%  perf-profile.children.cycles-pp.__put_partials
      9.16            -4.3        4.86        perf-profile.children.cycles-pp.vms_complete_munmap_vmas
      9.89 ?  2%      -4.2        5.74 ?  7%  perf-profile.children.cycles-pp.___slab_alloc
     10.12 ?  2%      -2.8        7.33 ?  4%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
     10.08 ?  2%      -2.8        7.29 ?  4%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      6.65            -2.8        3.89        perf-profile.children.cycles-pp.vma_merge_new_range
      5.20            -2.5        2.69        perf-profile.children.cycles-pp.vms_clear_ptes
      9.44 ?  2%      -2.5        6.94 ?  4%  perf-profile.children.cycles-pp.__irq_exit_rcu
      5.93            -2.4        3.51        perf-profile.children.cycles-pp.vma_expand
      4.49            -2.1        2.37        perf-profile.children.cycles-pp.mas_find
      7.36            -2.1        5.26        perf-profile.children.cycles-pp.mas_store_prealloc
      8.74            -2.1        6.65        perf-profile.children.cycles-pp.mas_wr_node_store
      4.82            -1.9        2.88        perf-profile.children.cycles-pp.commit_merge
      6.30 ?  4%      -1.9        4.44 ? 10%  perf-profile.children.cycles-pp.get_partial_node
      2.46 ?  2%      -1.5        0.97 ?  3%  perf-profile.children.cycles-pp.allocate_slab
      3.10            -1.5        1.61        perf-profile.children.cycles-pp.unmap_vmas
      5.63            -1.4        4.20        perf-profile.children.cycles-pp.vma_complete
      3.21            -1.4        1.82        perf-profile.children.cycles-pp.perf_event_mmap
      2.18 ?  2%      -1.3        0.86 ?  3%  perf-profile.children.cycles-pp.shuffle_freelist
      2.43            -1.2        1.26        perf-profile.children.cycles-pp.unmap_page_range
      2.71            -1.2        1.54        perf-profile.children.cycles-pp.perf_event_mmap_event
      2.85            -1.1        1.75        perf-profile.children.cycles-pp.__vma_start_write
      2.03            -1.0        1.07        perf-profile.children.cycles-pp.mas_walk
      2.22            -1.0        1.26        perf-profile.children.cycles-pp.vm_area_dup
      2.09            -0.9        1.15        perf-profile.children.cycles-pp.mas_wr_store_type
      2.09            -0.8        1.29        perf-profile.children.cycles-pp.mas_update_gap
      1.58            -0.8        0.80        perf-profile.children.cycles-pp.zap_pmd_range
      1.67            -0.6        1.05        perf-profile.children.cycles-pp.entry_SYSCALL_64
      1.35            -0.6        0.75        perf-profile.children.cycles-pp.mas_prev_slot
      1.37            -0.6        0.78 ?  3%  perf-profile.children.cycles-pp.__pi_memcpy
      1.16            -0.6        0.58        perf-profile.children.cycles-pp.zap_pte_range
      1.18            -0.6        0.61        perf-profile.children.cycles-pp.mas_next_slot
      1.15            -0.6        0.60        perf-profile.children.cycles-pp.free_pgtables
      1.27            -0.5        0.74        perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      1.48            -0.5        0.95        perf-profile.children.cycles-pp.mas_leaf_max_gap
      1.08 ?  7%      -0.5        0.56 ?  5%  perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.81 ?  2%      -0.5        0.30 ?  3%  perf-profile.children.cycles-pp.setup_object
      1.19            -0.5        0.70 ?  2%  perf-profile.children.cycles-pp.perf_iterate_sb
      1.07            -0.5        0.59        perf-profile.children.cycles-pp.__cond_resched
      1.14            -0.5        0.66        perf-profile.children.cycles-pp.check_brk_limits
      1.09            -0.4        0.66        perf-profile.children.cycles-pp.security_vm_enough_memory_mm
      0.99            -0.4        0.57        perf-profile.children.cycles-pp.__get_unmapped_area
      0.93            -0.4        0.52        perf-profile.children.cycles-pp.down_write_killable
      1.12 ?  2%      -0.3        0.80        perf-profile.children.cycles-pp.mas_pop_node
      1.05 ?  3%      -0.3        0.73        perf-profile.children.cycles-pp.mas_put_in_tree
      0.70            -0.3        0.38        perf-profile.children.cycles-pp.init_multi_vma_prep
      0.77            -0.3        0.47        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.58            -0.3        0.29        perf-profile.children.cycles-pp.__pte_offset_map_lock
      0.66 ?  5%      -0.3        0.37 ?  4%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.65 ?  4%      -0.3        0.37 ?  4%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.56            -0.3        0.30        perf-profile.children.cycles-pp.mas_wr_store_entry
      0.55 ?  4%      -0.3        0.30 ?  3%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.54 ?  4%      -0.3        0.29 ?  3%  perf-profile.children.cycles-pp.tick_nohz_handler
      0.56            -0.2        0.31        perf-profile.children.cycles-pp.rcu_all_qs
      0.48            -0.2        0.24 ?  2%  perf-profile.children.cycles-pp.mas_prev
      0.52 ?  4%      -0.2        0.28 ?  3%  perf-profile.children.cycles-pp.update_process_times
      0.52            -0.2        0.30        perf-profile.children.cycles-pp.up_read
      0.52            -0.2        0.30 ?  2%  perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.45            -0.2        0.24        perf-profile.children.cycles-pp.mas_prev_setup
      0.39            -0.2        0.20        perf-profile.children.cycles-pp.can_vma_merge_left
      0.51            -0.2        0.32 ?  2%  perf-profile.children.cycles-pp.mas_destroy
      0.44            -0.2        0.25 ?  2%  perf-profile.children.cycles-pp.perf_event_mmap_output
      0.39            -0.2        0.22        perf-profile.children.cycles-pp.tlb_finish_mmu
      0.36            -0.2        0.20 ?  2%  perf-profile.children.cycles-pp.sized_strscpy
      0.34            -0.2        0.18 ?  2%  perf-profile.children.cycles-pp.vma_adjust_trans_huge
      0.37 ?  2%      -0.2        0.21 ?  2%  perf-profile.children.cycles-pp.obj_cgroup_charge_account
      0.30            -0.2        0.14 ?  3%  perf-profile.children.cycles-pp.tlb_gather_mmu
      0.26            -0.2        0.10 ?  5%  perf-profile.children.cycles-pp.alloc_pages_mpol
      0.35            -0.2        0.20        perf-profile.children.cycles-pp.refill_obj_stock
      0.37 ?  3%      -0.2        0.22 ?  4%  perf-profile.children.cycles-pp.sched_tick
      0.24 ?  2%      -0.1        0.10 ?  5%  perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
      0.24 ?  2%      -0.1        0.09 ?  5%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.47            -0.1        0.32        perf-profile.children.cycles-pp.static_key_count
      0.32            -0.1        0.17        perf-profile.children.cycles-pp.kfree
      0.31 ?  2%      -0.1        0.17 ?  2%  perf-profile.children.cycles-pp.__vm_enough_memory
      0.30            -0.1        0.17 ?  2%  perf-profile.children.cycles-pp.vma_mark_detached
      0.30            -0.1        0.16 ?  3%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.30            -0.1        0.17        perf-profile.children.cycles-pp.__account_obj_stock
      0.24 ?  4%      -0.1        0.12 ?  2%  perf-profile.children.cycles-pp.__free_frozen_pages
      0.28            -0.1        0.16 ?  2%  perf-profile.children.cycles-pp.x64_sys_call
      0.29            -0.1        0.17        perf-profile.children.cycles-pp.up_write
      0.30            -0.1        0.18        perf-profile.children.cycles-pp.downgrade_write
      0.46            -0.1        0.34        perf-profile.children.cycles-pp.mas_prev_range
      0.28            -0.1        0.17 ?  2%  perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.25            -0.1        0.13 ?  3%  perf-profile.children.cycles-pp._raw_spin_lock
      0.26            -0.1        0.15 ?  2%  perf-profile.children.cycles-pp.mas_wr_slot_store
      0.24            -0.1        0.13 ?  2%  perf-profile.children.cycles-pp.vm_area_free
      0.24            -0.1        0.13 ?  3%  perf-profile.children.cycles-pp.remove_vma
      0.28            -0.1        0.18        perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
      0.28 ?  2%      -0.1        0.18 ?  3%  perf-profile.children.cycles-pp.may_expand_vm
      0.23 ?  2%      -0.1        0.13        perf-profile.children.cycles-pp.userfaultfd_unmap_complete
      0.18 ?  5%      -0.1        0.08 ?  5%  perf-profile.children.cycles-pp.free_frozen_page_commit
      0.15 ?  2%      -0.1        0.06 ?  5%  perf-profile.children.cycles-pp.rmqueue
      0.16 ?  6%      -0.1        0.08 ?  5%  perf-profile.children.cycles-pp.free_pcppages_bulk
      0.19            -0.1        0.10 ?  4%  perf-profile.children.cycles-pp.vm_area_init_from
      0.19 ?  2%      -0.1        0.10 ?  3%  perf-profile.children.cycles-pp.strnlen
      0.16 ?  3%      -0.1        0.08 ?  5%  perf-profile.children.cycles-pp.unlink_anon_vmas
      0.17            -0.1        0.09        perf-profile.children.cycles-pp.unmap_single_vma
      0.15 ?  2%      -0.1        0.07        perf-profile.children.cycles-pp.___pte_offset_map
      0.16            -0.1        0.08        perf-profile.children.cycles-pp.security_mmap_addr
      0.17            -0.1        0.09        perf-profile.children.cycles-pp.free_pgd_range
      0.17            -0.1        0.09        perf-profile.children.cycles-pp.vma_prepare
      0.13 ?  2%      -0.1        0.05        perf-profile.children.cycles-pp.__rmqueue_pcplist
      0.17 ?  2%      -0.1        0.10 ?  5%  perf-profile.children.cycles-pp.strlen
      0.12 ?  2%      -0.1        0.04 ? 37%  perf-profile.children.cycles-pp.rmqueue_bulk
      0.14 ?  3%      -0.1        0.07 ?  5%  perf-profile.children.cycles-pp.anon_vma_clone
      0.14            -0.1        0.07 ?  4%  perf-profile.children.cycles-pp.brk at plt
      0.16 ?  2%      -0.1        0.09 ?  3%  perf-profile.children.cycles-pp.finish_rcuwait
      0.14            -0.1        0.07 ?  6%  perf-profile.children.cycles-pp.is_mergeable_anon_vma
      0.15 ?  5%      -0.1        0.08 ?  7%  perf-profile.children.cycles-pp.cap_capable
      0.16 ?  4%      -0.1        0.10 ?  5%  perf-profile.children.cycles-pp.task_tick_fair
      0.13 ?  3%      -0.1        0.08 ?  7%  perf-profile.children.cycles-pp.update_rq_clock_task
      0.12            -0.0        0.07        perf-profile.children.cycles-pp.userfaultfd_unmap_prep
      0.10 ?  3%      -0.0        0.05        perf-profile.children.cycles-pp.mas_nomem
      0.14 ?  3%      -0.0        0.09        perf-profile.children.cycles-pp.mas_node_count_gfp
      0.11 ?  3%      -0.0        0.06 ?  6%  perf-profile.children.cycles-pp.khugepaged_enter_vma
      0.11 ?  4%      -0.0        0.07 ?  6%  perf-profile.children.cycles-pp.update_curr
      0.12            -0.0        0.08 ?  4%  perf-profile.children.cycles-pp.testcase
      0.07 ?  9%      -0.0        0.05 ?  8%  perf-profile.children.cycles-pp.record__mmap_read_evlist
      0.07            -0.0        0.05        perf-profile.children.cycles-pp.cap_vm_enough_memory
      0.08 ?  9%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.cmd_record
      0.08 ?  9%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.handle_internal_command
      0.08 ?  9%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.main
      0.08 ?  9%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.run_builtin
      0.07 ? 12%      -0.0        0.05 ?  8%  perf-profile.children.cycles-pp.perf_mmap__push
      0.08 ?  5%      -0.0        0.06 ?  5%  perf-profile.children.cycles-pp.call_rcu
      0.06            +0.1        0.11 ?  3%  perf-profile.children.cycles-pp.mt_free_rcu
      0.14 ?  3%      +0.2        0.31 ?  2%  perf-profile.children.cycles-pp.ksm_vma_flags
      2.45            +0.3        2.73 ?  2%  perf-profile.children.cycles-pp.__call_rcu_common
      1.77 ?  8%      +0.5        2.29 ? 11%  perf-profile.children.cycles-pp.get_any_partial
      1.25 ?  2%      +0.6        1.81 ?  2%  perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.00            +0.7        0.70 ?  6%  perf-profile.children.cycles-pp.build_detached_freelist
      0.00            +1.9        1.92 ?  5%  perf-profile.children.cycles-pp.barn_replace_full_sheaf
      0.00            +4.4        4.40 ?  4%  perf-profile.children.cycles-pp.barn_replace_empty_sheaf
      3.42 ?  3%      +4.5        7.92 ?  5%  perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.00            +6.6        6.61 ?  7%  perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.00            +8.6        8.60 ?  3%  perf-profile.children.cycles-pp.__kmem_cache_free_bulk
      0.00           +10.6       10.63        perf-profile.children.cycles-pp.__pcs_replace_full_main
     70.52           +11.3       81.78        perf-profile.children.cycles-pp.brk
     66.77           +12.8       79.57        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     66.50           +12.9       79.41        perf-profile.children.cycles-pp.do_syscall_64
     64.72           +13.7       78.40        perf-profile.children.cycles-pp.__do_sys_brk
      0.00           +16.0       15.98        perf-profile.children.cycles-pp.barn_put_empty_sheaf
      0.00           +17.8       17.78        perf-profile.children.cycles-pp.barn_get_empty_sheaf
     45.49           +21.6       67.13        perf-profile.children.cycles-pp.do_vmi_align_munmap
     25.36 ?  2%     +23.0       48.39        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     25.79 ?  2%     +23.2       48.98        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     23.37           +33.7       57.07        perf-profile.children.cycles-pp.vms_gather_munmap_vmas
     13.20           +34.3       47.52        perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
     11.72 ?  2%     +35.0       46.69        perf-profile.children.cycles-pp.mas_alloc_nodes
     18.22           +35.9       54.12        perf-profile.children.cycles-pp.__split_vma
      8.45           +39.1       47.55        perf-profile.children.cycles-pp.mas_preallocate
      0.00           +45.8       45.78        perf-profile.children.cycles-pp.__pcs_replace_empty_main
     12.21 ?  2%      -9.2        3.06 ?  2%  perf-profile.self.cycles-pp.__slab_free
      1.69 ?  2%      -1.0        0.74 ?  3%  perf-profile.self.cycles-pp.shuffle_freelist
      2.32            -0.8        1.47        perf-profile.self.cycles-pp.__vma_start_write
      1.72            -0.8        0.94        perf-profile.self.cycles-pp.mas_walk
      1.52            -0.8        0.76        perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      1.75            -0.8        1.00        perf-profile.self.cycles-pp.mas_wr_store_type
      1.75            -0.7        1.07        perf-profile.self.cycles-pp.brk
      1.39            -0.6        0.81        perf-profile.self.cycles-pp.mas_store_prealloc
      0.84 ?  2%      -0.6        0.27 ?  3%  perf-profile.self.cycles-pp.___slab_alloc
      1.49            -0.6        0.91        perf-profile.self.cycles-pp.mas_store_gfp
      1.40 ?  2%      -0.6        0.84 ?  2%  perf-profile.self.cycles-pp.mas_wr_node_store
      1.24            -0.5        0.72        perf-profile.self.cycles-pp.__do_sys_brk
      1.12 ?  2%      -0.5        0.62        perf-profile.self.cycles-pp.mas_preallocate
      1.11            -0.5        0.62        perf-profile.self.cycles-pp.mas_find
      1.13            -0.5        0.64        perf-profile.self.cycles-pp.mas_prev_slot
      1.11            -0.5        0.65 ?  2%  perf-profile.self.cycles-pp.__pi_memcpy
      0.96            -0.4        0.52        perf-profile.self.cycles-pp.mas_next_slot
      1.29            -0.4        0.85        perf-profile.self.cycles-pp.mas_leaf_max_gap
      1.05            -0.4        0.62        perf-profile.self.cycles-pp.do_syscall_64
      0.93            -0.4        0.57        perf-profile.self.cycles-pp.__call_rcu_common
      0.75            -0.3        0.41        perf-profile.self.cycles-pp.vms_gather_munmap_vmas
      0.90            -0.3        0.56        perf-profile.self.cycles-pp.do_brk_flags
      0.79            -0.3        0.46        perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.70            -0.3        0.38        perf-profile.self.cycles-pp.do_vmi_align_munmap
      0.63 ? 12%      -0.3        0.32 ?  8%  perf-profile.self.cycles-pp.__memcg_slab_free_hook
      0.76            -0.3        0.46        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.66            -0.3        0.37        perf-profile.self.cycles-pp.unmap_page_range
      0.93 ?  2%      -0.3        0.66        perf-profile.self.cycles-pp.mas_pop_node
      0.60            -0.3        0.34        perf-profile.self.cycles-pp.init_multi_vma_prep
      0.61            -0.3        0.36        perf-profile.self.cycles-pp.perf_event_mmap_event
      0.68 ?  2%      -0.3        0.42 ?  3%  perf-profile.self.cycles-pp.perf_iterate_sb
      0.56            -0.2        0.31        perf-profile.self.cycles-pp.down_write_killable
      0.88 ?  4%      -0.2        0.64 ?  2%  perf-profile.self.cycles-pp.mas_put_in_tree
      0.54            -0.2        0.30        perf-profile.self.cycles-pp.__split_vma
      0.52            -0.2        0.30        perf-profile.self.cycles-pp.__cond_resched
      0.53            -0.2        0.32        perf-profile.self.cycles-pp.mas_update_gap
      0.43            -0.2        0.22        perf-profile.self.cycles-pp.zap_pte_range
      0.45            -0.2        0.25        perf-profile.self.cycles-pp.mas_wr_store_entry
      0.44            -0.2        0.25        perf-profile.self.cycles-pp.perf_event_mmap
      0.27 ?  2%      -0.2        0.08 ?  5%  perf-profile.self.cycles-pp.setup_object
      0.44            -0.2        0.25        perf-profile.self.cycles-pp.commit_merge
      0.44            -0.2        0.27        perf-profile.self.cycles-pp.up_read
      0.41            -0.2        0.24 ?  2%  perf-profile.self.cycles-pp.security_vm_enough_memory_mm
      0.39            -0.2        0.23        perf-profile.self.cycles-pp.mas_alloc_nodes
      0.42            -0.2        0.26        perf-profile.self.cycles-pp.arch_get_unmapped_area_topdown
      0.40            -0.2        0.25        perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.32            -0.1        0.17 ?  2%  perf-profile.self.cycles-pp.mas_prev_setup
      0.27            -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.vms_clear_ptes
      0.32            -0.1        0.18        perf-profile.self.cycles-pp.rcu_all_qs
      0.35            -0.1        0.21 ?  2%  perf-profile.self.cycles-pp.perf_event_mmap_output
      0.34            -0.1        0.20        perf-profile.self.cycles-pp.vms_complete_munmap_vmas
      0.32            -0.1        0.18 ?  2%  perf-profile.self.cycles-pp.free_pgtables
      0.44            -0.1        0.31        perf-profile.self.cycles-pp.vma_complete
      0.32            -0.1        0.19 ?  2%  perf-profile.self.cycles-pp.tlb_finish_mmu
      0.28            -0.1        0.15        perf-profile.self.cycles-pp.vma_adjust_trans_huge
      0.29            -0.1        0.17 ?  2%  perf-profile.self.cycles-pp.sized_strscpy
      0.24            -0.1        0.12        perf-profile.self.cycles-pp.tlb_gather_mmu
      0.26            -0.1        0.14 ?  2%  perf-profile.self.cycles-pp.unmap_vmas
      0.37            -0.1        0.25 ?  2%  perf-profile.self.cycles-pp.mas_destroy
      0.29            -0.1        0.17        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      0.26            -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.zap_pmd_range
      0.23            -0.1        0.12        perf-profile.self.cycles-pp.mas_prev
      0.27            -0.1        0.16 ?  2%  perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.22 ?  2%      -0.1        0.11        perf-profile.self.cycles-pp.can_vma_merge_left
      0.23 ?  2%      -0.1        0.13        perf-profile.self.cycles-pp.x64_sys_call
      0.24            -0.1        0.14 ?  2%  perf-profile.self.cycles-pp.kfree
      0.19            -0.1        0.09 ?  3%  perf-profile.self.cycles-pp.__pte_offset_map_lock
      0.24            -0.1        0.14        perf-profile.self.cycles-pp.up_write
      0.24 ?  2%      -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.vma_mark_detached
      0.22            -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.vma_merge_new_range
      0.23 ?  2%      -0.1        0.14 ?  2%  perf-profile.self.cycles-pp.__account_obj_stock
      0.24 ?  2%      -0.1        0.15 ?  2%  perf-profile.self.cycles-pp.downgrade_write
      0.22            -0.1        0.13 ?  3%  perf-profile.self.cycles-pp.mas_wr_slot_store
      0.22            -0.1        0.13        perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.23 ?  2%      -0.1        0.14        perf-profile.self.cycles-pp.__get_unmapped_area
      0.19            -0.1        0.11 ?  3%  perf-profile.self.cycles-pp._raw_spin_lock
      0.16            -0.1        0.08 ?  5%  perf-profile.self.cycles-pp.__put_partials
      0.22 ?  2%      -0.1        0.14 ?  3%  perf-profile.self.cycles-pp.may_expand_vm
      0.18 ?  2%      -0.1        0.10 ?  4%  perf-profile.self.cycles-pp.refill_obj_stock
      0.18 ?  2%      -0.1        0.10 ?  3%  perf-profile.self.cycles-pp.userfaultfd_unmap_complete
      0.17 ?  2%      -0.1        0.10        perf-profile.self.cycles-pp.obj_cgroup_charge_account
      0.19            -0.1        0.12        perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
      0.15            -0.1        0.08 ?  5%  perf-profile.self.cycles-pp.strnlen
      0.14            -0.1        0.08 ?  6%  perf-profile.self.cycles-pp.free_pgd_range
      0.15            -0.1        0.09 ?  4%  perf-profile.self.cycles-pp.vm_area_init_from
      0.12 ?  2%      -0.1        0.06        perf-profile.self.cycles-pp.unlink_anon_vmas
      0.30            -0.1        0.24        perf-profile.self.cycles-pp.static_key_count
      0.14            -0.1        0.08        perf-profile.self.cycles-pp.strlen
      0.12            -0.1        0.06        perf-profile.self.cycles-pp.___pte_offset_map
      0.15            -0.1        0.09        perf-profile.self.cycles-pp.vma_expand
      0.13            -0.1        0.07 ?  4%  perf-profile.self.cycles-pp.unmap_single_vma
      0.12 ?  4%      -0.1        0.07        perf-profile.self.cycles-pp.__vm_enough_memory
      0.11 ?  4%      -0.1        0.06        perf-profile.self.cycles-pp.anon_vma_clone
      0.13 ?  3%      -0.1        0.08 ?  4%  perf-profile.self.cycles-pp.update_rq_clock_task
      0.12 ?  3%      -0.1        0.07 ?  5%  perf-profile.self.cycles-pp.finish_rcuwait
      0.11            -0.1        0.06        perf-profile.self.cycles-pp.is_mergeable_anon_vma
      0.11            -0.1        0.06        perf-profile.self.cycles-pp.vma_prepare
      0.13 ?  2%      -0.0        0.08        perf-profile.self.cycles-pp.check_brk_limits
      0.12 ?  4%      -0.0        0.07 ?  6%  perf-profile.self.cycles-pp.cap_capable
      0.11            -0.0        0.07 ?  6%  perf-profile.self.cycles-pp.vm_area_dup
      0.11 ?  3%      -0.0        0.07 ?  8%  perf-profile.self.cycles-pp.get_partial_node
      0.09            -0.0        0.05        perf-profile.self.cycles-pp.security_mmap_addr
      0.09 ?  5%      -0.0        0.06 ?  5%  perf-profile.self.cycles-pp.testcase
      0.10            -0.0        0.07 ?  4%  perf-profile.self.cycles-pp.mas_node_count_gfp
      0.08 ?  4%      -0.0        0.05        perf-profile.self.cycles-pp.remove_vma
      0.08 ?  4%      -0.0        0.05        perf-profile.self.cycles-pp.userfaultfd_unmap_prep
      0.22            -0.0        0.21        perf-profile.self.cycles-pp.mas_prev_range
      0.00            +0.1        0.08 ?  8%  perf-profile.self.cycles-pp.__pcs_replace_full_main
      0.00            +0.1        0.12        perf-profile.self.cycles-pp.barn_put_empty_sheaf
      0.11            +0.1        0.24 ?  2%  perf-profile.self.cycles-pp.ksm_vma_flags
      0.43 ?  2%      +0.2        0.59 ?  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.00            +0.2        0.16 ?  7%  perf-profile.self.cycles-pp.__kmem_cache_free_bulk
      0.16 ?  2%      +0.2        0.32 ?  3%  perf-profile.self.cycles-pp.rcu_do_batch
      0.00            +0.2        0.19 ?  4%  perf-profile.self.cycles-pp.barn_replace_empty_sheaf
      0.00            +0.2        0.25        perf-profile.self.cycles-pp.barn_get_empty_sheaf
      1.20 ?  2%      +0.6        1.79 ?  2%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.00            +0.7        0.70 ?  6%  perf-profile.self.cycles-pp.build_detached_freelist
      0.00            +0.8        0.82 ?  4%  perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
      1.27            +1.7        2.93 ?  3%  perf-profile.self.cycles-pp.kmem_cache_free
      3.38 ?  3%      +4.5        7.87 ?  5%  perf-profile.self.cycles-pp.rcu_cblist_dequeue
     25.36 ?  2%     +23.0       48.39        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



From harry.yoo at oracle.com  Wed Sep 17 06:34:10 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Wed, 17 Sep 2025 22:34:10 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
Message-ID: <aMq40h5iOjj8K7cc@hyeyoo>

On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> On 9/17/25 15:07, Harry Yoo wrote:
> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> >> On 9/17/25 13:32, Harry Yoo wrote:
> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >> >> >> +				sfw->skip = true;
> >> >> >> +				continue;
> >> >> >> +			}
> >> >> >>
> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >> >> >> +			sfw->skip = false;
> >> >> >> +			sfw->s = s;
> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >> >> >> +			flushed = true;
> >> >> >> +		}
> >> >> >> +
> >> >> >> +		for_each_online_cpu(cpu) {
> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> >> >> +			if (sfw->skip)
> >> >> >> +				continue;
> >> >> >> +			flush_work(&sfw->work);
> >> >> >> +		}
> >> >> >> +
> >> >> >> +		mutex_unlock(&flush_lock);
> >> >> >> +	}
> >> >> >> +
> >> >> >> +	mutex_unlock(&slab_mutex);
> >> >> >> +	cpus_read_unlock();
> >> >> >> +
> >> >> >> +	if (flushed)
> >> >> >> +		rcu_barrier();
> >> >> > 
> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> >> >> > 
> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> >> >> > so flushed == false but the rcu callback isn't processed yet
> >> >> > by the end of the function?
> >> >> > 
> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> >> >> > but still possible...
> >> >> 
> >> >> Yes also good point, will flush unconditionally.
> >> >> 
> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> >> >> local_unlock().
> >> >>
> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> >> > 
> >> > Makes sense to me.
> > 
> > Wait, I'm confused.
> > 
> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> > the object X to be freed before kvfree_rcu_barrier() returns?
> 
> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> sheaf also contains the object X, we should make sure that is flushed.

I was going to say "but we queue and wait for the flushing work to
complete, so the sheaf containing object X should be flushed?"

But nah, that's true only if we see pcs->rcu_free != NULL in
flush_all_rcu_sheaves().

You are right...

Hmm, maybe it's simpler to fix this by never skipping queueing the work
even when pcs->rcu_sheaf == NULL?

> > IOW if flush_all_rcu_sheaves() is called while __kfree_rcu_sheaf(s, X) was
> > running on another CPU, we don't have to guarantee that
> > flush_all_rcu_sheaves() returns after the object X is freed?
> > 
> >> >> But then rcu_barrier() itself probably won't mean we make sure such cpus
> >> >> finished the local_locked section, if we didn't queue work on them. So maybe
> >> >> we need synchronize_rcu()?
> > 
> > So... we don't need a synchronize_rcu() then?
> > 
> > Or my brain started malfunctioning again :D

-- 
Cheers,
Harry / Hyeonggon


From vbabka at suse.cz  Wed Sep 17 07:14:36 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Wed, 17 Sep 2025 16:14:36 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aMq40h5iOjj8K7cc@hyeyoo>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo> <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo>
Message-ID: <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>

On 9/17/25 15:34, Harry Yoo wrote:
> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
>> On 9/17/25 15:07, Harry Yoo wrote:
>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
>> >> On 9/17/25 13:32, Harry Yoo wrote:
>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> >> >> >> +				sfw->skip = true;
>> >> >> >> +				continue;
>> >> >> >> +			}
>> >> >> >>
>> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> >> >> >> +			sfw->skip = false;
>> >> >> >> +			sfw->s = s;
>> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>> >> >> >> +			flushed = true;
>> >> >> >> +		}
>> >> >> >> +
>> >> >> >> +		for_each_online_cpu(cpu) {
>> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >> >> >> +			if (sfw->skip)
>> >> >> >> +				continue;
>> >> >> >> +			flush_work(&sfw->work);
>> >> >> >> +		}
>> >> >> >> +
>> >> >> >> +		mutex_unlock(&flush_lock);
>> >> >> >> +	}
>> >> >> >> +
>> >> >> >> +	mutex_unlock(&slab_mutex);
>> >> >> >> +	cpus_read_unlock();
>> >> >> >> +
>> >> >> >> +	if (flushed)
>> >> >> >> +		rcu_barrier();
>> >> >> > 
>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
>> >> >> > 
>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>> >> >> > so flushed == false but the rcu callback isn't processed yet
>> >> >> > by the end of the function?
>> >> >> > 
>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
>> >> >> > but still possible...
>> >> >> 
>> >> >> Yes also good point, will flush unconditionally.
>> >> >> 
>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>> >> >> local_unlock().
>> >> >>
>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
>> >> > 
>> >> > Makes sense to me.
>> > 
>> > Wait, I'm confused.
>> > 
>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
>> > the object X to be freed before kvfree_rcu_barrier() returns?
>> 
>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
>> sheaf also contains the object X, we should make sure that is flushed.
> 
> I was going to say "but we queue and wait for the flushing work to
> complete, so the sheaf containing object X should be flushed?"
> 
> But nah, that's true only if we see pcs->rcu_free != NULL in
> flush_all_rcu_sheaves().
> 
> You are right...
> 
> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> even when pcs->rcu_sheaf == NULL?

I guess it's simpler, yeah.
We might have to think of something better once all caches have sheaves,
queueing and waiting for work to finish on each cpu, repeated for each
kmem_cache, might be just too much?

>> > IOW if flush_all_rcu_sheaves() is called while __kfree_rcu_sheaf(s, X) was
>> > running on another CPU, we don't have to guarantee that
>> > flush_all_rcu_sheaves() returns after the object X is freed?
>> > 
>> >> >> But then rcu_barrier() itself probably won't mean we make sure such cpus
>> >> >> finished the local_locked section, if we didn't queue work on them. So maybe
>> >> >> we need synchronize_rcu()?
>> > 
>> > So... we don't need a synchronize_rcu() then?
>> > 
>> > Or my brain started malfunctioning again :D
> 



From surenb at google.com  Wed Sep 17 09:14:37 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Wed, 17 Sep 2025 09:14:37 -0700
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <aMpE-oSjtlDU4TSl@pc636>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com> <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop> <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com> <aMpE-oSjtlDU4TSl@pc636>
Message-ID: <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>

On Tue, Sep 16, 2025 at 10:19?PM Uladzislau Rezki <urezki at gmail.com> wrote:
>
> On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> > On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > >
> > > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > > >>
> > > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > > >> >
> > > >> >Summary of the results:
> > >
> > > In any case, thanks a lot for the results!
> > >
> > > >> >- Significant change (meaning >10% difference
> > > >> >  between base and experiment) on will-it-scale
> > > >> >  tests in AMD.
> > > >> >
> > > >> >Summary of AMD will-it-scale test changes:
> > > >> >
> > > >> >Number of runs : 15
> > > >> >Direction      : + is good
> > > >>
> > > >> If STDDEV grows more than mean, there is more jitter,
> > > >> which is not "good".
> > > >
> > > > This is true.  On the other hand, the mean grew way more in absolute
> > > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> > >
> > > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > > there's always an improvement for this config. So jitter here means it's
> > > changing between better and more better :) and not between worse and (more)
> > > better.
> > >
> > > The annoying part of course is that for other configs it's consistently the
> > > opposite.
> >
> > Hi Vlastimil,
> > I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> > faulting them in then unmapping and timing only mmap and munmap calls.
> > This is not a realistic scenario but works well for A/B comparison.
> >
> > The numbers are below with sheaves showing a clear improvement:
> >
> > Baseline
> >             avg             stdev
> > mmap        2.621073        0.2525161631
> > munmap      2.292965        0.008831973052
> > total       4.914038        0.2572620923
> >
> > Sheaves
> >             avg            stdev           avg_diff        stdev_diff
> > mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> > munmap      2.042071       0.03603083448   -10.94%        307.96%
> > total       3.603291667    0.113209047     -26.67%        -55.99%
> >
> Could you run your test with dropping below patch?

Sure, will try later today and report.

>
> [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations
>
> mmap()/munmap(), i assume it is a duration time in average, is the time
> in microseconds?

Yeah, it ends up being in microseconds. The actual reported time is
the total time in seconds that all mmap/munmap in the test consumed.
With 20000 cycles of 50 mmap/munmap calls we end up with 1000000
syscalls, so the number can be considered as duration in microseconds
for a single call.

>
> Thank you.
>
> --
> Uladzislau Rezki


From surenb at google.com  Wed Sep 17 16:59:41 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Wed, 17 Sep 2025 16:59:41 -0700
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com> <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop> <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
 <aMpE-oSjtlDU4TSl@pc636> <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>
Message-ID: <CAJuCfpE48n=QM0nY8yE3drqZU0wgC76=70EyftL1WZewEcykyA@mail.gmail.com>

On Wed, Sep 17, 2025 at 9:14?AM Suren Baghdasaryan <surenb at google.com> wrote:
>
> On Tue, Sep 16, 2025 at 10:19?PM Uladzislau Rezki <urezki at gmail.com> wrote:
> >
> > On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> > > On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > > >
> > > > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > > > >>
> > > > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > > > >> >
> > > > >> >Summary of the results:
> > > >
> > > > In any case, thanks a lot for the results!
> > > >
> > > > >> >- Significant change (meaning >10% difference
> > > > >> >  between base and experiment) on will-it-scale
> > > > >> >  tests in AMD.
> > > > >> >
> > > > >> >Summary of AMD will-it-scale test changes:
> > > > >> >
> > > > >> >Number of runs : 15
> > > > >> >Direction      : + is good
> > > > >>
> > > > >> If STDDEV grows more than mean, there is more jitter,
> > > > >> which is not "good".
> > > > >
> > > > > This is true.  On the other hand, the mean grew way more in absolute
> > > > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> > > >
> > > > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > > > there's always an improvement for this config. So jitter here means it's
> > > > changing between better and more better :) and not between worse and (more)
> > > > better.
> > > >
> > > > The annoying part of course is that for other configs it's consistently the
> > > > opposite.
> > >
> > > Hi Vlastimil,
> > > I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> > > faulting them in then unmapping and timing only mmap and munmap calls.
> > > This is not a realistic scenario but works well for A/B comparison.
> > >
> > > The numbers are below with sheaves showing a clear improvement:
> > >
> > > Baseline
> > >             avg             stdev
> > > mmap        2.621073        0.2525161631
> > > munmap      2.292965        0.008831973052
> > > total       4.914038        0.2572620923
> > >
> > > Sheaves
> > >             avg            stdev           avg_diff        stdev_diff
> > > mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> > > munmap      2.042071       0.03603083448   -10.94%        307.96%
> > > total       3.603291667    0.113209047     -26.67%        -55.99%
> > >
> > Could you run your test with dropping below patch?
>
> Sure, will try later today and report.

Sheaves with [04/23] patch reverted:

            avg             avg_diff
mmap     2.143948        -18.20%
munmap     2.343707        2.21%
total     4.487655        -8.68%


>
> >
> > [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations
> >
> > mmap()/munmap(), i assume it is a duration time in average, is the time
> > in microseconds?
>
> Yeah, it ends up being in microseconds. The actual reported time is
> the total time in seconds that all mmap/munmap in the test consumed.
> With 20000 cycles of 50 mmap/munmap calls we end up with 1000000
> syscalls, so the number can be considered as duration in microseconds
> for a single call.
>
> >
> > Thank you.
> >
> > --
> > Uladzislau Rezki


From oliver.sang at intel.com  Thu Sep 18 00:04:14 2025
From: oliver.sang at intel.com (kernel test robot)
Date: Thu, 18 Sep 2025 15:04:14 +0800
Subject: [vbabka:slub-percpu-sheaves-v8r1] [maple_tree]  d446df8076:
 stress-ng.memfd.ops_per_sec 4.4% improvement
Message-ID: <202509180858.9aff849c-lkp@intel.com>



Hello,

kernel test robot noticed a 4.4% improvement of stress-ng.memfd.ops_per_sec on:


commit: d446df8076dc9c96a7b6d484789da814c19915ec ("maple_tree: Use kfree_rcu in ma_free_rcu")
https://git.kernel.org/cgit/linux/kernel/git/vbabka/linux.git slub-percpu-sheaves-v8r1

testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 192 threads 2 sockets Intel(R) Xeon(R) Platinum 8468V  CPU @ 2.4GHz (Sapphire Rapids) with 384G memory
parameters:

	nr_threads: 100%
	testtime: 60s
	test: memfd
	cpufreq_governor: performance


Details are as below:
-------------------------------------------------------------------------------------------------->


The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250918/202509180858.9aff849c-lkp at intel.com

=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
  gcc-14/performance/x86_64-rhel-9.4/100%/debian-13-x86_64-20250902.cgz/igk-spr-2sp2/memfd/stress-ng/60s

commit: 
  eb8b293b99 ("testing/radix-tree/maple: Hack around kfree_rcu not existing")
  d446df8076 ("maple_tree: Use kfree_rcu in ma_free_rcu")

eb8b293b995be7f2 d446df8076dc9c96a7b6d484789 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
     85615            +6.6%      91242        stress-ng.memfd.nanosecs_per_memfd_create_call
    216252            +4.4%     225722        stress-ng.memfd.ops
      3604            +4.4%       3762        stress-ng.memfd.ops_per_sec
     49.85            -0.6%      49.53        kmsg.timestamp:LKP:stdout:#:Kernel_tests:Boot_OK
      3.41            -0.2%       3.41        kmsg.timestamp:T0#cpu:SGX_disabled_or_unsupported_by_BIOS
      4.81 ?264%    +199.6%      14.42 ?129%  kmsg.timestamp:T11]#xxx#:#b:#:Failed_to_load_MMP_firmware_qat_4xxx_mmp.bin
      4.81 ?264%    +199.7%      14.43 ?129%  kmsg.timestamp:T11]#xxx#:#b:#:Failed_to_load_acceleration_FW
      0.00       +4.8e+102%       4.79 ?264%  kmsg.timestamp:T9]#xxx#:#b:#:Failed_to_load_MMP_firmware_qat_4xxx_mmp.bin
      0.00       +4.8e+102%       4.79 ?264%  kmsg.timestamp:T9]#xxx#:#b:#:Failed_to_load_acceleration_FW
     38.71            -0.4%      38.56        kmsg.timestamp:ice###:Fail_during_requesting_FW
     38.69            -0.4%      38.54        kmsg.timestamp:ice###:The_DDP_package_file_was_not_found_or_could_not_be_read.Entering_Safe_Mode
     99.48            -0.5%      99.02        kmsg.timestamp:last
     38.77            -0.4%      38.63        kmsg.timestamp:xxx###:Failed_to_load_MMP_firmware_qat_4xxx_mmp.bin
     38.78            -0.4%      38.65        kmsg.timestamp:xxx###:Failed_to_load_acceleration_FW
     33.73 ? 37%     -43.0%      19.22 ?100%  kmsg.timestamp:xxx#:#b:#:Failed_to_load_MMP_firmware_qat_4xxx_mmp.bin
     33.74 ? 37%     -43.0%      19.22 ?100%  kmsg.timestamp:xxx#:#b:#:Failed_to_load_acceleration_FW
     99.48            -0.5%      99.02        dmesg.timestamp:last
     48.70            -0.4%      48.49        boot-time.boot
      8629            -0.4%       8593        boot-time.idle
 3.091e+08 ?  6%      +3.5%    3.2e+08 ?  4%  cpuidle..time
     97898 ?  6%      -1.5%      96387 ?  7%  cpuidle..usage
    111.32            -0.2%     111.11        uptime.boot
      9125            -0.4%       9088        uptime.idle
  56649987            +4.8%   59352433        numa-numastat.node0.local_node
  56752404            +4.8%   59458171        numa-numastat.node0.numa_hit
     99218 ? 64%      +3.3%     102454 ? 77%  numa-numastat.node0.other_node
  57106292            +4.5%   59671441        numa-numastat.node1.local_node
  57211560            +4.5%   59770935        numa-numastat.node1.numa_hit
     99387 ? 64%      -3.2%      96240 ? 81%  numa-numastat.node1.other_node
      0.85 ?  4%     -40.5%       0.50 ?  3%  perf-sched.sch_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
     24.82 ? 39%     +98.5%      49.26 ? 88%  perf-sched.sch_delay.max.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
      0.85 ?  4%     -40.5%       0.50 ?  3%  perf-sched.total_sch_delay.average.ms
     24.82 ? 39%     +98.5%      49.26 ? 88%  perf-sched.total_sch_delay.max.ms
    106.45 ?  7%     -28.8%      75.76 ?  2%  perf-sched.total_wait_and_delay.average.ms
     20529 ? 13%     +78.5%      36649 ?  2%  perf-sched.total_wait_and_delay.count.ms
      3941 ?  8%      -6.2%       3695 ?  6%  perf-sched.total_wait_and_delay.max.ms
    105.60 ?  8%     -28.7%      75.26 ?  2%  perf-sched.total_wait_time.average.ms
      3940 ?  8%      -6.3%       3693 ?  6%  perf-sched.total_wait_time.max.ms
    106.45 ?  7%     -28.8%      75.76 ?  2%  perf-sched.wait_and_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
     20529 ? 13%     +78.5%      36649 ?  2%  perf-sched.wait_and_delay.count.[unknown].[unknown].[unknown].[unknown].[unknown]
      3941 ?  8%      -6.2%       3695 ?  6%  perf-sched.wait_and_delay.max.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
    105.60 ?  8%     -28.7%      75.26 ?  2%  perf-sched.wait_time.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
      3940 ?  8%      -6.3%       3693 ?  6%  perf-sched.wait_time.max.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
   1224398 ?  7%      -4.7%    1166533        meminfo.Active
   1224382 ?  7%      -4.7%    1166517        meminfo.Active(anon)
     16.31            +0.3%      16.36        meminfo.Active(file)
     50010 ? 15%      +6.5%      53276 ? 13%  meminfo.AnonHugePages
    711203            +0.2%     712524        meminfo.AnonPages
      4.15 ?  4%      -1.2%       4.10 ?  4%  meminfo.Buffers
   4328057 ?  2%      -1.4%    4269024        meminfo.Cached
 1.979e+08            +0.0%  1.979e+08        meminfo.CommitLimit
   4645104 ?  2%      -1.3%    4582942        meminfo.Committed_AS
 3.954e+08            -0.1%  3.949e+08        meminfo.DirectMap1G
   9654784 ? 20%      +6.9%   10318080 ? 14%  meminfo.DirectMap2M
    170664 ?  3%      -4.7%     162728 ?  7%  meminfo.DirectMap4k
     14.35 ?129%     -33.7%       9.52 ?173%  meminfo.Dirty
      2048            +0.0%       2048        meminfo.Hugepagesize
    435.15 ?  5%      -1.1%     430.48 ?  4%  meminfo.Inactive
    435.15 ?  5%      -1.1%     430.48 ?  4%  meminfo.Inactive(file)
    308772            -2.6%     300749        meminfo.KReclaimable
     40882            -0.1%      40837        meminfo.KernelStack
    156270 ? 12%      -2.3%     152694 ? 10%  meminfo.Mapped
 3.853e+08            +0.0%  3.854e+08        meminfo.MemAvailable
 3.868e+08            +0.0%  3.869e+08        meminfo.MemFree
 3.958e+08            +0.0%  3.958e+08        meminfo.MemTotal
   9011343            -0.9%    8931416        meminfo.Memused
     15.42            +0.2%      15.45        meminfo.Mlocked
     48551 ?  3%      +2.3%      49657 ?  3%  meminfo.PageTables
     85054            -0.3%      84768        meminfo.Percpu
    308772            -2.6%     300749        meminfo.SReclaimable
   1120534 ?  8%      +6.2%    1189775 ?  3%  meminfo.SUnreclaim
     20640            +0.0%      20640        meminfo.SecPageTables
    515962 ? 18%     -11.4%     456931 ?  2%  meminfo.Shmem
   1429306 ?  6%      +4.3%    1490525 ?  2%  meminfo.Slab
   3814747            -0.0%    3814744        meminfo.Unevictable
 1.374e+13            +0.0%  1.374e+13        meminfo.VmallocTotal
    287600            -0.0%     287511        meminfo.VmallocUsed
   9302056            -0.6%    9243159        meminfo.max_used_kB
      1.90            -2.2%       1.86        perf-stat.i.MPKI
 1.894e+10            +1.5%  1.922e+10        perf-stat.i.branch-instructions
      0.51 ?  2%      -0.0        0.50        perf-stat.i.branch-miss-rate%
  91284450 ?  2%      +0.6%   91827325        perf-stat.i.branch-misses
     43.38            +1.0       44.41        perf-stat.i.cache-miss-rate%
 1.634e+08            -0.5%  1.626e+08        perf-stat.i.cache-misses
  3.82e+08 ?  2%      -2.9%   3.71e+08        perf-stat.i.cache-references
      4610 ?  6%     +65.0%       7607        perf-stat.i.context-switches
      6.32            -1.7%       6.22        perf-stat.i.cpi
 1.921e+11            +0.0%  1.921e+11        perf-stat.i.cpu-clock
 5.479e+11            +0.1%  5.482e+11        perf-stat.i.cpu-cycles
    302.49 ?  2%      +6.8%     322.98        perf-stat.i.cpu-migrations
      3331            +0.5%       3349        perf-stat.i.cycles-between-cache-misses
 8.614e+10            +1.7%  8.763e+10        perf-stat.i.instructions
      0.16            +1.8%       0.16        perf-stat.i.ipc
      0.05 ? 65%     -54.7%       0.02 ? 62%  perf-stat.i.major-faults
     19.01            +4.4%      19.84        perf-stat.i.metric.K/sec
   1825035            +4.4%    1905175        perf-stat.i.minor-faults
   1825035            +4.4%    1905175        perf-stat.i.page-faults
 1.921e+11            +0.0%  1.921e+11        perf-stat.i.task-clock
      1.90            -2.2%       1.86        perf-stat.overall.MPKI
      0.48            -0.0        0.48        perf-stat.overall.branch-miss-rate%
     42.78            +1.1       43.84        perf-stat.overall.cache-miss-rate%
      6.36            -1.7%       6.26        perf-stat.overall.cpi
      3352            +0.6%       3371        perf-stat.overall.cycles-between-cache-misses
      0.16            +1.7%       0.16        perf-stat.overall.ipc
 1.857e+10            +1.8%   1.89e+10        perf-stat.ps.branch-instructions
  88831156            +1.4%   90109350        perf-stat.ps.branch-misses
 1.603e+08            -0.3%  1.599e+08        perf-stat.ps.cache-misses
  3.75e+08 ?  3%      -2.7%  3.648e+08        perf-stat.ps.cache-references
      4502 ?  6%     +66.0%       7475        perf-stat.ps.context-switches
 1.883e+11            +0.3%  1.888e+11        perf-stat.ps.cpu-clock
 5.375e+11            +0.3%  5.391e+11        perf-stat.ps.cpu-cycles
    295.46 ?  3%      +7.3%     317.16        perf-stat.ps.cpu-migrations
 8.447e+10            +2.0%  8.617e+10        perf-stat.ps.instructions
      0.05 ? 66%     -54.3%       0.02 ? 62%  perf-stat.ps.major-faults
   1790235            +4.6%    1873425        perf-stat.ps.minor-faults
   1790235            +4.6%    1873425        perf-stat.ps.page-faults
 1.883e+11            +0.3%  1.888e+11        perf-stat.ps.task-clock
 5.124e+12            +2.4%  5.245e+12        perf-stat.total.instructions
    561023 ? 42%     -42.9%     320294 ? 51%  numa-meminfo.node0.Active
    561013 ? 42%     -42.9%     320283 ? 51%  numa-meminfo.node0.Active(anon)
      9.65 ? 50%     +15.9%      11.18 ? 46%  numa-meminfo.node0.Active(file)
     36983 ? 43%     -31.0%      25528 ? 88%  numa-meminfo.node0.AnonHugePages
    448115 ? 43%     -47.8%     234001 ? 73%  numa-meminfo.node0.AnonPages
    473848 ? 40%     -47.4%     249438 ? 68%  numa-meminfo.node0.AnonPages.max
      9.71 ?173%     -50.2%       4.84 ?264%  numa-meminfo.node0.Dirty
   1077003 ?138%    +124.9%    2422692 ? 71%  numa-meminfo.node0.FilePages
    165.74 ?115%      +1.2%     167.68 ?116%  numa-meminfo.node0.Inactive
    165.74 ?115%      +1.2%     167.68 ?116%  numa-meminfo.node0.Inactive(file)
    137669 ? 14%      +8.4%     149177 ? 18%  numa-meminfo.node0.KReclaimable
     20933 ? 10%      +2.4%      21428 ?  6%  numa-meminfo.node0.KernelStack
     36471 ?106%     -35.1%      23658 ? 62%  numa-meminfo.node0.Mapped
 1.941e+08            -0.6%  1.929e+08        numa-meminfo.node0.MemFree
 1.977e+08            +0.0%  1.977e+08        numa-meminfo.node0.MemTotal
   3551800 ? 41%     +35.4%    4810308 ? 35%  numa-meminfo.node0.MemUsed
      3.84 ?173%    +152.1%       9.68 ? 77%  numa-meminfo.node0.Mlocked
     24380 ? 45%      +3.2%      25156 ? 40%  numa-meminfo.node0.PageTables
    137669 ? 14%      +8.4%     149177 ? 18%  numa-meminfo.node0.SReclaimable
    627399 ? 24%     +23.2%     772697 ?  6%  numa-meminfo.node0.SUnreclaim
     10320            +0.0%      10320        numa-meminfo.node0.SecPageTables
    115703 ? 82%     -23.0%      89066 ? 43%  numa-meminfo.node0.Shmem
    765069 ? 18%     +20.5%     921875 ?  8%  numa-meminfo.node0.Slab
    964208 ?152%    +142.3%    2336527 ? 74%  numa-meminfo.node0.Unevictable
    667037 ? 44%     +26.6%     844571 ? 19%  numa-meminfo.node1.Active
    667030 ? 44%     +26.6%     844565 ? 19%  numa-meminfo.node1.Active(anon)
      6.67 ? 73%     -22.5%       5.17 ? 99%  numa-meminfo.node1.Active(file)
     13072 ? 96%    +112.4%      27772 ? 77%  numa-meminfo.node1.AnonHugePages
    263798 ? 73%     +81.3%     478363 ? 35%  numa-meminfo.node1.AnonPages
    473428 ? 37%     +41.8%     671200 ? 24%  numa-meminfo.node1.AnonPages.max
      4.68 ?264%      +3.4%       4.84 ?264%  numa-meminfo.node1.Dirty
   3253707 ? 46%     -43.3%    1844595 ? 93%  numa-meminfo.node1.FilePages
    269.44 ? 71%      -2.4%     262.96 ? 77%  numa-meminfo.node1.Inactive
    269.44 ? 71%      -2.4%     262.96 ? 77%  numa-meminfo.node1.Inactive(file)
    171088 ? 12%     -11.3%     151793 ? 18%  numa-meminfo.node1.KReclaimable
     19948 ? 11%      -2.6%      19439 ?  6%  numa-meminfo.node1.KernelStack
    119520 ? 37%      +7.7%     128781 ? 17%  numa-meminfo.node1.Mapped
 1.926e+08            +0.7%   1.94e+08        numa-meminfo.node1.MemFree
 1.981e+08            -0.0%  1.981e+08        numa-meminfo.node1.MemTotal
   5463778 ? 27%     -24.6%    4120509 ? 40%  numa-meminfo.node1.MemUsed
     11.58 ? 57%     -50.2%       5.77 ?129%  numa-meminfo.node1.Mlocked
     24165 ? 47%      +2.1%      24662 ? 43%  numa-meminfo.node1.PageTables
    171088 ? 12%     -11.3%     151793 ? 18%  numa-meminfo.node1.SReclaimable
    494648 ? 18%     -15.6%     417516 ?  2%  numa-meminfo.node1.SUnreclaim
     10320            +0.0%      10320        numa-meminfo.node1.SecPageTables
    402897 ? 33%      -9.1%     366108 ? 11%  numa-meminfo.node1.Shmem
    665737 ? 11%     -14.5%     569309 ?  5%  numa-meminfo.node1.Slab
   2850540 ? 51%     -48.1%    1478215 ?117%  numa-meminfo.node1.Unevictable
     69.38 ?  4%      -6.1%      65.12 ?  8%  proc-vmstat.direct_map_level2_splits
      4.75 ? 39%     +13.2%       5.38 ? 26%  proc-vmstat.direct_map_level3_splits
    306122 ?  8%      -4.7%     291692        proc-vmstat.nr_active_anon
      4.08            +0.3%       4.09        proc-vmstat.nr_active_file
    177825            +0.2%     178172        proc-vmstat.nr_anon_pages
     24.42 ? 15%      +6.7%      26.05 ? 13%  proc-vmstat.nr_anon_transparent_hugepages
     16.50 ?129%     -30.3%      11.50 ?173%  proc-vmstat.nr_dirtied
      3.59 ?129%     -33.7%       2.38 ?173%  proc-vmstat.nr_dirty
   9614010            +0.0%    9616077        proc-vmstat.nr_dirty_background_threshold
  19251527            +0.0%   19255667        proc-vmstat.nr_dirty_threshold
   1082006 ?  2%      -1.4%    1067246        proc-vmstat.nr_file_pages
  96696183            +0.0%   96716889        proc-vmstat.nr_free_pages
  96587674            +0.0%   96591486        proc-vmstat.nr_free_pages_blocks
    108.79 ?  5%      -1.1%     107.62 ?  4%  proc-vmstat.nr_inactive_file
      5160            +0.0%       5160        proc-vmstat.nr_iommu_pages
      0.00          +4e+99%       0.00 ?173%  proc-vmstat.nr_isolated_anon
     40882            -0.1%      40836        proc-vmstat.nr_kernel_stack
     38804 ? 13%      -1.5%      38224 ? 10%  proc-vmstat.nr_mapped
   1571840            +0.0%    1571840        proc-vmstat.nr_memmap_boot_pages
      3.85            +0.2%       3.86        proc-vmstat.nr_mlock
     12136 ?  3%      +2.2%      12409 ?  3%  proc-vmstat.nr_page_table_pages
      5160            +0.0%       5160        proc-vmstat.nr_sec_page_table_pages
    128975 ? 19%     -11.4%     114219 ?  2%  proc-vmstat.nr_shmem
     77176            -2.7%      75106        proc-vmstat.nr_slab_reclaimable
    280110 ?  8%      +6.1%     297296 ?  3%  proc-vmstat.nr_slab_unreclaimable
    953686            -0.0%     953686        proc-vmstat.nr_unevictable
     16.50 ?129%     -30.3%      11.50 ?173%  proc-vmstat.nr_written
    306122 ?  8%      -4.7%     291692        proc-vmstat.nr_zone_active_anon
      4.08            +0.3%       4.09        proc-vmstat.nr_zone_active_file
    108.79 ?  5%      -1.1%     107.62 ?  4%  proc-vmstat.nr_zone_inactive_file
    953686            -0.0%     953686        proc-vmstat.nr_zone_unevictable
      3.59 ?129%     -33.7%       2.38 ?173%  proc-vmstat.nr_zone_write_pending
    123885 ? 19%      +2.1%     126487 ? 17%  proc-vmstat.numa_hint_faults
    114821 ? 16%      +5.2%     120838 ? 17%  proc-vmstat.numa_hint_faults_local
  1.14e+08            +4.7%  1.193e+08        proc-vmstat.numa_hit
     20.62 ?139%     +70.9%      35.25 ? 51%  proc-vmstat.numa_huge_pte_updates
 1.138e+08            +4.7%  1.191e+08        proc-vmstat.numa_local
    198605            +0.0%     198694        proc-vmstat.numa_other
     13219 ? 64%     -19.2%      10676 ?100%  proc-vmstat.numa_pages_migrated
    197242 ? 28%      -2.1%     193036 ? 23%  proc-vmstat.numa_pte_updates
 1.217e+08            +6.4%  1.295e+08        proc-vmstat.pgalloc_normal
 1.114e+08            +4.4%  1.162e+08        proc-vmstat.pgfault
 1.211e+08            +6.5%   1.29e+08        proc-vmstat.pgfree
     13219 ? 64%     -19.2%      10676 ?100%  proc-vmstat.pgmigrate_success
     18.00 ?136%     -22.2%      14.00 ?175%  proc-vmstat.pgpgin
    136.50 ?129%     -30.4%      95.00 ?173%  proc-vmstat.pgpgout
     29018 ?  3%      -2.7%      28248 ?  2%  proc-vmstat.pgreuse
     31.75 ? 35%     +18.5%      37.62 ? 28%  proc-vmstat.thp_collapse_alloc
      0.38 ?129%     -66.7%       0.12 ?264%  proc-vmstat.thp_deferred_split_page
      8.38 ?160%     +17.9%       9.88 ?130%  proc-vmstat.thp_migration_success
    798.00            -0.0%     797.75        proc-vmstat.unevictable_pgs_culled
      4.00            +0.0%       4.00        proc-vmstat.unevictable_pgs_mlocked
      1.00 ?173%    -100.0%       0.00        proc-vmstat.unevictable_pgs_munlocked
    140267 ? 41%     -42.8%      80195 ? 51%  numa-vmstat.node0.nr_active_anon
      2.41 ? 50%     +15.8%       2.79 ? 46%  numa-vmstat.node0.nr_active_file
    112035 ? 43%     -47.7%      58542 ? 73%  numa-vmstat.node0.nr_anon_pages
     18.07 ? 43%     -31.6%      12.36 ? 87%  numa-vmstat.node0.nr_anon_transparent_hugepages
     10.50 ?173%     -47.6%       5.50 ?264%  numa-vmstat.node0.nr_dirtied
      2.39 ?173%     -48.5%       1.23 ?264%  numa-vmstat.node0.nr_dirty
    269249 ?138%    +125.0%     605717 ? 71%  numa-vmstat.node0.nr_file_pages
  48535760            -0.6%   48221056        numa-vmstat.node0.nr_free_pages
  48476817            -0.7%   48156334        numa-vmstat.node0.nr_free_pages_blocks
     41.51 ?115%      +0.7%      41.82 ?116%  numa-vmstat.node0.nr_inactive_file
      2580            +0.0%       2580        numa-vmstat.node0.nr_iommu_pages
      0.00          +2e+99%       0.00 ?264%  numa-vmstat.node0.nr_isolated_anon
     20932 ? 10%      +2.6%      21467 ?  6%  numa-vmstat.node0.nr_kernel_stack
      9113 ?106%     -34.9%       5929 ? 62%  numa-vmstat.node0.nr_mapped
      0.96 ?173%    +152.0%       2.42 ? 77%  numa-vmstat.node0.nr_mlock
      6095 ? 45%      +4.2%       6349 ? 40%  numa-vmstat.node0.nr_page_table_pages
      2580            +0.0%       2580        numa-vmstat.node0.nr_sec_page_table_pages
     28921 ? 82%     -22.9%      22312 ? 44%  numa-vmstat.node0.nr_shmem
     34405 ? 14%      +8.6%      37362 ? 18%  numa-vmstat.node0.nr_slab_reclaimable
    156893 ? 24%     +22.7%     192446 ?  6%  numa-vmstat.node0.nr_slab_unreclaimable
    241052 ?152%    +142.3%     584131 ? 74%  numa-vmstat.node0.nr_unevictable
     10.50 ?173%     -47.6%       5.50 ?264%  numa-vmstat.node0.nr_written
    140280 ? 41%     -42.8%      80191 ? 51%  numa-vmstat.node0.nr_zone_active_anon
      2.41 ? 50%     +15.8%       2.79 ? 46%  numa-vmstat.node0.nr_zone_active_file
     41.51 ?115%      +0.7%      41.82 ?116%  numa-vmstat.node0.nr_zone_inactive_file
    241052 ?152%    +142.3%     584131 ? 74%  numa-vmstat.node0.nr_zone_unevictable
      2.39 ?173%     -48.5%       1.23 ?264%  numa-vmstat.node0.nr_zone_write_pending
  56752707            +4.3%   59196927        numa-vmstat.node0.numa_hit
  56650290            +4.3%   59091188        numa-vmstat.node0.numa_local
     99218 ? 64%      +3.3%     102454 ? 77%  numa-vmstat.node0.numa_other
    166637 ? 44%     +25.8%     209680 ? 19%  numa-vmstat.node1.nr_active_anon
      1.67 ? 73%     -22.6%       1.29 ? 99%  numa-vmstat.node1.nr_active_file
     65991 ? 73%     +80.6%     119160 ? 36%  numa-vmstat.node1.nr_anon_pages
      6.37 ? 96%    +111.4%      13.47 ? 77%  numa-vmstat.node1.nr_anon_transparent_hugepages
      6.00 ?264%      +0.0%       6.00 ?264%  numa-vmstat.node1.nr_dirtied
      1.17 ?264%      +5.1%       1.23 ?264%  numa-vmstat.node1.nr_dirty
    813249 ? 46%     -43.4%     460096 ? 93%  numa-vmstat.node1.nr_file_pages
  48158852            +0.7%   48495377        numa-vmstat.node1.nr_free_pages
  48110347            +0.7%   48436077        numa-vmstat.node1.nr_free_pages_blocks
     67.36 ? 71%      -2.6%      65.63 ? 77%  numa-vmstat.node1.nr_inactive_file
      2580            +0.0%       2580        numa-vmstat.node1.nr_iommu_pages
      0.00          +2e+99%       0.00 ?264%  numa-vmstat.node1.nr_isolated_anon
     19947 ? 11%      -2.4%      19466 ?  6%  numa-vmstat.node1.nr_kernel_stack
     29927 ? 37%      +7.7%      32231 ? 17%  numa-vmstat.node1.nr_mapped
      2.90 ? 57%     -50.2%       1.44 ?129%  numa-vmstat.node1.nr_mlock
      6042 ? 47%      +2.8%       6214 ? 43%  numa-vmstat.node1.nr_page_table_pages
      2580            +0.0%       2580        numa-vmstat.node1.nr_sec_page_table_pages
    100545 ? 34%     -10.0%      90476 ? 11%  numa-vmstat.node1.nr_shmem
     42772 ? 12%     -11.1%      38034 ? 18%  numa-vmstat.node1.nr_slab_reclaimable
    123737 ? 18%     -15.6%     104446 ?  2%  numa-vmstat.node1.nr_slab_unreclaimable
    712634 ? 51%     -48.1%     369553 ?117%  numa-vmstat.node1.nr_unevictable
      6.00 ?264%      +0.0%       6.00 ?264%  numa-vmstat.node1.nr_written
    166631 ? 44%     +25.8%     209683 ? 19%  numa-vmstat.node1.nr_zone_active_anon
      1.67 ? 73%     -22.6%       1.29 ? 99%  numa-vmstat.node1.nr_zone_active_file
     67.36 ? 71%      -2.6%      65.63 ? 77%  numa-vmstat.node1.nr_zone_inactive_file
    712634 ? 51%     -48.1%     369553 ?117%  numa-vmstat.node1.nr_zone_unevictable
      1.17 ?264%      +5.1%       1.23 ?264%  numa-vmstat.node1.nr_zone_write_pending
  57211551            +4.0%   59511604        numa-vmstat.node1.numa_hit
  57106283            +4.0%   59412110        numa-vmstat.node1.numa_local
     99387 ? 64%      -3.2%      96240 ? 81%  numa-vmstat.node1.numa_other
   5867348            -0.1%    5861257        sched_debug.cfs_rq:/.avg_vruntime.avg
   5956538            -0.2%    5942404        sched_debug.cfs_rq:/.avg_vruntime.max
   5386265 ?  9%      +2.8%    5535354 ?  2%  sched_debug.cfs_rq:/.avg_vruntime.min
     64630 ? 46%      -9.1%      58726 ? 14%  sched_debug.cfs_rq:/.avg_vruntime.stddev
      0.54 ?  2%      +1.4%       0.55        sched_debug.cfs_rq:/.h_nr_queued.avg
      1.69 ? 14%     +25.9%       2.12 ? 10%  sched_debug.cfs_rq:/.h_nr_queued.max
      0.50            +0.0%       0.50        sched_debug.cfs_rq:/.h_nr_queued.min
      0.19 ? 17%     +17.0%       0.22 ?  9%  sched_debug.cfs_rq:/.h_nr_queued.stddev
      0.54 ?  2%      +0.9%       0.54        sched_debug.cfs_rq:/.h_nr_runnable.avg
      1.69 ? 14%     +22.2%       2.06 ?  8%  sched_debug.cfs_rq:/.h_nr_runnable.max
      0.50            +0.0%       0.50        sched_debug.cfs_rq:/.h_nr_runnable.min
      0.19 ? 17%     +13.1%       0.21 ?  6%  sched_debug.cfs_rq:/.h_nr_runnable.stddev
     60057 ?159%     +14.5%      68780 ? 98%  sched_debug.cfs_rq:/.left_deadline.avg
   3456228 ? 79%     +27.6%    4408987 ? 57%  sched_debug.cfs_rq:/.left_deadline.max
      0.00            +0.0%       0.00        sched_debug.cfs_rq:/.left_deadline.min
    402169 ?103%     +27.6%     513361 ? 70%  sched_debug.cfs_rq:/.left_deadline.stddev
     60056 ?159%     +14.5%      68779 ? 98%  sched_debug.cfs_rq:/.left_vruntime.avg
   3456191 ? 79%     +27.6%    4408921 ? 57%  sched_debug.cfs_rq:/.left_vruntime.max
      0.00            +0.0%       0.00        sched_debug.cfs_rq:/.left_vruntime.min
    402167 ?103%     +27.6%     513350 ? 70%  sched_debug.cfs_rq:/.left_vruntime.stddev
      8634 ? 99%     +16.9%      10093 ? 66%  sched_debug.cfs_rq:/.load.avg
    342965 ? 73%     +57.5%     540080 ? 67%  sched_debug.cfs_rq:/.load.max
      2625 ?  4%      +1.9%       2676        sched_debug.cfs_rq:/.load.min
     39369 ? 93%     +45.3%      57212 ? 71%  sched_debug.cfs_rq:/.load.stddev
     31.94 ? 21%      -0.2%      31.86 ? 22%  sched_debug.cfs_rq:/.load_avg.avg
    778.38 ? 33%     -27.5%     564.44 ?  3%  sched_debug.cfs_rq:/.load_avg.max
      2.44 ?  6%      +2.6%       2.50        sched_debug.cfs_rq:/.load_avg.min
    115.08 ? 15%     -14.3%      98.57 ? 13%  sched_debug.cfs_rq:/.load_avg.stddev
   5867354            -0.1%    5861258        sched_debug.cfs_rq:/.min_vruntime.avg
   5956538            -0.2%    5942404        sched_debug.cfs_rq:/.min_vruntime.max
   5386265 ?  9%      +2.8%    5535354 ?  2%  sched_debug.cfs_rq:/.min_vruntime.min
     64632 ? 46%      -9.1%      58726 ? 14%  sched_debug.cfs_rq:/.min_vruntime.stddev
      0.53            +1.0%       0.54        sched_debug.cfs_rq:/.nr_queued.avg
      1.31 ? 18%     +14.3%       1.50 ? 23%  sched_debug.cfs_rq:/.nr_queued.max
      0.50            +0.0%       0.50        sched_debug.cfs_rq:/.nr_queued.min
      0.14 ? 26%     +13.8%       0.16 ? 23%  sched_debug.cfs_rq:/.nr_queued.stddev
     15.94 ? 45%      -5.0%      15.15 ? 35%  sched_debug.cfs_rq:/.removed.load_avg.avg
    512.00            +0.0%     512.00        sched_debug.cfs_rq:/.removed.load_avg.max
     86.73 ? 20%      -2.3%      84.74 ? 17%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      7.49 ? 46%      +0.5%       7.53 ? 36%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
    260.38            +0.3%     261.19        sched_debug.cfs_rq:/.removed.runnable_avg.max
     41.38 ? 20%      +2.1%      42.24 ? 18%  sched_debug.cfs_rq:/.removed.runnable_avg.stddev
      7.49 ? 46%      +0.5%       7.53 ? 36%  sched_debug.cfs_rq:/.removed.util_avg.avg
    260.38            +0.3%     261.12        sched_debug.cfs_rq:/.removed.util_avg.max
     41.38 ? 20%      +2.1%      42.23 ? 18%  sched_debug.cfs_rq:/.removed.util_avg.stddev
     60056 ?159%     +14.5%      68779 ? 98%  sched_debug.cfs_rq:/.right_vruntime.avg
   3456191 ? 79%     +27.6%    4408921 ? 57%  sched_debug.cfs_rq:/.right_vruntime.max
      0.00            +0.0%       0.00        sched_debug.cfs_rq:/.right_vruntime.min
    402167 ?103%     +27.6%     513350 ? 70%  sched_debug.cfs_rq:/.right_vruntime.stddev
    599.15            +0.2%     600.49        sched_debug.cfs_rq:/.runnable_avg.avg
      1504 ?  6%      +2.5%       1542 ?  9%  sched_debug.cfs_rq:/.runnable_avg.max
    510.25            +0.3%     512.00        sched_debug.cfs_rq:/.runnable_avg.min
    161.25 ?  9%      +1.3%     163.34 ? 11%  sched_debug.cfs_rq:/.runnable_avg.stddev
    587.54            +0.1%     587.96        sched_debug.cfs_rq:/.util_avg.avg
      1096 ?  5%      +2.2%       1120 ?  7%  sched_debug.cfs_rq:/.util_avg.max
    393.06 ? 39%     +18.2%     464.44 ? 12%  sched_debug.cfs_rq:/.util_avg.min
    128.19 ? 10%      -2.8%     124.61 ?  3%  sched_debug.cfs_rq:/.util_avg.stddev
     12.29 ? 26%      +3.5%      12.73 ? 23%  sched_debug.cfs_rq:/.util_est.avg
    974.50 ?  9%      +5.1%       1023        sched_debug.cfs_rq:/.util_est.max
     92.43 ? 16%      +4.9%      96.94 ? 11%  sched_debug.cfs_rq:/.util_est.stddev
   1186407 ?  5%      -0.1%    1185208 ?  5%  sched_debug.cpu.avg_idle.avg
   2925176 ?  3%      -2.7%    2846682 ?  5%  sched_debug.cpu.avg_idle.max
    294241 ? 26%     -27.6%     212924 ? 30%  sched_debug.cpu.avg_idle.min
    482686 ?  9%      +0.5%     485288 ? 12%  sched_debug.cpu.avg_idle.stddev
     80740            -0.3%      80525        sched_debug.cpu.clock.avg
     80766            -0.3%      80555        sched_debug.cpu.clock.max
     80713            -0.3%      80492        sched_debug.cpu.clock.min
     15.55 ? 12%     +14.4%      17.78 ? 10%  sched_debug.cpu.clock.stddev
     80052            -0.2%      79873        sched_debug.cpu.clock_task.avg
     80347            -0.3%      80077        sched_debug.cpu.clock_task.max
     65765            -0.0%      65765        sched_debug.cpu.clock_task.min
      1043            -1.6%       1026        sched_debug.cpu.clock_task.stddev
      5032            +0.1%       5034        sched_debug.cpu.curr->pid.avg
      9236 ?  2%      +1.0%       9326        sched_debug.cpu.curr->pid.max
      2399 ? 64%     -21.2%       1891 ? 90%  sched_debug.cpu.curr->pid.min
    914.85 ? 18%     +16.6%       1066 ? 23%  sched_debug.cpu.curr->pid.stddev
    591468 ?  3%      -0.3%     589731 ?  3%  sched_debug.cpu.max_idle_balance_cost.avg
   1297017 ?  4%      -1.9%    1272169 ?  2%  sched_debug.cpu.max_idle_balance_cost.max
    500000            +0.0%     500000        sched_debug.cpu.max_idle_balance_cost.min
    170705 ? 12%      -1.8%     167551 ? 13%  sched_debug.cpu.max_idle_balance_cost.stddev
      4294            -0.0%       4294        sched_debug.cpu.next_balance.avg
      4294            -0.0%       4294        sched_debug.cpu.next_balance.max
      4294            -0.0%       4294        sched_debug.cpu.next_balance.min
      0.00 ? 61%     -14.1%       0.00 ?  6%  sched_debug.cpu.next_balance.stddev
      0.54            +1.0%       0.54        sched_debug.cpu.nr_running.avg
      1.75 ? 14%     +17.9%       2.06 ?  8%  sched_debug.cpu.nr_running.max
      0.50            +0.0%       0.50        sched_debug.cpu.nr_running.min
      0.19 ? 16%     +12.3%       0.21 ? 10%  sched_debug.cpu.nr_running.stddev
      1980 ?  2%     +26.2%       2498 ?  2%  sched_debug.cpu.nr_switches.avg
     46888 ?  8%      -1.2%      46330 ?  6%  sched_debug.cpu.nr_switches.max
    481.50 ? 13%    +122.3%       1070 ?  4%  sched_debug.cpu.nr_switches.min
      3646 ?  7%      +0.4%       3661 ?  6%  sched_debug.cpu.nr_switches.stddev
      0.01 ? 36%     -11.1%       0.01 ? 25%  sched_debug.cpu.nr_uninterruptible.avg
     33.44 ? 19%     +12.9%      37.75 ? 24%  sched_debug.cpu.nr_uninterruptible.max
    -15.62            +9.2%     -17.06        sched_debug.cpu.nr_uninterruptible.min
      5.26 ? 12%     +16.3%       6.12 ? 14%  sched_debug.cpu.nr_uninterruptible.stddev
     80713            -0.3%      80494        sched_debug.cpu_clk
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.avg
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.max
    996147            +0.0%     996147        sched_debug.dl_rq:.dl_bw->bw.min
  10066176            +0.0%   10066176        sched_debug.dl_rq:.dl_bw->total_bw.avg
  10066176            +0.0%   10066176        sched_debug.dl_rq:.dl_bw->total_bw.max
  10066176            +0.0%   10066176        sched_debug.dl_rq:.dl_bw->total_bw.min
 4.295e+09            -0.0%  4.295e+09        sched_debug.jiffies
     79668            -0.3%      79450        sched_debug.ktime
      0.00 ?264%    -100.0%       0.00        sched_debug.rt_rq:.rt_nr_running.avg
      0.06 ?264%    -100.0%       0.00        sched_debug.rt_rq:.rt_nr_running.max
      0.00 ?264%    -100.0%       0.00        sched_debug.rt_rq:.rt_nr_running.stddev
     81956            -0.2%      81765        sched_debug.sched_clk
      1.00            +0.0%       1.00        sched_debug.sched_clock_stable()
      2.80            +0.0%       2.80        sched_debug.sysctl_sched.sysctl_sched_base_slice
  32696287            +0.0%   32696287        sched_debug.sysctl_sched.sysctl_sched_features
      1.00            +0.0%       1.00        sched_debug.sysctl_sched.sysctl_sched_tunable_scaling
      5.44 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.close_range
      5.44 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close_range
      5.44 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close_range
      5.43 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close_range
      5.43 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.task_work_run.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close_range
      5.43 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.42 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_user_mode_loop.do_syscall_64
      5.42 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.__dentry_kill.dput.__fput.task_work_run.exit_to_user_mode_loop
      5.38 ?264%      -5.4        0.00        perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dput.__fput.task_work_run
      5.26 ?264%      -5.3        0.00        perf-profile.calltrace.cycles-pp.memfd_create
      5.26 ?264%      -5.3        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.memfd_create
      5.26 ?264%      -5.3        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.memfd_create
      5.26 ?264%      -5.3        0.00        perf-profile.calltrace.cycles-pp.__do_sys_memfd_create.do_syscall_64.entry_SYSCALL_64_after_hwframe.memfd_create
      5.25 ?264%      -5.3        0.00        perf-profile.calltrace.cycles-pp.__shmem_file_setup.__do_sys_memfd_create.do_syscall_64.entry_SYSCALL_64_after_hwframe.memfd_create
      5.20 ?264%      -5.2        0.00        perf-profile.calltrace.cycles-pp.__shmem_get_inode.__shmem_file_setup.__do_sys_memfd_create.do_syscall_64.entry_SYSCALL_64_after_hwframe
      5.18 ?264%      -5.2        0.00        perf-profile.calltrace.cycles-pp.new_inode.__shmem_get_inode.__shmem_file_setup.__do_sys_memfd_create.do_syscall_64
      5.14 ?264%      -5.1        0.00        perf-profile.calltrace.cycles-pp.inode_sb_list_add.new_inode.__shmem_get_inode.__shmem_file_setup.__do_sys_memfd_create
      5.13 ?264%      -5.1        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.inode_sb_list_add.new_inode.__shmem_get_inode.__shmem_file_setup
      5.12 ?264%      -5.1        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.inode_sb_list_add.new_inode.__shmem_get_inode
      5.05 ?264%      -5.1        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock.evict.__dentry_kill.dput.__fput
      5.03 ?264%      -5.0        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.evict.__dentry_kill.dput
      1.19 ?264%      -1.2        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.18 ?264%      -1.2        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.13 ?264%      -1.1        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_fallocate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.12 ?264%      -1.1        0.00        perf-profile.calltrace.cycles-pp.vfs_fallocate.__x64_sys_fallocate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.12 ?264%      -1.1        0.00        perf-profile.calltrace.cycles-pp.shmem_fallocate.vfs_fallocate.__x64_sys_fallocate.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.08 ?264%      -1.1        0.00        perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_fallocate.vfs_fallocate.__x64_sys_fallocate.do_syscall_64
      0.75 ?264%      -0.7        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.folio_lruvec_lock_irqsave.__page_cache_release.folios_put_refs.shmem_undo_range
      0.74 ?264%      -0.7        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.folio_lruvec_lock_irqsave.__page_cache_release.folios_put_refs
      0.54 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.folios_put_refs.shmem_undo_range.shmem_fallocate.vfs_fallocate.__x64_sys_fallocate
      0.53 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.__folio_batch_release.shmem_undo_range.shmem_fallocate.vfs_fallocate.__x64_sys_fallocate
      0.53 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.folio_batch_move_lru.lru_add_drain_cpu.__folio_batch_release.shmem_undo_range.shmem_fallocate
      0.53 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.lru_add_drain_cpu.__folio_batch_release.shmem_undo_range.shmem_fallocate.vfs_fallocate
      0.51 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.__page_cache_release.folios_put_refs.shmem_undo_range.shmem_fallocate.vfs_fallocate
      0.50 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.folio_lruvec_lock_irqsave.__page_cache_release.folios_put_refs.shmem_undo_range.shmem_fallocate
      0.50 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.folio_lruvec_lock_irqsave.folio_batch_move_lru.lru_add_drain_cpu.__folio_batch_release
      0.50 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.folio_lruvec_lock_irqsave.folio_batch_move_lru.lru_add_drain_cpu.__folio_batch_release.shmem_undo_range
      0.49 ?264%      -0.5        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.folio_lruvec_lock_irqsave.folio_batch_move_lru.lru_add_drain_cpu
      0.32 ?264%      -0.3        0.00        perf-profile.calltrace.cycles-pp.shmem_evict_inode.evict.__dentry_kill.dput.__fput
      0.32 ?264%      -0.3        0.00        perf-profile.calltrace.cycles-pp.shmem_undo_range.shmem_evict_inode.evict.__dentry_kill.dput
      0.28 ?264%      -0.3        0.00        perf-profile.calltrace.cycles-pp.folios_put_refs.shmem_undo_range.shmem_evict_inode.evict.__dentry_kill
      0.26 ?264%      -0.3        0.00        perf-profile.calltrace.cycles-pp.__page_cache_release.folios_put_refs.shmem_undo_range.shmem_evict_inode.evict
      0.25 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.folio_lruvec_lock_irqsave.__page_cache_release.folios_put_refs.shmem_undo_range.shmem_evict_inode
      0.19 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.stress_memfd_child
      0.18 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.__mmap
      0.18 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.asm_exc_page_fault.stress_memfd_child
      0.16 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__mmap
      0.16 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      0.16 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      0.16 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
      0.15 ?264%      -0.2        0.00        perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.15 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.stress_memfd_child
      0.15 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.stress_memfd_child
      0.14 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.stress_memfd_child
      0.13 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
      0.13 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64
      0.13 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__mmap_region.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff
      0.12 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_pte_missing.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
      0.12 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_shared_fault.do_pte_missing.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__do_fault.do_shared_fault.do_pte_missing.__handle_mm_fault.handle_mm_fault
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.shmem_fault.__do_fault.do_shared_fault.do_pte_missing.__handle_mm_fault
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.shmem_get_folio_gfp.shmem_fault.__do_fault.do_shared_fault.do_pte_missing
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__munmap
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__mmap_new_vma.__mmap_region.mmap_region.do_mmap.vm_mmap_pgoff
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      0.10 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.inode_sb_list_add.new_inode
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.inode_sb_list_add
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap.do_syscall_64
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.evict.__dentry_kill
      0.09 ?264%      -0.1        0.00        perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.evict
     12.21 ?264%     -12.2        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     12.20 ?264%     -12.2        0.00        perf-profile.children.cycles-pp.do_syscall_64
     11.46 ?264%     -11.5        0.00        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     10.28 ?264%     -10.3        0.00        perf-profile.children.cycles-pp._raw_spin_lock
      5.48 ?264%      -5.5        0.00        perf-profile.children.cycles-pp.__fput
      5.48 ?264%      -5.5        0.00        perf-profile.children.cycles-pp.dput
      5.47 ?264%      -5.5        0.00        perf-profile.children.cycles-pp.__dentry_kill
      5.44 ?264%      -5.4        0.00        perf-profile.children.cycles-pp.close_range
      5.43 ?264%      -5.4        0.00        perf-profile.children.cycles-pp.exit_to_user_mode_loop
      5.43 ?264%      -5.4        0.00        perf-profile.children.cycles-pp.task_work_run
      5.43 ?264%      -5.4        0.00        perf-profile.children.cycles-pp.evict
      5.26 ?264%      -5.3        0.00        perf-profile.children.cycles-pp.memfd_create
      5.26 ?264%      -5.3        0.00        perf-profile.children.cycles-pp.__do_sys_memfd_create
      5.25 ?264%      -5.3        0.00        perf-profile.children.cycles-pp.__shmem_file_setup
      5.20 ?264%      -5.2        0.00        perf-profile.children.cycles-pp.__shmem_get_inode
      5.18 ?264%      -5.2        0.00        perf-profile.children.cycles-pp.new_inode
      5.14 ?264%      -5.1        0.00        perf-profile.children.cycles-pp.inode_sb_list_add
      1.40 ?264%      -1.4        0.00        perf-profile.children.cycles-pp.shmem_undo_range
      1.25 ?264%      -1.3        0.00        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.24 ?264%      -1.2        0.00        perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave
      1.13 ?264%      -1.1        0.00        perf-profile.children.cycles-pp.__x64_sys_fallocate
      1.12 ?264%      -1.1        0.00        perf-profile.children.cycles-pp.vfs_fallocate
      1.12 ?264%      -1.1        0.00        perf-profile.children.cycles-pp.shmem_fallocate
      0.82 ?264%      -0.8        0.00        perf-profile.children.cycles-pp.folios_put_refs
      0.77 ?264%      -0.8        0.00        perf-profile.children.cycles-pp.__page_cache_release
      0.53 ?264%      -0.5        0.00        perf-profile.children.cycles-pp.__folio_batch_release
      0.53 ?264%      -0.5        0.00        perf-profile.children.cycles-pp.lru_add_drain_cpu
      0.53 ?264%      -0.5        0.00        perf-profile.children.cycles-pp.folio_batch_move_lru
      0.32 ?264%      -0.3        0.00        perf-profile.children.cycles-pp.shmem_evict_inode
      0.22 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.stress_memfd_child
      0.22 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      0.21 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
      0.20 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.asm_exc_page_fault
      0.18 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.__mmap
      0.16 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.ksys_mmap_pgoff
      0.16 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.vm_mmap_pgoff
      0.15 ?264%      -0.2        0.00        perf-profile.children.cycles-pp.do_mmap
      0.15 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.exc_page_fault
      0.15 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.do_user_addr_fault
      0.14 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.handle_mm_fault
      0.13 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
      0.13 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.hrtimer_interrupt
      0.13 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__handle_mm_fault
      0.13 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.mmap_region
      0.13 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__mmap_region
      0.12 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.do_pte_missing
      0.12 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.do_shared_fault
      0.12 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.12 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.tick_nohz_handler
      0.11 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.update_process_times
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__do_fault
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.handle_softirqs
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.rcu_core
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.rcu_do_batch
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.shmem_fault
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.shmem_get_folio_gfp
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__munmap
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__mmap_new_vma
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__vm_munmap
      0.10 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__x64_sys_munmap
      0.09 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.do_vmi_munmap
      0.09 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.do_vmi_align_munmap
      0.08 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__irq_exit_rcu
      0.08 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.sched_tick
      0.07 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.kmem_cache_free
      0.06 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.shmem_alloc_and_add_folio
      0.06 ?264%      -0.1        0.00        perf-profile.children.cycles-pp.__x64_sys_close
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.llseek
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_mmap
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.touch_atime
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.unmap_page_range
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.vms_complete_munmap_vmas
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.zap_pmd_range
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.zap_pte_range
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.ftruncate64
      0.04 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.alloc_inode
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__slab_free
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.kmem_cache_alloc_lru_noprof
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_store_gfp
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_cblist_dequeue
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.alloc_file_pseudo
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.vms_clear_ptes
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__mem_cgroup_uncharge_folios
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.generic_update_time
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.update_rq_clock_task
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__mark_inode_dirty
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__x64_sys_ftruncate
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.do_sys_ftruncate
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.page_counter_uncharge
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.uncharge_batch
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.do_ftruncate
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.task_tick_fair
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.kthread
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.ret_from_fork
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.ret_from_fork_asm
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_add_to_page_cache
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.locked_inode_to_wb_and_lock_list
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.run_ksoftirqd
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.smpboot_thread_fn
      0.03 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.unmap_mapping_range
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.destroy_inode
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.unmap_vmas
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.update_curr
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__destroy_inode
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.do_truncate
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.truncate_inode_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.zap_page_range_single
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_alloc_nodes
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_store_prealloc
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_store
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__get_unmapped_area
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_get_unmapped_area
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.zap_page_range_single_batched
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__lruvec_stat_mod_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.filemap_remove_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.lru_add
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.___slab_alloc
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.unmapped_area_topdown
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.vm_unmapped_area
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.clear_nlink
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.find_lock_entries
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.get_jiffies_update
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.inode_init_always_gfp
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.ksys_lseek
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.lru_gen_del_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mod_memcg_lruvec_state
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.notify_change
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.tmigr_requires_handle_remote
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__filemap_remove_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__memcg_slab_free_hook
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_preallocate
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_alloc_inode
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__call_rcu_common
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__kmem_cache_free_bulk
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__pcs_replace_full_main
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.flush_tlb_mm_range
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_wr_node_store
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.native_irq_return_iret
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.perf_event_mmap
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.perf_event_mmap_event
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_alloc_folio
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.sync_regs
      0.02 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.update_se
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.alloc_pages_mpol
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.folio_alloc_mpol_noprof
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.lru_gen_add_folio
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_empty_area_rev
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_file_llseek
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.vms_gather_munmap_vmas
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_create
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_find
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__cond_resched
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.clear_page_erms
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.flush_tlb_func
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_wr_spanning_store
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.allocate_slab
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.atime_needs_update
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.clockevents_program_event
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.file_init_path
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.finish_fault
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.inode_set_ctime_current
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.ktime_get
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_find
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_rev_awalk
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_split
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_wr_bnode
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shuffle_freelist
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__pcs_replace_empty_main
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.errseq_sample
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.get_page_from_freelist
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_spanning_rebalance
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.native_flush_tlb_one_user
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.setup_object
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_expand
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__d_alloc
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.d_alloc_pseudo
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.inode_init_owner
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.kmem_cache_alloc_bulk_noprof
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_walk
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.simple_inode_init_ts
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.vm_area_alloc
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_alloc
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_load
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__account_obj_stock
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__put_partials
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.__vfs_getxattr
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.cap_inode_need_killpriv
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.fault_dirty_shared_page
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.filemap_unaccount_folio
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mapping_seek_hole_data
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_prev_slot
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.obj_cgroup_charge_account
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_all_qs
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.security_inode_need_killpriv
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_inode_acct_blocks
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_recalc_inode
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.zap_present_ptes
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.alloc_empty_file
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.cmd_record
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.down_write
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.free_pgtables
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.inode_init_once
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.lock_vma_under_rcu
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.mas_topiary_replace
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.memcg_list_lru_alloc
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.perf_iterate_sb
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.refill_obj_stock
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rmqueue
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.rmqueue_pcplist
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.shmem_free_in_core_inode
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.tlb_flush_rmap_batch
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.tlb_flush_rmaps
      0.01 ?264%      -0.0        0.00        perf-profile.children.cycles-pp.xas_start
     11.28 ?264%     -11.3        0.00        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
      0.06 ?264%      -0.1        0.00        perf-profile.self.cycles-pp._raw_spin_lock
      0.04 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.stress_memfd_child
      0.03 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_cblist_dequeue
      0.03 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__slab_free
      0.03 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.update_rq_clock_task
      0.03 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.page_counter_uncharge
      0.03 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.shmem_get_folio_gfp
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__destroy_inode
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.clear_nlink
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.get_jiffies_update
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.lru_gen_del_folio
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.native_irq_return_iret
      0.02 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.sync_regs
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.do_syscall_64
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.mod_memcg_lruvec_state
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.clear_page_erms
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.kmem_cache_free
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.lru_gen_add_folio
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__kmem_cache_alloc_bulk
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.inode_init_always_gfp
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.ktime_get
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.mas_rev_awalk
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.notify_change
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.update_se
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.atime_needs_update
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.errseq_sample
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.native_flush_tlb_one_user
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__call_rcu_common
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.inode_init_owner
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.inode_set_ctime_current
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.xas_find
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.__memcg_slab_free_hook
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.folios_put_refs
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.inode_sb_list_add
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.llseek
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.mas_walk
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.find_lock_entries
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.inode_init_once
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.shmem_free_in_core_inode
      0.01 ?264%      -0.0        0.00        perf-profile.self.cycles-pp.update_curr



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki



From vbabka at suse.cz  Thu Sep 18 01:09:34 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Thu, 18 Sep 2025 10:09:34 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo> <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo> <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
Message-ID: <40461105-a344-444f-834b-9559b6644710@suse.cz>

On 9/17/25 16:14, Vlastimil Babka wrote:
> On 9/17/25 15:34, Harry Yoo wrote:
>> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
>>> On 9/17/25 15:07, Harry Yoo wrote:
>>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
>>> >> On 9/17/25 13:32, Harry Yoo wrote:
>>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
>>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>>> >> >> >> +				sfw->skip = true;
>>> >> >> >> +				continue;
>>> >> >> >> +			}
>>> >> >> >>
>>> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>>> >> >> >> +			sfw->skip = false;
>>> >> >> >> +			sfw->s = s;
>>> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>>> >> >> >> +			flushed = true;
>>> >> >> >> +		}
>>> >> >> >> +
>>> >> >> >> +		for_each_online_cpu(cpu) {
>>> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
>>> >> >> >> +			if (sfw->skip)
>>> >> >> >> +				continue;
>>> >> >> >> +			flush_work(&sfw->work);
>>> >> >> >> +		}
>>> >> >> >> +
>>> >> >> >> +		mutex_unlock(&flush_lock);
>>> >> >> >> +	}
>>> >> >> >> +
>>> >> >> >> +	mutex_unlock(&slab_mutex);
>>> >> >> >> +	cpus_read_unlock();
>>> >> >> >> +
>>> >> >> >> +	if (flushed)
>>> >> >> >> +		rcu_barrier();
>>> >> >> > 
>>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
>>> >> >> > 
>>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
>>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>>> >> >> > so flushed == false but the rcu callback isn't processed yet
>>> >> >> > by the end of the function?
>>> >> >> > 
>>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
>>> >> >> > but still possible...
>>> >> >> 
>>> >> >> Yes also good point, will flush unconditionally.
>>> >> >> 
>>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>>> >> >> local_unlock().
>>> >> >>
>>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
>>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
>>> >> > 
>>> >> > Makes sense to me.
>>> > 
>>> > Wait, I'm confused.
>>> > 
>>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
>>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
>>> > the object X to be freed before kvfree_rcu_barrier() returns?
>>> 
>>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
>>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
>>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
>>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
>>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
>>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
>>> sheaf also contains the object X, we should make sure that is flushed.
>> 
>> I was going to say "but we queue and wait for the flushing work to
>> complete, so the sheaf containing object X should be flushed?"
>> 
>> But nah, that's true only if we see pcs->rcu_free != NULL in
>> flush_all_rcu_sheaves().
>> 
>> You are right...
>> 
>> Hmm, maybe it's simpler to fix this by never skipping queueing the work
>> even when pcs->rcu_sheaf == NULL?
> 
> I guess it's simpler, yeah.

So what about this? The unconditional queueing should cover all races with
__kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.

>From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka at suse.cz>
Date: Wed, 3 Sep 2025 14:59:46 +0200
Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations

Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
For caches with sheaves, on each cpu maintain a rcu_free sheaf in
addition to main and spare sheaves.

kfree_rcu() operations will try to put objects on this sheaf. Once full,
the sheaf is detached and submitted to call_rcu() with a handler that
will try to put it in the barn, or flush to slab pages using bulk free,
when the barn is full. Then a new empty sheaf must be obtained to put
more objects there.

It's possible that no free sheaves are available to use for a new
rcu_free sheaf, and the allocation in kfree_rcu() context can only use
GFP_NOWAIT and thus may fail. In that case, fall back to the existing
kfree_rcu() implementation.

Expected advantages:
- batching the kfree_rcu() operations, that could eventually replace the
  existing batching
- sheaves can be reused for allocations via barn instead of being
  flushed to slabs, which is more efficient
  - this includes cases where only some cpus are allowed to process rcu
    callbacks (Android)

Possible disadvantage:
- objects might be waiting for more than their grace period (it is
  determined by the last object freed into the sheaf), increasing memory
  usage - but the existing batching does that too.

Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
implementation favors smaller memory footprint over performance.

Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
contexts where kfree_rcu() is called might not be compatible with taking
a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
spinlock - the current kfree_rcu() implementation avoids doing that.

Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
that have them. This is not a cheap operation, but the barrier usage is
rare - currently kmem_cache_destroy() or on module unload.

Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
count how many kfree_rcu() used the rcu_free sheaf successfully and how
many had to fall back to the existing implementation.

Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
---
 mm/slab.h        |   3 +
 mm/slab_common.c |  26 +++++
 mm/slub.c        | 267 ++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 294 insertions(+), 2 deletions(-)

diff --git a/mm/slab.h b/mm/slab.h
index 206987ce44a4..e82e51c44bd0 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -435,6 +435,9 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
 	return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
 }
 
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
+void flush_all_rcu_sheaves(void);
+
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
 			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
diff --git a/mm/slab_common.c b/mm/slab_common.c
index e2b197e47866..005a4319c06a 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
 		kvfree_rcu_list(head);
 }
 
+static bool kfree_rcu_sheaf(void *obj)
+{
+	struct kmem_cache *s;
+	struct folio *folio;
+	struct slab *slab;
+
+	if (is_vmalloc_addr(obj))
+		return false;
+
+	folio = virt_to_folio(obj);
+	if (unlikely(!folio_test_slab(folio)))
+		return false;
+
+	slab = folio_slab(folio);
+	s = slab->slab_cache;
+	if (s->cpu_sheaves)
+		return __kfree_rcu_sheaf(s, obj);
+
+	return false;
+}
+
 static bool
 need_offload_krc(struct kfree_rcu_cpu *krcp)
 {
@@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
 	if (!head)
 		might_sleep();
 
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
+		return;
+
 	// Queue the object but don't yet schedule the batch.
 	if (debug_rcu_head_queue(ptr)) {
 		// Probable double kfree_rcu(), just leak.
@@ -2026,6 +2050,8 @@ void kvfree_rcu_barrier(void)
 	bool queued;
 	int i, cpu;
 
+	flush_all_rcu_sheaves();
+
 	/*
 	 * Firstly we detach objects and queue them over an RCU-batch
 	 * for all CPUs. Finally queued works are flushed for each CPU.
diff --git a/mm/slub.c b/mm/slub.c
index cba188b7e04d..171273f90efd 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -367,6 +367,8 @@ enum stat_item {
 	ALLOC_FASTPATH,		/* Allocation from cpu slab */
 	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
 	FREE_PCS,		/* Free to percpu sheaf */
+	FREE_RCU_SHEAF,		/* Free to rcu_free sheaf */
+	FREE_RCU_SHEAF_FAIL,	/* Failed to free to a rcu_free sheaf */
 	FREE_FASTPATH,		/* Free to cpu slab */
 	FREE_SLOWPATH,		/* Freeing not to cpu slab */
 	FREE_FROZEN,		/* Freeing to frozen slab */
@@ -461,6 +463,7 @@ struct slab_sheaf {
 		struct rcu_head rcu_head;
 		struct list_head barn_list;
 	};
+	struct kmem_cache *cache;
 	unsigned int size;
 	void *objects[];
 };
@@ -469,6 +472,7 @@ struct slub_percpu_sheaves {
 	local_trylock_t lock;
 	struct slab_sheaf *main; /* never NULL when unlocked */
 	struct slab_sheaf *spare; /* empty or full, may be NULL */
+	struct slab_sheaf *rcu_free; /* for batching kfree_rcu() */
 };
 
 /*
@@ -2531,6 +2535,8 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
 	if (unlikely(!sheaf))
 		return NULL;
 
+	sheaf->cache = s;
+
 	stat(s, SHEAF_ALLOC);
 
 	return sheaf;
@@ -2655,6 +2661,43 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	sheaf->size = 0;
 }
 
+static void __rcu_free_sheaf_prepare(struct kmem_cache *s,
+				     struct slab_sheaf *sheaf)
+{
+	bool init = slab_want_init_on_free(s);
+	void **p = &sheaf->objects[0];
+	unsigned int i = 0;
+
+	while (i < sheaf->size) {
+		struct slab *slab = virt_to_slab(p[i]);
+
+		memcg_slab_free_hook(s, slab, p + i, 1);
+		alloc_tagging_slab_free_hook(s, slab, p + i, 1);
+
+		if (unlikely(!slab_free_hook(s, p[i], init, true))) {
+			p[i] = p[--sheaf->size];
+			continue;
+		}
+
+		i++;
+	}
+}
+
+static void rcu_free_sheaf_nobarn(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+	s = sheaf->cache;
+
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	sheaf_flush_unused(s, sheaf);
+
+	free_empty_sheaf(s, sheaf);
+}
+
 /*
  * Caller needs to make sure migration is disabled in order to fully flush
  * single cpu's sheaves
@@ -2667,7 +2710,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
 static void pcs_flush_all(struct kmem_cache *s)
 {
 	struct slub_percpu_sheaves *pcs;
-	struct slab_sheaf *spare;
+	struct slab_sheaf *spare, *rcu_free;
 
 	local_lock(&s->cpu_sheaves->lock);
 	pcs = this_cpu_ptr(s->cpu_sheaves);
@@ -2675,6 +2718,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 	spare = pcs->spare;
 	pcs->spare = NULL;
 
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
 	local_unlock(&s->cpu_sheaves->lock);
 
 	if (spare) {
@@ -2682,6 +2728,9 @@ static void pcs_flush_all(struct kmem_cache *s)
 		free_empty_sheaf(s, spare);
 	}
 
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+
 	sheaf_flush_main(s);
 }
 
@@ -2698,6 +2747,11 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
 		free_empty_sheaf(s, pcs->spare);
 		pcs->spare = NULL;
 	}
+
+	if (pcs->rcu_free) {
+		call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+		pcs->rcu_free = NULL;
+	}
 }
 
 static void pcs_destroy(struct kmem_cache *s)
@@ -2723,6 +2777,7 @@ static void pcs_destroy(struct kmem_cache *s)
 		 */
 
 		WARN_ON(pcs->spare);
+		WARN_ON(pcs->rcu_free);
 
 		if (!WARN_ON(pcs->main->size)) {
 			free_empty_sheaf(s, pcs->main);
@@ -3780,7 +3835,7 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s)
 
 	pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
 
-	return (pcs->spare || pcs->main->size);
+	return (pcs->spare || pcs->rcu_free || pcs->main->size);
 }
 
 /*
@@ -3840,6 +3895,77 @@ static void flush_all(struct kmem_cache *s)
 	cpus_read_unlock();
 }
 
+static void flush_rcu_sheaf(struct work_struct *w)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_free;
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+
+	sfw = container_of(w, struct slub_flush_work, work);
+	s = sfw->s;
+
+	local_lock(&s->cpu_sheaves->lock);
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	rcu_free = pcs->rcu_free;
+	pcs->rcu_free = NULL;
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	if (rcu_free)
+		call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
+}
+
+
+/* needed for kvfree_rcu_barrier() */
+void flush_all_rcu_sheaves(void)
+{
+	struct slub_flush_work *sfw;
+	struct kmem_cache *s;
+	unsigned int cpu;
+
+	cpus_read_lock();
+	mutex_lock(&slab_mutex);
+
+	list_for_each_entry(s, &slab_caches, list) {
+		if (!s->cpu_sheaves)
+			continue;
+
+		mutex_lock(&flush_lock);
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+
+			/*
+			 * we don't check if rcu_free sheaf exists - racing
+			 * __kfree_rcu_sheaf() might have just removed it.
+			 * by executing flush_rcu_sheaf() on the cpu we make
+			 * sure the __kfree_rcu_sheaf() finished its call_rcu()
+			 */
+
+			INIT_WORK(&sfw->work, flush_rcu_sheaf);
+			sfw->skip = false;
+			sfw->s = s;
+			queue_work_on(cpu, flushwq, &sfw->work);
+		}
+
+		for_each_online_cpu(cpu) {
+			sfw = &per_cpu(slub_flush, cpu);
+			if (sfw->skip)
+				continue;
+			flush_work(&sfw->work);
+		}
+
+		mutex_unlock(&flush_lock);
+	}
+
+	mutex_unlock(&slab_mutex);
+	cpus_read_unlock();
+
+	rcu_barrier();
+}
+
 /*
  * Use the cpu notifier to insure that the cpu slabs are flushed when
  * necessary.
@@ -5413,6 +5539,134 @@ bool free_to_pcs(struct kmem_cache *s, void *object)
 	return true;
 }
 
+static void rcu_free_sheaf(struct rcu_head *head)
+{
+	struct slab_sheaf *sheaf;
+	struct node_barn *barn;
+	struct kmem_cache *s;
+
+	sheaf = container_of(head, struct slab_sheaf, rcu_head);
+
+	s = sheaf->cache;
+
+	/*
+	 * This may remove some objects due to slab_free_hook() returning false,
+	 * so that the sheaf might no longer be completely full. But it's easier
+	 * to handle it as full (unless it became completely empty), as the code
+	 * handles it fine. The only downside is that sheaf will serve fewer
+	 * allocations when reused. It only happens due to debugging, which is a
+	 * performance hit anyway.
+	 */
+	__rcu_free_sheaf_prepare(s, sheaf);
+
+	barn = get_node(s, numa_mem_id())->barn;
+
+	/* due to slab_free_hook() */
+	if (unlikely(sheaf->size == 0))
+		goto empty;
+
+	/*
+	 * Checking nr_full/nr_empty outside lock avoids contention in case the
+	 * barn is at the respective limit. Due to the race we might go over the
+	 * limit but that should be rare and harmless.
+	 */
+
+	if (data_race(barn->nr_full) < MAX_FULL_SHEAVES) {
+		stat(s, BARN_PUT);
+		barn_put_full_sheaf(barn, sheaf);
+		return;
+	}
+
+	stat(s, BARN_PUT_FAIL);
+	sheaf_flush_unused(s, sheaf);
+
+empty:
+	if (data_race(barn->nr_empty) < MAX_EMPTY_SHEAVES) {
+		barn_put_empty_sheaf(barn, sheaf);
+		return;
+	}
+
+	free_empty_sheaf(s, sheaf);
+}
+
+bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
+{
+	struct slub_percpu_sheaves *pcs;
+	struct slab_sheaf *rcu_sheaf;
+
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto fail;
+
+	pcs = this_cpu_ptr(s->cpu_sheaves);
+
+	if (unlikely(!pcs->rcu_free)) {
+
+		struct slab_sheaf *empty;
+		struct node_barn *barn;
+
+		if (pcs->spare && pcs->spare->size == 0) {
+			pcs->rcu_free = pcs->spare;
+			pcs->spare = NULL;
+			goto do_free;
+		}
+
+		barn = get_barn(s);
+
+		empty = barn_get_empty_sheaf(barn);
+
+		if (empty) {
+			pcs->rcu_free = empty;
+			goto do_free;
+		}
+
+		local_unlock(&s->cpu_sheaves->lock);
+
+		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
+
+		if (!empty)
+			goto fail;
+
+		if (!local_trylock(&s->cpu_sheaves->lock)) {
+			barn_put_empty_sheaf(barn, empty);
+			goto fail;
+		}
+
+		pcs = this_cpu_ptr(s->cpu_sheaves);
+
+		if (unlikely(pcs->rcu_free))
+			barn_put_empty_sheaf(barn, empty);
+		else
+			pcs->rcu_free = empty;
+	}
+
+do_free:
+
+	rcu_sheaf = pcs->rcu_free;
+
+	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
+
+	if (likely(rcu_sheaf->size < s->sheaf_capacity))
+		rcu_sheaf = NULL;
+	else
+		pcs->rcu_free = NULL;
+
+	/*
+	 * we flush before local_unlock to make sure a racing
+	 * flush_all_rcu_sheaves() doesn't miss this sheaf
+	 */
+	if (rcu_sheaf)
+		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
+
+	local_unlock(&s->cpu_sheaves->lock);
+
+	stat(s, FREE_RCU_SHEAF);
+	return true;
+
+fail:
+	stat(s, FREE_RCU_SHEAF_FAIL);
+	return false;
+}
+
 /*
  * Bulk free objects to the percpu sheaves.
  * Unlike free_to_pcs() this includes the calls to all necessary hooks
@@ -6909,6 +7163,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
 	struct kmem_cache_node *n;
 
 	flush_all_cpus_locked(s);
+
+	/* we might have rcu sheaves in flight */
+	if (s->cpu_sheaves)
+		rcu_barrier();
+
 	/* Attempt to free all objects */
 	for_each_kmem_cache_node(s, node, n) {
 		if (n->barn)
@@ -8284,6 +8543,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
 STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
 STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
 STAT_ATTR(FREE_PCS, free_cpu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf);
+STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail);
 STAT_ATTR(FREE_FASTPATH, free_fastpath);
 STAT_ATTR(FREE_SLOWPATH, free_slowpath);
 STAT_ATTR(FREE_FROZEN, free_frozen);
@@ -8382,6 +8643,8 @@ static struct attribute *slab_attrs[] = {
 	&alloc_fastpath_attr.attr,
 	&alloc_slowpath_attr.attr,
 	&free_cpu_sheaf_attr.attr,
+	&free_rcu_sheaf_attr.attr,
+	&free_rcu_sheaf_fail_attr.attr,
 	&free_fastpath_attr.attr,
 	&free_slowpath_attr.attr,
 	&free_frozen_attr.attr,
-- 
2.51.0




From urezki at gmail.com  Thu Sep 18 04:50:12 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Thu, 18 Sep 2025 13:50:12 +0200
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <CAJuCfpE48n=QM0nY8yE3drqZU0wgC76=70EyftL1WZewEcykyA@mail.gmail.com>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
 <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
 <aMpE-oSjtlDU4TSl@pc636>
 <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>
 <CAJuCfpE48n=QM0nY8yE3drqZU0wgC76=70EyftL1WZewEcykyA@mail.gmail.com>
Message-ID: <aMvx9GKgIWtrkZ_R@milan>

On Wed, Sep 17, 2025 at 04:59:41PM -0700, Suren Baghdasaryan wrote:
> On Wed, Sep 17, 2025 at 9:14?AM Suren Baghdasaryan <surenb at google.com> wrote:
> >
> > On Tue, Sep 16, 2025 at 10:19?PM Uladzislau Rezki <urezki at gmail.com> wrote:
> > >
> > > On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> > > > On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > > > >
> > > > > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > > > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > > > > >>
> > > > > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > > > > >> >
> > > > > >> >Summary of the results:
> > > > >
> > > > > In any case, thanks a lot for the results!
> > > > >
> > > > > >> >- Significant change (meaning >10% difference
> > > > > >> >  between base and experiment) on will-it-scale
> > > > > >> >  tests in AMD.
> > > > > >> >
> > > > > >> >Summary of AMD will-it-scale test changes:
> > > > > >> >
> > > > > >> >Number of runs : 15
> > > > > >> >Direction      : + is good
> > > > > >>
> > > > > >> If STDDEV grows more than mean, there is more jitter,
> > > > > >> which is not "good".
> > > > > >
> > > > > > This is true.  On the other hand, the mean grew way more in absolute
> > > > > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> > > > >
> > > > > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > > > > there's always an improvement for this config. So jitter here means it's
> > > > > changing between better and more better :) and not between worse and (more)
> > > > > better.
> > > > >
> > > > > The annoying part of course is that for other configs it's consistently the
> > > > > opposite.
> > > >
> > > > Hi Vlastimil,
> > > > I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> > > > faulting them in then unmapping and timing only mmap and munmap calls.
> > > > This is not a realistic scenario but works well for A/B comparison.
> > > >
> > > > The numbers are below with sheaves showing a clear improvement:
> > > >
> > > > Baseline
> > > >             avg             stdev
> > > > mmap        2.621073        0.2525161631
> > > > munmap      2.292965        0.008831973052
> > > > total       4.914038        0.2572620923
> > > >
> > > > Sheaves
> > > >             avg            stdev           avg_diff        stdev_diff
> > > > mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> > > > munmap      2.042071       0.03603083448   -10.94%        307.96%
> > > > total       3.603291667    0.113209047     -26.67%        -55.99%
> > > >
> > > Could you run your test with dropping below patch?
> >
> > Sure, will try later today and report.
> 
> Sheaves with [04/23] patch reverted:
> 
>             avg             avg_diff
> mmap     2.143948        -18.20%
> munmap     2.343707        2.21%
> total     4.487655        -8.68%
> 
With offloading over sheaves the mmap/munmap is faster, i assume it is
because of same objects are reused from the sheaves after reclaim. Whereas we,
kvfree_rcu() just free them.
 
Thank you for your results.

--
Uladzislau Rezki


From Liam.Howlett at oracle.com  Thu Sep 18 08:29:14 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 18 Sep 2025 11:29:14 -0400
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <aMvx9GKgIWtrkZ_R@milan>
References: <20250723-slub-percpu-caches-v5-0-b792cd830f5d@suse.cz>
 <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
 <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
 <aMpE-oSjtlDU4TSl@pc636>
 <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>
 <CAJuCfpE48n=QM0nY8yE3drqZU0wgC76=70EyftL1WZewEcykyA@mail.gmail.com>
 <aMvx9GKgIWtrkZ_R@milan>
Message-ID: <vhp6cqq6gx5xgs2ueo7az4wobs2rddjsucjwlxckx3us3p3pey@s62u2ruxkjw7>

* Uladzislau Rezki <urezki at gmail.com> [250918 07:50]:
> On Wed, Sep 17, 2025 at 04:59:41PM -0700, Suren Baghdasaryan wrote:
> > On Wed, Sep 17, 2025 at 9:14?AM Suren Baghdasaryan <surenb at google.com> wrote:
> > >
> > > On Tue, Sep 16, 2025 at 10:19?PM Uladzislau Rezki <urezki at gmail.com> wrote:
> > > >
> > > > On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> > > > > On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > > > > >
> > > > > > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > > > > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > > > > > >>
> > > > > > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > > > > > >> >
> > > > > > >> >Summary of the results:
> > > > > >
> > > > > > In any case, thanks a lot for the results!
> > > > > >
> > > > > > >> >- Significant change (meaning >10% difference
> > > > > > >> >  between base and experiment) on will-it-scale
> > > > > > >> >  tests in AMD.
> > > > > > >> >
> > > > > > >> >Summary of AMD will-it-scale test changes:
> > > > > > >> >
> > > > > > >> >Number of runs : 15
> > > > > > >> >Direction      : + is good
> > > > > > >>
> > > > > > >> If STDDEV grows more than mean, there is more jitter,
> > > > > > >> which is not "good".
> > > > > > >
> > > > > > > This is true.  On the other hand, the mean grew way more in absolute
> > > > > > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> > > > > >
> > > > > > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > > > > > there's always an improvement for this config. So jitter here means it's
> > > > > > changing between better and more better :) and not between worse and (more)
> > > > > > better.
> > > > > >
> > > > > > The annoying part of course is that for other configs it's consistently the
> > > > > > opposite.
> > > > >
> > > > > Hi Vlastimil,
> > > > > I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> > > > > faulting them in then unmapping and timing only mmap and munmap calls.
> > > > > This is not a realistic scenario but works well for A/B comparison.
> > > > >
> > > > > The numbers are below with sheaves showing a clear improvement:
> > > > >
> > > > > Baseline
> > > > >             avg             stdev
> > > > > mmap        2.621073        0.2525161631
> > > > > munmap      2.292965        0.008831973052
> > > > > total       4.914038        0.2572620923
> > > > >
> > > > > Sheaves
> > > > >             avg            stdev           avg_diff        stdev_diff
> > > > > mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> > > > > munmap      2.042071       0.03603083448   -10.94%        307.96%
> > > > > total       3.603291667    0.113209047     -26.67%        -55.99%
> > > > >
> > > > Could you run your test with dropping below patch?
> > >
> > > Sure, will try later today and report.
> > 
> > Sheaves with [04/23] patch reverted:
> > 
> >             avg             avg_diff
> > mmap     2.143948        -18.20%
> > munmap     2.343707        2.21%
> > total     4.487655        -8.68%
> > 
> With offloading over sheaves the mmap/munmap is faster, i assume it is
> because of same objects are reused from the sheaves after reclaim. Whereas we,
> kvfree_rcu() just free them.

Sorry, I am having trouble following where you think the speed up is
coming from.

Can you clarify what you mean by offloading and reclaim in this context?

Thanks,
Liam



From foxido at foxido.dev  Thu Sep 18 11:25:37 2025
From: foxido at foxido.dev (Gladyshev Ilya)
Date: Thu, 18 Sep 2025 21:25:37 +0300
Subject: [PATCH RESEND] maple: fix incorrect rewinding in mas_empty_area_rev
Message-ID: <20250918182544.1400748-1-foxido@foxido.dev>

Previously, mas_empty_area_rev was rewinding to the previous node if
some offset was cached. This could lead to incorrect results because a
useful gap could be skipped. However, this was never triggered in the
kernel because mm subsystem calls mas_empty_area_rev on non cached mas.

This patch unifies the rewind logic between mas_empty_area_rev and
mas_empty_area, so they both rewind in their correct directions.

Signed-off-by: Gladyshev Ilya <foxido at foxido.dev>
---
 lib/maple_tree.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a9..c7790fff4825 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -5138,15 +5138,15 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
 
 	if (mas_is_start(mas))
 		mas_start(mas);
-	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
-		return -EBUSY;
 
 	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
 		return mas_sparse_area(mas, min, max, size, false);
-	else if (mas->offset >= 2)
-		mas->offset -= 2;
-	else
+	else if (!mas->offset)
 		mas->offset = mas_data_end(mas);
+	else if (mas->offset <= mas_data_end(mas) - 2)
+		mas->offset = mas->offset + 2;
+	else if (!mas_skip_node(mas))
+		return -EBUSY;
 
 
 	/* The start of the window can only be within these values. */

base-commit: cbf658dd09419f1ef9de11b9604e950bdd5c170b
-- 
2.51.0



From Liam.Howlett at oracle.com  Thu Sep 18 13:20:32 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Thu, 18 Sep 2025 16:20:32 -0400
Subject: [PATCH RESEND] maple: fix incorrect rewinding in
 mas_empty_area_rev
In-Reply-To: <20250918182544.1400748-1-foxido@foxido.dev>
References: <20250918182544.1400748-1-foxido@foxido.dev>
Message-ID: <4a72mtf5n4mzirudx3z2z6b72ropqs7f4kp2dlezkib4k5em56@hiw63iieahph>

* Gladyshev Ilya <foxido at foxido.dev> [250918 14:26]:
> Previously, mas_empty_area_rev was rewinding to the previous node if
> some offset was cached. This could lead to incorrect results because a
> useful gap could be skipped. However, this was never triggered in the
> kernel because mm subsystem calls mas_empty_area_rev on non cached mas.

Can you please produce a test case, ideally in lib/test_maple_tree.c or
tools/testing/radix_tree/maple.c that triggers your issue?  I add all
new tests to one of these places so the error does not return.

You can build maple in tools/testing/radix_tree/ and run it to run the
tests.

It also helps understand the issue.

> 
> This patch unifies the rewind logic between mas_empty_area_rev and
> mas_empty_area, so they both rewind in their correct directions.

How are these unified?  They are still different functions..?  What is
the correct direction, in your opinion?

> 
> Signed-off-by: Gladyshev Ilya <foxido at foxido.dev>
> ---
>  lib/maple_tree.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index b4ee2d29d7a9..c7790fff4825 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -5138,15 +5138,15 @@ int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
>  
>  	if (mas_is_start(mas))
>  		mas_start(mas);
> -	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
> -		return -EBUSY;
>  
>  	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
>  		return mas_sparse_area(mas, min, max, size, false);
> -	else if (mas->offset >= 2)
> -		mas->offset -= 2;
> -	else

This does not make sense to me by inspection.
> +	else if (!mas->offset)
>  		mas->offset = mas_data_end(mas);

If we are at the start of the node, go to the end of the node.  If we
are continuing the search, why are we restarting the same node?

> +	else if (mas->offset <= mas_data_end(mas) - 2)
> +		mas->offset = mas->offset + 2;

If we are at the offset less than or equal 2 of the end of the node, we
advance two slots.  This is supposed to be going down but you are going
up?

> +	else if (!mas_skip_node(mas))
> +		return -EBUSY;

Go to the next node, which would be larger addresses or return -EBUSY?

>  
>  
>  	/* The start of the window can only be within these values. */
> 
> base-commit: cbf658dd09419f1ef9de11b9604e950bdd5c170b

This function is to find a gap from top (highest) down (to lower)
addresses.  On entry, it needs to decrement the offset or go to the
previous node.  You are resetting to start from the end of the node or
incrementing two slots.  Failing that (which will only happen when we
are within two of the end of a node), you will go to the NEXT node
which seems the wrong direction?

Can you produce a test case that shows that we are skipping a gap?

Thanks,
Liam


From harry.yoo at oracle.com  Thu Sep 18 23:47:34 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Fri, 19 Sep 2025 15:47:34 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <40461105-a344-444f-834b-9559b6644710@suse.cz>
References: <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo>
 <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz>
Message-ID: <aMz8hk3lrWiPQYnw@harry>

On Thu, Sep 18, 2025 at 10:09:34AM +0200, Vlastimil Babka wrote:
> On 9/17/25 16:14, Vlastimil Babka wrote:
> > On 9/17/25 15:34, Harry Yoo wrote:
> >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> >>> On 9/17/25 15:07, Harry Yoo wrote:
> >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> >>> >> On 9/17/25 13:32, Harry Yoo wrote:
> >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >>> >> >> >> +				sfw->skip = true;
> >>> >> >> >> +				continue;
> >>> >> >> >> +			}
> >>> >> >> >>
> >>> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >>> >> >> >> +			sfw->skip = false;
> >>> >> >> >> +			sfw->s = s;
> >>> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >>> >> >> >> +			flushed = true;
> >>> >> >> >> +		}
> >>> >> >> >> +
> >>> >> >> >> +		for_each_online_cpu(cpu) {
> >>> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
> >>> >> >> >> +			if (sfw->skip)
> >>> >> >> >> +				continue;
> >>> >> >> >> +			flush_work(&sfw->work);
> >>> >> >> >> +		}
> >>> >> >> >> +
> >>> >> >> >> +		mutex_unlock(&flush_lock);
> >>> >> >> >> +	}
> >>> >> >> >> +
> >>> >> >> >> +	mutex_unlock(&slab_mutex);
> >>> >> >> >> +	cpus_read_unlock();
> >>> >> >> >> +
> >>> >> >> >> +	if (flushed)
> >>> >> >> >> +		rcu_barrier();
> >>> >> >> > 
> >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> >>> >> >> > 
> >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> >>> >> >> > so flushed == false but the rcu callback isn't processed yet
> >>> >> >> > by the end of the function?
> >>> >> >> > 
> >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> >>> >> >> > but still possible...
> >>> >> >> 
> >>> >> >> Yes also good point, will flush unconditionally.
> >>> >> >> 
> >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> >>> >> >> local_unlock().
> >>> >> >>
> >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> >>> >> > 
> >>> >> > Makes sense to me.
> >>> > 
> >>> > Wait, I'm confused.
> >>> > 
> >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> >>> > the object X to be freed before kvfree_rcu_barrier() returns?
> >>> 
> >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> >>> sheaf also contains the object X, we should make sure that is flushed.
> >> 
> >> I was going to say "but we queue and wait for the flushing work to
> >> complete, so the sheaf containing object X should be flushed?"
> >> 
> >> But nah, that's true only if we see pcs->rcu_free != NULL in
> >> flush_all_rcu_sheaves().
> >> 
> >> You are right...
> >> 
> >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> >> even when pcs->rcu_sheaf == NULL?
> > 
> > I guess it's simpler, yeah.
> 
> So what about this? The unconditional queueing should cover all races with
> __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
> 
> From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
> From: Vlastimil Babka <vbabka at suse.cz>
> Date: Wed, 3 Sep 2025 14:59:46 +0200
> Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
> 
> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> addition to main and spare sheaves.
> 
> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> the sheaf is detached and submitted to call_rcu() with a handler that
> will try to put it in the barn, or flush to slab pages using bulk free,
> when the barn is full. Then a new empty sheaf must be obtained to put
> more objects there.
> 
> It's possible that no free sheaves are available to use for a new
> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> kfree_rcu() implementation.
> 
> Expected advantages:
> - batching the kfree_rcu() operations, that could eventually replace the
>   existing batching
> - sheaves can be reused for allocations via barn instead of being
>   flushed to slabs, which is more efficient
>   - this includes cases where only some cpus are allowed to process rcu
>     callbacks (Android)
> 
> Possible disadvantage:
> - objects might be waiting for more than their grace period (it is
>   determined by the last object freed into the sheaf), increasing memory
>   usage - but the existing batching does that too.
> 
> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> implementation favors smaller memory footprint over performance.
> 
> Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> contexts where kfree_rcu() is called might not be compatible with taking
> a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> spinlock - the current kfree_rcu() implementation avoids doing that.
> 
> Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> that have them. This is not a cheap operation, but the barrier usage is
> rare - currently kmem_cache_destroy() or on module unload.
> 
> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> many had to fall back to the existing implementation.
> 
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---

Looks good to me,
Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

with a nit:

> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
> +{
> +	struct slub_percpu_sheaves *pcs;
> +	struct slab_sheaf *rcu_sheaf;
> +
> +	if (!local_trylock(&s->cpu_sheaves->lock))
> +		goto fail;
> +
> +	pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +	if (unlikely(!pcs->rcu_free)) {
> +
> +		struct slab_sheaf *empty;
> +		struct node_barn *barn;
> +
> +		if (pcs->spare && pcs->spare->size == 0) {
> +			pcs->rcu_free = pcs->spare;
> +			pcs->spare = NULL;
> +			goto do_free;
> +		}
> +
> +		barn = get_barn(s);
> +
> +		empty = barn_get_empty_sheaf(barn);
> +
> +		if (empty) {
> +			pcs->rcu_free = empty;
> +			goto do_free;
> +		}
> +
> +		local_unlock(&s->cpu_sheaves->lock);
> +
> +		empty = alloc_empty_sheaf(s, GFP_NOWAIT);
> +
> +		if (!empty)
> +			goto fail;
> +
> +		if (!local_trylock(&s->cpu_sheaves->lock)) {
> +			barn_put_empty_sheaf(barn, empty);
> +			goto fail;
> +		}
> +
> +		pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +		if (unlikely(pcs->rcu_free))
> +			barn_put_empty_sheaf(barn, empty);
> +		else
> +			pcs->rcu_free = empty;
> +	}
> +
> +do_free:
> +
> +	rcu_sheaf = pcs->rcu_free;
> +
> +	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
> +
> +	if (likely(rcu_sheaf->size < s->sheaf_capacity))
> +		rcu_sheaf = NULL;
> +	else
> +		pcs->rcu_free = NULL;
> +
> +	/*
> +	 * we flush before local_unlock to make sure a racing
> +	 * flush_all_rcu_sheaves() doesn't miss this sheaf
> +	 */
> +	if (rcu_sheaf)
> +		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);

nit: now we don't have to put this inside local_lock()~local_unlock()?

-- 
Cheers,
Harry / Hyeonggon

> +	local_unlock(&s->cpu_sheaves->lock);
> +
> +	stat(s, FREE_RCU_SHEAF);
> +	return true;
> +
> +fail:
> +	stat(s, FREE_RCU_SHEAF_FAIL);
> +	return false;
> +}


From vbabka at suse.cz  Fri Sep 19 00:02:22 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Fri, 19 Sep 2025 09:02:22 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <aMz8hk3lrWiPQYnw@harry>
References: <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo> <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo> <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz> <aMz8hk3lrWiPQYnw@harry>
Message-ID: <a4ca0cf5-b7fa-48cb-abfb-1e1d016c30d0@suse.cz>

On 9/19/25 08:47, Harry Yoo wrote:
> On Thu, Sep 18, 2025 at 10:09:34AM +0200, Vlastimil Babka wrote:
>> On 9/17/25 16:14, Vlastimil Babka wrote:
>> > On 9/17/25 15:34, Harry Yoo wrote:
>> >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
>> >>> On 9/17/25 15:07, Harry Yoo wrote:
>> >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
>> >>> >> On 9/17/25 13:32, Harry Yoo wrote:
>> >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
>> >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
>> >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
>> >>> >> >> >> +				sfw->skip = true;
>> >>> >> >> >> +				continue;
>> >>> >> >> >> +			}
>> >>> >> >> >>
>> >>> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
>> >>> >> >> >> +			sfw->skip = false;
>> >>> >> >> >> +			sfw->s = s;
>> >>> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
>> >>> >> >> >> +			flushed = true;
>> >>> >> >> >> +		}
>> >>> >> >> >> +
>> >>> >> >> >> +		for_each_online_cpu(cpu) {
>> >>> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
>> >>> >> >> >> +			if (sfw->skip)
>> >>> >> >> >> +				continue;
>> >>> >> >> >> +			flush_work(&sfw->work);
>> >>> >> >> >> +		}
>> >>> >> >> >> +
>> >>> >> >> >> +		mutex_unlock(&flush_lock);
>> >>> >> >> >> +	}
>> >>> >> >> >> +
>> >>> >> >> >> +	mutex_unlock(&slab_mutex);
>> >>> >> >> >> +	cpus_read_unlock();
>> >>> >> >> >> +
>> >>> >> >> >> +	if (flushed)
>> >>> >> >> >> +		rcu_barrier();
>> >>> >> >> > 
>> >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
>> >>> >> >> > 
>> >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
>> >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
>> >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
>> >>> >> >> > so flushed == false but the rcu callback isn't processed yet
>> >>> >> >> > by the end of the function?
>> >>> >> >> > 
>> >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
>> >>> >> >> > but still possible...
>> >>> >> >> 
>> >>> >> >> Yes also good point, will flush unconditionally.
>> >>> >> >> 
>> >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
>> >>> >> >> local_unlock().
>> >>> >> >>
>> >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
>> >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
>> >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
>> >>> >> > 
>> >>> >> > Makes sense to me.
>> >>> > 
>> >>> > Wait, I'm confused.
>> >>> > 
>> >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
>> >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
>> >>> > the object X to be freed before kvfree_rcu_barrier() returns?
>> >>> 
>> >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
>> >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
>> >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
>> >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
>> >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
>> >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
>> >>> sheaf also contains the object X, we should make sure that is flushed.
>> >> 
>> >> I was going to say "but we queue and wait for the flushing work to
>> >> complete, so the sheaf containing object X should be flushed?"
>> >> 
>> >> But nah, that's true only if we see pcs->rcu_free != NULL in
>> >> flush_all_rcu_sheaves().
>> >> 
>> >> You are right...
>> >> 
>> >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
>> >> even when pcs->rcu_sheaf == NULL?
>> > 
>> > I guess it's simpler, yeah.
>> 
>> So what about this? The unconditional queueing should cover all races with
>> __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
>> 
>> From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
>> From: Vlastimil Babka <vbabka at suse.cz>
>> Date: Wed, 3 Sep 2025 14:59:46 +0200
>> Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
>> 
>> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
>> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
>> addition to main and spare sheaves.
>> 
>> kfree_rcu() operations will try to put objects on this sheaf. Once full,
>> the sheaf is detached and submitted to call_rcu() with a handler that
>> will try to put it in the barn, or flush to slab pages using bulk free,
>> when the barn is full. Then a new empty sheaf must be obtained to put
>> more objects there.
>> 
>> It's possible that no free sheaves are available to use for a new
>> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
>> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
>> kfree_rcu() implementation.
>> 
>> Expected advantages:
>> - batching the kfree_rcu() operations, that could eventually replace the
>>   existing batching
>> - sheaves can be reused for allocations via barn instead of being
>>   flushed to slabs, which is more efficient
>>   - this includes cases where only some cpus are allowed to process rcu
>>     callbacks (Android)
>> 
>> Possible disadvantage:
>> - objects might be waiting for more than their grace period (it is
>>   determined by the last object freed into the sheaf), increasing memory
>>   usage - but the existing batching does that too.
>> 
>> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
>> implementation favors smaller memory footprint over performance.
>> 
>> Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
>> contexts where kfree_rcu() is called might not be compatible with taking
>> a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
>> spinlock - the current kfree_rcu() implementation avoids doing that.
>> 
>> Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
>> that have them. This is not a cheap operation, but the barrier usage is
>> rare - currently kmem_cache_destroy() or on module unload.
>> 
>> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
>> count how many kfree_rcu() used the rcu_free sheaf successfully and how
>> many had to fall back to the existing implementation.
>> 
>> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
>> ---
> 
> Looks good to me,
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

Thanks.

>> +do_free:
>> +
>> +	rcu_sheaf = pcs->rcu_free;
>> +
>> +	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
>> +
>> +	if (likely(rcu_sheaf->size < s->sheaf_capacity))
>> +		rcu_sheaf = NULL;
>> +	else
>> +		pcs->rcu_free = NULL;
>> +
>> +	/*
>> +	 * we flush before local_unlock to make sure a racing
>> +	 * flush_all_rcu_sheaves() doesn't miss this sheaf
>> +	 */
>> +	if (rcu_sheaf)
>> +		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> 
> nit: now we don't have to put this inside local_lock()~local_unlock()?

I think we still need to? AFAICS I wrote before is still true:

The caller of kvfree_rcu(X, rhs) might have returned without filling up
the rcu_sheaf fully and thus without submitting it to call_rcu(), then
migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
up the rcu_sheaf fully and is about to call_rcu() on it.

If it can local_unlock() before doing the call_rcu(), it can local_unlock(),
get preempted, and our flush worqueue handler will only see there's no
rcu_free sheaf and do nothing.

If if must call_rcu() before local_unlock(), our flush workqueue handler
will not execute on the cpu until it performs the call_rcu() and
local_unlock(), because it can't preempt that section (!RT) or will have to
wait doing local_lock() in flush_rcu_sheaf() (RT) - here it's important it
takes the lock unconditionally.

Or am I missing something?


From harry.yoo at oracle.com  Fri Sep 19 01:59:54 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Fri, 19 Sep 2025 17:59:54 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <a4ca0cf5-b7fa-48cb-abfb-1e1d016c30d0@suse.cz>
References: <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo>
 <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz>
 <aMz8hk3lrWiPQYnw@harry>
 <a4ca0cf5-b7fa-48cb-abfb-1e1d016c30d0@suse.cz>
Message-ID: <aM0biub0a-px9Ldt@harry>

On Fri, Sep 19, 2025 at 09:02:22AM +0200, Vlastimil Babka wrote:
> On 9/19/25 08:47, Harry Yoo wrote:
> > On Thu, Sep 18, 2025 at 10:09:34AM +0200, Vlastimil Babka wrote:
> >> On 9/17/25 16:14, Vlastimil Babka wrote:
> >> > On 9/17/25 15:34, Harry Yoo wrote:
> >> >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> >> >>> On 9/17/25 15:07, Harry Yoo wrote:
> >> >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> >> >>> >> On 9/17/25 13:32, Harry Yoo wrote:
> >> >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> >> >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> >> >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >> >>> >> >> >> +				sfw->skip = true;
> >> >>> >> >> >> +				continue;
> >> >>> >> >> >> +			}
> >> >>> >> >> >>
> >> >>> >> >> >> +			INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >> >>> >> >> >> +			sfw->skip = false;
> >> >>> >> >> >> +			sfw->s = s;
> >> >>> >> >> >> +			queue_work_on(cpu, flushwq, &sfw->work);
> >> >>> >> >> >> +			flushed = true;
> >> >>> >> >> >> +		}
> >> >>> >> >> >> +
> >> >>> >> >> >> +		for_each_online_cpu(cpu) {
> >> >>> >> >> >> +			sfw = &per_cpu(slub_flush, cpu);
> >> >>> >> >> >> +			if (sfw->skip)
> >> >>> >> >> >> +				continue;
> >> >>> >> >> >> +			flush_work(&sfw->work);
> >> >>> >> >> >> +		}
> >> >>> >> >> >> +
> >> >>> >> >> >> +		mutex_unlock(&flush_lock);
> >> >>> >> >> >> +	}
> >> >>> >> >> >> +
> >> >>> >> >> >> +	mutex_unlock(&slab_mutex);
> >> >>> >> >> >> +	cpus_read_unlock();
> >> >>> >> >> >> +
> >> >>> >> >> >> +	if (flushed)
> >> >>> >> >> >> +		rcu_barrier();
> >> >>> >> >> > 
> >> >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> >> >>> >> >> > 
> >> >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> >> >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> >> >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> >> >>> >> >> > so flushed == false but the rcu callback isn't processed yet
> >> >>> >> >> > by the end of the function?
> >> >>> >> >> > 
> >> >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> >> >>> >> >> > but still possible...
> >> >>> >> >> 
> >> >>> >> >> Yes also good point, will flush unconditionally.
> >> >>> >> >> 
> >> >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> >> >>> >> >> local_unlock().
> >> >>> >> >>
> >> >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> >> >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> >> >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> >> >>> >> > 
> >> >>> >> > Makes sense to me.
> >> >>> > 
> >> >>> > Wait, I'm confused.
> >> >>> > 
> >> >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> >> >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> >> >>> > the object X to be freed before kvfree_rcu_barrier() returns?
> >> >>> 
> >> >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> >> >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> >> >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> >> >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> >> >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> >> >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> >> >>> sheaf also contains the object X, we should make sure that is flushed.
> >> >> 
> >> >> I was going to say "but we queue and wait for the flushing work to
> >> >> complete, so the sheaf containing object X should be flushed?"
> >> >> 
> >> >> But nah, that's true only if we see pcs->rcu_free != NULL in
> >> >> flush_all_rcu_sheaves().
> >> >> 
> >> >> You are right...
> >> >> 
> >> >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> >> >> even when pcs->rcu_sheaf == NULL?
> >> > 
> >> > I guess it's simpler, yeah.
> >> 
> >> So what about this? The unconditional queueing should cover all races with
> >> __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
> >> 
> >> From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
> >> From: Vlastimil Babka <vbabka at suse.cz>
> >> Date: Wed, 3 Sep 2025 14:59:46 +0200
> >> Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
> >> 
> >> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> >> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> >> addition to main and spare sheaves.
> >> 
> >> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> >> the sheaf is detached and submitted to call_rcu() with a handler that
> >> will try to put it in the barn, or flush to slab pages using bulk free,
> >> when the barn is full. Then a new empty sheaf must be obtained to put
> >> more objects there.
> >> 
> >> It's possible that no free sheaves are available to use for a new
> >> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> >> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> >> kfree_rcu() implementation.
> >> 
> >> Expected advantages:
> >> - batching the kfree_rcu() operations, that could eventually replace the
> >>   existing batching
> >> - sheaves can be reused for allocations via barn instead of being
> >>   flushed to slabs, which is more efficient
> >>   - this includes cases where only some cpus are allowed to process rcu
> >>     callbacks (Android)
> >> 
> >> Possible disadvantage:
> >> - objects might be waiting for more than their grace period (it is
> >>   determined by the last object freed into the sheaf), increasing memory
> >>   usage - but the existing batching does that too.
> >> 
> >> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> >> implementation favors smaller memory footprint over performance.
> >> 
> >> Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> >> contexts where kfree_rcu() is called might not be compatible with taking
> >> a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> >> spinlock - the current kfree_rcu() implementation avoids doing that.
> >> 
> >> Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> >> that have them. This is not a cheap operation, but the barrier usage is
> >> rare - currently kmem_cache_destroy() or on module unload.
> >> 
> >> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> >> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> >> many had to fall back to the existing implementation.
> >> 
> >> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> >> ---
> > 
> > Looks good to me,
> > Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> 
> Thanks.
> 
> >> +do_free:
> >> +
> >> +	rcu_sheaf = pcs->rcu_free;
> >> +
> >> +	rcu_sheaf->objects[rcu_sheaf->size++] = obj;
> >> +
> >> +	if (likely(rcu_sheaf->size < s->sheaf_capacity))
> >> +		rcu_sheaf = NULL;
> >> +	else
> >> +		pcs->rcu_free = NULL;
> >> +
> >> +	/*
> >> +	 * we flush before local_unlock to make sure a racing
> >> +	 * flush_all_rcu_sheaves() doesn't miss this sheaf
> >> +	 */
> >> +	if (rcu_sheaf)
> >> +		call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> > 
> > nit: now we don't have to put this inside local_lock()~local_unlock()?
> 
> I think we still need to? AFAICS I wrote before is still true:
> 
> The caller of kvfree_rcu(X, rhs) might have returned without filling up
> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> up the rcu_sheaf fully and is about to call_rcu() on it.
>
> If it can local_unlock() before doing the call_rcu(), it can local_unlock(),
> get preempted, and our flush worqueue handler will only see there's no
> rcu_free sheaf and do nothing.

Oops, you're right. So even if a previous kvfree_rcu() has returned
and then kvfree_rcu_barrier() is called, a later kvfree_rcu() call can
make the sheaf invisible to the flush workqueue handler if it calls
call_rcu() outside the critical section because it can be preempted by
the workqueue handler after local_unlock() but before calling
call_rcu().

> If if must call_rcu() before local_unlock(), our flush workqueue handler
> will not execute on the cpu until it performs the call_rcu() and
> local_unlock(), because it can't preempt that section (!RT) or will have to
> wait doing local_lock() in flush_rcu_sheaf() (RT) - here it's important it
> takes the lock unconditionally.

Right.

My nit was wrong and it looks good to me then!

-- 
Cheers,
Harry / Hyeonggon


From urezki at gmail.com  Fri Sep 19 08:07:01 2025
From: urezki at gmail.com (Uladzislau Rezki)
Date: Fri, 19 Sep 2025 17:07:01 +0200
Subject: Benchmarking [PATCH v5 00/14] SLUB percpu sheaves
In-Reply-To: <vhp6cqq6gx5xgs2ueo7az4wobs2rddjsucjwlxckx3us3p3pey@s62u2ruxkjw7>
References: <20250913000935.1021068-1-sudarsanm@google.com>
 <qs3967pq-4nq7-67pq-2025-r7259o0s52p4@vanv.qr>
 <f5792407-d2b9-42b3-bc85-ed14eac945ec@paulmck-laptop>
 <d1ef1cbb-c18d-4da6-b56b-342e86dca525@suse.cz>
 <CAJuCfpEQ=RUgcAvRzE5jRrhhFpkm8E2PpBK9e9GhK26ZaJQt=Q@mail.gmail.com>
 <aMpE-oSjtlDU4TSl@pc636>
 <CAJuCfpHQ_JedSRHKKoYXyVzaFOm=dDWzgFZwqerfEC1fn35j0w@mail.gmail.com>
 <CAJuCfpE48n=QM0nY8yE3drqZU0wgC76=70EyftL1WZewEcykyA@mail.gmail.com>
 <aMvx9GKgIWtrkZ_R@milan>
 <vhp6cqq6gx5xgs2ueo7az4wobs2rddjsucjwlxckx3us3p3pey@s62u2ruxkjw7>
Message-ID: <aM1xlYbvp13rOs6r@milan>

On Thu, Sep 18, 2025 at 11:29:14AM -0400, Liam R. Howlett wrote:
> * Uladzislau Rezki <urezki at gmail.com> [250918 07:50]:
> > On Wed, Sep 17, 2025 at 04:59:41PM -0700, Suren Baghdasaryan wrote:
> > > On Wed, Sep 17, 2025 at 9:14?AM Suren Baghdasaryan <surenb at google.com> wrote:
> > > >
> > > > On Tue, Sep 16, 2025 at 10:19?PM Uladzislau Rezki <urezki at gmail.com> wrote:
> > > > >
> > > > > On Tue, Sep 16, 2025 at 10:09:18AM -0700, Suren Baghdasaryan wrote:
> > > > > > On Mon, Sep 15, 2025 at 8:22?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > > > > > >
> > > > > > > On 9/15/25 14:13, Paul E. McKenney wrote:
> > > > > > > > On Mon, Sep 15, 2025 at 09:51:25AM +0200, Jan Engelhardt wrote:
> > > > > > > >>
> > > > > > > >> On Saturday 2025-09-13 02:09, Sudarsan Mahendran wrote:
> > > > > > > >> >
> > > > > > > >> >Summary of the results:
> > > > > > >
> > > > > > > In any case, thanks a lot for the results!
> > > > > > >
> > > > > > > >> >- Significant change (meaning >10% difference
> > > > > > > >> >  between base and experiment) on will-it-scale
> > > > > > > >> >  tests in AMD.
> > > > > > > >> >
> > > > > > > >> >Summary of AMD will-it-scale test changes:
> > > > > > > >> >
> > > > > > > >> >Number of runs : 15
> > > > > > > >> >Direction      : + is good
> > > > > > > >>
> > > > > > > >> If STDDEV grows more than mean, there is more jitter,
> > > > > > > >> which is not "good".
> > > > > > > >
> > > > > > > > This is true.  On the other hand, the mean grew way more in absolute
> > > > > > > > terms than did STDDEV.  So might this be a reasonable tradeoff?
> > > > > > >
> > > > > > > Also I'd point out that MIN of TEST is better than MAX of BASE, which means
> > > > > > > there's always an improvement for this config. So jitter here means it's
> > > > > > > changing between better and more better :) and not between worse and (more)
> > > > > > > better.
> > > > > > >
> > > > > > > The annoying part of course is that for other configs it's consistently the
> > > > > > > opposite.
> > > > > >
> > > > > > Hi Vlastimil,
> > > > > > I ran my mmap stress test that runs 20000 cycles of mmapping 50 VMAs,
> > > > > > faulting them in then unmapping and timing only mmap and munmap calls.
> > > > > > This is not a realistic scenario but works well for A/B comparison.
> > > > > >
> > > > > > The numbers are below with sheaves showing a clear improvement:
> > > > > >
> > > > > > Baseline
> > > > > >             avg             stdev
> > > > > > mmap        2.621073        0.2525161631
> > > > > > munmap      2.292965        0.008831973052
> > > > > > total       4.914038        0.2572620923
> > > > > >
> > > > > > Sheaves
> > > > > >             avg            stdev           avg_diff        stdev_diff
> > > > > > mmap        1.561220667    0.07748897037   -40.44%        -69.31%
> > > > > > munmap      2.042071       0.03603083448   -10.94%        307.96%
> > > > > > total       3.603291667    0.113209047     -26.67%        -55.99%
> > > > > >
> > > > > Could you run your test with dropping below patch?
> > > >
> > > > Sure, will try later today and report.
> > > 
> > > Sheaves with [04/23] patch reverted:
> > > 
> > >             avg             avg_diff
> > > mmap     2.143948        -18.20%
> > > munmap     2.343707        2.21%
> > > total     4.487655        -8.68%
> > > 
> > With offloading over sheaves the mmap/munmap is faster, i assume it is
> > because of same objects are reused from the sheaves after reclaim. Whereas we,
> > kvfree_rcu() just free them.
> 
> Sorry, I am having trouble following where you think the speed up is
> coming from.
> 
> Can you clarify what you mean by offloading and reclaim in this context?
> 
[1] <Sheaves series>
             avg            stdev           avg_diff        stdev_diff
 mmap        1.561220667    0.07748897037   -40.44%        -69.31%
 munmap      2.042071       0.03603083448   -10.94%        307.96%
 total       3.603291667    0.113209047     -26.67%        -55.99%
[1] <Sheaves series>

[2] <Sheaves series but with [04/23] patch reverted>
             avg             avg_diff
 mmap     2.143948        -18.20%
 munmap     2.343707        2.21%
 total     4.487655        -8.68%
[2] <Sheaves series but with [04/23] patch reverted>

I meant those two data results. It is comparison of freeing over
or to "sheaves" and without it in the kvfree_rcu() path.

--
Uladzislau Rezki


From foxido at foxido.dev  Fri Sep 19 10:14:30 2025
From: foxido at foxido.dev (Gladyshev Ilya)
Date: Fri, 19 Sep 2025 20:14:30 +0300
Subject: [PATCH RESEND] maple: fix incorrect rewinding in
 mas_empty_area_rev
In-Reply-To: <4a72mtf5n4mzirudx3z2z6b72ropqs7f4kp2dlezkib4k5em56@hiw63iieahph>
References: <20250918182544.1400748-1-foxido@foxido.dev>
 <4a72mtf5n4mzirudx3z2z6b72ropqs7f4kp2dlezkib4k5em56@hiw63iieahph>
Message-ID: <76e0b252-ecbe-4d86-b210-79eb803e0fd7@foxido.dev>

On 9/18/25 23:20, Liam R. Howlett wrote:

Sorry, I kinda overrushed with this patch and it turned out it wasn't 
complete. I apologize for that

Below you can find new patch with regression tests. I'll ignore some of 
your comments (and some of them are fixed) for now, so we can find 
common ground about semantics of empty_area* functions and later, if 
it's really is a bug and not my misunderstanding, I will send v2 for review.

 > * Gladyshev Ilya <foxido at foxido.dev> [250918 14:26]:
 >> Previously, mas_empty_area_rev was rewinding to the previous node if
 >> some offset was cached. This could lead to incorrect results because a
 >> useful gap could be skipped. However, this was never triggered in the
 >> kernel because mm subsystem calls mas_empty_area_rev on non cached mas.
 >
 > Can you please produce a test case, ideally in lib/test_maple_tree.c or
 > tools/testing/radix_tree/maple.c that triggers your issue?  I add all
 > new tests to one of these places so the error does not return.
 >
 > You can build maple in tools/testing/radix_tree/ and run it to run the
 > tests.
 >
 > It also helps understand the issue.
 >
 >>
 >> This patch unifies the rewind logic between mas_empty_area_rev and
 >> mas_empty_area, so they both rewind in their correct directions.
 >
 > How are these unified?  They are still different functions..?  What is
 > the correct direction, in your opinion?

For mas_empty_area we should move towards first slot, for 
mas_empty_area_rev we should move towards last slot. In both cases 
empty_area function will rescan last saved child / gap and won't miss 
anything.
 From 17707e1117a4d4be23f257c3b911c0a36f55b116 Mon Sep 17 00:00:00 2001
From: Gladyshev Ilya <foxido at foxido.dev>
Date: Fri, 19 Sep 2025 20:00:26 +0300
Subject: [PATCH] maple: fix incorrect rewinding in empty_area functions

todo: full description
Signed-off-by: Gladyshev Ilya <foxido at foxido.dev>
---
  lib/maple_tree.c                 | 40 ++++++++++++++--------
  tools/testing/radix-tree/maple.c | 57 ++++++++++++++++++++++++++++++++
  2 files changed, 84 insertions(+), 13 deletions(-)

diff --git a/lib/maple_tree.c b/lib/maple_tree.c
index b4ee2d29d7a9..3cc1596e231b 100644
--- a/lib/maple_tree.c
+++ b/lib/maple_tree.c
@@ -4985,13 +4985,10 @@ static inline bool mas_rewind_node(struct 
ma_state *mas)
   */
  static inline bool mas_skip_node(struct ma_state *mas)
  {
-	if (mas_is_err(mas))
-		return false;
-
  	do {
  		if (mte_is_root(mas->node)) {
  			if (mas->offset >= mas_data_end(mas)) {
-				mas_set_err(mas, -EBUSY);
+				mas->offset = mas_data_end(mas);
  				return false;
  			}
  		} else {
@@ -5003,6 +5000,22 @@ static inline bool mas_skip_node(struct ma_state 
*mas)
  	return true;
  }

+/*
+ * mas_skip_node_err() - Wraps mas_skip_node and errors if there is no 
next node
+ */
+static inline bool mas_skip_node_err(struct ma_state *mas)
+{
+	if (mas_is_err(mas))
+		return false;
+
+	if (!mas_skip_node(mas)) {
+		mas_set_err(mas, -EBUSY);
+		return false;
+	}
+
+	return true;
+}
+
  /*
   * mas_awalk() - Allocation walk.  Search from low address to high, 
for a gap of
   * @size
@@ -5024,7 +5037,7 @@ static inline void mas_awalk(struct ma_state *mas, 
unsigned long size)
  	 */
  	while (!mas_is_err(mas) && !mas_anode_descend(mas, size)) {
  		if (last == mas->node)
-			mas_skip_node(mas);
+			mas_skip_node_err(mas);
  		else
  			last = mas->node;
  	}
@@ -5089,8 +5102,8 @@ int mas_empty_area(struct ma_state *mas, unsigned 
long min,
  		mas_start(mas);
  	else if (mas->offset >= 2)
  		mas->offset -= 2;
-	else if (!mas_skip_node(mas))
-		return -EBUSY;
+	else
+		mas_rewind_node(mas);

  	/* Empty set */
  	if (mas_is_none(mas) || mas_is_ptr(mas))
@@ -5128,7 +5141,7 @@ EXPORT_SYMBOL_GPL(mas_empty_area);
  int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
  		unsigned long max, unsigned long size)
  {
-	struct maple_enode *last = mas->node;
+	struct maple_enode *last;

  	if (min > max)
  		return -EINVAL;
@@ -5138,21 +5151,22 @@ int mas_empty_area_rev(struct ma_state *mas, 
unsigned long min,

  	if (mas_is_start(mas))
  		mas_start(mas);
-	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
-		return -EBUSY;

  	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
  		return mas_sparse_area(mas, min, max, size, false);
-	else if (mas->offset >= 2)
-		mas->offset -= 2;
-	else
+	else if (!mas->offset)
  		mas->offset = mas_data_end(mas);
+	else if (mas->offset <= mas_data_end(mas) - 2)
+		mas->offset = mas->offset + 2;
+	else
+		mas_skip_node(mas);


  	/* The start of the window can only be within these values. */
  	mas->index = min;
  	mas->last = max;

+	last = mas->node;
  	while (!mas_rev_awalk(mas, size, &min, &max)) {
  		if (last == mas->node) {
  			if (!mas_rewind_node(mas))
diff --git a/tools/testing/radix-tree/maple.c 
b/tools/testing/radix-tree/maple.c
index 172700fb7784..bb424f94de1b 100644
--- a/tools/testing/radix-tree/maple.c
+++ b/tools/testing/radix-tree/maple.c
@@ -36655,9 +36655,66 @@ static void test_spanning_store_regression(void)
  	__mt_destroy(&tree);
  }

+static void test_cached_empty_area_rev_regression(void)
+{
+	struct maple_tree tree = MTREE_INIT(&tree, MT_FLAGS_ALLOC_RANGE);
+	MA_STATE(mas, &tree, 0, 0);
+	void *const dataptr = (void *)0x13370;
+
+	for (int i = 1; i <= 10; ++i) {
+		unsigned long start = i * 10000;
+		unsigned long end = start + 1000;
+		mas_set_range(&mas, start, end);
+		mas_store_gfp(&mas, dataptr, GFP_KERNEL);
+	}
+
+	/* NOTE: Still looks unneccecarry for me, but without it cached offset 
in mas::offset is too incorrect.
+	 * Probably couldn't be allowed without disabling caching at all
+	*/
+	mas_reset(&mas);
+
+	for (int i = 1; i < 10; ++i) {
+		/* NOTE: Wasn't working here without mas_reset */
+		// mas_reset(&mas);
+		mas_empty_area_rev(&mas, 0, 200000, 10);
+		BUG_ON(mas.index != 199991);
+	}
+
+	/* Cleanup. */
+	__mt_destroy(&tree);
+}
+
+static void test_cached_empty_area_regression(void)
+{
+	struct maple_tree tree = MTREE_INIT(&tree, MT_FLAGS_ALLOC_RANGE);
+	MA_STATE(mas, &tree, 0, 0);
+	void *const dataptr = (void *)0x13370;
+
+	for (int i = 1; i <= 10; ++i) {
+		unsigned long start = i * 10000;
+		unsigned long end = start + 1000;
+		mas_set_range(&mas, start, end);
+		mas_store_gfp(&mas, dataptr, GFP_KERNEL);
+	}
+
+	mas_reset(&mas);
+
+	for (int i = 1; i < 10; ++i) {
+		/* NOTE: Wasn't working here without mas_reset */
+		// mas_reset(&mas);
+		int res = mas_empty_area(&mas, 0, 200000, 10);
+		BUG_ON(mas.index != 0);
+		BUG_ON(res);
+	}
+
+	/* Cleanup. */
+	__mt_destroy(&tree);
+}
  static void regression_tests(void)
  {
  	test_spanning_store_regression();
+	test_cached_empty_area_regression();
+	test_cached_empty_area_rev_regression();
  }

  void maple_tree_tests(void)

base-commit: cbf658dd09419f1ef9de11b9604e950bdd5c170b
-- 
2.51.0



From Liam.Howlett at oracle.com  Fri Sep 19 11:43:54 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Fri, 19 Sep 2025 14:43:54 -0400
Subject: [PATCH RESEND] maple: fix incorrect rewinding in
 mas_empty_area_rev
In-Reply-To: <76e0b252-ecbe-4d86-b210-79eb803e0fd7@foxido.dev>
References: <20250918182544.1400748-1-foxido@foxido.dev>
 <4a72mtf5n4mzirudx3z2z6b72ropqs7f4kp2dlezkib4k5em56@hiw63iieahph>
 <76e0b252-ecbe-4d86-b210-79eb803e0fd7@foxido.dev>
Message-ID: <gna74zdgw7ob5sa4jvcrgr35wbrbxibtxum7ysdkqdhetkiyiv@gzvdwxybeb25>

* Gladyshev Ilya <foxido at foxido.dev> [250919 13:14]:
> On 9/18/25 23:20, Liam R. Howlett wrote:
> 
> Sorry, I kinda overrushed with this patch and it turned out it wasn't
> complete. I apologize for that

Well, don't expect me to spend much time on this either, then.

Your testcases are wrong.  You are not using the interface correctly and
have broken the tree with this patch to pass an invalid test.

> 
> Below you can find new patch with regression tests. I'll ignore some of your
> comments (and some of them are fixed) for now, so we can find common ground
> about semantics of empty_area* functions and later, if it's really is a bug
> and not my misunderstanding, I will send v2 for review.
> 
> > * Gladyshev Ilya <foxido at foxido.dev> [250918 14:26]:
> >> Previously, mas_empty_area_rev was rewinding to the previous node if
> >> some offset was cached. This could lead to incorrect results because a
> >> useful gap could be skipped. However, this was never triggered in the
> >> kernel because mm subsystem calls mas_empty_area_rev on non cached mas.
> >
> > Can you please produce a test case, ideally in lib/test_maple_tree.c or
> > tools/testing/radix_tree/maple.c that triggers your issue?  I add all
> > new tests to one of these places so the error does not return.
> >
> > You can build maple in tools/testing/radix_tree/ and run it to run the
> > tests.
> >
> > It also helps understand the issue.
> >
> >>
> >> This patch unifies the rewind logic between mas_empty_area_rev and
> >> mas_empty_area, so they both rewind in their correct directions.
> >
> > How are these unified?  They are still different functions..?  What is
> > the correct direction, in your opinion?
> 
> For mas_empty_area we should move towards first slot, for mas_empty_area_rev
> we should move towards last slot. In both cases empty_area function will
> rescan last saved child / gap and won't miss anything.
> From 17707e1117a4d4be23f257c3b911c0a36f55b116 Mon Sep 17 00:00:00 2001
> From: Gladyshev Ilya <foxido at foxido.dev>
> Date: Fri, 19 Sep 2025 20:00:26 +0300
> Subject: [PATCH] maple: fix incorrect rewinding in empty_area functions
> 
> todo: full description
> Signed-off-by: Gladyshev Ilya <foxido at foxido.dev>
> ---
>  lib/maple_tree.c                 | 40 ++++++++++++++--------
>  tools/testing/radix-tree/maple.c | 57 ++++++++++++++++++++++++++++++++
>  2 files changed, 84 insertions(+), 13 deletions(-)
> 
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index b4ee2d29d7a9..3cc1596e231b 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -4985,13 +4985,10 @@ static inline bool mas_rewind_node(struct ma_state
> *mas)
>   */
>  static inline bool mas_skip_node(struct ma_state *mas)
>  {
> -	if (mas_is_err(mas))
> -		return false;
> -
>  	do {
>  		if (mte_is_root(mas->node)) {
>  			if (mas->offset >= mas_data_end(mas)) {
> -				mas_set_err(mas, -EBUSY);
> +				mas->offset = mas_data_end(mas);
>  				return false;
>  			}
>  		} else {
> @@ -5003,6 +5000,22 @@ static inline bool mas_skip_node(struct ma_state
> *mas)
>  	return true;
>  }
> 
> +/*
> + * mas_skip_node_err() - Wraps mas_skip_node and errors if there is no next
> node
> + */
> +static inline bool mas_skip_node_err(struct ma_state *mas)
> +{
> +	if (mas_is_err(mas))
> +		return false;
> +
> +	if (!mas_skip_node(mas)) {
> +		mas_set_err(mas, -EBUSY);
> +		return false;
> +	}
> +
> +	return true;
> +}
> +
>  /*
>   * mas_awalk() - Allocation walk.  Search from low address to high, for a
> gap of
>   * @size
> @@ -5024,7 +5037,7 @@ static inline void mas_awalk(struct ma_state *mas,
> unsigned long size)
>  	 */
>  	while (!mas_is_err(mas) && !mas_anode_descend(mas, size)) {
>  		if (last == mas->node)
> -			mas_skip_node(mas);
> +			mas_skip_node_err(mas);
>  		else
>  			last = mas->node;
>  	}
> @@ -5089,8 +5102,8 @@ int mas_empty_area(struct ma_state *mas, unsigned long
> min,
>  		mas_start(mas);
>  	else if (mas->offset >= 2)
>  		mas->offset -= 2;
> -	else if (!mas_skip_node(mas))
> -		return -EBUSY;
> +	else
> +		mas_rewind_node(mas);
> 
>  	/* Empty set */
>  	if (mas_is_none(mas) || mas_is_ptr(mas))
> @@ -5128,7 +5141,7 @@ EXPORT_SYMBOL_GPL(mas_empty_area);
>  int mas_empty_area_rev(struct ma_state *mas, unsigned long min,
>  		unsigned long max, unsigned long size)
>  {
> -	struct maple_enode *last = mas->node;
> +	struct maple_enode *last;
> 
>  	if (min > max)
>  		return -EINVAL;
> @@ -5138,21 +5151,22 @@ int mas_empty_area_rev(struct ma_state *mas,
> unsigned long min,
> 
>  	if (mas_is_start(mas))
>  		mas_start(mas);
> -	else if ((mas->offset < 2) && (!mas_rewind_node(mas)))
> -		return -EBUSY;
> 
>  	if (unlikely(mas_is_none(mas) || mas_is_ptr(mas)))
>  		return mas_sparse_area(mas, min, max, size, false);
> -	else if (mas->offset >= 2)
> -		mas->offset -= 2;
> -	else
> +	else if (!mas->offset)
>  		mas->offset = mas_data_end(mas);
> +	else if (mas->offset <= mas_data_end(mas) - 2)
> +		mas->offset = mas->offset + 2;
> +	else
> +		mas_skip_node(mas);
> 
> 
>  	/* The start of the window can only be within these values. */
>  	mas->index = min;
>  	mas->last = max;
> 
> +	last = mas->node;
>  	while (!mas_rev_awalk(mas, size, &min, &max)) {
>  		if (last == mas->node) {
>  			if (!mas_rewind_node(mas))
> diff --git a/tools/testing/radix-tree/maple.c
> b/tools/testing/radix-tree/maple.c
> index 172700fb7784..bb424f94de1b 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -36655,9 +36655,66 @@ static void test_spanning_store_regression(void)
>  	__mt_destroy(&tree);
>  }
> 
> +static void test_cached_empty_area_rev_regression(void)
> +{
> +	struct maple_tree tree = MTREE_INIT(&tree, MT_FLAGS_ALLOC_RANGE);
> +	MA_STATE(mas, &tree, 0, 0);
> +	void *const dataptr = (void *)0x13370;
> +
> +	for (int i = 1; i <= 10; ++i) {
> +		unsigned long start = i * 10000;
> +		unsigned long end = start + 1000;
> +		mas_set_range(&mas, start, end);
> +		mas_store_gfp(&mas, dataptr, GFP_KERNEL);
> +	}
> +
> +	/* NOTE: Still looks unneccecarry for me, but without it cached offset in
> mas::offset is too incorrect.
> +	 * Probably couldn't be allowed without disabling caching at all
> +	*/
> +	mas_reset(&mas);
> +
> +	for (int i = 1; i < 10; ++i) {
> +		/* NOTE: Wasn't working here without mas_reset */
> +		// mas_reset(&mas);
> +		mas_empty_area_rev(&mas, 0, 200000, 10);
> +		BUG_ON(mas.index != 199991);
> +	}
> +
> +	/* Cleanup. */
> +	__mt_destroy(&tree);
> +}
> +
> +static void test_cached_empty_area_regression(void)
> +{
> +	struct maple_tree tree = MTREE_INIT(&tree, MT_FLAGS_ALLOC_RANGE);
> +	MA_STATE(mas, &tree, 0, 0);
> +	void *const dataptr = (void *)0x13370;
> +
> +	for (int i = 1; i <= 10; ++i) {
> +		unsigned long start = i * 10000;
> +		unsigned long end = start + 1000;
> +		mas_set_range(&mas, start, end);
> +		mas_store_gfp(&mas, dataptr, GFP_KERNEL);
> +	}
> +
> +	mas_reset(&mas);
> +
> +	for (int i = 1; i < 10; ++i) {
> +		/* NOTE: Wasn't working here without mas_reset */
> +		// mas_reset(&mas);
> +		int res = mas_empty_area(&mas, 0, 200000, 10);
> +		BUG_ON(mas.index != 0);
> +		BUG_ON(res);
> +	}
> +
> +	/* Cleanup. */
> +	__mt_destroy(&tree);
> +}
>  static void regression_tests(void)
>  {
>  	test_spanning_store_regression();
> +	test_cached_empty_area_regression();
> +	test_cached_empty_area_rev_regression();
>  }
> 
>  void maple_tree_tests(void)
> 
> base-commit: cbf658dd09419f1ef9de11b9604e950bdd5c170b
> -- 
> 2.51.0
> 


From miguel.ojeda.sandonis at gmail.com  Sun Sep 21 02:27:52 2025
From: miguel.ojeda.sandonis at gmail.com (Miguel Ojeda)
Date: Sun, 21 Sep 2025 11:27:52 +0200
Subject: [PATCH] rust: maple_tree: fix intra-doc link
In-Reply-To: <20250910140212.997771-1-ojeda@kernel.org>
References: <20250910140212.997771-1-ojeda@kernel.org>
Message-ID: <CANiq72kB5hCC9A9bXEx489E+SMMFcFMiJB68NHA7megz2aG5HA@mail.gmail.com>

On Wed, Sep 10, 2025 at 4:02?PM Miguel Ojeda <ojeda at kernel.org> wrote:
>
> Andrew: as usual, please feel free to rebase if that is what you
> normally do, of course. This was in next-20250910. Thanks!

(Somehow I didn't Cc akpm -- doing it now.)

Could this be picked up? It blocks generation of docs for linux-next. Thanks!

By the way, there was a rebase -- the tag should now be:

Fixes: 6a25c101c7a2 ("rust: maple_tree: add MapleTree")

Cheers,
Miguel


From surenb at google.com  Wed Sep 24 09:49:22 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Wed, 24 Sep 2025 09:49:22 -0700
Subject: [PATCH v8 01/23] locking/local_lock: Expose dep_map in local_trylock_t.
In-Reply-To: <20250910-slub-percpu-caches-v8-1-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-1-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpE0AQ+59=PLb0xWK28NVAn_BxG-X=EPxy+dcRn2xWb6=g@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: Alexei Starovoitov <ast at kernel.org>
>
> lockdep_is_held() macro assumes that "struct lockdep_map dep_map;"
> is a top level field of any lock that participates in LOCKDEP.
> Make it so for local_trylock_t.
>
> Reviewed-by: Sebastian Andrzej Siewior <bigeasy at linutronix.de>
> Signed-off-by: Alexei Starovoitov <ast at kernel.org>
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  include/linux/local_lock_internal.h | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/include/linux/local_lock_internal.h b/include/linux/local_lock_internal.h
> index d80b5306a2c0ccf95a3405b6b947b5f1f9a3bd38..949de37700dbc10feafc06d0b52382cf2e00c694 100644
> --- a/include/linux/local_lock_internal.h
> +++ b/include/linux/local_lock_internal.h
> @@ -17,7 +17,10 @@ typedef struct {
>
>  /* local_trylock() and local_trylock_irqsave() only work with local_trylock_t */
>  typedef struct {
> -       local_lock_t    llock;
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +       struct lockdep_map      dep_map;
> +       struct task_struct      *owner;
> +#endif
>         u8              acquired;
>  } local_trylock_t;
>
> @@ -31,7 +34,7 @@ typedef struct {
>         .owner = NULL,
>
>  # define LOCAL_TRYLOCK_DEBUG_INIT(lockname)            \
> -       .llock = { LOCAL_LOCK_DEBUG_INIT((lockname).llock) },
> +       LOCAL_LOCK_DEBUG_INIT(lockname)
>
>  static inline void local_lock_acquire(local_lock_t *l)
>  {
> @@ -81,7 +84,7 @@ do {                                                          \
>         local_lock_debug_init(lock);                            \
>  } while (0)
>
> -#define __local_trylock_init(lock) __local_lock_init(lock.llock)
> +#define __local_trylock_init(lock) __local_lock_init((local_lock_t *)lock)
>
>  #define __spinlock_nested_bh_init(lock)                                \
>  do {                                                           \
>
> --
> 2.51.0
>


From surenb at google.com  Wed Sep 24 09:52:55 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Wed, 24 Sep 2025 09:52:55 -0700
Subject: [PATCH v8 02/23] slab: simplify init_kmem_cache_nodes() error handling
In-Reply-To: <20250910-slub-percpu-caches-v8-2-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-2-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpHzNyQw6N5CjfNXsMLyUQem2noSm+_A3O2gm+skA7qDsA@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> We don't need to call free_kmem_cache_nodes() immediately when failing
> to allocate a kmem_cache_node, because when we return 0,
> do_kmem_cache_create() calls __kmem_cache_release() which also performs
> free_kmem_cache_nodes().
>
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/slub.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 30003763d224c2704a4b93082b8b47af12dcffc5..9f671ec76131c4b0b28d5d568aa45842b5efb6d4 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -5669,10 +5669,8 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
>                 n = kmem_cache_alloc_node(kmem_cache_node,
>                                                 GFP_KERNEL, node);
>
> -               if (!n) {
> -                       free_kmem_cache_nodes(s);
> +               if (!n)
>                         return 0;
> -               }
>
>                 init_kmem_cache_node(n);
>                 s->node[node] = n;
>
> --
> 2.51.0
>


From surenb at google.com  Wed Sep 24 21:35:05 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Wed, 24 Sep 2025 21:35:05 -0700
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations
In-Reply-To: <40461105-a344-444f-834b-9559b6644710@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz> <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz> <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz> <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz> <aMq40h5iOjj8K7cc@hyeyoo>
 <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz> <40461105-a344-444f-834b-9559b6644710@suse.cz>
Message-ID: <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>

On Thu, Sep 18, 2025 at 1:09?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> On 9/17/25 16:14, Vlastimil Babka wrote:
> > On 9/17/25 15:34, Harry Yoo wrote:
> >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> >>> On 9/17/25 15:07, Harry Yoo wrote:
> >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> >>> >> On 9/17/25 13:32, Harry Yoo wrote:
> >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> >>> >> >> >> +                          sfw->skip = true;
> >>> >> >> >> +                          continue;
> >>> >> >> >> +                  }
> >>> >> >> >>
> >>> >> >> >> +                  INIT_WORK(&sfw->work, flush_rcu_sheaf);
> >>> >> >> >> +                  sfw->skip = false;
> >>> >> >> >> +                  sfw->s = s;
> >>> >> >> >> +                  queue_work_on(cpu, flushwq, &sfw->work);
> >>> >> >> >> +                  flushed = true;
> >>> >> >> >> +          }
> >>> >> >> >> +
> >>> >> >> >> +          for_each_online_cpu(cpu) {
> >>> >> >> >> +                  sfw = &per_cpu(slub_flush, cpu);
> >>> >> >> >> +                  if (sfw->skip)
> >>> >> >> >> +                          continue;
> >>> >> >> >> +                  flush_work(&sfw->work);
> >>> >> >> >> +          }
> >>> >> >> >> +
> >>> >> >> >> +          mutex_unlock(&flush_lock);
> >>> >> >> >> +  }
> >>> >> >> >> +
> >>> >> >> >> +  mutex_unlock(&slab_mutex);
> >>> >> >> >> +  cpus_read_unlock();
> >>> >> >> >> +
> >>> >> >> >> +  if (flushed)
> >>> >> >> >> +          rcu_barrier();
> >>> >> >> >
> >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> >>> >> >> >
> >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> >>> >> >> > so flushed == false but the rcu callback isn't processed yet
> >>> >> >> > by the end of the function?
> >>> >> >> >
> >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> >>> >> >> > but still possible...
> >>> >> >>
> >>> >> >> Yes also good point, will flush unconditionally.
> >>> >> >>
> >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> >>> >> >> local_unlock().
> >>> >> >>
> >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> >>> >> >
> >>> >> > Makes sense to me.
> >>> >
> >>> > Wait, I'm confused.
> >>> >
> >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> >>> > the object X to be freed before kvfree_rcu_barrier() returns?
> >>>
> >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> >>> sheaf also contains the object X, we should make sure that is flushed.
> >>
> >> I was going to say "but we queue and wait for the flushing work to
> >> complete, so the sheaf containing object X should be flushed?"
> >>
> >> But nah, that's true only if we see pcs->rcu_free != NULL in
> >> flush_all_rcu_sheaves().
> >>
> >> You are right...
> >>
> >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> >> even when pcs->rcu_sheaf == NULL?
> >
> > I guess it's simpler, yeah.
>
> So what about this? The unconditional queueing should cover all races with
> __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
>
> From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
> From: Vlastimil Babka <vbabka at suse.cz>
> Date: Wed, 3 Sep 2025 14:59:46 +0200
> Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
>
> Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> addition to main and spare sheaves.
>
> kfree_rcu() operations will try to put objects on this sheaf. Once full,
> the sheaf is detached and submitted to call_rcu() with a handler that
> will try to put it in the barn, or flush to slab pages using bulk free,
> when the barn is full. Then a new empty sheaf must be obtained to put
> more objects there.
>
> It's possible that no free sheaves are available to use for a new
> rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> kfree_rcu() implementation.
>
> Expected advantages:
> - batching the kfree_rcu() operations, that could eventually replace the
>   existing batching
> - sheaves can be reused for allocations via barn instead of being
>   flushed to slabs, which is more efficient
>   - this includes cases where only some cpus are allowed to process rcu
>     callbacks (Android)

nit: I would say it's more CONFIG_RCU_NOCB_CPU related. Android is
just an instance of that.

>
> Possible disadvantage:
> - objects might be waiting for more than their grace period (it is
>   determined by the last object freed into the sheaf), increasing memory
>   usage - but the existing batching does that too.
>
> Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> implementation favors smaller memory footprint over performance.
>
> Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> contexts where kfree_rcu() is called might not be compatible with taking
> a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> spinlock - the current kfree_rcu() implementation avoids doing that.
>
> Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> that have them. This is not a cheap operation, but the barrier usage is
> rare - currently kmem_cache_destroy() or on module unload.
>
> Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> count how many kfree_rcu() used the rcu_free sheaf successfully and how
> many had to fall back to the existing implementation.
>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  mm/slab.h        |   3 +
>  mm/slab_common.c |  26 +++++
>  mm/slub.c        | 267 ++++++++++++++++++++++++++++++++++++++++++++++-
>  3 files changed, 294 insertions(+), 2 deletions(-)
>
> diff --git a/mm/slab.h b/mm/slab.h
> index 206987ce44a4..e82e51c44bd0 100644
> --- a/mm/slab.h
> +++ b/mm/slab.h
> @@ -435,6 +435,9 @@ static inline bool is_kmalloc_normal(struct kmem_cache *s)
>         return !(s->flags & (SLAB_CACHE_DMA|SLAB_ACCOUNT|SLAB_RECLAIM_ACCOUNT));
>  }
>
> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj);
> +void flush_all_rcu_sheaves(void);
> +
>  #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
>                          SLAB_CACHE_DMA32 | SLAB_PANIC | \
>                          SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index e2b197e47866..005a4319c06a 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1608,6 +1608,27 @@ static void kfree_rcu_work(struct work_struct *work)
>                 kvfree_rcu_list(head);
>  }
>
> +static bool kfree_rcu_sheaf(void *obj)
> +{
> +       struct kmem_cache *s;
> +       struct folio *folio;
> +       struct slab *slab;
> +
> +       if (is_vmalloc_addr(obj))
> +               return false;
> +
> +       folio = virt_to_folio(obj);
> +       if (unlikely(!folio_test_slab(folio)))
> +               return false;
> +
> +       slab = folio_slab(folio);
> +       s = slab->slab_cache;
> +       if (s->cpu_sheaves)
> +               return __kfree_rcu_sheaf(s, obj);
> +
> +       return false;
> +}
> +
>  static bool
>  need_offload_krc(struct kfree_rcu_cpu *krcp)
>  {
> @@ -1952,6 +1973,9 @@ void kvfree_call_rcu(struct rcu_head *head, void *ptr)
>         if (!head)
>                 might_sleep();
>
> +       if (!IS_ENABLED(CONFIG_PREEMPT_RT) && kfree_rcu_sheaf(ptr))
> +               return;
> +
>         // Queue the object but don't yet schedule the batch.
>         if (debug_rcu_head_queue(ptr)) {
>                 // Probable double kfree_rcu(), just leak.
> @@ -2026,6 +2050,8 @@ void kvfree_rcu_barrier(void)
>         bool queued;
>         int i, cpu;
>
> +       flush_all_rcu_sheaves();
> +
>         /*
>          * Firstly we detach objects and queue them over an RCU-batch
>          * for all CPUs. Finally queued works are flushed for each CPU.
> diff --git a/mm/slub.c b/mm/slub.c
> index cba188b7e04d..171273f90efd 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -367,6 +367,8 @@ enum stat_item {
>         ALLOC_FASTPATH,         /* Allocation from cpu slab */
>         ALLOC_SLOWPATH,         /* Allocation by getting a new cpu slab */
>         FREE_PCS,               /* Free to percpu sheaf */
> +       FREE_RCU_SHEAF,         /* Free to rcu_free sheaf */
> +       FREE_RCU_SHEAF_FAIL,    /* Failed to free to a rcu_free sheaf */
>         FREE_FASTPATH,          /* Free to cpu slab */
>         FREE_SLOWPATH,          /* Freeing not to cpu slab */
>         FREE_FROZEN,            /* Freeing to frozen slab */
> @@ -461,6 +463,7 @@ struct slab_sheaf {
>                 struct rcu_head rcu_head;
>                 struct list_head barn_list;
>         };
> +       struct kmem_cache *cache;
>         unsigned int size;
>         void *objects[];
>  };
> @@ -469,6 +472,7 @@ struct slub_percpu_sheaves {
>         local_trylock_t lock;
>         struct slab_sheaf *main; /* never NULL when unlocked */
>         struct slab_sheaf *spare; /* empty or full, may be NULL */
> +       struct slab_sheaf *rcu_free; /* for batching kfree_rcu() */
>  };
>
>  /*
> @@ -2531,6 +2535,8 @@ static struct slab_sheaf *alloc_empty_sheaf(struct kmem_cache *s, gfp_t gfp)
>         if (unlikely(!sheaf))
>                 return NULL;
>
> +       sheaf->cache = s;
> +
>         stat(s, SHEAF_ALLOC);
>
>         return sheaf;
> @@ -2655,6 +2661,43 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
>         sheaf->size = 0;
>  }
>
> +static void __rcu_free_sheaf_prepare(struct kmem_cache *s,
> +                                    struct slab_sheaf *sheaf)
> +{
> +       bool init = slab_want_init_on_free(s);
> +       void **p = &sheaf->objects[0];
> +       unsigned int i = 0;
> +
> +       while (i < sheaf->size) {
> +               struct slab *slab = virt_to_slab(p[i]);
> +
> +               memcg_slab_free_hook(s, slab, p + i, 1);
> +               alloc_tagging_slab_free_hook(s, slab, p + i, 1);
> +
> +               if (unlikely(!slab_free_hook(s, p[i], init, true))) {
> +                       p[i] = p[--sheaf->size];
> +                       continue;
> +               }
> +
> +               i++;
> +       }
> +}
> +
> +static void rcu_free_sheaf_nobarn(struct rcu_head *head)
> +{
> +       struct slab_sheaf *sheaf;
> +       struct kmem_cache *s;
> +
> +       sheaf = container_of(head, struct slab_sheaf, rcu_head);
> +       s = sheaf->cache;
> +
> +       __rcu_free_sheaf_prepare(s, sheaf);
> +
> +       sheaf_flush_unused(s, sheaf);
> +
> +       free_empty_sheaf(s, sheaf);
> +}
> +
>  /*
>   * Caller needs to make sure migration is disabled in order to fully flush
>   * single cpu's sheaves
> @@ -2667,7 +2710,7 @@ static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf)
>  static void pcs_flush_all(struct kmem_cache *s)
>  {
>         struct slub_percpu_sheaves *pcs;
> -       struct slab_sheaf *spare;
> +       struct slab_sheaf *spare, *rcu_free;
>
>         local_lock(&s->cpu_sheaves->lock);
>         pcs = this_cpu_ptr(s->cpu_sheaves);
> @@ -2675,6 +2718,9 @@ static void pcs_flush_all(struct kmem_cache *s)
>         spare = pcs->spare;
>         pcs->spare = NULL;
>
> +       rcu_free = pcs->rcu_free;
> +       pcs->rcu_free = NULL;
> +
>         local_unlock(&s->cpu_sheaves->lock);
>
>         if (spare) {
> @@ -2682,6 +2728,9 @@ static void pcs_flush_all(struct kmem_cache *s)
>                 free_empty_sheaf(s, spare);
>         }
>
> +       if (rcu_free)
> +               call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> +
>         sheaf_flush_main(s);
>  }
>
> @@ -2698,6 +2747,11 @@ static void __pcs_flush_all_cpu(struct kmem_cache *s, unsigned int cpu)
>                 free_empty_sheaf(s, pcs->spare);
>                 pcs->spare = NULL;
>         }
> +
> +       if (pcs->rcu_free) {
> +               call_rcu(&pcs->rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> +               pcs->rcu_free = NULL;
> +       }
>  }
>
>  static void pcs_destroy(struct kmem_cache *s)
> @@ -2723,6 +2777,7 @@ static void pcs_destroy(struct kmem_cache *s)
>                  */
>
>                 WARN_ON(pcs->spare);
> +               WARN_ON(pcs->rcu_free);
>
>                 if (!WARN_ON(pcs->main->size)) {
>                         free_empty_sheaf(s, pcs->main);
> @@ -3780,7 +3835,7 @@ static bool has_pcs_used(int cpu, struct kmem_cache *s)
>
>         pcs = per_cpu_ptr(s->cpu_sheaves, cpu);
>
> -       return (pcs->spare || pcs->main->size);
> +       return (pcs->spare || pcs->rcu_free || pcs->main->size);
>  }
>
>  /*
> @@ -3840,6 +3895,77 @@ static void flush_all(struct kmem_cache *s)
>         cpus_read_unlock();
>  }
>
> +static void flush_rcu_sheaf(struct work_struct *w)
> +{
> +       struct slub_percpu_sheaves *pcs;
> +       struct slab_sheaf *rcu_free;
> +       struct slub_flush_work *sfw;
> +       struct kmem_cache *s;
> +
> +       sfw = container_of(w, struct slub_flush_work, work);
> +       s = sfw->s;
> +
> +       local_lock(&s->cpu_sheaves->lock);
> +       pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +       rcu_free = pcs->rcu_free;
> +       pcs->rcu_free = NULL;
> +
> +       local_unlock(&s->cpu_sheaves->lock);
> +
> +       if (rcu_free)
> +               call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> +}
> +
> +
> +/* needed for kvfree_rcu_barrier() */
> +void flush_all_rcu_sheaves(void)
> +{
> +       struct slub_flush_work *sfw;
> +       struct kmem_cache *s;
> +       unsigned int cpu;
> +
> +       cpus_read_lock();
> +       mutex_lock(&slab_mutex);
> +
> +       list_for_each_entry(s, &slab_caches, list) {
> +               if (!s->cpu_sheaves)
> +                       continue;
> +
> +               mutex_lock(&flush_lock);
> +
> +               for_each_online_cpu(cpu) {
> +                       sfw = &per_cpu(slub_flush, cpu);
> +
> +                       /*
> +                        * we don't check if rcu_free sheaf exists - racing
> +                        * __kfree_rcu_sheaf() might have just removed it.
> +                        * by executing flush_rcu_sheaf() on the cpu we make
> +                        * sure the __kfree_rcu_sheaf() finished its call_rcu()
> +                        */
> +
> +                       INIT_WORK(&sfw->work, flush_rcu_sheaf);
> +                       sfw->skip = false;

I think you don't need this sfw->skip flag since you never skip anymore, right?

> +                       sfw->s = s;
> +                       queue_work_on(cpu, flushwq, &sfw->work);
> +               }
> +
> +               for_each_online_cpu(cpu) {
> +                       sfw = &per_cpu(slub_flush, cpu);
> +                       if (sfw->skip)
> +                               continue;
> +                       flush_work(&sfw->work);

I'm sure I'm missing something but why can't we execute call_rcu()
from here instead of queuing the work which does call_rcu() and then
flushing all the queued work? I'm sure you have a good reason which
I'm missing.

> +               }
> +
> +               mutex_unlock(&flush_lock);
> +       }
> +
> +       mutex_unlock(&slab_mutex);
> +       cpus_read_unlock();
> +
> +       rcu_barrier();
> +}
> +
>  /*
>   * Use the cpu notifier to insure that the cpu slabs are flushed when
>   * necessary.
> @@ -5413,6 +5539,134 @@ bool free_to_pcs(struct kmem_cache *s, void *object)
>         return true;
>  }
>
> +static void rcu_free_sheaf(struct rcu_head *head)
> +{
> +       struct slab_sheaf *sheaf;
> +       struct node_barn *barn;
> +       struct kmem_cache *s;
> +
> +       sheaf = container_of(head, struct slab_sheaf, rcu_head);
> +
> +       s = sheaf->cache;
> +
> +       /*
> +        * This may remove some objects due to slab_free_hook() returning false,
> +        * so that the sheaf might no longer be completely full. But it's easier
> +        * to handle it as full (unless it became completely empty), as the code
> +        * handles it fine. The only downside is that sheaf will serve fewer
> +        * allocations when reused. It only happens due to debugging, which is a
> +        * performance hit anyway.
> +        */
> +       __rcu_free_sheaf_prepare(s, sheaf);
> +
> +       barn = get_node(s, numa_mem_id())->barn;
> +
> +       /* due to slab_free_hook() */
> +       if (unlikely(sheaf->size == 0))
> +               goto empty;
> +
> +       /*
> +        * Checking nr_full/nr_empty outside lock avoids contention in case the
> +        * barn is at the respective limit. Due to the race we might go over the
> +        * limit but that should be rare and harmless.
> +        */
> +
> +       if (data_race(barn->nr_full) < MAX_FULL_SHEAVES) {
> +               stat(s, BARN_PUT);
> +               barn_put_full_sheaf(barn, sheaf);
> +               return;
> +       }
> +
> +       stat(s, BARN_PUT_FAIL);
> +       sheaf_flush_unused(s, sheaf);
> +
> +empty:
> +       if (data_race(barn->nr_empty) < MAX_EMPTY_SHEAVES) {
> +               barn_put_empty_sheaf(barn, sheaf);
> +               return;
> +       }
> +
> +       free_empty_sheaf(s, sheaf);
> +}
> +
> +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
> +{
> +       struct slub_percpu_sheaves *pcs;
> +       struct slab_sheaf *rcu_sheaf;
> +
> +       if (!local_trylock(&s->cpu_sheaves->lock))
> +               goto fail;
> +
> +       pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +       if (unlikely(!pcs->rcu_free)) {
> +
> +               struct slab_sheaf *empty;
> +               struct node_barn *barn;
> +
> +               if (pcs->spare && pcs->spare->size == 0) {
> +                       pcs->rcu_free = pcs->spare;
> +                       pcs->spare = NULL;
> +                       goto do_free;
> +               }
> +
> +               barn = get_barn(s);
> +
> +               empty = barn_get_empty_sheaf(barn);
> +
> +               if (empty) {
> +                       pcs->rcu_free = empty;
> +                       goto do_free;
> +               }
> +
> +               local_unlock(&s->cpu_sheaves->lock);
> +
> +               empty = alloc_empty_sheaf(s, GFP_NOWAIT);
> +
> +               if (!empty)
> +                       goto fail;
> +
> +               if (!local_trylock(&s->cpu_sheaves->lock)) {
> +                       barn_put_empty_sheaf(barn, empty);
> +                       goto fail;
> +               }
> +
> +               pcs = this_cpu_ptr(s->cpu_sheaves);
> +
> +               if (unlikely(pcs->rcu_free))
> +                       barn_put_empty_sheaf(barn, empty);
> +               else
> +                       pcs->rcu_free = empty;
> +       }
> +
> +do_free:
> +
> +       rcu_sheaf = pcs->rcu_free;
> +
> +       rcu_sheaf->objects[rcu_sheaf->size++] = obj;

nit: The above would result in OOB write if we ever reached here with
a full rcu_sheaf (rcu_sheaf->size == rcu_sheaf->sheaf_capacity) but I
think it's impossible. You always start with an empty rcu_sheaf and
objects are added only here with a following check for a full
rcu_sheaf. I think a short comment clarifying that would be nice.

> +
> +       if (likely(rcu_sheaf->size < s->sheaf_capacity))
> +               rcu_sheaf = NULL;
> +       else
> +               pcs->rcu_free = NULL;
> +
> +       /*
> +        * we flush before local_unlock to make sure a racing
> +        * flush_all_rcu_sheaves() doesn't miss this sheaf
> +        */
> +       if (rcu_sheaf)
> +               call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> +
> +       local_unlock(&s->cpu_sheaves->lock);
> +
> +       stat(s, FREE_RCU_SHEAF);
> +       return true;
> +
> +fail:
> +       stat(s, FREE_RCU_SHEAF_FAIL);
> +       return false;
> +}
> +
>  /*
>   * Bulk free objects to the percpu sheaves.
>   * Unlike free_to_pcs() this includes the calls to all necessary hooks
> @@ -6909,6 +7163,11 @@ int __kmem_cache_shutdown(struct kmem_cache *s)
>         struct kmem_cache_node *n;
>
>         flush_all_cpus_locked(s);
> +
> +       /* we might have rcu sheaves in flight */
> +       if (s->cpu_sheaves)
> +               rcu_barrier();
> +
>         /* Attempt to free all objects */
>         for_each_kmem_cache_node(s, node, n) {
>                 if (n->barn)
> @@ -8284,6 +8543,8 @@ STAT_ATTR(ALLOC_PCS, alloc_cpu_sheaf);
>  STAT_ATTR(ALLOC_FASTPATH, alloc_fastpath);
>  STAT_ATTR(ALLOC_SLOWPATH, alloc_slowpath);
>  STAT_ATTR(FREE_PCS, free_cpu_sheaf);
> +STAT_ATTR(FREE_RCU_SHEAF, free_rcu_sheaf);
> +STAT_ATTR(FREE_RCU_SHEAF_FAIL, free_rcu_sheaf_fail);
>  STAT_ATTR(FREE_FASTPATH, free_fastpath);
>  STAT_ATTR(FREE_SLOWPATH, free_slowpath);
>  STAT_ATTR(FREE_FROZEN, free_frozen);
> @@ -8382,6 +8643,8 @@ static struct attribute *slab_attrs[] = {
>         &alloc_fastpath_attr.attr,
>         &alloc_slowpath_attr.attr,
>         &free_cpu_sheaf_attr.attr,
> +       &free_rcu_sheaf_attr.attr,
> +       &free_rcu_sheaf_fail_attr.attr,
>         &free_fastpath_attr.attr,
>         &free_slowpath_attr.attr,
>         &free_frozen_attr.attr,
> --
> 2.51.0
>
>


From harry.yoo at oracle.com  Thu Sep 25 01:52:10 2025
From: harry.yoo at oracle.com (Harry Yoo)
Date: Thu, 25 Sep 2025 17:52:10 +0900
Subject: [PATCH v8 04/23] slab: add sheaf support for batching
 kfree_rcu() operations
In-Reply-To: <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>
References: <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo>
 <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz>
 <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>
Message-ID: <aNUCuvUh5Ugdnyxr@hyeyoo>

On Wed, Sep 24, 2025 at 09:35:05PM -0700, Suren Baghdasaryan wrote:
> On Thu, Sep 18, 2025 at 1:09?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> >
> > On 9/17/25 16:14, Vlastimil Babka wrote:
> > > On 9/17/25 15:34, Harry Yoo wrote:
> > >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> > >>> On 9/17/25 15:07, Harry Yoo wrote:
> > >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> > >>> >> On 9/17/25 13:32, Harry Yoo wrote:
> > >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> > >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> > >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> > >>> >> >> >> +                          sfw->skip = true;
> > >>> >> >> >> +                          continue;
> > >>> >> >> >> +                  }
> > >>> >> >> >>
> > >>> >> >> >> +                  INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > >>> >> >> >> +                  sfw->skip = false;
> > >>> >> >> >> +                  sfw->s = s;
> > >>> >> >> >> +                  queue_work_on(cpu, flushwq, &sfw->work);
> > >>> >> >> >> +                  flushed = true;
> > >>> >> >> >> +          }
> > >>> >> >> >> +
> > >>> >> >> >> +          for_each_online_cpu(cpu) {
> > >>> >> >> >> +                  sfw = &per_cpu(slub_flush, cpu);
> > >>> >> >> >> +                  if (sfw->skip)
> > >>> >> >> >> +                          continue;
> > >>> >> >> >> +                  flush_work(&sfw->work);
> > >>> >> >> >> +          }
> > >>> >> >> >> +
> > >>> >> >> >> +          mutex_unlock(&flush_lock);
> > >>> >> >> >> +  }
> > >>> >> >> >> +
> > >>> >> >> >> +  mutex_unlock(&slab_mutex);
> > >>> >> >> >> +  cpus_read_unlock();
> > >>> >> >> >> +
> > >>> >> >> >> +  if (flushed)
> > >>> >> >> >> +          rcu_barrier();
> > >>> >> >> >
> > >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> > >>> >> >> >
> > >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> > >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> > >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> > >>> >> >> > so flushed == false but the rcu callback isn't processed yet
> > >>> >> >> > by the end of the function?
> > >>> >> >> >
> > >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> > >>> >> >> > but still possible...
> > >>> >> >>
> > >>> >> >> Yes also good point, will flush unconditionally.
> > >>> >> >>
> > >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> > >>> >> >> local_unlock().
> > >>> >> >>
> > >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> > >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> > >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> > >>> >> >
> > >>> >> > Makes sense to me.
> > >>> >
> > >>> > Wait, I'm confused.
> > >>> >
> > >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> > >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> > >>> > the object X to be freed before kvfree_rcu_barrier() returns?
> > >>>
> > >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> > >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> > >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> > >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> > >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> > >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> > >>> sheaf also contains the object X, we should make sure that is flushed.
> > >>
> > >> I was going to say "but we queue and wait for the flushing work to
> > >> complete, so the sheaf containing object X should be flushed?"
> > >>
> > >> But nah, that's true only if we see pcs->rcu_free != NULL in
> > >> flush_all_rcu_sheaves().
> > >>
> > >> You are right...
> > >>
> > >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> > >> even when pcs->rcu_sheaf == NULL?
> > >
> > > I guess it's simpler, yeah.
> >
> > So what about this? The unconditional queueing should cover all races with
> > __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
> >
> > From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
> > From: Vlastimil Babka <vbabka at suse.cz>
> > Date: Wed, 3 Sep 2025 14:59:46 +0200
> > Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
> >
> > Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> > For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> > addition to main and spare sheaves.
> >
> > kfree_rcu() operations will try to put objects on this sheaf. Once full,
> > the sheaf is detached and submitted to call_rcu() with a handler that
> > will try to put it in the barn, or flush to slab pages using bulk free,
> > when the barn is full. Then a new empty sheaf must be obtained to put
> > more objects there.
> >
> > It's possible that no free sheaves are available to use for a new
> > rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> > GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> > kfree_rcu() implementation.
> >
> > Expected advantages:
> > - batching the kfree_rcu() operations, that could eventually replace the
> >   existing batching
> > - sheaves can be reused for allocations via barn instead of being
> >   flushed to slabs, which is more efficient
> >   - this includes cases where only some cpus are allowed to process rcu
> >     callbacks (Android)
> 
> nit: I would say it's more CONFIG_RCU_NOCB_CPU related. Android is
> just an instance of that.
> 
> >
> > Possible disadvantage:
> > - objects might be waiting for more than their grace period (it is
> >   determined by the last object freed into the sheaf), increasing memory
> >   usage - but the existing batching does that too.
> >
> > Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> > implementation favors smaller memory footprint over performance.
> >
> > Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> > contexts where kfree_rcu() is called might not be compatible with taking
> > a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> > spinlock - the current kfree_rcu() implementation avoids doing that.
> >
> > Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> > that have them. This is not a cheap operation, but the barrier usage is
> > rare - currently kmem_cache_destroy() or on module unload.
> >
> > Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> > count how many kfree_rcu() used the rcu_free sheaf successfully and how
> > many had to fall back to the existing implementation.
> >
> > Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> > ---
> >  mm/slab.h        |   3 +
> >  mm/slab_common.c |  26 +++++
> >  mm/slub.c        | 267 ++++++++++++++++++++++++++++++++++++++++++++++-
> >  3 files changed, 294 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/slub.c b/mm/slub.c
> > index cba188b7e04d..171273f90efd 100644
> > --- a/mm/slub.c
> > +++ b/mm/slub.c

[...snip...]

> > @@ -3840,6 +3895,77 @@ static void flush_all(struct kmem_cache *s)
> >         cpus_read_unlock();
> >  }
> >
> > +static void flush_rcu_sheaf(struct work_struct *w)
> > +{
> > +       struct slub_percpu_sheaves *pcs;
> > +       struct slab_sheaf *rcu_free;
> > +       struct slub_flush_work *sfw;
> > +       struct kmem_cache *s;
> > +
> > +       sfw = container_of(w, struct slub_flush_work, work);
> > +       s = sfw->s;
> > +
> > +       local_lock(&s->cpu_sheaves->lock);
> > +       pcs = this_cpu_ptr(s->cpu_sheaves);
> > +
> > +       rcu_free = pcs->rcu_free;
> > +       pcs->rcu_free = NULL;
> > +
> > +       local_unlock(&s->cpu_sheaves->lock);
> > +
> > +       if (rcu_free)
> > +               call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> > +}
> > +
> > +
> > +/* needed for kvfree_rcu_barrier() */
> > +void flush_all_rcu_sheaves(void)
> > +{
> > +       struct slub_flush_work *sfw;
> > +       struct kmem_cache *s;
> > +       unsigned int cpu;
> > +
> > +       cpus_read_lock();
> > +       mutex_lock(&slab_mutex);
> > +
> > +       list_for_each_entry(s, &slab_caches, list) {
> > +               if (!s->cpu_sheaves)
> > +                       continue;
> > +
> > +               mutex_lock(&flush_lock);
> > +
> > +               for_each_online_cpu(cpu) {
> > +                       sfw = &per_cpu(slub_flush, cpu);
> > +
> > +                       /*
> > +                        * we don't check if rcu_free sheaf exists - racing
> > +                        * __kfree_rcu_sheaf() might have just removed it.
> > +                        * by executing flush_rcu_sheaf() on the cpu we make
> > +                        * sure the __kfree_rcu_sheaf() finished its call_rcu()
> > +                        */
> > +
> > +                       INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > +                       sfw->skip = false;
> 
> I think you don't need this sfw->skip flag since you never skip anymore, right?

Yes, at least in flush_all_rcu_sheaves().
I'm fine with or without sfw->skip in this function.

> > +                       sfw->s = s;
> > +                       queue_work_on(cpu, flushwq, &sfw->work);
> > +               }
> > +
> > +               for_each_online_cpu(cpu) {
> > +                       sfw = &per_cpu(slub_flush, cpu);
> > +                       if (sfw->skip)
> > +                               continue;
> > +                       flush_work(&sfw->work);
> 
> I'm sure I'm missing something but why can't we execute call_rcu()
> from here instead of queuing the work which does call_rcu() and then
> flushing all the queued work? I'm sure you have a good reason which
> I'm missing.

Because a local lock cannot be held by other CPUs, you can't take off the
rcu_free sheaf remotely and call call_rcu(). That's why the work is
queued on each CPU, ensuring the rcu_free sheaf is flushed by its local CPU.

> > +               }
> > +
> > +               mutex_unlock(&flush_lock);
> > +       }
> > +
> > +       mutex_unlock(&slab_mutex);
> > +       cpus_read_unlock();
> > +
> > +       rcu_barrier();
> > +}
> > +
> >  /*
> >   * Use the cpu notifier to insure that the cpu slabs are flushed when
> >   * necessary.
> > +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
> > +{
> > +       struct slub_percpu_sheaves *pcs;
> > +       struct slab_sheaf *rcu_sheaf;
> > +
> > +       if (!local_trylock(&s->cpu_sheaves->lock))
> > +               goto fail;
> > +
> > +       pcs = this_cpu_ptr(s->cpu_sheaves);
> > +
> > +       if (unlikely(!pcs->rcu_free)) {
> > +
> > +               struct slab_sheaf *empty;
> > +               struct node_barn *barn;
> > +
> > +               if (pcs->spare && pcs->spare->size == 0) {
> > +                       pcs->rcu_free = pcs->spare;
> > +                       pcs->spare = NULL;
> > +                       goto do_free;
> > +               }
> > +
> > +               barn = get_barn(s);
> > +
> > +               empty = barn_get_empty_sheaf(barn);
> > +
> > +               if (empty) {
> > +                       pcs->rcu_free = empty;
> > +                       goto do_free;
> > +               }
> > +
> > +               local_unlock(&s->cpu_sheaves->lock);
> > +
> > +               empty = alloc_empty_sheaf(s, GFP_NOWAIT);
> > +
> > +               if (!empty)
> > +                       goto fail;
> > +
> > +               if (!local_trylock(&s->cpu_sheaves->lock)) {
> > +                       barn_put_empty_sheaf(barn, empty);
> > +                       goto fail;
> > +               }
> > +
> > +               pcs = this_cpu_ptr(s->cpu_sheaves);
> > +
> > +               if (unlikely(pcs->rcu_free))
> > +                       barn_put_empty_sheaf(barn, empty);
> > +               else
> > +                       pcs->rcu_free = empty;
> > +       }
> > +
> > +do_free:
> > +
> > +       rcu_sheaf = pcs->rcu_free;
> > +
> > +       rcu_sheaf->objects[rcu_sheaf->size++] = obj;
> 
> nit: The above would result in OOB write if we ever reached here with
> a full rcu_sheaf (rcu_sheaf->size == rcu_sheaf->sheaf_capacity) but I
> think it's impossible. You always start with an empty rcu_sheaf and
> objects are added only here with a following check for a full
> rcu_sheaf. I think a short comment clarifying that would be nice.

Sounds good to me.

-- 
Cheers,
Harry / Hyeonggon

> > +
> > +       if (likely(rcu_sheaf->size < s->sheaf_capacity))
> > +               rcu_sheaf = NULL;
> > +       else
> > +               pcs->rcu_free = NULL;
> > +
> > +       /*
> > +        * we flush before local_unlock to make sure a racing
> > +        * flush_all_rcu_sheaves() doesn't miss this sheaf
> > +        */
> > +       if (rcu_sheaf)
> > +               call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> > +
> > +       local_unlock(&s->cpu_sheaves->lock);
> > +
> > +       stat(s, FREE_RCU_SHEAF);
> > +       return true;
> > +
> > +fail:
> > +       stat(s, FREE_RCU_SHEAF_FAIL);
> > +       return false;
> > +}


From surenb at google.com  Thu Sep 25 06:38:36 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 25 Sep 2025 06:38:36 -0700
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations
In-Reply-To: <aNUCuvUh5Ugdnyxr@hyeyoo>
References: <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo> <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo> <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz> <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>
 <aNUCuvUh5Ugdnyxr@hyeyoo>
Message-ID: <CAJuCfpFG0wzY24xyXBjdRsHK3zxbVskU3raoQXF9k-gfnZEntQ@mail.gmail.com>

On Thu, Sep 25, 2025 at 1:52?AM Harry Yoo <harry.yoo at oracle.com> wrote:
>
> On Wed, Sep 24, 2025 at 09:35:05PM -0700, Suren Baghdasaryan wrote:
> > On Thu, Sep 18, 2025 at 1:09?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> > >
> > > On 9/17/25 16:14, Vlastimil Babka wrote:
> > > > On 9/17/25 15:34, Harry Yoo wrote:
> > > >> On Wed, Sep 17, 2025 at 03:21:31PM +0200, Vlastimil Babka wrote:
> > > >>> On 9/17/25 15:07, Harry Yoo wrote:
> > > >>> > On Wed, Sep 17, 2025 at 02:05:49PM +0200, Vlastimil Babka wrote:
> > > >>> >> On 9/17/25 13:32, Harry Yoo wrote:
> > > >>> >> > On Wed, Sep 17, 2025 at 11:55:10AM +0200, Vlastimil Babka wrote:
> > > >>> >> >> On 9/17/25 10:30, Harry Yoo wrote:
> > > >>> >> >> > On Wed, Sep 10, 2025 at 10:01:06AM +0200, Vlastimil Babka wrote:
> > > >>> >> >> >> +                          sfw->skip = true;
> > > >>> >> >> >> +                          continue;
> > > >>> >> >> >> +                  }
> > > >>> >> >> >>
> > > >>> >> >> >> +                  INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > > >>> >> >> >> +                  sfw->skip = false;
> > > >>> >> >> >> +                  sfw->s = s;
> > > >>> >> >> >> +                  queue_work_on(cpu, flushwq, &sfw->work);
> > > >>> >> >> >> +                  flushed = true;
> > > >>> >> >> >> +          }
> > > >>> >> >> >> +
> > > >>> >> >> >> +          for_each_online_cpu(cpu) {
> > > >>> >> >> >> +                  sfw = &per_cpu(slub_flush, cpu);
> > > >>> >> >> >> +                  if (sfw->skip)
> > > >>> >> >> >> +                          continue;
> > > >>> >> >> >> +                  flush_work(&sfw->work);
> > > >>> >> >> >> +          }
> > > >>> >> >> >> +
> > > >>> >> >> >> +          mutex_unlock(&flush_lock);
> > > >>> >> >> >> +  }
> > > >>> >> >> >> +
> > > >>> >> >> >> +  mutex_unlock(&slab_mutex);
> > > >>> >> >> >> +  cpus_read_unlock();
> > > >>> >> >> >> +
> > > >>> >> >> >> +  if (flushed)
> > > >>> >> >> >> +          rcu_barrier();
> > > >>> >> >> >
> > > >>> >> >> > I think we need to call rcu_barrier() even if flushed == false?
> > > >>> >> >> >
> > > >>> >> >> > Maybe a kvfree_rcu()'d object was already waiting for the rcu callback to
> > > >>> >> >> > be processed before flush_all_rcu_sheaves() is called, and
> > > >>> >> >> > in flush_all_rcu_sheaves() we skipped all (cache, cpu) pairs,
> > > >>> >> >> > so flushed == false but the rcu callback isn't processed yet
> > > >>> >> >> > by the end of the function?
> > > >>> >> >> >
> > > >>> >> >> > That sounds like a very unlikely to happen in a realistic scenario,
> > > >>> >> >> > but still possible...
> > > >>> >> >>
> > > >>> >> >> Yes also good point, will flush unconditionally.
> > > >>> >> >>
> > > >>> >> >> Maybe in __kfree_rcu_sheaf() I should also move the call_rcu(...) before
> > > >>> >> >> local_unlock().
> > > >>> >> >>
> > > >>> >> >> So we don't end up seeing a NULL pcs->rcu_free in
> > > >>> >> >> flush_all_rcu_sheaves() because __kfree_rcu_sheaf() already set it to NULL,
> > > >>> >> >> but didn't yet do the call_rcu() as it got preempted after local_unlock().
> > > >>> >> >
> > > >>> >> > Makes sense to me.
> > > >>> >
> > > >>> > Wait, I'm confused.
> > > >>> >
> > > >>> > I think the caller of kvfree_rcu_barrier() should make sure that it's invoked
> > > >>> > only after a kvfree_rcu(X, rhs) call has returned, if the caller expects
> > > >>> > the object X to be freed before kvfree_rcu_barrier() returns?
> > > >>>
> > > >>> Hmm, the caller of kvfree_rcu(X, rhs) might have returned without filling up
> > > >>> the rcu_sheaf fully and thus without submitting it to call_rcu(), then
> > > >>> migrated to another cpu. Then it calls kvfree_rcu_barrier() while another
> > > >>> unrelated kvfree_rcu(X, rhs) call on the previous cpu is for the same
> > > >>> kmem_cache (kvfree_rcu_barrier() is not only for cache destruction), fills
> > > >>> up the rcu_sheaf fully and is about to call_rcu() on it. And since that
> > > >>> sheaf also contains the object X, we should make sure that is flushed.
> > > >>
> > > >> I was going to say "but we queue and wait for the flushing work to
> > > >> complete, so the sheaf containing object X should be flushed?"
> > > >>
> > > >> But nah, that's true only if we see pcs->rcu_free != NULL in
> > > >> flush_all_rcu_sheaves().
> > > >>
> > > >> You are right...
> > > >>
> > > >> Hmm, maybe it's simpler to fix this by never skipping queueing the work
> > > >> even when pcs->rcu_sheaf == NULL?
> > > >
> > > > I guess it's simpler, yeah.
> > >
> > > So what about this? The unconditional queueing should cover all races with
> > > __kfree_rcu_sheaf() so there's just unconditional rcu_barrier() in the end.
> > >
> > > From 0722b29fa1625b31c05d659d1d988ec882247b38 Mon Sep 17 00:00:00 2001
> > > From: Vlastimil Babka <vbabka at suse.cz>
> > > Date: Wed, 3 Sep 2025 14:59:46 +0200
> > > Subject: [PATCH] slab: add sheaf support for batching kfree_rcu() operations
> > >
> > > Extend the sheaf infrastructure for more efficient kfree_rcu() handling.
> > > For caches with sheaves, on each cpu maintain a rcu_free sheaf in
> > > addition to main and spare sheaves.
> > >
> > > kfree_rcu() operations will try to put objects on this sheaf. Once full,
> > > the sheaf is detached and submitted to call_rcu() with a handler that
> > > will try to put it in the barn, or flush to slab pages using bulk free,
> > > when the barn is full. Then a new empty sheaf must be obtained to put
> > > more objects there.
> > >
> > > It's possible that no free sheaves are available to use for a new
> > > rcu_free sheaf, and the allocation in kfree_rcu() context can only use
> > > GFP_NOWAIT and thus may fail. In that case, fall back to the existing
> > > kfree_rcu() implementation.
> > >
> > > Expected advantages:
> > > - batching the kfree_rcu() operations, that could eventually replace the
> > >   existing batching
> > > - sheaves can be reused for allocations via barn instead of being
> > >   flushed to slabs, which is more efficient
> > >   - this includes cases where only some cpus are allowed to process rcu
> > >     callbacks (Android)
> >
> > nit: I would say it's more CONFIG_RCU_NOCB_CPU related. Android is
> > just an instance of that.
> >
> > >
> > > Possible disadvantage:
> > > - objects might be waiting for more than their grace period (it is
> > >   determined by the last object freed into the sheaf), increasing memory
> > >   usage - but the existing batching does that too.
> > >
> > > Only implement this for CONFIG_KVFREE_RCU_BATCHED as the tiny
> > > implementation favors smaller memory footprint over performance.
> > >
> > > Also for now skip the usage of rcu sheaf for CONFIG_PREEMPT_RT as the
> > > contexts where kfree_rcu() is called might not be compatible with taking
> > > a barn spinlock or a GFP_NOWAIT allocation of a new sheaf taking a
> > > spinlock - the current kfree_rcu() implementation avoids doing that.
> > >
> > > Teach kvfree_rcu_barrier() to flush all rcu_free sheaves from all caches
> > > that have them. This is not a cheap operation, but the barrier usage is
> > > rare - currently kmem_cache_destroy() or on module unload.
> > >
> > > Add CONFIG_SLUB_STATS counters free_rcu_sheaf and free_rcu_sheaf_fail to
> > > count how many kfree_rcu() used the rcu_free sheaf successfully and how
> > > many had to fall back to the existing implementation.
> > >
> > > Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> > > ---
> > >  mm/slab.h        |   3 +
> > >  mm/slab_common.c |  26 +++++
> > >  mm/slub.c        | 267 ++++++++++++++++++++++++++++++++++++++++++++++-
> > >  3 files changed, 294 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/mm/slub.c b/mm/slub.c
> > > index cba188b7e04d..171273f90efd 100644
> > > --- a/mm/slub.c
> > > +++ b/mm/slub.c
>
> [...snip...]
>
> > > @@ -3840,6 +3895,77 @@ static void flush_all(struct kmem_cache *s)
> > >         cpus_read_unlock();
> > >  }
> > >
> > > +static void flush_rcu_sheaf(struct work_struct *w)
> > > +{
> > > +       struct slub_percpu_sheaves *pcs;
> > > +       struct slab_sheaf *rcu_free;
> > > +       struct slub_flush_work *sfw;
> > > +       struct kmem_cache *s;
> > > +
> > > +       sfw = container_of(w, struct slub_flush_work, work);
> > > +       s = sfw->s;
> > > +
> > > +       local_lock(&s->cpu_sheaves->lock);
> > > +       pcs = this_cpu_ptr(s->cpu_sheaves);
> > > +
> > > +       rcu_free = pcs->rcu_free;
> > > +       pcs->rcu_free = NULL;
> > > +
> > > +       local_unlock(&s->cpu_sheaves->lock);
> > > +
> > > +       if (rcu_free)
> > > +               call_rcu(&rcu_free->rcu_head, rcu_free_sheaf_nobarn);
> > > +}
> > > +
> > > +
> > > +/* needed for kvfree_rcu_barrier() */
> > > +void flush_all_rcu_sheaves(void)
> > > +{
> > > +       struct slub_flush_work *sfw;
> > > +       struct kmem_cache *s;
> > > +       unsigned int cpu;
> > > +
> > > +       cpus_read_lock();
> > > +       mutex_lock(&slab_mutex);
> > > +
> > > +       list_for_each_entry(s, &slab_caches, list) {
> > > +               if (!s->cpu_sheaves)
> > > +                       continue;
> > > +
> > > +               mutex_lock(&flush_lock);
> > > +
> > > +               for_each_online_cpu(cpu) {
> > > +                       sfw = &per_cpu(slub_flush, cpu);
> > > +
> > > +                       /*
> > > +                        * we don't check if rcu_free sheaf exists - racing
> > > +                        * __kfree_rcu_sheaf() might have just removed it.
> > > +                        * by executing flush_rcu_sheaf() on the cpu we make
> > > +                        * sure the __kfree_rcu_sheaf() finished its call_rcu()
> > > +                        */
> > > +
> > > +                       INIT_WORK(&sfw->work, flush_rcu_sheaf);
> > > +                       sfw->skip = false;
> >
> > I think you don't need this sfw->skip flag since you never skip anymore, right?
>
> Yes, at least in flush_all_rcu_sheaves().
> I'm fine with or without sfw->skip in this function.
>
> > > +                       sfw->s = s;
> > > +                       queue_work_on(cpu, flushwq, &sfw->work);
> > > +               }
> > > +
> > > +               for_each_online_cpu(cpu) {
> > > +                       sfw = &per_cpu(slub_flush, cpu);
> > > +                       if (sfw->skip)
> > > +                               continue;
> > > +                       flush_work(&sfw->work);
> >
> > I'm sure I'm missing something but why can't we execute call_rcu()
> > from here instead of queuing the work which does call_rcu() and then
> > flushing all the queued work? I'm sure you have a good reason which
> > I'm missing.
>
> Because a local lock cannot be held by other CPUs, you can't take off the
> rcu_free sheaf remotely and call call_rcu(). That's why the work is
> queued on each CPU, ensuring the rcu_free sheaf is flushed by its local CPU.

Ah, yes, of course. I knew it was something obvious but my brain was
too tired. Thanks for the explanation, Harry!

>
> > > +               }
> > > +
> > > +               mutex_unlock(&flush_lock);
> > > +       }
> > > +
> > > +       mutex_unlock(&slab_mutex);
> > > +       cpus_read_unlock();
> > > +
> > > +       rcu_barrier();
> > > +}
> > > +
> > >  /*
> > >   * Use the cpu notifier to insure that the cpu slabs are flushed when
> > >   * necessary.
> > > +bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
> > > +{
> > > +       struct slub_percpu_sheaves *pcs;
> > > +       struct slab_sheaf *rcu_sheaf;
> > > +
> > > +       if (!local_trylock(&s->cpu_sheaves->lock))
> > > +               goto fail;
> > > +
> > > +       pcs = this_cpu_ptr(s->cpu_sheaves);
> > > +
> > > +       if (unlikely(!pcs->rcu_free)) {
> > > +
> > > +               struct slab_sheaf *empty;
> > > +               struct node_barn *barn;
> > > +
> > > +               if (pcs->spare && pcs->spare->size == 0) {
> > > +                       pcs->rcu_free = pcs->spare;
> > > +                       pcs->spare = NULL;
> > > +                       goto do_free;
> > > +               }
> > > +
> > > +               barn = get_barn(s);
> > > +
> > > +               empty = barn_get_empty_sheaf(barn);
> > > +
> > > +               if (empty) {
> > > +                       pcs->rcu_free = empty;
> > > +                       goto do_free;
> > > +               }
> > > +
> > > +               local_unlock(&s->cpu_sheaves->lock);
> > > +
> > > +               empty = alloc_empty_sheaf(s, GFP_NOWAIT);
> > > +
> > > +               if (!empty)
> > > +                       goto fail;
> > > +
> > > +               if (!local_trylock(&s->cpu_sheaves->lock)) {
> > > +                       barn_put_empty_sheaf(barn, empty);
> > > +                       goto fail;
> > > +               }
> > > +
> > > +               pcs = this_cpu_ptr(s->cpu_sheaves);
> > > +
> > > +               if (unlikely(pcs->rcu_free))
> > > +                       barn_put_empty_sheaf(barn, empty);
> > > +               else
> > > +                       pcs->rcu_free = empty;
> > > +       }
> > > +
> > > +do_free:
> > > +
> > > +       rcu_sheaf = pcs->rcu_free;
> > > +
> > > +       rcu_sheaf->objects[rcu_sheaf->size++] = obj;
> >
> > nit: The above would result in OOB write if we ever reached here with
> > a full rcu_sheaf (rcu_sheaf->size == rcu_sheaf->sheaf_capacity) but I
> > think it's impossible. You always start with an empty rcu_sheaf and
> > objects are added only here with a following check for a full
> > rcu_sheaf. I think a short comment clarifying that would be nice.
>
> Sounds good to me.
>
> --
> Cheers,
> Harry / Hyeonggon
>
> > > +
> > > +       if (likely(rcu_sheaf->size < s->sheaf_capacity))
> > > +               rcu_sheaf = NULL;
> > > +       else
> > > +               pcs->rcu_free = NULL;
> > > +
> > > +       /*
> > > +        * we flush before local_unlock to make sure a racing
> > > +        * flush_all_rcu_sheaves() doesn't miss this sheaf
> > > +        */
> > > +       if (rcu_sheaf)
> > > +               call_rcu(&rcu_sheaf->rcu_head, rcu_free_sheaf);
> > > +
> > > +       local_unlock(&s->cpu_sheaves->lock);
> > > +
> > > +       stat(s, FREE_RCU_SHEAF);
> > > +       return true;
> > > +
> > > +fail:
> > > +       stat(s, FREE_RCU_SHEAF_FAIL);
> > > +       return false;
> > > +}


From surenb at google.com  Thu Sep 25 09:14:50 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 25 Sep 2025 09:14:50 -0700
Subject: [PATCH v8 07/23] slab: skip percpu sheaves for remote object freeing
In-Reply-To: <20250910-slub-percpu-caches-v8-7-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-7-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpHmDxgFj+Jh9+qK1nwnnGWGBvPYEFrLSdooCUy5ODRboA@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> Since we don't control the NUMA locality of objects in percpu sheaves,
> allocations with node restrictions bypass them. Allocations without
> restrictions may however still expect to get local objects with high
> probability, and the introduction of sheaves can decrease it due to
> freed object from a remote node ending up in percpu sheaves.
>
> The fraction of such remote frees seems low (5% on an 8-node machine)
> but it can be expected that some cache or workload specific corner cases
> exist. We can either conclude that this is not a problem due to the low
> fraction, or we can make remote frees bypass percpu sheaves and go
> directly to their slabs. This will make the remote frees more expensive,
> but if if's only a small fraction, most frees will still benefit from

s/if's/it's

> the lower overhead of percpu sheaves.
>
> This patch thus makes remote object freeing bypass percpu sheaves,
> including bulk freeing, and kfree_rcu() via the rcu_free sheaf. However
> it's not intended to be 100% guarantee that percpu sheaves will only
> contain local objects. The refill from slabs does not provide that
> guarantee in the first place, and there might be cpu migrations
> happening when we need to unlock the local_lock. Avoiding all that could
> be possible but complicated so we can leave it for later investigation
> whether it would be worth it. It can be expected that the more selective
> freeing will itself prevent accumulation of remote objects in percpu
> sheaves so any such violations would have only short-term effects.
>
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/slab_common.c |  7 +++++--
>  mm/slub.c        | 42 ++++++++++++++++++++++++++++++++++++------
>  2 files changed, 41 insertions(+), 8 deletions(-)
>
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 005a4319c06a01d2b616a75396fcc43766a62ddb..b6601e0fe598e24bd8d456dce4fc82c65b342bfd 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -1623,8 +1623,11 @@ static bool kfree_rcu_sheaf(void *obj)
>
>         slab = folio_slab(folio);
>         s = slab->slab_cache;
> -       if (s->cpu_sheaves)
> -               return __kfree_rcu_sheaf(s, obj);
> +       if (s->cpu_sheaves) {
> +               if (likely(!IS_ENABLED(CONFIG_NUMA) ||
> +                          slab_nid(slab) == numa_mem_id()))
> +                       return __kfree_rcu_sheaf(s, obj);
> +       }
>
>         return false;
>  }
> diff --git a/mm/slub.c b/mm/slub.c
> index 35274ce4e709c9da7ac8f9006c824f28709e923d..9699d048b2cd08ee75c4cc3d1e460868704520b1 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -472,6 +472,7 @@ struct slab_sheaf {
>         };
>         struct kmem_cache *cache;
>         unsigned int size;
> +       int node; /* only used for rcu_sheaf */
>         void *objects[];
>  };
>
> @@ -5828,7 +5829,7 @@ static void rcu_free_sheaf(struct rcu_head *head)
>          */
>         __rcu_free_sheaf_prepare(s, sheaf);
>
> -       barn = get_node(s, numa_mem_id())->barn;
> +       barn = get_node(s, sheaf->node)->barn;
>
>         /* due to slab_free_hook() */
>         if (unlikely(sheaf->size == 0))
> @@ -5914,10 +5915,12 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
>
>         rcu_sheaf->objects[rcu_sheaf->size++] = obj;
>
> -       if (likely(rcu_sheaf->size < s->sheaf_capacity))
> +       if (likely(rcu_sheaf->size < s->sheaf_capacity)) {
>                 rcu_sheaf = NULL;
> -       else
> +       } else {
>                 pcs->rcu_free = NULL;
> +               rcu_sheaf->node = numa_mem_id();
> +       }
>
>         local_unlock(&s->cpu_sheaves->lock);
>
> @@ -5944,7 +5947,11 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
>         bool init = slab_want_init_on_free(s);
>         unsigned int batch, i = 0;
>         struct node_barn *barn;
> +       void *remote_objects[PCS_BATCH_MAX];
> +       unsigned int remote_nr = 0;
> +       int node = numa_mem_id();
>
> +next_remote_batch:
>         while (i < size) {
>                 struct slab *slab = virt_to_slab(p[i]);
>
> @@ -5954,7 +5961,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
>                 if (unlikely(!slab_free_hook(s, p[i], init, false))) {
>                         p[i] = p[--size];
>                         if (!size)
> -                               return;
> +                               goto flush_remote;
> +                       continue;
> +               }
> +
> +               if (unlikely(IS_ENABLED(CONFIG_NUMA) && slab_nid(slab) != node)) {
> +                       remote_objects[remote_nr] = p[i];
> +                       p[i] = p[--size];
> +                       if (++remote_nr >= PCS_BATCH_MAX)
> +                               goto flush_remote;
>                         continue;
>                 }
>
> @@ -6024,6 +6039,15 @@ static void free_to_pcs_bulk(struct kmem_cache *s, size_t size, void **p)
>          */
>  fallback:
>         __kmem_cache_free_bulk(s, size, p);
> +
> +flush_remote:
> +       if (remote_nr) {
> +               __kmem_cache_free_bulk(s, remote_nr, &remote_objects[0]);
> +               if (i < size) {
> +                       remote_nr = 0;
> +                       goto next_remote_batch;
> +               }
> +       }
>  }
>
>  #ifndef CONFIG_SLUB_TINY
> @@ -6115,8 +6139,14 @@ void slab_free(struct kmem_cache *s, struct slab *slab, void *object,
>         if (unlikely(!slab_free_hook(s, object, slab_want_init_on_free(s), false)))
>                 return;
>
> -       if (!s->cpu_sheaves || !free_to_pcs(s, object))
> -               do_slab_free(s, slab, object, object, 1, addr);
> +       if (s->cpu_sheaves && likely(!IS_ENABLED(CONFIG_NUMA) ||
> +                                    slab_nid(slab) == numa_mem_id())) {
> +               if (likely(free_to_pcs(s, object))) {

nit: no need for curly braces here.

> +                       return;
> +               }
> +       }
> +
> +       do_slab_free(s, slab, object, object, 1, addr);
>  }
>
>  #ifdef CONFIG_MEMCG
>
> --
> 2.51.0
>


From surenb at google.com  Thu Sep 25 09:27:29 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 25 Sep 2025 09:27:29 -0700
Subject: [PATCH v8 08/23] slab: allow NUMA restricted allocations to use
 percpu sheaves
In-Reply-To: <20250910-slub-percpu-caches-v8-8-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-8-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpEL1n+yVJDK0UWxP9CYgu28z6yLUzDWXBKyF0Oyukt1XA@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> Currently allocations asking for a specific node explicitly or via
> mempolicy in strict_numa node bypass percpu sheaves. Since sheaves
> contain mostly local objects, we can try allocating from them if the
> local node happens to be the requested node or allowed by the mempolicy.
> If we find the object from percpu sheaves is not from the expected node,
> we skip the sheaves - this should be rare.
>
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  mm/slub.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 46 insertions(+), 7 deletions(-)
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 9699d048b2cd08ee75c4cc3d1e460868704520b1..3746c0229cc2f9658a589416c63c21fbf2850c44 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -4888,18 +4888,43 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
>  }
>
>  static __fastpath_inline
> -void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
> +void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp, int node)
>  {
>         struct slub_percpu_sheaves *pcs;
> +       bool node_requested;
>         void *object;
>
>  #ifdef CONFIG_NUMA
> -       if (static_branch_unlikely(&strict_numa)) {
> -               if (current->mempolicy)
> -                       return NULL;
> +       if (static_branch_unlikely(&strict_numa) &&
> +                        node == NUMA_NO_NODE) {
> +
> +               struct mempolicy *mpol = current->mempolicy;
> +
> +               if (mpol) {
> +                       /*
> +                        * Special BIND rule support. If the local node
> +                        * is in permitted set then do not redirect
> +                        * to a particular node.
> +                        * Otherwise we apply the memory policy to get
> +                        * the node we need to allocate on.
> +                        */
> +                       if (mpol->mode != MPOL_BIND ||
> +                                       !node_isset(numa_mem_id(), mpol->nodes))
> +
> +                               node = mempolicy_slab_node();
> +               }
>         }
>  #endif
>
> +       node_requested = IS_ENABLED(CONFIG_NUMA) && node != NUMA_NO_NODE;
> +
> +       /*
> +        * We assume the percpu sheaves contain only local objects although it's
> +        * not completely guaranteed, so we verify later.
> +        */
> +       if (unlikely(node_requested && node != numa_mem_id()))
> +               return NULL;
> +
>         if (!local_trylock(&s->cpu_sheaves->lock))
>                 return NULL;
>
> @@ -4911,7 +4936,21 @@ void *alloc_from_pcs(struct kmem_cache *s, gfp_t gfp)
>                         return NULL;
>         }
>
> -       object = pcs->main->objects[--pcs->main->size];
> +       object = pcs->main->objects[pcs->main->size - 1];
> +
> +       if (unlikely(node_requested)) {
> +               /*
> +                * Verify that the object was from the node we want. This could
> +                * be false because of cpu migration during an unlocked part of
> +                * the current allocation or previous freeing process.
> +                */
> +               if (folio_nid(virt_to_folio(object)) != node) {
> +                       local_unlock(&s->cpu_sheaves->lock);
> +                       return NULL;
> +               }
> +       }
> +
> +       pcs->main->size--;
>
>         local_unlock(&s->cpu_sheaves->lock);
>
> @@ -5011,8 +5050,8 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list
>         if (unlikely(object))
>                 goto out;
>
> -       if (s->cpu_sheaves && node == NUMA_NO_NODE)
> -               object = alloc_from_pcs(s, gfpflags);
> +       if (s->cpu_sheaves)
> +               object = alloc_from_pcs(s, gfpflags, node);
>
>         if (!object)
>                 object = __slab_alloc_node(s, gfpflags, node, addr, orig_size);
>
> --
> 2.51.0
>


From surenb at google.com  Thu Sep 25 09:38:35 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 25 Sep 2025 09:38:35 -0700
Subject: [PATCH v8 11/23] maple_tree: Drop bulk insert support
In-Reply-To: <20250910-slub-percpu-caches-v8-11-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-11-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpG_3+D+dPN=kzYwSLJR76p6zCwMcA3gHVi1RhFPnEvKtg@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>
> Bulk insert mode was added to facilitate forking faster, but forking now
> uses __mt_dup() to duplicate the tree.
>
> The addition of sheaves has made the bulk allocations difficult to
> maintain - since the expected entries would preallocate into the maple
> state.  A big part of the maple state node allocation was the ability to
> push nodes back onto the state for later use, which was essential to the
> bulk insert algorithm.
>
> Remove mas_expected_entries() and mas_destroy_rebalance() functions as
> well as the MA_STATE_BULK and MA_STATE_REBALANCE maple state flags since
> there are no users anymore.  Drop the associated testing as well.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  lib/maple_tree.c                 | 270 +--------------------------------------
>  lib/test_maple_tree.c            | 137 --------------------
>  tools/testing/radix-tree/maple.c |  36 ------
>  3 files changed, 4 insertions(+), 439 deletions(-)

Awesome!

>
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index 38fb68c082915211c80f473d313159599fe97e2c..4f0e30b57b0cef9e5cf791f3f64f5898752db402 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -83,13 +83,9 @@
>
>  /*
>   * Maple state flags
> - * * MA_STATE_BULK             - Bulk insert mode
> - * * MA_STATE_REBALANCE                - Indicate a rebalance during bulk insert
>   * * MA_STATE_PREALLOC         - Preallocated nodes, WARN_ON allocation
>   */
> -#define MA_STATE_BULK          1
> -#define MA_STATE_REBALANCE     2
> -#define MA_STATE_PREALLOC      4
> +#define MA_STATE_PREALLOC      1
>
>  #define ma_parent_ptr(x) ((struct maple_pnode *)(x))
>  #define mas_tree_parent(x) ((unsigned long)(x->tree) | MA_ROOT_PARENT)
> @@ -1031,24 +1027,6 @@ static inline void mas_descend(struct ma_state *mas)
>         mas->node = mas_slot(mas, slots, mas->offset);
>  }
>
> -/*
> - * mte_set_gap() - Set a maple node gap.
> - * @mn: The encoded maple node
> - * @gap: The offset of the gap to set
> - * @val: The gap value
> - */
> -static inline void mte_set_gap(const struct maple_enode *mn,
> -                                unsigned char gap, unsigned long val)
> -{
> -       switch (mte_node_type(mn)) {
> -       default:
> -               break;
> -       case maple_arange_64:
> -               mte_to_node(mn)->ma64.gap[gap] = val;
> -               break;
> -       }
> -}
> -
>  /*
>   * mas_ascend() - Walk up a level of the tree.
>   * @mas: The maple state
> @@ -1878,21 +1856,7 @@ static inline int mab_calc_split(struct ma_state *mas,
>          * end on a NULL entry, with the exception of the left-most leaf.  The
>          * limitation means that the split of a node must be checked for this condition
>          * and be able to put more data in one direction or the other.
> -        */
> -       if (unlikely((mas->mas_flags & MA_STATE_BULK))) {
> -               *mid_split = 0;
> -               split = b_end - mt_min_slots[bn->type];
> -
> -               if (!ma_is_leaf(bn->type))
> -                       return split;
> -
> -               mas->mas_flags |= MA_STATE_REBALANCE;
> -               if (!bn->slot[split])
> -                       split--;
> -               return split;
> -       }
> -
> -       /*
> +        *
>          * Although extremely rare, it is possible to enter what is known as the 3-way
>          * split scenario.  The 3-way split comes about by means of a store of a range
>          * that overwrites the end and beginning of two full nodes.  The result is a set
> @@ -2039,27 +2003,6 @@ static inline void mab_mas_cp(struct maple_big_node *b_node,
>         }
>  }
>
> -/*
> - * mas_bulk_rebalance() - Rebalance the end of a tree after a bulk insert.
> - * @mas: The maple state
> - * @end: The maple node end
> - * @mt: The maple node type
> - */
> -static inline void mas_bulk_rebalance(struct ma_state *mas, unsigned char end,
> -                                     enum maple_type mt)
> -{
> -       if (!(mas->mas_flags & MA_STATE_BULK))
> -               return;
> -
> -       if (mte_is_root(mas->node))
> -               return;
> -
> -       if (end > mt_min_slots[mt]) {
> -               mas->mas_flags &= ~MA_STATE_REBALANCE;
> -               return;
> -       }
> -}
> -
>  /*
>   * mas_store_b_node() - Store an @entry into the b_node while also copying the
>   * data from a maple encoded node.
> @@ -2109,9 +2052,6 @@ static noinline_for_kasan void mas_store_b_node(struct ma_wr_state *wr_mas,
>         /* Handle new range ending before old range ends */
>         piv = mas_safe_pivot(mas, wr_mas->pivots, offset_end, wr_mas->type);
>         if (piv > mas->last) {
> -               if (piv == ULONG_MAX)
> -                       mas_bulk_rebalance(mas, b_node->b_end, wr_mas->type);
> -
>                 if (offset_end != slot)
>                         wr_mas->content = mas_slot_locked(mas, wr_mas->slots,
>                                                           offset_end);
> @@ -3011,126 +2951,6 @@ static inline void mas_rebalance(struct ma_state *mas,
>         return mas_spanning_rebalance(mas, &mast, empty_count);
>  }
>
> -/*
> - * mas_destroy_rebalance() - Rebalance left-most node while destroying the maple
> - * state.
> - * @mas: The maple state
> - * @end: The end of the left-most node.
> - *
> - * During a mass-insert event (such as forking), it may be necessary to
> - * rebalance the left-most node when it is not sufficient.
> - */
> -static inline void mas_destroy_rebalance(struct ma_state *mas, unsigned char end)
> -{
> -       enum maple_type mt = mte_node_type(mas->node);
> -       struct maple_node reuse, *newnode, *parent, *new_left, *left, *node;
> -       struct maple_enode *eparent, *old_eparent;
> -       unsigned char offset, tmp, split = mt_slots[mt] / 2;
> -       void __rcu **l_slots, **slots;
> -       unsigned long *l_pivs, *pivs, gap;
> -       bool in_rcu = mt_in_rcu(mas->tree);
> -       unsigned char new_height = mas_mt_height(mas);
> -
> -       MA_STATE(l_mas, mas->tree, mas->index, mas->last);
> -
> -       l_mas = *mas;
> -       mas_prev_sibling(&l_mas);
> -
> -       /* set up node. */
> -       if (in_rcu) {
> -               newnode = mas_pop_node(mas);
> -       } else {
> -               newnode = &reuse;
> -       }
> -
> -       node = mas_mn(mas);
> -       newnode->parent = node->parent;
> -       slots = ma_slots(newnode, mt);
> -       pivs = ma_pivots(newnode, mt);
> -       left = mas_mn(&l_mas);
> -       l_slots = ma_slots(left, mt);
> -       l_pivs = ma_pivots(left, mt);
> -       if (!l_slots[split])
> -               split++;
> -       tmp = mas_data_end(&l_mas) - split;
> -
> -       memcpy(slots, l_slots + split + 1, sizeof(void *) * tmp);
> -       memcpy(pivs, l_pivs + split + 1, sizeof(unsigned long) * tmp);
> -       pivs[tmp] = l_mas.max;
> -       memcpy(slots + tmp, ma_slots(node, mt), sizeof(void *) * end);
> -       memcpy(pivs + tmp, ma_pivots(node, mt), sizeof(unsigned long) * end);
> -
> -       l_mas.max = l_pivs[split];
> -       mas->min = l_mas.max + 1;
> -       old_eparent = mt_mk_node(mte_parent(l_mas.node),
> -                            mas_parent_type(&l_mas, l_mas.node));
> -       tmp += end;
> -       if (!in_rcu) {
> -               unsigned char max_p = mt_pivots[mt];
> -               unsigned char max_s = mt_slots[mt];
> -
> -               if (tmp < max_p)
> -                       memset(pivs + tmp, 0,
> -                              sizeof(unsigned long) * (max_p - tmp));
> -
> -               if (tmp < mt_slots[mt])
> -                       memset(slots + tmp, 0, sizeof(void *) * (max_s - tmp));
> -
> -               memcpy(node, newnode, sizeof(struct maple_node));
> -               ma_set_meta(node, mt, 0, tmp - 1);
> -               mte_set_pivot(old_eparent, mte_parent_slot(l_mas.node),
> -                             l_pivs[split]);
> -
> -               /* Remove data from l_pivs. */
> -               tmp = split + 1;
> -               memset(l_pivs + tmp, 0, sizeof(unsigned long) * (max_p - tmp));
> -               memset(l_slots + tmp, 0, sizeof(void *) * (max_s - tmp));
> -               ma_set_meta(left, mt, 0, split);
> -               eparent = old_eparent;
> -
> -               goto done;
> -       }
> -
> -       /* RCU requires replacing both l_mas, mas, and parent. */
> -       mas->node = mt_mk_node(newnode, mt);
> -       ma_set_meta(newnode, mt, 0, tmp);
> -
> -       new_left = mas_pop_node(mas);
> -       new_left->parent = left->parent;
> -       mt = mte_node_type(l_mas.node);
> -       slots = ma_slots(new_left, mt);
> -       pivs = ma_pivots(new_left, mt);
> -       memcpy(slots, l_slots, sizeof(void *) * split);
> -       memcpy(pivs, l_pivs, sizeof(unsigned long) * split);
> -       ma_set_meta(new_left, mt, 0, split);
> -       l_mas.node = mt_mk_node(new_left, mt);
> -
> -       /* replace parent. */
> -       offset = mte_parent_slot(mas->node);
> -       mt = mas_parent_type(&l_mas, l_mas.node);
> -       parent = mas_pop_node(mas);
> -       slots = ma_slots(parent, mt);
> -       pivs = ma_pivots(parent, mt);
> -       memcpy(parent, mte_to_node(old_eparent), sizeof(struct maple_node));
> -       rcu_assign_pointer(slots[offset], mas->node);
> -       rcu_assign_pointer(slots[offset - 1], l_mas.node);
> -       pivs[offset - 1] = l_mas.max;
> -       eparent = mt_mk_node(parent, mt);
> -done:
> -       gap = mas_leaf_max_gap(mas);
> -       mte_set_gap(eparent, mte_parent_slot(mas->node), gap);
> -       gap = mas_leaf_max_gap(&l_mas);
> -       mte_set_gap(eparent, mte_parent_slot(l_mas.node), gap);
> -       mas_ascend(mas);
> -
> -       if (in_rcu) {
> -               mas_replace_node(mas, old_eparent, new_height);
> -               mas_adopt_children(mas, mas->node);
> -       }
> -
> -       mas_update_gap(mas);
> -}
> -
>  /*
>   * mas_split_final_node() - Split the final node in a subtree operation.
>   * @mast: the maple subtree state
> @@ -3837,8 +3657,6 @@ static inline void mas_wr_node_store(struct ma_wr_state *wr_mas,
>
>         if (mas->last == wr_mas->end_piv)
>                 offset_end++; /* don't copy this offset */
> -       else if (unlikely(wr_mas->r_max == ULONG_MAX))
> -               mas_bulk_rebalance(mas, mas->end, wr_mas->type);
>
>         /* set up node. */
>         if (in_rcu) {
> @@ -4255,7 +4073,7 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
>         new_end = mas_wr_new_end(wr_mas);
>         /* Potential spanning rebalance collapsing a node */
>         if (new_end < mt_min_slots[wr_mas->type]) {
> -               if (!mte_is_root(mas->node) && !(mas->mas_flags & MA_STATE_BULK))
> +               if (!mte_is_root(mas->node))
>                         return  wr_rebalance;
>                 return wr_node_store;
>         }
> @@ -5562,25 +5380,7 @@ void mas_destroy(struct ma_state *mas)
>         struct maple_alloc *node;
>         unsigned long total;
>
> -       /*
> -        * When using mas_for_each() to insert an expected number of elements,
> -        * it is possible that the number inserted is less than the expected
> -        * number.  To fix an invalid final node, a check is performed here to
> -        * rebalance the previous node with the final node.
> -        */
> -       if (mas->mas_flags & MA_STATE_REBALANCE) {
> -               unsigned char end;
> -               if (mas_is_err(mas))
> -                       mas_reset(mas);
> -               mas_start(mas);
> -               mtree_range_walk(mas);
> -               end = mas->end + 1;
> -               if (end < mt_min_slot_count(mas->node) - 1)
> -                       mas_destroy_rebalance(mas, end);
> -
> -               mas->mas_flags &= ~MA_STATE_REBALANCE;
> -       }
> -       mas->mas_flags &= ~(MA_STATE_BULK|MA_STATE_PREALLOC);
> +       mas->mas_flags &= ~MA_STATE_PREALLOC;
>
>         total = mas_allocated(mas);
>         while (total) {
> @@ -5600,68 +5400,6 @@ void mas_destroy(struct ma_state *mas)
>  }
>  EXPORT_SYMBOL_GPL(mas_destroy);
>
> -/*
> - * mas_expected_entries() - Set the expected number of entries that will be inserted.
> - * @mas: The maple state
> - * @nr_entries: The number of expected entries.
> - *
> - * This will attempt to pre-allocate enough nodes to store the expected number
> - * of entries.  The allocations will occur using the bulk allocator interface
> - * for speed.  Please call mas_destroy() on the @mas after inserting the entries
> - * to ensure any unused nodes are freed.
> - *
> - * Return: 0 on success, -ENOMEM if memory could not be allocated.
> - */
> -int mas_expected_entries(struct ma_state *mas, unsigned long nr_entries)
> -{
> -       int nonleaf_cap = MAPLE_ARANGE64_SLOTS - 2;
> -       struct maple_enode *enode = mas->node;
> -       int nr_nodes;
> -       int ret;
> -
> -       /*
> -        * Sometimes it is necessary to duplicate a tree to a new tree, such as
> -        * forking a process and duplicating the VMAs from one tree to a new
> -        * tree.  When such a situation arises, it is known that the new tree is
> -        * not going to be used until the entire tree is populated.  For
> -        * performance reasons, it is best to use a bulk load with RCU disabled.
> -        * This allows for optimistic splitting that favours the left and reuse
> -        * of nodes during the operation.
> -        */
> -
> -       /* Optimize splitting for bulk insert in-order */
> -       mas->mas_flags |= MA_STATE_BULK;
> -
> -       /*
> -        * Avoid overflow, assume a gap between each entry and a trailing null.
> -        * If this is wrong, it just means allocation can happen during
> -        * insertion of entries.
> -        */
> -       nr_nodes = max(nr_entries, nr_entries * 2 + 1);
> -       if (!mt_is_alloc(mas->tree))
> -               nonleaf_cap = MAPLE_RANGE64_SLOTS - 2;
> -
> -       /* Leaves; reduce slots to keep space for expansion */
> -       nr_nodes = DIV_ROUND_UP(nr_nodes, MAPLE_RANGE64_SLOTS - 2);
> -       /* Internal nodes */
> -       nr_nodes += DIV_ROUND_UP(nr_nodes, nonleaf_cap);
> -       /* Add working room for split (2 nodes) + new parents */
> -       mas_node_count_gfp(mas, nr_nodes + 3, GFP_KERNEL);
> -
> -       /* Detect if allocations run out */
> -       mas->mas_flags |= MA_STATE_PREALLOC;
> -
> -       if (!mas_is_err(mas))
> -               return 0;
> -
> -       ret = xa_err(mas->node);
> -       mas->node = enode;
> -       mas_destroy(mas);
> -       return ret;
> -
> -}
> -EXPORT_SYMBOL_GPL(mas_expected_entries);
> -
>  static void mas_may_activate(struct ma_state *mas)
>  {
>         if (!mas->node) {
> diff --git a/lib/test_maple_tree.c b/lib/test_maple_tree.c
> index cb3936595b0d56a9682ff100eba54693a1427829..14fbbee32046a13d54d60dcac2b45be2bd190ac4 100644
> --- a/lib/test_maple_tree.c
> +++ b/lib/test_maple_tree.c
> @@ -2746,139 +2746,6 @@ static noinline void __init check_fuzzer(struct maple_tree *mt)
>         mtree_test_erase(mt, ULONG_MAX - 10);
>  }
>
> -/* duplicate the tree with a specific gap */
> -static noinline void __init check_dup_gaps(struct maple_tree *mt,
> -                                   unsigned long nr_entries, bool zero_start,
> -                                   unsigned long gap)
> -{
> -       unsigned long i = 0;
> -       struct maple_tree newmt;
> -       int ret;
> -       void *tmp;
> -       MA_STATE(mas, mt, 0, 0);
> -       MA_STATE(newmas, &newmt, 0, 0);
> -       struct rw_semaphore newmt_lock;
> -
> -       init_rwsem(&newmt_lock);
> -       mt_set_external_lock(&newmt, &newmt_lock);
> -
> -       if (!zero_start)
> -               i = 1;
> -
> -       mt_zero_nr_tallocated();
> -       for (; i <= nr_entries; i++)
> -               mtree_store_range(mt, i*10, (i+1)*10 - gap,
> -                                 xa_mk_value(i), GFP_KERNEL);
> -
> -       mt_init_flags(&newmt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN);
> -       mt_set_non_kernel(99999);
> -       down_write(&newmt_lock);
> -       ret = mas_expected_entries(&newmas, nr_entries);
> -       mt_set_non_kernel(0);
> -       MT_BUG_ON(mt, ret != 0);
> -
> -       rcu_read_lock();
> -       mas_for_each(&mas, tmp, ULONG_MAX) {
> -               newmas.index = mas.index;
> -               newmas.last = mas.last;
> -               mas_store(&newmas, tmp);
> -       }
> -       rcu_read_unlock();
> -       mas_destroy(&newmas);
> -
> -       __mt_destroy(&newmt);
> -       up_write(&newmt_lock);
> -}
> -
> -/* Duplicate many sizes of trees.  Mainly to test expected entry values */
> -static noinline void __init check_dup(struct maple_tree *mt)
> -{
> -       int i;
> -       int big_start = 100010;
> -
> -       /* Check with a value at zero */
> -       for (i = 10; i < 1000; i++) {
> -               mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -               check_dup_gaps(mt, i, true, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -       }
> -
> -       cond_resched();
> -       mt_cache_shrink();
> -       /* Check with a value at zero, no gap */
> -       for (i = 1000; i < 2000; i++) {
> -               mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -               check_dup_gaps(mt, i, true, 0);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -       }
> -
> -       cond_resched();
> -       mt_cache_shrink();
> -       /* Check with a value at zero and unreasonably large */
> -       for (i = big_start; i < big_start + 10; i++) {
> -               mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -               check_dup_gaps(mt, i, true, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -       }
> -
> -       cond_resched();
> -       mt_cache_shrink();
> -       /* Small to medium size not starting at zero*/
> -       for (i = 200; i < 1000; i++) {
> -               mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -               check_dup_gaps(mt, i, false, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -       }
> -
> -       cond_resched();
> -       mt_cache_shrink();
> -       /* Unreasonably large not starting at zero*/
> -       for (i = big_start; i < big_start + 10; i++) {
> -               mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -               check_dup_gaps(mt, i, false, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -               cond_resched();
> -               mt_cache_shrink();
> -       }
> -
> -       /* Check non-allocation tree not starting at zero */
> -       for (i = 1500; i < 3000; i++) {
> -               mt_init_flags(mt, 0);
> -               check_dup_gaps(mt, i, false, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -               cond_resched();
> -               if (i % 2 == 0)
> -                       mt_cache_shrink();
> -       }
> -
> -       mt_cache_shrink();
> -       /* Check non-allocation tree starting at zero */
> -       for (i = 200; i < 1000; i++) {
> -               mt_init_flags(mt, 0);
> -               check_dup_gaps(mt, i, true, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -               cond_resched();
> -       }
> -
> -       mt_cache_shrink();
> -       /* Unreasonably large */
> -       for (i = big_start + 5; i < big_start + 10; i++) {
> -               mt_init_flags(mt, 0);
> -               check_dup_gaps(mt, i, true, 5);
> -               mtree_destroy(mt);
> -               rcu_barrier();
> -               mt_cache_shrink();
> -               cond_resched();
> -       }
> -}
> -
>  static noinline void __init check_bnode_min_spanning(struct maple_tree *mt)
>  {
>         int i = 50;
> @@ -4077,10 +3944,6 @@ static int __init maple_tree_seed(void)
>         check_fuzzer(&tree);
>         mtree_destroy(&tree);
>
> -       mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
> -       check_dup(&tree);
> -       mtree_destroy(&tree);
> -
>         mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
>         check_bnode_min_spanning(&tree);
>         mtree_destroy(&tree);
> diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
> index 172700fb7784d29f9403003b4484a5ebd7aa316b..c0543060dae2510477963331fb0ccdffd78ea965 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -35455,17 +35455,6 @@ static void check_dfs_preorder(struct maple_tree *mt)
>         MT_BUG_ON(mt, count != e);
>         mtree_destroy(mt);
>
> -       mt_init_flags(mt, MT_FLAGS_ALLOC_RANGE);
> -       mas_reset(&mas);
> -       mt_zero_nr_tallocated();
> -       mt_set_non_kernel(200);
> -       mas_expected_entries(&mas, max);
> -       for (count = 0; count <= max; count++) {
> -               mas.index = mas.last = count;
> -               mas_store(&mas, xa_mk_value(count));
> -               MT_BUG_ON(mt, mas_is_err(&mas));
> -       }
> -       mas_destroy(&mas);
>         rcu_barrier();
>         /*
>          * pr_info(" ->seq test of 0-%lu %luK in %d active (%d total)\n",
> @@ -36454,27 +36443,6 @@ static inline int check_vma_modification(struct maple_tree *mt)
>         return 0;
>  }
>
> -/*
> - * test to check that bulk stores do not use wr_rebalance as the store
> - * type.
> - */
> -static inline void check_bulk_rebalance(struct maple_tree *mt)
> -{
> -       MA_STATE(mas, mt, ULONG_MAX, ULONG_MAX);
> -       int max = 10;
> -
> -       build_full_tree(mt, 0, 2);
> -
> -       /* erase every entry in the tree */
> -       do {
> -               /* set up bulk store mode */
> -               mas_expected_entries(&mas, max);
> -               mas_erase(&mas);
> -               MT_BUG_ON(mt, mas.store_type == wr_rebalance);
> -       } while (mas_prev(&mas, 0) != NULL);
> -
> -       mas_destroy(&mas);
> -}
>
>  void farmer_tests(void)
>  {
> @@ -36487,10 +36455,6 @@ void farmer_tests(void)
>         check_vma_modification(&tree);
>         mtree_destroy(&tree);
>
> -       mt_init(&tree);
> -       check_bulk_rebalance(&tree);
> -       mtree_destroy(&tree);
> -
>         tree.ma_root = xa_mk_value(0);
>         mt_dump(&tree, mt_dump_dec);
>
>
> --
> 2.51.0
>


From surenb at google.com  Thu Sep 25 09:38:47 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Thu, 25 Sep 2025 09:38:47 -0700
Subject: [PATCH v8 12/23] tools/testing/vma: Implement vm_refcnt reset
In-Reply-To: <20250910-slub-percpu-caches-v8-12-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-12-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpEBOAGN+hLssuAma473B5-m-cA4CN3xEyjoM8WXcQv-6Q@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>
> Add the reset of the ref count in vma_lock_init().  This is needed if
> the vma memory is not zeroed on allocation.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>



> ---
>  tools/testing/vma/vma_internal.h | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index f8cf5b184d5b51dd627ff440943a7af3c549f482..6b6e2b05918c9f95b537f26e20a943b34082825a 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -1373,6 +1373,8 @@ static inline void ksm_exit(struct mm_struct *mm)
>
>  static inline void vma_lock_init(struct vm_area_struct *vma, bool reset_refcnt)
>  {
> +       if (reset_refcnt)
> +               refcount_set(&vma->vm_refcnt, 0);
>  }
>
>  static inline void vma_numab_state_init(struct vm_area_struct *vma)
>
> --
> 2.51.0
>


From vbabka at suse.cz  Fri Sep 26 03:08:26 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Fri, 26 Sep 2025 12:08:26 +0200
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu()
 operations
In-Reply-To: <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz>
 <aMpxnACqmsQl-lp0@hyeyoo> <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz>
 <aMqcXyKRlZggLxu_@hyeyoo> <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz>
 <aMqyn6nenR8V_p1y@hyeyoo> <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz>
 <aMq40h5iOjj8K7cc@hyeyoo> <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz>
 <40461105-a344-444f-834b-9559b6644710@suse.cz>
 <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com>
Message-ID: <80f8bdf8-ce50-4682-9468-9f38e0817e89@suse.cz>

On 9/25/25 06:35, Suren Baghdasaryan wrote:
> On Thu, Sep 18, 2025 at 1:09?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>>
>> Expected advantages:
>> - batching the kfree_rcu() operations, that could eventually replace the
>>   existing batching
>> - sheaves can be reused for allocations via barn instead of being
>>   flushed to slabs, which is more efficient
>>   - this includes cases where only some cpus are allowed to process rcu
>>     callbacks (Android)
> 
> nit: I would say it's more CONFIG_RCU_NOCB_CPU related. Android is
> just an instance of that.

OK changed that.

Changes due to your other suggestions:

diff --git a/mm/slub.c b/mm/slub.c
index 8220ce095970..fec0cdc7ef37 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3945,15 +3945,12 @@ void flush_all_rcu_sheaves(void)
                         */
 
                        INIT_WORK(&sfw->work, flush_rcu_sheaf);
-                       sfw->skip = false;
                        sfw->s = s;
                        queue_work_on(cpu, flushwq, &sfw->work);
                }
 
                for_each_online_cpu(cpu) {
                        sfw = &per_cpu(slub_flush, cpu);
-                       if (sfw->skip)
-                               continue;
                        flush_work(&sfw->work);
                }
 
@@ -5643,6 +5640,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
 
        rcu_sheaf = pcs->rcu_free;
 
+       /*
+        * Since we flush immediately when size reaches capacity, we never reach
+        * this with size already at capacity, so no OOB write is possible.
+        */
        rcu_sheaf->objects[rcu_sheaf->size++] = obj;
 
        if (likely(rcu_sheaf->size < s->sheaf_capacity))



From surenb at google.com  Fri Sep 26 08:41:45 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 08:41:45 -0700
Subject: [PATCH v8 04/23] slab: add sheaf support for batching kfree_rcu() operations
In-Reply-To: <80f8bdf8-ce50-4682-9468-9f38e0817e89@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-4-ca3099d8352c@suse.cz> <aMpxnACqmsQl-lp0@hyeyoo>
 <bbda8c25-b575-4e98-b1ae-b103c6598d38@suse.cz> <aMqcXyKRlZggLxu_@hyeyoo>
 <6f92eca3-863e-4b77-b2df-dc2752c0ff4e@suse.cz> <aMqyn6nenR8V_p1y@hyeyoo>
 <bf8b4741-1025-4cc8-b9ed-5b566bdcb2d7@suse.cz> <aMq40h5iOjj8K7cc@hyeyoo>
 <a32bd837-2597-43d0-9da3-1ce5a53b15f4@suse.cz> <40461105-a344-444f-834b-9559b6644710@suse.cz>
 <CAJuCfpG6CSm2iZ3jrwLQA4vVbTMvC=B37q10OL+wLzm-DSRhFw@mail.gmail.com> <80f8bdf8-ce50-4682-9468-9f38e0817e89@suse.cz>
Message-ID: <CAJuCfpG4JYPE0YbzcLCZbtsqkqCrzaRENkBunpvzLC_B5q2vCA@mail.gmail.com>

On Fri, Sep 26, 2025 at 3:08?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> On 9/25/25 06:35, Suren Baghdasaryan wrote:
> > On Thu, Sep 18, 2025 at 1:09?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> >>
> >> Expected advantages:
> >> - batching the kfree_rcu() operations, that could eventually replace the
> >>   existing batching
> >> - sheaves can be reused for allocations via barn instead of being
> >>   flushed to slabs, which is more efficient
> >>   - this includes cases where only some cpus are allowed to process rcu
> >>     callbacks (Android)
> >
> > nit: I would say it's more CONFIG_RCU_NOCB_CPU related. Android is
> > just an instance of that.
>
> OK changed that.
>
> Changes due to your other suggestions:
>
> diff --git a/mm/slub.c b/mm/slub.c
> index 8220ce095970..fec0cdc7ef37 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3945,15 +3945,12 @@ void flush_all_rcu_sheaves(void)
>                          */
>
>                         INIT_WORK(&sfw->work, flush_rcu_sheaf);
> -                       sfw->skip = false;
>                         sfw->s = s;
>                         queue_work_on(cpu, flushwq, &sfw->work);
>                 }
>
>                 for_each_online_cpu(cpu) {
>                         sfw = &per_cpu(slub_flush, cpu);
> -                       if (sfw->skip)
> -                               continue;
>                         flush_work(&sfw->work);
>                 }
>
> @@ -5643,6 +5640,10 @@ bool __kfree_rcu_sheaf(struct kmem_cache *s, void *obj)
>
>         rcu_sheaf = pcs->rcu_free;
>
> +       /*
> +        * Since we flush immediately when size reaches capacity, we never reach
> +        * this with size already at capacity, so no OOB write is possible.
> +        */

Perfect!

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

>         rcu_sheaf->objects[rcu_sheaf->size++] = obj;
>
>         if (likely(rcu_sheaf->size < s->sheaf_capacity))
>


From surenb at google.com  Fri Sep 26 16:28:43 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 16:28:43 -0700
Subject: [PATCH v8 13/23] tools/testing: Add support for changes to slab
 for sheaves
In-Reply-To: <20250910-slub-percpu-caches-v8-13-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-13-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpGYgOVG7hpZJ7MED_ALXZfKgUuRcPN3YYjx6k_t-dGXhg@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>
>
> The slab changes for sheaves requires more effort in the testing code.
> Unite all the kmem_cache work into the tools/include slab header for
> both the vma and maple tree testing.
>
> The vma test code also requires importing more #defines to allow for
> seamless use of the shared kmem_cache code.
>
> This adds the pthread header to the slab header in the tools directory
> to allow for the pthread_mutex in linux.c.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

The patch does several things and could be split in 3 (code
refactoring, kmem_cache_create change, new definitions like
_slab_flag_bits) but I don't think it's worth a respin.

> ---
>  tools/include/linux/slab.h        | 137 ++++++++++++++++++++++++++++++++++++--
>  tools/testing/shared/linux.c      |  26 ++------
>  tools/testing/shared/maple-shim.c |   1 +
>  tools/testing/vma/vma_internal.h  |  92 +------------------------
>  4 files changed, 142 insertions(+), 114 deletions(-)
>
> diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
> index c87051e2b26f5a7fee0362697fae067076b8e84d..c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08 100644
> --- a/tools/include/linux/slab.h
> +++ b/tools/include/linux/slab.h
> @@ -4,11 +4,31 @@
>
>  #include <linux/types.h>
>  #include <linux/gfp.h>
> +#include <pthread.h>
>
> -#define SLAB_PANIC 2
>  #define SLAB_RECLAIM_ACCOUNT    0x00020000UL            /* Objects are reclaimable */
>
>  #define kzalloc_node(size, flags, node) kmalloc(size, flags)
> +enum _slab_flag_bits {
> +       _SLAB_KMALLOC,
> +       _SLAB_HWCACHE_ALIGN,
> +       _SLAB_PANIC,
> +       _SLAB_TYPESAFE_BY_RCU,
> +       _SLAB_ACCOUNT,
> +       _SLAB_FLAGS_LAST_BIT
> +};
> +
> +#define __SLAB_FLAG_BIT(nr)    ((unsigned int __force)(1U << (nr)))
> +#define __SLAB_FLAG_UNUSED     ((unsigned int __force)(0U))
> +
> +#define SLAB_HWCACHE_ALIGN     __SLAB_FLAG_BIT(_SLAB_HWCACHE_ALIGN)
> +#define SLAB_PANIC             __SLAB_FLAG_BIT(_SLAB_PANIC)
> +#define SLAB_TYPESAFE_BY_RCU   __SLAB_FLAG_BIT(_SLAB_TYPESAFE_BY_RCU)
> +#ifdef CONFIG_MEMCG
> +# define SLAB_ACCOUNT          __SLAB_FLAG_BIT(_SLAB_ACCOUNT)
> +#else
> +# define SLAB_ACCOUNT          __SLAB_FLAG_UNUSED
> +#endif
>
>  void *kmalloc(size_t size, gfp_t gfp);
>  void kfree(void *p);
> @@ -23,6 +43,86 @@ enum slab_state {
>         FULL
>  };
>
> +struct kmem_cache {
> +       pthread_mutex_t lock;
> +       unsigned int size;
> +       unsigned int align;
> +       unsigned int sheaf_capacity;
> +       int nr_objs;
> +       void *objs;
> +       void (*ctor)(void *);
> +       bool non_kernel_enabled;
> +       unsigned int non_kernel;
> +       unsigned long nr_allocated;
> +       unsigned long nr_tallocated;
> +       bool exec_callback;
> +       void (*callback)(void *);
> +       void *private;
> +};
> +
> +struct kmem_cache_args {
> +       /**
> +        * @align: The required alignment for the objects.
> +        *
> +        * %0 means no specific alignment is requested.
> +        */
> +       unsigned int align;
> +       /**
> +        * @sheaf_capacity: The maximum size of the sheaf.
> +        */
> +       unsigned int sheaf_capacity;
> +       /**
> +        * @useroffset: Usercopy region offset.
> +        *
> +        * %0 is a valid offset, when @usersize is non-%0
> +        */
> +       unsigned int useroffset;
> +       /**
> +        * @usersize: Usercopy region size.
> +        *
> +        * %0 means no usercopy region is specified.
> +        */
> +       unsigned int usersize;
> +       /**
> +        * @freeptr_offset: Custom offset for the free pointer
> +        * in &SLAB_TYPESAFE_BY_RCU caches
> +        *
> +        * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
> +        * outside of the object. This might cause the object to grow in size.
> +        * Cache creators that have a reason to avoid this can specify a custom
> +        * free pointer offset in their struct where the free pointer will be
> +        * placed.
> +        *
> +        * Note that placing the free pointer inside the object requires the
> +        * caller to ensure that no fields are invalidated that are required to
> +        * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
> +        * details).
> +        *
> +        * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
> +        * is specified, %use_freeptr_offset must be set %true.
> +        *
> +        * Note that @ctor currently isn't supported with custom free pointers
> +        * as a @ctor requires an external free pointer.
> +        */
> +       unsigned int freeptr_offset;
> +       /**
> +        * @use_freeptr_offset: Whether a @freeptr_offset is used.
> +        */
> +       bool use_freeptr_offset;
> +       /**
> +        * @ctor: A constructor for the objects.
> +        *
> +        * The constructor is invoked for each object in a newly allocated slab
> +        * page. It is the cache user's responsibility to free object in the
> +        * same state as after calling the constructor, or deal appropriately
> +        * with any differences between a freshly constructed and a reallocated
> +        * object.
> +        *
> +        * %NULL means no constructor.
> +        */
> +       void (*ctor)(void *);
> +};
> +
>  static inline void *kzalloc(size_t size, gfp_t gfp)
>  {
>         return kmalloc(size, gfp | __GFP_ZERO);
> @@ -37,9 +137,38 @@ static inline void *kmem_cache_alloc(struct kmem_cache *cachep, int flags)
>  }
>  void kmem_cache_free(struct kmem_cache *cachep, void *objp);
>
> -struct kmem_cache *kmem_cache_create(const char *name, unsigned int size,
> -                       unsigned int align, unsigned int flags,
> -                       void (*ctor)(void *));
> +
> +struct kmem_cache *
> +__kmem_cache_create_args(const char *name, unsigned int size,
> +               struct kmem_cache_args *args, unsigned int flags);
> +
> +/* If NULL is passed for @args, use this variant with default arguments. */
> +static inline struct kmem_cache *
> +__kmem_cache_default_args(const char *name, unsigned int size,
> +               struct kmem_cache_args *args, unsigned int flags)
> +{
> +       struct kmem_cache_args kmem_default_args = {};
> +
> +       return __kmem_cache_create_args(name, size, &kmem_default_args, flags);
> +}
> +
> +static inline struct kmem_cache *
> +__kmem_cache_create(const char *name, unsigned int size, unsigned int align,
> +               unsigned int flags, void (*ctor)(void *))
> +{
> +       struct kmem_cache_args kmem_args = {
> +               .align  = align,
> +               .ctor   = ctor,
> +       };
> +
> +       return __kmem_cache_create_args(name, size, &kmem_args, flags);
> +}
> +
> +#define kmem_cache_create(__name, __object_size, __args, ...)           \
> +       _Generic((__args),                                              \
> +               struct kmem_cache_args *: __kmem_cache_create_args,     \
> +               void *: __kmem_cache_default_args,                      \
> +               default: __kmem_cache_create)(__name, __object_size, __args, __VA_ARGS__)
>
>  void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
>  int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
> diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
> index 0f97fb0d19e19c327aa4843a35b45cc086f4f366..97b8412ccbb6d222604c7b397c53c65618d8d51b 100644
> --- a/tools/testing/shared/linux.c
> +++ b/tools/testing/shared/linux.c
> @@ -16,21 +16,6 @@ int nr_allocated;
>  int preempt_count;
>  int test_verbose;
>
> -struct kmem_cache {
> -       pthread_mutex_t lock;
> -       unsigned int size;
> -       unsigned int align;
> -       int nr_objs;
> -       void *objs;
> -       void (*ctor)(void *);
> -       unsigned int non_kernel;
> -       unsigned long nr_allocated;
> -       unsigned long nr_tallocated;
> -       bool exec_callback;
> -       void (*callback)(void *);
> -       void *private;
> -};
> -
>  void kmem_cache_set_callback(struct kmem_cache *cachep, void (*callback)(void *))
>  {
>         cachep->callback = callback;
> @@ -234,23 +219,26 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
>  }
>
>  struct kmem_cache *
> -kmem_cache_create(const char *name, unsigned int size, unsigned int align,
> -               unsigned int flags, void (*ctor)(void *))
> +__kmem_cache_create_args(const char *name, unsigned int size,
> +                         struct kmem_cache_args *args,
> +                         unsigned int flags)
>  {
>         struct kmem_cache *ret = malloc(sizeof(*ret));
>
>         pthread_mutex_init(&ret->lock, NULL);
>         ret->size = size;
> -       ret->align = align;
> +       ret->align = args->align;
> +       ret->sheaf_capacity = args->sheaf_capacity;
>         ret->nr_objs = 0;
>         ret->nr_allocated = 0;
>         ret->nr_tallocated = 0;
>         ret->objs = NULL;
> -       ret->ctor = ctor;
> +       ret->ctor = args->ctor;
>         ret->non_kernel = 0;
>         ret->exec_callback = false;
>         ret->callback = NULL;
>         ret->private = NULL;
> +
>         return ret;
>  }
>
> diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
> index 640df76f483e09f3b6f85612786060dd273e2362..9d7b743415660305416e972fa75b56824211b0eb 100644
> --- a/tools/testing/shared/maple-shim.c
> +++ b/tools/testing/shared/maple-shim.c
> @@ -3,5 +3,6 @@
>  /* Very simple shim around the maple tree. */
>
>  #include "maple-shared.h"
> +#include <linux/slab.h>
>
>  #include "../../../lib/maple_tree.c"
> diff --git a/tools/testing/vma/vma_internal.h b/tools/testing/vma/vma_internal.h
> index 6b6e2b05918c9f95b537f26e20a943b34082825a..d5b87fa6a133f6d676488de2538c509e0f0e1d54 100644
> --- a/tools/testing/vma/vma_internal.h
> +++ b/tools/testing/vma/vma_internal.h
> @@ -26,6 +26,7 @@
>  #include <linux/mm.h>
>  #include <linux/rbtree.h>
>  #include <linux/refcount.h>
> +#include <linux/slab.h>
>
>  extern unsigned long stack_guard_gap;
>  #ifdef CONFIG_MMU
> @@ -509,65 +510,6 @@ struct pagetable_move_control {
>                 .len_in = len_,                                         \
>         }
>
> -struct kmem_cache_args {
> -       /**
> -        * @align: The required alignment for the objects.
> -        *
> -        * %0 means no specific alignment is requested.
> -        */
> -       unsigned int align;
> -       /**
> -        * @useroffset: Usercopy region offset.
> -        *
> -        * %0 is a valid offset, when @usersize is non-%0
> -        */
> -       unsigned int useroffset;
> -       /**
> -        * @usersize: Usercopy region size.
> -        *
> -        * %0 means no usercopy region is specified.
> -        */
> -       unsigned int usersize;
> -       /**
> -        * @freeptr_offset: Custom offset for the free pointer
> -        * in &SLAB_TYPESAFE_BY_RCU caches
> -        *
> -        * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
> -        * outside of the object. This might cause the object to grow in size.
> -        * Cache creators that have a reason to avoid this can specify a custom
> -        * free pointer offset in their struct where the free pointer will be
> -        * placed.
> -        *
> -        * Note that placing the free pointer inside the object requires the
> -        * caller to ensure that no fields are invalidated that are required to
> -        * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
> -        * details).
> -        *
> -        * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
> -        * is specified, %use_freeptr_offset must be set %true.
> -        *
> -        * Note that @ctor currently isn't supported with custom free pointers
> -        * as a @ctor requires an external free pointer.
> -        */
> -       unsigned int freeptr_offset;
> -       /**
> -        * @use_freeptr_offset: Whether a @freeptr_offset is used.
> -        */
> -       bool use_freeptr_offset;
> -       /**
> -        * @ctor: A constructor for the objects.
> -        *
> -        * The constructor is invoked for each object in a newly allocated slab
> -        * page. It is the cache user's responsibility to free object in the
> -        * same state as after calling the constructor, or deal appropriately
> -        * with any differences between a freshly constructed and a reallocated
> -        * object.
> -        *
> -        * %NULL means no constructor.
> -        */
> -       void (*ctor)(void *);
> -};
> -
>  static inline void vma_iter_invalidate(struct vma_iterator *vmi)
>  {
>         mas_pause(&vmi->mas);
> @@ -652,38 +594,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
>         vma->vm_lock_seq = UINT_MAX;
>  }
>
> -struct kmem_cache {
> -       const char *name;
> -       size_t object_size;
> -       struct kmem_cache_args *args;
> -};
> -
> -static inline struct kmem_cache *__kmem_cache_create(const char *name,
> -                                                    size_t object_size,
> -                                                    struct kmem_cache_args *args)
> -{
> -       struct kmem_cache *ret = malloc(sizeof(struct kmem_cache));
> -
> -       ret->name = name;
> -       ret->object_size = object_size;
> -       ret->args = args;
> -
> -       return ret;
> -}
> -
> -#define kmem_cache_create(__name, __object_size, __args, ...)           \
> -       __kmem_cache_create((__name), (__object_size), (__args))
> -
> -static inline void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
> -{
> -       return calloc(1, s->object_size);
> -}
> -
> -static inline void kmem_cache_free(struct kmem_cache *s, void *x)
> -{
> -       free(x);
> -}
> -
>  /*
>   * These are defined in vma.h, but sadly vm_stat_account() is referenced by
>   * kernel/fork.c, so we have to these broadly available there, and temporarily
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 16:45:30 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 16:45:30 -0700
Subject: [PATCH v8 16/23] tools/testing: include maple-shim.c in maple.c
In-Reply-To: <20250910-slub-percpu-caches-v8-16-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-16-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpF8TDS0bjksssRJyvxnr1Rbb2SpgRfRAxJmPS5oQz88dg@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> There's some duplicated code and we are about to add more functionality
> in maple-shared.h that we will need in the userspace maple test to be
> available, so include it via maple-shim.c
>
> Co-developed-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  tools/testing/radix-tree/maple.c | 12 +++---------
>  1 file changed, 3 insertions(+), 9 deletions(-)
>
> diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
> index c0543060dae2510477963331fb0ccdffd78ea965..4a35e1e7c64b7ce347cbd1693beeaacb0c4c330e 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -8,14 +8,6 @@
>   * difficult to handle in kernel tests.
>   */
>
> -#define CONFIG_DEBUG_MAPLE_TREE
> -#define CONFIG_MAPLE_SEARCH
> -#define MAPLE_32BIT (MAPLE_NODE_SLOTS > 31)
> -#include "test.h"
> -#include <stdlib.h>
> -#include <time.h>
> -#include <linux/init.h>
> -
>  #define module_init(x)
>  #define module_exit(x)
>  #define MODULE_AUTHOR(x)
> @@ -23,7 +15,9 @@
>  #define MODULE_LICENSE(x)
>  #define dump_stack()   assert(0)
>
> -#include "../../../lib/maple_tree.c"
> +#include "test.h"
> +
> +#include "../shared/maple-shim.c"
>  #include "../../../lib/test_maple_tree.c"
>
>  #define RCU_RANGE_COUNT 1000
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 16:53:43 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 16:53:43 -0700
Subject: [PATCH v8 17/23] testing/radix-tree/maple: Hack around kfree_rcu
 not existing
In-Reply-To: <20250910-slub-percpu-caches-v8-17-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-17-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpFrFn01Dx88vYaDnByRraE3qPvO4rfMaortgwYyVsybZg@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>
> liburcu doesn't have kfree_rcu (or anything similar). Despite that, we
> can hack around it in a trivial fashion, by adding a wrapper.
>
> The wrapper only works for maple_nodes because we cannot get the
> kmem_cache pointer any other way in the test code.
>
> Link: https://lore.kernel.org/all/20250812162124.59417-1-pfalcato at suse.de/
> Suggested-by: Pedro Falcato <pfalcato at suse.de>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

With one nit below:

> ---
>  tools/testing/shared/maple-shared.h | 11 +++++++++++
>  tools/testing/shared/maple-shim.c   |  6 ++++++
>  2 files changed, 17 insertions(+)
>
> diff --git a/tools/testing/shared/maple-shared.h b/tools/testing/shared/maple-shared.h
> index dc4d30f3860b9bd23b4177c7d7926ac686887815..2a1e9a8594a2834326cd9374738b2a2c7c3f9f7c 100644
> --- a/tools/testing/shared/maple-shared.h
> +++ b/tools/testing/shared/maple-shared.h
> @@ -10,4 +10,15 @@
>  #include <time.h>
>  #include "linux/init.h"
>
> +void maple_rcu_cb(struct rcu_head *head);
> +#define rcu_cb         maple_rcu_cb
> +
> +#define kfree_rcu(_struct, _memb)              \
> +do {                                            \
> +    typeof(_struct) _p_struct = (_struct);      \

Maybe add an assertion that (typeof(_struct) == typeof(struct
maple_node)) to make sure kfree_rcu() is not used for anything else in
the tests?

> +                                                \
> +    call_rcu(&((_p_struct)->_memb), rcu_cb);    \
> +} while(0);
> +
> +
>  #endif /* __MAPLE_SHARED_H__ */
> diff --git a/tools/testing/shared/maple-shim.c b/tools/testing/shared/maple-shim.c
> index 9d7b743415660305416e972fa75b56824211b0eb..16252ee616c0489c80490ff25b8d255427bf9fdc 100644
> --- a/tools/testing/shared/maple-shim.c
> +++ b/tools/testing/shared/maple-shim.c
> @@ -6,3 +6,9 @@
>  #include <linux/slab.h>
>
>  #include "../../../lib/maple_tree.c"
> +
> +void maple_rcu_cb(struct rcu_head *head) {
> +       struct maple_node *node = container_of(head, struct maple_node, rcu);
> +
> +       kmem_cache_free(maple_node_cache, node);
> +}
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 17:05:19 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 17:05:19 -0700
Subject: [PATCH v8 18/23] maple_tree: Use kfree_rcu in ma_free_rcu
In-Reply-To: <aMqfmzwCu1N_L1qp@hyeyoo>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-18-ca3099d8352c@suse.cz> <aMqfmzwCu1N_L1qp@hyeyoo>
Message-ID: <CAJuCfpF52wCCrddQizzsMHF3myuD7JsH6p8DZJgN8J+DGHx2WA@mail.gmail.com>

On Wed, Sep 17, 2025 at 4:46?AM Harry Yoo <harry.yoo at oracle.com> wrote:
>
> On Wed, Sep 10, 2025 at 10:01:20AM +0200, Vlastimil Babka wrote:
> > From: Pedro Falcato <pfalcato at suse.de>
> >
> > kfree_rcu is an optimized version of call_rcu + kfree. It used to not be
> > possible to call it on non-kmalloc objects, but this restriction was
> > lifted ever since SLOB was dropped from the kernel, and since commit
> > 6c6c47b063b5 ("mm, slab: call kvfree_rcu_barrier() from kmem_cache_destroy()").
> >
> > Thus, replace call_rcu + mt_free_rcu with kfree_rcu.
> >
> > Signed-off-by: Pedro Falcato <pfalcato at suse.de>
> > Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> > ---
>
> Looks good to me,
> Reviewed-by: Harry Yoo <harry.yoo at oracle.com>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

>
> --
> Cheers,
> Harry / Hyeonggon


From surenb at google.com  Fri Sep 26 17:06:46 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 17:06:46 -0700
Subject: [PATCH v8 19/23] maple_tree: Replace mt_free_one() with kfree()
In-Reply-To: <20250910-slub-percpu-caches-v8-19-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-19-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpEJ=-=NqmbZpQOGKsxfW1DQsn3_gusQxeJG8E6hVthgmQ@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: Pedro Falcato <pfalcato at suse.de>
>
> kfree() is a little shorter and works with kmem_cache_alloc'd pointers
> too. Also lets us remove one more helper.
>
> Signed-off-by: Pedro Falcato <pfalcato at suse.de>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  lib/maple_tree.c | 13 ++++---------
>  1 file changed, 4 insertions(+), 9 deletions(-)
>
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index c706e2e48f884fd156e25be2b17eb5e154774db7..0439aaacf6cb1f39d0d23af2e2a5af1d27ab32be 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -177,11 +177,6 @@ static inline int mt_alloc_bulk(gfp_t gfp, size_t size, void **nodes)
>         return kmem_cache_alloc_bulk(maple_node_cache, gfp, size, nodes);
>  }
>
> -static inline void mt_free_one(struct maple_node *node)
> -{
> -       kmem_cache_free(maple_node_cache, node);
> -}
> -
>  static inline void mt_free_bulk(size_t size, void __rcu **nodes)
>  {
>         kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
> @@ -5092,7 +5087,7 @@ static void mt_free_walk(struct rcu_head *head)
>         mt_free_bulk(node->slot_len, slots);
>
>  free_leaf:
> -       mt_free_one(node);
> +       kfree(node);
>  }
>
>  static inline void __rcu **mte_destroy_descend(struct maple_enode **enode,
> @@ -5176,7 +5171,7 @@ static void mt_destroy_walk(struct maple_enode *enode, struct maple_tree *mt,
>
>  free_leaf:
>         if (free)
> -               mt_free_one(node);
> +               kfree(node);
>         else
>                 mt_clear_meta(mt, node, node->type);
>  }
> @@ -5385,7 +5380,7 @@ void mas_destroy(struct ma_state *mas)
>                         mt_free_bulk(count, (void __rcu **)&node->slot[1]);
>                         total -= count;
>                 }
> -               mt_free_one(ma_mnode_ptr(node));
> +               kfree(ma_mnode_ptr(node));
>                 total--;
>         }
>
> @@ -6373,7 +6368,7 @@ static void mas_dup_free(struct ma_state *mas)
>         }
>
>         node = mte_to_node(mas->node);
> -       mt_free_one(node);
> +       kfree(node);
>  }
>
>  /*
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 17:28:33 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 17:28:33 -0700
Subject: [PATCH v8 20/23] tools/testing: Add support for prefilled slab sheafs
In-Reply-To: <20250910-slub-percpu-caches-v8-20-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-20-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpEQzHM8hwTLXj8evuWHcUWvsM+HecwkLgP3i-A30rAtHQ@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>
> Add the prefilled sheaf structs to the slab header and the associated
> functions to the testing/shared/linux.c file.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  tools/include/linux/slab.h   | 28 ++++++++++++++
>  tools/testing/shared/linux.c | 89 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 117 insertions(+)
>
> diff --git a/tools/include/linux/slab.h b/tools/include/linux/slab.h
> index c5c5cc6db5668be2cc94c29065ccfa7ca7b4bb08..94937a699402bd1f31887dfb52b6fd0a3c986f43 100644
> --- a/tools/include/linux/slab.h
> +++ b/tools/include/linux/slab.h
> @@ -123,6 +123,18 @@ struct kmem_cache_args {
>         void (*ctor)(void *);
>  };
>
> +struct slab_sheaf {
> +       union {
> +               struct list_head barn_list;
> +               /* only used for prefilled sheafs */
> +               unsigned int capacity;
> +       };
> +       struct kmem_cache *cache;
> +       unsigned int size;
> +       int node; /* only used for rcu_sheaf */
> +       void *objects[];
> +};
> +
>  static inline void *kzalloc(size_t size, gfp_t gfp)
>  {
>         return kmalloc(size, gfp | __GFP_ZERO);
> @@ -173,5 +185,21 @@ __kmem_cache_create(const char *name, unsigned int size, unsigned int align,
>  void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list);
>  int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
>                           void **list);
> +struct slab_sheaf *
> +kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size);
> +
> +void *
> +kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
> +               struct slab_sheaf *sheaf);
> +
> +void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
> +               struct slab_sheaf *sheaf);
> +int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
> +               struct slab_sheaf **sheafp, unsigned int size);
> +
> +static inline unsigned int kmem_cache_sheaf_size(struct slab_sheaf *sheaf)
> +{
> +       return sheaf->size;
> +}
>
>  #endif         /* _TOOLS_SLAB_H */
> diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
> index 97b8412ccbb6d222604c7b397c53c65618d8d51b..4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1 100644
> --- a/tools/testing/shared/linux.c
> +++ b/tools/testing/shared/linux.c
> @@ -137,6 +137,12 @@ void kmem_cache_free_bulk(struct kmem_cache *cachep, size_t size, void **list)
>         if (kmalloc_verbose)
>                 pr_debug("Bulk free %p[0-%zu]\n", list, size - 1);
>
> +       if (cachep->exec_callback) {
> +               if (cachep->callback)
> +                       cachep->callback(cachep->private);
> +               cachep->exec_callback = false;
> +       }
> +
>         pthread_mutex_lock(&cachep->lock);
>         for (int i = 0; i < size; i++)
>                 kmem_cache_free_locked(cachep, list[i]);
> @@ -242,6 +248,89 @@ __kmem_cache_create_args(const char *name, unsigned int size,
>         return ret;
>  }
>
> +struct slab_sheaf *
> +kmem_cache_prefill_sheaf(struct kmem_cache *s, gfp_t gfp, unsigned int size)
> +{
> +       struct slab_sheaf *sheaf;
> +       unsigned int capacity;
> +
> +       if (s->exec_callback) {
> +               if (s->callback)
> +                       s->callback(s->private);
> +               s->exec_callback = false;
> +       }
> +
> +       capacity = max(size, s->sheaf_capacity);
> +
> +       sheaf = calloc(1, sizeof(*sheaf) + sizeof(void *) * capacity);
> +       if (!sheaf)
> +               return NULL;
> +
> +       sheaf->cache = s;
> +       sheaf->capacity = capacity;
> +       sheaf->size = kmem_cache_alloc_bulk(s, gfp, size, sheaf->objects);
> +       if (!sheaf->size) {
> +               free(sheaf);
> +               return NULL;
> +       }
> +
> +       return sheaf;
> +}
> +
> +int kmem_cache_refill_sheaf(struct kmem_cache *s, gfp_t gfp,
> +                struct slab_sheaf **sheafp, unsigned int size)
> +{
> +       struct slab_sheaf *sheaf = *sheafp;
> +       int refill;
> +
> +       if (sheaf->size >= size)
> +               return 0;
> +
> +       if (size > sheaf->capacity) {
> +               sheaf = kmem_cache_prefill_sheaf(s, gfp, size);
> +               if (!sheaf)
> +                       return -ENOMEM;
> +
> +               kmem_cache_return_sheaf(s, gfp, *sheafp);
> +               *sheafp = sheaf;
> +               return 0;
> +       }
> +
> +       refill = kmem_cache_alloc_bulk(s, gfp, size - sheaf->size,
> +                                      &sheaf->objects[sheaf->size]);
> +       if (!refill)
> +               return -ENOMEM;
> +
> +       sheaf->size += refill;
> +       return 0;
> +}
> +
> +void kmem_cache_return_sheaf(struct kmem_cache *s, gfp_t gfp,
> +                struct slab_sheaf *sheaf)
> +{
> +       if (sheaf->size)
> +               kmem_cache_free_bulk(s, sheaf->size, &sheaf->objects[0]);
> +
> +       free(sheaf);
> +}
> +
> +void *
> +kmem_cache_alloc_from_sheaf(struct kmem_cache *s, gfp_t gfp,
> +               struct slab_sheaf *sheaf)
> +{
> +       void *obj;
> +
> +       if (sheaf->size == 0) {
> +               printf("Nothing left in sheaf!\n");
> +               return NULL;
> +       }
> +
> +       obj = sheaf->objects[--sheaf->size];
> +       sheaf->objects[sheaf->size] = NULL;
> +
> +       return obj;
> +}
> +
>  /*
>   * Test the test infrastructure for kem_cache_alloc/free and bulk counterparts.
>   */
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 18:08:38 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 18:08:38 -0700
Subject: [PATCH v8 21/23] maple_tree: Prefilled sheaf conversion and testing
In-Reply-To: <20250910-slub-percpu-caches-v8-21-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-21-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpGBg7xO6-swzvk3vLhMxO+F=C6udEiNmAAC6LEA7vA9=Q@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>
> Use prefilled sheaves instead of bulk allocations. This should speed up
> the allocations and the return path of unused allocations.
>
> Remove the push and pop of nodes from the maple state as this is now
> handled by the slab layer with sheaves.
>
> Testing has been removed as necessary since the features of the tree
> have been reduced.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>

Couple nits but otherwise looks great!

Reviewed-by: Suren Baghdasaryan <surenb at google.com>

> ---
>  include/linux/maple_tree.h       |   6 +-
>  lib/maple_tree.c                 | 326 ++++++---------------------
>  tools/testing/radix-tree/maple.c | 461 ++-------------------------------------
>  tools/testing/shared/linux.c     |   5 +-
>  4 files changed, 88 insertions(+), 710 deletions(-)
>
> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
> index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
> --- a/include/linux/maple_tree.h
> +++ b/include/linux/maple_tree.h
> @@ -442,7 +442,8 @@ struct ma_state {
>         struct maple_enode *node;       /* The node containing this entry */
>         unsigned long min;              /* The minimum index of this node - implied pivot min */
>         unsigned long max;              /* The maximum index of this node - implied pivot max */
> -       struct maple_alloc *alloc;      /* Allocated nodes for this operation */
> +       struct slab_sheaf *sheaf;       /* Allocated nodes for this operation */
> +       unsigned long node_request;

No comment for this poor fella?

>         enum maple_status status;       /* The status of the state (active, start, none, etc) */
>         unsigned char depth;            /* depth of tree descent during write */
>         unsigned char offset;
> @@ -490,7 +491,8 @@ struct ma_wr_state {
>                 .status = ma_start,                                     \
>                 .min = 0,                                               \
>                 .max = ULONG_MAX,                                       \
> -               .alloc = NULL,                                          \
> +               .node_request= 0,                                       \

Missing space in assignment.

> +               .sheaf = NULL,                                          \
>                 .mas_flags = 0,                                         \
>                 .store_type = wr_invalid,                               \
>         }
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index 0439aaacf6cb1f39d0d23af2e2a5af1d27ab32be..a3fcb20227e506ed209554cc8c041a53f7ef4903 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -182,6 +182,22 @@ static inline void mt_free_bulk(size_t size, void __rcu **nodes)
>         kmem_cache_free_bulk(maple_node_cache, size, (void **)nodes);
>  }
>
> +static void mt_return_sheaf(struct slab_sheaf *sheaf)
> +{
> +       kmem_cache_return_sheaf(maple_node_cache, GFP_NOWAIT, sheaf);
> +}
> +
> +static struct slab_sheaf *mt_get_sheaf(gfp_t gfp, int count)
> +{
> +       return kmem_cache_prefill_sheaf(maple_node_cache, gfp, count);
> +}
> +
> +static int mt_refill_sheaf(gfp_t gfp, struct slab_sheaf **sheaf,
> +               unsigned int size)
> +{
> +       return kmem_cache_refill_sheaf(maple_node_cache, gfp, sheaf, size);
> +}
> +
>  /*
>   * ma_free_rcu() - Use rcu callback to free a maple node
>   * @node: The node to free
> @@ -574,67 +590,6 @@ static __always_inline bool mte_dead_node(const struct maple_enode *enode)
>         return ma_dead_node(node);
>  }
>
> -/*
> - * mas_allocated() - Get the number of nodes allocated in a maple state.
> - * @mas: The maple state
> - *
> - * The ma_state alloc member is overloaded to hold a pointer to the first
> - * allocated node or to the number of requested nodes to allocate.  If bit 0 is
> - * set, then the alloc contains the number of requested nodes.  If there is an
> - * allocated node, then the total allocated nodes is in that node.
> - *
> - * Return: The total number of nodes allocated
> - */
> -static inline unsigned long mas_allocated(const struct ma_state *mas)
> -{
> -       if (!mas->alloc || ((unsigned long)mas->alloc & 0x1))
> -               return 0;
> -
> -       return mas->alloc->total;
> -}
> -
> -/*
> - * mas_set_alloc_req() - Set the requested number of allocations.
> - * @mas: the maple state
> - * @count: the number of allocations.
> - *
> - * The requested number of allocations is either in the first allocated node,
> - * located in @mas->alloc->request_count, or directly in @mas->alloc if there is
> - * no allocated node.  Set the request either in the node or do the necessary
> - * encoding to store in @mas->alloc directly.
> - */
> -static inline void mas_set_alloc_req(struct ma_state *mas, unsigned long count)
> -{
> -       if (!mas->alloc || ((unsigned long)mas->alloc & 0x1)) {
> -               if (!count)
> -                       mas->alloc = NULL;
> -               else
> -                       mas->alloc = (struct maple_alloc *)(((count) << 1U) | 1U);
> -               return;
> -       }
> -
> -       mas->alloc->request_count = count;
> -}
> -
> -/*
> - * mas_alloc_req() - get the requested number of allocations.
> - * @mas: The maple state
> - *
> - * The alloc count is either stored directly in @mas, or in
> - * @mas->alloc->request_count if there is at least one node allocated.  Decode
> - * the request count if it's stored directly in @mas->alloc.
> - *
> - * Return: The allocation request count.
> - */
> -static inline unsigned int mas_alloc_req(const struct ma_state *mas)
> -{
> -       if ((unsigned long)mas->alloc & 0x1)
> -               return (unsigned long)(mas->alloc) >> 1;
> -       else if (mas->alloc)
> -               return mas->alloc->request_count;
> -       return 0;
> -}
> -
>  /*
>   * ma_pivots() - Get a pointer to the maple node pivots.
>   * @node: the maple node
> @@ -1120,77 +1075,15 @@ static int mas_ascend(struct ma_state *mas)
>   */
>  static inline struct maple_node *mas_pop_node(struct ma_state *mas)
>  {
> -       struct maple_alloc *ret, *node = mas->alloc;
> -       unsigned long total = mas_allocated(mas);
> -       unsigned int req = mas_alloc_req(mas);
> +       struct maple_node *ret;
>
> -       /* nothing or a request pending. */
> -       if (WARN_ON(!total))
> +       if (WARN_ON_ONCE(!mas->sheaf))
>                 return NULL;
>
> -       if (total == 1) {
> -               /* single allocation in this ma_state */
> -               mas->alloc = NULL;
> -               ret = node;
> -               goto single_node;
> -       }
> -
> -       if (node->node_count == 1) {
> -               /* Single allocation in this node. */
> -               mas->alloc = node->slot[0];
> -               mas->alloc->total = node->total - 1;
> -               ret = node;
> -               goto new_head;
> -       }
> -       node->total--;
> -       ret = node->slot[--node->node_count];
> -       node->slot[node->node_count] = NULL;
> -
> -single_node:
> -new_head:
> -       if (req) {
> -               req++;
> -               mas_set_alloc_req(mas, req);
> -       }
> -
> +       ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
>         memset(ret, 0, sizeof(*ret));
> -       return (struct maple_node *)ret;
> -}
> -
> -/*
> - * mas_push_node() - Push a node back on the maple state allocation.
> - * @mas: The maple state
> - * @used: The used maple node
> - *
> - * Stores the maple node back into @mas->alloc for reuse.  Updates allocated and
> - * requested node count as necessary.
> - */
> -static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
> -{
> -       struct maple_alloc *reuse = (struct maple_alloc *)used;
> -       struct maple_alloc *head = mas->alloc;
> -       unsigned long count;
> -       unsigned int requested = mas_alloc_req(mas);
>
> -       count = mas_allocated(mas);
> -
> -       reuse->request_count = 0;
> -       reuse->node_count = 0;
> -       if (count) {
> -               if (head->node_count < MAPLE_ALLOC_SLOTS) {
> -                       head->slot[head->node_count++] = reuse;
> -                       head->total++;
> -                       goto done;
> -               }
> -               reuse->slot[0] = head;
> -               reuse->node_count = 1;
> -       }
> -
> -       reuse->total = count + 1;
> -       mas->alloc = reuse;
> -done:
> -       if (requested > 1)
> -               mas_set_alloc_req(mas, requested - 1);
> +       return ret;
>  }
>
>  /*
> @@ -1200,75 +1093,32 @@ static inline void mas_push_node(struct ma_state *mas, struct maple_node *used)
>   */
>  static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
>  {
> -       struct maple_alloc *node;
> -       unsigned long allocated = mas_allocated(mas);
> -       unsigned int requested = mas_alloc_req(mas);
> -       unsigned int count;
> -       void **slots = NULL;
> -       unsigned int max_req = 0;
> -
> -       if (!requested)
> -               return;
> +       if (unlikely(mas->sheaf)) {
> +               unsigned long refill = mas->node_request;
>
> -       mas_set_alloc_req(mas, 0);
> -       if (mas->mas_flags & MA_STATE_PREALLOC) {
> -               if (allocated)
> +               if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
> +                       mas->node_request = 0;
>                         return;
> -               WARN_ON(!allocated);
> -       }
> -
> -       if (!allocated || mas->alloc->node_count == MAPLE_ALLOC_SLOTS) {
> -               node = (struct maple_alloc *)mt_alloc_one(gfp);
> -               if (!node)
> -                       goto nomem_one;
> -
> -               if (allocated) {
> -                       node->slot[0] = mas->alloc;
> -                       node->node_count = 1;
> -               } else {
> -                       node->node_count = 0;
>                 }
>
> -               mas->alloc = node;
> -               node->total = ++allocated;
> -               node->request_count = 0;
> -               requested--;
> -       }
> +               if (mt_refill_sheaf(gfp, &mas->sheaf, refill))
> +                       goto error;
>
> -       node = mas->alloc;
> -       while (requested) {
> -               max_req = MAPLE_ALLOC_SLOTS - node->node_count;
> -               slots = (void **)&node->slot[node->node_count];
> -               max_req = min(requested, max_req);
> -               count = mt_alloc_bulk(gfp, max_req, slots);
> -               if (!count)
> -                       goto nomem_bulk;
> -
> -               if (node->node_count == 0) {
> -                       node->slot[0]->node_count = 0;
> -                       node->slot[0]->request_count = 0;
> -               }
> +               mas->node_request = 0;
> +               return;
> +       }
>
> -               node->node_count += count;
> -               allocated += count;
> -               /* find a non-full node*/
> -               do {
> -                       node = node->slot[0];
> -               } while (unlikely(node->node_count == MAPLE_ALLOC_SLOTS));
> -               requested -= count;
> +       mas->sheaf = mt_get_sheaf(gfp, mas->node_request);
> +       if (likely(mas->sheaf)) {
> +               mas->node_request = 0;
> +               return;
>         }
> -       mas->alloc->total = allocated;
> -       return;
>
> -nomem_bulk:
> -       /* Clean up potential freed allocations on bulk failure */
> -       memset(slots, 0, max_req * sizeof(unsigned long));
> -       mas->alloc->total = allocated;
> -nomem_one:
> -       mas_set_alloc_req(mas, requested);
> +error:
>         mas_set_err(mas, -ENOMEM);
>  }
>
> +
>  /*
>   * mas_free() - Free an encoded maple node
>   * @mas: The maple state
> @@ -1279,42 +1129,7 @@ static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
>   */
>  static inline void mas_free(struct ma_state *mas, struct maple_enode *used)
>  {
> -       struct maple_node *tmp = mte_to_node(used);
> -
> -       if (mt_in_rcu(mas->tree))
> -               ma_free_rcu(tmp);
> -       else
> -               mas_push_node(mas, tmp);
> -}
> -
> -/*
> - * mas_node_count_gfp() - Check if enough nodes are allocated and request more
> - * if there is not enough nodes.
> - * @mas: The maple state
> - * @count: The number of nodes needed
> - * @gfp: the gfp flags
> - */
> -static void mas_node_count_gfp(struct ma_state *mas, int count, gfp_t gfp)
> -{
> -       unsigned long allocated = mas_allocated(mas);
> -
> -       if (allocated < count) {
> -               mas_set_alloc_req(mas, count - allocated);
> -               mas_alloc_nodes(mas, gfp);
> -       }
> -}
> -
> -/*
> - * mas_node_count() - Check if enough nodes are allocated and request more if
> - * there is not enough nodes.
> - * @mas: The maple state
> - * @count: The number of nodes needed
> - *
> - * Note: Uses GFP_NOWAIT for gfp flags.
> - */
> -static void mas_node_count(struct ma_state *mas, int count)
> -{
> -       return mas_node_count_gfp(mas, count, GFP_NOWAIT);
> +       ma_free_rcu(mte_to_node(used));
>  }
>
>  /*
> @@ -2451,10 +2266,7 @@ static inline void mas_topiary_node(struct ma_state *mas,
>         enode = tmp_mas->node;
>         tmp = mte_to_node(enode);
>         mte_set_node_dead(enode);
> -       if (in_rcu)
> -               ma_free_rcu(tmp);
> -       else
> -               mas_push_node(mas, tmp);
> +       ma_free_rcu(tmp);
>  }
>
>  /*
> @@ -3980,7 +3792,7 @@ static inline void mas_wr_prealloc_setup(struct ma_wr_state *wr_mas)
>   *
>   * Return: Number of nodes required for preallocation.
>   */
> -static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
> +static inline void mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
>  {
>         struct ma_state *mas = wr_mas->mas;
>         unsigned char height = mas_mt_height(mas);
> @@ -4026,7 +3838,7 @@ static inline int mas_prealloc_calc(struct ma_wr_state *wr_mas, void *entry)
>                 WARN_ON_ONCE(1);
>         }
>
> -       return ret;
> +       mas->node_request = ret;
>  }
>
>  /*
> @@ -4087,15 +3899,15 @@ static inline enum store_type mas_wr_store_type(struct ma_wr_state *wr_mas)
>   */
>  static inline void mas_wr_preallocate(struct ma_wr_state *wr_mas, void *entry)
>  {
> -       int request;
> +       struct ma_state *mas = wr_mas->mas;
>
>         mas_wr_prealloc_setup(wr_mas);
> -       wr_mas->mas->store_type = mas_wr_store_type(wr_mas);
> -       request = mas_prealloc_calc(wr_mas, entry);
> -       if (!request)
> +       mas->store_type = mas_wr_store_type(wr_mas);
> +       mas_prealloc_calc(wr_mas, entry);
> +       if (!mas->node_request)
>                 return;
>
> -       mas_node_count(wr_mas->mas, request);
> +       mas_alloc_nodes(mas, GFP_NOWAIT);
>  }
>
>  /**
> @@ -5208,7 +5020,6 @@ static inline void mte_destroy_walk(struct maple_enode *enode,
>   */
>  void *mas_store(struct ma_state *mas, void *entry)
>  {
> -       int request;
>         MA_WR_STATE(wr_mas, mas, entry);
>
>         trace_ma_write(__func__, mas, 0, entry);
> @@ -5238,11 +5049,11 @@ void *mas_store(struct ma_state *mas, void *entry)
>                 return wr_mas.content;
>         }
>
> -       request = mas_prealloc_calc(&wr_mas, entry);
> -       if (!request)
> +       mas_prealloc_calc(&wr_mas, entry);
> +       if (!mas->node_request)
>                 goto store;
>
> -       mas_node_count(mas, request);
> +       mas_alloc_nodes(mas, GFP_NOWAIT);
>         if (mas_is_err(mas))
>                 return NULL;
>
> @@ -5330,20 +5141,19 @@ EXPORT_SYMBOL_GPL(mas_store_prealloc);
>  int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
>  {
>         MA_WR_STATE(wr_mas, mas, entry);
> -       int ret = 0;
> -       int request;
>
>         mas_wr_prealloc_setup(&wr_mas);
>         mas->store_type = mas_wr_store_type(&wr_mas);
> -       request = mas_prealloc_calc(&wr_mas, entry);
> -       if (!request)
> +       mas_prealloc_calc(&wr_mas, entry);
> +       if (!mas->node_request)
>                 goto set_flag;
>
>         mas->mas_flags &= ~MA_STATE_PREALLOC;
> -       mas_node_count_gfp(mas, request, gfp);
> +       mas_alloc_nodes(mas, gfp);
>         if (mas_is_err(mas)) {
> -               mas_set_alloc_req(mas, 0);
> -               ret = xa_err(mas->node);
> +               int ret = xa_err(mas->node);
> +
> +               mas->node_request = 0;
>                 mas_destroy(mas);
>                 mas_reset(mas);
>                 return ret;
> @@ -5351,7 +5161,7 @@ int mas_preallocate(struct ma_state *mas, void *entry, gfp_t gfp)
>
>  set_flag:
>         mas->mas_flags |= MA_STATE_PREALLOC;
> -       return ret;
> +       return 0;
>  }
>  EXPORT_SYMBOL_GPL(mas_preallocate);
>
> @@ -5365,26 +5175,13 @@ EXPORT_SYMBOL_GPL(mas_preallocate);
>   */
>  void mas_destroy(struct ma_state *mas)
>  {
> -       struct maple_alloc *node;
> -       unsigned long total;
> -
>         mas->mas_flags &= ~MA_STATE_PREALLOC;
>
> -       total = mas_allocated(mas);
> -       while (total) {
> -               node = mas->alloc;
> -               mas->alloc = node->slot[0];
> -               if (node->node_count > 1) {
> -                       size_t count = node->node_count - 1;
> -
> -                       mt_free_bulk(count, (void __rcu **)&node->slot[1]);
> -                       total -= count;
> -               }
> -               kfree(ma_mnode_ptr(node));
> -               total--;
> -       }
> +       mas->node_request = 0;
> +       if (mas->sheaf)
> +               mt_return_sheaf(mas->sheaf);
>
> -       mas->alloc = NULL;
> +       mas->sheaf = NULL;
>  }
>  EXPORT_SYMBOL_GPL(mas_destroy);
>
> @@ -6019,7 +5816,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
>                 mas_alloc_nodes(mas, gfp);
>         }
>
> -       if (!mas_allocated(mas))
> +       if (!mas->sheaf)
>                 return false;
>
>         mas->status = ma_start;
> @@ -7414,8 +7211,9 @@ void mas_dump(const struct ma_state *mas)
>
>         pr_err("[%u/%u] index=%lx last=%lx\n", mas->offset, mas->end,
>                mas->index, mas->last);
> -       pr_err("     min=%lx max=%lx alloc=" PTR_FMT ", depth=%u, flags=%x\n",
> -              mas->min, mas->max, mas->alloc, mas->depth, mas->mas_flags);
> +       pr_err("     min=%lx max=%lx sheaf=" PTR_FMT ", request %lu depth=%u, flags=%x\n",
> +              mas->min, mas->max, mas->sheaf, mas->node_request, mas->depth,
> +              mas->mas_flags);
>         if (mas->index > mas->last)
>                 pr_err("Check index & last\n");
>  }
> diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
> index 4a35e1e7c64b7ce347cbd1693beeaacb0c4c330e..72a8fe8e832a4150c6567b711768eba6a3fa6768 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -57,430 +57,6 @@ struct rcu_reader_struct {
>         struct rcu_test_struct2 *test;
>  };
>
> -static int get_alloc_node_count(struct ma_state *mas)
> -{
> -       int count = 1;
> -       struct maple_alloc *node = mas->alloc;
> -
> -       if (!node || ((unsigned long)node & 0x1))
> -               return 0;
> -       while (node->node_count) {
> -               count += node->node_count;
> -               node = node->slot[0];
> -       }
> -       return count;
> -}
> -
> -static void check_mas_alloc_node_count(struct ma_state *mas)
> -{
> -       mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 1, GFP_KERNEL);
> -       mas_node_count_gfp(mas, MAPLE_ALLOC_SLOTS + 3, GFP_KERNEL);
> -       MT_BUG_ON(mas->tree, get_alloc_node_count(mas) != mas->alloc->total);
> -       mas_destroy(mas);
> -}
> -
> -/*
> - * check_new_node() - Check the creation of new nodes and error path
> - * verification.
> - */
> -static noinline void __init check_new_node(struct maple_tree *mt)
> -{
> -
> -       struct maple_node *mn, *mn2, *mn3;
> -       struct maple_alloc *smn;
> -       struct maple_node *nodes[100];
> -       int i, j, total;
> -
> -       MA_STATE(mas, mt, 0, 0);
> -
> -       check_mas_alloc_node_count(&mas);
> -
> -       /* Try allocating 3 nodes */
> -       mtree_lock(mt);
> -       mt_set_non_kernel(0);
> -       /* request 3 nodes to be allocated. */
> -       mas_node_count(&mas, 3);
> -       /* Allocation request of 3. */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 3);
> -       /* Allocate failed. */
> -       MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 3);
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       MT_BUG_ON(mt, mn == NULL);
> -       MT_BUG_ON(mt, mas.alloc == NULL);
> -       MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
> -       mas_push_node(&mas, mn);
> -       mas_reset(&mas);
> -       mas_destroy(&mas);
> -       mtree_unlock(mt);
> -
> -
> -       /* Try allocating 1 node, then 2 more */
> -       mtree_lock(mt);
> -       /* Set allocation request to 1. */
> -       mas_set_alloc_req(&mas, 1);
> -       /* Check Allocation request of 1. */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
> -       mas_set_err(&mas, -ENOMEM);
> -       /* Validate allocation request. */
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       /* Eat the requested node. */
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       MT_BUG_ON(mt, mn == NULL);
> -       MT_BUG_ON(mt, mn->slot[0] != NULL);
> -       MT_BUG_ON(mt, mn->slot[1] != NULL);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -
> -       mn->parent = ma_parent_ptr(mn);
> -       ma_free_rcu(mn);
> -       mas.status = ma_start;
> -       mas_destroy(&mas);
> -       /* Allocate 3 nodes, will fail. */
> -       mas_node_count(&mas, 3);
> -       /* Drop the lock and allocate 3 nodes. */
> -       mas_nomem(&mas, GFP_KERNEL);
> -       /* Ensure 3 are allocated. */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 3);
> -       /* Allocation request of 0. */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 0);
> -
> -       MT_BUG_ON(mt, mas.alloc == NULL);
> -       MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
> -       MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
> -       /* Ensure we counted 3. */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 3);
> -       /* Free. */
> -       mas_reset(&mas);
> -       mas_destroy(&mas);
> -
> -       /* Set allocation request to 1. */
> -       mas_set_alloc_req(&mas, 1);
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
> -       mas_set_err(&mas, -ENOMEM);
> -       /* Validate allocation request. */
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 1);
> -       /* Check the node is only one node. */
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -       MT_BUG_ON(mt, mn == NULL);
> -       MT_BUG_ON(mt, mn->slot[0] != NULL);
> -       MT_BUG_ON(mt, mn->slot[1] != NULL);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -       mas_push_node(&mas, mn);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 1);
> -       MT_BUG_ON(mt, mas.alloc->node_count);
> -
> -       mas_set_alloc_req(&mas, 2); /* request 2 more. */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 2);
> -       mas_set_err(&mas, -ENOMEM);
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 3);
> -       MT_BUG_ON(mt, mas.alloc == NULL);
> -       MT_BUG_ON(mt, mas.alloc->slot[0] == NULL);
> -       MT_BUG_ON(mt, mas.alloc->slot[1] == NULL);
> -       for (i = 2; i >= 0; i--) {
> -               mn = mas_pop_node(&mas);
> -               MT_BUG_ON(mt, mas_allocated(&mas) != i);
> -               MT_BUG_ON(mt, !mn);
> -               MT_BUG_ON(mt, not_empty(mn));
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -       }
> -
> -       total = 64;
> -       mas_set_alloc_req(&mas, total); /* request 2 more. */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != total);
> -       mas_set_err(&mas, -ENOMEM);
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       for (i = total; i > 0; i--) {
> -               unsigned int e = 0; /* expected node_count */
> -
> -               if (!MAPLE_32BIT) {
> -                       if (i >= 35)
> -                               e = i - 34;
> -                       else if (i >= 5)
> -                               e = i - 4;
> -                       else if (i >= 2)
> -                               e = i - 1;
> -               } else {
> -                       if (i >= 4)
> -                               e = i - 3;
> -                       else if (i >= 1)
> -                               e = i - 1;
> -                       else
> -                               e = 0;
> -               }
> -
> -               MT_BUG_ON(mt, mas.alloc->node_count != e);
> -               mn = mas_pop_node(&mas);
> -               MT_BUG_ON(mt, not_empty(mn));
> -               MT_BUG_ON(mt, mas_allocated(&mas) != i - 1);
> -               MT_BUG_ON(mt, !mn);
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -       }
> -
> -       total = 100;
> -       for (i = 1; i < total; i++) {
> -               mas_set_alloc_req(&mas, i);
> -               mas_set_err(&mas, -ENOMEM);
> -               MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -               for (j = i; j > 0; j--) {
> -                       mn = mas_pop_node(&mas);
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
> -                       MT_BUG_ON(mt, !mn);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       mas_push_node(&mas, mn);
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != j);
> -                       mn = mas_pop_node(&mas);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != j - 1);
> -                       mn->parent = ma_parent_ptr(mn);
> -                       ma_free_rcu(mn);
> -               }
> -               MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -
> -               mas_set_alloc_req(&mas, i);
> -               mas_set_err(&mas, -ENOMEM);
> -               MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -               for (j = 0; j <= i/2; j++) {
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
> -                       nodes[j] = mas_pop_node(&mas);
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
> -               }
> -
> -               while (j) {
> -                       j--;
> -                       mas_push_node(&mas, nodes[j]);
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
> -               }
> -               MT_BUG_ON(mt, mas_allocated(&mas) != i);
> -               for (j = 0; j <= i/2; j++) {
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != i - j);
> -                       mn = mas_pop_node(&mas);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       mn->parent = ma_parent_ptr(mn);
> -                       ma_free_rcu(mn);
> -                       MT_BUG_ON(mt, mas_allocated(&mas) != i - j - 1);
> -               }
> -               mas_reset(&mas);
> -               MT_BUG_ON(mt, mas_nomem(&mas, GFP_KERNEL));
> -               mas_destroy(&mas);
> -
> -       }
> -
> -       /* Set allocation request. */
> -       total = 500;
> -       mas_node_count(&mas, total);
> -       /* Drop the lock and allocate the nodes. */
> -       mas_nomem(&mas, GFP_KERNEL);
> -       MT_BUG_ON(mt, !mas.alloc);
> -       i = 1;
> -       smn = mas.alloc;
> -       while (i < total) {
> -               for (j = 0; j < MAPLE_ALLOC_SLOTS; j++) {
> -                       i++;
> -                       MT_BUG_ON(mt, !smn->slot[j]);
> -                       if (i == total)
> -                               break;
> -               }
> -               smn = smn->slot[0]; /* next. */
> -       }
> -       MT_BUG_ON(mt, mas_allocated(&mas) != total);
> -       mas_reset(&mas);
> -       mas_destroy(&mas); /* Free. */
> -
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -       for (i = 1; i < 128; i++) {
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
> -               for (j = i; j > 0; j--) { /*Free the requests */
> -                       mn = mas_pop_node(&mas); /* get the next node. */
> -                       MT_BUG_ON(mt, mn == NULL);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       mn->parent = ma_parent_ptr(mn);
> -                       ma_free_rcu(mn);
> -               }
> -               MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -       }
> -
> -       for (i = 1; i < MAPLE_NODE_MASK + 1; i++) {
> -               MA_STATE(mas2, mt, 0, 0);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               MT_BUG_ON(mt, mas_allocated(&mas) != i); /* check request filled */
> -               for (j = 1; j <= i; j++) { /* Move the allocations to mas2 */
> -                       mn = mas_pop_node(&mas); /* get the next node. */
> -                       MT_BUG_ON(mt, mn == NULL);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       mas_push_node(&mas2, mn);
> -                       MT_BUG_ON(mt, mas_allocated(&mas2) != j);
> -               }
> -               MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -               MT_BUG_ON(mt, mas_allocated(&mas2) != i);
> -
> -               for (j = i; j > 0; j--) { /*Free the requests */
> -                       MT_BUG_ON(mt, mas_allocated(&mas2) != j);
> -                       mn = mas_pop_node(&mas2); /* get the next node. */
> -                       MT_BUG_ON(mt, mn == NULL);
> -                       MT_BUG_ON(mt, not_empty(mn));
> -                       mn->parent = ma_parent_ptr(mn);
> -                       ma_free_rcu(mn);
> -               }
> -               MT_BUG_ON(mt, mas_allocated(&mas2) != 0);
> -       }
> -
> -
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
> -       MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
> -       MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
> -
> -       mn = mas_pop_node(&mas); /* get the next node. */
> -       MT_BUG_ON(mt, mn == NULL);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS);
> -       MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS - 1);
> -
> -       mas_push_node(&mas, mn);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
> -       MT_BUG_ON(mt, mas.alloc->node_count != MAPLE_ALLOC_SLOTS);
> -
> -       /* Check the limit of pop/push/pop */
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 2); /* Request */
> -       MT_BUG_ON(mt, mas_alloc_req(&mas) != 1);
> -       MT_BUG_ON(mt, mas.node != MA_ERROR(-ENOMEM));
> -       MT_BUG_ON(mt, !mas_nomem(&mas, GFP_KERNEL));
> -       MT_BUG_ON(mt, mas_alloc_req(&mas));
> -       MT_BUG_ON(mt, mas.alloc->node_count != 1);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
> -       MT_BUG_ON(mt, mas.alloc->node_count  != MAPLE_ALLOC_SLOTS);
> -       mas_push_node(&mas, mn);
> -       MT_BUG_ON(mt, mas.alloc->node_count != 1);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 2);
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, not_empty(mn));
> -       mn->parent = ma_parent_ptr(mn);
> -       ma_free_rcu(mn);
> -       for (i = 1; i <= MAPLE_ALLOC_SLOTS + 1; i++) {
> -               mn = mas_pop_node(&mas);
> -               MT_BUG_ON(mt, not_empty(mn));
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -       }
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 0);
> -
> -
> -       for (i = 3; i < MAPLE_NODE_MASK * 3; i++) {
> -               mas.node = MA_ERROR(-ENOMEM);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mas_push_node(&mas, mn); /* put it back */
> -               mas_destroy(&mas);
> -
> -               mas.node = MA_ERROR(-ENOMEM);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn2 = mas_pop_node(&mas); /* get the next node. */
> -               mas_push_node(&mas, mn); /* put them back */
> -               mas_push_node(&mas, mn2);
> -               mas_destroy(&mas);
> -
> -               mas.node = MA_ERROR(-ENOMEM);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn2 = mas_pop_node(&mas); /* get the next node. */
> -               mn3 = mas_pop_node(&mas); /* get the next node. */
> -               mas_push_node(&mas, mn); /* put them back */
> -               mas_push_node(&mas, mn2);
> -               mas_push_node(&mas, mn3);
> -               mas_destroy(&mas);
> -
> -               mas.node = MA_ERROR(-ENOMEM);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -               mas_destroy(&mas);
> -
> -               mas.node = MA_ERROR(-ENOMEM);
> -               mas_node_count(&mas, i); /* Request */
> -               mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -               mn = mas_pop_node(&mas); /* get the next node. */
> -               mn->parent = ma_parent_ptr(mn);
> -               ma_free_rcu(mn);
> -               mas_destroy(&mas);
> -       }
> -
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, 5); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 5);
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, 10); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       mas.status = ma_start;
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 10);
> -       mas_destroy(&mas);
> -
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS - 1); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS - 1);
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, 10 + MAPLE_ALLOC_SLOTS - 1); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       mas.status = ma_start;
> -       MT_BUG_ON(mt, mas_allocated(&mas) != 10 + MAPLE_ALLOC_SLOTS - 1);
> -       mas_destroy(&mas);
> -
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS + 1); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS + 1);
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 2); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       mas.status = ma_start;
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 2);
> -       mas_destroy(&mas);
> -
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 2 + 1); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 2 + 1);
> -       mas.node = MA_ERROR(-ENOMEM);
> -       mas_node_count(&mas, MAPLE_ALLOC_SLOTS * 3 + 2); /* Request */
> -       mas_nomem(&mas, GFP_KERNEL); /* Fill request */
> -       mas.status = ma_start;
> -       MT_BUG_ON(mt, mas_allocated(&mas) != MAPLE_ALLOC_SLOTS * 3 + 2);
> -       mas_destroy(&mas);
> -
> -       mtree_unlock(mt);
> -}
> -
>  /*
>   * Check erasing including RCU.
>   */
> @@ -35507,6 +35083,13 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
>         return vacant_height;
>  }
>
> +static int mas_allocated(struct ma_state *mas)
> +{
> +       if (mas->sheaf)
> +               return kmem_cache_sheaf_size(mas->sheaf);
> +
> +       return 0;
> +}
>  /* Preallocation testing */
>  static noinline void __init check_prealloc(struct maple_tree *mt)
>  {
> @@ -35525,7 +35108,10 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
>
>         /* Spanning store */
>         mas_set_range(&mas, 470, 500);
> -       MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
> +
> +       mas_wr_preallocate(&wr_mas, ptr);
> +       MT_BUG_ON(mt, mas.store_type != wr_spanning_store);
> +       MT_BUG_ON(mt, mas_is_err(&mas));
>         allocated = mas_allocated(&mas);
>         height = mas_mt_height(&mas);
>         vacant_height = get_vacant_height(&wr_mas, ptr);
> @@ -35535,6 +35121,7 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
>         allocated = mas_allocated(&mas);
>         MT_BUG_ON(mt, allocated != 0);
>
> +       mas_wr_preallocate(&wr_mas, ptr);
>         MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
>         allocated = mas_allocated(&mas);
>         height = mas_mt_height(&mas);
> @@ -35575,20 +35162,6 @@ static noinline void __init check_prealloc(struct maple_tree *mt)
>         mn->parent = ma_parent_ptr(mn);
>         ma_free_rcu(mn);
>
> -       MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
> -       allocated = mas_allocated(&mas);
> -       height = mas_mt_height(&mas);
> -       vacant_height = get_vacant_height(&wr_mas, ptr);
> -       MT_BUG_ON(mt, allocated != 1 + (height - vacant_height) * 3);
> -       mn = mas_pop_node(&mas);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != allocated - 1);
> -       mas_push_node(&mas, mn);
> -       MT_BUG_ON(mt, mas_allocated(&mas) != allocated);
> -       MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
> -       mas_destroy(&mas);
> -       allocated = mas_allocated(&mas);
> -       MT_BUG_ON(mt, allocated != 0);
> -
>         MT_BUG_ON(mt, mas_preallocate(&mas, ptr, GFP_KERNEL) != 0);
>         allocated = mas_allocated(&mas);
>         height = mas_mt_height(&mas);
> @@ -36389,11 +35962,17 @@ static void check_nomem_writer_race(struct maple_tree *mt)
>         check_load(mt, 6, xa_mk_value(0xC));
>         mtree_unlock(mt);
>
> +       mt_set_non_kernel(0);
>         /* test for the same race but with mas_store_gfp() */
>         mtree_store_range(mt, 0, 5, xa_mk_value(0xA), GFP_KERNEL);
>         mtree_store_range(mt, 6, 10, NULL, GFP_KERNEL);
>
>         mas_set_range(&mas, 0, 5);
> +
> +       /* setup writer 2 that will trigger the race condition */
> +       mt_set_private(mt);
> +       mt_set_callback(writer2);
> +
>         mtree_lock(mt);
>         mas_store_gfp(&mas, NULL, GFP_KERNEL);
>
> @@ -36508,10 +36087,6 @@ void farmer_tests(void)
>         check_erase_testset(&tree);
>         mtree_destroy(&tree);
>
> -       mt_init_flags(&tree, 0);
> -       check_new_node(&tree);
> -       mtree_destroy(&tree);
> -
>         if (!MAPLE_32BIT) {
>                 mt_init_flags(&tree, MT_FLAGS_ALLOC_RANGE);
>                 check_rcu_simulated(&tree);
> diff --git a/tools/testing/shared/linux.c b/tools/testing/shared/linux.c
> index 4ceff7969b78cf8e33cd1e021c68bc9f8a02a7a1..8c72571559583759456c2b469a2abc2611117c13 100644
> --- a/tools/testing/shared/linux.c
> +++ b/tools/testing/shared/linux.c
> @@ -64,7 +64,8 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
>
>         if (!(gfp & __GFP_DIRECT_RECLAIM)) {
>                 if (!cachep->non_kernel) {
> -                       cachep->exec_callback = true;
> +                       if (cachep->callback)
> +                               cachep->exec_callback = true;
>                         return NULL;
>                 }
>
> @@ -210,6 +211,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *cachep, gfp_t gfp, size_t size,
>                 for (i = 0; i < size; i++)
>                         __kmem_cache_free_locked(cachep, p[i]);
>                 pthread_mutex_unlock(&cachep->lock);
> +               if (cachep->callback)
> +                       cachep->exec_callback = true;
>                 return 0;
>         }
>
>
> --
> 2.51.0
>


From surenb at google.com  Fri Sep 26 18:17:08 2025
From: surenb at google.com (Suren Baghdasaryan)
Date: Fri, 26 Sep 2025 18:17:08 -0700
Subject: [PATCH v8 22/23] maple_tree: Add single node allocation support
 to maple state
In-Reply-To: <20250910-slub-percpu-caches-v8-22-ca3099d8352c@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz> <20250910-slub-percpu-caches-v8-22-ca3099d8352c@suse.cz>
Message-ID: <CAJuCfpGhv15YLyAm7X63t5gT9hkw+6PK3GZbWmOTng+AG7rEsg@mail.gmail.com>

On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>
> From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>
>
> The fast path through a write will require replacing a single node in
> the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
> special case the node store operation by just allocating one node in the
> maple state.
>
> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> ---
>  include/linux/maple_tree.h       |  4 +++-
>  lib/maple_tree.c                 | 47 +++++++++++++++++++++++++++++++++++-----
>  tools/testing/radix-tree/maple.c |  9 ++++++--
>  3 files changed, 51 insertions(+), 9 deletions(-)
>
> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
> index 166fd67e00d882b1e6de1f80c1b590bba7497cd3..562a1e9e5132b5b1fa8f8402a7cadd8abb65e323 100644
> --- a/include/linux/maple_tree.h
> +++ b/include/linux/maple_tree.h
> @@ -443,6 +443,7 @@ struct ma_state {
>         unsigned long min;              /* The minimum index of this node - implied pivot min */
>         unsigned long max;              /* The maximum index of this node - implied pivot max */
>         struct slab_sheaf *sheaf;       /* Allocated nodes for this operation */
> +       struct maple_node *alloc;       /* allocated nodes */
>         unsigned long node_request;
>         enum maple_status status;       /* The status of the state (active, start, none, etc) */
>         unsigned char depth;            /* depth of tree descent during write */
> @@ -491,8 +492,9 @@ struct ma_wr_state {
>                 .status = ma_start,                                     \
>                 .min = 0,                                               \
>                 .max = ULONG_MAX,                                       \
> -               .node_request= 0,                                       \
>                 .sheaf = NULL,                                          \
> +               .alloc = NULL,                                          \
> +               .node_request= 0,                                       \
>                 .mas_flags = 0,                                         \
>                 .store_type = wr_invalid,                               \
>         }
> diff --git a/lib/maple_tree.c b/lib/maple_tree.c
> index a3fcb20227e506ed209554cc8c041a53f7ef4903..a912e6a1d4378e72b967027b60f8f564476ad14e 100644
> --- a/lib/maple_tree.c
> +++ b/lib/maple_tree.c
> @@ -1073,16 +1073,23 @@ static int mas_ascend(struct ma_state *mas)
>   *
>   * Return: A pointer to a maple node.
>   */
> -static inline struct maple_node *mas_pop_node(struct ma_state *mas)
> +static __always_inline struct maple_node *mas_pop_node(struct ma_state *mas)
>  {
>         struct maple_node *ret;
>
> +       if (mas->alloc) {
> +               ret = mas->alloc;
> +               mas->alloc = NULL;
> +               goto out;
> +       }
> +
>         if (WARN_ON_ONCE(!mas->sheaf))
>                 return NULL;
>
>         ret = kmem_cache_alloc_from_sheaf(maple_node_cache, GFP_NOWAIT, mas->sheaf);
> -       memset(ret, 0, sizeof(*ret));
>
> +out:
> +       memset(ret, 0, sizeof(*ret));
>         return ret;
>  }
>
> @@ -1093,9 +1100,34 @@ static inline struct maple_node *mas_pop_node(struct ma_state *mas)
>   */
>  static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
>  {
> -       if (unlikely(mas->sheaf)) {
> -               unsigned long refill = mas->node_request;
> +       if (!mas->node_request)
> +               return;
> +
> +       if (mas->node_request == 1) {
> +               if (mas->sheaf)
> +                       goto use_sheaf;

Hmm, I don't get the above logic. One node is requested and instead of
using possibly available mas->alloc, we jump to using mas->sheaf and
freeing mas->alloc... That does not sound efficient. What am I
missing?

> +
> +               if (mas->alloc)
> +                       return;
>
> +               mas->alloc = mt_alloc_one(gfp);
> +               if (!mas->alloc)
> +                       goto error;
> +
> +               mas->node_request = 0;
> +               return;
> +       }
> +
> +use_sheaf:
> +       if (unlikely(mas->alloc)) {
> +               kfree(mas->alloc);
> +               mas->alloc = NULL;
> +       }
> +
> +       if (mas->sheaf) {
> +               unsigned long refill;
> +
> +               refill = mas->node_request;
>                 if(kmem_cache_sheaf_size(mas->sheaf) >= refill) {
>                         mas->node_request = 0;
>                         return;
> @@ -5180,8 +5212,11 @@ void mas_destroy(struct ma_state *mas)
>         mas->node_request = 0;
>         if (mas->sheaf)
>                 mt_return_sheaf(mas->sheaf);
> -
>         mas->sheaf = NULL;
> +
> +       if (mas->alloc)
> +               kfree(mas->alloc);
> +       mas->alloc = NULL;
>  }
>  EXPORT_SYMBOL_GPL(mas_destroy);
>
> @@ -5816,7 +5851,7 @@ bool mas_nomem(struct ma_state *mas, gfp_t gfp)
>                 mas_alloc_nodes(mas, gfp);
>         }
>
> -       if (!mas->sheaf)
> +       if (!mas->sheaf && !mas->alloc)
>                 return false;
>
>         mas->status = ma_start;
> diff --git a/tools/testing/radix-tree/maple.c b/tools/testing/radix-tree/maple.c
> index 72a8fe8e832a4150c6567b711768eba6a3fa6768..83260f2efb1990b71093e456950069c24d75560e 100644
> --- a/tools/testing/radix-tree/maple.c
> +++ b/tools/testing/radix-tree/maple.c
> @@ -35085,10 +35085,15 @@ static unsigned char get_vacant_height(struct ma_wr_state *wr_mas, void *entry)
>
>  static int mas_allocated(struct ma_state *mas)
>  {
> +       int total = 0;
> +
> +       if (mas->alloc)
> +               total++;
> +
>         if (mas->sheaf)
> -               return kmem_cache_sheaf_size(mas->sheaf);
> +               total += kmem_cache_sheaf_size(mas->sheaf);
>
> -       return 0;
> +       return total;
>  }
>  /* Preallocation testing */
>  static noinline void __init check_prealloc(struct maple_tree *mt)
>
> --
> 2.51.0
>


From vbabka at suse.cz  Mon Sep 29 00:30:30 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 29 Sep 2025 09:30:30 +0200
Subject: [PATCH v8 21/23] maple_tree: Prefilled sheaf conversion and
 testing
In-Reply-To: <CAJuCfpGBg7xO6-swzvk3vLhMxO+F=C6udEiNmAAC6LEA7vA9=Q@mail.gmail.com>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-21-ca3099d8352c@suse.cz>
 <CAJuCfpGBg7xO6-swzvk3vLhMxO+F=C6udEiNmAAC6LEA7vA9=Q@mail.gmail.com>
Message-ID: <e8735d27-2917-402a-86fa-7446b1acbfcc@suse.cz>

On 9/27/25 03:08, Suren Baghdasaryan wrote:
> On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>>
>> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
>>
>> Use prefilled sheaves instead of bulk allocations. This should speed up
>> the allocations and the return path of unused allocations.
>>
>> Remove the push and pop of nodes from the maple state as this is now
>> handled by the slab layer with sheaves.
>>
>> Testing has been removed as necessary since the features of the tree
>> have been reduced.
>>
>> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
>> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> 
> Couple nits but otherwise looks great!
> 
> Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> 
>> ---
>>  include/linux/maple_tree.h       |   6 +-
>>  lib/maple_tree.c                 | 326 ++++++---------------------
>>  tools/testing/radix-tree/maple.c | 461 ++-------------------------------------
>>  tools/testing/shared/linux.c     |   5 +-
>>  4 files changed, 88 insertions(+), 710 deletions(-)
>>
>> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
>> index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
>> --- a/include/linux/maple_tree.h
>> +++ b/include/linux/maple_tree.h
>> @@ -442,7 +442,8 @@ struct ma_state {
>>         struct maple_enode *node;       /* The node containing this entry */
>>         unsigned long min;              /* The minimum index of this node - implied pivot min */
>>         unsigned long max;              /* The maximum index of this node - implied pivot max */
>> -       struct maple_alloc *alloc;      /* Allocated nodes for this operation */
>> +       struct slab_sheaf *sheaf;       /* Allocated nodes for this operation */
>> +       unsigned long node_request;
> 
> No comment for this poor fella?

adding: /* The number of nodes to allocate for this operation */



From vbabka at suse.cz  Mon Sep 29 00:39:57 2025
From: vbabka at suse.cz (Vlastimil Babka)
Date: Mon, 29 Sep 2025 09:39:57 +0200
Subject: [PATCH v8 22/23] maple_tree: Add single node allocation support
 to maple state
In-Reply-To: <CAJuCfpGhv15YLyAm7X63t5gT9hkw+6PK3GZbWmOTng+AG7rEsg@mail.gmail.com>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-22-ca3099d8352c@suse.cz>
 <CAJuCfpGhv15YLyAm7X63t5gT9hkw+6PK3GZbWmOTng+AG7rEsg@mail.gmail.com>
Message-ID: <6b983596-e521-4714-8a5b-758da76eb1c7@suse.cz>

On 9/27/25 03:17, Suren Baghdasaryan wrote:
> On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
>>
>> From: "Liam R. Howlett" <Liam.Howlett at Oracle.com>
>>
>> The fast path through a write will require replacing a single node in
>> the tree.  Using a sheaf (32 nodes) is too heavy for the fast path, so
>> special case the node store operation by just allocating one node in the
>> maple state.
>>
>> Signed-off-by: Liam R. Howlett <Liam.Howlett at Oracle.com>
>> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
>> ---
>>  include/linux/maple_tree.h       |  4 +++-
>>  lib/maple_tree.c                 | 47 +++++++++++++++++++++++++++++++++++-----
>>  tools/testing/radix-tree/maple.c |  9 ++++++--
>>  3 files changed, 51 insertions(+), 9 deletions(-)
>>
>> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
>> index 166fd67e00d882b1e6de1f80c1b590bba7497cd3..562a1e9e5132b5b1fa8f8402a7cadd8abb65e323 100644
>> --- a/include/linux/maple_tree.h
>> +++ b/include/linux/maple_tree.h
>> @@ -443,6 +443,7 @@ struct ma_state {
>>         unsigned long min;              /* The minimum index of this node - implied pivot min */
>>         unsigned long max;              /* The maximum index of this node - implied pivot max */
>>         struct slab_sheaf *sheaf;       /* Allocated nodes for this operation */
>> +       struct maple_node *alloc;       /* allocated nodes */

Replacing with: /* A single allocated node for fast path writes */

since I'm touching it anyway due to previous patch fixup.

>>
>> @@ -1093,9 +1100,34 @@ static inline struct maple_node *mas_pop_node(struct ma_state *mas)
>>   */
>>  static inline void mas_alloc_nodes(struct ma_state *mas, gfp_t gfp)
>>  {
>> -       if (unlikely(mas->sheaf)) {
>> -               unsigned long refill = mas->node_request;
>> +       if (!mas->node_request)
>> +               return;
>> +
>> +       if (mas->node_request == 1) {
>> +               if (mas->sheaf)
>> +                       goto use_sheaf;
> 
> Hmm, I don't get the above logic. One node is requested and instead of
> using possibly available mas->alloc, we jump to using mas->sheaf and
> freeing mas->alloc... That does not sound efficient. What am I
> missing?

I'm not changing it now due to merge window, only cosmetic changes and r-b
tags. Can leave it to a follow-up optimization.


From Liam.Howlett at oracle.com  Mon Sep 29 09:51:49 2025
From: Liam.Howlett at oracle.com (Liam R. Howlett)
Date: Mon, 29 Sep 2025 12:51:49 -0400
Subject: [PATCH v8 21/23] maple_tree: Prefilled sheaf conversion and
 testing
In-Reply-To: <e8735d27-2917-402a-86fa-7446b1acbfcc@suse.cz>
References: <20250910-slub-percpu-caches-v8-0-ca3099d8352c@suse.cz>
 <20250910-slub-percpu-caches-v8-21-ca3099d8352c@suse.cz>
 <CAJuCfpGBg7xO6-swzvk3vLhMxO+F=C6udEiNmAAC6LEA7vA9=Q@mail.gmail.com>
 <e8735d27-2917-402a-86fa-7446b1acbfcc@suse.cz>
Message-ID: <4oaulium6rn7xzio3dg3lxjc7qrlt27dl2bs3dlmmkytjkiqlg@zwphzvpirbes>

* Vlastimil Babka <vbabka at suse.cz> [250929 03:30]:
> On 9/27/25 03:08, Suren Baghdasaryan wrote:
> > On Wed, Sep 10, 2025 at 1:01?AM Vlastimil Babka <vbabka at suse.cz> wrote:
> >>
> >> From: "Liam R. Howlett" <Liam.Howlett at oracle.com>
> >>
> >> Use prefilled sheaves instead of bulk allocations. This should speed up
> >> the allocations and the return path of unused allocations.
> >>
> >> Remove the push and pop of nodes from the maple state as this is now
> >> handled by the slab layer with sheaves.
> >>
> >> Testing has been removed as necessary since the features of the tree
> >> have been reduced.
> >>
> >> Signed-off-by: Liam R. Howlett <Liam.Howlett at oracle.com>
> >> Signed-off-by: Vlastimil Babka <vbabka at suse.cz>
> > 
> > Couple nits but otherwise looks great!
> > 
> > Reviewed-by: Suren Baghdasaryan <surenb at google.com>
> > 
> >> ---
> >>  include/linux/maple_tree.h       |   6 +-
> >>  lib/maple_tree.c                 | 326 ++++++---------------------
> >>  tools/testing/radix-tree/maple.c | 461 ++-------------------------------------
> >>  tools/testing/shared/linux.c     |   5 +-
> >>  4 files changed, 88 insertions(+), 710 deletions(-)
> >>
> >> diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h
> >> index bafe143b1f783202e27b32567fffee4149e8e266..166fd67e00d882b1e6de1f80c1b590bba7497cd3 100644
> >> --- a/include/linux/maple_tree.h
> >> +++ b/include/linux/maple_tree.h
> >> @@ -442,7 +442,8 @@ struct ma_state {
> >>         struct maple_enode *node;       /* The node containing this entry */
> >>         unsigned long min;              /* The minimum index of this node - implied pivot min */
> >>         unsigned long max;              /* The maximum index of this node - implied pivot max */
> >> -       struct maple_alloc *alloc;      /* Allocated nodes for this operation */
> >> +       struct slab_sheaf *sheaf;       /* Allocated nodes for this operation */
> >> +       unsigned long node_request;
> > 
> > No comment for this poor fella?
> 
> adding: /* The number of nodes to allocate for this operation */
> 

Thanks.  That sounds better than my planned /* requested nodes */


