[PATCH v7 07/12] mm: multigenerational LRU: support page table walks

[Yu Zhao]

On Tue, Feb 08, 2022 at 01:18:57AM -0700, Yu Zhao wrote:
> To avoid confusions, the term "iteration" specifically means the
> traversal of an entire mm_struct list; the term "walk" will be applied
> to page tables and the rmap, as usual.
> 
> To further exploit spatial locality, the aging prefers to walk page
> tables to search for young PTEs and promote hot pages. A runtime
> switch will be added in the next patch to enable or disable this
> feature. Without it, the aging relies on the rmap only.

Clarified that page table scanning is optional as requested here:
https://lore.kernel.org/linux-mm/YdxEqFPLDf+wI0xX@dhcp22.suse.cz/

> NB: this feature has nothing similar with the page table scanning in
> the 2.4 kernel [1], which searches page tables for old PTEs, adds cold
> pages to swapcache and unmap them.
> 
> An mm_struct list is maintained for each memcg, and an mm_struct
> follows its owner task to the new memcg when this task is migrated.
> Given an lruvec, the aging iterates lruvec_memcg()->mm_list and calls
> walk_page_range() with each mm_struct on this list to promote hot
> pages before it increments max_seq.
> 
> When multiple page table walkers (threads) iterate the same list, each
> of them gets a unique mm_struct; therefore they can run concurrently.
> Page table walkers ignore any misplaced pages, e.g., if an mm_struct
> was migrated, pages it left in the previous memcg won't be promoted
> when its current memcg is under reclaim. Similarly, page table walkers
> won't promote pages from nodes other than the one under reclaim.

Clarified the interaction between task migration and reclaim as requested here:
https://lore.kernel.org/linux-mm/YdxPEdsfl771Z7IX@dhcp22.suse.cz/

<snipped>

> Server benchmark results:
>   Single workload:
>     fio (buffered I/O): no change
> 
>   Single workload:
>     memcached (anon): +[5.5, 7.5]%
>                 Ops/sec      KB/sec
>       patch1-6: 1015292.83   39490.38
>       patch1-7: 1080856.82   42040.53
> 
>   Configurations:
>     no change
> 
> Client benchmark results:
>   kswapd profiles:
>     patch1-6
>       45.49%  lzo1x_1_do_compress (real work)
>        7.38%  page_vma_mapped_walk
>        7.24%  _raw_spin_unlock_irq
>        2.64%  ptep_clear_flush
>        2.31%  __zram_bvec_write
>        2.13%  do_raw_spin_lock
>        2.09%  lru_gen_look_around
>        1.89%  free_unref_page_list
>        1.85%  memmove
>        1.74%  obj_malloc
> 
>     patch1-7
>       47.73%  lzo1x_1_do_compress (real work)
>        6.84%  page_vma_mapped_walk
>        6.14%  _raw_spin_unlock_irq
>        2.86%  walk_pte_range
>        2.79%  ptep_clear_flush
>        2.24%  __zram_bvec_write
>        2.10%  do_raw_spin_lock
>        1.94%  free_unref_page_list
>        1.80%  memmove
>        1.75%  obj_malloc
> 
>   Configurations:
>     no change

Added benchmark results to show the difference between page table
scanning and no page table scanning, as requested here:
https://lore.kernel.org/linux-mm/Ye6xS6xUD1SORdHJ@dhcp22.suse.cz/

<snipped>

> +static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lru_gen_mm_walk *walk)
> +{
> +	static const struct mm_walk_ops mm_walk_ops = {
> +		.test_walk = should_skip_vma,
> +		.p4d_entry = walk_pud_range,
> +	};
> +
> +	int err;
> +	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
> +
> +	walk->next_addr = FIRST_USER_ADDRESS;
> +
> +	do {
> +		err = -EBUSY;
> +
> +		/* folio_update_gen() requires stable folio_memcg() */
> +		if (!mem_cgroup_trylock_pages(memcg))
> +			break;

Added a comment on the stable folio_memcg() requirement as requested
here:
https://lore.kernel.org/linux-mm/Yd6q0QdLVTS53vu4@dhcp22.suse.cz/

<snipped>

> +static struct lru_gen_mm_walk *alloc_mm_walk(void)
> +{
> +	if (current->reclaim_state && current->reclaim_state->mm_walk)
> +		return current->reclaim_state->mm_walk;
> +
> +	return kzalloc(sizeof(struct lru_gen_mm_walk),
> +		       __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
> +}

Replaced kvzalloc() with kzalloc() as requested here:
https://lore.kernel.org/linux-mm/Yd6tafG3CS7BoRYn@dhcp22.suse.cz/

Replaced GFP_KERNEL with __GFP_HIGH|__GFP_NOMEMALLOC|__GFP_NOWARN as
requested here:
https://lore.kernel.org/linux-mm/YefddYm8FAfJalNa@dhcp22.suse.cz/

<snipped>