SPARSEMEM memory needs?
Joakim Tjernlund
Joakim.Tjernlund at infinera.com
Mon Jun 6 13:37:03 PDT 2022
On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> Hello Joakim,
>
> On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> <Joakim.Tjernlund at infinera.com> wrote:
> >
> > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> >
> > [ 0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > [ 0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > [ 0.000000] ------------[ cut here ]------------
> > [ 0.000000] Call trace:
> > [ 0.000000] vmemmap_alloc_block+0xc4/0xe8
> > [ 0.000000] vmemmap_pud_populate+0x24/0xb8
> > [ 0.000000] vmemmap_populate+0xa4/0x180
> > [ 0.000000] __populate_section_memmap+0x50/0x70
> > [ 0.000000] sparse_init_nid+0x164/0x1d4
> > [ 0.000000] sparse_init+0xb0/0x224
> > [ 0.000000] bootmem_init+0x40/0x80
> > [ 0.000000] setup_arch+0x244/0x540
> > [ 0.000000] start_kernel+0x60/0x804
> > [ 0.000000] __primary_switched+0xa0/0xa8
> > [ 0.000000] ---[ end trace 0000000000000000 ]---
> > [ 0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [ 0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
>
> I'm not sure which backtrace belongs with which memblock debug
> message, but something looks wrong here. vmemmap_pud_populate() does
> an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> here.
I have added this to get a BT, I trimmed the regs ways and just kept the BT
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index bdce883f9286..f8326e1295ed 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
warned = true;
}
return NULL;
- } else
+ } else {
+ WARN_ON(1);
return __earlyonly_bootmem_alloc(node, size, size,
__pa(MAX_DMA_ADDRESS));
+ }
}
I guess I may have trimmed the log a bit too much. How does this look?
[ 0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
[ 0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
[ 0.000000] Call trace:
[ 0.000000] vmemmap_alloc_block+0xc4/0xe8
[ 0.000000] vmemmap_pud_populate+0x24/0xb8
[ 0.000000] vmemmap_populate+0xa4/0x180
[ 0.000000] __populate_section_memmap+0x50/0x70
[ 0.000000] sparse_init_nid+0x164/0x1d4
[ 0.000000] sparse_init+0xb0/0x224
[ 0.000000] bootmem_init+0x40/0x80
[ 0.000000] setup_arch+0x244/0x540
[ 0.000000] start_kernel+0x60/0x804
[ 0.000000] __primary_switched+0xa0/0xa8
[ 0.000000] ---[ end trace 0000000000000000 ]---
[ 0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[ 0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134
[ 0.000000] Call trace:
[ 0.000000] vmemmap_alloc_block+0xc4/0xe8
[ 0.000000] vmemmap_alloc_block_buf+0xfc/0x100
[ 0.000000] vmemmap_populate+0xd0/0x180
[ 0.000000] __populate_section_memmap+0x50/0x70
[ 0.000000] sparse_init_nid+0x164/0x1d4
[ 0.000000] sparse_init+0xb0/0x224
[ 0.000000] bootmem_init+0x40/0x80
[ 0.000000] setup_arch+0x244/0x540
[ 0.000000] start_kernel+0x60/0x804
[ 0.000000] __primary_switched+0xa0/0xa8
[ 0.000000] ---[ end trace f68728a0d3053b52 ]---
[ 0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[ 0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
[ 0.000000] Call trace:
[ 0.000000] vmemmap_alloc_block+0xc4/0xe8
[ 0.000000] vmemmap_alloc_block_buf+0xfc/0x100
[ 0.000000] vmemmap_populate+0xd0/0x180
[ 0.000000] __populate_section_memmap+0x50/0x70
[ 0.000000] sparse_init_nid+0x164/0x1d4
[ 0.000000] sparse_init+0xb0/0x224
[ 0.000000] bootmem_init+0x40/0x80
[ 0.000000] setup_arch+0x244/0x540
[ 0.000000] start_kernel+0x60/0x804
[ 0.000000] __primary_switched+0xa0/0xa8
[ 0.000000] ---[ end trace f68728a0d3053b53 ]---
[ 0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[ 0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000060000000-0x00000000703fffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000060000000-0x0000000061ffffff]
[ 0.000000] node 0: [mem 0x0000000070000000-0x0000000070000fff]
[ 0.000000] node 0: [mem 0x0000000070001000-0x00000000703fffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000060000000-0x00000000703fffff]
[ 0.000000] MEMBLOCK configuration:
[ 0.000000] memory size = 0x0000000002400000 reserved size = 0x0000000000de991f
[ 0.000000] memory.cnt = 0x3
[ 0.000000] memory[0x0] [0x0000000060000000-0x0000000061ffffff], 0x0000000002000000 bytes flags: 0x0
[ 0.000000] memory[0x1] [0x0000000070000000-0x0000000070000fff], 0x0000000000001000 bytes flags: 0x4
[ 0.000000] memory[0x2] [0x0000000070001000-0x00000000703fffff], 0x00000000003ff000 bytes flags: 0x0
[ 0.000000] reserved.cnt = 0x9
[ 0.000000] reserved[0x0] [0x0000000060010000-0x00000000605cefff], 0x00000000005bf000 bytes flags: 0x0
[ 0.000000] reserved[0x1] [0x00000000605d1000-0x00000000605dffff], 0x000000000000f000 bytes flags: 0x0
[ 0.000000] reserved[0x2] [0x0000000061800000-0x0000000061ffffff], 0x0000000000800000 bytes flags: 0x0
[ 0.000000] reserved[0x3] [0x0000000070377000-0x0000000070379fff], 0x0000000000003000 bytes flags: 0x0
[ 0.000000] reserved[0x4] [0x00000000703e7000-0x00000000703e7fff], 0x0000000000001000 bytes flags: 0x0
[ 0.000000] reserved[0x5] [0x00000000703e86c0-0x00000000703e86ef], 0x0000000000000030 bytes flags: 0x0
[ 0.000000] reserved[0x6] [0x00000000703e8700-0x00000000703e97ff], 0x0000000000001100 bytes flags: 0x0
[ 0.000000] reserved[0x7] [0x00000000703e9810-0x00000000703e983e], 0x000000000000002f bytes flags: 0x0
[ 0.000000] reserved[0x8] [0x00000000703e9840-0x00000000703fffff], 0x00000000000167c0 bytes flags: 0x0
>
> Which kernel version are you using? Do you have any out of tree
> changes related to memory management? I assume you are using the
> standard 4k page size, right?
5.15.26, no MM changes and using 4KB. Oh a small MM change that didn't do much I had added to this run:
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -941,6 +941,9 @@ choice
config ARM64_PA_BITS_48
bool "48-bit"
+config ARM64_PA_BITS_40
+ bool "40-bit"
+
config ARM64_PA_BITS_52
bool "52-bit (ARMv8.2)"
depends on ARM64_64K_PAGES
@@ -957,6 +960,7 @@ endchoice
config ARM64_PA_BITS
int
+ default 40 if ARM64_PA_BITS_40
default 48 if ARM64_PA_BITS_48
default 52 if ARM64_PA_BITS_52
>
>
> > [ 0.000000] ------------[ cut here ]------------
> > [ 0.000000] Call trace:
> > [ 0.000000] vmemmap_alloc_block+0xc4/0xe8
> > [ 0.000000] vmemmap_alloc_block_buf+0xfc/0x100
> > [ 0.000000] vmemmap_populate+0xd0/0x180
> > [ 0.000000] __populate_section_memmap+0x50/0x70
> > [ 0.000000] sparse_init_nid+0x164/0x1d4
> > [ 0.000000] sparse_init+0xb0/0x224
> > [ 0.000000] bootmem_init+0x40/0x80
> > [ 0.000000] setup_arch+0x244/0x540
> > [ 0.000000] start_kernel+0x60/0x804
> > [ 0.000000] __primary_switched+0xa0/0xa8
> > [ 0.000000] ---[ end trace f68728a0d3053b53 ]---
> > [ 0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [ 0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
> >
> > It seems AARCH64 SPARSEMEM uses some RAM and I wonder if this use can be somewhat reduced?
> > Looking at this code I have not yet figured out how these allocations work.
> >
> > If SPARSEMEM really needs all that RAM, could a simpler memory model like FLATMEM be supported?
> >
> > Jocke
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel at lists.infradead.org
> > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C7fe98adbbd1f4582486608da47f05071%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637901394719615340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=irSjvXlqRsP%2FoIKjcZjC%2Bg5hfMJZBkWaJbkJTLAQk%2Bs%3D&reserved=0
More information about the linux-arm-kernel
mailing list