[PATCH] arm64, numa: Add cpu_to_node() implementation.
Robert Richter
rric at kernel.org
Tue Sep 20 03:43:48 PDT 2016
David,
On 19.09.16 11:49:30, David Daney wrote:
> Fix by supplying a cpu_to_node() implementation that returns correct
> node mappings.
> +int cpu_to_node(int cpu)
> +{
> + int nid;
> +
> + /*
> + * Return 0 for unknown mapping so that we report something
> + * sensible if firmware doesn't supply a proper mapping.
> + */
> + if (cpu < 0 || cpu >= NR_CPUS)
> + return 0;
> +
> + nid = cpu_to_node_map[cpu];
> + if (nid == NUMA_NO_NODE)
> + nid = 0;
> + return nid;
> +}
> +EXPORT_SYMBOL(cpu_to_node);
this implementation fixes the per-cpu workqueue initialization, but I
don't think a cpu_to_node() implementation private to arm64 is the
proper solution.
Apart from better using generic code, the cpu_to_node() function is
called in the kernel's fast path. I think your implementation is too
expensive and also does not consider per-cpu data access for the
lookup as the generic code does. Secondly, numa_off is not considered
at all.
Instead we need to make sure the set_*numa_node() functions are called
earlier before secondary cpus are booted. My suggested change for that
is this:
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d93d43352504..952365c2f100 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -204,7 +204,6 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
static void smp_store_cpu_info(unsigned int cpuid)
{
store_cpu_topology(cpuid);
- numa_store_cpu_info(cpuid);
}
/*
@@ -719,6 +718,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
continue;
set_cpu_present(cpu, true);
+ numa_store_cpu_info(cpu);
}
}
I have tested the code and it properly sets up all per-cpu workqueues.
Unfortunately either your nor my code does fix the BUG_ON() I see with
the numa kernel:
kernel BUG at mm/page_alloc.c:1848!
See below for the core dump. It looks like this happens due to moving
a mem block where first and last page are mapped to different numa
nodes, thus, triggering the BUG_ON().
Continuing with my investigations...
-Robert
[ 9.674272] ------------[ cut here ]------------
[ 9.678881] kernel BUG at mm/page_alloc.c:1848!
[ 9.683406] Internal error: Oops - BUG: 0 [#1] SMP
[ 9.688190] Modules linked in:
[ 9.691247] CPU: 77 PID: 1 Comm: swapper/0 Tainted: G W 4.8.0-rc5.vanilla5-00030-ga2b86cb3ce72 #38
[ 9.701322] Hardware name: www.cavium.com ThunderX CRB-2S/ThunderX CRB-2S, BIOS 0.3 Aug 24 2016
[ 9.710008] task: ffff800fe4561400 task.stack: ffff800ffbe0c000
[ 9.715939] PC is at move_freepages+0x160/0x168
[ 9.720460] LR is at move_freepages+0x160/0x168
[ 9.724979] pc : [<ffff0000081ec7d0>] lr : [<ffff0000081ec7d0>] pstate: 600000c5
[ 9.732362] sp : ffff800ffbe0f510
[ 9.735666] x29: ffff800ffbe0f510 x28: ffff7fe043f80020
[ 9.740975] x27: ffff7fe043f80000 x26: 000000000000000c
[ 9.746283] x25: 000000000000000c x24: ffff810ffffaf0e0
[ 9.751591] x23: 0000000000000001 x22: 0000000000000000
[ 9.756898] x21: ffff7fe043ffffc0 x20: ffff810ffffaeb00
[ 9.762206] x19: ffff7fe043f80000 x18: 0000000000000010
[ 9.767513] x17: 0000000000000000 x16: 0000000100000000
[ 9.772821] x15: ffff000088f03f37 x14: 6e2c303d64696e2c
[ 9.778128] x13: 3038336566666666 x12: 6630303866666666
[ 9.783436] x11: 3d656e6f7a203a64 x10: 0000000000000536
[ 9.788744] x9 : 0000000000000060 x8 : 3030626561666666
[ 9.794051] x7 : 6630313866666666 x6 : ffff000008f03f97
[ 9.799359] x5 : 0000000000000006 x4 : 000000000000000c
[ 9.804667] x3 : 0000000000010000 x2 : 0000000000010000
[ 9.809975] x1 : ffff000008da7be0 x0 : 0000000000000050
[ 10.517213] Call trace:
[ 10.519651] Exception stack(0xffff800ffbe0f340 to 0xffff800ffbe0f470)
[ 10.526081] f340: ffff7fe043f80000 0001000000000000 ffff800ffbe0f510 ffff0000081ec7d0
[ 10.533900] f360: ffff000008f03988 0000000008da7bc8 ffff800ffbe0f410 ffff0000081275fc
[ 10.541718] f380: ffff800ffbe0f470 ffff000008ac5a00 ffff7fe043ffffc0 0000000000000000
[ 10.549536] f3a0: 0000000000000001 ffff810ffffaf0e0 000000000000000c 000000000000000c
[ 10.557355] f3c0: ffff7fe043f80000 ffff7fe043f80020 0000000000000030 0000000000000000
[ 10.565173] f3e0: 0000000000000050 ffff000008da7be0 0000000000010000 0000000000010000
[ 10.572991] f400: 000000000000000c 0000000000000006 ffff000008f03f97 6630313866666666
[ 10.580809] f420: 3030626561666666 0000000000000060 0000000000000536 3d656e6f7a203a64
[ 10.588628] f440: 6630303866666666 3038336566666666 6e2c303d64696e2c ffff000088f03f37
[ 10.596446] f460: 0000000100000000 0000000000000000
[ 10.601316] [<ffff0000081ec7d0>] move_freepages+0x160/0x168
[ 10.606879] [<ffff0000081ec880>] move_freepages_block+0xa8/0xb8
[ 10.612788] [<ffff0000081ecf80>] __rmqueue+0x610/0x670
[ 10.617918] [<ffff0000081ee2e4>] get_page_from_freelist+0x3cc/0xb40
[ 10.624174] [<ffff0000081ef05c>] __alloc_pages_nodemask+0x12c/0xd40
[ 10.630438] [<ffff000008244cd0>] alloc_page_interleave+0x60/0xb0
[ 10.636434] [<ffff000008245398>] alloc_pages_current+0x108/0x168
[ 10.642430] [<ffff0000081e49ac>] __page_cache_alloc+0x104/0x140
[ 10.648339] [<ffff0000081e4b00>] pagecache_get_page+0x118/0x2e8
[ 10.654248] [<ffff0000081e4d18>] grab_cache_page_write_begin+0x48/0x68
[ 10.660769] [<ffff000008298c08>] simple_write_begin+0x40/0x150
[ 10.666591] [<ffff0000081e47c0>] generic_perform_write+0xb8/0x1a0
[ 10.672674] [<ffff0000081e6228>] __generic_file_write_iter+0x178/0x1c8
[ 10.679191] [<ffff0000081e6344>] generic_file_write_iter+0xcc/0x1c8
[ 10.685448] [<ffff00000826d12c>] __vfs_write+0xcc/0x140
[ 10.690663] [<ffff00000826de08>] vfs_write+0xa8/0x1c0
[ 10.695704] [<ffff00000826ee34>] SyS_write+0x54/0xb0
[ 10.700666] [<ffff000008bf2008>] xwrite+0x34/0x7c
[ 10.705359] [<ffff000008bf20ec>] do_copy+0x9c/0xf4
[ 10.710140] [<ffff000008bf1dc4>] write_buffer+0x34/0x50
[ 10.715354] [<ffff000008bf1e28>] flush_buffer+0x48/0xb8
[ 10.720579] [<ffff000008c1faa0>] __gunzip+0x27c/0x324
[ 10.725620] [<ffff000008c1fb60>] gunzip+0x18/0x20
[ 10.730314] [<ffff000008bf26dc>] unpack_to_rootfs+0x168/0x280
[ 10.736049] [<ffff000008bf2864>] populate_rootfs+0x70/0x138
[ 10.741615] [<ffff000008082ff4>] do_one_initcall+0x44/0x138
[ 10.747179] [<ffff000008bf0d0c>] kernel_init_freeable+0x1ac/0x24c
[ 10.753267] [<ffff000008859f78>] kernel_init+0x20/0xf8
[ 10.758395] [<ffff000008082b80>] ret_from_fork+0x10/0x50
[ 10.763698] Code: 17fffff2 b00046c0 91280000 97ffd47d (d4210000)
[ 10.769834] ---[ end trace 972d622f64fd69c0 ]---
More information about the linux-arm-kernel
mailing list