sched/fair: NULL deref in pick_next_task_fair (v4.19) -- tasks_timeline empty, nr_running == -1 (two production dumps)
Qiurong Fang
fangqiurong at kylinos.cn
Mon Aug 24 01:19:59 PDT 2026
Hello,
We have encountered the same crash signature in two unrelated production
environments, each with a kdump-captured vmcore. Both crashes occurred
after long uptime (738 days / 290 days) and dereferenced NULL in
pick_next_task_fair(). The root cfs_rq state on the crashing CPU is
identical in both cases.
=== 1. Environment ===
Site A:
- aarch64, HiSilicon Kunpeng 920 (HIP08), 96 CPUs(NUMA: 4 nodes, 24 CPUs/node), 382 GB
- Kernel 4.19.90-23.43.v2101 (vendor distro kernel based on 4.19), GCC 7.3
- Uptime 738 days;
Site B:
- aarch64, HiSilicon Kunpeng 920 (HIP08), 128 CPUs(NUMA: 4 nodes, 32 CPUs/node), 1 TB
- Kernel 4.19.90-89.17.v2401 (newer vendor build), GCC 7.3
- Uptime 290 days
=== 2. Crash site ===
In both cases the NULL comes from the same place: the non-zero count
prevents the idle path, so pick_next_entity() takes rb_leftmost from
an empty tree and gets NULL.
again:
if (!cfs_rq->nr_running) /* 0xFFFFFFFF != 0, no idle */
goto idle;
--- Site A dmesg ---
[63777450.752746] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000160
[63777450.753146] Mem abort info:
[63777450.753299] ESR = 0x96000005
...
[63777450.755251] Internal error: Oops: 96000005 [#1] SMP
[63777450.755489] Modules linked in: iptable_filter udp_diag raw_diag unix_diag af_packet_diag netlink_diag tcp_diag inet_diag bonding vfat fat ipmi_ssif aes_ce_blk crypto_simd cryptd aes_ce_cipher crct10dif_ce ghash_ce sha2_ce sha256_arm64 sha1_ce hns_roce_hw_v2 hns_roce ib_uverbs ses sbsa_gwdt ib_core enclosure ofpart cmdlinepart ipmi_si ipmi_devintf hi_sfc ipmi_msghandler mtd spi_dw_mmio sch_fq_codel binfmt_misc ip_tables realtek hinic hns3 hclge megaraid_sas hisi_sas_v3_hw hnae3 hisi_sas_main host_edma_drv
[63777450.757458] Process ksoftirqd/65 (pid: 339, stack limit = 0x000000001f6f62e2)
[63777450.764478] CPU: 65 PID: 339 Comm: ksoftirqd/65 Kdump: loaded Not tainted 4.19.90-23.43.v2101.ky10.aarch64 #1
[63777450.778683] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDDA, BIOS 1.38 07/04/2020
[63777450.793264] pstate: 60c00089 (nZCv daIf +PAN +UAO)
[63777450.800561] pc : pick_next_task_fair+0x88/0x640
[63777450.807710] lr : pick_next_task_fair+0x88/0x640
[63777450.814599] sp : ffffa037c3d7bd10
[63777450.821394] x29: ffffa037c3d7bd10 x28: ffff000048c5e248
...
[63777450.911587] x1 : 0000000000000000 x0 : 0000000000000000
[63777450.917053] Call trace:
[63777450.922275] pick_next_task_fair+0x88/0x640
[63777450.927392] __schedule+0x71c/0x958
[63777450.932458] schedule+0x2c/0x88
[63777450.937190] smpboot_thread_fn+0x1b4/0x1b8
[63777450.941813] kthread+0x134/0x138
[63777450.946248] ret_from_fork+0x10/0x18
[63777450.950520] Code: 54000221 aa1403e0 aa1303e1 97ffcf29 (f940b014)
Corresponding source (4.19 pick_next_task_fair):
se = pick_next_entity(cfs_rq, curr); /* returns NULL */
cfs_rq = group_cfs_rq(se); /* NULL->my_q, +0x160, crash */
--- Site B dmesg ---
[25097461.360687] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000040
[25097461.370128] Mem abort info:
[25097461.373601] ESR = 0x96000005
...
[25097461.417991] Internal error: Oops: 96000005 [#1] SMP
[25097461.423539] Modules linked in: nfsd nfs_acl binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc ipmi_ssif vfat fat aes_ce_blk crypto_simd joydev cryptd aes_ce_cipher crct10dif_ce ofpart ghash_ce cmdlinepart ipmi_si sha1_ce ses sbsa_gwdt ky_lpcmux enclosure hi_sfc ipmi_devintf mtd ipmi_msghandler spi_dw_mmio sch_fq_codel ip_tables sha2_ce nvme hinic sha256_arm64 nvme_core nfit hisi_sas_v3_hw nvme_common ngbe megaraid_sas libnvdimm hisi_sas_main host_edma_drv dm_mirror dm_region_hash dm_log
[25097461.471273] Process T1004_LockWaitM (pid: 4090813, stack limit = 0x0000000096656505)
[25097461.479672] CPU: 92 PID: 4090813 Comm: T1004_LockWaitM Kdump: loaded Not tainted 4.19.90-89.17.v2401.ky10.aarch64 #1
[25097461.497852] Hardware name: PowerLeader PR210K/BC82AMDYA, BIOS 6.57 05/17/2023
[25097461.505644] pstate: 60400089 (nZCv daIf +PAN -UAO)
[25097461.511111] pc : set_next_entity+0x20/0x6f8
[25097461.515968] lr : pick_next_task_fair+0x2e8/0x548
[25097461.521255] sp : ffff8041a56d7c60
[25097461.525245] x29: ffff8041a56d7c60 x28: ffff8040a2593800
...
[25097461.549161] x21: 0000000000000000 x20: 0000000000000000
...
[25097461.608946] x1 : 0000000000000000 x0 : ffff561d3afdf1d8
[25097461.614926] Call trace:
[25097461.618054] set_next_entity+0x20/0x6f8
[25097461.622563] pick_next_task_fair+0x2e8/0x548
[25097461.627509] __schedule+0x6e4/0x840
[25097461.631673] schedule+0x30/0xe0
[25097461.635493] do_nanosleep+0x78/0x168
[25097461.639747] hrtimer_nanosleep+0x88/0x100
[25097461.644429] sys_nanosleep+0x94/0xb8
[25097461.648680] el0_svc_naked+0x44/0x48
[25097461.652931] Code: aa0103f4 aa0003f3 aa1e03e0 d503201f (b9404280)
Corresponding source (4.19 pick_next_task_fair, simple label):
se = pick_next_entity(cfs_rq, NULL); /* returns NULL */
set_next_entity(cfs_rq, se); /* se->on_rq, +0x40, crash */
=== 3. Root cfs_rq / rq state from vmcore ===
field observed value
------------------------------------------- --------------------
rq->nr_running 0
rq->load.weight 2^64 - 1048576
cfs_rq->nr_running 0xFFFFFFFF (-1)
cfs_rq->h_nr_running 0xFFFFFFFF (-1)
cfs_rq->load.weight 0
tasks_timeline->rb_root.rb_node/rb_leftmost NULL / NULL
rq->cfs_tasks list length 0
rq->curr task se (se->on_rq=0)
The two sites use hardware from different vendors and kernel builds
about a year apart. This issue has only been seen on Kunpeng 920 arm64
(kernel 4.19); it has not appeared on x86 (kernel 4.19). The corruption
affects three fields (cfs.nr -1, cfs.h_nr -1, rq.load -1048576, the
latter being exactly one nice-0 task weight NICE_0_LOAD).
--- Site A (cpu 65) ---
struct rq (cpu 65):
.nr_running = (unsigned int)0
.load.weight = (unsigned long)18446744073708503040
.nr_switches = (u64)113264737287
...
.curr->pid = (pid_t)339
.curr->on_rq = (int)0
.cfs_tasks len = 0
struct cfs_rq (root, cpu 65):
(struct cfs_rq){
.load = (struct load_weight){
.weight = (unsigned long)0,
.inv_weight = (u32)0,
},
.runnable_weight = (unsigned long)0,
.nr_running = (unsigned int)4294967295,
.h_nr_running = (unsigned int)4294967295,
.exec_clock = (u64)0,
.min_vruntime = (u64)12868847681674577,
.tasks_timeline = (struct rb_root_cached){
.rb_root = (struct rb_root){
.rb_node = (struct rb_node *)0x0,
},
.rb_leftmost = (struct rb_node *)0x0,
},
.curr = (struct sched_entity *)0xffffa037c3d83ec0,
.next = (struct sched_entity *)0x0,
.last = (struct sched_entity *)0x0,
.skip = (struct sched_entity *)0x0,
...
.tg = (struct task_group *)root_task_group+0x0 = 0xffff000049a364c0,
.runtime_enabled = (int)0,
.throttled = (int)0,
.throttle_count = (int)0,
...
}
--- Site B (cpu 92) ---
struct rq (cpu 92):
.nr_running = (unsigned int)0
.load.weight = (unsigned long)18446744073708503040
.nr_switches = (u64)77284029461
...
.curr->pid = (pid_t)4090813
.curr->on_rq = (int)0
.cfs_tasks len = 0
struct cfs_rq (root, cpu 92):
(struct cfs_rq){
.load = (struct load_weight){
.weight = (unsigned long)0,
.inv_weight = (u32)0,
},
.runnable_weight = (unsigned long)0,
.nr_running = (unsigned int)4294967295,
.h_nr_running = (unsigned int)4294967295,
.exec_clock = (u64)0,
.min_vruntime = (u64)1181660162701980,
.tasks_timeline = (struct rb_root_cached){
.rb_root = (struct rb_root){
.rb_node = (struct rb_node *)0x0,
},
.rb_leftmost = (struct rb_node *)0x0,
},
.curr = (struct sched_entity *)0x0,
.next = (struct sched_entity *)0x0,
.last = (struct sched_entity *)0x0,
.skip = (struct sched_entity *)0x0,
...
.tg = (struct task_group *)root_task_group+0x0 = 0xffff561d3d1b6100,
.runtime_enabled = (int)0,
.throttled = (int)0,
.throttle_count = (int)0,
.idle_h_nr_running = (unsigned int)0,
...
}
=== 4. Code analysis ===
Verified against the kernel source tag: the sole writers of root
cfs_rq.nr_running, root cfs_rq.load and rq.load in the entire scheduler
are the following two functions. Both are called under rq->lock, so no
race condition is expected:
account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
update_load_add(&cfs_rq->load, se->load.weight);
if (!parent_entity(se))
update_load_add(&rq_of(cfs_rq)->load, se->load.weight);
... /* cfs_tasks list, numa accounting, irrelevant here */
cfs_rq->nr_running++;
}
account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
update_load_sub(&cfs_rq->load, se->load.weight);
if (!parent_entity(se))
update_load_sub(&rq_of(cfs_rq)->load, se->load.weight);
...
cfs_rq->nr_running--;
}
For the three fields visible on a given rq, each accounting call has one
of three effects, and (-1, 0, -w) is not in the reachable set:
(a) root-level entity enqueue: (cfs.nr +1, cfs.load +w, rq.load +w)
(b) root-level entity dequeue: (cfs.nr -1, cfs.load -w, rq.load -w)
(c) non-root-level entity: (0, 0, 0)
>From source inspection, the sole writers of nr_running and load are
account_entity_enqueue / account_entity_dequeue,both called under
rq->lock protection, which makes the observed field state hard to
explain.
Has anyone seen a similar issue on arm64? Any debugging suggestions
would be appreciated.
The full panic dmesg for both crashes is posted above. Both crashes have
kdump-captured vmcores; any data from them (struct fields, disassembly,
etc.) can be queried on request and reported back.
--
Thanks
More information about the linux-arm-kernel
mailing list