[FS#804] mt7621: kernel errors - rcu_sched detected stalls on CPUs/tasks

Tue Jul 25 06:32:57 PDT 2017

The following task has a new comment added:

FS#804 - mt7621: kernel errors - rcu_sched detected stalls on CPUs/tasks
User who did this - camel (camel)

----------
meanwhile .. i tested with traffic shapping ... and produced a lot of traff ic on:
.) LAN _=> ~10GB transfer
.) WLAN 5Ghz _=> ~15GB transfer
.) WLAN 2,4Ghz => ~5GB transfer

and got few errors ....

Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.520000] luci invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), nodemask=0, order=1, oom_score_adj=0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.530000] COMPACTION is disabled!!!
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] CPU: 2 PID: 32685 Comm: luci Not tainted 4.9.37 #0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.540000] Stack : 00000000 00000000 80537b2a 00000032 803f4084 00000000 00000000 80530000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.550000]         81fa462c 804d7da7 8046dff0 00000002 00007fad 80533824 00000001 00200000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.560000]         00001321 80069890 00000000 800696b0 00000000 00000004 80472c00 82745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.570000]         00000000 800a5d98 00000000 00000000 80537b2a 00000000 82745d28 00745c3c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000]         00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.580000]         ...
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] Call Trace:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [] show_stack+0x54/0x88
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.590000] [] dump_stack+0x84/0xc0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [] dump_header.isra.4+0x84/0x1b4
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.600000] [] oom_kill_process+0xd0/0x484
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [] out_of_memory+0x3bc/0x3fc
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.610000] [] __alloc_pages_nodemask+0x5e4/0xa58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [] copy_process.isra.8.part.9+0x10c/0x1300
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.620000] [] _do_fork+0xcc/0x2d8
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [] SyS_clone+0x20/0x2c
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.630000] [] syscall_common+0x34/0x58
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] Mem-Info:
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000] active_anon:13977 inactive_anon:1643 isolated_anon:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  active_file:805 inactive_file:2995 isolated_file:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  unevictable:1 dirty:2 writeback:0 unstable:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  slab_reclaimable:3926 slab_unreclaimable:35491
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  mapped:3433 shmem:8024 pagetables:227 bounce:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.640000]  free:49832 free_pcp:28 free_cma:0
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.670000] Node 0 active_anon:55908kB inactive_anon:6572kB active_file:3220kB inactive_file:11980kB unevictable:4kB isolated(anon):0kB isolated(file):0kB mapped:13732kB dirty:8kB writeback:0kB shmem:32096kB writeback_tmp:0kB unstable:0kB pages_scanned:0 all_unreclaimable? no
Tue Jul 25 12:50:04 2017 kern.warn kernel: [ 1065.700000] Normal free:20008kB min:16384kB low:20480kB high:24576kB active_anon:0kB inactive_anon:0kB active_file:4kB inactive_file:84kB unevictable:0kB writepending:8kB present:262144kB managed:251688kB mlocked:0kB slab_reclaimable:15704kB slab_unreclaimable:141964kB kernel_stack:56192kB pagetables:908kB bounce:0kB free_pcp:232kB local_pcp:0kB free_cma:0kB
Tue Jul 25 12:50:05 2017 kern.emerg kernel: lowmem_reserve[]: 0 2048 2048

Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.840000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.840000] 	3-...: (0 ticks this GP) idle=c95/140000000000000/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.850000] 	(detected by 2, t=6003 jiffies, g=66650, c=66649, q=150961)
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.860000] Task dump for CPU 3:
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.860000] luci            R  running task        0 21284   1964 0x08100000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.870000] Stack : 00000000 8baf8c38 80528b80 8c0a9a98 8c0a9a98 8fc02380 8d0fe180 000000bb
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.870000]         8fc02680 8ecc0080 8ecc2180 800e5fd0 8ecc2180 800e79c0 00000001 811dc800
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.880000]         8c0a9b20 800af53c 00000000 804e0000 8149499c ffffffff 00000001 00008159
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.890000]         8d5676a8 0000005a 00000058 800e55c0 024102c0 802d9e80 00210d00 804d55f4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.900000]         00000000 800afce0 026012d0 80479f14 811dc800 0000000f 00210d00 804d55f4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000]         ...
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000] Call Trace:
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.910000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:13:39 2017 kern.err kernel: [ 2479.920000] rcu_sched kthread starved for 2050 jiffies! g66650 c66649 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:13:39 2017 kern.info kernel: [ 2479.930000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.930000] Stack : 814a9bc0 8fc61250 8fc89e50 814b4320 8fc89e50 00000000 00000001 8007b658
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.940000]         814b4320 00034b87 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.950000]         804dd5f0 803dc574 00034b87 804d0000 8fc89e50 814b4320 00034b87 803deed8
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.960000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b4710
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.970000]         00034b87 8007bb54 8fc61250 39800003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.970000]         ...
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] Call Trace:
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.980000] [] schedule+0x6c/0x84
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.990000] [] schedule_timeout+0x160/0x19c
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2479.990000] [] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2480.000000] [] kthread+0xd8/0xec
Tue Jul 25 13:13:39 2017 kern.warn kernel: [ 2480.000000] [] ret_from_kernel_thread+0x14/0x1c

Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.280000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.280000] 	3-...: (0 ticks this GP) idle=9c0/0/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.290000] 	(detected by 1, t=6003 jiffies, g=66656, c=66655, q=340412)
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.290000] Task dump for CPU 3:
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.300000] swapper/3       R  running task        0     0      1 0x00100000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.300000] Stack : 00000068 8fc62498 00000001 84ca6177 00000279 00000000 0000022e 009a3220
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.310000]         804d548c 00000008 80472554 804e0000 00000001 00000001 bfbf8000 00000003
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.320000]         00000000 803dc574 00000000 00000000 8fc94000 804d548c 00000008 800c5534
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.330000]         804d548c 00000008 80472554 804e0000 804d548c 8006304c 00000001 00000001
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.340000]         00000001 00000001 00000001 00000001 00000001 00000001 00000001 9fc0046c
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000]         ...
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000] Call Trace:
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.350000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.360000] [] schedule+0x6c/0x84
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.360000] [] quiet_vmstat+0x54/0x80
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.370000] [] cpu_startup_entry+0xec/0x150
Tue Jul 25 13:17:39 2017 kern.err kernel: [ 2720.370000] rcu_sched kthread starved for 2496 jiffies! g66656 c66655 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:17:39 2017 kern.info kernel: [ 2720.380000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.390000] Stack : 814a9bc0 8fc61250 8fc89e50 814b4320 8fc89e50 804d0000 00000001 8007b658
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.390000]         814b4320 0003a7b6 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.400000]         804dd5f0 803dc574 0003a7b6 804d0000 8fc89e50 814b4320 0003a7b6 803deed8
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.410000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b46c8
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.420000]         0003a7b6 8007bb54 8fc61250 35000003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000]         ...
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000] Call Trace:
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.430000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.440000] [] schedule+0x6c/0x84
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.440000] [] schedule_timeout+0x160/0x19c
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.450000] [] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.450000] [] kthread+0xd8/0xec
Tue Jul 25 13:17:39 2017 kern.warn kernel: [ 2720.460000] [] ret_from_kernel_thread+0x14/0x1c

Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.490000] INFO: rcu_sched detected stalls on CPUs/tasks:
Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.490000] 	3-...: (0 ticks this GP) idle=69c/0/0 softirq=302510/302510 fqs=1 
Tue Jul 25 13:19:54 2017 kern.err kernel: [ 2855.500000] 	(detected by 1, t=6003 jiffies, g=66673, c=66672, q=275515)
Tue Jul 25 13:19:54 2017 kern.info kernel: [ 2855.500000] Task dump for CPU 3:
Tue Jul 25 13:19:54 2017 kern.info kernel: [ 2855.510000] swapper/3       R  running task        0     0      1 0x00100000
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.510000] Stack : 00000068 00000001 00000001 001fd953 00000000 77edc2c0 00000000 00000000
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.520000]         804d548c 00000008 80472554 804e0000 00000001 00000001 bfbf8000 00000003
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.530000]         00000000 8000c1cc 00000000 00000000 8fc94000 8fc95f00 00000000 80063054
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.540000]         1100fc03 00000001 00000001 804d548c 804d548c 8006304c 00000001 00000001
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.550000]         00000001 00000001 00000001 00000001 00000001 00000001 00000001 9fc0046c
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.560000]         ...
Tue Jul 25 13:19:54 2017 kern.warn kernel: [ 2855.560000] Call Trace:
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.560000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.570000] [] r4k_wait_irqoff+0x0/0x20
Tue Jul 25 13:19:55 2017 kern.err kernel: [ 2855.570000] rcu_sched kthread starved for 3051 jiffies! g66673 c66672 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
Tue Jul 25 13:19:55 2017 kern.info kernel: [ 2855.580000] rcu_sched       S    0     7      2 0x00100000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.590000] Stack : 814b5bc0 8fc61250 8fc89e50 814b4320 8fc89e50 804d0000 00000001 8007b658
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.590000]         814b4320 0003da5b 814b4320 8fc89e50 804d0000 00000000 804e0000 804d0000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.600000]         804dd5f0 803dc574 0003da5b 804d0000 8fc89e50 814b4320 0003da5b 803deed8
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.610000]         804d55f4 804e0000 00000001 804d0000 814b6240 00000001 00000000 814b4870
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.620000]         0003da5b 8007bb54 8fc61250 4f800003 804dd4c0 00000001 804d0000 00000000
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000]         ...
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000] Call Trace:
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.630000] [] __schedule+0x5d4/0x7a4
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.640000] [] schedule+0x6c/0x84
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.640000] [] schedule_timeout+0x160/0x19c
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.650000] [] rcu_gp_kthread+0x7f4/0x7fc
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.650000] [] kthread+0xd8/0xec
Tue Jul 25 13:19:55 2017 kern.warn kernel: [ 2855.660000] [] ret_from_kernel_thread+0x14/0x1c

Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: Logfile: opening log stdio:/var/logs/netdb.state
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: ERROR: logfileOpen stdio:/var/logs/netdb.state: (2) No such file or directory
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: logfileOpen: stdio:/var/logs/netdb.state: couldn't open!
Tue Jul 25 13:20:21 2017 daemon.notice squid[2635]: netdbSaveState: stdio:/var/logs/netdb.state: (2) No such file or directory

----------

More information can be found at the following URL:
https://bugs.lede-project.org/index.php?do=details&task_id=804#comment3116