[FS#972] Ubiquiti EdgeRouter X (ER-X) port connection interruption & kernel crash (sometimes)

Sun Aug 20 23:52:26 PDT 2017

A new Flyspray task has been opened.  Details are below. 

User who did this - Guenther (guenti_r) 

Attached to Project - LEDE Project
Summary - Ubiquiti EdgeRouter X (ER-X) port connection interruption & kernel crash (sometimes)
Task Type - Bug Report
Category - Base system
Status - Unconfirmed
Assigned To - 
Operating System - All
Severity - Medium
Priority - Very Low
Reported Version - lede-17.01
Due in Version - Undecided
Due Date - Undecided
Details - 
 - Device: Ubiquiti EdgeRouter X (ER-X)
 - LEDE version: 17.01.2

We have several devices running, with 17.01.1 all is rock solid.
After upgrading to 17.01.2 we have problems with port connection interruption and sometimes kernel crashes.

Other people have similar experiences -> [[https://forum.lede-project.org/t/er-x-lan-port-connection-interruption/5709]]

Port connection interruption log:

Mon Aug 21 07:28:57 2017 kern.info kernel: [69826.410000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link down
Mon Aug 21 07:28:59 2017 kern.info kernel: [69828.710000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link up
Mon Aug 21 07:29:05 2017 kern.info kernel: [69833.950000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link down
Mon Aug 21 07:29:34 2017 kern.info kernel: [69863.260000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link up
Mon Aug 21 07:29:53 2017 kern.info kernel: [69882.840000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link down
Mon Aug 21 07:30:01 2017 kern.info kernel: [69890.820000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link up
Mon Aug 21 07:30:02 2017 kern.info kernel: [69891.570000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link down
Mon Aug 21 07:30:19 2017 kern.info kernel: [69908.040000] mtk_soc_eth 1e100000.ethernet eth0: port 3 link up

Kernel crash log:

Sun Aug 20 15:39:45 2017 kern.err kernel: [12873.930000] INFO: rcu_sched detected stalls on CPUs/tasks:
Sun Aug 20 15:39:45 2017 kern.err kernel: [12873.940000] 	3-...: (0 ticks this GP) idle=88c/0/0 softirq=786546/786546 fqs=0 
Sun Aug 20 15:39:45 2017 kern.err kernel: [12873.950000] 	(detected by 1, t=6004 jiffies, g=23478, c=23477, q=130)
Sun Aug 20 15:39:45 2017 kern.info kernel: [12873.960000] Task dump for CPU 3:
Sun Aug 20 15:39:45 2017 kern.info kernel: [12873.970000] swapper/3       R running      0     0      1 0x00100000
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] Stack : 00000000 8fc53c80 00000bb8 ffffffff 00000a4f 00000000 804762a4 80420000
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 80420000 8000c0e0
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8000c410 1100fc03 00000003
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] 	  804762a4 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12873.980000] 	  ...
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.050000] Call Trace:[] 0x8000c0e0
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.060000] [] 0x8000c410
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.070000] [] 0x8005dc70
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.080000] [] 0x8001acc0
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.080000] [] 0x8001acc8
Sun Aug 20 15:39:45 2017 kern.warn kernel: [12874.090000] 
Sun Aug 20 15:39:45 2017 kern.err kernel: [12874.090000] rcu_sched kthread starved for 6018 jiffies! g23478 c23477 f0x0 s3 ->state=0x1
Sun Aug 20 15:40:47 2017 kern.err kernel: [12936.490000] INFO: rcu_sched detected stalls on CPUs/tasks:
Sun Aug 20 15:40:47 2017 kern.err kernel: [12936.500000] 	3-...: (0 ticks this GP) idle=a0c/0/0 softirq=786546/786546 fqs=0 
Sun Aug 20 15:40:47 2017 kern.err kernel: [12936.510000] 	(detected by 1, t=6004 jiffies, g=23490, c=23489, q=159)
Sun Aug 20 15:40:47 2017 kern.info kernel: [12936.520000] Task dump for CPU 3:
Sun Aug 20 15:40:47 2017 kern.info kernel: [12936.530000] swapper/3       R running      0     0      1 0x00100000
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] Stack : 00000000 8fc53c80 00000bc6 ffffffff 00000a59 00000000 804762a4 80420000
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 80420000 8000c0e0
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8000c410 1100fc03 00000003
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] 	  804762a4 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.540000] 	  ...
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.610000] Call Trace:[] 0x8000c0e0
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.620000] [] 0x8000c410
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.630000] [] 0x8005dc70
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.640000] [] 0x8001acc0
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.640000] [] 0x8001acc8
Sun Aug 20 15:40:47 2017 kern.warn kernel: [12936.650000] 
Sun Aug 20 15:40:47 2017 kern.err kernel: [12936.650000] rcu_sched kthread starved for 6018 jiffies! g23490 c23489 f0x0 s3 ->state=0x1
Sun Aug 20 15:41:52 2017 kern.err kernel: [13000.910000] INFO: rcu_sched detected stalls on CPUs/tasks:
Sun Aug 20 15:41:52 2017 kern.err kernel: [13000.920000] 	3-...: (0 ticks this GP) idle=b20/0/0 softirq=786546/786546 fqs=0 
Sun Aug 20 15:41:52 2017 kern.err kernel: [13000.930000] 	(detected by 1, t=6004 jiffies, g=23496, c=23495, q=184)
Sun Aug 20 15:41:52 2017 kern.info kernel: [13000.940000] Task dump for CPU 3:
Sun Aug 20 15:41:52 2017 kern.info kernel: [13000.950000] swapper/3       R running      0     0      1 0x00100000
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] Stack : 00000000 0e77bc1c 00000bd5 ffffffff 00000a65 00401b75 804762a4 80420000
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 000010d9 80013554
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8005dc78 1100fc03 00000003
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] 	  00000000 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13000.960000] 	  ...
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.030000] Call Trace:[] 0x80013554
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.040000] [] 0x8005dc78
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.050000] [] 0x8005dc70
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.060000] [] 0x8001acc0
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.060000] [] 0x8001acc8
Sun Aug 20 15:41:52 2017 kern.warn kernel: [13001.070000] 
Sun Aug 20 15:41:52 2017 kern.err kernel: [13001.070000] rcu_sched kthread starved for 6018 jiffies! g23496 c23495 f0x0 s3 ->state=0x1
Sun Aug 20 15:43:01 2017 kern.err kernel: [13069.970000] INFO: rcu_sched detected stalls on CPUs/tasks:
Sun Aug 20 15:43:01 2017 kern.err kernel: [13069.980000] 	3-...: (0 ticks this GP) idle=c38/0/0 softirq=786546/786546 fqs=0 
Sun Aug 20 15:43:01 2017 kern.err kernel: [13069.990000] 	(detected by 1, t=6004 jiffies, g=23508, c=23507, q=179)
Sun Aug 20 15:43:01 2017 kern.info kernel: [13070.000000] Task dump for CPU 3:
Sun Aug 20 15:43:01 2017 kern.info kernel: [13070.010000] swapper/3       R running      0     0      1 0x00100000
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] Stack : 00000000 8fc53c80 00000be3 ffffffff 00000a77 00000000 804762a4 80420000
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 80420000 8000c0e0
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8000c410 1100fc03 00000003
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] 	  804762a4 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.020000] 	  ...
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.090000] Call Trace:[] 0x8000c0e0
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.100000] [] 0x8000c410
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.110000] [] 0x8005dc70
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.120000] [] 0x8001acc0
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.120000] [] 0x8001acc8
Sun Aug 20 15:43:01 2017 kern.warn kernel: [13070.130000] 
Sun Aug 20 15:43:01 2017 kern.err kernel: [13070.130000] rcu_sched kthread starved for 6018 jiffies! g23508 c23507 f0x0 s3 ->state=0x1
Mon Aug 21 01:52:45 2017 kern.err kernel: [49654.240000] INFO: rcu_sched detected stalls on CPUs/tasks:
Mon Aug 21 01:52:45 2017 kern.err kernel: [49654.250000] 	3-...: (0 ticks this GP) idle=8be/0/0 softirq=3072493/3072493 fqs=0 
Mon Aug 21 01:52:45 2017 kern.err kernel: [49654.260000] 	(detected by 1, t=6004 jiffies, g=90843, c=90842, q=104)
Mon Aug 21 01:52:45 2017 kern.info kernel: [49654.270000] Task dump for CPU 3:
Mon Aug 21 01:52:45 2017 kern.info kernel: [49654.280000] swapper/3       R running      0     0      1 0x00100000
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] Stack : 00000000 00000001 00000005 00000000 00000000 00000001 804762a4 80420000
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 00000000 80013554
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8005dc78 1100fc03 00000003
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] 	  00000000 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.290000] 	  ...
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.360000] Call Trace:[] 0x80013554
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.370000] [] 0x8005dc78
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.380000] [] 0x8005dc70
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.390000] [] 0x8001acc0
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.390000] [] 0x8001acc8
Mon Aug 21 01:52:45 2017 kern.warn kernel: [49654.400000] 
Mon Aug 21 01:52:45 2017 kern.err kernel: [49654.400000] rcu_sched kthread starved for 6018 jiffies! g90843 c90842 f0x0 s3 ->state=0x1
Mon Aug 21 01:53:51 2017 kern.err kernel: [49720.510000] INFO: rcu_sched detected stalls on CPUs/tasks:
Mon Aug 21 01:53:51 2017 kern.err kernel: [49720.520000] 	3-...: (0 ticks this GP) idle=a3e/0/0 softirq=3072493/3072493 fqs=0 
Mon Aug 21 01:53:51 2017 kern.err kernel: [49720.530000] 	(detected by 1, t=6004 jiffies, g=90851, c=90850, q=139)
Mon Aug 21 01:53:51 2017 kern.info kernel: [49720.540000] Task dump for CPU 3:
Mon Aug 21 01:53:51 2017 kern.info kernel: [49720.550000] swapper/3       R running      0     0      1 0x00100000
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] Stack : 00000000 00000001 00000005 00000000 00000000 00000001 804762a4 80420000
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] 	  8042475c 00000001 00000001 80424680 80424724 80420000 00000000 80013554
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] 	  1100fc03 00000003 8fc74000 8fc75ec0 80420000 8005dc78 1100fc03 00000003
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] 	  00000000 80420000 804762a4 8005dc70 80420000 8001acc0 1100fc03 00000000
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] 	  00000004 804244a0 000000a0 8001acc8 f99b5fff 5dfcf51e fccb7bd5 ff7fca95
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.560000] 	  ...
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.630000] Call Trace:[] 0x80013554
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.640000] [] 0x8005dc78
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.650000] [] 0x8005dc70
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.660000] [] 0x8001acc0
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.660000] [] 0x8001acc8
Mon Aug 21 01:53:51 2017 kern.warn kernel: [49720.670000] 
Mon Aug 21 01:53:51 2017 kern.err kernel: [49720.670000] rcu_sched kthread starved for 6018 jiffies! g90851 c90850 f0x0 s3 ->state=0x1

More information can be found at the following URL:
https://bugs.lede-project.org/index.php?do=details&task_id=972