Unnecessary Thread Migrations on multi-core CPU

[Eric Magil]

Hello, I have been investigating a network performance problem on kernel 4.14.226-rt109 (RT patchset) running on a custom 4-core ARMv7 (32-bit) SoC.

We have the following setup to test network performance to the ARM board  in UDP mode:
- ARM is running iperf -s -u -i5
- A desktop machine is running the iperf client: iperf -c 192.168.1.100 -u -b1000m
- They are connected by an unmanaged gigabit switch over ethernet

Normally, we observe about ~10% packet loss with our kernel. However, after about 5 minutes, the packet loss spikes to 50% and stays there until we reboot the system. This coincides with the moment the jiffies wrap-around to 0.

The packet loss seems to be caused by a number of redundant task migrations of iperf, as observed using ftrace:

     ksoftirqd/3-39    [003]   320.481021: sched_migrate_task:   comm=iperf pid=1577 prio=120 orig_cpu=0 dest_cpu=3
     ksoftirqd/0-8     [000]   320.481203: sched_migrate_task:   comm=iperf pid=1577 prio=120 orig_cpu=3 dest_cpu=0

One the migration is caused by run_rebalance_domains(), the second is caused by wake_wide() in kernel/sched/fair.c.

The first issue seems to be  that time_after(jiffies, current->wakee_flip_decay_ts + HZ)  in fair.c:5337 is always false until the jiffies wrap around, since wakee_flip_decay_ts is initialized to 0 and a long is 32 bits on this system.

The second issue is that wake_wide() appears to reschedule the task on the CPU it was just migrated from. I found the following patch to address this from 2017, which seems to solve the issue: http://lkml.iu.edu/hypermail/linux/kernel/1708.0/00269.html

I just wanted to know if anyone else has seen similar behavior? It seems like this involves scheduling code that has been around for some time (and hasn't changed in the latest upstream kernel), so I wanted to know if I'm missing something obvious that could be causing this issue. 

Thank you for taking the time,

-Eric