[LEDE-DEV] ramips: Error in RXCSUM setting? Or, performance improvement.

[rosenp at gmail.com]

I received my new mt7621 toy today and noticed something interesting
when playing with ethtool. I use ethtool to remove most offloads in the
driver as offloads increase latency. But something caught my eye. If I
run "ethtool -K eth0 rx off", iperf performance improves from ~350 to
~450. I ran multiple times to make sure. I looked at the code and it
seems that for mt7621, it's supposed to disable RXCSUM but for some
reason it fails. Anyone have any insight into this?

Anyway, here are results in case nobody believes me. Anomalies are also
present for some reason.

rx = on (default).

root at LEDE:/# iperf -e -s
------------------------------------------------------------
Server listening on TCP port 5001 with pid 1232
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55880
[ ID]
Interval        Transfer    Bandwidth       Reads   Dist(bin=16.0K)
[  4] 0.00-10.23 sec   730 MBytes   599
Mbits/sec  8041    1322:274:576:283:167:215:161:5043
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55882
[  5] 0.00-10.23 sec   417 MBytes   342
Mbits/sec  4172    117:751:78:34:29:31:32:3100
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55884
[  4] 0.00-10.23 sec   383 MBytes   314
Mbits/sec  4460    294:1116:81:87:83:70:84:2645
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55886
[  5] 0.00-10.24 sec   424 MBytes   347
Mbits/sec  5318    381:1479:261:140:115:96:93:2753
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55888
[  4] 0.00-10.23 sec   375 MBytes   307
Mbits/sec  4628    324:1317:156:85:72:78:78:2518
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55892
[  5] 0.00-10.23 sec   469 MBytes   385
Mbits/sec  4859    229:792:130:105:93:85:74:3351
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55894
[  4] 0.00-10.23 sec   385 MBytes   316
Mbits/sec  4372    227:1064:110:79:68:75:68:2681
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55896
[  5] 0.00-10.23 sec   365 MBytes   299
Mbits/sec  4724    358:1460:175:93:87:72:87:2392
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55900
[  4] 0.00-10.23 sec   368 MBytes   302
Mbits/sec  4653    347:1238:151:157:160:135:134:2331
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55902
[  5] 0.00-10.23 sec   679 MBytes   557
Mbits/sec  9340    2272:401:1382:377:232:300:234:4142
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55904
[  4] 0.00-10.23 sec   406 MBytes   333
Mbits/sec  4314    207:829:88:100:86:63:67:2874

rx off.

root at LEDE:/# ethtool -K eth0 rx off
root at LEDE:/# iperf -e -s
------------------------------------------------------------
Server listening on TCP port 5001 with pid 1224
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55930
[ ID]
Interval        Transfer    Bandwidth       Reads   Dist(bin=16.0K)
[  4] 0.00-10.23 sec   568 MBytes   466
Mbits/sec  19631    3180:6239:10139:59:6:2:0:6
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55940
[  5] 0.00-10.23 sec   520 MBytes   426
Mbits/sec  16300    2111:3702:9653:531:145:55:36:67
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55942
[  4] 0.00-10.23 sec   573 MBytes   470
Mbits/sec  17210    1954:2241:12278:663:65:5:4:0
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55944
[  5] 0.00-10.23 sec   530 MBytes   434
Mbits/sec  18502    3268:5288:9772:168:3:0:2:1
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55946
[  4] 0.00-10.24 sec   541 MBytes   443
Mbits/sec  17707    3066:2646:11882:103:0:0:0:10
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55948
[  5] 0.00-10.23 sec   570 MBytes   467
Mbits/sec  17166    2053:2517:11497:1089:6:2:0:2
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55950
[  4] 0.00-10.23 sec   543 MBytes   446
Mbits/sec  16826    2718:3605:9600:697:8:8:4:186
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55952
[  5] 0.00-10.23 sec   568 MBytes   466
Mbits/sec  17187    2004:2412:11973:792:4:2:0:0
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55954
[  4] 0.00-10.24 sec   581 MBytes   476
Mbits/sec  16425    1503:2227:10757:1891:28:14:4:1
[  5] local 192.168.1.1 port 5001 connected with 192.168.1.130 port
55956
[  5] 0.00-10.23 sec   541 MBytes   443
Mbits/sec  16189    1861:2487:11090:748:0:2:1:0