Slow ramp-up for single-stream TCP throughput on 4.2 kernel.

[Neal Cardwell]

On Sat, Oct 3, 2015 at 6:46 PM, Ben Greear <greearb at candelatech.com> wrote:
>
>
> On 10/03/2015 09:29 AM, Neal Cardwell wrote:
>>
>> On Fri, Oct 2, 2015 at 8:21 PM, Ben Greear <greearb at candelatech.com>
>> wrote:
>>>
>>> Gah, seems 'cubic' related.  That is the default tcp cong ctrl
>>> I was using (same in 3.17, for that matter).
>>
>>
>> There have been recent changes to CUBIC that may account for this. If
>> you could repeat your test with more instrumentation, eg "nstat", that
>> would be very helpful.
>>
>> nstat > /dev/null
>> # run one test
>> nstat
>>
>> Also, if you could take a sender-side tcpdump trace of the test, that
>> would be very useful (default capture length, grabbing just headers,
>> is fine).
>
>
> Here is nstat output:
>
> [root at ben-ota-1 ~]# nstat
> #kernel
> IpInReceives                    14507              0.0
> IpInDelivers                    14507              0.0
> IpOutRequests                   49531              0.0
> TcpActiveOpens                  3                  0.0
> TcpPassiveOpens                 2                  0.0
> TcpInSegs                       14498              0.0
> TcpOutSegs                      50269              0.0
> UdpInDatagrams                  9                  0.0
> UdpOutDatagrams                 1                  0.0
> TcpExtDelayedACKs               43                 0.0
> TcpExtDelayedACKLost            5                  0.0
> TcpExtTCPHPHits                 483                0.0
> TcpExtTCPPureAcks               918                0.0
> TcpExtTCPHPAcks                 12758              0.0
> TcpExtTCPDSACKOldSent           5                  0.0
> TcpExtTCPRcvCoalesce            49                 0.0
> TcpExtTCPAutoCorking            3                  0.0
> TcpExtTCPOrigDataSent           49776              0.0
> TcpExtTCPHystartTrainDetect     1                  0.0
> TcpExtTCPHystartTrainCwnd       16                 0.0
> IpExtInBcastPkts                8                  0.0
> IpExtInOctets                   2934274            0.0
> IpExtOutOctets                  74817312           0.0
> IpExtInBcastOctets              640                0.0
> IpExtInNoECTPkts                14911              0.0
> [root at ben-ota-1 ~]#
>
>
> And, you can find the pcap here:
>
> http://www.candelatech.com/downloads/cubic.pcap.bz2
>
> Let me know if you need anything else.

Thanks! This is very useful. It looks like the sender is sending 3
(and later 4) packets every ~1.5ms for the entirety of the trace. 3
packets per burst is usually a hint that this may be related to TSQ.

This slow-and-steady behavior triggers CUBIC's Hystart Train Detection
to enter congestion avoidance at a cwnd of 16, which probably in turn
leads to slow cwnd growth, since the sending is not cwnd-limited, but
probably TSQ-limited, so cwnd does not grow in congestion avoidance
mode. Probably most of the other congestion control modules do better
because they stay in slow-start, which has a more aggressive criterion
for growing cwnd.

So this is probably at root due to the known issue with an interaction
between the ath10k driver and the following change in 3.19:

  605ad7f tcp: refine TSO autosizing

There has been a lot of discussion about how to address the
TSQ-related issues with this driver. For example, you might consider:

  https://patchwork.ozlabs.org/patch/438322/

But I am not sure of the latest status of that effort. Perhaps someone
on the ath10k list will know.

neal