[LEDE-DEV] RFC: Throughput testing results.

[David Lang]

On Fri, 6 May 2016, Ben Greear wrote:

> On 05/06/2016 12:05 PM, David Lang wrote:
>> On Fri, 6 May 2016, Ben Greear wrote:
>> 
>>> On 05/06/2016 10:20 AM, Toke Høiland-Jørgensen wrote:
>>>> Ben Greear <greearb at candelatech.com> writes:
>>>> 
>>>>> I am interested in feedback on the testing output. My goal is to add a
>>>>> few more different hardware configurations and then do nightly (or at
>>>>> least weekly) tests.
>>>> 
>>>> So that is showing up to 10s of *seconds* of latency, right? (I'm not
>>>> sure I'm reading the units right).
>>> 
>>> Yes, 10 seconds of latency.  My traffic generator is using pfifo-fast,
>>> RENO, and default socket sizes, so it can be at least part of the problem.
>> 
>> That's so much latency that you may as well be down.
>> 
>> Please look at the make-wifi-fast mailing list and the tests that are being 
>> done there. they show latency spikes as well as throughput, and show how it 
>> is very
>> possible to get low latency without affecting throughput (in some cases, 
>> throughput actually increases)
>
> I understand that.  My test case is fairly abusive, and my generator is not 
> optimized for
> low-latency at this time.
>
> In many cases, throughput does go down though, so I have been slow to try the 
> buffer bloat stuff.  I can run some tests with codel enabled sometime soon.
>
> I can also run my capacity test with UDP only.  That might be better for pure 
> throughput testing.  My hope is to be able to show regressions/improvements 
> over time as LEDE changes...

This is a good idea, but it is going to be very specific to the exact setup you 
use for the testing. Use different hardware, or add/remove nodes from the test 
(or have someone nearby create additional noise) and you can end up with very 
different results.

I think it would be a bad idea to setup something that encourages chasing 
benchmarks. I agree it's a good idea to watch out for regressions. The hard 
thing is doing the latter without the former :-)

David Lang