[LEDE-DEV] RFC: Throughput testing results.

[Ben Greear]

On 05/06/2016 12:26 PM, David Lang wrote:
> On Fri, 6 May 2016, Ben Greear wrote:
>
>> On 05/06/2016 12:05 PM, David Lang wrote:
>>> On Fri, 6 May 2016, Ben Greear wrote:
>>>
>>>> On 05/06/2016 10:20 AM, Toke Høiland-Jørgensen wrote:
>>>>> Ben Greear <greearb at candelatech.com> writes:
>>>>>
>>>>>> I am interested in feedback on the testing output. My goal is to add a
>>>>>> few more different hardware configurations and then do nightly (or at
>>>>>> least weekly) tests.
>>>>>
>>>>> So that is showing up to 10s of *seconds* of latency, right? (I'm not
>>>>> sure I'm reading the units right).
>>>>
>>>> Yes, 10 seconds of latency.  My traffic generator is using pfifo-fast,
>>>> RENO, and default socket sizes, so it can be at least part of the problem.
>>>
>>> That's so much latency that you may as well be down.
>>>
>>> Please look at the make-wifi-fast mailing list and the tests that are being done there. they show latency spikes as well as throughput, and show how it is very
>>> possible to get low latency without affecting throughput (in some cases, throughput actually increases)
>>
>> I understand that.  My test case is fairly abusive, and my generator is not optimized for
>> low-latency at this time.
>>
>> In many cases, throughput does go down though, so I have been slow to try the buffer bloat stuff.  I can run some tests with codel enabled sometime soon.
>>
>> I can also run my capacity test with UDP only.  That might be better for pure throughput testing.  My hope is to be able to show regressions/improvements over
>> time as LEDE changes...
>
> This is a good idea, but it is going to be very specific to the exact setup you use for the testing. Use different hardware, or add/remove nodes from the test
> (or have someone nearby create additional noise) and you can end up with very different results.
>
> I think it would be a bad idea to setup something that encourages chasing benchmarks. I agree it's a good idea to watch out for regressions. The hard thing is
> doing the latter without the former :-)

We have isolation chambers too, so at least if I dedicate some effort and/or hardware to it, I should be
able to get pretty repeatable results.

I will add some additional notes to the test to specify the traffic generator's qdisc, and will also
be more specific about whether it is in an isolation chamber or not.

Either way, I expect some jitter in the results, so one bad (or good) run doesn't necessarily mean
a lot...

Thanks,
Ben


-- 
Ben Greear <greearb at candelatech.com>
Candela Technologies Inc  http://www.candelatech.com