getting users involved in on-device testing

[David Lang]

On Thu, 5 May 2016, Michal Hrusecky wrote:

> John Crispin -  8:56  5.05.16 wrote:
>> Hi,
>>
>> It would be nice if there was a process in place that would allow
>> community members to easily be able to test images on devices and report
>> the results some place. after some time thinking of different ways to do
>> this I came up with one possible solution and wanted to know what others
>> think about this.
>>
>> we setup a file similar to how the MAINTAINERS file in the kernel.
>> anyone can add his name/email and boards he would like to be a tester
>> of. every time an image is built for said board, an email is generated
>> and sent out to the according persons. this email would contains an otp.
>> with this credential you could then log into some web frontend and get a
>> simple mask along the lines of
>>
>> * wifi worked
>> * ethernet worked
>> * leds worked
>> * buttons worked
>> * iperf
>> * ssl benchmark
>> ...
>>
>> this could all be done in a rather trivial manner and i dont think the
>> work behind such a setup is really that huge. using otps would for
>> example eliminate the need for user credential management. results could
>> simply be stored in files on the backend and then harvested later by a
>> secondary script etc etc...
>>
>> next it would be possible to generate static content based on aggregated
>> data that will show a traffic light style support status for various
>> boards, listing when it was last tested, with what revision and what the
>> test results were.
>>
>> would something like this make sense ?
>
> It kinda makes sense, although it would be even nicer if the tests were
> automated as well. That is something that Purple is working on. There is a
> BoardFarm[1] test framework that is designed to test routers and there could be
> created test suites that reflects some of the criteria you described and in the
> best case, whenever image is produced, automatic test could be triggered and
> results uploaded. It outputs junit and there is plenty of parsers of junit - we
> have it nicely integrated in Jenkins.
>
> [1] https://github.com/qca/boardfarm

That's good when you have the board as part of your buildfarm. But what about 
the hardware that doesn't make it into your buildfarm (like the routers that 
will be released next week :-)

having a way for people to test and report will give you testers with different 
build configs, different use cases, different clients (how many routers have had 
lockup problems when used with apple devices and/or multiple SSIDs in the last 
few years?)

you can never have too much testing. Make it easy for people to test and report 
the results of the test. Include a N/A option for each thing as well.

Along similar lines, setup a way for people to know what hardware you developers 
have, what you are especially interested in getting your hands on, and what you 
won't bother with even if someone donates it.

Setup a way for people to donate small amounts monthly to the project (to fund 
such devices and/or hosting, trips to DC to argue with/brief the FCC, etc)

David Lang