getting users involved in on-device testing

[John Crispin]

On 05/05/2016 09:10, Michal Hrusecky wrote:
> John Crispin -  8:56  5.05.16 wrote:
>> Hi,
>>
>> It would be nice if there was a process in place that would allow
>> community members to easily be able to test images on devices and report
>> the results some place. after some time thinking of different ways to do
>> this I came up with one possible solution and wanted to know what others
>> think about this.
>>
>> we setup a file similar to how the MAINTAINERS file in the kernel.
>> anyone can add his name/email and boards he would like to be a tester
>> of. every time an image is built for said board, an email is generated
>> and sent out to the according persons. this email would contains an otp.
>> with this credential you could then log into some web frontend and get a
>> simple mask along the lines of
>>
>> * wifi worked
>> * ethernet worked
>> * leds worked
>> * buttons worked
>> * iperf
>> * ssl benchmark
>> ...
>>
>> this could all be done in a rather trivial manner and i dont think the
>> work behind such a setup is really that huge. using otps would for
>> example eliminate the need for user credential management. results could
>> simply be stored in files on the backend and then harvested later by a
>> secondary script etc etc...
>>
>> next it would be possible to generate static content based on aggregated
>> data that will show a traffic light style support status for various
>> boards, listing when it was last tested, with what revision and what the
>> test results were.
>>
>> would something like this make sense ?
> 
> It kinda makes sense, although it would be even nicer if the tests were
> automated as well. That is something that Purple is working on. There is a
> BoardFarm[1] test framework that is designed to test routers and there could be
> created test suites that reflects some of the criteria you described and in the
> best case, whenever image is produced, automatic test could be triggered and
> results uploaded. It outputs junit and there is plenty of parsers of junit - we
> have it nicely integrated in Jenkins.
> 
> [1] https://github.com/qca/boardfarm

i doubt they will test the 900+ boards. i would expect them to test
their own boards.

additionally the proposal does not tell you how to test. it describes a
way to notify when testing is required and where the images are located.
i think you misread/understood the purpose of what i proposed or i
worded it badly.

	John