getting users involved in on-device testing

[John Crispin]

On 05/05/2016 11:47, Hauke Mehrtens wrote:
> On 05/05/2016 08:56 AM, John Crispin wrote:
>> Hi,
>>
>> It would be nice if there was a process in place that would allow
>> community members to easily be able to test images on devices and report
>> the results some place. after some time thinking of different ways to do
>> this I came up with one possible solution and wanted to know what others
>> think about this.
>>
>> we setup a file similar to how the MAINTAINERS file in the kernel.
>> anyone can add his name/email and boards he would like to be a tester
>> of. every time an image is built for said board, an email is generated
>> and sent out to the according persons. this email would contains an otp.
>> with this credential you could then log into some web frontend and get a
>> simple mask along the lines of
>>
>> * wifi worked
>> * ethernet worked
>> * leds worked
>> * buttons worked
>> * iperf
>> * ssl benchmark
>> ...
>>
>> this could all be done in a rather trivial manner and i dont think the
>> work behind such a setup is really that huge. using otps would for
>> example eliminate the need for user credential management. results could
>> simply be stored in files on the backend and then harvested later by a
>> secondary script etc etc...
>>
>> next it would be possible to generate static content based on aggregated
>> data that will show a traffic light style support status for various
>> boards, listing when it was last tested, with what revision and what the
>> test results were.
>>
>> would something like this make sense ?
>>
>> 	John
> 
> I would like such a system. This could also help people to decide which
> device to buy.

yep, that is what i am hoping for. we could list devices by their
support status, this is what i meant by the traffic light style. green
means "fully supported and functional" ....

> I think we should add a free text comment section for some special
> remarks that do not fit into the yes/no/not available questions, we
> should then later extend the yes/no/na questions to reflect stuff that
> is often added to the comments.

yeah, i have actually spent a bit of time thi8nking about details but
before publishing them i wanted to get a feel for the general attitude
toward such a system. the questions should not be y/n but y/n/na/free-text


> 
> We could also try to integrate this with some automated testing.

indeed, we should allow people to use the system of choice but provide
info how to do so. i use a trivial etup locally that involves a 40 euro
usb power cord, a small router, usb hub with 4 usb2serial dongles
attached and a switch. whole setup cost me around 80 euros and i use it
for testing and remote work.

> I think it would be nice to have this as an open source software and
> then extend this according to our needs.

preferably something simple written in for example python


> Is there anybody interested in developing such a system and releasing it
> under an open source license so it can later easily be extended?
>