getting users involved in on-device testing

[Hauke Mehrtens]

On 05/05/2016 08:56 AM, John Crispin wrote:
> Hi,
> 
> It would be nice if there was a process in place that would allow
> community members to easily be able to test images on devices and report
> the results some place. after some time thinking of different ways to do
> this I came up with one possible solution and wanted to know what others
> think about this.
> 
> we setup a file similar to how the MAINTAINERS file in the kernel.
> anyone can add his name/email and boards he would like to be a tester
> of. every time an image is built for said board, an email is generated
> and sent out to the according persons. this email would contains an otp.
> with this credential you could then log into some web frontend and get a
> simple mask along the lines of
> 
> * wifi worked
> * ethernet worked
> * leds worked
> * buttons worked
> * iperf
> * ssl benchmark
> ...
> 
> this could all be done in a rather trivial manner and i dont think the
> work behind such a setup is really that huge. using otps would for
> example eliminate the need for user credential management. results could
> simply be stored in files on the backend and then harvested later by a
> secondary script etc etc...
> 
> next it would be possible to generate static content based on aggregated
> data that will show a traffic light style support status for various
> boards, listing when it was last tested, with what revision and what the
> test results were.
> 
> would something like this make sense ?
> 
> 	John

I would like such a system. This could also help people to decide which
device to buy.

I think we should add a free text comment section for some special
remarks that do not fit into the yes/no/not available questions, we
should then later extend the yes/no/na questions to reflect stuff that
is often added to the comments.

We could also try to integrate this with some automated testing.

I think it would be nice to have this as an open source software and
then extend this according to our needs.

Is there anybody interested in developing such a system and releasing it
under an open source license so it can later easily be extended?

Hauke