sort hwsim test sequence

[Jouni Malinen]

On Sat, Mar 01, 2014 at 05:26:18PM -0400, Eduardo Abinader wrote:
> 1) Running each test_yyy.py in shuffle order should produce different
> results from those reported by buildbot?

That depends on what you mean with different results.. The test cases
are supposed to give the same result regardless of which order they are
executed in. Sure, there may be issues that are specific to the order,
but that is certainly not by design.

> 2) The only way I'm finding myself to achieve the same results from
> buildbot is by sorting os.listdir(".") and also using vm-run.sh with kernel
> 3.14-rc4. Is that ok? Could someone report your configurations to achieve
> same results of buildbot.

I don't think it would be possible to reproduce the exact same results
taken into account that timing varies based on the load on the server.
Anyway, if you do not specifically order the test cases in the same way,
you will likely see different order and the buildbot server does not
order these in anyway specific way (i.e., it depends on how the files
happen to show up in the directory).. I don't see any particular harm in
this; in fact, this can increase testing coverage by using different
execution order (and that was one of the reasons for adding that shuffle
option in the first place).

The current buildbot setup uses 32-bit Ubuntu 13.10 with recent snapshot
of wireless-testing.git -based Backports for wireless components. This
full OS runs on a virtual server, so vm-run.sh is not used.

-- 
Jouni Malinen                                            PGP id EFC895FA