[ARM ATTEND] Interested in R/M-class (!MMU), automated testing
jonathan.austin at arm.com
Mon Aug 5 06:08:27 EDT 2013
Hi Olof, Robert
(+Paul Walmsley on CC as I mention his recent email below)
On 05/08/13 07:49, Olof Johansson wrote:
> [+ksummit-discuss which had fallen off]
> On Sun, Aug 4, 2013 at 11:49 PM, Olof Johansson <olof at lixom.net> wrote:
>> On Sun, Aug 4, 2013 at 5:25 AM, Robert Schwebel
>> <r.schwebel at pengutronix.de> wrote:
>>>> - Ways to do more automated testing of ARM kernels, including hearing
>>>> from other people what they use to build/boot/test/benchmark their
>>>> code (perhaps even get a discussion going about using the diverse
>>>> range of hardware people have around put to use as test-machines).
>>> That's pretty interesting; we have a test farm at PTX, but it mainly
>>> does nightly build + boot tests on mainline, for systems we care of.
>> I have a small (but growing) farm of boards here that I do some (so
>> far) limited boot testing, but I do it a few times a day on mainline,
>> and nightly on linux-next and arm-soc for-next branches. I also do it
>> for -stable but right now only for published releases since I so far
>> only handle pulling whole git branches.
>> I also build every defconfig on arm at the same cadence. Everything's
>> handled by a couple of scripts that emails me the results (as compared
>> to building a status webpage that I have to go check). So essentially
>> every morning I have an email with the fresh -next breakage waiting
>> for me to look at.
>> The hardware comes from various sources; some I've bought myself,
>> other I've been sent by kind vendors. All of them are "modern" though,
>> i.e. v7-class hardware with decent amount of memory, etc. I netboot
>> everything with local rootfs at the moment.
>> I honestly don't know if there's much point in doing complex
>> centralized testing/reporting. I've found that I never go look for the
>> results. I don't regularly check Russell's autobuilder status, for
I broadly agree with you here - I find I tend to go and look there
when things are broken ("when did this first break?"), but honestly
wasn't aware of it until very recently and don't poll it.
I think that some level of centralised regression testing on
performance might be nice - if there was notification when things
jumped in a way that bucked the trend, for example, it might be good
to know about. Do your scripts monitor that sort of thing?
The other reason I mention sharing (which I guess implies something
a little centralised) is for the case where the code you are working
on has implications for a core you don't have - I'm not sure there
are many people with 11MPCores around, for example!
(However, I don't think anything that leads people to abdicate their
personal responsibility for testing patches before they go out the door
is a good idea!)
>> example. Nor the kisskb -next build status. But having my own scripts
>> email me has been useful. I do hope that most ARM subplatform
>> maintainers have some semi-automated setup for frequent testing as
>> well, but I know that reality is that far from all do.
>> It could make sense to show what some of us use to give others
>> examples of what might work for them. There's a thousand ways to build
>> these kind of things, some elaborate and others quite simple. I've
>> definitely started at the simple end myself. :)
I think this is more the sort of thing I was hoping to discuss.
I know Paul Walmsley has some pretty functional scripts that he's recently
offered to share part of (assuming he gets time to tidy them up):
So that seems to be at least three separate approaches. Further, I took a
look recently at the ktest.pl script, which looked handy but not so
targetted at the ARM world, where you care about testing the same thing on
a wide array of platforms, might want an NFS root file-system, etc.
Here we tend to have a lot of hardware on our desks, and we test a lot
but mainly on an ad-hoc basis/where we see changes will intersect
with any specific board or core's quirks. Hearing how other people handle
automation would be great :)
More information about the linux-arm-kernel