[PATCH v3 9/9] KVM: selftests: Provide README.rst for KVM selftests runner

[Sean Christopherson]

On Fri, Oct 10, 2025, Brendan Jackman wrote:
> On Tue Sep 30, 2025 at 4:36 PM UTC, Vipin Sharma wrote:
> > @@ -0,0 +1,54 @@
> > +KVM Selftest Runner
> > +===================
> > +
> > +KVM selftest runner is highly configurable test executor that allows to run
> > +tests with different configurations (not just the default), parallely, save
> > +output to disk hierarchically, control what gets printed on console, provide
> > +execution status.

...

> I understand that for reasons of velocity

It's not just velocity, it's also for stability and maintainability.  Selftests are
the wild, wild west; there's no central authority, and many subsystems have "needs"
and opinions.  E.g. tools/testing/selftests/kselftest_harness.h is quite opionated,
_and_ it has fatally been broken multiple due to one subsystem making changes that
broke usage for other subsystems.  Obviously those bugs got sorted out, but it's a
painful experience.

I guess you could say those things are all about velocity in the end; but I want
to call out that it's not just about the initial velocity of landing the series,
it's also about the long-term velocity of being able to make changes to fit KVM's
needs without getting bogged down due to other susbystems adding requirements and
use cases that are irrelevant or at odds with KVM's.

> it might make sense to do this as a KVM-specific thing, but IIUC very little
> of this has anything to do with KVM in particular, right?

The actual implementation doesn't have any dependencies on KVM, but the design
and its goal are tailored to the needs of KVM.

> Is there an expectation to evolve in a more KVM-specific direction?

Sort of?  I don't think we'll ever pick up direct dependencies, but I do think
we'll continue to tailor the runner to the needs of the KVM community.

> (One thing that might be KVM-specific is the concurrency. I assume there
> are a bunch of KVM tests that are pretty isolated from one another and
> reasonable to run in parallel.

Every KVM selftest should be able to run in parallel.  That's actually a very
intentional design property of the runner: any system-level configuration needs
to be done by a "higher" authority, e.g. the human manually running the test, a
wrapper script, some form of CI infrastructure, etc.

> Testing _the_ mm like that just isn't gonna work most of the time. I still
> think this is really specific to individual sets of tests though, in a more
> mature system there would be a metadata mechanism for marking tests as
> parallelisable wrt each other.

Dependency and friendliness tracking is again something we specifically avoided
doing, because the KVM selftests need to be self-contained anyways.  E.g. if a
test requires KVM module param X to be enabled, then the test needs to skip.
The runner takes advantage of that behavior in order to simplify the code; it
really is just a "dumb" executor.

> I guess this patchset is part of an effort to have a more mature system that
> enables that kind of thing.).

Sort of?  My response to Marc covered more of the goals in detail:

https://lore.kernel.org/all/aN8gkEMHuvIVPcCt@google.com

> To avoid confusing people and potentially leave the door open to a
> cleaner integration, please can you add some bits here about how this
> relates to the rest of the kselftest infrastructure? Some questions I
> think are worth answering:
> 
> - As someone who runs KVM selftests, but doesn't work specifically on
>   KVM, to what extent do I need to know about this tool? Can I still run
>   the selftests "the old fashioned way" and if so what do I lose as
>   compared to using the KVM runner?

The runner is purely optional.  You'll lose whatever you don't have, that the
runner provides.  E.g. I have (hacky) scripts to run KVM selftests in parallel,
but without much of the niceties provided by this runner.

> - Does this system change the "data model" of the selftests at all, and
>   if so how? I.e. I think (but honestly I'm not sure) that kselftests
>   are a 2-tier hierarchy of $suite:$test without any further
>   parameterisation or nesting (where there is more detail, it's hidden
>   as implementation details of individual $tests). Do the KVM selftests
>   have this structure?

More or less.

>   If it differs, how does that effect the view from run_kselftest.sh?

AFAIK, nothing in KVM selftests is at odds with run_kselftest.sh.

> - I think (again, not very sure) that in kselftest that each $test is a
>   command executing a process. And this process communicates its status
>   by printing KTAP and returning an exit code. Is that stuff the same
>   for this runner?

Yes?  Except most KVM selftests don't support TAP (yet).