[PATCH v4 1/9] KVM: selftest: Create KVM selftest runner

[Ackerley Tng]

Sean Christopherson <seanjc at google.com> writes:

> On Fri, Jun 12, 2026, Ackerley Tng wrote:
>> Sean Christopherson <seanjc at google.com> writes:
>>
>> > On Thu, Jun 11, 2026, Ackerley Tng wrote:
>> >> Sean Christopherson <seanjc at google.com> writes:
>> >>
>> >> > On Wed, Jun 10, 2026, Ackerley Tng wrote:
>> >> >> Vipin Sharma <vipinsh at google.com> writes:
>> >> >> My (future) use case is that with hugepages, I want to run something
>> >> >> like
>> >> >>
>> >> >>   ./guest_memfd_test --order=0
>> >> >>   ./guest_memfd_test --order=9
>> >> >>   ./guest_memfd_test --order=18
>> >> >>
>> >> >> And 0, 9 and 18 are the supported HugeTLB orders on the machine being
>> >> >> tested. I'd like to iterate over supported HugeTLB orders at runner
>> >> >> runtime instead of at build time.
>> >> >
>> >> > No.  The right way to handle this is to define testcases for the "interesting"
>> >> > sizes, and then rely on the test itself to SKIP if the size is unsupported.  This
>> >> > is no different than a test that requires EPT, or nested VMX, or nested SVM, etc.
>> >>
>> >> That should work too. So at build time I'd make it define all the
>> >> possible HugeTLB sizes on every arch, and then skip as necessary.
>> >
>> > Not necessarily at "build time", the testcases can also come from your local
>> > environment.
>> >
>> >> Why though, why not find the supported sizes at runtime?
>> >
>> > You can find the supported sizes at runtime, just not in the test runner.  I want
>> > the runner itself to be largely oblivious to what's its running.  Disallowing
>> > more or less _any_ test specific configuration/setup in the runner is the only
>> > way I see of keeping the runner strictly focused on running tests/testcases.
>>
>> The runner should 100% focus on running tests, I think it's hard for the
>> runner to avoid the process of test discovery though.
>>
>> The current test discovery process is to go down the tree of directories
>> and find files, 1 file == 1 testcase. Each file should (statically)
>> contain a single command.
>>
>> Since you're not opposed to runtime discovery of test cases, how about
>> something like:
>>
>>     if the test case file has executable permissions
>>         execute it and get back a list of test cases to be run.
>
> I don't hate it, but I still dislike the idea.  I am all in favor of runtime
> *discovery* of testcases, but I am staunchly opposed to runtime *definition* of
> testcases.
>
> If I define guest_memfd testcases for 4KiB, 2MiB, and 1GiB pages, but the
> underlying system only support 4KiB and 2MiB, then I want to see a SKIP for the
> 1GiB testcase.  If the definition is dynamic, then the 1GiB testcase simply won't
> exist.
>

Seeing the SKIP is a huge advantage over dynamic definition. Thanks for
explaining. Makes a lot of sense now to have everything defined statically.

>>     else
>>         read it as a single test case.
>>
>> I'd then use the executable version, check HugeTLB setup on the machine
>> under test and return bunch of separate commands to be run (each with a
>> different HugeTLB size).
>>
>> The runner still doesn't need to deal test-specific config, that's part
>> of the executable that tells the runner what to run.