FW: [EXTERNAL] Re: [PATCH v2 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
Steve VanderLeest
steven.h.vanderleest at gmail.com
Thu Sep 5 05:24:56 PDT 2024
Resending as plain text so that lists can accept.
On Thu, Sep 5, 2024 at 8:21 AM Vanderleest (US), Steven H
<steven.h.vanderleest at boeing.com> wrote:
>
> From: Vanderleest (US), Steven H <steven.h.vanderleest at boeing.com>
> Date: Thursday, September 5, 2024 at 8:13 AM
> To: Peter Zijlstra <peterz at infradead.org>, Wentao Zhang <wentaoz5 at illinois.edu>
> Cc: Kelly (US), Matt <Matt.Kelly2 at boeing.com>, akpm at linux-foundation.org <akpm at linux-foundation.org>, Oppelt (US), Andrew J <andrew.j.oppelt at boeing.com>, anton.ivanov at cambridgegreys.com <anton.ivanov at cambridgegreys.com>, ardb at kernel.org <ardb at kernel.org>, arnd at arndb.de <arnd at arndb.de>, bhelgaas at google.com <bhelgaas at google.com>, bp at alien8.de <bp at alien8.de>, Wolber (US), Chuck <chuck.wolber at boeing.com>, dave.hansen at linux.intel.com <dave.hansen at linux.intel.com>, dvyukov at google.com <dvyukov at google.com>, hpa at zytor.com <hpa at zytor.com>, jinghao7 at illinois.edu <jinghao7 at illinois.edu>, johannes at sipsolutions.net <johannes at sipsolutions.net>, jpoimboe at kernel.org <jpoimboe at kernel.org>, justinstitt at google.com <justinstitt at google.com>, kees at kernel.org <kees at kernel.org>, kent.overstreet at linux.dev <kent.overstreet at linux.dev>, linux-arch at vger.kernel.org <linux-arch at vger.kernel.org>, linux-efi at vger.kernel.org <linux-efi at vger.kernel.org>, linux-kbuild at vger.kernel.org <linux-kbuild at vger.kernel.org>, linux-kernel at vger.kernel.org <linux-kernel at vger.kernel.org>, linux-trace-kernel at vger.kernel.org <linux-trace-kernel at vger.kernel.org>, linux-um at lists.infradead.org <linux-um at lists.infradead.org>, llvm at lists.linux.dev <llvm at lists.linux.dev>, luto at kernel.org <luto at kernel.org>, marinov at illinois.edu <marinov at illinois.edu>, masahiroy at kernel.org <masahiroy at kernel.org>, maskray at google.com <maskray at google.com>, mathieu.desnoyers at efficios.com <mathieu.desnoyers at efficios.com>, Weber (US), Matthew L <matthew.l.weber3 at boeing.com>, mhiramat at kernel.org <mhiramat at kernel.org>, mingo at redhat.com <mingo at redhat.com>, morbo at google.com <morbo at google.com>, nathan at kernel.org <nathan at kernel.org>, ndesaulniers at google.com <ndesaulniers at google.com>, oberpar at linux.ibm.com <oberpar at linux.ibm.com>, paulmck at kernel.org <paulmck at kernel.org>, richard at nod.at <richard at nod.at>, rostedt at goodmis.org <rostedt at goodmis.org>, samitolvanen at google.com <samitolvanen at google.com>, Sarkisian (US), Samuel <samuel.sarkisian at boeing.com>, tglx at linutronix.de <tglx at linutronix.de>, tingxur at illinois.edu <tingxur at illinois.edu>, tyxu at illinois.edu <tyxu at illinois.edu>, x86 at kernel.org <x86 at kernel.org>
> Subject: Re: [EXTERNAL] Re: [PATCH v2 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
>
> I’ll answer Peter’s last question: “What is the impact on certification of not covering the noinstr code.”
>
>
>
> Any code in the target image that is not executed by a test (and thus not covered) must be analyzed and justified as an exception. For example, defensive code is often impossible to exercise by test, but can be included in the image with a justification to the regulatory authority such as the Federal Aviation Administration (FAA). In practice, this means the number of unique instances of non-instrumented code needs to be manageable. I say “unique instances” because there may be many instances of a particular category, but justified by the same analysis/rationale. Where we specifically mark a section of code with noinstr, it is typically because the instrumentation would change the behavior of the code, perturbing the test results. With some analysis for each distinct category of this issue, we could then write justification(s) to show the overall coverage is sufficient.
>
>
>
> Regards,
>
> Steve
>
>
>
>
>
> From: Peter Zijlstra <peterz at infradead.org>
> Date: Thursday, September 5, 2024 at 7:42 AM
> To: Wentao Zhang <wentaoz5 at illinois.edu>
> Cc: Kelly (US), Matt <Matt.Kelly2 at boeing.com>, akpm at linux-foundation.org <akpm at linux-foundation.org>, Oppelt (US), Andrew J <andrew.j.oppelt at boeing.com>, anton.ivanov at cambridgegreys.com <anton.ivanov at cambridgegreys.com>, ardb at kernel.org <ardb at kernel.org>, arnd at arndb.de <arnd at arndb.de>, bhelgaas at google.com <bhelgaas at google.com>, bp at alien8.de <bp at alien8.de>, Wolber (US), Chuck <chuck.wolber at boeing.com>, dave.hansen at linux.intel.com <dave.hansen at linux.intel.com>, dvyukov at google.com <dvyukov at google.com>, hpa at zytor.com <hpa at zytor.com>, jinghao7 at illinois.edu <jinghao7 at illinois.edu>, johannes at sipsolutions.net <johannes at sipsolutions.net>, jpoimboe at kernel.org <jpoimboe at kernel.org>, justinstitt at google.com <justinstitt at google.com>, kees at kernel.org <kees at kernel.org>, kent.overstreet at linux.dev <kent.overstreet at linux.dev>, linux-arch at vger.kernel.org <linux-arch at vger.kernel.org>, linux-efi at vger.kernel.org <linux-efi at vger.kernel.org>, linux-kbuild at vger.kernel.org <linux-kbuild at vger.kernel.org>, linux-kernel at vger.kernel.org <linux-kernel at vger.kernel.org>, linux-trace-kernel at vger.kernel.org <linux-trace-kernel at vger.kernel.org>, linux-um at lists.infradead.org <linux-um at lists.infradead.org>, llvm at lists.linux.dev <llvm at lists.linux.dev>, luto at kernel.org <luto at kernel.org>, marinov at illinois.edu <marinov at illinois.edu>, masahiroy at kernel.org <masahiroy at kernel.org>, maskray at google.com <maskray at google.com>, mathieu.desnoyers at efficios.com <mathieu.desnoyers at efficios.com>, Weber (US), Matthew L <matthew.l.weber3 at boeing.com>, mhiramat at kernel.org <mhiramat at kernel.org>, mingo at redhat.com <mingo at redhat.com>, morbo at google.com <morbo at google.com>, nathan at kernel.org <nathan at kernel.org>, ndesaulniers at google.com <ndesaulniers at google.com>, oberpar at linux.ibm.com <oberpar at linux.ibm.com>, paulmck at kernel.org <paulmck at kernel.org>, richard at nod.at <richard at nod.at>, rostedt at goodmis.org <rostedt at goodmis.org>, samitolvanen at google.com <samitolvanen at google.com>, Sarkisian (US), Samuel <samuel.sarkisian at boeing.com>, Vanderleest (US), Steven H <steven.h.vanderleest at boeing.com>, tglx at linutronix.de <tglx at linutronix.de>, tingxur at illinois.edu <tingxur at illinois.edu>, tyxu at illinois.edu <tyxu at illinois.edu>, x86 at kernel.org <x86 at kernel.org>
> Subject: [EXTERNAL] Re: [PATCH v2 0/4] Enable measuring the kernel's Source-based Code Coverage and MC/DC with Clang
>
> EXT email: be mindful of links/attachments.
>
>
>
> On Wed, Sep 04, 2024 at 11:32:41PM -0500, Wentao Zhang wrote:
> > From: Wentao Zhang <zhangwt1997 at gmail.com>
> >
> > This series adds support for building x86-64 kernels with Clang's Source-
> > based Code Coverage[1] in order to facilitate Modified Condition/Decision
> > Coverage (MC/DC)[2] that provably correlates to source code for all levels
> > of compiler optimization.
> >
> > The newly added kernel/llvm-cov/ directory complements the existing gcov
> > implementation. Gcov works at the object code level which may better
> > reflect actual execution. However, Gcov lacks the necessary information to
> > correlate coverage measurement with source code location when compiler
> > optimization level is non-zero (which is the default when building the
> > kernel). In addition, gcov reports are occasionally ambiguous when
> > attempting to compare with source code level developer intent.
> >
> > In the following gcov example from drivers/firmware/dmi_scan.c, an
> > expression with four conditions is reported to have six branch outcomes,
> > which is not ideally informative in many safety related use cases, such as
> > automotive, medical, and aerospace.
> >
> > 5: 1068: if (s == e || *e != '/' || !month || month > 12) {
> > branch 0 taken 5 (fallthrough)
> > branch 1 taken 0
> > branch 2 taken 5 (fallthrough)
> > branch 3 taken 0
> > branch 4 taken 0 (fallthrough)
> > branch 5 taken 5
> >
> > On the other hand, Clang's Source-based Code Coverage instruments at the
> > compiler frontend which maintains an accurate mapping from coverage
> > measurement to source code location. Coverage reports reflect exactly how
> > the code is written regardless of optimization and can present advanced
> > metrics like branch coverage and MC/DC in a clearer way. Coverage report
> > for the same snippet by llvm-cov would look as follows:
> >
> > 1068| 5| if (s == e || *e != '/' || !month || month > 12) {
> > ------------------
> > | Branch (1068:6): [True: 0, False: 5]
> > | Branch (1068:16): [True: 0, False: 5]
> > | Branch (1068:29): [True: 0, False: 5]
> > | Branch (1068:39): [True: 0, False: 5]
> > ------------------
> >
> > Clang has added MC/DC support as of its 18.1.0 release. MC/DC is a fine-
> > grained coverage metric required by many automotive and aviation industrial
> > standards for certifying mission-critical software [3].
> >
> > In the following example from arch/x86/events/probe.c, llvm-cov gives the
> > MC/DC measurement for the compound logic decision at line 43.
> >
> > 43| 12| if (msr[bit].test && !msr[bit].test(bit, data))
> > ------------------
> > |---> MC/DC Decision Region (43:8) to (43:50)
> > |
> > | Number of Conditions: 2
> > | Condition C1 --> (43:8)
> > | Condition C2 --> (43:25)
> > |
> > | Executed MC/DC Test Vectors:
> > |
> > | C1, C2 Result
> > | 1 { T, F = F }
> > | 2 { T, T = T }
> > |
> > | C1-Pair: not covered
> > | C2-Pair: covered: (1,2)
> > | MC/DC Coverage for Decision: 50.00%
> > |
> > ------------------
> > 44| 5| continue;
> >
> > As the results suggest, during the span of measurement, only condition C2
> > (!msr[bit].test(bit, data)) is covered. That means C2 was evaluated to both
> > true and false, and in those test vectors C2 affected the decision outcome
> > independently. Therefore MC/DC for this decision is 1 out of 2 (50.00%).
> >
> > To do a full kernel measurement, instrument the kernel with
> > LLVM_COV_KERNEL_MCDC enabled, and optionally set a
> > LLVM_COV_KERNEL_MCDC_MAX_CONDITIONS value (the default is six). Run the
> > testsuites, and collect the raw profile data under
> > /sys/kernel/debug/llvm-cov/profraw. Such raw profile data can be merged and
> > indexed, and used for generating coverage reports in various formats.
> >
> > $ cp /sys/kernel/debug/llvm-cov/profraw vmlinux.profraw
> > $ llvm-profdata merge vmlinux.profraw -o vmlinux.profdata
> > $ llvm-cov show --show-mcdc --show-mcdc-summary \
> > --format=text --use-color=false -output-dir=coverage_reports \
> > -instr-profile vmlinux.profdata vmlinux
> >
> > The first two patches enable the llvm-cov infrastructure, where the first
> > enables source based code coverage and the second adds MC/DC support. The
> > next patch disables instrumentation for curve25519-x86_64.c for the same
> > reason as gcov. The final patch enables coverage for x86-64.
> >
> > The choice to use a new Makefile variable LLVM_COV_PROFILE, instead of
> > reusing GCOV_PROFILE, was deliberate. More work needs to be done to
> > determine if it is appropriate to reuse the same flag. In addition, given
> > the fundamentally different approaches to instrumentation and the resulting
> > variation in coverage reports, there is a strong likelihood that coverage
> > type will need to be managed separately.
> >
> > This work reuses code from a previous effort by Sami Tolvanen et al. [4].
> > Our aim is for source-based *code coverage* required for high assurance
> > (MC/DC) while [4] focused more on performance optimization.
> >
> > This initial submission is restricted to x86-64. Support for other
> > architectures would need a bit more Makefile & linker script modification.
> > Informally we've confirmed that arm64 works and more are being tested.
> >
> > Note that Source-based Code Coverage is Clang-specific and isn't compatible
> > with Clang's gcov support in kernel/gcov/. Currently, kernel/gcov/ is not
> > able to measure MC/DC without modifying CFLAGS_GCOV and it would face the
> > same issues in terms of source correlation as gcov in general does.
> >
> > Some demo and results can be found in [5]. We will talk about this patch
> > series in the Refereed Track at LPC 2024 [6].
> >
> > Known Limitations:
> >
> > Kernel code with logical expressions exceeding
> > LVM_COV_KERNEL_MCDC_MAX_CONDITIONS will produce a compiler warning.
> > Expressions with up to 47 conditions are found in the Linux kernel source
> > tree (as of v6.11), but 46 seems to be the max value before the build fails
> > due to kernel size. As of LLVM 19 the max number of conditions possible is
> > 32767.
> >
> > As of LLVM 19, certain expressions are still not covered, and will produce
> > build warnings when they are encountered:
> >
> > "[...] if a boolean expression is embedded in the nest of another boolean
> > expression but separated by a non-logical operator, this is also not
> > supported. For example, in x = (a && b && c && func(d && f)), the d && f
> > case starts a new boolean expression that is separated from the other
> > conditions by the operator func(). When this is encountered, a warning
> > will be generated and the boolean expression will not be
> > instrumented." [7]
> >
>
> What does this actually look like in the generated code?
>
> Where is the modification to noinstr ?
>
> What is the impact on certification of not covering the noinstr code.
More information about the linux-um
mailing list