[PATCH v2] test-patchset: allow proper support for randconfig

[Jacob Keller]

Make randconfig pre-generate a configuration so that the pre and post
patch series builds use the same configuration (vs using random configs
each time). In addition, append the random configuration whenever there
is a build diff, so that the user can see.

Signed-off-by: Jacob Keller <jacob.e.keller at intel.com>
---
 aiaiai-test-patchset | 37 ++++++++++++++++++++++++++++++-------
 doc/README           |  8 ++++++++
 2 files changed, 38 insertions(+), 7 deletions(-)

diff --git a/aiaiai-test-patchset b/aiaiai-test-patchset
index d65f699c3251..9e1c7d265577 100755
--- a/aiaiai-test-patchset
+++ b/aiaiai-test-patchset
@@ -39,20 +39,26 @@ The configurations are specified as follows:
 
 where
 
-  * "defconfig" is the name kernel defconfig file to test
+  * "defconfig" is the name kernel defconfig file to test (or "randconfig")
   * "arch" is the architecture (translates to Kbuild's ARCH variable value)
   * "cross" is the cross-compiler prefix (translates to Kbuild's CROSS_COMPILE
      variable value).
 
 For example, the following list of configurations:
 
-  omap2_defconfig,arm,arm-eabi- i386_defconfig,i386 defconfig,s390,s390x-linux-
+  omap2_defconfig,arm,arm-eabi- i386_defconfig,i386 defconfig,s390,s390x-linux- randconfig,arm
 
 will make $PROG to test the patch in 3 configurations:
 
   1. defconfig "omap2_defconfig" with ARCH=arm and CROSS_COMPILE=arm-eabi-
   2. defconfig "i386_defconfig" with ARCH=arm and the default CROSS_COMPILE
   3. defconfig "defconfig" with ARCH=s390 and CROSS_COMPILE=s390x-linux-
+  4. a randomly generated configuration with ARCH=arm
+
+Note that randconfig pre-generates a single random configuration rather than
+using randconfig for each kernel compile. In this way, we guarantee that the
+same kernel configuration is used each build, rather than completely random
+each time.
 
 By default, $PROG assumes that defconfigs are part of the
 kernel tree, unless --confdir option is specified, in which case the defconfig
@@ -160,11 +166,6 @@ test_configuration()
 
 	[ -z "$arch" ] || arch_opt="-a $arch${cross:+",$cross"}"
 
-	if [ -n "$confdir" ]; then
-		defconfig="$confdir/$defconfig"
-		[ -f "$defconfig" ] || die "$defconfig is not available"
-	fi
-
 	if [ -n "$bisectability" ]; then
 		# We share the same source tree with 'aiaiai-test-bisectability', so
 		# wait for it (it could be run at the previous iteration)
@@ -179,6 +180,23 @@ test_configuration()
 
 	git --git-dir="$(git_dir "$cloned_kernel1")" reset $quiet --hard "$commit_id1" >&2
 
+	# Enable use of a random condiguration that is stable between the
+	# before and after tests. If we just used 'randconfig' by default, then
+	# we wouldn't be testing the before and after kernels with the same
+	# configuration. We do this even if we have specified a confdir.
+	if [ "$defconfig" = "randconfig" ]; then
+			# Generate a random configuration
+			make -C "$cloned_kernel1" ${arch:+ARCH="$arch"} \
+				${cross:+CROSS_COMPILE="$cross"} -- "$defconfig"
+
+			# Store this configuration in our temporary work dir
+			defconfig="$tmpdir/$config.random"
+			cp .config "$defconfig"
+	elif [ -n "$confdir" ]; then
+		defconfig="$confdir/$defconfig"
+		[ -f "$defconfig" ] || die "$defconfig is not available"
+	fi
+
 	verbose "Build non-patched kernel (\"$cloned_kernel1\", configuration \"$config\")"
 
 	local log1="$logdir1/$config"
@@ -524,6 +542,11 @@ for defconfig in $defconfigs; do
 		fi
 
 		cat "$tmpdir/$defconfig.stderr.diff"
+
+		if [ -f "$tmpdir/$defconfig.config" ]; then
+			printf "\n%s\n" "For reference, the random configuration used was:"
+			cat "$tmpdir/$defconfig.config"
+		fi
 	else
 		printf "\n%s\n" "Successfully built configuration \"$defconfig\", no issues."
 	fi
diff --git a/doc/README b/doc/README
index c10d691afd02..0e699a209253 100644
--- a/doc/README
+++ b/doc/README
@@ -257,6 +257,14 @@ Here are some examples.
    IOW, just specify defconfigs, architectures, and cross-compiler prefixes at
    the end.
 
+4. Test a patch or a patch-set p.mbox against the net-next kernel tree,
+   against the 'origin/master' branch, using a randomly generated config, the
+   architecture is i386 (translates to 'make ARCH=i386'). Use 16 jobs.
+
+   cat p.mbox | ./aiaiai-test-patchset --bisectability --sparse --smatch \
+       --cppcheck --coccinelle -j 16 -c origin/master \
+       /<path>/net-next randconfig,i386
+
 Note, the 'aiaiai-test-patchset' script accepts one mbox file via stdio or the
 '-i' option. If you have several files belonging to one patch-set and you want
 to test them all, you need to concatenate them. Use the 'aiaiai-concat-mboxes'
-- 
2.0.1.475.g9b8d714