[PATCH uci 4/6] tests: cram: add uci import testing on fuzzer corpus

[Petr Štetiar]

Hauke Mehrtens <hauke at hauke-m.de> [2020-10-04 17:10:59]:

> On 10/3/20 9:48 AM, Petr Štetiar wrote:
> > Use valgrind and uci cli compiled with undefined, address and leak
> > sanitizers.
> ....
> > diff --git a/tests/cram/test-san_uci_import.t b/tests/cram/test-san_uci_import.t
> > new file mode 100644
> > index 000000000000..7faed221f93e
> > --- /dev/null
> > +++ b/tests/cram/test-san_uci_import.t
> > @@ -0,0 +1,15 @@
> > +check that uci import is producing expected results:
> > +
> > +  $ [ -n "$BUILD_BIN_DIR" ] && export PATH="$BUILD_BIN_DIR:$PATH"
> > +  $ export TEST_INPUTS="$TESTDIR/inputs"
> > +  $ export FUZZ_CORPUS="$TESTDIR/../fuzz/corpus"
> > +
> > +  $ for file in $(LC_ALL=C find $FUZZ_CORPUS -type f | sort ); do
> > +  >   uci-san import -f $file; \
> > +  > done

BTW this above is a "standard" multiline shell construct (> prefix), so you
can just add simple "echo $file" and see what is going on, like this:

 diff --git a/tests/cram/test_uci_import.t b/tests/cram/test_uci_import.t
 index 8d5ab74ed4e7..e086ecf11ab1 100644
 --- a/tests/cram/test_uci_import.t
 +++ b/tests/cram/test_uci_import.t
 @@ -5,6 +5,7 @@ check that uci import is producing expected results:
    $ export FUZZ_CORPUS="$TESTDIR/../fuzz/corpus"
 
    $ for file in $(LC_ALL=C find $FUZZ_CORPUS -type f | sort ); do
 +  >   echo "testing $file"; \
    >   valgrind --quiet --leak-check=full uci import -f $file; \
    > done
    uci: I/O error

> > +  uci-san: I/O error
> > +  uci-san: Parse error (invalid command) at line 0, byte 0
> > +  uci-san: Parse error (invalid command) at line 1, byte 18
> 
> I do not get this error. Which test case should trigger it?

That's quite strange as I'm getting consistent results with gcc 8,9,10 and
clang 10 (the same version used on CI
https://gitlab.com/ynezz/openwrt-uci/-/jobs/771521825).  Both cram/shuni2 test
suites are run always for complete compiler matrix, fuzzer only under clang
compiler for obvious reasons.

Anyway, here is some clue, hope it helps:

  ynezz at ntbk:/opt/devel/openwrt/c/uci/build/tests/cram$ make test CTEST_OUTPUT_ON_FAILURE=1
  Running tests...
  Test project /opt/devel/openwrt/c/uci/build/tests/cram
      Start 1: cram
  1/1 Test #1: cram .............................***Failed    8.78 sec
  !
  --- /opt/devel/openwrt/c/uci/tests/cram/test_uci_import.t
  +++ /opt/devel/openwrt/c/uci/tests/cram/test_uci_import.t.err
  @@ -8,9 +8,21 @@
     >   echo "testing $file"; \
     >   valgrind --quiet --leak-check=full uci import -f $file; \
     > done
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/231ee80a172b8e1749b9d91867989d88e4faf7bb
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/26a6253fc1eb695b61a2fc7640ee4c03c19e438e
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/29a6e206439d792afba5e8e9c1fdf55e65a1145d
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/2e18ecc3a759dedc9357b1298e9269eccc5c5a6b
  +  uci: Parse error (package without name) at line 0, byte 68
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/51045ac5401085f5727c6d3c1cac5f8cc32a2927
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/845dcf3f15f3c28235e6be148a690e7f03b07f65
     uci: I/O error
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/bb589d0621e5472f470fa3425a234c74b1e202e8
     uci: Parse error (invalid command) at line 0, byte 0
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/ea387894a296772f96706df8b999a52d9334c746
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/id-000000,sig-06,src-000079,time-22005942,op-ext_AO,pos-8
     uci: Parse error (invalid command) at line 1, byte 18
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/id:000000,sig:11,src:000001,op:flip1,pos:24
     uci: I/O error
  +  testing /opt/devel/openwrt/c/uci/tests/cram/../fuzz/corpus/id:000008,sig:11,src:000022,op:arith8,pos:42,val:+26
     uci: I/O error
     [1]
  ..!

And it seems correct when run manually:

  ynezz at ntbk:/opt/devel/openwrt/c-projects/uci/build/tests/cram$ ../../uci import -f ../../../tests/fuzz/corpus/id-000000,sig-06,src-000079,time-22005942,op-ext_AO,pos-8
  ../../uci: Parse error (invalid command) at line 1, byte 18

Cheers,

Petr