compiler regression testing

(home) 2020-12-13

There comes a time in compiler development when you've written enough code that you'd rather not break it. You want an automated testing program which tells you if your most recent change has broken something other than the thing you're concentrating on right now. To answer the question "have I just made things worse instead of better?". Have I caused the state to regress?

That kind of test is called a regression test, but the names of tests are less important than the fact that you have them and that they are fully automatic.

automatic testing

Tests should be easy to run (and preferably quick), so that you can run them after each code change you make.

So let's install npm testamajig and get going.... joke.

$ make test

This runs a test program which tests two kinds of things, so far:

$ cat makefile

This lets me write tests like these:

wasm_test malloc
sugar_test '(+ 3 1)' "t1() => i32:4"
sugar_test '(+ 3 -15)' "t1() => i32:`dec 0xfffffff4`"
sugar_test '(+ #xffffffff 1)' "t1() => i32:0"


The astute among you will have noticed that above script is written in python prevaricator (v3 only).... yes, it's just shell script.
$ mk test
GOOD all tests passed

One question might be, "how do we get our known-good .wasm file?". The trick is that it doesn't really have to be good. You're just looking for changes/regressions. Did something change when I didn't expect it to? You may find later that your good .wasm file also had a bug.

You can just change it later. In this way every time you fix your known good version it remains the paradigmatic, exemplar that it always was (that's what version control systems are for).

code templating with npm plate-o

Someone mentioned templates? I know! Let's install npm plate-o... I SED that as a joke...

I wrote a sugar module with a few globals and some functions. Then I tested to make sure it compiled and ran. Finally, in the entry function, I added a line like this:


Et Viola! I have a template.

Now I just run sed s/TESTCASE/$1/ on it, where $1 is my code snippet, and I have a valid .sugar program ready to compile and run. Here are the details of the shell script function. It uses make & wasm-interp to compile and run the .sugar file.

  sed "s/TESTCASE/$1/g" good/testing.sugar.template >good/testee.sugar
  make good/testee.wasm
  output=`wasm-interp --host-print --run-all-exports testee.wasm`
  if [ "$output" = "$2" ]; then
    echo "GOOD - $1 passed"
    echo "FAIL - $1 FAILED '$output' /= '$2'"
    failcnt=`expr ${failcnt} + 1`

So, now I have a useful regression testing system, all in only 4G of code and dependencies. Oh wait, that's 4K.

$ wc -c regress
4609 regress

the code



Tags: testing compilers automation unix-scripting makefile-advocacy unix-tool-power npm-baiting bloat (home)