There comes a time in compiler development when you've written enough code that you'd rather not break it. You want an automated testing program which tells you if your most recent change has broken something other than the thing you're concentrating on right now. To answer the question "have I just made things worse instead of better?". Have I caused the state to regress?
That kind of test is called a regression test, but the names of tests are less important than the fact that you have them and that they are fully automatic.
Tests should be easy to run (and preferably quick), so that you can run them after each code change you make.
So let's install npm testamajig and get going.... joke.
$ make test
This runs a test program which tests two kinds of things, so far:
$ cat makefile test: ./regress
This lets me write tests like these:
wasm_test malloc sugar_test '(+ 3 1)' "t1() => i32:4" sugar_test '(+ 3 -15)' "t1() => i32:`dec 0xfffffff4`" sugar_test '(+ #xffffffff 1)' "t1() => i32:0"
Where
wasm_test malloc will compile malloc.sugar to malloc.wasm and compare with good/malloc.wasm.
sugar_test sugar-code expected-result will insert the code snippet in a sugar template module, compile and run the result,
then check the output text against the given expected text.
$ mk test ... GOOD all tests passed
One question might be, "how do we get our known-good .wasm file?". The trick is that it doesn't really have to be good. You're just looking for changes/regressions. Did something change when I didn't expect it to? You may find later that your good .wasm file also had a bug.
You can just change it later. In this way every time you fix your known good version it remains the paradigmatic, exemplar that it always was (that's what version control systems are for).
Someone mentioned templates? I know! Let's install npm plate-o... I SED that as a joke...
I wrote a sugar module with a few globals and some functions. Then I tested to make sure it compiled and ran. Finally, in the entry function, I added a line like this:
TESTCASE
Et Viola! I have a template.
Now I just run sed s/TESTCASE/$1/ on it, where $1 is my code snippet, and I have a valid .sugar program ready to compile and run.
Here are the details of the shell script function. It uses make & wasm-interp to compile and run the .sugar file.
sugar_test(){
sed "s/TESTCASE/$1/g" good/testing.sugar.template >good/testee.sugar
make good/testee.wasm
output=`wasm-interp --host-print --run-all-exports testee.wasm`
if [ "$output" = "$2" ]; then
echo "GOOD - $1 passed"
else
echo "FAIL - $1 FAILED '$output' /= '$2'"
failcnt=`expr ${failcnt} + 1`
fi
}
So, now I have a useful regression testing system, all in only 4G of code and dependencies. Oh wait, that's 4K.
$ wc -c regress 4609 regress