tree d3dd5b5741ba5e576814b8879319e8d85a076f7b
parent def4ad4bcefbe065bc3446b20fb8b32769158425
author Neels Hofmeyr <neels@hofmeyr.de> 1522940198 +0200
committer Neels Hofmeyr <neels@hofmeyr.de> 1522969635 +0200

add compare-results.sh, call from start-testsuite.sh

Compare current test results to the expected results, and exit in error on
discrepancies.

Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.

Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".

In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.

In start-testsuite.sh, after running the tests, invoke the results comparison.

Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.

Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d