add compare-results.sh, call from start-testsuite.sh
Compare current test results to the expected results, and exit in error on
discrepancies.
Add compare-result.sh: (trivially) grep junit xml output to determine which
tests passed and which didn't, and compare against an expected-result.log,
another junit file from a previous run. Summarize and determine success.
Include an "xfail" feature: tests that are expected to fail are marked as
"xfail", unexpected failures as "FAIL".
In various subdirs, copy the current jenkins jobs' junit xml outputs as
expected-results.log, so that we will start getting useful output in both
jenkins runs and manual local runs.
In start-testsuite.sh, after running the tests, invoke the results comparison.
Due to the single-line parsing nature, the script so far does not distinguish
between error and failure. I doubt that we actually need to do that though.
Related: OS#3136
Change-Id: I87d62a8be73d73a5eeff61a842e7c27a0066079d
diff --git a/hlr/expected-results.log b/hlr/expected-results.log
new file mode 100644
index 0000000..be8000c
--- /dev/null
+++ b/hlr/expected-results.log
@@ -0,0 +1,23 @@
+<?xml version="1.0"?>
+<testsuite name='Titan' tests='10' failures='2' errors='0' skipped='0' inconc='0' time='118.00'>
+ <testcase classname='HLR_Tests' name='TC_gsup_sai_err_invalid_imsi' time='0.008962'>
+ <failure type='fail-verdict'>Unexpected SAI ERROR Cause
+ HLR_Tests.ttcn:558 HLR_Tests control part
+ HLR_Tests.ttcn:364 TC_gsup_sai_err_invalid_imsi testcase
+ </failure>
+ </testcase>
+ <testcase classname='HLR_Tests' name='TC_gsup_sai' time='0.431623'/>
+ <testcase classname='HLR_Tests' name='TC_gsup_ul_unknown_imsi' time='0.009124'/>
+ <testcase classname='HLR_Tests' name='TC_gsup_sai_err_unknown_imsi' time='0.008056'/>
+ <testcase classname='HLR_Tests' name='TC_gsup_ul' time='0.402349'/>
+ <testcase classname='HLR_Tests' name='TC_vty' time='0.151754'/>
+ <testcase classname='HLR_Tests' name='TC_vty_msisdn_isd' time='5.076033'>
+ <failure type='fail-verdict'>Timeout waiting for ISD.req
+ HLR_Tests.ttcn:564 HLR_Tests control part
+ HLR_Tests.ttcn:485 TC_vty_msisdn_isd testcase
+ </failure>
+ </testcase>
+ <testcase classname='HLR_Tests' name='TC_gsup_purge_cs' time='0.459958'/>
+ <testcase classname='HLR_Tests' name='TC_gsup_purge_ps' time='0.467724'/>
+ <testcase classname='HLR_Tests' name='TC_gsup_purge_unknown' time='0.008034'/>
+</testsuite>