Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 1 | #!/usr/bin/env python3 |
| 2 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 3 | # osmo-gsm-tester.py: main program file |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 4 | # |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 5 | # Copyright (C) 2016-2020 sysmocom - s.f.m.c. GmbH <info@sysmocom.de> |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 6 | # |
| 7 | # Author: Neels Hofmeyr <neels@hofmeyr.de> |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 8 | # Author: Pau Espin Pedrol <pespin@sysmocom.de> |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 9 | # |
| 10 | # This program is free software: you can redistribute it and/or modify |
Harald Welte | 2720534 | 2017-06-03 09:51:45 +0200 | [diff] [blame] | 11 | # it under the terms of the GNU General Public License as |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 12 | # published by the Free Software Foundation, either version 3 of the |
| 13 | # License, or (at your option) any later version. |
| 14 | # |
| 15 | # This program is distributed in the hope that it will be useful, |
| 16 | # but WITHOUT ANY WARRANTY; without even the implied warranty of |
| 17 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
Harald Welte | 2720534 | 2017-06-03 09:51:45 +0200 | [diff] [blame] | 18 | # GNU General Public License for more details. |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 19 | # |
Harald Welte | 2720534 | 2017-06-03 09:51:45 +0200 | [diff] [blame] | 20 | # You should have received a copy of the GNU General Public License |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 21 | # along with this program. If not, see <http://www.gnu.org/licenses/>. |
| 22 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 23 | '''osmo-gsm-tester.py: invoke a single test run |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 24 | |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 25 | Examples: |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 26 | |
Pau Espin Pedrol | e972c9c | 2020-05-12 15:06:55 +0200 | [diff] [blame] | 27 | ./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_dir/ -s osmo_trx |
| 28 | ./osmo-gsm-tester.py -c doc/examples/2g_osmocom/main.conf ~/my_trial_dir/ -s sms_tests:dyn_ts+eu_band+bts_sysmo |
| 29 | ./osmo-gsm-tester.py -c sysmocom/main.conf ~/my_trial_dir/ -s sms_tests/mo_mt_sms:bts_trx |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 30 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 31 | (The names for test suites and scenarios used in these examples must be defined |
| 32 | by the osmo-gsm-tester configuration.) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 33 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 34 | A trial package contains binaries (usually built by a jenkins job) of software |
| 35 | to be run by Osmo-Gsm-Tester, like core network programs as well as binaries for |
| 36 | the various BTS models on a 2G network. |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 37 | |
| 38 | A test suite defines specific actions to be taken and verifies their outcome. |
| 39 | Such a test suite may leave certain aspects of a setup undefined, e.g. it may |
| 40 | be BTS model agnostic or does not care which voice codecs are chosen. |
| 41 | |
| 42 | A test scenario completes the picture in that it defines which specific choices |
| 43 | shall be made to run a test suite. Any one test suite may thus run on any |
| 44 | number of different scenarios, e.g. to test various voice codecs. |
| 45 | |
| 46 | Test scenarios may be combined. For example, one scenario may define a timeslot |
| 47 | configuration to use, while another scenario may define the voice codec |
| 48 | configuration. |
| 49 | |
| 50 | There may still be aspects that are neither required by a test suite nor |
| 51 | strictly defined by a scenario, which will be resolved automatically, e.g. by |
| 52 | choosing the first available item that matches the other constraints. |
| 53 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 54 | A test run thus needs to define: |
| 55 | * A trial package containing built binaries |
| 56 | * A set of test suites, each with its combinations of scenarios |
Pau Espin Pedrol | 6c6c0e8 | 2020-05-11 18:30:58 +0200 | [diff] [blame] | 57 | * A main configuration file specifying paths to other files containing sets of |
| 58 | resources, default configurations and paths on where to find suites, |
| 59 | scenarios, etc. |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 60 | |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 61 | If no combination of suites and scenarios is provided, the default list of |
| 62 | suites will be run as defined in the osmo-gsm-tester configuration. |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 63 | |
| 64 | The scenarios and suites run for a given trial will be recorded in a trial |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 65 | package's directory: Upon launch, a '$trial_dir/run.<date>' directory will be |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 66 | created, which will collect logs and reports. |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 67 | ''' |
| 68 | |
Neels Hofmeyr | e352844 | 2017-04-08 21:18:30 +0200 | [diff] [blame] | 69 | import sys |
Neels Hofmeyr | e60df69 | 2017-05-14 03:05:48 +0200 | [diff] [blame] | 70 | import argparse |
Pau Espin Pedrol | 469316f | 2017-05-17 14:51:31 +0200 | [diff] [blame] | 71 | from signal import * |
Neels Hofmeyr | d46ea13 | 2017-04-08 15:56:31 +0200 | [diff] [blame] | 72 | from osmo_gsm_tester import __version__ |
Pau Espin Pedrol | ea8c3d4 | 2020-05-04 12:05:05 +0200 | [diff] [blame] | 73 | from osmo_gsm_tester.core import log |
Pau Espin Pedrol | f574a46 | 2020-05-05 12:18:35 +0200 | [diff] [blame] | 74 | from osmo_gsm_tester.core import trial |
| 75 | from osmo_gsm_tester.core import suite |
Pau Espin Pedrol | 0dd4c0b | 2020-05-06 14:04:18 +0200 | [diff] [blame] | 76 | from osmo_gsm_tester.core import config |
Pau Espin Pedrol | ea8c3d4 | 2020-05-04 12:05:05 +0200 | [diff] [blame] | 77 | from osmo_gsm_tester.core.schema import generate_schemas |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 78 | |
Pau Espin Pedrol | 469316f | 2017-05-17 14:51:31 +0200 | [diff] [blame] | 79 | def sig_handler_cleanup(signum, frame): |
| 80 | print("killed by signal %d" % signum) |
| 81 | # This sys.exit() will raise a SystemExit base exception at the current |
| 82 | # point of execution. Code must be prepared to clean system-wide resources |
| 83 | # by using the "finally" section. This allows at the end 'atexit' hooks to |
| 84 | # be called before exiting. |
| 85 | sys.exit(1) |
| 86 | |
Neels Hofmeyr | 2123541 | 2017-05-14 03:00:21 +0200 | [diff] [blame] | 87 | def main(): |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 88 | |
Pau Espin Pedrol | 469316f | 2017-05-17 14:51:31 +0200 | [diff] [blame] | 89 | for sig in (SIGINT, SIGTERM, SIGQUIT, SIGPIPE, SIGHUP): |
| 90 | signal(sig, sig_handler_cleanup) |
| 91 | |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 92 | parser = argparse.ArgumentParser(epilog=__doc__, formatter_class=argparse.RawTextHelpFormatter) |
Neels Hofmeyr | cf0304f | 2017-05-06 22:07:39 +0200 | [diff] [blame] | 93 | # Note: since we're using RawTextHelpFormatter to keep nicely separate |
| 94 | # paragraphs in the long help text, we unfortunately also need to take care |
| 95 | # of line wraps in the shorter cmdline options help. |
| 96 | # The line width here is what remains of screen width after the list of |
| 97 | # options placed by ArgumentParser. That's unfortunately subject to change |
| 98 | # and undefined, so when things change, just run a local |
| 99 | # ./osmo-gsm-tester.py --help and try to keep everything in 80 chars width. |
| 100 | # The help text is indented automatically, but line width is manual. |
| 101 | # Using multi-line strings here -- doesn't look nice in the python flow but |
| 102 | # is easiest to maintain. |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 103 | parser.add_argument('-V', '--version', action='store_true', |
| 104 | help='Show version') |
Pau Espin Pedrol | 6c6c0e8 | 2020-05-11 18:30:58 +0200 | [diff] [blame] | 105 | parser.add_argument('-c', '--conf-path', dest='conf_path', |
| 106 | help='''Specify main configuration file path''') |
Pau Espin Pedrol | e972c9c | 2020-05-12 15:06:55 +0200 | [diff] [blame] | 107 | parser.add_argument('trial_dir', nargs='?', default=None, |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 108 | help='Directory containing binaries to test') |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 109 | parser.add_argument('-s', '--suite-scenario', dest='suite_scenario', action='append', |
Neels Hofmeyr | cf0304f | 2017-05-06 22:07:39 +0200 | [diff] [blame] | 110 | help='''A suite-scenarios combination |
| 111 | like suite:scenario+scenario''') |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 112 | parser.add_argument('-t', '--test', dest='test', action='append', |
Neels Hofmeyr | cf0304f | 2017-05-06 22:07:39 +0200 | [diff] [blame] | 113 | help='''Run only tests matching this name. |
| 114 | Any test name that contains the given string is run. |
| 115 | To get an exact match, prepend a "=" like |
| 116 | "-t =my_exact_name". The ".py" suffix is always |
| 117 | optional.''') |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 118 | parser.add_argument('-l', '--log-level', dest='log_level', choices=log.LEVEL_STRS.keys(), |
| 119 | default=None, |
| 120 | help='Set logging level for all categories (on stdout)') |
| 121 | parser.add_argument('-T', '--traceback', dest='trace', action='store_true', |
Neels Hofmeyr | 42bebfa | 2017-06-06 19:41:42 +0200 | [diff] [blame] | 122 | help='Enable stdout logging of tracebacks') |
| 123 | parser.add_argument('-R', '--source', dest='source', action='store_true', |
| 124 | help='Enable stdout logging of source file') |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 125 | args = parser.parse_args() |
| 126 | |
| 127 | if args.version: |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 128 | print(__version__) |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 129 | exit(0) |
| 130 | |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 131 | print('combinations:', repr(args.suite_scenario)) |
Pau Espin Pedrol | e972c9c | 2020-05-12 15:06:55 +0200 | [diff] [blame] | 132 | print('trial:', repr(args.trial_dir)) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 133 | print('tests:', repr(args.test)) |
| 134 | |
Neels Hofmeyr | f816688 | 2017-05-05 19:48:35 +0200 | [diff] [blame] | 135 | # create a default log to stdout |
Neels Hofmeyr | 9576f5f | 2017-05-24 18:31:01 +0200 | [diff] [blame] | 136 | log.LogTarget().style(all_origins_on_levels=(log.L_ERR, log.L_TRACEBACK), src=False) |
Neels Hofmeyr | f816688 | 2017-05-05 19:48:35 +0200 | [diff] [blame] | 137 | |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 138 | if args.log_level: |
| 139 | log.set_all_levels(log.LEVEL_STRS.get(args.log_level)) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 140 | if args.trace: |
| 141 | log.style_change(trace=True) |
Neels Hofmeyr | 42bebfa | 2017-06-06 19:41:42 +0200 | [diff] [blame] | 142 | if args.source: |
| 143 | log.style_change(src=True) |
Pau Espin Pedrol | 6c6c0e8 | 2020-05-11 18:30:58 +0200 | [diff] [blame] | 144 | if args.conf_path: |
| 145 | config.override_conf = args.conf_path |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 146 | |
Pau Espin Pedrol | e972c9c | 2020-05-12 15:06:55 +0200 | [diff] [blame] | 147 | if args.trial_dir is not None: |
| 148 | trial_dir = args.trial_dir |
| 149 | else: |
| 150 | trial_dir = config.get_main_config_value(config.CFG_TRIAL_DIR) |
| 151 | |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 152 | combination_strs = list(args.suite_scenario or []) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 153 | |
| 154 | if not combination_strs: |
Pau Espin Pedrol | 6c6c0e8 | 2020-05-11 18:30:58 +0200 | [diff] [blame] | 155 | combination_strs = config.read_config_file(config.CFG_DEFAULT_SUITES_CONF, if_missing_return=[]) |
Neels Hofmeyr | d46ea13 | 2017-04-08 15:56:31 +0200 | [diff] [blame] | 156 | |
| 157 | if combination_strs: |
| 158 | print('Running default suites:\n ' + ('\n '.join(combination_strs))) |
Your Name | 3c6673a | 2017-04-08 18:52:39 +0200 | [diff] [blame] | 159 | else: |
Pau Espin Pedrol | 6c6c0e8 | 2020-05-11 18:30:58 +0200 | [diff] [blame] | 160 | print('Failed to load default suites (%r)' % config.get_main_config_value(config.DEFAULT_SUITES_CONF, fail_if_missing=False)) |
Your Name | 3c6673a | 2017-04-08 18:52:39 +0200 | [diff] [blame] | 161 | |
Neels Hofmeyr | d46ea13 | 2017-04-08 15:56:31 +0200 | [diff] [blame] | 162 | |
| 163 | if not combination_strs: |
Pau Espin Pedrol | 0bd048c | 2020-05-07 19:32:21 +0200 | [diff] [blame] | 164 | raise RuntimeError('Need at least one suite:scenario to run') |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 165 | |
Pau Espin Pedrol | ea8c3d4 | 2020-05-04 12:05:05 +0200 | [diff] [blame] | 166 | # Generate supported schemas dynamically from objects: |
| 167 | generate_schemas() |
| 168 | |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 169 | # make sure all suite:scenarios exist |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 170 | suite_scenarios = [] |
| 171 | for combination_str in combination_strs: |
| 172 | suite_scenarios.append(suite.load_suite_scenario_str(combination_str)) |
| 173 | |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 174 | # pick tests and make sure they exist |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 175 | test_names = [] |
| 176 | for test_name in (args.test or []): |
| 177 | found = False |
Neels Hofmeyr | f9de78f | 2017-05-06 16:04:37 +0200 | [diff] [blame] | 178 | if test_name.startswith('=') and not test_name.endswith('.py'): |
| 179 | test_name = test_name + '.py' |
Neels Hofmeyr | 930ac95 | 2017-05-06 15:49:53 +0200 | [diff] [blame] | 180 | for suite_scenario_str, suite_def, scenarios in suite_scenarios: |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 181 | for def_test_name in suite_def.test_basenames: |
Neels Hofmeyr | f9de78f | 2017-05-06 16:04:37 +0200 | [diff] [blame] | 182 | if test_name.startswith('='): |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 183 | match = test_name[1:] == def_test_name |
Neels Hofmeyr | f9de78f | 2017-05-06 16:04:37 +0200 | [diff] [blame] | 184 | else: |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 185 | match = test_name in def_test_name |
Neels Hofmeyr | f9de78f | 2017-05-06 16:04:37 +0200 | [diff] [blame] | 186 | if match: |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 187 | found = True |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 188 | test_names.append(def_test_name) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 189 | if not found: |
| 190 | raise RuntimeError('No test found for %r' % test_name) |
| 191 | if test_names: |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 192 | test_names = sorted(set(test_names)) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 193 | print(repr(test_names)) |
| 194 | |
Pau Espin Pedrol | e972c9c | 2020-05-12 15:06:55 +0200 | [diff] [blame] | 195 | with trial.Trial(trial_dir) as current_trial: |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 196 | current_trial.verify() |
| 197 | for suite_scenario_str, suite_def, scenarios in suite_scenarios: |
| 198 | current_trial.add_suite_run(suite_scenario_str, suite_def, scenarios) |
| 199 | current_trial.run_suites(test_names) |
Neels Hofmeyr | 3531a19 | 2017-03-28 14:30:28 +0200 | [diff] [blame] | 200 | |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 201 | if current_trial.status != trial.Trial.PASS: |
| 202 | return 1 |
| 203 | return 0 |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 204 | |
Neels Hofmeyr | 2123541 | 2017-05-14 03:00:21 +0200 | [diff] [blame] | 205 | if __name__ == '__main__': |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 206 | rc = 2 |
Neels Hofmeyr | 2123541 | 2017-05-14 03:00:21 +0200 | [diff] [blame] | 207 | try: |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 208 | rc = main() |
Neels Hofmeyr | 2123541 | 2017-05-14 03:00:21 +0200 | [diff] [blame] | 209 | except: |
| 210 | # Tell the log to show the exception, then terminate the program with the exception anyway. |
| 211 | # Since exceptions within test runs should be caught and evaluated, this is basically about |
| 212 | # exceptions during command line parsing and such, so it's appropriate to abort immediately. |
| 213 | log.log_exn() |
| 214 | raise |
Neels Hofmeyr | 6ccda11 | 2017-06-06 19:41:17 +0200 | [diff] [blame] | 215 | exit(rc) |
Neels Hofmeyr | 2123541 | 2017-05-14 03:00:21 +0200 | [diff] [blame] | 216 | |
Neels Hofmeyr | dae3d3c | 2017-03-28 12:16:58 +0200 | [diff] [blame] | 217 | # vim: expandtab tabstop=4 shiftwidth=4 |