| == Introduction with Examples |
| |
| The osmo-gsm-tester is software to run automated tests of real GSM hardware, |
| foremost to verify that ongoing Osmocom software development continues to work |
| with various BTS models, while being flexibly configurable and extendable. |
| |
| A 'main unit' (general purpose computer) is connected via ethernet and/or USB to |
| any number of BTS models and to any number of GSM modems via USB. The modems |
| and BTS instances' RF transceivers are typically wired directly to each other |
| via RF distribution chambers to bypass the air medium and avoid disturbing real |
| production cellular networks. Furthermore, the setup may include adjustable RF |
| attenuators to model various distances between modems and base stations. |
| |
| The osmo-gsm-tester software runs on the main unit to orchestrate the various |
| GSM hardware and run predefined test scripts. It typically receives binary |
| packages from a jenkins build service. It then automatically configures and |
| launches an Osmocom core network on the main unit and sets up and runs BTS |
| models as well as modems to form a complete ad-hoc GSM network. On this setup, |
| predefined test suites, combined with various scenario definitions, are run to |
| verify stability of the system. |
| |
| The osmo-gsm-tester is implemented in Python (version 3). It uses the ofono |
| daemon to control the modems connected via USB. BTS software is either run |
| directly on the main unit (e.g. for osmo-bts-trx, osmo-bts-octphy), run via SSH |
| (e.g. for a sysmoBTS) or assumed to run on a connected BTS model (e.g. for |
| ip.access nanoBTS). |
| |
| .Typical osmo-gsm-tester setup |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| jenkins |
| subgraph cluster_gsm_hardware { |
| label = "GSM Hardware"; |
| style=dotted |
| |
| modem0 [shape=box label="Modems..."] |
| modem1 [shape=box label="Modems..."] |
| modem2 [shape=box label="Modems..."] |
| osmo_bts_sysmo [label="sysmocom sysmoBTS\nrunning osmo-bts-sysmo" shape=box] |
| B200 [label="Ettus B200" shape=box] |
| octphy [label="Octasic octphy BTS" shape=box] |
| nanoBTS [label="ip.access nanoBTS" shape=box] |
| rf_distribution [label="RF distribution"] |
| |
| {modem0 modem1 modem2 osmo_bts_sysmo B200 octphy nanoBTS}->rf_distribution [dir=both arrowhead="curve" arrowtail="curve"] |
| } |
| subgraph cluster_main_unit { |
| label = "Main Unit" |
| osmo_gsm_tester [label="Osmo-GSM-Tester\ntest suites\n& scenarios"] |
| subgraph { |
| rank=same |
| ofono [label="ofono daemon"] |
| OsmoNITB |
| osmo_bts_trx [label="osmo-bts-trx\n+ osmo-trx"] |
| osmo_bts_octphy [label="osmo-bts-octphy"] |
| } |
| } |
| |
| |
| jenkins->osmo_gsm_tester [label="trial\n(binaries)"] |
| osmo_gsm_tester->jenkins [label="results"] |
| ofono->{modem0 modem1 modem2} [label="USB"] |
| osmo_gsm_tester-> {OsmoNITB osmo_bts_trx osmo_bts_octphy} |
| osmo_gsm_tester-> osmo_bts_sysmo [taillabel="SSH"] |
| osmo_gsm_tester-> ofono [taillabel="DBus"] |
| osmo_bts_trx->B200 [label="USB"] |
| osmo_bts_octphy->octphy [label="raw eth"] |
| {osmo_bts_sysmo nanoBTS}->OsmoNITB [label="eth"] |
| {B200 octphy}->OsmoNITB [label="eth" style=invis] |
| {osmo_bts_trx osmo_bts_octphy}->OsmoNITB |
| } |
| ---- |
| |
| .Example of how to select resources and configurations: scenarios may pick specific resources (here BTS and ARFCN), remaining requirements are picked as available (here two modems and a NITB interface) |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=TB; |
| |
| suite_scenarios [label="Suite+Scenarios selection\nsms:sysmo+band1800"] |
| |
| subgraph { |
| rank=same; |
| suite |
| scenarios |
| } |
| |
| subgraph cluster_scenarios { |
| label = "Scenarios"; |
| u_sysmoBTS [label="Scenario: sysmo\nbts: type: osmo-bts-sysmo"] |
| u_trx [label="Scenario: trx\nbts: type: osmo-bts-trx"] |
| u_arfcn [label="Scenario: band1800\narfcn: band: GSM-1800"] |
| } |
| |
| subgraph cluster_suite { |
| label = "Suite: sms"; |
| requires [label="Requirements (suite.conf):\nmodem: times: 2\nbts\nip_address\narfcn"] |
| subgraph cluster_tests { |
| label = "Test Scripts (py)"; |
| mo_mt_sms |
| etc |
| } |
| } |
| |
| subgraph cluster_resources { |
| label = "Resources"; |
| rankdir=TB; |
| nitb_addr1 [label="NITB interface addr\n10.42.42.1"] |
| nitb_addr2 [label="NITB interface addr\n10.42.42.2"] |
| Modem0 |
| Modem1 |
| Modem2 |
| sysmoBTS [label="osmo-bts-sysmo"] |
| osmo_bts_trx [label="osmo-bts-trx"] |
| arfcn1 [label="arfcn: 512\nband: GSM-1800"] |
| arfcn2 [label="arfcn: 540\nband: GSM-1900"] |
| |
| arfcn1->arfcn2 [style=invis] |
| nitb_addr1->nitb_addr2 [style=invis] |
| Modem0 -> Modem1 -> Modem2 [style=invis] |
| sysmoBTS -> osmo_bts_trx [style=invis] |
| } |
| |
| suite_scenarios -> {suite scenarios} |
| scenarios -> { u_arfcn u_sysmoBTS } |
| |
| suite -> requires |
| requires -> Modem0 |
| requires -> Modem1 |
| requires -> sysmoBTS |
| requires -> arfcn1 |
| requires -> nitb_addr1 |
| |
| { u_sysmoBTS u_arfcn } -> requires [label="influences\nresource\nselection"] |
| } |
| ---- |
| |
| .Example of a "trial" containing binaries built by a jenkins |
| [graphviz] |
| ---- |
| digraph G { |
| subgraph cluster_trial { |
| label = "Trial (binaries)" |
| sysmo [label="osmo-bts-sysmo.build-23.tgz\n(osmo-bts-sysmo\n+ deps\ncompiled for sysmoBTS)"] |
| trx [label="osmo-bts-trx.build-5.tgz\n(osmo-trx + osmo-bts-trx\n+ deps\ncompiled for main unit)"] |
| nitb [label="osmo-nitb.build-42.tgz\n(osmo-nitb\n+ deps\ncompiled for main unit)"] |
| checksums [label="checksums.md5"] |
| |
| checksums -> {sysmo trx nitb} |
| } |
| } |
| ---- |
| |
| === Typical Test Script |
| |
| A typical single test script (part of a suite) may look like this: |
| |
| ---- |
| #!/usr/bin/env python3 |
| from osmo_gsm_tester.test import * |
| |
| nitb = suite.nitb() |
| bts = suite.bts() |
| ms_mo = suite.modem() |
| ms_mt = suite.modem() |
| |
| print('start nitb and bts...') |
| nitb.bts_add(bts) |
| nitb.start() |
| bts.start() |
| |
| nitb.subscriber_add(ms_mo) |
| nitb.subscriber_add(ms_mt) |
| |
| ms_mo.connect(nitb.mcc_mnc()) |
| ms_mt.connect(nitb.mcc_mnc()) |
| |
| print('waiting for modems to attach...') |
| wait(ms_mo.is_connected, nitb.mcc_mnc()) |
| wait(ms_mt.is_connected, nitb.mcc_mnc()) |
| wait(nitb.subscriber_attached, ms_mo, ms_mt) |
| |
| sms = ms_mo.sms_send(ms_mt) |
| wait(ms_mt.sms_was_received, sms) |
| ---- |
| |
| === Resource Resolution |
| |
| - A global configuration defines which hardware is connected to the |
| osmo-gsm-tester main unit. |
| - Each suite contains a number of test scripts. The amount of resources a test |
| may use is defined by the test suite's 'suite.conf'. |
| - Which specific modems, BTS models, NITB IP addresses etc. are made available |
| to a test run is typically determined by a combination of scenario |
| configurations -- or picked automatically if not. |
| |
| [[resources_conf_example]] |
| === Typical 'resources.conf' |
| |
| A global configuration of hardware may look like below; for details, see |
| <<resources_conf>>. |
| |
| ---- |
| ip_address: |
| - addr: 10.42.42.1 |
| - addr: 10.42.42.2 |
| - addr: 10.42.42.3 |
| |
| bts: |
| - label: sysmoBTS 1002 |
| type: osmo-bts-sysmo |
| addr: 10.42.42.114 |
| band: GSM-1800 |
| |
| - label: octBTS 3000 |
| type: osmo-bts-octphy |
| addr: 10.42.42.115 |
| band: GSM-1800 |
| trx_list: |
| - hw_addr: 00:0c:90:32:b5:8a |
| net_device: eth0.2342 |
| |
| - label: Ettus B210 |
| type: osmo-bts-trx |
| addr: 10.42.42.116 |
| band: GSM-1800 |
| |
| - label: nanoBTS 1900 |
| type: nanobts |
| addr: 10.42.42.190 |
| band: GSM-1900 |
| trx_list: |
| - hw_addr: 00:02:95:00:41:b3 |
| |
| arfcn: |
| - arfcn: 512 |
| band: GSM-1800 |
| - arfcn: 514 |
| band: GSM-1800 |
| |
| - arfcn: 540 |
| band: GSM-1900 |
| - arfcn: 542 |
| band: GSM-1900 |
| |
| modem: |
| - label: m7801 |
| path: '/wavecom_0' |
| imsi: 901700000007801 |
| ki: D620F48487B1B782DA55DF6717F08FF9 |
| |
| - label: m7802 |
| path: '/wavecom_1' |
| imsi: 901700000007802 |
| ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3 |
| |
| - label: m7803 |
| path: '/wavecom_2' |
| imsi: 901700000007803 |
| ki: ABBED4C91417DF710F60675B6EE2C8D2 |
| ---- |
| |
| === Typical 'suites/*/suite.conf' |
| |
| The configuration that reserves a number of resources for a test suite may look |
| like this: |
| |
| ---- |
| resources: |
| ip_address: |
| - times: 1 |
| bts: |
| - times: 1 |
| modem: |
| - times: 2 |
| ---- |
| |
| It may also request e.g. specific BTS models, but this is typically left to |
| scenario configurations. |
| |
| === Typical 'scenarios/*.conf' |
| |
| For a suite as above run as-is, any available resources are picked. This may be |
| combined with any number of scenario definitions to constrain which specific |
| resources should be used, e.g.: |
| |
| ---- |
| resources: |
| bts: |
| - type: osmo-bts-sysmo |
| ---- |
| |
| Which 'ip_address' or 'modem' is used in particular doesn't really matter, so |
| it can be left up to the osmo-gsm-tester to pick these automatically. |
| |
| Any number of such scenario configurations can be combined in the form |
| '<suite_name>:<scenario>+<scenario>+...', e.g. 'my_suite:sysmo+tch_f+amr'. |
| |
| === Typical Invocations |
| |
| Each invocation of osmo-gsm-tester deploys a set of pre-compiled binaries for |
| the Osmocom core network as well as for the Osmocom based BTS models. To create |
| such a set of binaries, see <<trials>>. |
| |
| Examples for launching test trials: |
| |
| - Run the default suites (see <<default_suites>>) on a given set of binaries: |
| |
| ---- |
| osmo-gsm-tester.py path/to/my-trial |
| ---- |
| |
| - Run an explicit choice of 'suite:scenario' combinations: |
| |
| ---- |
| osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -s sms:trx -s sms:nanobts |
| ---- |
| |
| - Run one 'suite:scenario' combination, setting log level to 'debug' and |
| enabling logging of full python tracebacks, and also only run just the |
| 'mo_mt_sms.py' test from the suite, e.g. to investigate a test failure: |
| |
| ---- |
| osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -l dbg -T -t mo_mt |
| ---- |
| |
| A test script may also be run step-by-step in a python debugger, see |
| <<debugging>>. |
| |
| === Resource Reservation for Concurrent Trials |
| |
| While a test suite runs, the used resources are noted in a global state |
| directory in a reserved-resources file. This way, any number of trials may be |
| run consecutively without resource conflicts. Any test trial will only use |
| resources that are currently not reserved by any other test suite. The |
| reservation state is human readable. |
| |
| The global state directory is protected by a file lock to allow access by |
| separate processes. |
| |
| Also, the binaries from a trial are never installed system-wide, but are run |
| with a specific 'LD_LIBRARY_PATH' pointing at the trial's 'inst', so that |
| several trials can run consecutively without conflicting binary versions. |
| |
| Once a test suite run is complete, all its reserved resources are torn down (if |
| the test scripts have not done so already), and the reservations are released |
| automatically. |
| |
| If required resources are unavailable, the test trial fails. For consecutive |
| test trials, a test run needs to either wait for resources to become available, |
| or test suites need to be scheduled to make sense. (*<- TODO*) |