== Introduction with Examples

The osmo-gsm-tester is software to run automated tests of real GSM hardware,
foremost to verify that ongoing Osmocom software development continues to work
with various BTS models, while being flexibly configurable and extendable.

A 'main unit' (general purpose computer) is connected via ethernet and/or USB to
any number of BTS models and to any number of GSM modems via USB. The modems
and BTS instances' RF transceivers are typically wired directly to each other
via RF distribution chambers to bypass the air medium and avoid disturbing real
production cellular networks. Furthermore, the setup may include adjustable RF
attenuators to model various distances between modems and base stations.

The osmo-gsm-tester software runs on the main unit to orchestrate the various
GSM hardware and run predefined test scripts. It typically receives binary
packages from a jenkins build service. It then automatically configures and
launches an Osmocom core network on the main unit and sets up and runs BTS
models as well as modems to form a complete ad-hoc GSM network. On this setup,
predefined test suites, combined with various scenario definitions, are run to
verify stability of the system.

The osmo-gsm-tester is implemented in Python (version 3). It uses the ofono
daemon to control the modems connected via USB. BTS software is either run
directly on the main unit (e.g. for osmo-bts-trx, osmo-bts-octphy), run via SSH
(e.g. for a sysmoBTS) or assumed to run on a connected BTS model (e.g. for
ip.access nanoBTS).

.Typical osmo-gsm-tester setup
[graphviz]
----
digraph G {
	rankdir=LR;
	jenkins
	subgraph cluster_gsm_hardware {
		label = "GSM Hardware";
		style=dotted

		modem0 [shape=box label="Modems..."]
		modem1 [shape=box label="Modems..."]
		modem2 [shape=box label="Modems..."]
		osmo_bts_sysmo [label="sysmocom sysmoBTS\nrunning osmo-bts-sysmo" shape=box]
		B200 [label="Ettus B200" shape=box]
		octphy [label="Octasic octphy BTS" shape=box]
		nanoBTS [label="ip.access nanoBTS" shape=box]
		rf_distribution [label="RF distribution"]

		{modem0 modem1 modem2 osmo_bts_sysmo B200 octphy nanoBTS}->rf_distribution [dir=both arrowhead="curve" arrowtail="curve"]
	}
	subgraph cluster_main_unit {
	  label = "Main Unit"
	  osmo_gsm_tester [label="Osmo-GSM-Tester\ntest suites\n& scenarios"]
	  subgraph {
		rank=same
		ofono [label="ofono daemon"]
		OsmoNITB
		osmo_bts_trx [label="osmo-bts-trx\n+ osmo-trx"]
		osmo_bts_octphy [label="osmo-bts-octphy"]
	  }
	}


	jenkins->osmo_gsm_tester [label="trial\n(binaries)"]
	osmo_gsm_tester->jenkins [label="results"]
	ofono->{modem0 modem1 modem2} [label="USB"]
	osmo_gsm_tester-> {OsmoNITB osmo_bts_trx osmo_bts_octphy}
	osmo_gsm_tester-> osmo_bts_sysmo [taillabel="SSH"]
	osmo_gsm_tester-> ofono [taillabel="DBus"]
	osmo_bts_trx->B200 [label="USB"]
	osmo_bts_octphy->octphy [label="raw eth"]
	{osmo_bts_sysmo nanoBTS}->OsmoNITB [label="eth"]
	{B200 octphy}->OsmoNITB [label="eth" style=invis]
	{osmo_bts_trx osmo_bts_octphy}->OsmoNITB
}
----

.Example of how to select resources and configurations: scenarios may pick specific resources (here BTS and ARFCN), remaining requirements are picked as available (here two modems and a NITB interface)
[graphviz]
----
digraph G {
	rankdir=TB;

	suite_scenarios [label="Suite+Scenarios selection\nsms:sysmo+band1800"]

	subgraph {
		rank=same;
		suite
		scenarios
	}

	subgraph cluster_scenarios {
		label = "Scenarios";
		u_sysmoBTS [label="Scenario: sysmo\nbts: type: osmo-bts-sysmo"]
		u_trx [label="Scenario: trx\nbts: type: osmo-bts-trx"]
		u_arfcn [label="Scenario: band1800\narfcn: band: GSM-1800"]
	}

	subgraph cluster_suite {
		label = "Suite: sms";
		requires [label="Requirements (suite.conf):\nmodem: times: 2\nbts\nip_address\narfcn"]
		subgraph cluster_tests {
			label = "Test Scripts (py)";
			mo_mt_sms
			etc
		}
	}

	subgraph cluster_resources {
		label = "Resources";
		rankdir=TB;
			nitb_addr1 [label="NITB interface addr\n10.42.42.1"]
			nitb_addr2 [label="NITB interface addr\n10.42.42.2"]
			Modem0
			Modem1
			Modem2
			sysmoBTS [label="osmo-bts-sysmo"]
			osmo_bts_trx [label="osmo-bts-trx"]
			arfcn1 [label="arfcn: 512\nband: GSM-1800"]
			arfcn2 [label="arfcn: 540\nband: GSM-1900"]

			arfcn1->arfcn2 [style=invis]
			nitb_addr1->nitb_addr2 [style=invis]
			Modem0 -> Modem1 -> Modem2 [style=invis]
			sysmoBTS -> osmo_bts_trx [style=invis]
	}

	suite_scenarios -> {suite scenarios}
	scenarios -> { u_arfcn u_sysmoBTS }

	suite -> requires
	requires -> Modem0
	requires -> Modem1
	requires -> sysmoBTS
	requires -> arfcn1
	requires -> nitb_addr1

	{ u_sysmoBTS u_arfcn } -> requires [label="influences\nresource\nselection"]
}
----

.Example of a "trial" containing binaries built by a jenkins
[graphviz]
----
digraph G {
	subgraph cluster_trial {
		label = "Trial (binaries)"
		sysmo [label="osmo-bts-sysmo.build-23.tgz\n(osmo-bts-sysmo\n+ deps\ncompiled for sysmoBTS)"]
		trx [label="osmo-bts-trx.build-5.tgz\n(osmo-trx + osmo-bts-trx\n+ deps\ncompiled for main unit)"]
		nitb [label="osmo-nitb.build-42.tgz\n(osmo-nitb\n+ deps\ncompiled for main unit)"]
		checksums [label="checksums.md5"]

		checksums -> {sysmo trx nitb}
	}
}
----

=== Typical Test Script

A typical single test script (part of a suite) may look like this:

----
#!/usr/bin/env python3
from osmo_gsm_tester.test import *

nitb = suite.nitb()
bts = suite.bts()
ms_mo = suite.modem()
ms_mt = suite.modem()

print('start nitb and bts...')
nitb.bts_add(bts)
nitb.start()
bts.start()

nitb.subscriber_add(ms_mo)
nitb.subscriber_add(ms_mt)

ms_mo.connect(nitb.mcc_mnc())
ms_mt.connect(nitb.mcc_mnc())

print('waiting for modems to attach...')
wait(ms_mo.is_connected, nitb.mcc_mnc())
wait(ms_mt.is_connected, nitb.mcc_mnc())
wait(nitb.subscriber_attached, ms_mo, ms_mt)

sms = ms_mo.sms_send(ms_mt)
wait(ms_mt.sms_was_received, sms)
----

=== Resource Resolution

- A global configuration defines which hardware is connected to the
  osmo-gsm-tester main unit.
- Each suite contains a number of test scripts. The amount of resources a test
  may use is defined by the test suite's 'suite.conf'.
- Which specific modems, BTS models, NITB IP addresses etc. are made available
  to a test run is typically determined by a combination of scenario
  configurations -- or picked automatically if not.

[[resources_conf_example]]
=== Typical 'resources.conf'

A global configuration of hardware may look like below; for details, see
<<resources_conf>>.

----
ip_address:
- addr: 10.42.42.1
- addr: 10.42.42.2
- addr: 10.42.42.3

bts:
- label: sysmoBTS 1002
  type: osmo-bts-sysmo
  addr: 10.42.42.114
  band: GSM-1800

- label: octBTS 3000
  type: osmo-bts-octphy
  addr: 10.42.42.115
  band: GSM-1800
  trx_list:
  - hw_addr: 00:0c:90:32:b5:8a
    net_device: eth0.2342

- label: Ettus B210
  type: osmo-bts-trx
  addr: 10.42.42.116
  band: GSM-1800

- label: nanoBTS 1900
  type: nanobts
  addr: 10.42.42.190
  band: GSM-1900
  trx_list:
  - hw_addr: 00:02:95:00:41:b3

arfcn:
  - arfcn: 512
    band: GSM-1800
  - arfcn: 514
    band: GSM-1800

  - arfcn: 540
    band: GSM-1900
  - arfcn: 542
    band: GSM-1900

modem:
- label: m7801
  path: '/wavecom_0'
  imsi: 901700000007801
  ki: D620F48487B1B782DA55DF6717F08FF9

- label: m7802
  path: '/wavecom_1'
  imsi: 901700000007802
  ki: 47FDB2D55CE6A10A85ABDAD034A5B7B3

- label: m7803
  path: '/wavecom_2'
  imsi: 901700000007803
  ki: ABBED4C91417DF710F60675B6EE2C8D2
----

=== Typical 'suites/*/suite.conf'

The configuration that reserves a number of resources for a test suite may look
like this:

----
resources:
  ip_address:
  - times: 1
  bts:
  - times: 1
  modem:
  - times: 2
----

It may also request e.g. specific BTS models, but this is typically left to
scenario configurations.

=== Typical 'scenarios/*.conf'

For a suite as above run as-is, any available resources are picked. This may be
combined with any number of scenario definitions to constrain which specific
resources should be used, e.g.:

----
resources:
  bts:
  - type: osmo-bts-sysmo
----

Which 'ip_address' or 'modem' is used in particular doesn't really matter, so
it can be left up to the osmo-gsm-tester to pick these automatically.

Any number of such scenario configurations can be combined in the form
'<suite_name>:<scenario>+<scenario>+...', e.g. 'my_suite:sysmo+tch_f+amr'.

=== Typical Invocations

Each invocation of osmo-gsm-tester deploys a set of pre-compiled binaries for
the Osmocom core network as well as for the Osmocom based BTS models. To create
such a set of binaries, see <<trials>>.

Examples for launching test trials:

- Run the default suites (see <<default_suites>>) on a given set of binaries:

----
osmo-gsm-tester.py path/to/my-trial
----

- Run an explicit choice of 'suite:scenario' combinations:

----
osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -s sms:trx -s sms:nanobts
----

- Run one 'suite:scenario' combination, setting log level to 'debug' and
  enabling logging of full python tracebacks, and also only run just the
  'mo_mt_sms.py' test from the suite, e.g. to investigate a test failure:

----
osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -l dbg -T -t mo_mt
----

A test script may also be run step-by-step in a python debugger, see
<<debugging>>.

=== Resource Reservation for Concurrent Trials

While a test suite runs, the used resources are noted in a global state
directory in a reserved-resources file. This way, any number of trials may be
run consecutively without resource conflicts. Any test trial will only use
resources that are currently not reserved by any other test suite. The
reservation state is human readable.

The global state directory is protected by a file lock to allow access by
separate processes.

Also, the binaries from a trial are never installed system-wide, but are run
with a specific 'LD_LIBRARY_PATH' pointing at the trial's 'inst', so that
several trials can run consecutively without conflicting binary versions.

Once a test suite run is complete, all its reserved resources are torn down (if
the test scripts have not done so already), and the reservations are released
automatically.

If required resources are unavailable, the test trial fails. For consecutive
test trials, a test run needs to either wait for resources to become available,
or test suites need to be scheduled to make sense. (*<- TODO*)