== Introduction with Examples

The osmo-gsm-tester is software to run automated tests of real GSM hardware,
foremost to verify that ongoing Osmocom software development continues to work
with various BTS models, while being flexibly configurable and extendable.

A 'main unit' (general purpose computer) is connected via ethernet and/or USB to
any number of BTS models and to any number of GSM modems via USB. The modems
and BTS instances' RF transceivers are typically wired directly to each other
via RF distribution chambers to bypass the air medium and avoid disturbing real
production cellular networks. Furthermore, the setup may include adjustable RF
attenuators to model various distances between modems and base stations.

The osmo-gsm-tester software runs on the main unit to orchestrate the various
GSM hardware and run predefined test scripts. It typically receives binary
packages from a jenkins build service. It then automatically configures and
launches an Osmocom core network on the main unit and sets up and runs BTS
models as well as modems to form a complete ad-hoc GSM network. On this setup,
predefined test suites, combined with various scenario definitions, are run to
verify stability of the system.

The osmo-gsm-tester is implemented in Python (version 3). It uses the ofono
daemon to control the modems connected via USB. BTS software is either run
directly on the main unit (e.g. for osmo-bts-trx, osmo-bts-octphy), run via SSH
(e.g. for a sysmoBTS) or assumed to run on a connected BTS model (e.g. for
ip.access nanoBTS).

.Typical osmo-gsm-tester setup
[graphviz]
----
digraph G {
	rankdir=LR;
	jenkins
	subgraph cluster_gsm_hardware {
		label = "GSM Hardware";
		style=dotted

		modem0 [shape=box label="Modems..."]
		modem1 [shape=box label="Modems..."]
		osmo_bts_sysmo [label="sysmocom sysmoBTS\nrunning osmo-bts-sysmo" shape=box]
		B200 [label="Ettus B200" shape=box]
		sysmoCell5K [label="sysmocom sysmoCell5000" shape=box]
		octphy [label="Octasic octphy BTS" shape=box]
		nanoBTS [label="ip.access nanoBTS" shape=box]
		rf_distribution [label="RF distribution"]

		{modem0 modem1 osmo_bts_sysmo B200 octphy nanoBTS sysmoCell5K}->rf_distribution [dir=both arrowhead="curve" arrowtail="curve"]
	}
	subgraph cluster_main_unit {
	  label = "Main Unit"
	  osmo_gsm_tester [label="Osmo-GSM-Tester\ntest suites\n& scenarios"]
	  subgraph {
		rank=same
		ofono [label="ofono daemon"]
		osmo_trx [label="osmo-trx"]
		osmo_bts_trx [label="osmo-bts-trx"]
		osmo_bts_octphy [label="osmo-bts-octphy"]
		OsmoNITB [label="BSC + Core Network\n(Osmo{NITB,MSC,BSC,HLR,...})"]
	  }
	}


	jenkins->osmo_gsm_tester [label="trial\n(binaries)"]
	osmo_gsm_tester->jenkins [label="results"]
	ofono->{modem0 modem1} [label="USB"]
	osmo_gsm_tester->{OsmoNITB osmo_bts_trx osmo_bts_octphy}
	osmo_gsm_tester->osmo_bts_sysmo [taillabel="SSH"]
	osmo_gsm_tester->ofono [taillabel="DBus"]
	osmo_trx->B200 [label="USB"]
	osmo_bts_trx->{osmo_trx sysmoCell5K} [dir=both label="UDP"]
	osmo_bts_octphy->octphy [label="raw eth"]
	{osmo_bts_sysmo nanoBTS}->OsmoNITB [label="IP"]
	{B200 octphy}->OsmoNITB [label="eth" style=invis]
	{osmo_bts_trx osmo_bts_octphy}->OsmoNITB
}
----

.Example of how to select resources and configurations: scenarios may pick specific resources (here BTS and ARFCN), remaining requirements are picked as available (here two modems and a NITB interface)
[graphviz]
----
digraph G {
	rankdir=TB;

	suite_scenarios [label="Suite+Scenarios selection\nsms:sysmo+band1800"]

	subgraph {
		rank=same;
		suite
		scenarios
	}

	subgraph cluster_scenarios {
		label = "Scenarios";
		u_sysmoBTS [label="Scenario: sysmo\nbts: type: osmo-bts-sysmo"]
		u_trx [label="Scenario: trx\nbts: type: osmo-bts-trx"]
		u_arfcn [label="Scenario: band1800\narfcn: band: GSM-1800"]
	}

	subgraph cluster_suite {
		label = "Suite: sms";
		requires [label="Requirements (suite.conf):\nmodem: times: 2\nbts\nip_address\narfcn"]
		subgraph cluster_tests {
			label = "Test Scripts (py)";
			mo_mt_sms
			etc
		}
	}

	subgraph cluster_resources {
		label = "Resources";
		rankdir=TB;
			nitb_addr1 [label="NITB interface addr\n10.42.42.1"]
			nitb_addr2 [label="NITB interface addr\n10.42.42.2"]
			Modem0
			Modem1
			Modem2
			sysmoBTS [label="osmo-bts-sysmo"]
			osmo_bts_trx [label="osmo-bts-trx"]
			arfcn1 [label="arfcn: 512\nband: GSM-1800"]
			arfcn2 [label="arfcn: 540\nband: GSM-1900"]

			arfcn1->arfcn2 [style=invis]
			nitb_addr1->nitb_addr2 [style=invis]
			Modem0 -> Modem1 -> Modem2 [style=invis]
			sysmoBTS -> osmo_bts_trx [style=invis]
	}

	suite_scenarios -> {suite scenarios}
	scenarios -> { u_arfcn u_sysmoBTS }

	suite -> requires
	requires -> Modem0
	requires -> Modem1
	requires -> sysmoBTS
	requires -> arfcn1
	requires -> nitb_addr1

	{ u_sysmoBTS u_arfcn } -> requires [label="influences\nresource\nselection"]
}
----

.Example of a "trial" containing binaries built by a jenkins
[graphviz]
----
digraph G {
	subgraph cluster_trial {
		label = "Trial (binaries)"
		sysmo [label="osmo-bts-sysmo.build-23.tgz\n(osmo-bts-sysmo\n+ deps\ncompiled for sysmoBTS)"]
		trx [label="osmo-bts.build-5.tgz\n(osmo-bts-octphy + osmo-bts-trx\n+ deps\ncompiled for main unit)"]
		nitb [label="osmo-nitb.build-42.tgz\n(osmo-nitb\n+ deps\ncompiled for main unit)"]
		checksums [label="checksums.md5"]

		checksums -> {sysmo trx nitb}
	}
}
----

=== Typical Test Script

A typical single test script (part of a suite) may look like this:

----
#!/usr/bin/env python3
from osmo_gsm_tester.test import *

hlr = suite.hlr()
bts = suite.bts()
mgcpgw = suite.mgcpgw(bts_ip=bts.remote_addr())
msc = suite.msc(hlr, mgcpgw)
bsc = suite.bsc(msc)
stp = suite.stp()
ms_mo = suite.modem()
ms_mt = suite.modem()

hlr.start()
stp.start()
msc.start()
mgcpgw.start()

bsc.bts_add(bts)
bsc.start()

bts.start()

hlr.subscriber_add(ms_mo)
hlr.subscriber_add(ms_mt)

ms_mo.connect(msc.mcc_mnc())
ms_mt.connect(msc.mcc_mnc())

ms_mo.log_info()
ms_mt.log_info()

print('waiting for modems to attach...')
wait(ms_mo.is_connected, msc.mcc_mnc())
wait(ms_mt.is_connected, msc.mcc_mnc())
wait(msc.subscriber_attached, ms_mo, ms_mt)

sms = ms_mo.sms_send(ms_mt)
wait(ms_mt.sms_was_received, sms)
----

=== Resource Resolution

- A global configuration 'resources.conf' defines which hardware is connected to the
  osmo-gsm-tester main unit.
- Each suite contains a number of test scripts. The amount of resources a test
  may use is defined by the test suite's 'suite.conf'.
- Which specific modems, BTS models, NITB IP addresses etc. are made available
  to a test run is typically determined by 'suite.conf' and a combination of scenario
  configurations -- or picked automatically if not.

[[resources_conf_example]]
=== Typical 'resources.conf'

A global configuration of hardware may look like below; for details, see
<<resources_conf>>.

----
ip_address:
- addr: 10.42.42.2
- addr: 10.42.42.3
- addr: 10.42.42.4
- addr: 10.42.42.5
- addr: 10.42.42.6

bts:
- label: sysmoBTS 1002
  type: osmo-bts-sysmo
  ipa_unit_id: 1
  addr: 10.42.42.114
  band: GSM-1800
  ciphers:
  - a5_0
  - a5_1
  - a5_3

- label: Ettus B200
  type: osmo-bts-trx
  ipa_unit_id: 6
  addr: 10.42.42.50
  band: GSM-1800
  launch_trx: true
  ciphers:
  - a5_0
  - a5_1

- label: sysmoCell 5000
  type: osmo-bts-trx
  ipa_unit_id: 7
  addr: 10.42.42.51
  band: GSM-1800
  trx_remote_ip: 10.42.42.112
  ciphers:
  - a5_0
  - a5_1

- label: OCTBTS 3500
  type: osmo-bts-octphy
  ipa_unit_id: 8
  addr: 10.42.42.52
  band: GSM-1800
  trx_list:
  - hw_addr: 00:0c:90:2e:80:1e
    net_device: eth1
  - hw_addr: 00:0c:90:2e:87:52
    net_device: eth1

arfcn:
  - arfcn: 512
    band: GSM-1800
  - arfcn: 514
    band: GSM-1800
  - arfcn: 516
    band: GSM-1800
  - arfcn: 546
    band: GSM-1900
  - arfcn: 548
    band: GSM-1900

modem:
- label: sierra_1
  path: '/sierra_1'
  imsi: '901700000009031'
  ki: '80A37E6FDEA931EAC92FFA5F671EFEAD'
  auth_algo: 'xor'
  ciphers:
  - a5_0
  - a5_1
  features:
  - 'sms'
  - 'voice'

- label: gobi_0
  path: '/gobi_0'
  imsi: '901700000009030'
  ki: 'BB70807226393CDBAC8DD3439FF54252'
  auth_algo: 'xor'
  ciphers:
  - a5_0
  - a5_1
  features:
  - 'sms'
----

=== Typical 'suites/*/suite.conf'

The configuration that reserves a number of resources for a test suite may look
like this:

----
resources:
  ip_address:
  - times: 1
  bts:
  - times: 1
  modem:
  - times: 2
    features:
    - sms
----

It may also request e.g. specific BTS models, but this is typically left to
scenario configurations.

=== Typical 'scenarios/*.conf'

For a suite as above run as-is, any available resources are picked. This may be
combined with any number of scenario definitions to constrain which specific
resources should be used, e.g.:

----
resources:
  bts:
  - type: osmo-bts-sysmo
----

Which 'ip_address' or 'modem' is used in particular doesn't really matter, so
it can be left up to the osmo-gsm-tester to pick these automatically.

Any number of such scenario configurations can be combined in the form
'<suite_name>:<scenario>+<scenario>+...', e.g. 'my_suite:sysmo+tch_f+amr'.

=== Typical Invocations

Each invocation of osmo-gsm-tester deploys a set of pre-compiled binaries for
the Osmocom core network as well as for the Osmocom based BTS models. To create
such a set of binaries, see <<trials>>.

Examples for launching test trials:

- Run the default suites (see <<default_suites>>) on a given set of binaries:

----
osmo-gsm-tester.py path/to/my-trial
----

- Run an explicit choice of 'suite:scenario' combinations:

----
osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -s sms:trx -s sms:nanobts
----

- Run one 'suite:scenario' combination, setting log level to 'debug' and
  enabling logging of full python tracebacks, and also only run just the
  'mo_mt_sms.py' test from the suite, e.g. to investigate a test failure:

----
osmo-gsm-tester.py path/to/my-trial -s sms:sysmo -l dbg -T -t mo_mt
----

A test script may also be run step-by-step in a python debugger, see
<<debugging>>.

=== Resource Reservation for Concurrent Trials

While a test suite runs, the used resources are noted in a global state
directory in a reserved-resources file. This way, any number of trials may be
run consecutively without resource conflicts. Any test trial will only use
resources that are currently not reserved by any other test suite. The
reservation state is human readable.

The global state directory is protected by a file lock to allow access by
separate processes.

Also, the binaries from a trial are never installed system-wide, but are run
with a specific 'LD_LIBRARY_PATH' pointing at the trial's 'inst', so that
several trials can run consecutively without conflicting binary versions. For
some specific binaries which require extra permissions (such as osmo-bts-octphy
requiring 'CAP_NET_RAW'), 'patchelf' program is used to modify the binary
'RPATH' field instead because the OS dynamic linker skips 'LD_LIBRARY_PATH' for
binaries with special permissions.

Once a test suite run is complete, all its reserved resources are torn down (if
the test scripts have not done so already), and the reservations are released
automatically.

If required resources are unavailable, the test trial fails. For consecutive
test trials, a test run needs to either wait for resources to become available,
or test suites need to be scheduled to make sense. (*<- TODO*)