doc/manuals/chapters/trial.adoc

[[trials]]
== Trial: Binaries to be Tested

A trial is a set of pre-built sysroot archives to be tested. They are typically built
by jenkins using the build scripts found in osmo-gsm-tester's source in the
'contrib/' dir, see <<install_add_jenkins_slave>>.

A trial comes in the form of a directory containing a number of '<inst-name>.*tgz' tar
archives (containing different sysroots) as well as a 'checksums.md5' file to
verify the tar archives' integrity.

.Example of a "trial" containing binaries built by a jenkins job
[graphviz]
----
digraph G {
	subgraph cluster_trial {
		label = "Trial (binaries)"
		sysmo [label="osmo-bts-sysmo.build-23.tgz\n(osmo-bts-sysmo\n+ deps\ncompiled for sysmoBTS)"]
		trx [label="osmo-bts.build-5.tgz\n(osmo-bts-octphy + osmo-bts-trx\n+ deps\ncompiled for main unit)"]
		nitb [label="osmo-nitb.build-42.tgz\n(osmo-nitb\n+ deps\ncompiled for main unit)"]
		checksums [label="checksums.md5"]

		checksums -> {sysmo trx nitb}
	}
}
----

When the osmo-gsm-tester is invoked to run on such a trial directory, it will
create a sub directory named 'inst' and unpack the tar archives into it.

For each test run on this trial, a new subdirectory in the trial dir is
created, named in the form of 'run.<timestamp>'. A symbolic link 'last-run'
will point at the most recently created run dir. This run dir will accumulate:

* the rendered configuration files used to run the binaries
* stdout and stderr outputs of the binaries
* pcap files for processes doing relevant network communication
* a test log
* jenkins parsable XML (Junit) reports

The script in 'contrib/jenkins-run.sh' takes care of related tasks such as

* creating the dir structure,
* generating md5 sums for the various tar.gz containing software builds to be tested,
* cleaning up after the build,
* saving extra logs such as journalctl output from ofonod,
* generating a final .tar.gz file with all the logs and reports to store as jenkins archives.

{app-name} tests create objects to manage the allocated resources during test
lifetime. These objects, in turn, usually run and manage processes started from
the trial's sysroot binaries. {app-name} provide APIs for those object classes
to discover, unpack and run those binaries. An object class simply needs to
request the name of the sysroot it wants to use (for instance 'osmo-bsc'), and
{app-name} will take care of preparing everything and providing the sysroot path
to it. It's a duty of the resource class to copy over the sysroot to the
destination if the intention is to run the binary remotely on another host.

Moreover, {app-name} supports working with several different versions of a given
sysroot by means of storing them in different subdirectories, which are later
referenced by an object's class 'run_label' attribute named after that
subdirectory. This way, for instance, a sysroot can be provided containing
binaries linked against libraries present on a CentOS distribution, and other
sysroots with the same name can also be provided which are linked against
different versions of CentOS, or a different distro like Debian, or even a
different arch like ARM.

When seeking a sysroot of a given name '<inst-name>' in the 'inst/' directory,
{app-name} will look for 'tgz' files starting with the pattern '<inst-name>.'
(up to the first dot). That means, suffixes are available for {app-name} user to
identify the content, for instance having an incrementing version counter or a
commit hash. Hence, these example files are considered valid and will be
selected by {app-name} for 'osmo-bsc': 'osmo-bsc.tgz', 'osmo-bsc.build-23.tgz',
'osmo-bsc.5f3e0dd2.tgz', 'osmo-bsc.armv7.build-2.tgz'. If either none or more
than one valid file is found matching the pattern, an exception will be thrown.
If a 'run_label=foobar' is provided, {app-name} will look up for the 'tgz' in
exactly the same way, but this time in 'inst/foobar/' directory instead of
'inst/'.