| Osmocom TTCN-3 Test Suites |
| ========================== |
| :author: Harald Welte <laforge@gnumonks.org> |
| :copyright: 2018 by Harald Welte (License: CC-BY-SA) |
| :backend: slidy |
| :max-width: 45em |
| |
| |
| == Osmocom TTCN-3 Test Suites |
| |
| * developed in 2017+2018 |
| * compiled using Eclipse TITAN |
| ** uses just a command-line compiler + Makefiles |
| ** no IDE needed at all, don't let _Eclipse_ fool you |
| * containerized in Docker |
| * executed by Jenkins CI |
| |
| == Terminology |
| |
| ATS:: Abstract Test Suite |
| MTC:: Main Test Component |
| PTC:: Parallel Test Component |
| IUT:: Implementation Under Test |
| |
| == Test Suite Philosophy |
| |
| * test one network element (our IUT) |
| * test external behavior (3GPP and non-3GPP) |
| * emulate entire environment from TTCN-3 |
| * don't reuse Osmocom C-code protocol implementations in the tests |
| * test against independent TTCN-3 implementations! |
| |
| == What to test? |
| |
| * successful cases |
| * erroneous cases (no answer, NACK, ...) |
| ** many difficult to reproduce with real phones/devices |
| * load / resource exhaustion |
| * spec compliance |
| * focus on functionality actually relevant to IUT |
| |
| == Why TTCN-3 + TITAN |
| |
| * TTCN-3 specifically designed for telecom protocol testing |
| * TITAN team released many telecom protocols in TTCN-3, such as |
| ** BSSAP, L3 (RR/MM/CC), SMS (CP/RP/TP), SS, M3UA, SCCP, GTP, NS, BSSGP, ... |
| ** shortens our test development cycle |
| ** permits us to test against known working industry implementations |
| |
| == Test suites for Osmocom CNI components |
| |
| * `osmo-bts` |
| * `osmo-bsc` |
| * `osmo-msc` |
| * `osmo-mgw` |
| * `osmo-hlr` |
| * `osmo-sip-connector` |
| * `osmo-sgsn` |
| * `osmo-ggsn` |
| |
| == Test suites in progress |
| |
| * `osmo-pcu` |
| * `osmo-bsc_nat` |
| |
| |
| |
| |
| == BTS_Tests.ttcn |
| |
| * external interfaces |
| ** A-bis side: RSL (emulates BSC-side server) |
| ** Um side: L1CTL to control MS |
| ** PCU side: pcu_socket |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| { rank=same; BTS, BSC}; |
| BTS [label="IUT\nosmo-bts-trx",shape="box"]; |
| ATS [label="ATS\nBTS_Tests.ttcn"]; |
| BSC [label="osmo-bsc\nOML only"]; |
| BTS -> fake_trx [label="bursts"]; |
| fake_trx -> trxcon [label="bursts"]; |
| trxcon -> ATS [label="bursts"]; |
| |
| BTS -> BSC [label="A-bis OML"]; |
| BTS -> ATS [label="A-bis RSL"]; |
| |
| ATS -> BTS [label="pcu_sock"]; |
| ATS -> BSC [label="VTY"]; |
| ATS -> BTS [label="CTRL"]; |
| } |
| ---- |
| |
| |
| |
| == BSC_Tests.ttcn |
| |
| * external interfaces |
| ** A-bis side: RSL (emulates BTS-side client) |
| ** A-side: BSSAP/SCCP/M3UA (emulates MSC-side) |
| ** MGW side: MGCP (emulates MGW side) |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| { rank=same; BTS; STP; }; |
| BSC [label="IUT\nosmo-bsc",shape="box"]; |
| ATS [label="ATS\nBSC_Tests.ttcn"]; |
| BTS [label="osmo-bts-omldummy\nOML only"]; |
| |
| BTS -> BSC [label="A-bis OML"]; |
| ATS -> BSC [label="A-bis RSL"]; |
| ATS -> BSC [label="CTRL"]; |
| ATS -> BSC [label="VTY"]; |
| ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| BSC -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| } |
| ---- |
| |
| == MSC_Tests.ttcn |
| |
| * external interfaces |
| ** A: BSSAP/SCCP/M3UA (emulates BSC-side) |
| ** MNCC: MNCC/unix-domain (emulates ext. MNCC side) |
| ** MGW: MGCP (emulates MGW side) |
| ** GSUP (emulates HLR side) |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| MSC [label="IUT\nosmo-msc",shape="box"]; |
| ATS [label="ATS\nMSC_Tests.ttcn"]; |
| |
| ATS -> MSC [label="MNCC"]; |
| ATS -> MSC [label="SMPP",style="dashed"]; |
| ATS -> MSC [label="CTRL"]; |
| ATS -> MSC [label="VTY"]; |
| MSC -> ATS [label="GSUP"]; |
| ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| MSC -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| } |
| ---- |
| |
| |
| == MGCP_Test.ttcn |
| |
| * external interfaces |
| ** MGCP (emulates call agent) |
| ** RTP (stream source/sink) |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| MGW [label="IUT\nosmo-mgw",shape="box"]; |
| ATS [label="ATS\nMGCP_Test.ttcn"]; |
| |
| ATS -> MGW [label="RTP"]; |
| ATS -> MGW [label="MGCP"]; |
| MGW -> ATS [label="RTP"]; |
| } |
| ---- |
| |
| == HLR_Tests.ttcn |
| |
| * external interfaces |
| ** GSUP (emulates VLR/SGSN side) |
| ** VTY |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| HLR [label="IUT\nosmo-hlr",shape="box"]; |
| ATS [label="ATS\nHLR_Tests.ttcn"]; |
| |
| ATS -> HLR [label="GSUP"]; |
| ATS -> HLR [label="VTY"]; |
| } |
| ---- |
| |
| |
| == SIP_Tests.ttcn |
| |
| * external interfaces |
| ** MNCC (emulates MSC side) |
| ** SIP (emulates SIP switch) |
| ** VTY |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| SIP [label="SIP\nosmo-sip-connector",shape="box"]; |
| ATS [label="ATS\nSIP_Tests.ttcn"]; |
| |
| ATS -> SIP [label="MNCC"]; |
| ATS -> SIP [label="SIP"]; |
| ATS -> SIP [label="VTY"]; |
| } |
| ---- |
| |
| |
| == SGSN_Tests.ttcn |
| |
| * external interfaces |
| ** Gb (emulates PCU side NS/BSSGP + MS) |
| ** GSUP (emulates HLR) |
| ** VTY |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| SGSN [label="SGSN\nosmo-sgsn",shape="box"]; |
| ATS [label="ATS\nSGSN_Tests.ttcn"]; |
| |
| ATS -> SGSN [label="Gb"]; |
| SGSN-> ATS [label="Gp (GTP)"]; |
| ATS -> SGSN [label="VTY"]; |
| } |
| ---- |
| |
| |
| == GGSN_Tests.ttcn |
| |
| * external interfaces |
| ** Gp: GTP (emulates SGSN) |
| ** Gi: IP (emulates Internet) |
| |
| [graphviz] |
| ---- |
| digraph G { |
| rankdir=LR; |
| GGSN [label="GGSN\nosmo-ggsn",shape="box"]; |
| ATS [label="ATS\nGGSN_Tests.ttcn"]; |
| |
| ATS -> GGSN [label="Gp (GTP)"]; |
| GGSN -> ATS [label="Gi (IP)"]; |
| ATS -> GGSN [label="VTY"]; |
| } |
| ---- |
| |
| |
| |
| == Dockerized Setup |
| |
| * one process per container |
| * packages either |
| ** IUT (e.g. `osmo-bsc`) |
| ** ATS (compiled docker test suite) |
| ** other utility (e.g. `trxcon` or `osmo-bts-omldummy`) |
| * why? |
| ** no need for local ip/network configuration |
| ** standardized / packaged setup on every machine |
| ** run older/newer versions of ATS against older/newer IUT |
| |
| == Jenkins CI Execution |
| |
| . update `docker-playground.git` |
| .. contains `Dockerfile` for ATS + IUT |
| . rebuild IUT container[s] (e.g. `osmo-bts-master`) |
| .. git magic ensures re-build only if `osmo-bts.git` master changed |
| . rebuild ATS container (e.g. `ttcn3-bts-test`) |
| .. git magic ensures re-build only if `osmo-ttcn3-hacks.git` master changed |
| . run `docker-playground/ttcn3-bts-test/jenkins.sh` |
| .. creates docker network |
| .. starts IUT + ATS docker containers |
| .. collects test results |
| |
| |
| == Jenkins CI Reporting |
| |
| * junit-xml generation |
| * store artefacts |
| ** pcap file of every test case |
| ** ATS log file (TTCN-3 testsuite) |
| ** IUT log file[s] (`osmo-*.log`) |
| ** IUT config file[s] (`osmo-*.cfg`) |
| * see https://jenkins.osmocom.org/jenkins/view/TTCN3/ |
| |
| |
| == Further Reading |
| |
| * http://git.osmocom.org/osmo-ttcn3-hacks/ |
| * http://git.osmocom.org/docker-playground/ |
| * http://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Notes |
| |
| == EOF |
| |
| End of File |