Harald Welte | 3db0311 | 2018-08-30 20:55:54 +0200 | [diff] [blame] | 1 | Osmocom TTCN-3 Test Suites |
| 2 | ========================== |
| 3 | :author: Harald Welte <laforge@gnumonks.org> |
| 4 | :copyright: 2018 by Harald Welte (License: CC-BY-SA) |
| 5 | :backend: slidy |
| 6 | :max-width: 45em |
| 7 | |
| 8 | |
| 9 | == Osmocom TTCN-3 Test Suites |
| 10 | |
| 11 | * developed in 2017+2018 |
| 12 | * compiled using Eclipse TITAN |
| 13 | ** uses just a command-line compiler + Makefiles |
| 14 | ** no IDE needed at all, don't let _Eclipse_ fool you |
| 15 | * containerized in Docker |
| 16 | * executed by Jenkins CI |
| 17 | |
| 18 | == Terminology |
| 19 | |
| 20 | ATS:: Abstract Test Suite |
| 21 | MTC:: Main Test Component |
| 22 | PTC:: Parallel Test Component |
| 23 | IUT:: Implementation Under Test |
| 24 | |
| 25 | == Test Suite Philosophy |
| 26 | |
| 27 | * test one network element (our IUT) |
| 28 | * test external behavior (3GPP and non-3GPP) |
| 29 | * emulate entire environment from TTCN-3 |
| 30 | * don't reuse Osmocom C-code protocol implementations in the tests |
| 31 | * test against independent TTCN-3 implementations! |
| 32 | |
| 33 | == What to test? |
| 34 | |
| 35 | * successful cases |
| 36 | * erroneous cases (no answer, NACK, ...) |
| 37 | ** many difficult to reproduce with real phones/devices |
| 38 | * load / resource exhaustion |
| 39 | * spec compliance |
| 40 | * focus on functionality actually relevant to IUT |
| 41 | |
| 42 | == Why TTCN-3 + TITAN |
| 43 | |
| 44 | * TTCN-3 specifically designed for telecom protocol testing |
| 45 | * TITAN team released many telecom protocols in TTCN-3, such as |
| 46 | ** BSSAP, L3 (RR/MM/CC), SMS (CP/RP/TP), SS, M3UA, SCCP, GTP, NS, BSSGP, ... |
| 47 | ** shortens our test development cycle |
| 48 | ** permits us to test against known working industry implementations |
| 49 | |
| 50 | == Test suites for Osmocom CNI components |
| 51 | |
| 52 | * `osmo-bts` |
| 53 | * `osmo-bsc` |
| 54 | * `osmo-msc` |
| 55 | * `osmo-mgw` |
| 56 | * `osmo-hlr` |
| 57 | * `osmo-sip-connector` |
| 58 | * `osmo-sgsn` |
| 59 | * `osmo-ggsn` |
| 60 | |
| 61 | == Test suites in progress |
| 62 | |
| 63 | * `osmo-pcu` |
| 64 | * `osmo-bsc_nat` |
| 65 | |
| 66 | |
| 67 | |
| 68 | |
| 69 | == BTS_Tests.ttcn |
| 70 | |
| 71 | * external interfaces |
| 72 | ** A-bis side: RSL (emulates BSC-side server) |
| 73 | ** Um side: L1CTL to control MS |
| 74 | ** PCU side: pcu_socket |
| 75 | |
| 76 | [graphviz] |
| 77 | ---- |
| 78 | digraph G { |
| 79 | rankdir=LR; |
| 80 | { rank=same; BTS, BSC}; |
| 81 | BTS [label="IUT\nosmo-bts-trx",shape="box"]; |
| 82 | ATS [label="ATS\nBTS_Tests.ttcn"]; |
| 83 | BSC [label="osmo-bsc\nOML only"]; |
| 84 | BTS -> fake_trx [label="bursts"]; |
| 85 | fake_trx -> trxcon [label="bursts"]; |
| 86 | trxcon -> ATS [label="bursts"]; |
| 87 | |
| 88 | BTS -> BSC [label="A-bis OML"]; |
| 89 | BTS -> ATS [label="A-bis RSL"]; |
| 90 | |
| 91 | ATS -> BTS [label="pcu_sock"]; |
| 92 | ATS -> BSC [label="VTY"]; |
| 93 | ATS -> BTS [label="CTRL"]; |
| 94 | } |
| 95 | ---- |
| 96 | |
| 97 | |
| 98 | |
| 99 | == BSC_Tests.ttcn |
| 100 | |
| 101 | * external interfaces |
| 102 | ** A-bis side: RSL (emulates BTS-side client) |
| 103 | ** A-side: BSSAP/SCCP/M3UA (emulates MSC-side) |
| 104 | ** MGW side: MGCP (emulates MGW side) |
| 105 | |
| 106 | [graphviz] |
| 107 | ---- |
| 108 | digraph G { |
| 109 | rankdir=LR; |
| 110 | { rank=same; BTS; STP; }; |
| 111 | BSC [label="IUT\nosmo-bsc",shape="box"]; |
| 112 | ATS [label="ATS\nBSC_Tests.ttcn"]; |
| 113 | BTS [label="osmo-bts-omldummy\nOML only"]; |
| 114 | |
| 115 | BTS -> BSC [label="A-bis OML"]; |
| 116 | ATS -> BSC [label="A-bis RSL"]; |
| 117 | ATS -> BSC [label="CTRL"]; |
| 118 | ATS -> BSC [label="VTY"]; |
| 119 | ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| 120 | BSC -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| 121 | } |
| 122 | ---- |
| 123 | |
| 124 | == MSC_Tests.ttcn |
| 125 | |
| 126 | * external interfaces |
| 127 | ** A: BSSAP/SCCP/M3UA (emulates BSC-side) |
| 128 | ** MNCC: MNCC/unix-domain (emulates ext. MNCC side) |
| 129 | ** MGW: MGCP (emulates MGW side) |
| 130 | ** GSUP (emulates HLR side) |
| 131 | |
| 132 | [graphviz] |
| 133 | ---- |
| 134 | digraph G { |
| 135 | rankdir=LR; |
| 136 | MSC [label="IUT\nosmo-msc",shape="box"]; |
| 137 | ATS [label="ATS\nMSC_Tests.ttcn"]; |
| 138 | |
| 139 | ATS -> MSC [label="MNCC"]; |
| 140 | ATS -> MSC [label="SMPP",style="dashed"]; |
| 141 | ATS -> MSC [label="CTRL"]; |
| 142 | ATS -> MSC [label="VTY"]; |
| 143 | MSC -> ATS [label="GSUP"]; |
| 144 | ATS -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| 145 | MSC -> STP [label="A BSSAP\nSCCP/M3UA"]; |
| 146 | } |
| 147 | ---- |
| 148 | |
| 149 | |
| 150 | == MGCP_Test.ttcn |
| 151 | |
| 152 | * external interfaces |
| 153 | ** MGCP (emulates call agent) |
| 154 | ** RTP (stream source/sink) |
| 155 | |
| 156 | [graphviz] |
| 157 | ---- |
| 158 | digraph G { |
| 159 | rankdir=LR; |
| 160 | MGW [label="IUT\nosmo-mgw",shape="box"]; |
| 161 | ATS [label="ATS\nMGCP_Test.ttcn"]; |
| 162 | |
| 163 | ATS -> MGW [label="RTP"]; |
| 164 | ATS -> MGW [label="MGCP"]; |
| 165 | MGW -> ATS [label="RTP"]; |
| 166 | } |
| 167 | ---- |
| 168 | |
| 169 | == HLR_Tests.ttcn |
| 170 | |
| 171 | * external interfaces |
| 172 | ** GSUP (emulates VLR/SGSN side) |
| 173 | ** VTY |
| 174 | |
| 175 | [graphviz] |
| 176 | ---- |
| 177 | digraph G { |
| 178 | rankdir=LR; |
| 179 | HLR [label="IUT\nosmo-hlr",shape="box"]; |
| 180 | ATS [label="ATS\nHLR_Tests.ttcn"]; |
| 181 | |
| 182 | ATS -> HLR [label="GSUP"]; |
| 183 | ATS -> HLR [label="VTY"]; |
| 184 | } |
| 185 | ---- |
| 186 | |
| 187 | |
| 188 | == SIP_Tests.ttcn |
| 189 | |
| 190 | * external interfaces |
| 191 | ** MNCC (emulates MSC side) |
| 192 | ** SIP (emulates SIP switch) |
| 193 | ** VTY |
| 194 | |
| 195 | [graphviz] |
| 196 | ---- |
| 197 | digraph G { |
| 198 | rankdir=LR; |
| 199 | SIP [label="SIP\nosmo-sip-connector",shape="box"]; |
| 200 | ATS [label="ATS\nSIP_Tests.ttcn"]; |
| 201 | |
| 202 | ATS -> SIP [label="MNCC"]; |
| 203 | ATS -> SIP [label="SIP"]; |
| 204 | ATS -> SIP [label="VTY"]; |
| 205 | } |
| 206 | ---- |
| 207 | |
| 208 | |
| 209 | == SGSN_Tests.ttcn |
| 210 | |
| 211 | * external interfaces |
| 212 | ** Gb (emulates PCU side NS/BSSGP + MS) |
| 213 | ** GSUP (emulates HLR) |
| 214 | ** VTY |
| 215 | |
| 216 | [graphviz] |
| 217 | ---- |
| 218 | digraph G { |
| 219 | rankdir=LR; |
| 220 | SGSN [label="SGSN\nosmo-sgsn",shape="box"]; |
| 221 | ATS [label="ATS\nSGSN_Tests.ttcn"]; |
| 222 | |
| 223 | ATS -> SGSN [label="Gb"]; |
| 224 | SGSN-> ATS [label="Gp (GTP)"]; |
| 225 | ATS -> SGSN [label="VTY"]; |
| 226 | } |
| 227 | ---- |
| 228 | |
| 229 | |
| 230 | == GGSN_Tests.ttcn |
| 231 | |
| 232 | * external interfaces |
| 233 | ** Gp: GTP (emulates SGSN) |
| 234 | ** Gi: IP (emulates Internet) |
| 235 | |
| 236 | [graphviz] |
| 237 | ---- |
| 238 | digraph G { |
| 239 | rankdir=LR; |
| 240 | GGSN [label="GGSN\nosmo-ggsn",shape="box"]; |
| 241 | ATS [label="ATS\nGGSN_Tests.ttcn"]; |
| 242 | |
| 243 | ATS -> GGSN [label="Gp (GTP)"]; |
| 244 | GGSN -> ATS [label="Gi (IP)"]; |
| 245 | ATS -> GGSN [label="VTY"]; |
| 246 | } |
| 247 | ---- |
| 248 | |
| 249 | |
| 250 | |
| 251 | == Dockerized Setup |
| 252 | |
| 253 | * one process per container |
| 254 | * packages either |
| 255 | ** IUT (e.g. `osmo-bsc`) |
| 256 | ** ATS (compiled docker test suite) |
| 257 | ** other utility (e.g. `trxcon` or `osmo-bts-omldummy`) |
| 258 | * why? |
| 259 | ** no need for local ip/network configuration |
| 260 | ** standardized / packaged setup on every machine |
| 261 | ** run older/newer versions of ATS against older/newer IUT |
| 262 | |
| 263 | == Jenkins CI Execution |
| 264 | |
| 265 | . update `docker-playground.git` |
| 266 | .. contains `Dockerfile` for ATS + IUT |
| 267 | . rebuild IUT container[s] (e.g. `osmo-bts-master`) |
| 268 | .. git magic ensures re-build only if `osmo-bts.git` master changed |
| 269 | . rebuild ATS container (e.g. `ttcn3-bts-test`) |
| 270 | .. git magic ensures re-build only if `osmo-ttcn3-hacks.git` master changed |
| 271 | . run `docker-playground/ttcn3-bts-test/jenkins.sh` |
| 272 | .. creates docker network |
| 273 | .. starts IUT + ATS docker containers |
| 274 | .. collects test results |
| 275 | |
| 276 | |
| 277 | == Jenkins CI Reporting |
| 278 | |
| 279 | * junit-xml generation |
| 280 | * store artefacts |
| 281 | ** pcap file of every test case |
| 282 | ** ATS log file (TTCN-3 testsuite) |
| 283 | ** IUT log file[s] (`osmo-*.log`) |
| 284 | ** IUT config file[s] (`osmo-*.cfg`) |
| 285 | * see https://jenkins.osmocom.org/jenkins/view/TTCN3/ |
| 286 | |
| 287 | |
| 288 | == Further Reading |
| 289 | |
| 290 | * http://git.osmocom.org/osmo-ttcn3-hacks/ |
| 291 | * http://git.osmocom.org/docker-playground/ |
| 292 | * http://osmocom.org/projects/cellular-infrastructure/wiki/Titan_TTCN3_Notes |
| 293 | |
| 294 | == EOF |
| 295 | |
| 296 | End of File |