Pau Espin Pedrol | c5f623f | 2024-03-26 16:17:59 +0100 | [diff] [blame] | 1 | [[ipc_if]] |
| 2 | == osmo-trx-ipc IPC Interface |
| 3 | |
| 4 | This interface is the one used by _osmo_trx_ipc_ backend to communicate to a |
| 5 | third party process in charge of driving the lowest layer device-specific bits |
| 6 | (from now on the Driver). |
| 7 | |
| 8 | It consists of a set of Unix Domain (UD) sockets for the control plane, plus a |
| 9 | shared memory region for the data plane. |
| 10 | |
| 11 | Related code can be found in the |
| 12 | https://gitea.osmocom.org/cellular-infrastructure/osmo-trx/src/branch/master/Transceiver52M/device/ipc[Transceiver52M/device/ipc/] |
| 13 | directory in _osmo-trx.git_. |
| 14 | |
| 15 | If you are a potential driver implementator, the |
| 16 | various primitives and data structures are publicly available in header file |
| 17 | https://gitea.osmocom.org/cellular-infrastructure/osmo-trx/src/branch/master/Transceiver52M/device/ipc/shm.h[Transceiver52M/device/ipc/shm.h]. |
| 18 | |
| 19 | === Control plane |
| 20 | |
| 21 | Control plane protocol is transmitted over Unix Domain (UD) sockets using |
| 22 | message based primitives. Each primitive has a type identified by an integer, |
| 23 | and each type of primitive has a number of extra attributes attached to it. The |
| 24 | IPC interface consists of 2 types of UD sockets: |
| 25 | |
| 26 | * _Master_ UD socket: One per osmo-trx-ipc process. |
| 27 | |
| 28 | * _Channel_ UD socket: One for each channel managed by osmo-trx-ipc process. |
| 29 | |
| 30 | The _Driver_ is in all cases expected to take the server role when creating UD |
| 31 | sockets, while _osmo-trx-ipc_ takes the client role and connects to sockets |
| 32 | provided by the driver. |
| 33 | |
| 34 | === Master UD socket |
| 35 | |
| 36 | During startup, _osmo-trx-ipc_ will try connecting to the _Driver_ Master UD |
| 37 | socket located in the path provided by its own (VTY) configuration. As a result, |
| 38 | it means the _Driver_ process must be running and listening on the Master UD |
| 39 | socket before _osmo-trx-ipc_ is started, otherwise _osmo-trx-ipc_ will fail and |
| 40 | exit. |
| 41 | |
| 42 | Once connected, _osmo-trx-ipc_ will submit a `GREETING_REQ` message primitive |
| 43 | announcing the maximum supported protocol version (first version ever is `1`, |
| 44 | increasing over time). |
| 45 | |
| 46 | The _Driver_ shall then answer in `GREETING_CNF` message primitive with its own |
| 47 | maximum supported version (`<=` version received), providing 0 if none is |
| 48 | supported. |
| 49 | |
| 50 | If _osmo-trx-ipc_ receives back the requested version, then both sides agreed |
| 51 | on the protocol version to use. |
| 52 | If _osmo-trx-ipc_ receives back a lower version, it shall decide to continue |
| 53 | with version negotiation using a lower version, until a supported version or 0 |
| 54 | is received. If finally 0 is received, _osmo-trx-ipc_ will disconnect and exit |
| 55 | with failure. |
| 56 | |
| 57 | Once the version is negotiated (`v1` as of current date), _osmo-trx-ipc_ will |
| 58 | ask for device information and available characeristics to the _Driver_ using |
| 59 | the `INFO_REQ` message primitive. |
| 60 | |
| 61 | The _Driver_ shall then answer with a `INFO_CNF` message |
| 62 | containing information, such as: |
| 63 | |
| 64 | * String containing device description |
| 65 | |
| 66 | * Available reference clocks, |
| 67 | |
| 68 | * {rx,tx} I/Q scaling factors |
| 69 | |
| 70 | * Maximum number of channels supported |
| 71 | |
| 72 | * for each channel: |
| 73 | |
| 74 | ** List of available {rx,tx} paths/antennas. |
| 75 | |
| 76 | ** {min,max}{rx,tx} gains |
| 77 | |
| 78 | ** Nominal transmit power |
| 79 | |
| 80 | All the information received from the _Driver_ during `INFO_CNF` will be used by |
| 81 | _osmo-trx-ipc_ to decide whether it can fullfil the requested configuration from |
| 82 | the user, and proceed to open the device, or exit with a failure (for instance |
| 83 | number of channels, referece clock or tx/rx antenna selected by the user cannot |
| 84 | be fullfilled). |
| 85 | |
| 86 | _osmo-trx-ipc_ will then proceed to open the device and do an initial |
| 87 | configuration using an `OPEN_REQ` message, where it will provide the _Driver_ |
| 88 | with the desired selected configuration (such as number of channels, rx/tx |
| 89 | paths, clock reference, bandwidth filters, etc.). |
| 90 | |
| 91 | The _Driver_ shall then configure the device and send back a `OPEN_CNF` with: |
| 92 | |
| 93 | * `return_code` integer attribute set to `0` on success or `!0` on error. |
| 94 | |
| 95 | * Name of the Posix Shared Memory region where data plane is going to be |
| 96 | transmitted. |
| 97 | |
| 98 | * One path for each channel, containing the just-created UD socket to manage |
| 99 | that channel (for instance by taking Master UD socket path and appending |
| 100 | `_$chan_idx`). |
| 101 | |
| 102 | * Path Delay: this is the loopback path delay in samples (= used as a timestamp |
| 103 | offset internally by _osmo-trx-ipc_), this value contains the analog delay as |
| 104 | well as the delay introduced by the digital filters in the fpga in the sdr |
| 105 | devices, and is therefore device type and bandwidth/sample rate dependant. This |
| 106 | can not be omitted, wrong values will lead to a _osmo-trx-ipc_ that just doesn't |
| 107 | detect any bursts. |
| 108 | |
| 109 | Finally, _osmo-trx-ipc_ will connect to each channel's UD socket (see next |
| 110 | section). |
| 111 | |
| 112 | Upon _osmo-trx-ipc_ closing the UD master socket connection, the _Driver_ shall |
| 113 | go into _closed_ state: stop all processing and instruct the device to power |
| 114 | off. |
| 115 | |
| 116 | TIP: See |
| 117 | https://gitea.osmocom.org/cellular-infrastructure/osmo-trx/src/branch/master/Transceiver52M/device/ipc/shm.h[Transceiver52M/device/ipc/shm.h] |
| 118 | for the detailed definition of all the related message primitives and data |
| 119 | types for this socket. |
| 120 | |
| 121 | === Channel UD Socket |
| 122 | |
| 123 | This socket can be used by _osmo-trx-ipc_ to start/stop data plane processing or |
| 124 | change channel's parameters such as Rx/Tx Frequency, Rx/Tx gains, etc. |
| 125 | |
| 126 | A channel can be either in _started_ or _stopped_ state. When a channel is |
| 127 | created (during `OPEN_REQ` in the Master UD Socket), it's by default in |
| 128 | _stopped_ state. `START_REQ` and `STOP_REQ` messages control this state, and |
| 129 | eventual failures can be reported through `START_CNF` and `STOP_CNF` by the |
| 130 | _Driver_. |
| 131 | |
| 132 | The message `START_REQ` instructs the _Driver_ to start processing data in the |
| 133 | data plane. Similary, `STOP_REQ` instructs the _Driver_ to stop processing data |
| 134 | in the data plane. |
| 135 | |
| 136 | Some parameters are usually changed only when the channel is in stopped mode, |
| 137 | for instance Rx/Tx Frequency. |
| 138 | |
| 139 | TIP: See |
| 140 | https://gitea.osmocom.org/cellular-infrastructure/osmo-trx/src/branch/master/Transceiver52M/device/ipc/shm.h[Transceiver52M/device/ipc/shm.h] |
| 141 | for the detailed definition of all the related message primitives and data |
| 142 | types for this socket. |
| 143 | |
| 144 | === Data Plane |
| 145 | |
| 146 | Data plane protocol is implemented by means of a ring buffer structure on top of |
| 147 | Posix Shared Memory (see `man 7 shm_overview`) between _osmo-trx-ipc_ process |
| 148 | and the _Driver_. |
| 149 | |
| 150 | The Posix Shared Memory region is created and its memory structure prepared by |
| 151 | the _Driver_ and its name shared with _osmo-trx-ipc_ during _OPEN_CNF_ message |
| 152 | in the Master UD Socket from the Control Plane. Resource allocation for the |
| 153 | shared memory area and cleanup is up to the ipc server, as is mutex |
| 154 | initialization for the buffers. |
| 155 | |
| 156 | ==== Posix Shared Memory structure |
| 157 | |
| 158 | [[fig-shm-structure]] |
| 159 | .General overview of Posix Shared Memory structure |
| 160 | [graphviz] |
| 161 | ---- |
| 162 | digraph hierarchy { |
| 163 | node[shape=record,style=filled,fillcolor=gray95] |
| 164 | edge[dir=back, arrowtail=empty] |
| 165 | |
| 166 | SHM[label = "{Posix Shared Memory region|+ num_chans\l+ Channels[]\l}"] |
| 167 | CHAN0[label = "{Channel 0|...}"] |
| 168 | CHAN1[label = "{Channel 1|...}"] |
| 169 | CHANN[label = "{Channel ...|}"] |
| 170 | STREAM0_UL[label = "{UL Stream|+ semaphore\l+ read_next\l+ write_next\l+ buffer_size /* In samples */\l+ num_buffers\l+ sample_buffers[]\l}"] |
| 171 | STREAM0_DL[label = "{DL Stream|+ semaphore\l+ read_next\l+ write_next\l+ buffer_size /* In samples */\l+ num_buffers\l+ sample_buffers[]\l}"] |
| 172 | STREAM1_UL[label = "{UL Stream|...}"] |
| 173 | STREAM1_DL[label = "{DL Stream|...}"] |
| 174 | STREAMN_UL[label = "{UL Stream|...}"] |
| 175 | STREAMN_DL[label = "{DL Stream|...}"] |
| 176 | BUF_0DL0[label = "{DL Sample Buffer 0|+ timestamp\l+ buffer_size /* In samples */\l+ samples[] = [16bit I + 16bit Q,...]\l}"] |
| 177 | BUF_0DLN[label = "{DL Sample Buffer ....|...}"] |
| 178 | BUF_0UL0[label = "{UL Sample Buffer 0|+ timestamp\l+ buffer_size /* In samples */\l+ samples[] = [16bit I + 16bit Q,...]\l}"] |
| 179 | BUF_0ULN[label = "{UL Sample Buffer ...|...}"] |
| 180 | |
| 181 | SHM->CHAN0 |
| 182 | SHM->CHAN1 |
| 183 | SHM->CHANN |
| 184 | |
| 185 | CHAN0->STREAM0_DL |
| 186 | CHAN0->STREAM0_UL |
| 187 | STREAM0_DL->BUF_0DL0 |
| 188 | STREAM0_DL->BUF_0DLN |
| 189 | STREAM0_UL->BUF_0UL0 |
| 190 | STREAM0_UL->BUF_0ULN |
| 191 | |
| 192 | CHAN1->STREAM1_UL |
| 193 | CHAN1->STREAM1_DL |
| 194 | |
| 195 | CHANN->STREAMN_UL |
| 196 | CHANN->STREAMN_DL |
| 197 | } |
| 198 | ---- |
| 199 | |
| 200 | The Posix Shared Memory region contains an array of _Channels_. |
| 201 | |
| 202 | Each _Channel_ contains 2 Streams: |
| 203 | |
| 204 | * Downlink _Stream_ |
| 205 | |
| 206 | * Uplink _Stream_ |
| 207 | |
| 208 | Each _Stream_ handles a ring buffer, which is implemented as: |
| 209 | |
| 210 | * An array of pointers to _Sample Buffer_ structures. |
| 211 | |
| 212 | * Variables containing the number of buffers in the array, as well as the |
| 213 | maximum size in samples for each Sample Buffer. |
| 214 | |
| 215 | * Variables containing `next_read` and `next_write` _Sample Buffer_ (its index |
| 216 | in the array of pointers). |
| 217 | |
| 218 | * Unnamed Posix semaphores to do the required locking while using the ring |
| 219 | buffer. |
| 220 | |
| 221 | Each _Sample Buffer_ contains: |
| 222 | |
| 223 | * A `timestamp` variable, containing the position in the stream of the first |
| 224 | sample in the buffer |
| 225 | |
| 226 | * A `data_len` variable, containing the amount of samples available to process |
| 227 | in the buffer |
| 228 | |
| 229 | * An array of samples of size specified by the stream struct it is part of. |
| 230 | |
| 231 | ==== Posix Shared Memory format |
| 232 | |
| 233 | The Posix Shared memory region shall be formatted applying the following |
| 234 | considerations: |
| 235 | |
| 236 | * All pointers in the memory region are encoded as offsets from the start |
| 237 | address of the region itself, to allow different processes with different |
| 238 | address spaces to decode them. |
| 239 | |
| 240 | * All structs must be force-aligned to 8 bytes |
| 241 | |
| 242 | * Number of buffers must be power of 2 (2,4,8,16,...) - 4 appears to be plenty |
| 243 | |
| 244 | * IQ samples format: One (complex) sample consists of 16bit i + 16bit q, so the |
| 245 | buffer size is number of IQ pairs. |
| 246 | |
| 247 | * A reasonable per-buffer size (in samples) is 2500, since this happens to be |
| 248 | the ususal TX (downlink) buffer size used by _osmo-trx-ipc_ with the b210 (rx |
| 249 | over-the-wire packet size for the b210 is 2040 samples, so the larger value of |
| 250 | both is convenient). |
| 251 | |
| 252 | TIP: See |
| 253 | https://gitea.osmocom.org/cellular-infrastructure/osmo-trx/src/branch/master/Transceiver52M/device/ipc/shm.h[Transceiver52M/device/ipc/shm.h] |
| 254 | for the detailed definition of all the objects being part of the Posix Shared |
| 255 | memory region structure |
| 256 | |
| 257 | ==== Posix Shared Memory procedures |
| 258 | |
| 259 | The queue in the shared memory area is not supposed to be used for actual |
| 260 | buffering of data, only for exchange, so the general expectation is that it is |
| 261 | mostly empty. The only exception to that might be minor processing delays, and |
| 262 | during startup. |
| 263 | |
| 264 | Care must be taken to ensure that only timed waits for the mutex protecting it |
| 265 | and the condition variables are used, in order to ensure that no deadlock occurs |
| 266 | should the other side die/quit. |
| 267 | |
| 268 | Thread cancellation should be disabled during reads/writes from/to the queue. In |
| 269 | general a timeout can be considered a non recoverable error during regular |
| 270 | processing after startup, at least with the current timeout value of one second. |
| 271 | |
| 272 | Should over- or underflows occur a corresponding message should be sent towards |
| 273 | _osmo-trx-ipc_. |
| 274 | |
| 275 | Upon **read** of `N` samples, the reader does something like: |
| 276 | |
| 277 | . Acquire the semaphore in the channel's stream object. |
| 278 | |
| 279 | . Read `stream->next_read`, if `next_read==next_write`, become blocked in |
| 280 | another sempahore (unlocking the previous one) until writer signals us, then |
| 281 | `buff = stream->buffers[next_read]` |
| 282 | |
| 283 | . Read `buff->data_len` samples, reset the buffer data (`data_len=0`), |
| 284 | increment `next_read` and if read samples is `<N`, continue with next buffer |
| 285 | until `next_read==next_write`, then block again or if timeout elapsed, then we |
| 286 | reach conditon buffer underflow and `return len < N`. |
| 287 | |
| 288 | . Release the semaphore |
| 289 | |
| 290 | Upon **write** of `N` samples, the writer does something like: |
| 291 | |
| 292 | . Acquire the semapore in the channel's stream object. |
| 293 | |
| 294 | . Write samples to `buff = stream->buffers[next_write]`. If `data_len!=0`, |
| 295 | signal `buffer_overflow` (increase field in stream object) and probably |
| 296 | increase next_read`. |
| 297 | |
| 298 | . Increase `next_write`. |
| 299 | |
| 300 | . If `next_write` was `== next_read`, signal the reader through the other |
| 301 | semaphore that it can continue reading. |