libosmogsm: Allow auth API caller to specify RES length

There are 3G algorithms which support different lengths of RES values
(4, 8, 16 byte).  For MILENAGE, we never really had to bother, as
the 4-byte RES is simply the first 4 bytes of the 8-byte RES.

However, for TUAK, the expected RES length is an input parameter to
the Keccak crypto functions, so the result of all parameters (including
CK, IK, ...) will be completely different for RES length 4 than RES
length 8.

So let's permit the caller of the osmocom auth API to specify the
requested RES length via the osmo_auth_vector.res_len parameter.

For backwards compatibility of callers of the old osmo_auth_gen_vec/
osmo_auth_gen_vec_auts API: Always force the res_len to 8 in this case,
which was the hard-coded length before this patch.

Change-Id: Ic662843fbe8b5c58e4af39ea630ad5ac13fd6bef
diff --git a/src/gsm/auth_core.c b/src/gsm/auth_core.c
index 421ecee..2b6f35a 100644
--- a/src/gsm/auth_core.c
+++ b/src/gsm/auth_core.c
@@ -167,7 +167,7 @@
 }
 
 /*! Generate authentication vector
- *  \param[out] vec Generated authentication vector
+ *  \param[out] vec Generated authentication vector. See below!
  *  \param[in] aud Subscriber-specific key material
  *  \param[in] _rand Random challenge to be used
  *  \returns 0 on success, negative error on failure
@@ -176,7 +176,12 @@
  * computing authentication triples/quintuples based on the permanent
  * subscriber data and a random value.  The result is what is forwarded
  * by the AUC via HLR and VLR to the MSC which will then be able to
- * invoke authentication with the MS
+ * invoke authentication with the MS.
+ *
+ * Contrary to the older osmo_auth_gen_vec(), the caller must specify
+ * the desired RES length in the vec->res_len field prior to calling
+ * this function.  The requested length must match the capabilities of
+ * the chosen algorithm (e.g. 4/8 for MILENAGE).
  */
 int osmo_auth_gen_vec2(struct osmo_auth_vector *vec,
 		       struct osmo_sub_auth_data2 *aud,
@@ -216,6 +221,12 @@
 	struct osmo_sub_auth_data2 aud2;
 	int rc;
 
+	if (aud->type == OSMO_AUTH_TYPE_UMTS) {
+		/* old API callers are not expected to initialize this struct field,
+		 * and always expect an 8-byte RES value */
+		vec->res_len = 8;
+	}
+
 	rc = auth_data2auth_data2(&aud2, aud);
 	if (rc < 0)
 		return rc;
@@ -228,7 +239,7 @@
 }
 
 /*! Generate authentication vector and re-sync sequence
- *  \param[out] vec Generated authentication vector
+ *  \param[out] vec Generated authentication vector. See below!
  *  \param[in] aud Subscriber-specific key material
  *  \param[in] auts AUTS value sent by the SIM/MS
  *  \param[in] rand_auts RAND value sent by the SIM/MS
@@ -241,6 +252,11 @@
  * AUTS and RAND values returned by the SIM/MS.  This special variant is
  * needed if the sequence numbers between MS and AUC have for some
  * reason become different.
+ *
+ * Contrary to the older osmo_auth_gen_vec_auts(), the caller must specify
+ * the desired RES length in the vec->res_len field prior to calling
+ * this function.  The requested length must match the capabilities of
+ * the chosen algorithm (e.g. 4/8 for MILENAGE).
  */
 int osmo_auth_gen_vec_auts2(struct osmo_auth_vector *vec,
 			    struct osmo_sub_auth_data2 *aud,
@@ -285,6 +301,12 @@
 	struct osmo_sub_auth_data2 aud2;
 	int rc;
 
+	if (aud->type == OSMO_AUTH_TYPE_UMTS) {
+		/* old API callers are not expected to initialize this struct field,
+		 * and always expect an 8-byte RES value */
+		vec->res_len = 8;
+	}
+
 	rc = auth_data2auth_data2(&aud2, aud);
 	if (rc < 0)
 		return rc;