libosmogsm: Allow auth API caller to specify RES length

There are 3G algorithms which support different lengths of RES values
(4, 8, 16 byte).  For MILENAGE, we never really had to bother, as
the 4-byte RES is simply the first 4 bytes of the 8-byte RES.

However, for TUAK, the expected RES length is an input parameter to
the Keccak crypto functions, so the result of all parameters (including
CK, IK, ...) will be completely different for RES length 4 than RES
length 8.

So let's permit the caller of the osmocom auth API to specify the
requested RES length via the osmo_auth_vector.res_len parameter.

For backwards compatibility of callers of the old osmo_auth_gen_vec/
osmo_auth_gen_vec_auts API: Always force the res_len to 8 in this case,
which was the hard-coded length before this patch.

Change-Id: Ic662843fbe8b5c58e4af39ea630ad5ac13fd6bef
diff --git a/src/gsm/auth_milenage.c b/src/gsm/auth_milenage.c
index 9e94293..a439b15 100644
--- a/src/gsm/auth_milenage.c
+++ b/src/gsm/auth_milenage.c
@@ -62,6 +62,8 @@
 		return -EINVAL;
 	if (aud->u.umts.opc_len != 16)
 		return -EINVAL;
+	if (vec->res_len != 4 && vec->res_len != 8)
+		return -EINVAL;
 
 	opc = gen_opc_if_needed(aud, gen_opc);
 	if (!opc)
@@ -135,7 +137,7 @@
 	milenage_generate(opc, aud->u.umts.amf, aud->u.umts.k,
 			  sqn, _rand,
 			  vec->autn, vec->ik, vec->ck, vec->res, &res_len);
-	vec->res_len = res_len;
+
 	rc = gsm_milenage(opc, aud->u.umts.k, _rand, vec->sres, vec->kc);
 	if (rc < 0)
 		return rc;