[PATCH] fix DTLS_OVERHEAD and GlobalProtect ESP overhead calculation

[Daniel Lenski]

GlobalProtect doesn't try to calculate MTU until after it has information on
the ESP ciphersuite, so it can use the real HMAC/encryption key lengths when
calculating ESP overhead.  In practice, I have never seen or heard of a GP
VPN that uses anything other than AES128+SHA1, but both the clients and
servers appear to include support for AES256.

DTLS_OVERHEAD was not correctly accounting for possibility of AES256
(32-byte IV).

Signed-off-by: Daniel Lenski <dlenski at gmail.com>
---
 gpst.c                 | 9 +++++----
 openconnect-internal.h | 2 +-
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/gpst.c b/gpst.c
index 71bd968..8eb925a 100644
--- a/gpst.c
+++ b/gpst.c
@@ -267,9 +267,8 @@ out:
 }
 
 #define ESP_OVERHEAD (4 /* SPI */ + 4 /* sequence number */ + \
-         20 /* biggest supported MAC (SHA1) */ + 16 /* biggest supported IV (AES-128) */ + \
-	 1 /* pad length */ + 1 /* next header */ + \
-         16 /* max padding */ )
+                      1 /* pad length */ + 1 /* next header */ + \
+                      16 /* max padding */ )
 #define UDP_HEADER_SIZE 8
 #define IPV4_HEADER_SIZE 20
 #define IPV6_HEADER_SIZE 40
@@ -323,7 +322,9 @@ static int calculate_mtu(struct openconnect_info *vpninfo)
 
 	if (!mtu) {
 		/* remove IP/UDP and ESP overhead from base MTU to calculate tunnel MTU */
-		mtu = base_mtu - ESP_OVERHEAD - UDP_HEADER_SIZE;
+		mtu = ( base_mtu - UDP_HEADER_SIZE - ESP_OVERHEAD
+		        - (vpninfo->hmac_key_len ? : 20) /* biggest supported MAC (SHA1) */
+		        - (vpninfo->enc_key_len ? : 32) /* biggest supported IV (AES-256) */ );
 		if (vpninfo->peer_addr->sa_family == AF_INET6)
 			mtu -= IPV6_HEADER_SIZE;
 		else
diff --git a/openconnect-internal.h b/openconnect-internal.h
index 3526ce6..af35791 100644
--- a/openconnect-internal.h
+++ b/openconnect-internal.h
@@ -334,7 +334,7 @@ static inline void init_pkt_queue(struct pkt_q *q)
 }
 
 #define DTLS_OVERHEAD (1 /* packet + header */ + 13 /* DTLS header */ + \
-	 20 /* biggest supported MAC (SHA1) */ +  16 /* biggest supported IV (AES-128) */ + \
+	 20 /* biggest supported MAC (SHA1) */ +  32 /* biggest supported IV (AES-256) */ + \
 	 16 /* max padding */)
 
 struct esp {
-- 
2.7.4