[PATCH] fix DTLS_OVERHEAD and GlobalProtect ESP overhead calculation

[David Woodhouse]

On Tue, 2017-08-15 at 11:42 -0700, Daniel Lenski wrote:
> 
>  #define ESP_OVERHEAD (4 /* SPI */ + 4 /* sequence number */ + \
> -         20 /* biggest supported MAC (SHA1) */ + 16 /* biggest supported IV (AES-128) */ + \
> -        1 /* pad length */ + 1 /* next header */ + \
> -         16 /* max padding */ )
> +                      1 /* pad length */ + 1 /* next header */ + \
> +                      16 /* max padding */ )
>  #define UDP_HEADER_SIZE 8
>  #define IPV4_HEADER_SIZE 20
>  #define IPV6_HEADER_SIZE 40
> @@ -323,7 +322,9 @@ static int calculate_mtu(struct openconnect_info *vpninfo)
>  
>         if (!mtu) {
>                 /* remove IP/UDP and ESP overhead from base MTU to calculate tunnel MTU */
> -               mtu = base_mtu - ESP_OVERHEAD - UDP_HEADER_SIZE;
> +               mtu = ( base_mtu - UDP_HEADER_SIZE - ESP_OVERHEAD
> +                       - (vpninfo->hmac_key_len ? : 20) /* biggest supported MAC (SHA1) */
> +                       - (vpninfo->enc_key_len ? : 32) /* biggest supported IV (AES-256) */ );
>                 if (vpninfo->peer_addr->sa_family == AF_INET6)
>                         mtu -= IPV6_HEADER_SIZE;

Better... except that you left the padding hard-coded to 16. That would
be the same as the cipher blocksize (== IV size).... except you don't
actually *need* to be pessimistic. It can be calculated exactly, just
as dtls_get_data_mtu() does in openssl-dtls.c

From payload MTU, add at least 1 byte of padding, round up to the next
multiple of the blocksize. Add the MAC size, and other headers. That's
the packet on the wire.

So from wire packet MTU, subtract headers and MAC and IV, round *down*
to a multiple of blocksize, subtract one byte for the *minimal*
padding, and that's the largest payload you can carry.

This is all assuming you only do CBC and don't support any AEAD
ciphersuites. Which is true for now.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 4938 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/openconnect-devel/attachments/20170815/efeef50a/attachment.bin>