Unable to establish an ssh connection through an openconnect tunnel

[Jean-Gabriel Gill-Couture]

Hi,

I have this new issue since last Friday where I am unable to establish
an ssh connection through on of my openconnect tunnels. It is still
working on the other openconnect tunnel I use for that job. Other
protocols are also working fine such as ping, http, https.

This issue seems to be affecting all new openconnect connections
established to that vpn server, as one of my co-workers is having the
same issue but another, that is using a connection established before
it stopped working, is still able to ssh through that tunnel. It is
also working fine for Cisco Anyconnect users on OSX and Windows. I did
not test with openconnect on other platforms than Linux (Ubuntu 14.04
and Archlinux).

So it is very likely related to a configuration change on the vpn
side, I just asked the administrators about it and will follow up here
with any information that they provide me.

Actually the ssh handshake is started but stops at the key exchange,
at which point the server does not respond and timeouts after a while.

Here is the debug output of openconnect and ssh :

********************* Openconnect output ***************************

sudo openconnect -u myuser --authgroup=somegroup vpn1.server.domain
--no-cert-check -vvvvvv
POST https://vpn1.server.domain/
Attempting to connect to server 111.111.11.60:443
SSL negotiation with vpn1.server.domain
Server certificate verify failed: signer not found
Connected to HTTPS on vpn1.server.domain
Got HTTP response: HTTP/1.0 302 Object Moved
Content-Type: text/html; charset=utf-8
Content-Length: 0
Cache-Control: no-cache
Pragma: no-cache
Connection: Close
Date: Mon, 04 Apr 2016 14:18:21 GMT
Location: /+webvpn+/index.html
Set-Cookie: tg=; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; secure
HTTP body length:  (0)
GET https://vpn1.server.domain/
Attempting to connect to server 111.111.11.60:443
SSL negotiation with vpn1.server.domain
Server certificate verify failed: signer not found
Connected to HTTPS on vpn1.server.domain
Got HTTP response: HTTP/1.0 302 Object Moved
Content-Type: text/html; charset=utf-8
Content-Length: 0
Cache-Control: no-cache
Pragma: no-cache
Connection: Close
Date: Mon, 04 Apr 2016 14:18:22 GMT
Location: /+webvpn+/index.html
Set-Cookie: tg=; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; secure
HTTP body length:  (0)
GET https://vpn1.server.domain/+webvpn+/index.html
SSL negotiation with vpn1.server.domain
Server certificate verify failed: signer not found
Connected to HTTPS on vpn1.server.domain
Got HTTP response: HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Cache-Control: max-age=0
Set-Cookie: webvpn=; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; secure
Set-Cookie: webvpnc=; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; secure
Set-Cookie: webvpnlogin=1; secure
X-Transcend-Version: 1
HTTP body chunked (-2)
Please enter your username and password.
Please enter your username and password.
Password:
POST https://vpn1.server.domain/+webvpn+/index.html
Got HTTP response: HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: text/xml
Cache-Control: max-age=0
Set-Cookie: webvpnlogin=; expires=Thu, 01 Jan 1970 22:00:00 GMT; path=/; secure
Set-Cookie: webvpn=<elided>; path=/; secure
Set-Cookie: webvpnc=bu:/CACHE/stc/&p:t&iu:1/&sh:<truncated>&lu:/+CSCOT+/translation-table?textdomain%3DAnyConnect%26type%3Dmanifest;
path=/; secure
Set-Cookie: webvpnx=
Set-Cookie: webvpnaac=1; path=/; secure
X-Transcend-Version: 1
HTTP body chunked (-2)
Got CONNECT response: HTTP/1.1 200 OK
X-CSTP-Version: 1
X-CSTP-Protocol: Copyright (c) 2004 Cisco Systems, Inc.
X-CSTP-Address: 172.24.250.24
X-CSTP-Netmask: 255.255.255.0
X-CSTP-Hostname: hq-fw.corp.server.domain
X-CSTP-DNS: 172.24.1.220
X-CSTP-DNS: 172.24.1.221
X-CSTP-Lease-Duration: 1209600
X-CSTP-Session-Timeout: none
X-CSTP-Idle-Timeout: 1800
X-CSTP-Disconnected-Timeout: 1800
X-CSTP-Default-Domain: corp.server.domain
X-CSTP-Split-Include: 172.24.0.0/255.255.240.0
X-CSTP-Split-Include: <truncated_public_ip>.60/255.255.255.255
X-CSTP-Split-Include: 10.250.0.0/255.255.0.0
X-CSTP-Split-Include: <truncated_public_ip>.240/255.255.255.255
X-CSTP-Split-Include: <truncated_public_ip>.89/255.255.255.255
X-CSTP-Keep: true
X-CSTP-Tunnel-All-DNS: false
X-CSTP-DPD: 30
X-CSTP-Keepalive: 20
X-CSTP-MSIE-Proxy-Lockdown: true
X-CSTP-Smartcard-Removal-Disconnect: true
X-DTLS-Session-ID: <truncated_session_id>
X-DTLS-Port: 443
X-DTLS-Keepalive: 20
X-DTLS-DPD: 30
X-CSTP-MTU: 1406
X-DTLS-CipherSuite: AES128-SHA
X-CSTP-Routing-Filtering-Ignore: false
X-CSTP-Quarantine: false
X-CSTP-Disable-Always-On-VPN: false
X-CSTP-Client-Bypass-Protocol: false
X-CSTP-TCP-Keepalive: true
CSTP connected. DPD 30, Keepalive 20
CSTP Ciphersuite: (TLS1.0)-(DHE-RSA-1024)-(AES-128-CBC)-(SHA1)
DTLS option X-DTLS-Session-ID : <truncated_session_id>
DTLS option X-DTLS-Port : 443
DTLS option X-DTLS-Keepalive : 20
DTLS option X-DTLS-DPD : 30
DTLS option X-DTLS-CipherSuite : AES128-SHA
DTLS initialised. DPD 30, Keepalive 20
Connected tun0 as 172.24.250.24, using SSL
No work to do; sleeping for 20000 ms...
No work to do; sleeping for 19000 ms...
Established DTLS connection (using GnuTLS). Ciphersuite
(DTLS0.9)-(RSA)-(AES-128-CBC)-(SHA1).
No work to do; sleeping for 19000 ms...
Sent DTLS packet of 73 bytes; DTLS send returned 74
No work to do; sleeping for 19000 ms...
Sent DTLS packet of 82 bytes; DTLS send returned 83
Sent DTLS packet of 80 bytes; DTLS send returned 81
Sent DTLS packet of 81 bytes; DTLS send returned 82
No work to do; sleeping for 19000 ms...
Received DTLS packet 0x00 of 158 bytes
No work to do; sleeping for 18000 ms...


********************* ssh output ***************************

ssh name-company at name-app1.server.domain -vvv
OpenSSH_7.2p2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolving "name-app1.server.domain" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to name-app1.server.domain [<server_ip>] port 22.
debug1: Connection established.
debug1: identity file <homedir>.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file <homedir>.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to name-app1.server.domain:22 as 'name-company'
debug3: hostkeys_foreach: reading file "<homedir>.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file <homedir>.ssh/known_hosts:94
debug3: load_hostkeys: loaded 1 keys from name-app1.server.domain
debug3: order_hostkeyalgs: prefer hostkeyalgs:
ssh-rsa-cert-v01 at openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms:
curve25519-sha256 at libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms:
ssh-rsa-cert-v01 at openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01 at openssh.com,ecdsa-sha2-nistp384-cert-v01 at openssh.com,ecdsa-sha2-nistp521-cert-v01 at openssh.com,ssh-ed25519-cert-v01 at openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos:
chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc:
chacha20-poly1305 at openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm at openssh.com,aes256-gcm at openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos:
umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc:
umac-64-etm at openssh.com,umac-128-etm at openssh.com,hmac-sha2-256-etm at openssh.com,hmac-sha2-512-etm at openssh.com,hmac-sha1-etm at openssh.com,umac-64 at openssh.com,umac-128 at openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib at openssh.com,zlib
debug2: compression stoc: none,zlib at openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
debug2: ciphers stoc:
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc at lysator.liu.se
debug2: MACs ctos:
hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc:
hmac-md5,hmac-sha1,umac-64 at openssh.com,hmac-ripemd160,hmac-ripemd160 at openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib at openssh.com
debug2: compression stoc: none,zlib at openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC:
umac-64 at openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC:
umac-64 at openssh.com compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
Connection closed by <server_ip> port 22


Is that an issue already seen before? As it is working fine for
Anyconnect clients it really feels like an openconnect bug.

Thanks for any help!

Jean-Gabriel