TCP Sessions get disconnected at 6, 9 hours

[Cline, Wade]

On Tue, Feb 27, 2024 at 04:31:04PM -0800, Daniel Lenski wrote:
> On Tue, Feb 27, 2024 at 3:58 PM Larry Ploetz <lploetz at gmail.com> wrote:
> >
> > On 2024-02-25 11:03, Larry Ploetz wrote:
> > >> Are the users of the official PAN GP clients keeping SSH sessions open
> > >> for 6+ hours like you are?
> > >
> > >
> > > Yes, I believe so. I'll verify.
> >
> > Yes, ssh as well as other TCP connections are staying open for more than
> > 6 hours.
> >
> >
> > > I'll get back with more information.
> >
> >
> > No indication of any packets in openconnect's stderr, only routing
> > changes being made (add host/add net), and those are on startup - the
> > timestamps on the redirected stdout/stderr files are when openconnect
> > was started + 11 seconds.
> 
> You say you're collecting logs with maximum debugging verbosity
> (`--vvv --dump-http-traffic --timestamp`)… but you see *nothing at
> all* in the logs around 6 hours? 🤷🏻‍♂️
> 
> That makes no sense.
> 
> With either the ESP tunnel
> (https://gitlab.com/openconnect/openconnect/blob/master/esp.c#L217-432)
> or with the TLS tunnel
> (https://gitlab.com/openconnect/openconnect/blob/master/gpst.c#L1224-1364)
> you should be getting a log message with every single packet sent or
> received over the tunnel, including keepalive/DPD packets.
> 
> Your initial command line included `--syslog`, so the logs are
> certainly *not going to stderr* after the connection is established.
> https://www.infradead.org/openconnect/manual.html#opt-syslog
> 
> Are you sure you're looking at the right logs, in the right place?

I've read through the thread and this sounds exactly like the issue I
filed here:

	https://gitlab.com/openconnect/openconnect/-/issues/683

In addition to what I've mentioned in the issue, we've since found that
the issue always happens at half the timeout interval.  There are also
certain settings for the rekey interval which cause a total disconnection.
I've compared a "good" rekey to a "bad" rekey and the only differences in
XML are timer values and IPSec secrets; nothing to indicate a bad rekey.
One additional thing I've noticed is that *receiving* tends to function
for about 15 seconds or so before *sending* fails (I'll be in a call
with people and will be able to hear them but they won't hear me).
We're tried playing around with the MTU but that hasn't helped.

I've tried to MitM GP (6.1.0-c45) in order to see if its doing something
else at the halfway mark but have only been partially successful.
On first run, I have to set 'stream_large_bodies' to '1K' otherwise
GP thinks it never got a response to '/global-protect/getconfig.esp',
but I can then unset the option for subsequent runs and it gets past
this point (I think it caches the result).  The GP client then gets
past '/ssl-vpn/getconfig.esp' and connects to the VPN, after which all
subsequent connections fail with:

	warn: 192.168.10.186:55182: Client Handshake failed. The client may not trust the proxy's certificate for REDACTED.
	debug: 192.168.10.186:55182: ClientHandshakeException('Cannot establish TLS with client (sni: REDACTED): TlsException("SSL handshake error: Error([(\'SSL routines\', \'\', \'sslv3 alert certificate unknown\')])")')

I tried following the advice in order to defeat anti-MitM measures[1] but
there is no 'root-ca' sent so that technique did not work.  Most suspect
is in the request to 'ssl-vpn/login.esp' there is an 'argument' tag with
what appears to be a SHA1 hash, but it does not appear to correspond
to any certificate fingerprint or public key.  So I haven't yet figured
out how to successfully compare what GP is doing with what OC is doing.

Anyways, attached is some rough data on timeouts that we've gathered;
a quick legend is:

	Date - HTTP Connection: Date an HTTPS connection was established (empty if never)
	Date - Timeout Reached: Date the connection timed out naturally (empty if never)
	Date - Tunnel Error: Date the connection broke before natural timeout (empty if never)
	ESO Duration: Duration of ESP portion of the connection

Will keep triaging the issue as time permits.

Regards,
Wade

[1] https://www.infradead.org/openconnect/mitm.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: data.csv
Type: text/csv
Size: 11117 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/openconnect-devel/attachments/20240228/91938d21/attachment.csv>