stuck workers?
Yick Xie
yick.xie at gmail.com
Sun Jan 17 01:30:03 PST 2016
Hi,
I updated the ocserv to commit
3eb2b903ed8c9cd4e10304e319465e1e50424f97(worker: Prevent any frozen
worker processes by killing them on inactivity), and found sometimes
the ocserv worker still become inactive. #A For those "active"
connections, there is no more effective requests(cannot ping gateway),
even though by tcpdump it's seen there was still some repeated
communication between the server and the client, then after couples of
minutes the problem was gone and a reconnection was launched
automatically. #B For those already disconnections, the occtl cannot
disconnect them as before said, once again several minutes later this
worker was destoryed automatically. Please have a check with both.
#A "Active" wokrer gdb:
gdb /usr/bin/ocserv 16553
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/usr/bin/ocserv: No such file or directory.
Attaching to process 16553
Reading symbols from /usr/local/sbin/ocserv...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libopts.so.25...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libopts.so.25
Reading symbols from /usr/local/lib/libgnutls.so.28...done.
Loaded symbols for /usr/local/lib/libgnutls.so.28
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpam.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libseccomp.so.2...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libseccomp.so.2
Reading symbols from /lib/x86_64-linux-gnu/libwrap.so.0...(no
debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libwrap.so.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from
/usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1
Reading symbols from /lib/libfreeradius-client.so.2...done.
Loaded symbols for /lib/libfreeradius-client.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libev.so.4...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libev.so.4
Reading symbols from /usr/lib/libpcl.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no
debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libaudit.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from
/lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
0x00007fac5291de1b in __pselect (nfds=15,
readfds=readfds at entry=0x7ffff8899e80, writefds=writefds at entry=0x0,
exceptfds=exceptfds at entry=0x0, timeout=<optimized out>,
timeout at entry=0x7ffff8899e40,
sigmask=sigmask at entry=0x7ffff8899f00) at
../sysdeps/unix/sysv/linux/pselect.c:71
71 ../sysdeps/unix/sysv/linux/pselect.c: No such file or directory.
(gdb) bt full
#0 0x00007fac5291de1b in __pselect (nfds=15,
readfds=readfds at entry=0x7ffff8899e80, writefds=writefds at entry=0x0,
exceptfds=exceptfds at entry=0x0, timeout=<optimized out>,
timeout at entry=0x7ffff8899e40,
sigmask=sigmask at entry=0x7ffff8899f00) at
../sysdeps/unix/sysv/linux/pselect.c:71
resultvar = 18446744073709551102
tval = {tv_sec = 9, tv_nsec = 126225802}
data = {ss = 140737363156736, ss_len = 8}
result = <optimized out>
#1 0x000000000040b6b4 in connect_handler (ws=ws at entry=0xff9c30) at
worker-vpn.c:1922
req = 0xff9c60
rfds = {fds_bits = {16387, 0 <repeats 15 times>}}
e = <optimized out>
max = 14
ret = <optimized out>
t = 2048
p = <optimized out>
rnd = 1335569060
tv = {tv_sec = 10, tv_nsec = 0}
tls_pending = 0
dtls_pending = 0
i = <optimized out>
tnow = {tv_sec = 1453020235, tv_nsec = 813061000}
ip6 = <optimized out>
sl = 4
emptyset = {__val = {0 <repeats 16 times>}}
blockset = {__val = {16384, 0 <repeats 15 times>}}
#2 0x000000000040d69c in vpn_server (ws=ws at entry=0xff9c30) at worker-vpn.c:571
ret = <optimized out>
nparsed = <optimized out>
nrecvd = <optimized out>
session = 0x1017010
parser = {type = 0 '\000', flags = 0 '\000', state = 17
'\021', header_state = 0 '\000', index = 0 '\000', nread = 0,
content_length = 18446744073709551615, http_major = 1,
http_minor = 1, status_code = 0, method = 5 '\005',
http_errno = 0 '\000', upgrade = 1 '\001', data = 0xff9c30}
settings = {on_message_begin = 0x0, on_url = 0x41db60
<http_url_cb>, on_status_complete = 0x0, on_header_field = 0x41dbb0
<http_header_field_cb>,
on_header_value = 0x41dcc0 <http_header_value_cb>,
on_headers_complete = 0x41ddc0 <http_header_complete_cb>, on_body =
0x41de50 <http_body_cb>,
on_message_complete = 0x41de30 <http_message_complete_cb>}
fn = <optimized out>
requests_left = 15
#3 0x0000000000408b87 in listen_watcher_cb (loop=0x7fac53002900,
w=<optimized out>, revents=<optimized out>) at main.c:1061
stype = 0
s = 0xfe59e0
ltmp = <optimized out>
ctmp = 0x0
ws = 0xff9c30
fd = 0
ret = <optimized out>
cmd_fd = {13, 14}
pid = <optimized out>
#4 0x00007fac52df8e45 in ev_invoke_pending () from
/usr/lib/x86_64-linux-gnu/libev.so.4
No symbol table info available.
#5 0x00007fac52dfbfe7 in ev_run () from /usr/lib/x86_64-linux-gnu/libev.so.4
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#6 0x0000000000406b42 in main (argc=<optimized out>, argv=<optimized
out>) at main.c:1342
e = <optimized out>
ltmp = 0xfe5a50
ret = <optimized out>
flags = <optimized out>
p = <optimized out>
worker_pool = <optimized out>
main_pool = 0xfe5920
s = 0xfe59e0
creds = {xcred = 0xfe9f10, cprio = 0xfea6e0, dh_params = 0x0}
#B Disconnection worker gdb:
gdb /usr/bin/ocserv 15964
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
/usr/bin/ocserv: No such file or directory.
Attaching to process 15964
Reading symbols from /usr/local/sbin/ocserv...done.
Reading symbols from /usr/lib/x86_64-linux-gnu/libopts.so.25...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libopts.so.25
Reading symbols from /usr/local/lib/libgnutls.so.28...done.
Loaded symbols for /usr/local/lib/libgnutls.so.28
Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libpam.so.0
Reading symbols from /usr/lib/x86_64-linux-gnu/libseccomp.so.2...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libseccomp.so.2
Reading symbols from /lib/x86_64-linux-gnu/libwrap.so.0...(no
debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libwrap.so.0
Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1
Reading symbols from
/usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1...(no debugging
symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1
Reading symbols from /lib/libfreeradius-client.so.2...done.
Loaded symbols for /lib/libfreeradius-client.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4
Reading symbols from /usr/lib/x86_64-linux-gnu/libev.so.4...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libev.so.4
Reading symbols from /usr/lib/libpcl.so.1...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libpcl.so.1
Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6
Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging
symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1
Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2
Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no
debugging symbols found)...done.
Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10
Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no
debugging symbols found)...done.
Loaded symbols for /lib/x86_64-linux-gnu/libaudit.so.1
Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols
from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6
Reading symbols from
/lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from
/usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2
Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading
symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.19.so...done.
done.
Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2
0x00007fac5291d417 in __libc_writev (fd=0, vector=0x7ffff8899a90,
count=1) at ../sysdeps/unix/sysv/linux/writev.c:49
49 ../sysdeps/unix/sysv/linux/writev.c: No such file or directory.
(gdb) bt full
#0 0x00007fac5291d417 in __libc_writev (fd=0, vector=0x7ffff8899a90,
count=1) at ../sysdeps/unix/sysv/linux/writev.c:49
resultvar = 12
result = <optimized out>
#1 0x00007fac53ed9c58 in _gnutls_writev (total=58, giovec_cnt=1,
giovec=0x7ffff8899a90, session=0x1016010) at gnutls_buffers.c:483
i = <optimized out>
is_dtls = false
no_writev = 0
fd = 0x0
#2 _gnutls_io_write_flush (session=session at entry=0x1016010) at
gnutls_buffers.c:683
msg = {data = 0x0, size = 0}
send_buffer = 0x10168f0
ret = <optimized out>
sent = 0
tosend = 58
iovec = {{iov_base = 0x10276d0, iov_len = 58}, {iov_base =
0x10, iov_len = 16883320}, {iov_base = 0x1018d60, iov_len =
1647603077}, {iov_base = 0x10276fa, iov_len = 16883288}, {
iov_base = 0x10, iov_len = 16883320}, {iov_base =
0x7ffff8899b10, iov_len = 140378103648064}, {iov_base = 0x10276fa,
iov_len = 140378103671172}, {iov_base = 0x414a,
iov_len = 16878944}, {iov_base = 0x8e1b82e2935c2b50,
iov_len = 12934948170159259015}, {iov_base = 0x0, iov_len =
18319672542873517056}, {iov_base = 0x10276dd,
iov_len = 16669880}, {iov_base = 0x0, iov_len = 29},
{iov_base = 0x10276dd, iov_len = 16753016}, {iov_base = 0x414a,
iov_len = 140378119245405}, {iov_base = 0xfe5cb8,
iov_len = 140378119245632}, {iov_base = 0x35, iov_len =
16714}, {iov_base = 0x1016010, iov_len = 16938624}, {iov_base = 0x0,
iov_len = 140378122284096}, {iov_base = 0x35,
iov_len = 16714}, {iov_base = 0x1016010, iov_len =
140378119177280}, {iov_base = 0x107, iov_len = 16669880}, {iov_base =
0x7fff0000001d, iov_len = 16938717}, {iov_base = 0x0,
iov_len = 4}, {iov_base = 0x8, iov_len = 511101108253},
{iov_base = 0xffa178, iov_len = 140737363156075}, {iov_base =
0x100000000, iov_len = 0}, {iov_base = 0x40000000c,
iov_len = 140737363156064}, {iov_base = 0xfe5990, iov_len
= 140378119163654}, {iov_base = 0x40, iov_len = 16}, {iov_base =
0x5e3c000000000000, iov_len = 139762530976535}, {
iov_base = 0x88874e38, iov_len = 1580990464}, {iov_base =
0xffa178, iov_len = 18319672542873517056}}
i = 1
cur = <optimized out>
#3 0x00007fac53eda00c in _gnutls_io_write_buffered
(session=session at entry=0x1016010, bufel=<optimized out>,
mflag=mflag at entry=1) at gnutls_buffers.c:639
send_buffer = 0x10168f0
#4 0x00007fac53ed477c in _gnutls_send_tlen_int
(session=session at entry=0x1016010,
type=type at entry=GNUTLS_APPLICATION_DATA, htype=htype at entry=4294967295,
epoch_rel=epoch_rel at entry=70001,
_data=_data at entry=0x1016010, data_size=data_size at entry=29,
min_pad=min_pad at entry=0, mflags=mflags at entry=1) at gnutls_record.c:551
bufel = <optimized out>
cipher_size = <optimized out>
retval = 29
ret = <optimized out>
send_data_size = 29
data = 0x1016010 "\001"
record_params = 0xfe5990
max_send_size = 12
record_state = 0xfe5c88
#5 0x00007fac53ed6a3a in _gnutls_send_int (mflags=1, data_size=29,
_data=0x1016010, epoch_rel=70001, htype=4294967295,
type=GNUTLS_APPLICATION_DATA, session=0x1016010)
at gnutls_record.h:43
No locals.
#6 gnutls_record_send (session=0x1016010, data=data at entry=0xffa178,
data_size=data_size at entry=29) at gnutls_record.c:1564
No locals.
#7 0x0000000000410c4c in cstp_send (ws=ws at entry=0xff9c30,
data=data at entry=0xffa178, data_size=data_size at entry=29) at tlslib.c:92
ret = <optimized out>
left = 29
p = 0xffa178 "STF\001"
#8 0x000000000040be7d in tun_mainloop (tnow=0x7ffff8899e50,
ws=0xff9c30) at worker-vpn.c:1245
ret = <optimized out>
---Type <return> to continue, or q <return> to quit---
e = <optimized out>
cstp_type = 0
dtls_to_send = {data = 0xffa178 "STF\001", size = 21}
l = <optimized out>
tls_retry = <optimized out>
dtls_type = 0
cstp_to_send = {data = 0xffa178 "STF\001", size = 21}
#9 connect_handler (ws=ws at entry=0xff9c30) at worker-vpn.c:1949
req = 0xff9c60
rfds = {fds_bits = {2, 0 <repeats 15 times>}}
e = <optimized out>
max = 13
ret = <optimized out>
t = 2048
p = <optimized out>
rnd = 648176375
tv = {tv_sec = 10, tv_nsec = 0}
tls_pending = 0
dtls_pending = 0
i = <optimized out>
tnow = {tv_sec = 1453019195, tv_nsec = 285061000}
ip6 = <optimized out>
sl = 4
emptyset = {__val = {0 <repeats 16 times>}}
blockset = {__val = {16384, 0 <repeats 15 times>}}
#10 0x000000000040d69c in vpn_server (ws=ws at entry=0xff9c30) at worker-vpn.c:571
ret = <optimized out>
nparsed = <optimized out>
nrecvd = <optimized out>
session = 0x1016010
parser = {type = 0 '\000', flags = 0 '\000', state = 17
'\021', header_state = 0 '\000', index = 0 '\000', nread = 0,
content_length = 18446744073709551615, http_major = 1,
http_minor = 1, status_code = 0, method = 5 '\005',
http_errno = 0 '\000', upgrade = 1 '\001', data = 0xff9c30}
settings = {on_message_begin = 0x0, on_url = 0x41db60
<http_url_cb>, on_status_complete = 0x0, on_header_field = 0x41dbb0
<http_header_field_cb>,
on_header_value = 0x41dcc0 <http_header_value_cb>,
on_headers_complete = 0x41ddc0 <http_header_complete_cb>, on_body =
0x41de50 <http_body_cb>,
on_message_complete = 0x41de30 <http_message_complete_cb>}
fn = <optimized out>
requests_left = 15
#11 0x0000000000408b87 in listen_watcher_cb (loop=0x7fac53002900,
w=<optimized out>, revents=<optimized out>) at main.c:1061
stype = 0
s = 0xfe59e0
ltmp = <optimized out>
ctmp = 0x0
ws = 0xff9c30
fd = 0
ret = <optimized out>
cmd_fd = {1, 13}
pid = <optimized out>
#12 0x00007fac52df8e45 in ev_invoke_pending () from
/usr/lib/x86_64-linux-gnu/libev.so.4
No symbol table info available.
---Type <return> to continue, or q <return> to quit---
#13 0x00007fac52dfbfe7 in ev_run () from /usr/lib/x86_64-linux-gnu/libev.so.4
No symbol table info available.
#14 0x0000000000406b42 in main (argc=<optimized out>, argv=<optimized
out>) at main.c:1342
e = <optimized out>
ltmp = 0xfe5a50
ret = <optimized out>
flags = <optimized out>
p = <optimized out>
worker_pool = <optimized out>
main_pool = 0xfe5920
s = 0xfe59e0
creds = {xcred = 0xfe9f10, cprio = 0xfea6e0, dh_params = 0x0}
More information about the openconnect-devel
mailing list