very low throughput on Mac OS X

[Tom Rodriguez]

> On Aug 18, 2016, at 2:58 AM, David Woodhouse <dwmw2 at infradead.org> wrote:
> 
> On Wed, 2016-08-17 at 09:26 -0700, Tom Rodriguez wrote:
>>> I’ve been using open connect on Mac OS X and noticed that I’m
>> getting much worse throughput than the same setup running on linux on
>> a local virtual machine.  Almost an order of magnitude worse.  I’ve
>> googled for help on the issue but without any luck, so here I am.  
> 
> Under Linux I'd be pointing you at the profiling tools so you can see
> precisely where the *system* is spending its time.
> 
> For OS X I am unfamiliar with any such tools. One thing which comes to
> mind is the different tun devices. We'll be using utun on OS X these
> days by default... does it get better if you install the Tun/Tap driver
> and use that instead? There's plenty of scope for screwing performance
> up in the user/kernel interface...
> 
> 
>> I’ve tried a few different command line options and adjusting the MTU
>> but without luck.  Are there any known issues concerning this?  I do
>> get a lot of "Failed to write incoming packet: No buffer space
>> available” on the mac but it didn’t seem like enough to explain the
>> low throughput. 
> 
> That's on receiving a packet from the VPN, and feeding it to the kernel
> through (presumably) the utun interface. Which tends to support my
> first supposition above…

I’m actually using ocproxy to handle the traffic so I don’t think I’m going through the tunnel device.  So does that mean the packets are being lost on the datagram socket connecting me to ocproxy?  My actual command line is this:

openconnect -q --compression=none --script-tun --script "ocproxy -D 9998”  <host>

Maybe buffering in the unix socket talking to ocproxy is to blame?  In the output below  fd 9 is the UDP connection to the VPN and 8 is one half of the unix socket talking to ocproxy.

select(0xA, 0x7FFF50ABFDE0, 0x7FFF50ABFD60, 0x7FFF50ABFCE0, 0x7FFF50ABFE60)		 = 1 0
recvfrom(0x9, 0x7FC8BD800053, 0x495C)		 = 1117 0
recvfrom(0x9, 0x7FC8BD804A53, 0x495C)		 = 1117 0
recvfrom(0x9, 0x7FC8BD800053, 0x495C)		 = 1117 0
recvfrom(0x9, 0x7FC8BD804A53, 0x495C)		 = 1117 0
recvfrom(0x9, 0x7FC8BD800053, 0x495C)		 = 1117 0
recvfrom(0x9, 0x7FC8BD804A53, 0x495C)		 = -1 Err#35
recvfrom(0x6, 0x7FC8BD804A5B, 0x5)		 = -1 Err#35
read(0x8, "\0", 0x514)		 = -1 Err#35
write(0x8, "E\0", 0x428)		 = 1064 0
write(0x8, "E\0", 0x428)		 = 1064 0
write(0x8, "E\0", 0x428)		 = 1064 0
write(0x8, "E\0", 0x428)		 = 1064 0
write(0x8, "\0", 0x428)		 = -1 Err#55
write_nocancel(0x2, "Failed to write incoming packet: No buffer space available\n\0", 0x3B)		 = 59 0

Some googling led me to net.local.dgram.recvspace to tune the buffer space for unix domain sockets.  Running this:

sudo sysctl -w net.local.dgram.recvspace=82320

brought my mac performance back up.  The default value of that tunable is 8192 which I guess is too low.  Maybe setsockopt should be used to increase it on the Mac?  Presumably linux has a higher default which is why it works ok there.

tom

> 
> Note that we *drop* the packet when that happens. See the code at
> http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/tun.c#l551
> http://git.infradead.org/users/dwmw2/openconnect.git/blob/HEAD:/mainloop.c#l95
> 
> So... the tun interface's queue is full and it stops accepting packets
> from us. We *drop* them, and that's going to screw your TCP connections
> up quite a lot.
> 
> In an ideal world, the poll() system call would let us know when the
> tun is writeable again, and we'd start shovelling packets then. In
> Linux that doesn't work so as the comments note, we don't bother with
> that. But perhaps we should, for OS X? You'd make os_write_tun()
> actually return -EAGAIN, and make mainlop.c react accordingly by
> calling monitor_write_fd() on the tun fd and only ever trying to write
> to it when it's... well, writeable.
> 
> And we'd still need a hack for Linux so that we don't just busy-wait
> trying to write the same packet over and over again until it's
> accepted. But first check if the utun interface really *does* support
> poll() for writeability at all, perhaps...
> 
> -- 
> dwmw2