nl_recv() and handling truncated messages

[Jan Gutter]

Hi,

I've run into a rather interesting problem recently where a weird
interaction between libnl3 and libvirt caused some difficult-to-debug
issues. From libnl3's side, the issue was that a netlink response was
much larger than the pagesize. If anyone's interested, the relevant
function is src/util/virnetlink.c:virNetlinkCommand(). When called in
a system where the number of PCIe Virtual Functions are more than 30
for a given Physical Function, the netlink response is larger than 4k,
meaning that a message is received with MSG_TRUNC set.

virNetlinkCommand() calls lib/nl.c:nl_recv() which in turn calls
recvmsg(). Unfortunately, it seems nl_recv() does not return an error
in this case. From tracing through the code, it seems that it assumes
that recvmsg() will return the actual size of the message. From the
man pages, this seems to be only true if flags have been set to
MSG_TRUNC when it was first called. If not, the rest of the message is
lost and the size of the buffer written to is returned.

nl_recv() works as expected if MSG_PEEK has been set, though, so it's
possible to fix in libvirt by either making the buffer bigger, or
enabling peeking.

However, it would be nice if the error is passed up to the user of libnl3.

There's a couple of options open:
1) Return NLE_MSG_TRUNC when a truncated message is received.
2) Enable MSG_TRUNC for all recvmsg calls and resize the buffer as
needed (meaning that NLE_MSG_TRUNC won't happen, but receive buffer
sizes can grow until realloc fails or a limit is hit...).
3) Make 2) optional (similar to enabling peeking) and return
NLE_MSG_TRUNC if it's not enabled.

I'm really not sure how much software is currently in the wild and
relies on this silent failure.

Jan