[REGRESSION] Virtio networking issues on v6.17-rc1
Cristian Marussi
cristian.marussi at arm.com
Wed Aug 13 14:04:54 PDT 2025
On Wed, Aug 13, 2025 at 10:41:28PM +0200, Paolo Abeni wrote:
> On 8/13/25 10:31 PM, Cristian Marussi wrote:
> > On Wed, Aug 13, 2025 at 09:51:30PM +0200, Paolo Abeni wrote:
> >> On 8/13/25 8:09 PM, Cristian Marussi wrote:
> >>> On Wed, Aug 13, 2025 at 03:07:34PM +0100, Cristian Marussi wrote:
> >>>> Not sure if it has been already reported but in a kvmtool/guest setup moving
> >>>> the guest kernel from v6.16 to v6.17-rc1 I completely lost host-guest network
> >>>> functionality....in a very funny way, though, I'd say...
> >>>>
> >>>> In fact NO error is apparently reported in the guest kernel log and the
> >>>> interfaces seems perfectly up an running both sides, but looking at the
> >>>> host/guest interfaces you can see that ALL received frames are indeed dropped:
> >>>>
> >>>>
> >>>> enp0s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> >>>> ....
> >>>> ether 02:15:15:15:15:15 txqueuelen 1000 (Ethernet) <<<<<<<<<<<<<<<<
> >>>> RX packets 125 bytes 17948 (17.5 KiB)
> >>>> RX errors 0 dropped 125 overruns 0 frame 0
> >>>> TX packets 1207 bytes 51182 (49.9 KiB)
> >>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> >>>>
> >>>>
> >>>> ...on the host same..(taken later on...)
> >>>>
> >>>> tap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> >>>> inet 192.168.33.1 netmask 255.255.255.0 broadcast 192.168.33.255
> >>>> ether 8a:10:f6:df:a1:70 txqueuelen 1000 (Ethernet)
> >>>> RX packets 804 bytes 43904 (42.8 KiB)
> >>>> RX errors 0 dropped 804 overruns 0 frame 0
> >>>> TX packets 101 bytes 14408 (14.0 KiB)
> >>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> >>>>
> >>>> ....and for a good reason, apparently, since sniffing around on the Host TAP
> >>>> interface I can see a never ending stream of:
> >>>>
> >>>> $ sudo tcpdump -i tap0
> >>>> listening on tap0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
> >>>> 22:40:42.309158 00:00:00:00:00:00 (oui Ethernet) > 00:00:00:00:00:00 (oui Ethernet), ethertype Unknown (0xffff), length 54:
> >>>> 0x0000: ffff ffff 0215 1515 1515 0806 0001 0800 ................ <<<<<<<<<<<<<
> >>>> 0x0010: 0604 0001 0215 1515 1515 c0a8 2102 0000 ............!...
> >>>> 0x0020: 0000 0000 c0a8 2101 ......!.
> >>>>
> >>>> ... DST/SRC Macs are just all zeros WHILE in the payload you can spot my guest
> >>>> SRC mac address 0215 1515 1515 :P
> >>>>
> >>>
> >>> I bisected this regression to:
> >>>
> >>> 56a06bd40fab64448aa6b84aa06b3dc470c1254a is the first bad commit
> >>>
> >>> commit 56a06bd40fab64448aa6b84aa06b3dc470c1254a
> >>> Author: Paolo Abeni <pabeni at redhat.com>
> >>> Date: Tue Jul 8 17:55:02 2025 +0200
> >>>
> >>> virtio_net: enable gso over UDP tunnel support.
> >>>
> >>> If the related virtio feature is set, enable transmission and reception
> >>> of gso over UDP tunnel packets.
> >>>
> >>> Most of the work is done by the previously introduced helper, just need
> >>> to determine the UDP tunnel features inside the virtio_net_hdr and
> >>> update accordingly the virtio net hdr size.
> >>>
> >>> Acked-by: Jason Wang <jasowang at redhat.com>
> >>> Signed-off-by: Paolo Abeni <pabeni at redhat.com>
> >>>
> >>> drivers/net/virtio_net.c | 85 ++++++++++++++++++++++++++++++++++++------------
> >>>
> >>> Reverting this commit on top of v6.17-rc1 solves for me and network works fine again.
> >>
> >> Thanks for reporting, I was not aware yet of this regression.
> >>
> >
> > Hi Paolo,
> >
> > I spotted this issue on my setup this morning.
> >
> >> Apparently there is mismatch between the negotiated features (that
> >> determinate the virtio net header size) and the actually virtio header
> >> used by both the guest and the host, which is totally unexpected.
> >>
> >
> > Have the UAPI headers been updated ? ... I could not see any change...
> > (not that I am very familiar with virtio core :D)
>
> The virtio feature space has been extended from 64 bits to 128 and the
> blamed commit is the first using the 'extended' virtio features
> negotiation.
Understood, thanks for the explanation.
>
> But, modulo bugs, the extended space should be used only if the
> hypervisors exposes the relevant capabilities. kvmtool does not do that.
>
Ah, ok...I was hoping it was just a matter of kvmtool still to have to
update headers but maybe there's something more.
> >> Could you please share the host kernel version, the arch and the kvmtool
> >> version? Also I understand you only upgraded the guest kernel, without
> >> any other setup changes, am I correct?
> >>
> >
> > The arch is arm64, I could reproduce on a Raspberry Pi5 with host v6.6 and on
> > an Apple M1 (running Linux natively) with a v6.12.
> >
> > Both host kernels are from the original distros unchanged (Debian and Fedora)
> >
> > In both cases I compiled from scratch (make clean && make) kvmtool on the
> > respective arm64 Host using the current top of tree from origin/master at:
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/will/kvmtool.git/log/
> >
> > commit ba6830eec0f5 ("vfio: include libgen.h (for musl compatibility)")
> >
> > Run kvmtoool baer simple with
> >
> > lkvm run -c 4 -m 4G -k $IMAGE_DEF --network virtio -d $ROOTFS_DEF -p "earlycon loglevel=8"
> >
> > All was fine till v6.16, and it is my usual setup and update procedure that never failed really
> > as long as I update kvmtool in sync.
> > Not sure if something is still missing from kvmtool master repo for v6.17 but it seems reasonably
> > updated (July25).
> >
> > I wonder if no one else had this issue on arm64/kvmtool....that would made my 'regression'
> > suspiciously tied tomy setup..
>
> Thanks for the detailed reproduction steps.
>
> I agree it should be some bad iteration with the arm arch and/or
> kvmtool. I hope an arm system is not required to reproduce the issue,
> because I don't have any of them handy.
Thanks for having a look, hope not to have made noise for nothing and
maybe it is just a matter of a pending udpate from kvmtool guys...indeed
looking at their mailing list there is a pending patch to update headers
to v6.16 (for unrelated reasons)...
https://lore.kernel.org/kvm/20250729095745.3148294-2-andre.przywara@arm.com/T/#u
....but I suppose it would not be enough to catch with v6.17 changes
anyway.
Anyway if you need to test any fix on arm64 just let me know and I can do it
(...modulo my incoming holiday end-of-next-week :P)
>
> I should be able to dig into this tomorrow afternoon my time.
>
No worries...no rush...I have a good workaround for now...I just wanted to
raise a flag to make sure I am not hallucinating on my own :D
Thanks for you help
Cheers,
Cristian
More information about the linux-arm-kernel
mailing list