2nd RTM_DELLINK notification is always 1 second delayed

[Tom, Deepak Abraham]

Hi!

I have a system configured with 2 physical eth interfaces connected to a switch.
When I reboot the switch, I see that the userspace RTM_DELLINK notifications for the interfaces are always 1 second apart although both links actually go down almost simultaneously!

>From /var/log/messages (logs have the granularity of a second, but can see both links go down almost simultaneously) -
Apr 3 12:20:49 m2-dl360g10-244 kernel: mlx5_core 0000:5d:00.0 eno5np0: Link down
Apr 3 12:20:49 m2-dl360g10-244 kernel: mlx5_core 0000:5d:00.1 eno6np1: Link down


Monitoring userspace link events with ip command (linkdown is always 1 second apart; order varies from run to run) -
[root]$ ip -t monitor link
Timestamp: Wed Apr 3 12:20:49 2024 365932 usec
6: eno5np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default
link/ether 88:e9:a4:54:89:4c brd ff:ff:ff:ff:ff:ff
altname enp93s0f0np0

Timestamp: Wed Apr 3 12:20:50 2024 382616 usec
7: eno6np1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default
link/ether 88:e9:a4:54:89:4d brd ff:ff:ff:ff:ff:ff
altname enp93s0f1np1


I wrote a simple test program that uses a netlink socket to get these link event notifications in the userspace. The exact same behavior can be seen there as well - 2nd interfaces' userspace notification is always delayed by 1 second.

This behavior is consistent across Debian 11, 12 and RHEL9 distros (with their corresponding stock libnl). This behavior is also not specific to a particular system. It should be easily reproducible on all hardware.

The subsequent RTM_NEWLINK notifications when the switch comes back up are however only delayed by a few microseconds between each other, which is as expected.

Anyone know why there is this 1 sec delay between RTM_DELLINK notifications in the userspace always, although both physical interfaces go down almost simultaneously? Also is there a way to not have this delay?
Any help is appreciated.

Thanks,
Deepak