[LSF/MM/BPF TOPIC] NVMe over MPTCP: Multi-Fold Acceleration for NVMe over TCP in Multi-NIC Environments

[Geliang Tang]

Hi Ming,

On Wed, 2026-02-25 at 13:57 +0800, Ming Lei wrote:
> Hi Geliang,
> 
> Looks one interesting topic!

Thanks for your reply.

> 
> On Thu, Jan 29, 2026 at 12:13:25PM +0800, Geliang Tang wrote:
> > As one of the MPTCP upstream developers, I'm recently working on
> > adding
> > MPTCP support to 'NVMe over TCP'. This approach achieves a multi-
> > fold
> > performance improvement over using standard TCP. The implementation
> > and
> > testing phases are largely complete. The code is currently in the
> > RFC
> > stage and has undergone several rounds of discussion and iteration
> > on
> > the MPTCP mailing list [1]. It will be sent to the NVMe mailing
> > list
> > shortly.
> > 
> > 1. Introduction to MPTCP
> > 
> > Multipath TCP (MPTCP), standardized in RFC 8684, represents a major
> > evolution of the TCP protocol. It enables a single transport
> > connection
> > to utilize multiple network paths simultaneously, providing
> > benefits in
> > redundancy, resilience, and bandwidth aggregation. Since its
> > introduction in Linux kernel v5.6, it has become a key technology
> > for
> > modern networking, particularly in multi-NIC environments.
> > 
> > On a supported system such as Linux, an MPTCP socket is created by
> > specifying the IPPROTO_MPTCP protocol in the socket() system call:
> > 
> > 	int fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);
> > 
> > This creates a socket that appears as a standard TCP socket to the
> > application but uses the MPTCP protocol stack underneath.
> > 
> > For more details, please visit the project website:
> > https://mptcp.dev.
> > 
> > 2. Implementation
> > 
> > 'NVMe over TCP' establishes multiple TCP connections between the
> > target
> > and host for data transfer. This includes one admin queue
> > connection
> > for management traffic and multiple I/O queue connections for data
> > traffic, with the number typically scaling with available CPU
> > cores.
> > While these multiple TCP connections (using the same IP address but
> > different ports) help distribute computational load across CPUs,
> > all
> > data traffic still flows through a single network interface card
> > (NIC),
> > even in multi-NIC environments.
> > 
> > The 'NVMe over MPTCP' solution enhances 'NVMe over TCP' by
> > replacing
> > the multiple TCP connections with multiple MPTCP connections,
> > leaving
> > other mechanisms unchanged. Internally, each MPTCP connection can
> > establish multiple subflows based on the number of configured NICs.
> > This distributes data traffic across all available NICs, thereby
> > increasing aggregate transmission speed.
> 
> NVMe supports multipath, which can apply load balance or sort of
> algorithm
> to maximize network link/bandwidth too.
> 
> Maybe you can compare mptcp with multipath in this viewpoint.

Indeed worth comparing. Although they work at different layers, their
goals share similarities. I'll compare them in a follow-up and get back
to you.

Thanks,
-Geliang

> 
> 
> 
> Thanks,
> Ming
>