What's the upstream disposition with nvme-stas?

[John Meneghini]

Christoph, Sagi, Keith, et al,

There has been an ongoing series of 3-way private conversations about the status of nvme-stas and I am sending this email to put 
these conversations into the public.

Background:

If my memory serves me correctly: after several failed attempts to add an "NVMe/TCP auto-discovery" feature to Linux by 
leveraging an avahi based mDNS/SD mechanism (one by Sagi, circa 2018, and one by Martin Wilck, circa 2021) TP-8009/8010 was 
created at NVMexpress.org. At that time Martin Belanger asked Keith Busch for a repository in the linux-nvme domain on Github 
and nvme-stas was born. My understanding was that the goal of the nvme-stas repository was to stage the nvme-cli code changes 
that would be needed to eventually support TP-8009/8010. While some of that work has been done and nvme-cli now supports TP-8010 
compliant discovery controllers (e.g. with the addition of the nvme dim - Send Discovery Information Management - command) there 
is still much functionality in nvme-stas that has not been merged into nvme-cli, including the mDNS/SD based functionality 
needed to support TP-8009. This remaining nvme-stas functionality, and the status of any effort to merge it into nvme-cli, is 
the source of all the private conversations and debates.

Current Status:

At LSFMM this year I spoke with privately with Keith Bush and others about the status of nvme-stas. When Keith and I spoke he 
told me that Dell had talked with him privately about nvme-stas and asked if the nvme-stas functionality can be merged into 
nvme-cli or otherwise accepted upstream.  Keith said that he told Dell, after looking at the nvme-stas implementation (again), 
the answer was no.

When I spoke with Hannes about nvme-stas he agreed with me that the utility is, at best, a stop-gap solution and that we need to 
continue to look for a better solution to the "NVMe/TCP auto-discovery" problem. We brain stormed about implementing an nvme-tcp 
discovery listener socket in Linux that leverages the Kickstart Discovery Request (KDReq) PDU from TP-8010.  We also 
brainstormed about this topic at the 2022 ALPSS conference last October. I think people in the upstream community are aware that 
some improvements with NVMe/TCP discovery are needed, but nobody, that I am aware of, see's nvme-stas as a solution to the 
problem. Nobody really cares about nvme-stas... but then that's the purpose of this email. I want to stop talking about what 
other people think.  They can speak for themselves. And if we are going to talk about the short comings and problems with 
nvme-stats, let's do that publicly, not privately.  I am tired of the hear-say game that happens with private conversations.

So what's the debate?

I work for Red Hat and Dell has asked Red Hat to support nvme-stas as a Red Hat product. Red Hat is not inclined to do this 
because we currently support discovering and managing all nvme and nvme-of devices with the nvme-cli utility and we don't want 
to support two different utilities to do this. Our general policy at Red Hat is: upstream first.  We don't want to support any 
package or feature that is not first supported upstream. So the push back from Red Hat to Dell, privately, has been: please get 
your nvme-stas functionality merged into the nvme-cli utility upstream. This is the source of all the three way private 
conversations with Dell and everyone else. Anyways, at this point it looks like a complete merge of nvme-stas with nvme-cli is 
not going to happen, so if there is any conversation about why or how or what needs to be done to change that, let's have that 
conversation here.

So what does "supported by" mean any way?

I believe the goal of the upstream linux-nvme project is to support one general purpose Linux utility used for discovering and 
managing nvme and nvme-of devices. This is the nvme-cli utility. This is what's "supported upstream" and I'd like to end any 
debate about whether or not the upstream community supports nvme-stas. Nvme-stas is supported by the upstream community to the 
extent that it uses the kernel and libnvme APIs to access nvme devices; just as any other third party user space application 
that exploits these kernel and user space APIs is supported. Other than this, nvme-stas is not "supported upstream".

Notwithstanding the fact that Dell's CDC is compliant the NVMe specification, nvme-stas is arguably little more than a Dell 
specific utility which is designed to optimize nvme-of discovery with Dell's products. There are no upstream or open source CDC 
implementations available (that I am aware of), and there are no commercial CDC implementations available other than Dell's 
(that I am aware of). So, at this point, nvme-stas's primary purpose is to optimize and enable the special nvme-of discovery 
features of Dell's CDC product. With this being the case, shouldn't nvme-stas be supported by Dell?  It was written and designed 
by Dell to support their products. Until this changes I really don't see why any entity other than Dell should be called upon to 
"support nvme-stas".

In summary:

- The linux-nvme project supports the kernel and library APIs implemented in libnvme in support of the nvme-cli utility.
- The linux-nvme project supports the nvme-cli utility for the general purpose of managing nvme and nvme-of devices.
- Anyone is free to contribute to libnvme and nvme-cli to the extent that their contributions support this goal with nvme-cli
- Projects like nvme-stas are free to use libnvme and nvme-cli to support their specific use case,
   but this does not mean their use case is supported by the upstream linux-nvme project
- It has been agreed that the remaining features and functions of nvme-stas cannot be merged into or subsumed by nvme-cli
- The upstream linux-nvme project neither approves of or disapproves of nvme-stas and the URL of the nvme-stas project
   on Github in the linux-nvme domain should not be seen as an endorsement by the linux-nvme project.
- The linux-nvme upstream community is aware of the problems with NVMe/TCP discovery and we are searching for a solution.
   We are also not sure we want to support mDNS/SD as a solution - TP-8009 notwithstanding.

Finally, whether or not any utility like nvme-stats or is supported by any Linux distribution is an independant business 
decision. These business decisions and support statements stand apart from any support statement made by the upstream linux-nvme 
project, and each Linux distribution has their own separate policies and standards for distributing and supporting Linux 
upstream packages, utilities, functions and features. We can't assume support by one distribution == support upstream == support 
by another distribution.

Thanks for your attention.

-- 
John Meneghini
RHEL SST - Platform Storage Group
jmeneghi at redhat.com