What's the upstream disposition with nvme-stas?

[smith, erik]

John, first and foremost, the conversation between Dell and Red Hat the other day was under NDA. Because of this, I will not debate what you said on a point by point basis, but it's fair to state we remember things differently. 

What I can say publicly is this... We asked you to start a thread with Hannes and Keith because, despite being "tired of the hear-say game that happens with private conversations.", you were attributing all kinds of statements to them that did not align with what we had heard.  

All of that said, we just want a plan to move forward with the functionality our customers need. The way it gets delivered is a great discussion to have.

Regards, Erik 


Internal Use - Confidential

-----Original Message-----
From: John Meneghini <jmeneghi at redhat.com> 
Sent: Thursday, June 22, 2023 1:42 PM
To: linux-nvme at lists.infradead.org; Christoph Hellwig <hch at lst.de>; Sagi Grimberg <sagi at grimberg.me>; Keith Busch <kbusch at kernel.org>
Cc: smith, erik <Erik.Smith at dell.com>; Rose, Charles <Charles_Rose at Dell.com>; Belanger, Martin <Martin_Belanger at Dell.com>; Hannes Reinecke <hare at suse.de>; Martin Wilck <mwilck at suse.com>; Daniel Wagner <dwagner at suse.de>
Subject: What's the upstream disposition with nvme-stas?


[EXTERNAL EMAIL] 

Christoph, Sagi, Keith, et al,

There has been an ongoing series of 3-way private conversations about the status of nvme-stas and I am sending this email to put these conversations into the public.

Background:

If my memory serves me correctly: after several failed attempts to add an "NVMe/TCP auto-discovery" feature to Linux by leveraging an avahi based mDNS/SD mechanism (one by Sagi, circa 2018, and one by Martin Wilck, circa 2021) TP-8009/8010 was created at NVMexpress.org. At that time Martin Belanger asked Keith Busch for a repository in the linux-nvme domain on Github and nvme-stas was born. My understanding was that the goal of the nvme-stas repository was to stage the nvme-cli code changes that would be needed to eventually support TP-8009/8010. While some of that work has been done and nvme-cli now supports TP-8010 compliant discovery controllers (e.g. with the addition of the nvme dim - Send Discovery Information Management - command) there is still much functionality in nvme-stas that has not been merged into nvme-cli, including the mDNS/SD based functionality needed to support TP-8009. This remaining nvme-stas functionality, and the status of any effort to merge it into nvme-cli, is the source of all the private conversations and debates.

Current Status:

At LSFMM this year I spoke with privately with Keith Bush and others about the status of nvme-stas. When Keith and I spoke he told me that Dell had talked with him privately about nvme-stas and asked if the nvme-stas functionality can be merged into nvme-cli or otherwise accepted upstream.  Keith said that he told Dell, after looking at the nvme-stas implementation (again), the answer was no.

When I spoke with Hannes about nvme-stas he agreed with me that the utility is, at best, a stop-gap solution and that we need to continue to look for a better solution to the "NVMe/TCP auto-discovery" problem. We brain stormed about implementing an nvme-tcp discovery listener socket in Linux that leverages the Kickstart Discovery Request (KDReq) PDU from TP-8010.  We also brainstormed about this topic at the 2022 ALPSS conference last October. I think people in the upstream community are aware that some improvements with NVMe/TCP discovery are needed, but nobody, that I am aware of, see's nvme-stas as a solution to the problem. Nobody really cares about nvme-stas... but then that's the purpose of this email. I want to stop talking about what other people think.  They can speak for themselves. And if we are going to talk about the short comings and problems with nvme-stats, let's do that publicly, not privately.  I am tired of the hear-say game that happens with private conversations.

So what's the debate?

I work for Red Hat and Dell has asked Red Hat to support nvme-stas as a Red Hat product. Red Hat is not inclined to do this because we currently support discovering and managing all nvme and nvme-of devices with the nvme-cli utility and we don't want to support two different utilities to do this. Our general policy at Red Hat is: upstream first.  We don't want to support any package or feature that is not first supported upstream. So the push back from Red Hat to Dell, privately, has been: please get your nvme-stas functionality merged into the nvme-cli utility upstream. This is the source of all the three way private conversations with Dell and everyone else. Anyways, at this point it looks like a complete merge of nvme-stas with nvme-cli is not going to happen, so if there is any conversation about why or how or what needs to be done to change that, let's have that conversation here.

So what does "supported by" mean any way?

I believe the goal of the upstream linux-nvme project is to support one general purpose Linux utility used for discovering and managing nvme and nvme-of devices. This is the nvme-cli utility. This is what's "supported upstream" and I'd like to end any debate about whether or not the upstream community supports nvme-stas. Nvme-stas is supported by the upstream community to the extent that it uses the kernel and libnvme APIs to access nvme devices; just as any other third party user space application that exploits these kernel and user space APIs is supported. Other than this, nvme-stas is not "supported upstream".

Notwithstanding the fact that Dell's CDC is compliant the NVMe specification, nvme-stas is arguably little more than a Dell specific utility which is designed to optimize nvme-of discovery with Dell's products. There are no upstream or open source CDC implementations available (that I am aware of), and there are no commercial CDC implementations available other than Dell's (that I am aware of). So, at this point, nvme-stas's primary purpose is to optimize and enable the special nvme-of discovery features of Dell's CDC product. With this being the case, shouldn't nvme-stas be supported by Dell?  It was written and designed by Dell to support their products. Until this changes I really don't see why any entity other than Dell should be called upon to "support nvme-stas".

In summary:

- The linux-nvme project supports the kernel and library APIs implemented in libnvme in support of the nvme-cli utility.
- The linux-nvme project supports the nvme-cli utility for the general purpose of managing nvme and nvme-of devices.
- Anyone is free to contribute to libnvme and nvme-cli to the extent that their contributions support this goal with nvme-cli
- Projects like nvme-stas are free to use libnvme and nvme-cli to support their specific use case,
   but this does not mean their use case is supported by the upstream linux-nvme project
- It has been agreed that the remaining features and functions of nvme-stas cannot be merged into or subsumed by nvme-cli
- The upstream linux-nvme project neither approves of or disapproves of nvme-stas and the URL of the nvme-stas project
   on Github in the linux-nvme domain should not be seen as an endorsement by the linux-nvme project.
- The linux-nvme upstream community is aware of the problems with NVMe/TCP discovery and we are searching for a solution.
   We are also not sure we want to support mDNS/SD as a solution - TP-8009 notwithstanding.

Finally, whether or not any utility like nvme-stats or is supported by any Linux distribution is an independant business decision. These business decisions and support statements stand apart from any support statement made by the upstream linux-nvme project, and each Linux distribution has their own separate policies and standards for distributing and supporting Linux upstream packages, utilities, functions and features. We can't assume support by one distribution == support upstream == support by another distribution.

Thanks for your attention.

--
John Meneghini
RHEL SST - Platform Storage Group
jmeneghi at redhat.com