Sysfs paths to NVME devices

[Keith Busch]

On Thu, Nov 18, 2021 at 03:28:26PM +0100, Jean Delvare wrote:
> Hi all,
> 
> I have a few questions related to the sysfs paths to NVME devices. We
> have a dracut module using udevadm on block devices (under
> /sys/dev/block) to figure out which drivers should be included in the
> initrd, and noticed that it does not always work for NVME devices. Upon
> investigation, it was discovered that the link in /sys/dev/block leads
> to either a physical NVME device (e.g.
> /sys/devices/pci0000:00/0000:00:06.0/nvme/nvme0/nvme0n1) or a virtual
> NVME device (e.g. /sys/devices/virtual/nvme-subsystem/nvme-
> subsys0/nvme0n1). The latter case is problematic because virtual NVME
> devices do not have a driver attached to them.
> 
> First of all I would like to understand what is the deciding factor
> inside the kernel to go for virtual devices or physical devices. At
> first I thought it was related to CONFIG_NVME_MULTIPATH, but in fact
> all our systems have that option enabled, still some have virtual
> devices and others have physical devices.

If you're using nvme native multipathing, and your namespace reports
that it is multipath capable (ID_NS.NMIC), then the driver will set up
the virtual device for the visible block device.

If your namespace isn't multipath capable, you will only get the
physical device. That's just for pci, though; fabrics targets always
link to a virtual nvme subsystem device.

In the case you have a multipath namespace, the driver will also create
"hidden" block devices for each controller path that it found. As an
exampe, if you have multipath nvme /sys/block/nvme0n1, there should be a
/sys/block/nvme0c0n1, which should link to a physical device in sysfs
for pci.
 
> Secondly, I would like to know if there's a chance to have a consistent
> behavior where the paths would be the same on all systems, so that
> user-space only has to deal with one naming scheme instead of two. It
> would be nice not to have to deal with exceptions in dracut and udev.
> 
> Lastly, in the case of virtual NVME device paths (which I suspect can't
> be avoided in multipath scenarios), could you suggest a reliable way to
> figure out which drivers are being used? Multipath existed before NVME
> so I suppose there's a way to do it already, maybe the NVME subsystem
> needs to be adjusted to do it the same way other subsystems (SCSI) do
> it?
> 
> Thanks for any insight on this topic,
> -- 
> Jean Delvare
> SUSE L3 Support
>