nvme/pcie hot plug results in /dev name change

[Christoph Hellwig]

On Fri, Jan 20, 2023 at 02:42:23PM -0700, Keith Busch wrote:
> That is correct. We don't know the identity of the device at the point
> we have to assign it an instance number, so the hot added one will just
> get the first available unique number. If you need a consistent name, we
> have the persistent naming rules that should create those links in
> /dev/disk/by-id/.

Note that this a bit of a problem under a file system or stacking driver
that handles failing drives (e.g. btrfs or md raid), that holds ontop
the "old" device file, and then fails to find the new one.  I had a
customer complaint for that as well :)

The first hack was to force run the multipath code that can keep the
node alive.  That works, but is really ugly especially when dealing
with corner cases such as overlapping nsids between different
controllers.

In the long run I think we'll need to:
 - send a notification to the holder if a device is hot removed from
   the block layer so that it can clean up
 - make the upper layers look for the replugged devie

I've been working on some of this for a while but haven't made much
progress due to other committments.