[LSF/MM TOPIC] NVMe over Fabrics auto-discovery in Linux

[Sagi Grimberg]

>> Partially beside the point.
>>
>> The problem currently is that FC-NVMe is the only transport which
>> implements dev_loss_tmo, causing connections to be dropped completely
>> after a certain time.
>> After that the user has to manually re-establish the connection via
>> nvme-cli, or one has to create some udev/systemd interaction (cf the
>> thread "nvme/fc: add 'discovery' sysfs attribute to fc transport
>> devices" and others).
>>
>> The other transports just keep the reconnection loop running, and the
>> user has to manually _disconnect_ here.
>>
>> So we have a difference in user experience, which should be reconciled.
> 
> This is incorrect. Rdma (and FC too) have the reconnect_delay timer that 
> caps, on a per-controller basis, how long the reconnection loop will run 
> before the controller is deleted.

echo that...

> As stated by Johannes, the real difference in behavior is establishing 
> the initial connectivity as well as those auto-reconnect behaviors where 
> connectivity was lost and later regained.

I agree that this is a problem for IP based transports. So until we
have discovery enhancements we probably need to poll predefined
discovery subsystems periodically and cross check with what we are
connected to like iscsi discoveryd..

>> Also, a user-space based rediscovery/reconnect will get tricky during
>> path failover, as one might end up with all connections down and no way
>> of ever being _able_ to call nvme-cli as the root fs in inaccessible.
>> But that might be another topic.

Again, polling discovery subsystems every X time will overcome that...

> I don't disagree with this, and do believe, that to support low-memory 
> issues during failover and reconnectivity (as we've seen in the past), 
> as well as to make booting easier (stop futzing with ramdisk), it will 
> likely require some amount of nvme discovery engine within the kernel. 
> It's difficult as without things like SSDP or LSP, rdma and tcp can't 
> really do this without admin help. But as you say - this can be a later 
> topic.

I hope fabrics 1.1 discovery enhancements will tackle all that...