Reconnect on RDMA device reset

[Oren Duer]

On Tue, Jan 30, 2018 at 12:28 AM, Doug Ledford <dledford at redhat.com> wrote:
> On Mon, 2018-01-29 at 22:36 +0200, Sagi Grimberg wrote:
>> >
>> That is the case for nvme as well, but I was merely saying that device
>> reset is not really a device removal. And this makes it hard for the ULP
>> to understand what to do (or for me at least...)
>
> OK, I get that the difference between the two is making it hard to
> understand what to do.  But, the truth of the issue is that whether you
> are doing a reset or a remove/add cycle, what *your* code needs to do
> doesn't change.  For both cases, your code must A) drop everything on
> the floor like a hot potato and B) restart from scratch.

Fully agree here. You don't want different code flow for reset. You already
have remove/add flows, which require to to act the right way as Doug
described. A reset is exactly remove and add of the same device.

> The only thing
> that's confusing you is that it's more or less assumed on a reset that
> you would auto-restart, where as it isn't so clear that you would want
> to do the same on a remove/add cycle.  I think the answer to your
> question is: if the same device comes back that went away, then yes,
> auto-restart would seem appropriate.  If you make that policy decision,
> then the *only* difference between device reset and device hot-replug is
> that you actually have to verify that the same device came back as went
> away.
>
> As an optional item, you could start a timer when the device disappears,
> and if it takes more than, say, 10 minutes to reappear, you could cancel
> the auto-restart on the basis that someone probably physically unplugged
> and replugged the card and they might not want that.  But really, aside
> from the fact that the hot plug flow needs you to check the same device
> comes back, reset and hot plug have the exact same requirements/needs
> and can be serviced by a single code path.

Not sure why we need to keep track whether it is the same device or not.
I fail to understand why we trust the system admin to create the connections
at the beginning, but we should not trust him anymore if the device was
removed, a new device was added in place of it, and it was configured with the
same network IP/subnet. To me it looks like the admin wanted exactly that:
for the connections to be restored over the new device.

If the admin does not want the reconnections to happen, he has the option
to explicitly request to disconnect. Same if the device was removed forever.

And to help us with all that, we have rdma_cm. As long as we repeat the
rdma_connect() on the initiator side and rdma_bind/resolve_addr()... on
the target side, we'll get exactly this behaviour.

If we want this using userspace daemons - that's fine. We'll need one for the
initiator stack and one for the target stack. And we'll probably need some
missing udev events and configfs entry?

-- 
Oren