[PATCH 1/1] Add 'Transport Interface' (triface) option. This can be used to specify the IP interface to use for the connection. The driver uses that to set SO_BINDTODEVICE on the socket before connecting.

[Michael Christie]

> On May 5, 2021, at 3:32 PM, Sagi Grimberg <sagi at grimberg.me> wrote:
> 
> 
>>>> Given that this was the original intent for host_traddr, why not have
>>>> host_traddr resolve the iface from the address and set sockopt
>>>> SO_BINDTODEVICE on it?
>>>> 
>>> That was my question, too.
>>> 
>>> I would vastly prefer to not have another option to deal with (as it raises the
>>> question whether to add it eg during 'nvme connect-all') And one could
>>> argue that this was the intention of _having_ the host_traddr argument in
>>> the first place ...
>>> 
>>> Cheers,
>>> 
>>> Hannes
>>> --
>>> Dr. Hannes Reinecke		        Kernel Storage Architect
>>> hare at suse.de			               +49 911 74053 688
>>> SUSE Software Solutions Germany GmbH, 90409 Nürnberg
>>> GF: F. Imendörffer, HRB 36809 (AG Nürnberg)
>> Hi Sagi and Hannes,
>> Correct me if I'm wrong, but it sounds like host_traddr was primarily added for FC (at least it wasn't tested for TCP since it does not work in its current state). I'm not an expert on FC and maybe specifying an address is the right (and only) way to specify and interface for FC. For TCP, however, it's not advisable. Specifying an interface by its associated IP address is less intuitive than specifying the actual interface name and, in some cases, it simply won't work. That's because the association between interfaces and IP addresses is not predictable. IP addresses can be changed or can change by themselves over time (e.g. DHCP). Interface names are predictable [1] and will persist over time. Consider the following configuration.
>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
>>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>>     inet 100.0.0.100/24 scope global lo
>>        valid_lft forever preferred_lft forever
>> 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
>>     link/ether 08:00:27:21:65:ec brd ff:ff:ff:ff:ff:ff
>>     inet 100.0.0.100/24 scope global enp0s3
>>        valid_lft forever preferred_lft forever
>> 3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
>>     link/ether 08:00:27:4f:95:5c brd ff:ff:ff:ff:ff:ff
>>     inet 100.0.0.100/24 scope global enp0s8
>>        valid_lft forever preferred_lft forever
>> The above is a VM that I configured with the same IP address (100.0.0.100) on all interfaces. Doing a reverse lookup to identify the unique interface associated with 100.0.0.100 would simply not work here. And this is why the option host_iface is required. I understand that the above config does not represent a standard host system, but I'm using this to prove a point: "we can never know how a user will configure their system and the above configuration is perfectly fine by Linux".
> 
> Is this a common setting? I'd say that we should probably see a real
> life need for it before adding a user interface for it.

For iscsi, people used SO_BINDTODEVICE because for whatver reason they
really like putting multiple interfaces on the same subnet. They
would then want to have connection1 use enp0s0 and connection2 use enp0s1,
etc.

I can’t remember why (it could be a bug or maybe the user can’t config the
routing table) but just doing a bind() will sometimes not work
because packets will come in on enp0s0 but when you do a send() it would
go out on enp0s. But enp0s1 might be connected in way it can only reach
certain target ports. The target would then not see the response.