TLS over TCP connection failure

[Nir Rigai]

Hi all,

We failed to enable TLS over TCP with the latest 6.8-rc3 from upstream.
We have followed the guidelines outlined in https://lwn.net/Articles/941139/.
The NVMe connection failed post-authentication

The steps and the outputs from the process:  
# Target creation
modprobe nvmet-tcp
mkdir /sys/kernel/config/nvmet/ports/10
echo -n "127.0.0.1" > /sys/kernel/config/nvmet/ports/10/addr_traddr
echo -n ipv4 > /sys/kernel/config/nvmet/ports/10/addr_adrfam
echo -n tcp > /sys/kernel/config/nvmet/ports/10/addr_trtype
echo -n 4420 > /sys/kernel/config/nvmet/ports/10/addr_trsvcid
echo tls1.3 > /sys/kernel/config/nvmet/ports/10/addr_tsas
mkdir /sys/kernel/config/nvmet/subsystems/nqn.test
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/attr_allow_any_host
mkdir /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1
echo "/dev/nvme1n1" > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/device_path
echo 1 > /sys/kernel/config/nvmet/subsystems/nqn.test/namespaces/1/enable
ln -s /sys/kernel/config/nvmet/subsystems/nqn.test /sys/kernel/config/nvmet/ports/10/subsystems/
  # Create keys and start the tlshd service:
modprobe nvme-tcp
nvme gen-tls-key --subsysnqn=nqn.test -i
nvme gen-tls-key --subsysnqn=nqn.2014-08.org.nvmexpress.discovery -i
#tlshd -c /etc/tlshd.conf
systemctl start tlshd.service
nvme connect -t tcp -a 127.0.0.1 -s 4420 -n nqn.test --tls

------------------------

# Outcome of enabling TLS over TCP – While authentication succeeded as expected,
we encountered an issue where the NVMe connection failed:

root at ip-10-0-40-151:~/work# nvme list-subsys
nvme-subsys2 - NQN=nqn.test
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme2 tcp traddr=127.0.0.1,trsvcid=4420 connecting  nvme-subsys1 - NQN=nqn:2008-08.com.amazon.aws:ebs:vol0ada4ab89cbeec6a8
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme1 pcie 0000:00:1f.0 live
nvme-subsys0 - NQN=nqn:2008-08.com.amazon.aws:ebs:vol0867e65caaa7cb8ca
               hostnqn=nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133
               iopolicy=numa
\
+- nvme0 pcie 0000:00:04.0 live

# The initiator syslog:
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.496761] nvmet: creating nvm controller 1 for subsystem nqn.test for NQN nqn.2014-08.org.nvmexpress:uuid:ec20ce15-ac61-d1d3-0b2e-dcf390b05133.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.497751] nvme nvme2: creating 32 I/O queues.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.646851] nvme nvme2: mapped 32/0/0 default/read/poll queues.
Feb  5 16:17:36 ip-10-0-40-151 kernel: [ 2067.647105] nvme nvme2: Successfully reconnected (1 attempt)
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.076927] nvme nvme2: I/O tag 0 (a000) type 4 opcode 0x18 (Admin Cmd) QID 0 timeout
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.076940] nvme nvme2: starting error recovery
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.078899] nvme nvme2: failed nvme_keep_alive_end_io error=10
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.080682] nvme nvme2: reading non-mdts-limits failed: -4
Feb  5 16:17:43 ip-10-0-40-151 kernel: [ 2075.092748] nvme nvme2: Reconnecting in 10 seconds…
  # tlshd configuration:
[authentication]
keyrings = .nvme
[debug]
loglevel=255
tls=255

We would greatly appreciate your assistance with this matter. Your response
would be highly valued.

Thanks,
Nir Rigai