Re: blktests nvme/062 lockdep — defer TLS inline send to io_work?

[Hannes Reinecke]

On 6/26/26 3:19 AM, Xixin Liu wrote:
> Hi,
> 
> I'm looking at the nvme/062 failure Shinichiro reported on linux-7.1.y
> (PROVE_LOCKING=y, blktests v5a62429536b1) and would like to sanity-check
> a fix direction before sending a formal patch.
> 
> Link: https://lore.kernel.org/linux-nvme/ajtk1CaN1pBreS4O@shinmob/T/
> 
> Problem (reproduced on linux-next, same test)
> ---------------------------------------------
> With PROVE_LOCKING=y, nvme/062 (TLS connect) fails with dmesg:
> 
>    WARNING: possible circular locking dependency detected
>    ...
>    tx_lock (tls_sw_sendmsg)
>    ...
>    &set->srcu (blk_mq / kblockd context)
> 
> The #0 stack is nvme_tcp_queue_request() doing an inline nvme_tcp_send_all()
> from queue_rq, which calls tls_sw_sendmsg() while blk_mq still holds srcu.
> Lockdep already knows tx_lock -> elevator_lock -> srcu, so the reverse
> order triggers the cycle.
> 
> Teardown fixes (__fput_sync, memalloc_noio_save, 26bb12b9caaf) are already
> present in current trees; 062 still fails — this looks like the TLS send
> path on top of blk_mq, not the disconnect/teardown chain.
> 
> Proposed fix (one line + comment)
> ---------------------------------
> Skip the queue_rq inline-send optimization when nvme_tcp_queue_tls(queue):
> 
>    if (queue->io_cpu == raw_smp_processor_id() &&
>        !nvme_tcp_queue_tls(queue) &&
>        empty && mutex_trylock(&queue->send_mutex)) {
>            nvme_tcp_send_all(queue);
>            ...
>    }
> 
> TLS traffic would always go through nvme_tcp_io_work on nvme_tcp_wq (same
> as write_space / backpressure paths already do).  Lab check: nvme/062
> passes, no lockdep splat in dmesg.
> 
> Does skipping inline send for TLS queues (plain TCP unchanged) look like a
> reasonable fix to you?  I can post [PATCH] if this direction sounds OK.
> 

I always found the call to nvme_tcp_send_all() a bit questionable here. 
Technically we should be submitting the first PDU of the request only; 
with the current setup we'll be sending the first PDU of this request
_and_ all other requests which might be added to the queue while we've
been sending.
That causes quite some unfairness in I/O submission, and play
havoc for someone trying to do QoS here.

One could try to use 'nvme_tcp_try_send()' instead of 
'nvme_tcp_try_send_all()',  but not sure if that would
help for the lockdep splat.

You could remove that call (it's an optimisation, after all),
but that might cause a latency degradation.
So if you do that please run some performance testing to check
if the latency is affected?

Cheers,

Hannes
-- 
Dr. Hannes Reinecke                  Kernel Storage Architect
hare at suse.de                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich