ncme-tcp: io_work NULL pointer when racing with queue stop

[Sagi Grimberg]

> On Tue, Jan 18, 2022 at 3:32 PM Sagi Grimberg <sagi at grimberg.me> wrote:
>>
>>> Hi,
>>
>> Hey Chris,
>>
>>> I'm looking at a reported NULL pointer fault with controller reset
>>> testing under load
>>
>> Just to understand, is this upstream kernel?
> 
> This was a CentOS Stream 9 kernel, which should be largely in sync
> with upstream currently.

Understood, can you perhaps share a reference to the bug report?

>>> and it appears that the nvme_tcp_stop_queue
>>> sequence is racing against scheduled io_work.  The fault occurs when
>>> kernel_sendpage is called from the workqueue context and hits a NULL
>>> sock->ops pointer, which is cleared by kernel_sock_shutdown before the
>>> call to cancel_work_sync.
>>
>> Hmm, can you point me to where kernel_sock_shutdown is clearing
>> sock->ops? The assumption that nvme-tcp takes is that accessing
>> the socket is not allowed only after sock_release() but
>> still valid after kernel_sock_shutdown.
> 
> Looks like I need to apologize Sagi.  I'm not sure where I got turned
> around looking at the socket code, but you're right and
> kernel_sock_shutdown doesn't look to be the culprit here.
> 
> Thank you for the following detailed description. I'm going to go back
> to my crash report and take another look at this one.

No worries Chris, perhaps I can assist.

Is the dmesg log prior to the BUG available? Does it tell us anything
to what was going on leading to this?

Any more information about the test case? (load + controller reset)
Is the reset in a loop? Any more info about the load?
Any other 'interference' during the test?
How reproducible is this?
Is this Linux nvmet as the controller?
How many queues does the controller have? (it will help me understand
how easy it is to reproduce on a vm setup)

Cheers,
Sagi