kdump: crash_kexec()-smp_send_stop() race in panic

[Eric W. Biederman]

Américo Wang <xiyou.wangcong at gmail.com> writes:

> On Mon, Oct 24, 2011 at 11:14 PM, Eric W. Biederman
> <ebiederm at xmission.com> wrote:
>> Michael Holzheu <holzheu at linux.vnet.ibm.com> writes:
>>
>>> Hello Vivek,
>>>
>>> In our tests we ran into the following scenario:
>>>
>>> Two CPUs have called panic at the same time. The first CPU called
>>> crash_kexec() and the second CPU called smp_send_stop() in panic()
>>> before crash_kexec() finished on the first CPU. So the second CPU
>>> stopped the first CPU and therefore kdump failed.
>>>
>>> 1st CPU:
>>> panic()->crash_kexec()->mutex_trylock(&kexec_mutex)-> do kdump
>>>
>>> 2nd CPU:
>>> panic()->crash_kexec()->kexec_mutex already held by 1st CPU
>>>        ->smp_send_stop()-> stop CPU 1 (stop kdump)
>>>
>>> How should we fix this problem? One possibility could be to do
>>> smp_send_stop() before we call crash_kexec().
>>>
>>> What do you think?
>>
>> smp_send_stop is insufficiently reliable to be used before crash_kexec.
>>
>> My first reaction would be to test oops_in_progress and wait until
>> oops_in_progress == 1 before calling smp_send_stop.
>>
>
> +1
>
> One of my colleague mentioned the same problem with me inside
> RH, given the fact that the race condition window is small, it would
> not be easy to reproduce this scenario.

As for reproducing it I have a hunch you could hack up something
horrible with smp_call_function and kprobes.


On a little more reflection we can't wait until oops_in_progress goes
to 1 before calling smp_send_stop.  Because if crash_kexec is not
involved nothing we will never call smp_send_stop. 

So my second thought is to introduce another atomic variable
panic_in_progress, visible only in panic.  The cpu that sets
increments panic_in_progress can call smp_send_stop.  The rest of
the cpus can just go into a busy wait.  That should stop nasty
fights about who is going to come out of smp_send_stop first.

Eric