[tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path

[HATAYAMA Daisuke]

2012/2/17 Eric W. Biederman <ebiederm at xmission.com>:
> ebiederm at xmission.com (Eric W. Biederman) writes:
>
>> Don Zickus <dzickus at redhat.com> writes:
>>
>>> On Thu, Feb 16, 2012 at 01:53:29PM -0800, Yinghai Lu wrote:
>>>> On Thu, Feb 16, 2012 at 9:27 AM, Don Zickus <dzickus at redhat.com> wrote:
>>>>
>>>> > So I think I figured it out.  I went through and commented out code in
>>>> > disable_local_APIC until I narrowed it down to the piece of code that
>>>> > needs to be disabled for it to work.
>>>> >
>>>> > Surprise, surprise... its LVTPC or perf! :-)  Actually it is the
>>>> > nmi_watchdog which uses perf.  My theory is NMIs are not disabled and one
>>>> > is generated by the local apic during decompression (just bad timing) and
>>>> > *splat*.
>>>> >
>>>> > Yinghai, you can probably prove this by
>>>> >
>>>> > echo 0 > /proc/sys/kernel/nmi_watchdog
>>>> >
>>>> > then do your kdump crash test.
>>>>
>>>> yes.  that will make kdump crash working.
>>>
>>> Cool.  Thanks.
>>>
>>> Eric,
>>>
>>> Just let me know how you want to handle disabling NMIs in the kexec in
>>> panic shutdown case.
>>
>> Interesting.  Apparently we have been avoiding this problem by accident.
>>
>> Thanks for hunting this down.
>>
>> The options I can see are:
>> - Ensure we can handle and ignore exceptions like this.
>> - Always shutoff the lapic and ioapic entries that can generate this.
>>
>> The good news is that both solutions should be lock free.
>>
>> The current kernel boot code relies on the assumption that all
>> interrupts can be disabled.  In this case with nmi's that is clearly not
>> the case.
>>
>> The most robust solution and what we want to do long term is to
>> install an idt that will simply ignore all interrupts until the
>> idt is replaced.  Since really all we need to deal with is the NMI
>> vector, which is vector #2, we can have a very small interrupt
>> descriptor table.
>>
>> Unfortunately we go through some cpu mode switches in /sbin/kexec,
>> allowing us to enter the kernels 32bit entry point before we
>> run the decompresser, so at first glance both /sbin/kexec and the
>> kernel need to be fixed in a coordinated fashion.
>>
>> There are two was I can see of removing the need for an exactly
>> coordinated release.
>> - Document that an old /sbin/kexec userspace requires you not to
>>   use the nmi watchdog with modern kernels.
>> - For a short while simply retain code that stomps the nmi watchdog.
>>   (But still leaves us open to other kinds of nmi's).
>>
>> Grr.  Looking a little more closely, all throughout the linux kernel's
>> boot there is the assumption that any interrupt during boot is a failure
>> of some kind, and except for an errant nmi watchdog that is a true
>> assumption.
>>
>> Don I guess I really have to recommend disabling the nmi watchdog in the
>> kexec on panic path if we can do so at all reasonably.
>>
>> I like the idea of ignoring nmis during boot but that seems to be a
>> slightly larger project and with little practical improvement in kexec
>> on panic quality.  Other than getting what should be one or two
>> i/o writes out of the kexec on panic path.
>
> Hmm.
>
> Thinking about it a little more.  The kernel's boot path is inconsistent
> with the rest of the kernel's nmi handling.  For anything exception
> except an nmi stopping and giving up is fine.  For an nmi it is very
> rare for an NMI to signal a truly nasty failure (usually it just means
> someone saw a bitflip somewhere), and we can almost always continue
> without problem.
>
> I think in practice we really should make our boot path consistent with
> the rest of the kernel and ignore/log/report nmis but not fail on them.
> Triple faulting (trigger a cpu reset) as we do today just seems like a
> recipe for deep and confusing mystery, and not being helpful to the
> user.
>
> My preferred fix would be to fix the boot path and /sbin/kexec to ignore
> and report nmis as we boot, as that is really what we want long term and
> it gives us the most robust solution.
>
> The fix with a guarantee of no more scope creep is to just disable the
> nmi watchdog on the kexec on panic path.
>
> Don if you have time please figure out is needed to ignore nmi's and
> possible record and/or report them while we boot, otherwise please cook
> up a patch that just disables the nmi watchdog wherever we are sending
> it from (the local apic or the ioapic).
>
> Eric

A few days ago I investigted the case where system is reseted due to
triple fault caused by the NMI after idt is disabled in
machine_kexec. I didn't see the reset when trigering the kdump with
NMI since the NMI is masked until next iret instruction executed as
described in 6.7.2. Handling Multiple NMIs of Intel Manual Vol.3A.
The NMI mask remains untill the first iret execution on the 2nd
kernel: just the return path of the first kernel_thread invocation for
init process. The exact path is:

  switch_to
  -> ret_from_fork
     -> int_ret_from_sys_call
        -> retint_restore_args
           -> irq_return

At that phase idt is already set up and kdump works.

>From the discussion I interpret kdump doesn't assume this behaviour,
right?

BTW, does anyone know the detail of the NMI mask? I couldn't figure
out about it from the Intel spec more than ``certain hardware
conditions''... I expect those who look at here are x86 NMI experts.

Thanks.
HATAYAMA, Daisuke