[tip:x86/debug] x86/kdump: No need to disable ioapic/ lapic in crash path
Ingo Molnar
mingo at elte.hu
Sun Feb 12 06:12:50 EST 2012
* Yinghai Lu <yinghai at kernel.org> wrote:
> On Sat, Feb 11, 2012 at 3:09 PM, tip-bot for Don Zickus
> <dzickus at redhat.com> wrote:
> > Commit-ID: d9bc9be89629445758670220787683e37c93f6c1
> > Gitweb: http://git.kernel.org/tip/d9bc9be89629445758670220787683e37c93f6c1
> > Author: Don Zickus <dzickus at redhat.com>
> > AuthorDate: Thu, 9 Feb 2012 16:53:41 -0500
> > Committer: Ingo Molnar <mingo at elte.hu>
> > CommitDate: Sat, 11 Feb 2012 15:38:53 +0100
> >
> > x86/kdump: No need to disable ioapic/lapic in crash path
> >
> > A customer of ours noticed when their machine crashed, kdump did
> > not work but hung instead. Using their firmware dumping
> > solution they grabbed a vmcore and decoded the stacks on the
> > cpus. What they noticed seemed to be a rare deadlock with the
> > ioapic_lock.
> >
> > CPU4:
> > machine_crash_shutdown
> > -> machine_ops.crash_shutdown
> > -> native_machine_crash_shutdown
> > -> kdump_nmi_shootdown_cpus ------> Send NMI to other CPUs
> > -> disable_IO_APIC
> > -> clear_IO_APIC
> > -> clear_IO_APIC_pin
> > -> ioapic_read_entry
> > -> spin_lock_irqsave(&ioapic_lock, flags)
> > ---Infinite loop here---
> >
> > CPU0:
> > do_IRQ
> > -> handle_irq
> > -> handle_edge_irq
> > -> ack_apic_edge
> > -> move_native_irq
> > -> mask_IO_APIC_irq
> > -> mask_IO_APIC_irq_desc
> > -> spin_lock_irqsave(&ioapic_lock, flags)
> > ---Receive NMI here after getting spinlock---
> > -> nmi
> > -> do_nmi
> > -> crash_nmi_callback
> > ---Infinite loop here---
> >
> > The problem is that although kdump tries to shutdown minimal
> > hardware, it still needs to disable the IO APIC. This requires
> > spinlocks which may be held by another cpu. This other cpu is
> > being held infinitely in an NMI context by kdump in order to
> > serialize the crashing path. Instant deadlock.
> >
> > Eric brought up a point that because the boot code was
> > restructured we may not need to disable the io apic any more in
> > the crash path. The original concern that led to the
> > development of disable_IO_APIC, was that the jiffies calibration
> > on boot up relied on the PIT timer for reference. Access to the
> > PIT required 8259 interrupts to be working. This wouldn't work
> > if the ioapic needed to be configured. So on panic path, the
> > ioapic was reconfigured to use virtual wire mode to allow the 8259 to passthrough.
> >
> > Those concerns don't hold true now, thanks to the jiffies
> > calibration code not needing the PIT. As a result, we can
> > remove this call and simplify the locking needed in the panic
> > path.
> >
> > The same work allowed us to remove the need to disable the local
> > apic on shutdown too. This should allow us to jump to the
> > second a little faster.
> >
> > I tested kdump on an Ivy Bridge platform, a Pentium4 and an old
> > athlon that did not have an ioapic. All three were successful.
> >
> > I also tested using lkdtm that would use jprobes to panic the
> > system when entering do_IRQ. The idea was to see how the system
> > reacted with an interrupt pending in the second kernel. My
> > core2 quad successfully kdump'd 3 times in a row with no issues.
> >
> > v2: removed the disable lapic code too
>
> with this commit, kdump is not working anymore on my setups with
> Nehalem, Westmere, sandbridge.
> these setup all have VT-d enabled.
>
> After reverting this commit, kdump is working again.
>
> So assume you need to drop this patch.
Dropped the patch, thanks for reporting this.
Don, Eric?
Ingo
More information about the kexec
mailing list