kexec: purgatory hang

Wed Jun 12 12:40:00 EDT 2013

On Tue, Jun 11, 2013 at 06:24:24PM -0700, Eric W. Biederman wrote:
> Cliff Wickman <cpw at sgi.com> writes:
> 
> > I'm getting a hang when trying to enter a high-memory crash kernel,
> > and I'm at a loss as to how to debug this.
> >
> > This is a 3.10.0-rc3 kernel, and set up as the crash kernel by kexec 2.0.4.
> > The machine is an SGI UV1000.
> >
> > [  164.027275] SysRq : Trigger a crash
> > [  164.031136] BUG: unable to handle kernel NULL pointer dereference at           (null)
> > [  164.031136] IP: [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [  164.031136] PGD 1fbe835067 PUD 1fbc2e8067 PMD 0 
> > [  164.031136] Oops: 0002 [#1] SMP 
> > [  164.031136] xpc : all partitions have deactivated
> > [  164.031136] Modules linked in: autofs4 binfmt_misc af_packet rdma_ucm rdma_cm iw_cm ib_addr ib_srp scsi_transport_srp scsi_tgt ib_ipoib ib_cm ib_uverbs ib_umad iw_cxgb3 cxgb3 mdio mlx4_en mlx4_ib ib_sa mlx4_core ib_mthca ib_mad ib_core fuse nls_iso8859_1 nls_cp437 vfat fat loop uv_mmtimer dm_mod sr_mod cdrom usb_storage iTCO_wdt iTCO_vendor_support coretemp mperf kvm_intel ipv6 kvm igb sg crc32c_intel lpc_ich pcspkr mptctl i2c_algo_bit ptp i2c_i801 microcode xhci_hcd joydev ioatdma ehci_pci hid_generic pps_core i2c_core rtc_cmos mfd_core button dca usbhid hid uhci_hcd ehci_hcd usbcore usb_common sd_mod crc_t10dif scsi_dh_hp_sw scsi_dh_alua scsi_dh_emc scsi_dh_rdac scsi_dh thermal sata_nv processor piix mptsas mptscsih scsi_transport_sas mptbase megaraid_sas fan thermal_sys hwmon ext3 jbd ata_piix ahci libahci libata scsi_mod
> > [  164.031136] CPU: 10 PID: 9299 Comm: dopanic Not tainted 3.10.0-rc3-linus-cpw+ #17
> > [  164.031136] Hardware name: Intel Corp. Stoutland Platform, BIOS 2.16 UEFI2.10 PI1.0 X64 2012-04-27
> > [  164.031136] task: ffff88203df94440 ti: ffff88203d5c2000 task.ti: ffff88203d5c2000
> > [  164.031136] RIP: 0010:[<ffffffff81397771>]  [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [  164.031136] RSP: 0018:ffff88203d5c3e68  EFLAGS: 00010092
> > [  164.031136] RAX: 000000000000000f RBX: ffffffff81a974e0 RCX: 0000000000000004
> > [  164.031136] RDX: 0000000000000000 RSI: ffff881fffd0ef48 RDI: 0000000000000063
> > [  164.031136] RBP: ffff88203d5c3e68 R08: ffff881fffd0d3e8 R09: 000000000004268c
> > [  164.031136] R10: 0000000000000b8b R11: 0000000000000000 R12: 0000000000000063
> > [  164.031136] R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000296
> > [  164.031136] FS:  00007ffff7fb5700(0000) GS:ffff881fffd00000(0000) knlGS:0000000000000000
> > [  164.031136] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  164.031136] CR2: 0000000000000000 CR3: 0000001fbea6c000 CR4: 00000000000007e0
> > [  164.031136] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [  164.031136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [  164.031136] Stack:
> > [  164.031136]  ffff88203d5c3ea8 ffffffff81398008 01ff88203d5c3e88 0000000000000002
> > [  164.031136]  ffff895f9d478380 ffff88203d5c3f40 00007ffff7ff8000 ffff88203d5c3f40
> > [  164.031136]  ffff88203d5c3ec8 ffffffff813980ad ffff88203d5c3ee8 fffffffffffffffb
> > [  164.031136] Call Trace:
> > [  164.031136]  [<ffffffff81398008>] __handle_sysrq+0x128/0x190
> > [  164.031136]  [<ffffffff813980ad>] write_sysrq_trigger+0x3d/0x40
> > [  164.031136]  [<ffffffff811c323f>] proc_reg_write+0x4f/0x80
> > [  164.031136]  [<ffffffff8115f107>] vfs_write+0xe7/0x190
> > [  164.031136]  [<ffffffff8115f8ec>] SyS_write+0x5c/0xa0
> > [  164.031136]  [<ffffffff8153c092>] system_call_fastpath+0x16/0x1b
> > [  164.031136] Code: 00 48 8b 75 e8 48 81 c7 08 08 00 00 e8 09 c6 19 00 31 d2 eb 95 90 90 90 90 90 55 c7 05 f5 74 96 00 01 00 00 00 48 89 e5 0f ae f8 <c6> 04 25 00 00 00 00 01 c9 c3 0f 1f 44 00 00 8d 47 d0 55 83 f8 
> > [  164.031136] RIP  [<ffffffff81397771>] sysrq_handle_crash+0x11/0x20
> > [  164.031136]  RSP <ffff88203d5c3e68>
> > [  164.031136] CR2: 0000000000000000
> >
> > This is always the last output.
> >
> > Can anyone suggest any way to debug this problem?  
> >
> > I suppose I can hang the processor just before it executes machine_kexec()
> > and look at it with crash.  Any suggestions as to what to look at?

Hi Eric,
Thanks for the reply.
> Hmm.  You can enable print statements in purgatory.c.  There is a
> command line switch that allows pugatory to print to a serial console.
> That should be a simple easy thing to try.

Do you recall what that switch is?  I don't see any condition in
purgatory.c.  But the existing printf's don't give me anything.

> I am totally lost as to the status of the patches to make all of this
> work right.  But the change to let purgator work above 4G was merged
> early so hopefully it is not a problem in kexec.
It works on a UV2000 or a whitebox, so it must be close.
> 
> You might also want to enable early printk in the crash dump kernel.
> Sometimes kernels get confused on the way up and we hang there.
Yes, I'm using earlyprintk.

-Cliff
-- 
Cliff Wickman
SGI
cpw at sgi.com
(651) 683-3824