Capturing crash with 4.6.0 and above kernel does not work

[Baoquan He]

On 08/26/16 at 04:47pm, Himanshu Madhani wrote:
> 
> 
> On 8/25/16, 10:39 PM, "Baoquan He" <bhe at redhat.com> wrote:
> 
> >On 08/25/16 at 05:45pm, Himanshu Madhani wrote:
> >> 
> >
> >crashkernel has been reserved successfully.
> >Aug 25 10:36:44 dut4062 kernel: Reserving 512MB of memory at 368MB for crashkernel (System RAM: 130847MB)
> >
> >So what's the result of executing "grep Crash /porc/iomem" on your 
> >system?
> 
> root at dut4062:~# grep Crash /proc/iomem
>   17000000-36ffffff : Crash kernel
> 
> Which does indicate that 512MB is set aside for crash dump

Please get a kexec-tools code and debug it. The message is printed in
main() of kexec.

git://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git

It should be easy to add debug printing or gdb. Now the latest is
2.0.13. What's your version?

> 
> 
> >
> >> 
> >> On 8/25/16, 1:10 AM, "Michal Hocko" <mhocko at kernel.org> wrote:
> >> 
> >> >[Let's add kdump people]
> >> >
> >> >On Wed 24-08-16 16:38:56, Himanshu Madhani wrote:
> >> >> Hello list,
> >> >> 
> >> >> I am wondering if anybody has issue capturing crash dump with the 4.6.0 and above kenrel.
> >> >> 
> >> >> I have a system, when booted in 4.5.7 kernel is able to capture crash dump.
> >> >> However, when I boot this system in 4.6.4 and 4.7.2 kernel, crash dump is not 
> >> >> able to capture any crash. 
> >> >> 
> >> >> I am still facing same issue with 4.8.0-rc2+ kernel and from the error at the command prompt, 
> >> >> it seems like kexec is ignoring “crashkenrel” parameter. 
> >> >> 
> >> >> I added below information in https://bugzilla.kernel.org/show_bug.cgi?id=119291. 
> >> >> 
> >> >> # uname -r
> >> >> 4.8.0-rc2+
> >> >> 
> >> >> # cat /proc/cmdline 
> >> >> ro root=/dev/mapper/vg_dut4110-lv_root rd_NO_LUKS  KEYBOARDTYPE=pc KEYTABLE=us LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=512M rd_LVM_LV=vg_dut4110/lv_swap rd_LVM_LV=vg_dut4110/lv_root rd_NO_DM rhgb quiet
> >> >> 
> >> >> # service kdump status
> >> >> Kdump is not operational
> >> >> 
> >> >> # service kdump start
> >> >> Memory for crashkernel is not reserved
> >> >> Please reserve memory by passing "crashkernel=X at Y" parameter to the kernel
> >> >> Starting kdump:                                            [FAILED]
> >> >
> >> >It smells like the crash kernel reservation has failed. Could you
> >> >provide the full kernel log?
> >> 
> >> Attached is kernel log from fresh kernel compile of 4.6.5 (linux-stable) tree
> >> 
> >> root at dut4062:~# cat /proc/cmdline 
> >> ro root=/dev/mapper/VolGroup-lv_root rd_NO_LUKS LANG=en_US.UTF-8 rd_NO_MD rd_LVM_LV=VolGroup/lv_swap SYSFONT=latarcyrheb-sun16 crashkernel=512M rd_LVM_LV=VolGroup/lv_root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb console=ttyS0 console=ttyS0,115200n8
> >> 
> >> root at dut4062:~# service kdump status
> >> 
> >> Kdump is not operational
> >> 
> >> root at dut4062:~# service kdump start
> >> Memory for crashkernel is not reserved
> >> Please reserve memory by passing "crashkernel=X at Y" parameter to the kernel
> >> Starting kdump:                                            [FAILED]
> >> root at dut4062:~# 
> >> 
> >> Messages in log fil
> >> 
> >> Aug 25 10:42:16 dut4062 kdump: kexec: failed to load kdump kernel
> >> Aug 25 10:42:16 dut4062 kdump: failed to start up
> >> 
> >> 
> >> root at dut4062:~# kexec --ver
> >> kexec-tools 2.0.0 released 19th July 2008
> >> root at dut4062:~# 
> >> 
> >> root at dut4062:~# crash --version
> >> 
> >> crash 7.1.5++
> >> Copyright (C) 2002-2016  Red Hat, Inc.
> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> >> Copyright (C) 1999-2006  Hewlett-Packard Co
> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> >> Copyright (C) 2005, 2011  NEC Corporation
> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> >> This program is free software, covered by the GNU General Public License,
> >> and you are welcome to change it and/or distribute copies of it under
> >> certain conditions.  Enter "help copying" to see the conditions.
> >> This program has absolutely no warranty.  Enter "help warranty" for details.
> >>  
> >> GNU gdb (GDB) 7.6
> >> Copyright (C) 2013 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "x86_64-unknown-linux-gnu".
> >> 
> >> Let me know if any other details are needed. I have multiple servers which exhibit same issues.
> >> 
> >> 
> >> 
> >> 
> >> 
> >> 
> >> >
> >> >> message file confirms that kexec was not able to start the service 
> >> >> 
> >> >> Aug 15 10:41:17 dut4110 kdump: kexec: failed to load kdump kernel
> >> >> Aug 15 10:41:17 dut4110 kdump: failed to start up
> >> >> 
> >> >> Note, that same option is able to load kdump service for 4.5.7 kernel.  
> >> >> 
> >> >> I can provide any details needed to help resolve this issue. 
> >> >> 
> >> >> Thanks,
> >> >> - Himanshu
> >> >> 
> >> >
> >> >-- 
> >> >Michal Hocko
> >> >SUSE Labs
> >
> >