/var/log/messages doesn't have crash info when kernel gets panic/oops/crash

Neil Horman nhorman at redhat.com
Tue Jun 16 07:24:25 EDT 2009 

On Mon, Jun 15, 2009 at 08:43:02PM -0800, Dharmosoth Seetharam wrote:
> Hi,
> 
> I have experimented few things and results are below.
> 
> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>         When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
>         and then continue(not switching to dump-saving kernel).
> 
>         What I am looking is - I want both the things to be happen.
>         that is - it should log the appropriate messages in /var/log/messsages and
>         should be able to take kernel dump.
> 
> case 2) In general, When panic has occurred we won't log any type of messages in
>         /var/log/messages ... am I right ?
>         If KEXEC is enabled and dump-saving kernel is loaded then 
>         will switch to it and save the kernel dump.
> 
>         I have added the dump_stack() function call in the panic() function 
>         before calling crash_kexec with few seconds of delay. It is behaving 
>         like randomly (ie. some time stack traces are logged in /var/log/messages and 
>         not for rest of the time)
> 
>         What I am looking is - I want to log the  appropriate messages in /var/log/messages
>         and then switch to dump-saving kernel.
> 
Then what you want to do is, enable kdump, and set your kernel up to
panic_on_oops.  Then, configure kdump to extract the dmesg log from
/proc/vmcore, and write that to a file (either a file you create, or to the end
of /var/log/messages).  Most distributions kdump configurations allow for
something simmilar to this.  The new config method I've got in rawhide should
make this very easy.

The problem is that when you panic the box (which is what has to happen for
kdump to start), the syslog daemon doesn't get a chance to run first, so theres
no way to write the oops backtrace to var/log/messages.  You'll need to tell
kdump to do that for you after the fact.

> case 3) In general, When die/nmi_die called we used to log the appropriate messages in /var/log/message
>         and then saving kernel dump will be depends on KEXEC enable and loading dump-saving kernel.
>         --- am I right ?
>         
>         I have added the delay in die() function with few seconds before calling crash_kexec()
>         But I never seen messages logged in /var/log/messages, I can see those on serial console.
> 
See above, when we printk those messages, klogd/syslogd needs to run to get them
from the console buffer and write them to /var/log/messages.  Theres no
guarantee that will ever happen.

>         Here also my intention is want both, ie. log the appropriate messages in /var/log/messages
>         and then take the action as per KEXEC enabled.
> 
> 
> 
> Please advice me, What we can do to achieve those things.
> 
See above, use makedumpfile to extract the dmesg log from /proc/vmcore in kdump,
then use a custom script to append the log to /var/log/messages.  In fedora that
will be equivalent to using the core_collector and kdump_post options in
/etc/kdump.conf.

Neil

> Thanks for helping me here.
> 
> 
> regards,
> Seetharam
> 
> 
> 
> 
> 
> > -----Original Message-----
> > From: nhorman at redhat.com
> > Sent: Sat, 13 Jun 2009 14:07:36 -0400
> > To: dseetharam at inbox.com
> > Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> > panic/oops/crash
> > 
> > On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
> >> 
> >> Hi Neil,
> >> 
> >> Thanks for your quick reply.
> >> 
> >> What you have suggested is nice when we got/save crash dump (i.e vmcore)
> >> What if we failed to save/capture vmcore info or vmcore is corrupted
> >> etc. then we dont get any information
> >> regarding the panic/crash/oops at anywhere?
> >> 
> > If the vmcore is corrupted to the point where makedumpfile can't extract
> > the
> > dmesg log, then its certain that syslog wouldn't have been able to write
> > anything to /var/log/messages either.  As a last resort you can try run
> > strings
> > on the corrupted vmcore and see if the dmesg log is still in there
> > somewhere.
> > 
> > If you forgot to capture the vmcore, theres not much more help I can give
> > you,
> > Im afraid.  You'll just have to make sure you capture it next time.  If
> > you only
> > need the backtrace, and don't want the rest of the vmcore, you can make
> > kdump
> > just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
> > 
> > Neil
> > 
> 
> 
> >> 
> >> 
> >> 
> >> Thanks and Regards
> >> 
> >>  Seetharam
> >> 
> >> 
> >>> -----Original Message-----
> >>> From: nhorman at redhat.com
> >>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
> >>> To: dseetharam at inbox.com
> >>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> >>> panic/oops/crash
> >>> 
> >>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
> >>>> Hi,
> >>>> 
> >>>> I have encounter a problem called /var/log/messages does not contain
> >>>> any
> >>>> information
> >>>> on kernel crash dump.
> >>>> 
> >>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
> >>>> /var/log/messages.
> >>>> After enabling KEXEC feature those info was missed.
> >>>> 
> >>>> How can we get those missed info in /var/log/messages even when KEXEC
> >>>> feature is enabled.
> >>>> 
> >>>> Any help will appreciate.
> >>>> 
> >>> run crash on the resultant vmcore file that kexec can produce/save and
> >>> get the
> >>> dmesg log out of it.
> >>> alternatively you can use makedumpfile with the --dump-dmesg log
> >>> 
> >>> Neil
> >>> 
> >>>> 
> >>>> Thanks and Regards
> >>>> Seetharam
> >>>> 
> >>>> ____________________________________________________________
> >>>> Receive Notifications of Incoming Messages
> >>>> Easily monitor multiple email accounts & access them with a click.
> >>>> Visit http://www.inbox.com/notifier and check it out!
> >>>> 
> >>>> _______________________________________________
> >>>> kexec mailing list
> >>>> kexec at lists.infradead.org
> >>>> http://lists.infradead.org/mailman/listinfo/kexec
> >> 
> >> ____________________________________________________________
> >> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
> >> http://www.inbox.com/smileys
> >> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
> >> and most webmails

More information about the kexec mailing list