/var/log/messages doesn't have crash info when kernel gets panic/oops/crash
Dharmosoth Seetharam
dseetharam at inbox.com
Tue Jun 16 07:47:01 EDT 2009
Thanks a lot.
Seetharam
> -----Original Message-----
> From: nhorman at redhat.com
> Sent: Tue, 16 Jun 2009 07:24:25 -0400
> To: dseetharam at inbox.com
> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
> panic/oops/crash
>
> On Mon, Jun 15, 2009 at 08:43:02PM -0800, Dharmosoth Seetharam wrote:
>> Hi,
>>
>> I have experimented few things and results are below.
>>
>> Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
>> When BUG/oops occurred it is able to log the diagnostics to
>> /var/log/messages
>> and then continue(not switching to dump-saving kernel).
>>
>> What I am looking is - I want both the things to be happen.
>> that is - it should log the appropriate messages in
>> /var/log/messsages and
>> should be able to take kernel dump.
>>
>> case 2) In general, When panic has occurred we won't log any type of
>> messages in
>> /var/log/messages ... am I right ?
>> If KEXEC is enabled and dump-saving kernel is loaded then
>> will switch to it and save the kernel dump.
>>
>> I have added the dump_stack() function call in the panic()
>> function
>> before calling crash_kexec with few seconds of delay. It is
>> behaving
>> like randomly (ie. some time stack traces are logged in
>> /var/log/messages and
>> not for rest of the time)
>>
>> What I am looking is - I want to log the appropriate messages
>> in /var/log/messages
>> and then switch to dump-saving kernel.
>>
> Then what you want to do is, enable kdump, and set your kernel up to
> panic_on_oops. Then, configure kdump to extract the dmesg log from
> /proc/vmcore, and write that to a file (either a file you create, or to
> the end
> of /var/log/messages). Most distributions kdump configurations allow for
> something simmilar to this. The new config method I've got in rawhide
> should
> make this very easy.
>
> The problem is that when you panic the box (which is what has to happen
> for
> kdump to start), the syslog daemon doesn't get a chance to run first, so
> theres
> no way to write the oops backtrace to var/log/messages. You'll need to
> tell
> kdump to do that for you after the fact.
>
>> case 3) In general, When die/nmi_die called we used to log the
>> appropriate messages in /var/log/message
>> and then saving kernel dump will be depends on KEXEC enable and
>> loading dump-saving kernel.
>> --- am I right ?
>>
>> I have added the delay in die() function with few seconds before
>> calling crash_kexec()
>> But I never seen messages logged in /var/log/messages, I can see
>> those on serial console.
>>
> See above, when we printk those messages, klogd/syslogd needs to run to
> get them
> from the console buffer and write them to /var/log/messages. Theres no
> guarantee that will ever happen.
>
>> Here also my intention is want both, ie. log the appropriate
>> messages in /var/log/messages
>> and then take the action as per KEXEC enabled.
>>
>>
>>
>> Please advice me, What we can do to achieve those things.
>>
> See above, use makedumpfile to extract the dmesg log from /proc/vmcore in
> kdump,
> then use a custom script to append the log to /var/log/messages. In
> fedora that
> will be equivalent to using the core_collector and kdump_post options in
> /etc/kdump.conf.
>
> Neil
>
>> Thanks for helping me here.
>>
>>
>> regards,
>> Seetharam
>>
>>
>>
>>
>>
>>> -----Original Message-----
>>> From: nhorman at redhat.com
>>> Sent: Sat, 13 Jun 2009 14:07:36 -0400
>>> To: dseetharam at inbox.com
>>> Subject: Re: /var/log/messages doesn't have crash info when kernel gets
>>> panic/oops/crash
>>>
>>> On Fri, Jun 12, 2009 at 11:01:35PM -0800, Dharmosoth Seetharam wrote:
>>>>
>>>> Hi Neil,
>>>>
>>>> Thanks for your quick reply.
>>>>
>>>> What you have suggested is nice when we got/save crash dump (i.e
>>>> vmcore)
>>>> What if we failed to save/capture vmcore info or vmcore is corrupted
>>>> etc. then we dont get any information
>>>> regarding the panic/crash/oops at anywhere?
>>>>
>>> If the vmcore is corrupted to the point where makedumpfile can't
>>> extract
>>> the
>>> dmesg log, then its certain that syslog wouldn't have been able to
>>> write
>>> anything to /var/log/messages either. As a last resort you can try run
>>> strings
>>> on the corrupted vmcore and see if the dmesg log is still in there
>>> somewhere.
>>>
>>> If you forgot to capture the vmcore, theres not much more help I can
>>> give
>>> you,
>>> Im afraid. You'll just have to make sure you capture it next time. If
>>> you only
>>> need the backtrace, and don't want the rest of the vmcore, you can make
>>> kdump
>>> just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
>>>
>>> Neil
>>>
>>
>>
>>>>
>>>>
>>>>
>>>> Thanks and Regards
>>>>
>>>> Seetharam
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: nhorman at redhat.com
>>>>> Sent: Fri, 12 Jun 2009 16:05:13 -0400
>>>>> To: dseetharam at inbox.com
>>>>> Subject: Re: /var/log/messages doesn't have crash info when kernel
>>>>> gets
>>>>> panic/oops/crash
>>>>>
>>>>> On Fri, Jun 12, 2009 at 01:25:49AM -0800, Dharmosoth Seetharam wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have encounter a problem called /var/log/messages does not contain
>>>>>> any
>>>>>> information
>>>>>> on kernel crash dump.
>>>>>>
>>>>>> Prior to enabling KEXEC we saw stack traces and bug info etc in
>>>>>> /var/log/messages.
>>>>>> After enabling KEXEC feature those info was missed.
>>>>>>
>>>>>> How can we get those missed info in /var/log/messages even when
>>>>>> KEXEC
>>>>>> feature is enabled.
>>>>>>
>>>>>> Any help will appreciate.
>>>>>>
>>>>> run crash on the resultant vmcore file that kexec can produce/save
>>>>> and
>>>>> get the
>>>>> dmesg log out of it.
>>>>> alternatively you can use makedumpfile with the --dump-dmesg log
>>>>>
>>>>> Neil
>>>>>
>>>>>>
>>>>>> Thanks and Regards
>>>>>> Seetharam
>>>>>>
>>>>>> ____________________________________________________________
>>>>>> Receive Notifications of Incoming Messages
>>>>>> Easily monitor multiple email accounts & access them with a click.
>>>>>> Visit http://www.inbox.com/notifier and check it out!
>>>>>>
>>>>>> _______________________________________________
>>>>>> kexec mailing list
>>>>>> kexec at lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/kexec
>>>>
>>>> ____________________________________________________________
>>>> GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
>>>> http://www.inbox.com/smileys
>>>> Works with AIM®, MSN® Messenger, Yahoo!® Messenger, ICQ®, Google Talk™
>>>> and most webmails
More information about the kexec
mailing list