[Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?
Dave Anderson
anderson at redhat.com
Wed May 8 09:29:51 EDT 2013
----- Original Message -----
> On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm at gmail.com>
> wrote:
> >> Hi ,
> >> Sorry about re posting this as i did not find solution ...
> >>
> >> I m facing a issue where on
> >>
> >> #crash /data/linux-2.6.30.8/vmlinux /proc/vmcore
> >
> > Do yourself a favor and copy /proc/vmcore to somewhere on disk. Then
> > reboot the system into the original kernel, and run crash on the saved
> > vmcore. I've never seen crash run on a /proc/vmcore file from the
> > secondary kernel.
> >
> > Also, after rebooting the the original kernel, please confirm that
> > crash runs OK on the live system.
> >
> >
> >> crash 6.1.6
> >> Copyright (C) 2002-2013 Red Hat, Inc.
> >> Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
> >> Copyright (C) 1999-2006 Hewlett-Packard Co
> >> Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
> >> Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
> >> Copyright (C) 2005, 2011 NEC Corporation
> >> Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
> >> Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
> >> This program is free software, covered by the GNU General Public License,
> >> and you are welcome to change it and/or distribute copies of it under
> >> certain conditions. Enter "help copying" to see the conditions.
> >> This program has absolutely no warranty. Enter "help warranty" for
> >> details.
> >>
> >> GNU gdb (GDB) 7.3.1
> >> Copyright (C) 2011 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "i686-pc-linux-gnu"...
> >>
> >> crash: read error: kernel virtual address: c127a9c8 type: "cpu_possible_mask"
> >
> > Please show the full output of "crash -d8 vmlinux vmcore".
>
> After rebooting to the primary / live kernel
>
> I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt
Looking at the ELF header data in the debug output, your configuration
does require special handling. X86 kernels are typically not configured
with CONFIG_PHYSICAL_START larger than CONFIG_PHYSICAL_ALIGN, because
when that is done, the vmlinux unity-mapped symbol values do not match
their relocated values when the kernel is loaded. This entry in the
crash.changelog describes the situation:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
4.0-4.5 - Addresses FC7/upstream x86 kernels that have been configured such
that the vmlinux symbol values do not match their relocated values
when loaded. If CONFIG_PHYSICAL_START contains a value that is
greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
Since the crash utility and its embedded gdb have always expected
that the compiled-in kernel symbol addresses are "real", the virtual
to physical translation fails, leading to an initialization-time
failure with the message: "crash: vmlinux and /dev/crash do not
match!" (/dev/mem or the dumpfile name may replace /dev/crash).
To deal with this issue, there are several alternatives:
1) Configure the kernel with CONFIG_PHYSICAL_START less than
or equal to CONFIG_PHYSICAL_ALIGN. Having done that, there
is no problem; the resultant vmlinux file will be loaded at
the address for which it was compiled, which has always
been the case.
2) Since /proc/kallsyms uses the same format as a System.map file,
and since it reflects the relocated symbol addresses, it
can be placed on the crash command line as if it were
a System.map file. (Note that the System.map file created
by these relocated kernels contains the same "wrong" symbol
values as the vmlinux file from which it was created.)
3) On a live system that has /proc/kallsyms (i.e., the kernel was
configured with CONFIG_KALLSYMS), this version of the crash
utility will replace/patch the vmlinux symbol values with those
seen in /proc/kallsyms. The relocation value will be displayed
as a WARNING message during initialization.
4) On a dumpfile, the relocation will not be performed automatically
as on a live system. It will require the addition of the
/proc/kallsyms on the command line, or if run on a different
host, a copy of the crashed system's /proc/kallsyms may be
used.
5) Alternatively on a dumpfile, a new command line option has been
created to specify the relocation amount. For example, if a
kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
of 12MB. To specify that, enter "crash --reloc=12m ..." on the
command line. (Recall that if crash is run on the live system,
a WARNING message will specify the relocation amount.)
Using /proc/kallsyms or a --reloc=[size] as a command line argument
is similar to using a System.map file, in that it results in the loss
of the use of line number debug data. (anderson at redhat.com)
...
>> My current config for
>>
>> CONFIG_PHYSICAL_START=0x1000000
>> CONFIG_RELOCATABLE=y
>> CONFIG_PHYSICAL_ALIGN=0x400000
>>
>> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
>> On the System Kernel it starts from 0x400000 .On the Debug kernel it
>> starts from 0x1000000.
So your configuration exactly matches paragraph 5) above, you should
be able to come up OK by entering either:
$ crash vmlinux vmcore --reloc=12m
or
$ crash vmlinux vmcore /proc/kallsyms
where the /proc/kallsyms is from the *primary* kernel, so option #2
will *not* work when running from the secondary kernel.
You didn't show any output of a live crash session (while running
on the primary kernel), but I believe it should work without using
the /proc/kallsyms as an argument or using --reloc=12m option. There
was a further fix for these configurations for running on live systems:
http://people.redhat.com/anderson/crash.changelog.html#5_1_4
5.1.4 - Fix for RT kernels in which the schedule() function has become a
... [ cut] ...
- Fix for running against live x86 kernels that were configured with
CONFIG_PHYSICAL_START containing a value that is greater than its
CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by
/proc/kallsyms is not "_text". Without the patch, the crash session
fails during invocation with the error message "crash: vmlinux and
/dev/mem do not match!" (or "/dev/crash" if applicable). As a work-
around, "/proc/kallsyms" can be entered on the command line, or the
"--reloc=<size>" option could be used, but the fix obviates that
requirement for live systems. It should be noted that dumpfiles of
kernels configured that way still do require that "/proc/kallsyms",
or a copy of it, or alternatively the "--reloc=<size>" option, to
be entered on the command line, as detailed in this changelog entry:
http://people.redhat.com/anderson/crash.changelog.html#4_0_4_5
(anderson at redhat.com)
So on the live system on the primary kernel, I believe this should just work:
$ crash vmlinux
Normally you don't even need to put the vmlinux file on the command line for
running on the live system because it will be "found" automatically unless
you've put it in some non-standard location.
Dave
More information about the kexec
mailing list