[Crash-utility] crash read error: kernel virtual address / vmcore address mismatch ?

Dave Anderson anderson at redhat.com
Wed May 8 09:29:51 EDT 2013

----- Original Message -----
> On Wed, May 8, 2013 at 11:23 AM, Anand Raj Manickam <anandrm at gmail.com>
> wrote:
> >> Hi ,
> >> Sorry about re posting this as i did not find solution ...
> >>
> >> I m facing a issue where on
> >>
> >> #crash /data/linux- /proc/vmcore
> >
> > Do yourself a favor and copy /proc/vmcore to somewhere on disk.  Then
> > reboot the system into the original kernel, and run crash on the saved
> > vmcore.  I've never seen crash run on a /proc/vmcore file from the
> > secondary kernel.
> >
> > Also, after rebooting the the original kernel, please confirm that
> > crash runs OK on the live system.
> >
> >
> >> crash 6.1.6
> >> Copyright (C) 2002-2013  Red Hat, Inc.
> >> Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
> >> Copyright (C) 1999-2006  Hewlett-Packard Co
> >> Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
> >> Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
> >> Copyright (C) 2005, 2011  NEC Corporation
> >> Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
> >> Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
> >> This program is free software, covered by the GNU General Public License,
> >> and you are welcome to change it and/or distribute copies of it under
> >> certain conditions.  Enter "help copying" to see the conditions.
> >> This program has absolutely no warranty.  Enter "help warranty" for
> >> details.
> >>
> >> GNU gdb (GDB) 7.3.1
> >> Copyright (C) 2011 Free Software Foundation, Inc.
> >> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> >> This is free software: you are free to change and redistribute it.
> >> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> >> and "show warranty" for details.
> >> This GDB was configured as "i686-pc-linux-gnu"...
> >>
> >> crash: read error: kernel virtual address: c127a9c8  type: "cpu_possible_mask"
> >
> > Please show the full output of "crash -d8 vmlinux vmcore".
> After rebooting to the primary / live kernel
>    I have attached the output for "crash -d8 vmlinux /data/vmcore" in d8.txt

Looking at the ELF header data in the debug output, your configuration 
does require special handling.  X86 kernels are typically not configured
when that is done, the vmlinux unity-mapped symbol values do not match
their relocated values when the kernel is loaded.  This entry in the 
crash.changelog describes the situation:


4.0-4.5  - Addresses FC7/upstream x86 kernels that have been configured such
           that the vmlinux symbol values do not match their relocated values
           when loaded.  If CONFIG_PHYSICAL_START contains a value that is
           greater then CONFIG_PHYSICAL_ALIGN, then this mismatch occurs.
           Since the crash utility and its embedded gdb have always expected
           that the compiled-in kernel symbol addresses are "real", the virtual
           to physical translation fails, leading to an initialization-time
           failure with the message: "crash: vmlinux and /dev/crash do not
           match!" (/dev/mem or the dumpfile name may replace /dev/crash).
           To deal with this issue, there are several alternatives:

            1) Configure the kernel with CONFIG_PHYSICAL_START less than
               or equal to CONFIG_PHYSICAL_ALIGN.  Having done that, there
               is no problem; the resultant vmlinux file will be loaded at
               the address for which it was compiled, which has always
               been the case.
            2) Since /proc/kallsyms uses the same format as a System.map file,
               and since it reflects the relocated symbol addresses, it
               can be placed on the crash command line as if it were
               a System.map file.  (Note that the System.map file created
               by these relocated kernels contains the same "wrong" symbol
               values as the vmlinux file from which it was created.)
            3) On a live system that has /proc/kallsyms (i.e., the kernel was
               configured with CONFIG_KALLSYMS), this version of the crash
               utility will replace/patch the vmlinux symbol values with those
               seen in /proc/kallsyms.  The relocation value will be displayed
               as a WARNING message during initialization.
            4) On a dumpfile, the relocation will not be performed automatically
               as on a live system.  It will require the addition of the 
               /proc/kallsyms on the command line, or if run on a different
               host, a copy of the crashed system's /proc/kallsyms may be
            5) Alternatively on a dumpfile, a new command line option has been
               created to specify the relocation amount.  For example, if a  
               kernel was configured with a CONFIG_PHYSICAL_START value of 16MB
               and a CONFIG_PHYSICAL_ALIGN of 4MB, that results in a relocation
               of 12MB.  To specify that, enter "crash --reloc=12m ..." on the
               command line.  (Recall that if crash is run on the live system,
               a WARNING message will specify the relocation amount.)

           Using /proc/kallsyms or a --reloc=[size] as a command line argument
           is similar to using a System.map file, in that it results in the loss
           of the use of line number debug data.  (anderson at redhat.com)


>> My current config for
>> The /proc/kallsyms is diffrent between the System Kernel and Debug Kernel.
>> On the System Kernel it starts from 0x400000 .On the Debug kernel it
>> starts from 0x1000000.

So your configuration exactly matches paragraph 5) above, you should
be able to come up OK by entering either:

 $ crash vmlinux vmcore --reloc=12m


 $ crash vmlinux vmcore /proc/kallsyms

where the /proc/kallsyms is from the *primary* kernel, so option #2
will *not* work when running from the secondary kernel.

You didn't show any output of a live crash session (while running
on the primary kernel), but I believe it should work without using 
the /proc/kallsyms as an argument or using --reloc=12m option.  There
was a further fix for these configurations for running on live systems:


5.1.4    - Fix for RT kernels in which the schedule() function has become a 
           ... [ cut] ...

         - Fix for running against live x86 kernels that were configured with
           CONFIG_PHYSICAL_START containing a value that is greater than its
           CONFIG_PHYSICAL_ALIGN value, and where the first symbol listed by
           /proc/kallsyms is not "_text".  Without the patch, the crash session
           fails during invocation with the error message "crash: vmlinux and
           /dev/mem do not match!" (or "/dev/crash" if applicable).  As a work-
           around, "/proc/kallsyms" can be entered on the command line, or the
           "--reloc=<size>" option could be used, but the fix obviates that
           requirement for live systems.  It should be noted that dumpfiles of
           kernels configured that way still do require that "/proc/kallsyms",
           or a copy of it, or alternatively the "--reloc=<size>" option, to
           be entered on the command line, as detailed in this changelog entry:
           (anderson at redhat.com)

So on the live system on the primary kernel, I believe this should just work:

 $ crash vmlinux

Normally you don't even need to put the vmlinux file on the command line for
running on the live system because it will be "found" automatically unless
you've put it in some non-standard location.


More information about the kexec mailing list