VMCore from makedumpfile -d 2 Failed to be Read by Crash

[Dave Anderson]

----- "Dave Anderson" <anderson at redhat.com> wrote:

> ----- "Neil Horman" <nhorman at redhat.com> wrote:
> 
> > On Sat, Oct 10, 2009 at 10:56:32AM +0800, CAI Qian wrote:
> > > Hallo!
> > >
> > > The crash utility failed to initialize when read the VMCore from
> > > makedumpfile -d 2.
> > >
> > > # crash -d 7 vmlinux vmcore.2
> > >
> > > <readmem: ffffffff818ba450, KVADDR, "__per_cpu_offset", 32768, (FOE), a31fc8>
> > > GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer i use)
> > > GNU_GET_DATATYPE[runqueue]: returned via gdb_error_hook (1 buffer in use)
> > > GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
> > > GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
> > > GNU_GET_DATATYPE[prio_array]: returned via gdb_error_hook (1 buffer in use)
> > > <readmem: ffff88000c60e2f8, KVADDR, "cpu number (per_cpu)", 4, (FOE), 7fff04f30c1c>
> > > <readmem: ffff88000c7ed2f8, KVADDR, "cpu number (per_cpu)", 4, (FOE), 7fff04f30c1c>
> > > <readmem: ffffffff818d92f8, KVADDR, "cpu number (per_cpu)", 4, (FOE), 7fff04f30c1c>
> > > crash: page excluded: kernel virtual address: ffffffff818d92f8 type: "cpu number (per_cpu)"
> > -d 2 is exclude cached pages, I don't rightly see how any of these data structures
> > can be in pagecache.  Not sure if that means that makedumpfile filtered them
> > inappropriately, or if crash is looking for them in the wrong place.
> >
> > Neil
> 
> Given that the full vmcore shows that there are 2 cpus, and that the
> failure above is on an attempt to read/verify a 3rd cpu, it's pretty
> clear that this is a crash bug.  There have been a few of these type
> of bugs that have recently been discovered due to the movement of the
> the x8664_pda data structures to per-cpu data variables in 2.6.30.
> Unfortunately, the x86_64_per_cpu_init() is trying to figure out
> how many cpus are there, and it's currently doing a "read-and-verify"
> kind of thing, but will need to be changed to recognize a readmem failure
> as the end of the line.
> 
> I'll be back on Tuesday -- Cai, can you contact me offline and
> make the vmcore/vmlinux pair available?

This will be fixed in the next upstream crash utility release.

In the meantime, "page excluded" errors can be worked around by using 
the --zero_excluded command line option, as in:

 # crash --zero_excluded vmlinux vmcore.2

Dave