[Crash-utility] "cannot access vmalloc'd module memory" when loading kdump'ed vmcore in crash
Dave Anderson
anderson at redhat.com
Mon Oct 6 15:39:05 EDT 2008
----- "Kevin Worth" <kevin.worth at hp.com> wrote:
> Dave,
>
> That does seem pretty strange that the physical address is coming out
> beyond the 4GB mark and that the read actually succeeds. Just checked
> on the Ubuntu patches to the 2.6.20 kernel (
> http://archive.ubuntu.com/ubuntu/pool/main/l/linux-source-2.6.20/linux-source-2.6.20_2.6.20-17.39.diff.gz
> ) and no mention of mem.c or either of those two functions.
Hmmm -- I do see one thing with the /dev/mem driver that could
be an explanation. Maybe...
Prior to the read() call to /dev/mem, crash does an llseek() to
the target physical address, which gets stored in the open file
structure's file.f_pos member, which is a 64-bit loff_t. Then when
the subsequent read() call is made, the file.f_pos member gets
passed by reference to the /dev/mem driver's read_mem() function
via the "ppos" argument:
static ssize_t read_mem(struct file * file, char __user * buf,
size_t count, loff_t *ppos)
{
unsigned long p = *ppos;
ssize_t read, sz;
char *ptr;
if (!valid_phys_addr_range(p, count))
return -EFAULT;
But its value is then pulled from *ppos into a 32-bit unsigned long
"p" variable, which is what gets used from then on. So it looks like
the high 1-bit from a greater-than-4GB (0x100000000) physical address
would get stripped, and therefore would erroneously bypass the
valid_phys_addr_range() check.
So in your case, physical addresses from ~3GB-up-to-4GB would
be rejected, but those at and above 4GB would be inadvertently
accepted. However, if that were the case, the *wrong* physical address
would be accessed -- but your "module" reads seemingly return the correct
data! So I still don't get it...
I haven't tinkered with the 32-bit /dev/mem driver in years, because
Red Hat not only has the "high_memory" restriction, it also has a
devmem_is_allowed() function that further restricts /dev/mem to the
first 256 pages (1MB) of physical memory. (I note that upstream kernels
have recently added a CONFIG_STRICT_DEVMEM config option to do the same
thing.) And, FYI, the Red Hat /dev/crash "replacement-for-/dev/mem" driver
correctly reads *ppos into a u64.
So when you test this again on your live system, after printing the
module via "p <virtual-address-of-module>", do a vtop of the
<virtual-address-of-module>, take the translated-to physical address
and dump it to verify the contents. Like this:
crash> p modules
modules = $2 = {
next = 0xf8bf5904,
prev = 0xf8836004
}
crash> module 0xf8bf5900
struct module {
state = MODULE_STATE_LIVE,
list = {
next = 0xf8a60d84,
prev = 0xc06787b0
},
name = "crash"
mkobj = {
kobj = {
k_name = 0xf8bf594c "crash",
name = "crash",
kref = {
refcount = {
counter = 2
}
},
...
crash> vtop 0xf8bf5900
VIRTUAL PHYSICAL
f8bf5900 2412c900
...
crash> rd -p 2412c900 30
2412c900: 00000000 f8a60d84 c06787b0 73617263 ..........g.cras
2412c910: 00000068 00000000 00000000 00000000 h...............
2412c920: 00000000 00000000 00000000 00000000 ................
2412c930: 00000000 00000000 00000000 00000000 ................
2412c940: 00000000 00000000 f8bf594c 73617263 ........LY..cras
2412c950: 00000068 00000000 00000000 00000000 h...............
2412c960: 00000002 c06783e8 f8a60de4 c06783f4 ......g.......g.
2412c970: c06783e0 00000000 ..g.....
crash>
Lastly, try this set of crash commands on your live system:
rd -p 0
rd -p 0x20000000
rd -p 0x40000000
rd -p 0x60000000
rd -p 0x80000000
rd -p 0xa0000000
rd -p 0xb8000000
rd -p 0xc0000000
rd -p 0xe0000000
rd -p 0x100000000
rd -p 0x120000000
rd -p 0x140000000
Theoretically, anything at and above 0xb8000000 should fail.
> Let me try the kexec PAGE_OFFSET modification today or tomorrow and
> reply back on how it goes. If that produces no change I'll try do a
> re-run of the previous email's process with some more careful
> attention paid (that I get a vtop of everything and that my context
> examples are the same process).
OK fine...
Thanks,
Dave
More information about the kexec
mailing list