Kexec & Memory Zones question

[Vivek Goyal]

On Wed, May 04, 2011 at 11:35:46AM -0700, Sujit V wrote:
> On our x86_64 NUMA hardware running linux 2.6.23 with two memory nodes
>  have the following zone layout
> DMA   0 - 16MB
> DMA32   16MB to 4GB
> NORMAL 4GB to 96GB
> 
> We had the crashkernel boot param as 128M at 16M.
> I am using kexec-tools-2.0.0
> 
> We observed that the reserved crashkernel memory was getting used by
> the system.
> We found out by kdb  that around 17MB of physical memory the memory
> contents were changing which proved that the system was
> using the memory which is actually reserved for crash kernel.
> [ If I load crash kernel using kexec then the system would crash. The
> back trace was always in the megasas driver. ]

This is problematic. This memory has been reserved by kernel in early
boot and nobody else should be using this memory. This sounds like a
but somewhere.

> 
> My Question is
> 1) Should the crashkernel memory be located past the DMA32 zone.?
> 
> I have tried the following
> (1) crashkernel=128M at 4GB  ( So the memory reservation is past DMA32)
> In this scenario the kexec tools gave an error "Could not find a free
> area of memory of xyz bytes"
> 
> (2) I changed the max of DMA32 to 1GB
>      crashkernel=128M at 1G
> Still kexec gave the same error "Could not find a free area of memory
> of xyz bytes"

We have discussed this in the past and due to various reasons the max
amount of RAM you can boot your kernel from seems to be 896MB for
x86_64 and 512MB for 32bit. I shall have to open a previous thread
with hpa to get exact numbers. So loading kernel even higher is not
the solution.

We need to figure out who is using this memory and where is the bug
and fix it.

Thanks
Vivek