896MB address limit

[Cliff Wickman]

On Tue, Sep 25, 2012 at 08:10:04AM -0700, Eric W. Biederman wrote:
> Cliff Wickman <cpw at sgi.com> writes:
> 
> > Hi Eric, and all,
> >
> > On Mon, Sep 24, 2012 at 08:11:12PM -0700, Eric W. Biederman wrote:
> >> Cliff Wickman <cpw at sgi.com> writes:
> >> 
> >> > Gentlemen,
> >> >
> >> > In dumping very large memories we are running up against the 896MB 
> >> > limit in SLES11SP2 (3.0.38 kernel).
> >> 
> >> Odd.  That limit should be the maximum address in memory to load the
> >> crash kernel.  Tha limit should have nothing to do with the dump process
> >> itself.
> >> 
> >> Are you saying you need more that 512MiB reserved for the crash kernel
> >> to be able to dump all of the memory in your system?
> >> 
> >> Eric
> >
> > As I noted to Eric privately, yes we need to bump up to crashkernel=1G
> > or more for some very large memories.
> >
> > As an experiment I bumped
> > +++ linux/arch/x86/kernel/setup.c
> > @@ -528,7 +528,7 @@ static inline unsigned long long get_tot
> >  #ifdef CONFIG_X86_32
> >  # define CRASH_KERNEL_ADDR_MAX (512 << 20)
> >  #else
> > -# define CRASH_KERNEL_ADDR_MAX (896 << 20)
> > +# define CRASH_KERNEL_ADDR_MAX (1700 << 20)
> >
> > And that seems to work.  i.e. I'm currently dumping a system where
> > crashkernel=1G and it seems to be working.
> >
> > Am I just living dangerously? 
> 
> So fundamentally this should work.  However there have been a lot of
> kinks and silly limitations in the x86 boot protocol.
> 
> So it used to be that the bootloader protocol variable ramdisk_max was
> set to 896M for 32bit kernels.  Because the ramdisk could not be located
> in high memory.
> 
> Looking today it appears that ramdisk_max has been upped to 4G.
> 
> I will let you look through the /sbin/kexec source code.
> 
> As for testing I would up the limit to 4G on x86_64 and see how far
> you get.
> 
> The practical question does the system still work with crashkernel=32M
> when you have raised the limit much higher. 
> 
> So I would test with crashkernel=1G at 2G and see if that works.  If that
> works I figure that in practice all of the bugs are historical and we
> can forget them.  But a sweep through the /sbin/kexec code for the magic
> number 896 might not be out of order.
> 
> Eric

I did try setting the limit to 8G. The crashkernel did get loaded there
but it would not execute there.

It works fine on a UV to set the limit to 4G and use a
crashkernel=1280M.  We have a hole of almost 2G there.

The memory at 2G is already in use so I can't explicitly place it there.

The kernel patch looks like this:
Index: linux/arch/x86/kernel/setup.c
===================================================================
--- linux.orig/arch/x86/kernel/setup.c
+++ linux/arch/x86/kernel/setup.c
@@ -522,13 +522,12 @@ static inline unsigned long long get_tot
 /*
  * Keep the crash kernel below this limit.  On 32 bits earlier kernels
  * would limit the kernel to the low 512 MiB due to mapping
  * restrictions.
- * On 64 bits, kexec-tools currently limits us to 896 MiB; increase
  this
- * limit once kexec-tools are fixed.
+ * On 64 bits, the boot protocol limits us to 4G.
  */
 #ifdef CONFIG_X86_32
 # define CRASH_KERNEL_ADDR_MAX (512 << 20)
 #else
-# define CRASH_KERNEL_ADDR_MAX (896 << 20)
+# define CRASH_KERNEL_ADDR_MAX (1UL << 32)
 #endif

 static void __init reserve_crashkernel(void)

-Cliff
-- 
Cliff Wickman
SGI
cpw at sgi.com
(651) 683-3824