32TB kdump

[Cliff Wickman]

I have been testing recent kernel and kexec-tools for doing kdump of large
memories, and found good results.

--------------------------------
UV2000  memory: 32TB  crashkernel=2G at 4G
command line  /usr/bin/makedumpfile --non-cylic -c --message-level 23 -d 31 \
   --map-size 4096 -x /boot/vmlinux-3.10.0-rc5-linus-cpw+ /proc/vmcore \
   /tmp/cpw/dumpfile

page scanning  570 sec.
copying data  5795 sec. (72G)
(The data copy ran out of disk space at 23%, so the time and size above are
 extrapolated.)

--------------------------------
UV1000  memory: 8.85TB  crashkernel=1G at 5G
command line  /usr/bin/makedumpfile --non-cylic -c --message-level 23 -d 31 \
   --map-size 4096 -x /boot/vmlinux-3.9.6-cpw-medusa /proc/vmcore \
   /tmp/cpw/dumpfile

page scanning  175 sec.
copying data  2085 sec. (15G)
(The data copy ran out of disk space at 60%, so the time and size above are
 extrapolated.)

Notes/observations:
- These systems were idle, so this is the capture of basically system
  memory only.
- Both stable 3.9.6 and 3.10.0-rc5 worked.
- Use of crashkernel=1G,high was usually problematic.  I assume some problem
  with a conflict with something else using high memory.  I always use
  the form like 1G at 5G, finding memory by examining /proc/iomem.
- Time for copying data is dominated by data compression.  Writing 15G of
  compressed data to /dev/null takes about 35min.  Writing the same data
  but uncompressed (140G) to /dev/null takes about 6min.
  So a good workaround for a very large system might be to dump uncompressed
  to an SSD.
  The multi-threading of the crash kernel would produce a big gain.
- Use of mmap on /proc/vmcore increased page scanning speed from 4.4 minutes
  to 3 minutes.  It also increased data copying speed (unexpectedly) from
  38min. to 35min.
  So I think it is worthwhile to push Hatayama's 9-patch set into the kernel.
- I applied a 5-patch set from Takao Indoh to fix reset_devices handling of
  PCI devices.
  And I applied 3 kernel hacks of my own:
    - making a "Crash kernel low" section in /proc/iomem
    - make crashkernel avoid some things in pci_swiotlb_detect_override(),
      pci_swiotlb_detect_4gb() and register_mem_sect_under_node()
    - doing a crashkernel return from cpu_up()
  I don't understand why these should be necessary for my kernels but are
  not reported as problems elsewhere. I'm still investigating and will discuss
  those patches separately.
- my makedumpfile is an mmap-using version, with about 10 patches applied. I'll
  check which of those are not in the common version and discuss separately.
- my kexec is version 2.0.4 with 3 patches applied. I'll check which of those
  are not in the common version and discuss separately.

-Cliff