[RFC] makedumpfile, crash: LZO compression support

[HATAYAMA Daisuke]

Hello,

This is a RFC patch set that adds LZO compression support to
makedumpfile and crash utility. LZO is as good as in size but by far
better in speed than ZLIB, leading to reducing down time during
generation of crash dump and refiltering.

How to build:

  1. Get LZO library, which is provided as lzo-devel package on recent
  linux distributions, and is also available on author's website:
  http://www.oberhumer.com/opensource/lzo/.

  2. Apply the patch set to makedumpfile v1.4.0 and crash v6.0.0.

  3. Build both using make. But for crash, do the following now:

    $ make CFLAGS="-llzo2"

How to use:

  I've newly used -l option for lzo compression in this patch. So for
  example, do as follows:

  $ makedumpfile -l vmcore dumpfile
  $ crash vmlinux dumpfile

Request of configure-like feature for crash utility:

  I would like configure-like feature on crash utility for users to
  select wheather to add LZO feature actually or not in build-time,
  that is: ./configure --enable-lzo or ./configure --disable-lzo.

  The reason is that support staff often downloads and installs the
  latest version of crash utility on machines where lzo library is not
  provided.

  Looking at the source code, it looks to me that crash does some kind
  of configuration processing in a local manner, around configure.c,
  and I guess it's difficult to use autoconf tools directly.

  Or is there another better way?

Performance Comparison:

  Sample Data

    Ideally, I must have measured the performance for many enough
    vmcores generated from machines that was actually running, but now
    I don't have enough sample vmcores, I couldn't do so. So this
    comparison doesn't answer question on I/O time improvement. This
    is TODO for now.

    Instead, I choosed worst and best cases regarding compression
    ratio and speed only. Specifically, the former is /dev/urandom and
    the latter is /dev/zero.

    I get the sample data of 10MB, 100MB and 1GB by doing like this:

      $ dd bs=4096 count=$((1024*1024*1024/4096)) if=/dev/urandom of=urandom.1GB

  How to measure

    Then I performed compression for each block, 4096 bytes, and
    measured total compression time and output size. See attached
    mycompress.c.

  Result

    See attached file result.txt.

  Discussion

    For both kinds of data, lzo's compression was considerably quicker
    than zlib's. Compression ratio is about 37% for urandom data, and
    about 8.5% for zero data. Actual situation of physical memory
    would be in between the two cases, and so I guess average
    compression time ratio is between 37% and 8.5%.

    Although beyond the topic of this patch set, we can estimate worst
    compression time on more data size since compression is performed
    block size wise and the compression time increases
    linearly. Estimated worst time on 2TB memory is about 15 hours for
    lzo and about 40 hours for zlib. In this case, compressed data
    size is larger than the original, so they are really not used,
    compression time is fully meaningless. I think compression must be
    done in parallel, and I'll post such patch later.

Diffstat

  * makedumpfile

 diskdump_mod.h |    3 +-
 makedumpfile.c |   98 +++++++++++++++++++++++++++++++++++++++++++++++++------
 makedumpfile.h |   12 +++++++
 3 files changed, 101 insertions(+), 12 deletions(-)

  * crash

 defs.h     |    1 +
 diskdump.c |   20 +++++++++++++++++++-
 diskdump.h |    3 ++-
 3 files changed, 22 insertions(+), 2 deletions(-)

TODO

  * evaluation including I/O time using actual vmcores

Thanks.
HATAYAMA, Daisuke
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mycompress.c
Type: text/x-csrc
Size: 3452 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20111117/9e5ddfe3/attachment-0001.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: result.txt
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20111117/9e5ddfe3/attachment-0001.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Add-LZO-Support.patch
Type: application/octet-stream
Size: 6319 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20111117/9e5ddfe3/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Support-LZO-compression-format.patch
Type: application/octet-stream
Size: 2598 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20111117/9e5ddfe3/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Add-Compression-and-IO-time-report.patch
Type: application/octet-stream
Size: 5142 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20111117/9e5ddfe3/attachment-0005.obj>