[PATCH 00/11] makedumpfile: Add zstd support for makedumpfile

[HAGIO KAZUHITO(萩尾 一仁)]

Hi Tao Liu,

Thanks for the patchset!

-----Original Message-----
> This patch set adds ZSTD compression support to makedumpfile. With ZSTD compression
> support, the vmcore dump size and time consumption can have a better balance than
> zlib/lzo/snappy.
> 
> How to build:
> 
>   Build using make:
>     $ make USEZSTD=on
> 
> Performance Comparison:
> 
>   How to measure
> 
>     I took a x86_64 machine which had 4T memory, and the compression level
>     range from (-3 to 4) for ZSTD, as well as zlib/lzo/snappy compression.
>     All testing was done by makedumpfile single thread mode.
> 
>     As for compression performance testing, in order to avoid the performance
>     bottle neck of disk I/O, I used the following makedumpfile cmd, which took
>     lzo compression as an example. "--dry-run" will not write any data to disk,
>     "--show-stat" will output the vmcore size after compression, and the time
>     consumption can be collected from the output logs.
> 
>     $ makedumpfile -d 0 -l /proc/kcore vmcore --dry-run --show-stat
> 
> 
>     As for decompression performance testing, I only tested the (-d 31) case,
>     because the vmcore size of (-d 0) case is too big to fit in the disk, in
>     addtion, to read a oversized file from disk will encounter the disk I/O
>     bottle neck.
> 
>     I triggered a kernel crash and collected a vmcore. Then I converted the
>     vmcore into specific compression format using the following makedumpfile
>     cmd, which would get a lzo format vmcore as an example:
> 
>     $ makedumpfile -l vmcore vmcore.lzo
> 
>     After all vmcores were ready, I used the following cmd to perform the
>     decompression, the time consumption can be collected from the logs.
> 
>     $ makedumpfile -F vmcore.lzo --dry-run --show-stat
> 
> 
>   Result charts
> 
>     For compression:
> 
>             makedumpfile -d31			|  makedumpfile -d0
>             Compression time	vmcore size	|  Compression time  vmcore size
>     zstd-3  325.516446	        5285179595	|  8205.452248	     51715430204
>     zstd-2  332.069432	        5319726604	|  8057.381371	     51732062793
>     zstd-1  309.942773	        5730516274	|  8138.060786	     52136191571
>     zstd0   439.773076	        4673859661	|  8873.059963	     50993669657
>     zstd1   406.68036	        4700959521	|  8259.417132	     51036900055
>     zstd2   397.195643	        4699263608	|  8230.308291	     51030410942
>     zstd3   436.491632	        4673306398	|  8803.970103	     51043393637
>     zstd4   543.363928	        4668419304	|  8991.240244	     51058088514
>     zlib    561.217381	        8514803195      | 14381.755611	     78199283893
>     lzo	    248.175953	       16696411879	|  6057.528781	     90020895741
>     snappy  231.868312	       11782236674	|  5290.919894	    245661288355
> 
>     For decompression:
> 
>             makedumpfile -d31
>             decompress time	   vmcore size
>     zstd-3	477.543396	       5289373448
>     zstd-2	478.034534	       5327454123
>     zstd-1	459.066807	       5748037931
>     zstd0	561.687525	       4680009013
>     zstd1	547.248917	       4706358547
>     zstd2	544.219758	       4704780719
>     zstd3	555.726343	       4680009013
>     zstd4	558.031721	       4675545933
>     zlib	630.965426	       8555376229
>     lzo	    	427.292107	      16849457649
>     snappy	446.542806	      11841407957
> 
>   Discussion
> 
>     For zstd range from -3 to 4, compression level 2 (ZSTD_dfast) has
>     the best time consumption and vmcore dump size balance.

Do you have a result of -d 1 compression test?  I think -d 0 is not
practical, I would like to see a -d 1 result of such a large vmcore.

And just out of curiosity, what version of zstd are you using?
When I tested zstd last time, compression level 1 was faster than 2, iirc.

btw, ZSTD_dfast is an enum of ZSTD_strategy, not for compression level?
(no need to update for now, I will review later)

Thanks,
Kazu

> 
>     For zstd2/zlib/lzo/snappy, zstd2 has the smallest vmcore size, also
>     the best time consumption and vmcore dump size balance.
> 
> Tao Liu (11):
>   Add dump header for zstd.
>   Add command-line processing for zstd
>   Add zstd build support
>   Notify zstd unsupporting when disabled
>   Add single thread zstd compression processing
>   Add parallel threads zstd compression processing
>   Add single thread zstd uncompression processing
>   Add parallel threads zstd uncompression processing
>   Add zstd help message
>   Add zstd manual description
>   Add zstd README description
> 
>  Makefile       |   5 +++
>  README         |   5 ++-
>  diskdump_mod.h |   1 +
>  makedumpfile.8 |   7 ++--
>  makedumpfile.c | 101 +++++++++++++++++++++++++++++++++++++++++++++----
>  makedumpfile.h |  10 +++++
>  print_info.c   |  30 ++++++++++-----
>  7 files changed, 138 insertions(+), 21 deletions(-)
> 
> --
> 2.29.2