makedumpfile mmap() benchmark
d.hatayama at jp.fujitsu.com
Tue May 7 04:47:34 EDT 2013
(2013/05/04 4:10), Cliff Wickman wrote:
>> Jingbai Ma wote on 27 Mar 2013:
>> I have tested the makedumpfile mmap patch on a machine with 2TB memory,
>> here is testing results:
>> Test environment:
>> Machine: HP ProLiant DL980 G7 with 2TB RAM.
>> CPU: Intel(R) Xeon(R) CPU E7- 2860 @ 2.27GHz (8 sockets, 10 cores)
>> (Only 1 cpu was enabled the 2nd kernel)
>> Kernel: 3.9.0-rc3+ with mmap kernel patch v3
>> vmcore size: 2.0TB
>> Dump file size: 3.6GB
>> makedumpfile mmap branch with parameters: -c --message-level 23 -d 31
>> --map-size <map-size>
>> All measured time from debug message of makedumpfile.
>> As a comparison, I also have tested with original kernel and original
>> makedumpfile 1.5.1 and 1.5.3.
>> I added all [Excluding unnecessary pages] and [Excluding free pages]
>> time together as "Filter Pages", and [Copyying Data] as "Copy data" here.
>> makedumjpfile Kernel map-size (KB) Filter pages (s) Copy data (s) Total (s)
>> 1.5.1 3.7.0-0.36.el7.x86_64 N/A 940.28 1269.25 2209.53
>> 1.5.3 3.7.0-0.36.el7.x86_64 N/A 380.09 992.77 1372.86
>> 1.5.3 v3.9-rc3 N/A 197.77 892.27 1090.04
>> 1.5.3+mmap v3.9-rc3+mmap 0 164.87 606.06 770.93
>> 1.5.3+mmap v3.9-rc3+mmap 4 88.62 576.07 664.69
>> 1.5.3+mmap v3.9-rc3+mmap 1024 83.66 477.23 560.89
>> 1.5.3+mmap v3.9-rc3+mmap 2048 83.44 477.21 560.65
>> 1.5.3+mmap v3.9-rc3+mmap 10240 83.84 476.56 560.4
> I have also tested the makedumpfile mmap patch on a machine with 2TB memory,
> here are the results:
> Test environment:
> Machine: SGI UV1000 with 2TB RAM.
> CPU: Intel(R) Xeon(R) CPU E7- 8837 @ 2.67GHz
> (only 1 cpu was enabled in the 2nd kernel)
> Kernel: 3.0.13 with mmap kernel patch v3 (I had to tweak the patch a bit)
> vmcore size: 2.0TB
> Dump file size: 3.6GB
> makedumpfile mmap branch with parameters: -c --message-level 23 -d 31
> --map-size <map-size>
> All measured times are actual clock times.
> All tests are noncyclic. Crash kernel memory: crashkernel=512M
> As did Jingbai Ma, I also tested with an unpatched kernel and
> makedumpfile 1.5.1 and 1.5.3. But they do 2 filtering scans: unnecessary
> pages and free pages; here added together as filter pages time.
> Filter Copy
> makedumpfile Kernel map-size(KB) pages(s) data(s) Total(s)
> 1.5.1 3.0.13 N/A 671 511 1182
> 1.5.3 3.0.13 N/A 294 535 829
> 1.5.3+mmap 3.0.13+mmap 0 54 506 560
> 1.5.3+mmap 3.0.13+mmap 4096 40 416 456
> 1.5.3+mmap 3.0.13+mmap 10240 37 424 461
> Using mmap for the copy data as well as for filtering pages did little:
> 1.5.3+mmap 3.0.13+mmap 4096 37 414 451
> My results are quite similar to Jingbai Ma's.
> The mmap patch to the kernel greatly speeds the filtering of pages, so
> we at SGI would very much like to see this patch in the 3.10 kernel.
> What puzzles me is that the patch greatly speeds the read's of /proc/vmcore
> (where map-size is 0) as well as providing the mmap ability. I can now
> seek/read page structures almost as fast as mmap'ing and copying them.
> (versus Jingbai Ma's results where mmap almost doubled the speed of reads)
> I have put counters in to verify, and we are doing several million
> seek/read's vs. a few thousand mmap's. Yet the performance is similar
> (54sec vs. 37sec, above). I can't rationalize that much improvement.
The change between 1.5.3+mmap between 1.5.3 that might be affecting the
result I guess is the below only.
Author: HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com>
Date: Wed Feb 20 20:13:07 2013 +0900
[PATCH] Clean up readmem() by removing its recursive call.
In addition to your and Ma's results, my result also showed similar
result: 100 secs for read() and 70 secs for mmap() with 4KB map. See:
So I think:
- the performance degradation not only had come from many
ioremap/iounmap calls but also from the way makedumpfile was implemented.
- The changes of makedumpfile that impacted performance gain are the
- Implement 8-entry cache for readmem() by Petr Tesarik, and
- The above clean up patch that removes unnecessary recursive call of
- Even by these changes only, we can get enough performance gain.
Further, using mmap allows us to get the performance close to
kernel-side processing; this might be unnecessary in practice but might
be meaningful in kdump's design that uses user-space tools as a part of
More information about the kexec