[RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

[Atsushi Kumagai]

Hello HATAYAMA-san,

On Thu, 30 Aug 2012 09:55:06 +0900 (JST)
HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:

> From: Vivek Goyal <vgoyal at redhat.com>
> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> Date: Wed, 29 Aug 2012 08:35:39 -0400
> 
> > On Wed, Aug 29, 2012 at 11:50:31AM +0900, HATAYAMA Daisuke wrote:
> >> From: Atsushi Kumagai <kumagai-atsushi at mxc.nes.nec.co.jp>
> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> >> Date: Wed, 15 Aug 2012 15:27:10 +0900
> >> 
> >> > Hello HATAYAMA-san,
> >> > 
> >> > On Tue, 14 Aug 2012 20:55:32 +0900 (JST)
> >> > HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:
> >> > 
> >> >> From: Vivek Goyal <vgoyal at redhat.com>
> >> >> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
> >> >> Date: Fri, 10 Aug 2012 10:36:55 -0400
> >> >> 
> >> >> > On Fri, Aug 10, 2012 at 05:39:38PM +0900, HATAYAMA Daisuke wrote:
> >> >> > 
> >> >> > [..]
> >> >> >> 
> >> >> >> I finished benchmarking filtering time and demonstrate the result.
> >> >> >> But I failed to collect amount of memory consumption by my mistake. If
> >> >> >> they are necessary, I'll again try to collect them. But we have 9 days
> >> >> >> vacation starting tommorow, so I'll do that after the vacation.
> >> >> >> 
> >> > 
> >> > Thank you for your help.
> >> > Could you continue to measure amount of memory consumption ?
> >> > 
> >> 
> >> Hello Kumagai-san, here is the benchmark result for the amount of
> >> actual memory consumption when there are multiple child processes
> >> using --split option.

Thank you for all your help, I could make sure that cyclic mode is effective.

> > 
> > Hi,
> > 
> > Thanks for testing results. I had few questions.
> > 
> > - What's the objective of this testing? Are we just trying to figure out
> >   memory footprint of makedumpfile in cyclic mode using struct page 
> >   filtering and compare with free list filtering?
> > 
> 
> Basically yes, and in addition to that, I wanted to see when using
> --split because then makedumpfile process forks and memory usage
> increases.
> 
> > - Why are we testing using --split option. How does that help. In kdump
> >   kernel we boot only 1 cpu. And if all the dump files are being saved
> >   to same disk, it might not give lot of performance boost.
> >  
> >   Even if does give performance boost, this seems to be orthogonal to 
> >   the idea of going using struct page for filtering.  Will single thread
> >   dumping not give a good idea about memory footprint?
> > 
> 
> On this benchmark, I didn't aim at seeing performance gain here, only
> seeing memory footprint, so I used only one disk and one cpu to write
> without any special configuration.
> 
> In cyclic mode, each child process refers to different part of
> physical memory, and the bitmaps each process handles are also
> different each other; in normal mode, each child process shares the
> unique two bitmaps. This bench aims at evaluating amount of memory
> footprint for the former case precisely.
> 
> > 
> > - What's the conclusion of below measurements and numbers.
> > 
> 
> These are expected results, I think. Please forcus on buffer size and
> value of child's PRI. Each child process has two bitmaps and each
> bitmap has buffer size length. So, for 1MB buffer size, PRI finally
> amounts to 2.38 MB where 2.00 MB is the 2 bitmap part, and for other
> buffer sizes, these also hold similarly, and there rest of bitmaps
> part is all 0.37 or 0.38 MB.

Yes, I think so. According to the results, we can estimate memory footprint
with the buffer size even when using --split option. It's what I expected.


BTW, I'll post the patchset as v1.5.0-rc soon.


Thanks
Atsushi Kumagai