[RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.

[HATAYAMA Daisuke]

From: Atsushi Kumagai <kumagai-atsushi at mxc.nes.nec.co.jp>
Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
Date: Fri, 13 Jul 2012 17:10:39 +0900

> Hello HATAYAMA-san,
> 
> On Fri, 13 Jul 2012 14:18:01 +0900 (JST)
> HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:
> 
>> From: Atsushi Kumagai <kumagai-atsushi at mxc.nes.nec.co.jp>
>> Subject: Re: [RFC PATCH v2 0/10] makedumpfile: cyclic processing to keep memory consumption.
>> Date: Fri, 13 Jul 2012 09:36:11 +0900
>> 
>> > Hello HATAYAMA-san,
>> > 
>> > On Wed, 11 Jul 2012 14:23:59 +0900 (JST)
>> > HATAYAMA Daisuke <d.hatayama at jp.fujitsu.com> wrote:
>> > 
>> > [...]
>> >> >>> Hi Atushi san,
>> >> >>> 
>> >> >>> Good to see this work making progress. I have few queries.
>> >> >>> 
>> >> >>> - Do you have some numbers for bigger machines like 1TB or higher memory. 
>> >> >>>   I am curious to know how bad is the time penalty.
>> >> >> 
>> >> >> I'm afraid that I don't have such a large machine, so I need someone who can
>> >> >> measure execution time in large machine.
>> >> >> 
>> >> > 
>> >> > I can prepare such machine but I cannot say I can use the machine on
>> >> > this day precisely for example. But I think it must be until the end
>> >> > of this month at most. So, I would like to fix content of benchmark
>> >> > first.
>> >> > 
>> >> 
>> >> Hello Kumagai-san,
>> >> 
>> >> I'm now trying to reserve the machine with big memory, but I'm not
>> >> accurately sure when I can use the machine; expecting within this
>> >> month? In advance, I want to make sure what I measure in this
>> >> benchmark in more detail.
>> > 
>> > OK, I'm glad for your help.
>> > 
>> >> I'm going to collect at least what you showed in your benchmark: RSS
>> >> size for no option, -cd31 and -Ed31. Is there another to collect?
>> > 
>> > Would you collect RSS also for --split ?
>> > While I expect the RSS will be increase based on number of child processes,
>> > I want to see measured data.
>> > 
>> 
>> Yes, I'll collect RSS information. But due to COW, child processes
>> share parent process's memory until they modify their memory, so
>> simply looking value of RSS is not enough. We need to measure size of
>> shared part e.g. by looking at /proc/<PID>/smaps.
> 
> You're right.  Would you choose the way which can correct effective
> memory consumption ?
> 

OK. PSS in smaps seems proper here, and though /proc/<PID>/pagemap
would also allow us to evaluate exact RSS, it's very slow. This is not
RSS, valgrind seems useful for evaluating usage of heap memory.

I'll consider how to evalute..

>> And, what version of makedumpfile should I use for this bench? If you
>> have additonal changes locally, it might be better to use the version.
> 
> Please use v1.4.4 + v2 patch of cyclic + performance improvement patch which
> I sent today, it's the latest version internally.
> 

I'll use this version.

Thanks.
HATAYAMA, Daisuke