makedumpfile 1.5.0 takes much more time to dump

[Atsushi Kumagai]

Hello,

> >> > Hi Atsushi san,
> >> > 
> >> > We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress
> >> > badly. We reserved 192MB of memory and following are test results.
> >> > 
> >> > #1. makedumpfile-1.4.2 -E --message-level 1 -d 31
> >> > real     3m47.520s
> >> > user     0m56.543s
> >> > sys	 2m41.631s
> >> > 
> >> > #2. makedumpfile-1.5.0 -E --message-level 1 -d 31
> >> > real     52m25.262s
> >> > user     32m51.310s
> >> > sys	 18m53.265s
> >> > 
> >> > #3. makedumpfile-1.4.2 -c --message-level 1 -d 31
> >> > real     8m49.107s
> >> > user     4m34.180s
> >> > sys	 4m8.691s
> >> > 
> >> > #4. makedumpfile-1.5.0 -c --message-level 1 -d 31
> >> > real     46m48.985s
> >> > user     29m35.203s
> >> > sys	 16m43.149s
> >> > 
> >> 
> >> Hello Vivek,
> >> 
> >> On v1.5.0 we cannot filter free pages in constant space becuase we
> >> have yet to test it. Instead, the existing method is used here, which
> >> repeats walking on a whole page frames the number of cycles times.
> >> 
> >> As Kumagai-san explains, the number of cycles can be calculated by the
> >> following expression:
> >> 
> >>   N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)
> >> 
> >> So,
> >> 
> >>   N = 2TB / (4KB * 8 * 1MB) = 64 cycles.
> >> 
> >> I guess on this environment, it took about 50 seconds to filter free
> >> pages in one cycle.
> > 
> > Ok, so once we have your walking through page struct patches in, hopefully
> > this problem will be gone?
> Yes, then free page filtering is integrated in other mem_map array
> logics. You can see how it takes from the following message.
> 
> Excluding unnecessary pages        : [100 %] STEP [Excluding unnecessary pages] : 0.204574 seconds
> 
> New logic takes equal to or quicker than the total time indicated by
> the above kind of messages now.
> 
> > If that's going to take time, can we make using of new logic conditional
> > on a command line option. So that user has the option of using old
> > logic.
> > 
> 
> Kumagai-san should decide this. 

I'm not planning to make the cyclic mode optional.
However, I think the performance issue should be improved even without
mem_map array logic.

So, I will try to reduce the number of cycles as few times as possible for v1.5.1,
the performance issue will be improved.
To make sure of it, would you re-test with --cyclic-buffer 32768 (32MB), Vivek ?
Then the result of v1.5.0 is still too bad, I will consider using the old logic 
as default logic.


> BTW, Kumagai-san, in what version do you plan release of this new
> mem_map logic? Around v1.5.2? I want to re-send a new patch set to
> which I add small changes and comments for maintainability.

Yes, I plan to merge it in v1.5.2. 
Would you re-send the new patch set based on v1.5.1 ?

Additionally, I'm afraid that I haven't reviewed the most of patches
for v1.5.1 yet. So I can't tell you when v1.5.2 will be released.


Thanks
Atsushi Kumagai