[PATCH v4] Improve the performance of --num-threads -d 31

[Atsushi Kumagai]

Hello,

Let me get this straight.

>> >>Hello Minfei,
>> >>
>> >>Thanks for your results.
>> >>And I have some questions.
>> >>
>> >>On 03/31/2016 04:38 PM, Minfei Huang wrote:
>> >>>Hi, Zhou.
>> >>>
>> >>>I have tested the increasing patch on 4T memory machine.
>> >>>
>> >>>makedumpfile fails to dump vmcore, if there are about 384M memory in 2nd
>> >>>kernel which is reserved by crashkernel=auto. But once the reserved
>> >>>memory is enlarged up to 10G, makedumpfile can dump vmcore successfully.
>> >>>
>> >>
>> >>Will it fail with patch v3? or just v4?
>> >
>> >Both v3 and v4 can work well, once reserved memory is enlarged manually.
>> >
>> >>I don't think it is a problem.
>> >>If 128 cpus are enabled in second kernel, there won't be much memory left if total memory is 384M.
>> >
>> >Enable 128 CPUs with 1GB reserved memory.
>> >kdump:/# /sysroot/bin/free -m
>> >               total        used        free      shared  buff/cache   available
>> >Mem:            976          97         732           6         146         774
>> >
>> >Enable 1 CPU with 1GB reserved memory.
>> >kdump:/# /sysroot/bin/free -m
>> >               total        used        free      shared  buff/cache   available
>> >Mem:            991          32         873           6          85         909
>> >
>> >Extra enabled 127 CPUs will consume 65MB. So I think it is acceptable
>> >in kdump kernel.
>> >
>> >The major memory is consumed by makedumpfile from the test result.
>> >crashkernel=auto doesn't work any more, if option --num-threads is
>> >set. Even more, there is no warning to let user enlarge the reserved
>> >memory.
>> >
>>
>> Yes, we should remind user if they want to use too much threads.

After all, the ENOMEM issue isn't peculiar to a specific patch but
just a matter of course "more threads consume more memory" ?
(At least I expect so.)

>This new feature multi-threads will consume more memory during dumping
>vmcore in 2nd kernel. Is it possible to improve it?

makedumpfile is designed to work in limited memory space, but the extra
memory consumption for each thread isn't considered.
I think it can be estimated by checking code and measured by actual test.

e.g. If we find that each thread requires 100KB more memory, 100KB * num-thread
should be reserved before filtering processing.
As I said to Zhou, the logic should be inserted into the cyclic buffer
allocation method (calculate_cyclic_buffer_size()).

>> >>And I think it will also work if the reserved memory is set to 1G.
>> >
>> >Yes, makedumpfile can work well under 1GB reserved memory.
>> >
>> >>
>> >>>The cache should be dropped before testing, otherwise makedumpfile will
>> >>>fail to dump vmcore.
>> >>>echo 3 > /proc/sys/vm/drop_caches
>> >>>Maybe there is something cleanup we can do to avoid this.
>> >>>
>> >>>Following is the result with different parameter for option
>> >>>--num-threads.
>> >>>
>> >>>makedumpfile -l --num-threads 128 --message-level 1 -d 31 /proc/vmcore a.128
>> >>>real    5m34.116s
>> >>>user    103m42.531s
>> >>>sys 86m12.586s
>> >[ snip ]
>> >>>makedumpfile -l --num-threads 0 --message-level 1 -d 31 /proc/vmcore a.0
>> >>>real    3m46.531s
>> >>>user    3m29.371s
>> >>>sys 0m16.909s
>> >>>
>> >>>makedumpfile.back -l --message-level 1 -d 31 /proc/vmcore a
>> >>>real    3m55.712s
>> >>>user    3m39.254s
>> >>>sys 0m16.287s
>> >>>
>> >>>Once the reserved memory is enlarged, makedumpfile works well with or
>> >>>without this increaseing patch.
>> >>>
>> >>>But there is an another issue I found during testing. makedumpfile may
>> >>>hang in about 24%. And with option --num-threads 64, this issue is also
>> >>>occured.
>> >>>
>> >>
>> >>Will it occur with patch v3?
>> >>If it not occurs, then neither of the previous two increasing patches will work?
>> >>
>> >>And did you test it with or without the increasing patch?
>> >
>> >without this increasing patch, v4 works well.
>> >
>>
>> Do you mean makedumpfile won't hang without the increasing patch?
>
>Seem that, but I cann't confirm it, since this issue occurs very
>occasionally.

I want to know how many times were the two cases (v4 and v4+inc) tested
and how often did them hang.


Thanks,
Atsushi Kumagai

>Thanks
>Minfei
>
>>
>> --
>> Thanks
>> Zhou
>> >>
>> >>>makedumpfile -l --num-threads 128 --message-level 1 -d 31 /proc/vmcore a.128
>> >>>Excluding unnecessary pages        : [100.0 %] |
>> >>>Excluding unnecessary pages        : [100.0 %] /
>> >>>Excluding unnecessary pages        : [100.0 %] -
>> >>>Copying data                       : [ 11.2 %] |
>> >>>Copying data                       : [ 12.4 %] -
>> >>>Excluding unnecessary pages        : [100.0 %] \
>> >>>Excluding unnecessary pages        : [100.0 %] |
>> >>>Copying data                       : [ 23.6 %] -
>> >>>Copying data                       : [ 24.4 %] /
>> >>>
>> >>
>> >>Could you help me find which line of the code is running at when it hanging?
>> >>makedumpfile may be in a loop and can't go out by some bugs.
>> >
>> >This issue happens very occasionally. I can update it once meet it.
>> >
>> >Thanks
>> >Minfei
>> >
>> >
>>
>>
>>
>>
>>
>
>_______________________________________________
>kexec mailing list
>kexec at lists.infradead.org
>http://lists.infradead.org/mailman/listinfo/kexec