[PATCH RFC 00/11] makedumpfile: parallel processing

[Atsushi Kumagai]

Hello Qiao,

>> On 06/10/2015 02:06 PM, Atsushi Kumagai wrote:
>>> Hello Zhou,
>>>
>
>Hello Atsushi,
>
>>>> This patch set implements parallel processing by means of multiple threads.
>>>> With this patch set, it is available to use multiple threads to read
>>>> and compress pages. This parallel process will save time.
>>>> This feature only supports creating dumpfile in kdump-compressed format from
>>>> vmcore in kdump-compressed format or elf format. Currently, sadump and
>>>> xen kdump are not supported.
>>>
>>> makedumpfile already has a parallel processing feature (--split),
>>> it parallelizes not only page compression but also disk i/o, so
>>> I think --split includes what you want to do by this patch.
>>>
>>> In what case do you think this patch will be effective, what is
>>> the advantage of this patch ?
>
>Since commit 428a5e99eea929639ab9c761f33743f78a961b1a(kdumpctl: Pass
>disable_cpu_apicid to kexec of capture kernel) has been merged. It is possible for
>us to use multiple cpus in 2nd kernel.
>
>Using multiple threads is trying to take advantage of multiple cpus in 2nd kernel.
>Since memory becomes bigger and bigger, dumping spends more time. Why not take
>advantage of multiple cpus?
>
>OTOH, --split does a lot help to improve performance. But more processes
>means more files, saving multiple files and managing those files is not that
>convenient.

I see, actually I guess some users may feel lazy to use --split since
it requires concatenation for analyzing, and it seems that some improvements
by using multiple threads can be expected at least in the zlib case.
So I agree with the concept.

>Multiple threads do have some merit in improving performance. And later, as zhou
>said, we can also try to combine --split with multiple threads to save more time.

At first I thought it's enough to modify --split path to generate single vmcore.
However, if the compression process is the bottleneck, we should allot multiple
cpus to each i/o process when doing parallel i/o. For that reason, it's good to
introduce the new feature to create multiple threads in addition to --split.

Just one thing, when you make the complete version, please make it on the devel
branch because cyclic/non-cyclic codes have been changed from v1.5.8.


Thanks
Atsushi Kumagai


>--
>Regards
>Qiao Nuohan
>
>>>
>>>
>>> Thanks
>>> Atsushi Kumagai
>>>
>>>>
>>>> Qiao Nuohan (11):
>>>>   Add readpage_kdump_compressed_parallel
>>>>   Add mappage_elf_parallel
>>>>   Add readpage_elf_parallel
>>>>   Add read_pfn_parallel
>>>>   Add function to initial bitmap for parallel use
>>>>   Add filter_data_buffer_parallel
>>>>   Add write_kdump_pages_parallel to allow parallel process
>>>>   Add write_kdump_pages_parallel_cyclic to allow parallel process in
>>>>     cyclic_mode
>>>>   Initial and free data used for parallel process
>>>>   Make makedumpfile available to read and compress pages parallelly
>>>>   Add usage and manual about multiple threads process
>>>>
>>>> Makefile       |    2 +
>>>> erase_info.c   |   29 +-
>>>> erase_info.h   |    2 +
>>>> makedumpfile.8 |   24 +
>>>> makedumpfile.c | 1505 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>>>> makedumpfile.h |   79 +++
>>>> print_info.c   |   16 +
>>>> 7 files changed, 1652 insertions(+), 5 deletions(-)
>>>>
>>>>
>>>> _______________________________________________
>>>> kexec mailing list
>>>> kexec at lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/kexec
>>
>>