[LEDE-DEV] [PATCH] sdk: automatically use all CPU cores for xz

[Jonas Gorski]

Hi,

On 17 August 2017 at 15:05, Karl Palsson <karlp at etactica.com> wrote:
> xz has supported multithreaded compression since 5.2 in 2014.  Enable
> it's automatic support for this via the "-T 0" flag.

its ;p

>
> Previously: (xz -7e)
> real    3m13.631s
>
> Now: (xz -T 0 -7e)
> real    1m23.051s

After playing around with it, it seems enabling multiple threads makes
it compress slightly worse, at least in case of the linux kernel
sources (86.8 MiB single thread vs. 87.7 MiB multithread).

This means that the archive will have a different checksum, and thus
affects reproducability. [1] suggests this is because of a different
block size default, but unfortunately there seems to be no way to set
the block size in an attempt to make it use the same one regardless of
threads.

If we still do this, I would suggest taking over the value from -j, so
we don't use more cores than allowed, and you can force single
threaded mode on a multi core system. Also ImageBuilder and Toolchain
might profit from it as well.


Regards
Jonas

[1] https://lists.debian.org/debian-dpkg/2016/10/msg00008.html