[LEDE-DEV] [PATCH] sdk: automatically use all CPU cores for xz

[Baptiste Jonglez]

On 18-08-17, Karl Palsson wrote:
> > This means that the archive will have a different checksum, and
> > thus affects reproducability. [1] suggests this is because of a
> > different block size default, but unfortunately there seems to
> > be no way to set the block size in an attempt to make it use
> > the same one regardless of threads.
> 
> Further in that same thread, it seems to suggest that for _any_
> build with >1 core, it will always use the same block size, so
> I'm not _entirely_ convinced this is actually a real problem with
> todays computers.

Indeed, the result is reproducible for any thread number >1.  On a
many-cores machines:

55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T0.tar.xz
c5c7259f2781cddfaaae92b6b7fd7c17a8b1012a  linux-T1.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T2.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T4.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T7.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T8.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T11.tar.xz

On a single-core machine:

c5c7259f2781cddfaaae92b6b7fd7c17a8b1012a  linux-T0.tar.xz
c5c7259f2781cddfaaae92b6b7fd7c17a8b1012a  linux-T1.tar.xz
55395e07f991cb8057c73d19bbe2030b716f4a90  linux-T2.tar.xz

If we really wanted a reproducible output whatever -j option is passed, we
could enforce a minimum of -T2.  But this is a bad idea for single core
machines, because it uses a lot more RAM and is twice as slow:

$ time xz -T1 < linux.tar > linux-T1.tar.xz
real           5m48.682s
user           5m37.036s
sys            0m0.932s

$ time xz -T2 < linux.tar > linux-T2.tar.xz
real           11m20.137s
user           5m46.528s
sys            0m0.472s

I think the significant reduction in build time outweighs the slight
decrease in reproducibility (especially given the broad availability of
multi-core machines).

My 2 cents,
Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/lede-dev/attachments/20170818/8e664a70/attachment.sig>