[LEDE-DEV] [PATCH] sdk: automatically use all CPU cores for xz

[Karl Palsson]

Jonas Gorski <jonas.gorski at gmail.com> wrote:
> >
> > Previously: (xz -7e)
> > real    3m13.631s
> >
> > Now: (xz -T 0 -7e)
> > real    1m23.051s
> 
> After playing around with it, it seems enabling multiple
> threads makes it compress slightly worse, at least in case of
> the linux kernel sources (86.8 MiB single thread vs. 87.7 MiB
> multithread).

I consider that to be lost in the noise when compared to bz2. T
 
> This means that the archive will have a different checksum, and
> thus affects reproducability. [1] suggests this is because of a
> different block size default, but unfortunately there seems to
> be no way to set the block size in an attempt to make it use
> the same one regardless of threads.

Further in that same thread, it seems to suggest that for _any_
build with >1 core, it will always use the same block size, so
I'm not _entirely_ convinced this is actually a real problem with
todays computers.

I've tried asking in both #debian-reproducible and
#reproducible-builds but not gotten any meaningful feedback.
(beyond: "it might")

Further...
https://tests.reproducible-builds.org/lede/lede_ar71xx.html isn't
even building the sdk or image builder or toolchain.... _and_
they don't change the cpu either. Is this really an issue that
matters?
 
> If we still do this, I would suggest taking over the value from
> -j, so we don't use more cores than allowed, and you can force
> single threaded mode on a multi core system.

If you know a nice way of getting the actual number in make,
please, go right ahead. "-T 0" was _vastly_ simpler than the sort
of jiggerypokery to get a sane number out of this optional make
argument.

> Also ImageBuilder and Toolchain might profit from it as well.

Indeed, I don't have those enabled in my build, so only noticed
the SDK being the massive timesuck. If we can get something like
this in, it could definitely be added to any place that uses xz.
(related, but not, we build a host 'xz' but then this place just
uses whatever xz the shell provides.....)

> 
> [1] https://lists.debian.org/debian-dpkg/2016/10/msg00008.html
> 

Cheers,
Karl P
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.html
Type: application/pgp-signature
Size: 1161 bytes
Desc: OpenPGP Digital Signature
URL: <http://lists.infradead.org/pipermail/lede-dev/attachments/20170818/ce42aec0/attachment.sig>