[OpenWrt-Devel] [PATCH v2 0/3] build: Avoid 'Argument list too long' error

[Josef Schlehofer]

Hello,


Thank you for your patches, Eneas! We (for Turris routers) experienced
the same that you are describing once we did an update on build servers
from Debian Stretch to Debian Buster. It is present also for OpenWrt
19.07, where we were able to see it first because currently OpenWrt
master we are not able to compile because hostapd fails. More about
it https://lists.infradead.org/pipermail/openwrt-devel/2020-June/023976.html


So all these tested patches were tested and feel free to add my:
Tested-by: Josef Schlehofer <josef.schlehofer at nic.cz> 


Regards,

Josef

On 20. 02. 20 22:29, Eneas U de Queiroz wrote:
> TLDR for those who have read v1:  I've come up with a good-enough, but
> not perfect wildcard pattern, and added it in a separate commit.
>
> After building all of the luci packages with all of the translations,
> if I either run:
> 'make package/luci/clean' or 'make package/luci/compile', I get:
> make[2]: execvp: /usr/bin/env: Argument list too long
> make[2]: *** [../../luci.mk:285: luci-clean] Error 127
>
> This is caused by the call to scripts/ipkg-remove with a list of over
> 2,300 packages matching a luci* wildcard:
> $ ll bin/packages/arm_cortex-a9_vfpv3/luci/luci*.ipk | wc -l
> 2307
>
> My first attempt to circumvent this was using xargs.  However, using
> echo from the Makefile results in make calling the shell with the same
> 2,307 file names.  The solution is to have make write the list to a file
> and then feed it to xargs.
>
> To avoid creating a file every time, xargs is only used when the number
> of files is >=512.
>
> As an optimization, to avoid calling wildcard twice, I've defined a
> RemoveOpkgPackageFiles function, and added the check for an empty list
> there, so that the call to opkg_package_files would only be done when
> the new function was called.
>
> I've put them in separate commits to ease an eventual reversal or
> rejection.
>
> If we change the wildcard pattern that selects the files, we can
> eliminate the problems with xargs, and avoid 4,612 runs of 'tar -Ozxf'
> when making package/luci/compile.
>
> There is a caveat; it will not remove the .ipk file if the version of
> a package whose name ends in a digit (e.g. nghttp2) that was currently
> built with an ABI_VERSION, but the version of the new build does not
> have an ABI_VERSION.  Then, make package/nghttp2/clean will not remove
> the old ipk file.
>
> I consider this extremely minor.  Nonetheless, I will leave the
> intermediate commit, since it works in 100% of the cases, making it
> easier to revert this.  If I should rather squash them, let me know
> which commits should be kept and I'll send a v3.
>
> This was compile-tested for mvebu, and checked by adding $(info ...)
> tracers to the new functions and to opkg_package_files in
> include/feeds.mk.
>
> To check how many ipk files each package was selecting with the new
> wildcard, I've run the following script in a directory containing all
> packages that I've build for mvebu:
>
> #!/bin/bash
> total=0
> packages=0
> for f in *.ipk; do
>   PKG=$(tar -Ozxf "$f" ./control.tar.gz | tar -Ozxf - ./control \
>         | sed -ne '/^Package:/{s,.*: ,,; p}')
>   SRC=$(tar -Ozxf "$f" ./control.tar.gz | tar -Ozxf - ./control \
>         | sed -ne '/^SourceName:/{s,.*: ,,; p}')
>   if [ "${SRC}" = "${PKG}" ]; then
>     files="${PKG}[^a-z-]*_*.ipk"
>   else
>     case "${SRC}" in
>        *[0-9] )
>           files="${SRC}*_*.ipk"
>           ;;
>        * )
>           files="${SRC}[^a-z-]*_*.ipk"
>     esac
>   fi
>   n=$(echo ${files} | wc -w)
>   if [ "$n" -ne 1 ]; then
>     echo pkg=${SRC} - n=${n}
>     ls -1 ${files}
>   fi
>   ((total = total + n))
>   ((packages++))
> done
> echo Total Packages=${packages}. Total lookups=${total}
>
> If you want to see the old wildcard, always use `files=${SRC}*_*.ipk`.
>
> Before the change:
> Total Packages=8213. Total lookups=16689
> After:
> Total Packages=8213. Total lookups=8838
>
> ChangeLog:
>
> v1->v2:
>   * Renamed the new functions using lowercase and underscores
>   * Used '< file' instead of 'cat file |' to pass the files to xargs
>   * Added a commit changing the wildcard pattern, reverting the use of
>     xargs.
>
> Eneas U de Queiroz (3):
>   build: package-ipkg: avoid calling wildcard twice
>   build: call ipkg-remove using xargs if #args>=512
>   build: reduce number of files passed to ipk-remove
>
>  include/package-ipkg.mk | 17 +++++++++++++++--
>  1 file changed, 15 insertions(+), 2 deletions(-)
>
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.infradead.org/pipermail/openwrt-devel/attachments/20200618/d8fb1943/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 659 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/openwrt-devel/attachments/20200618/d8fb1943/attachment.sig>
-------------- next part --------------
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel