[LEDE-DEV] make -j 4: race between Image/Prepare and device images?

[Rafał Miłecki]

Hi,

I just discovered why I was getting some crazy results during my various kernel
tests. It seems that LEDE doesn't rebuild images as expected with make -j N.

I think that some/all images may be generated before Image/Prepare step.


// I did initial build with CONFIG_KERNEL_KALLSYMS=y
$ ls -l build_dir/target-*/linux-*/vmlinux.lzma
-rw-r--r-- 1 zajec users 1229437 Oct 25 10:10 build_dir/target-mipsel_mips32_musl-1.1.15/linux-brcm47xx_legacy/vmlinux.lzma
$ ls -l bin/targets/brcm47xx/legacy/*standard*trx bin/targets/brcm47xx/legacy/*wrt300n*trx
-rw-r--r-- 1 zajec users 3608576 Oct 25 10:11 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-linksys-wrt300n-v1-squashfs.trx
-rw-r--r-- 1 zajec users 4132864 Oct 25 10:11 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-noloader-gz-squashfs.trx
-rw-r--r-- 1 zajec users 3608576 Oct 25 10:11 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-squashfs.trx


// I disabled CONFIG_KERNEL_KALLSYMS and decided to rebuild
$ make -j 4
  make[1] world
  make[2] package/cleanup
  make[2] target/compile
  make[3] -C target/linux compile
  make[2] package/compile
  make[3] -C package/kernel/gpio-button-hotplug compile
  make[3] -C package/libs/toolchain compile
  make[3] -C package/system/usign host-compile
  make[3] -C package/libs/libjson-c compile
  make[3] -C package/libs/libnl-tiny compile
  make[3] -C package/utils/lua compile
  make[3] -C package/libs/lzo compile
  make[3] -C package/libs/zlib compile
  make[3] -C package/utils/util-linux compile
  make[3] -C package/system/lede-keyring compile
  make[3] -C package/firmware/b43legacy-firmware compile
  make[3] -C package/firmware/linux-firmware compile
  make[3] -C package/firmware/prism54-firmware compile
  make[3] -C package/network/utils/iw compile
  make[3] -C package/libs/mbedtls compile
  make[3] -C package/libs/polarssl compile
  make[3] -C package/network/ipv6/odhcp6c compile
  make[3] -C package/network/services/dnsmasq compile
  make[3] -C package/network/services/dropbear compile
  make[3] -C package/libs/libpcap compile
  make[3] -C package/network/utils/linux-atm compile
  make[3] -C package/network/utils/resolveip compile
  make[3] -C package/utils/busybox compile
  make[3] -C package/utils/nvram compile
  make[3] -C package/utils/otrx compile
  make[3] -C package/libs/libubox compile
  make[3] -C package/utils/mtd-utils compile
  make[3] -C package/kernel/linux compile
  make[3] -C package/system/ubus compile
  make[3] -C package/system/uci compile
  make[3] -C package/utils/jsonfilter compile
  make[3] -C package/system/usign compile
  make[3] -C package/libs/ustream-ssl compile
  make[3] -C package/network/config/swconfig compile
  make[3] -C package/network/services/odhcpd compile
  make[3] -C package/network/utils/iwinfo compile
  make[3] -C package/system/mtd compile
  make[3] -C package/network/config/netifd compile
  make[3] -C package/system/ubox compile
  make[3] -C package/network/services/hostapd compile
  make[3] -C package/libs/uclient compile
  make[3] -C package/system/fstools compile
  make[3] -C package/system/procd compile
  make[3] -C package/system/opkg compile
  make[3] -C package/base-files compile
  make[3] -C package/kernel/mac80211 compile
  make[3] -C package/network/utils/iptables compile
  make[3] -C package/network/services/ppp compile
  make[3] -C package/network/config/firewall compile
  make[2] package/install
  make[3] -C package/system/opkg host-install
  make[3] package/preconfig
  make[2] target/install
  make[3] -C target/linux install
  make[6] -C target/linux/brcm47xx/image/lzma-loader clean install
  make[2] package/index
  make[2] checksum


// vmlinux.lzma is now smaller as expected, but TRX images remained big!
$ ls -l build_dir/target-*/linux-*/vmlinux.lzma
-rw-r--r-- 1 zajec users 1140021 Oct 25 10:17 build_dir/target-mipsel_mips32_musl-1.1.15/linux-brcm47xx_legacy/vmlinux.lzma
$ ls -l bin/targets/brcm47xx/legacy/*standard*trx bin/targets/brcm47xx/legacy/*wrt300n*trx
-rw-r--r-- 1 zajec users 3608576 Oct 25 10:17 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-linksys-wrt300n-v1-squashfs.trx
-rw-r--r-- 1 zajec users 4132864 Oct 25 10:17 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-noloader-gz-squashfs.trx
-rw-r--r-- 1 zajec users 3608576 Oct 25 10:17 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-squashfs.trx


// Another try
$ make -j 4
  make[1] world
  make[2] package/cleanup
  make[2] target/compile
  make[3] -C target/linux compile
  make[2] package/compile
  make[3] -C package/system/usign host-compile
  make[3] -C package/kernel/gpio-button-hotplug compile
  make[3] -C package/libs/toolchain compile
  make[3] -C package/utils/lua compile
  make[3] -C package/libs/lzo compile
  make[3] -C package/libs/libnl-tiny compile
  make[3] -C package/libs/libjson-c compile
  make[3] -C package/libs/zlib compile
  make[3] -C package/utils/util-linux compile
  make[3] -C package/system/lede-keyring compile
  make[3] -C package/firmware/b43legacy-firmware compile
  make[3] -C package/firmware/linux-firmware compile
  make[3] -C package/firmware/prism54-firmware compile
  make[3] -C package/network/utils/iw compile
  make[3] -C package/libs/mbedtls compile
  make[3] -C package/libs/polarssl compile
  make[3] -C package/network/ipv6/odhcp6c compile
  make[3] -C package/network/services/dnsmasq compile
  make[3] -C package/network/services/dropbear compile
  make[3] -C package/libs/libpcap compile
  make[3] -C package/network/utils/linux-atm compile
  make[3] -C package/network/utils/resolveip compile
  make[3] -C package/utils/busybox compile
  make[3] -C package/utils/nvram compile
  make[3] -C package/utils/otrx compile
  make[3] -C package/libs/libubox compile
  make[3] -C package/utils/mtd-utils compile
  make[3] -C package/kernel/linux compile
  make[3] -C package/system/uci compile
  make[3] -C package/system/ubus compile
  make[3] -C package/utils/jsonfilter compile
  make[3] -C package/system/usign compile
  make[3] -C package/libs/ustream-ssl compile
  make[3] -C package/network/config/swconfig compile
  make[3] -C package/network/services/odhcpd compile
  make[3] -C package/network/utils/iwinfo compile
  make[3] -C package/system/mtd compile
  make[3] -C package/network/config/netifd compile
  make[3] -C package/system/ubox compile
  make[3] -C package/network/services/hostapd compile
  make[3] -C package/libs/uclient compile
  make[3] -C package/system/fstools compile
  make[3] -C package/system/procd compile
  make[3] -C package/system/opkg compile
  make[3] -C package/base-files compile
  make[3] -C package/kernel/mac80211 compile
  make[3] -C package/network/services/ppp compile
  make[3] -C package/network/utils/iptables compile
  make[3] -C package/network/config/firewall compile
  make[2] package/install
  make[3] -C package/system/opkg host-install
  make[3] package/preconfig
  make[2] target/install
  make[3] -C target/linux install
  make[6] -C target/linux/brcm47xx/image/lzma-loader clean install
  make[2] package/index
  make[2] checksum


// Now TRX files got smaller as expected
$ ls -l build_dir/target-*/linux-*/vmlinux.lzma
-rw-r--r-- 1 zajec users 1140021 Oct 25 10:18 build_dir/target-mipsel_mips32_musl-1.1.15/linux-brcm47xx_legacy/vmlinux.lzma
$ ls -l bin/targets/brcm47xx/legacy/*standard*trx bin/targets/brcm47xx/legacy/*wrt300n*trx
-rw-r--r-- 1 zajec users 3543040 Oct 25 10:18 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-linksys-wrt300n-v1-squashfs.trx
-rw-r--r-- 1 zajec users 4001792 Oct 25 10:18 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-noloader-gz-squashfs.trx
-rw-r--r-- 1 zajec users 3543040 Oct 25 10:18 bin/targets/brcm47xx/legacy/lede-brcm47xx-legacy-standard-squashfs.trx