Bug#1033301: linux: arm64 kernel size increased from 31 to 39 MB, causing u-boot-rpi to fail

Fri Mar 24 13:58:35 PDT 2023

Adding u-boot maintainers for rpi (Matthias Brugger, Peter Robinson)
platforms and u-boot list to CC.

On 2023-03-22, Salvatore Bonaccorso wrote:
> Thanks for tracking this down. I would like to loop in Masahiro and
> upstream to see if something can/should be done on upstream side.
>
> Masahiro, in short, upstream change 994b7ac1697b ("arm64: remove
> special treatment for the link order of head.o") (which got backported
> as well to 6.1.14) caused the vmlinuz size to icrease significantly,
> causing boot failures on Raspberry Pi 3 Model B Plus with u-boot
> parameters previously working. Full quoting the Debian report below

In general it would be nice to not have the kernel grow nearly 25% in
size from a single commit; was that an expected outcome from the above
upstream change? Was the "special treament" originally done to keep the
kernel size down?

As for u-boot, It seems u-boot might need to update the various load
addresses for the kernel, device tree and ramdisk at some point
regardless of weather this particular issue gets fixed in the kernel, as
the kernel will likely continue to grow a bit over time...

Aurelien Jarno included a patch referenced below which bumps the
tolerances in u-boot from 36MB to 42MB.

> On Tue, Mar 21, 2023 at 11:11:13PM +0100, Aurelien Jarno wrote:
>> Following the upgrade of the kernel from 6.1.12-1 to 6.1.15-1 on a
>> Raspberry Pi 3 Model B Plus, u-boot (from the u-boot-rpi package) failed
>> to boot with:
>> 
>> | 40175552 bytes read in 1695 ms (23 MiB/s)
>> | 43794863 bytes read in 1817 ms (23 MiB/s)
>> | Moving Image from 0x80000 to 0x200000, end=2990000
>> | ERROR: RD image overlaps OS image (OS=0x200000..0x2990000)
>> 
>> I tracked the issue to a significant increase of the kernel size between
>> version 6.1.12-1 and 6.15-1:
>> 
>> | 31492   /boot/vmlinuz-6.1.0-5-arm64
>> | 39236   /boot/vmlinuz-6.1.0-6-arm64
>> 
>> This is more than the 36MB that is allowed by u-boot with the default
>> load addresses. A workaround is to shift the load addresses at the
>> u-boot level as in the attached patch.
>> 
>> I have tracked issue on the upstream kernel side to the following commit
>> on the stable tree:
>> 
>> | commit 3e3e4d234d46e48480a7c7c35399fa811182e8ef
>> | Author: Masahiro Yamada <masahiroy at kernel.org>
>> | Date:   Thu Oct 13 08:35:00 2022 +0900
>> | 
>> |     arm64: remove special treatment for the link order of head.o
>> |     
>> |     commit 994b7ac1697b4581b7726d2ac64321e3c840229b upstream.
>> |     
>> |     In the previous discussion (see the Link tag), Ard pointed out that
>> |     arm/arm64/kernel/head.o does not need any special treatment - the only
>> |     piece that must appear right at the start of the binary image is the
>> |     image header which is emitted into .head.text.
>> |     
>> |     The linker script does the right thing to do. The build system does
>> |     not need to manipulate the link order of head.o.
>> |     
>> |     Link: https://lore.kernel.org/lkml/CAMj1kXH77Ja8bSsq2Qj8Ck9iSZKw=1F8Uy-uAWGVDm4-CG=EuA@mail.gmail.com/
>> |     Suggested-by: Ard Biesheuvel <ardb at kernel.org>
>> |     Signed-off-by: Masahiro Yamada <masahiroy at kernel.org>
>> |     Reviewed-by: Nicolas Schier <nicolas at fjasle.eu>
>> |     Link: https://lore.kernel.org/r/20221012233500.156764-1-masahiroy@kernel.org
>> |     Signed-off-by: Will Deacon <will at kernel.org>
>> |     Signed-off-by: Tom Saeger <tom.saeger at oracle.com>
>> |     Signed-off-by: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
>> 
>> The problem is still reproducible on Linus' master.
>> 
>> I am reporting the bug to the linux package as I believed there is no
>> real reason for such an increase in the kernel size. In case I missed
>> something and this is actually wanted, the bug can be reassigned to the
>> u-boot package.
>> 
>> Regards
>> Aurelien
>
>> --- u-boot-2023.01+dfsg.orig/include/configs/rpi.h
>> +++ u-boot-2023.01+dfsg/include/configs/rpi.h
>> @@ -95,32 +95,32 @@
>>   *   text_offset bytes (specified in the header of the Image) into a 2MB
>>   *   boundary. The 'booti' command relocates the image if necessary. Linux uses
>>   *   a default text_offset of 0x80000.  In summary, loading at 0x80000
>> - *   satisfies all these constraints and reserving memory up to 0x02400000
>> - *   permits fairly large (roughly 36M) kernels.
>> + *   satisfies all these constraints and reserving memory up to 0x02a00000
>> + *   permits fairly large (roughly 42M) kernels.
>>   *
>>   * scriptaddr and pxefile_addr_r can be pretty much anywhere that doesn't
>>   * conflict with something else. Reserving 1M for each of them at
>> - * 0x02400000-0x02500000 and 0x02500000-0x02600000 should be plenty.
>> + * 0x02a00000-0x02b00000 and 0x02c00000-0x02d00000 should be plenty.
>>   *
>>   * On ARM, both the DTB and any possible initrd must be loaded such that they
>>   * fit inside the lowmem mapping in Linux. In practice, this usually means not
>>   * more than ~700M away from the start of the kernel image but this number can
>>   * be larger OR smaller depending on e.g. the 'vmalloc=xxxM' command line
>>   * parameter given to the kernel. So reserving memory from low to high
>> - * satisfies this constraint again. Reserving 1M at 0x02600000-0x02700000 for
>> - * the DTB leaves rest of the free RAM to the initrd starting at 0x02700000.
>> + * satisfies this constraint again. Reserving 1M at 0x02c00000-0x02d00000 for
>> + * the DTB leaves rest of the free RAM to the initrd starting at 0x02d00000.
>>   * Even with the smallest possible CPU-GPU memory split of the CPU getting
>> - * only 64M, the remaining 25M starting at 0x02700000 should allow quite
>> + * only 64M, the remaining 19M starting at 0x02d00000 should allow quite
>>   * large initrds before they start colliding with U-Boot.
>>   */
>>  #define ENV_MEM_LAYOUT_SETTINGS \
>>  	"fdt_high=" FDT_HIGH "\0" \
>>  	"initrd_high=" INITRD_HIGH "\0" \
>>  	"kernel_addr_r=0x00080000\0" \
>> -	"scriptaddr=0x02400000\0" \
>> -	"pxefile_addr_r=0x02500000\0" \
>> -	"fdt_addr_r=0x02600000\0" \
>> -	"ramdisk_addr_r=0x02700000\0"
>> +	"scriptaddr=0x02a00000\0" \
>> +	"pxefile_addr_r=0x02b00000\0" \
>> +	"fdt_addr_r=0x02c00000\0" \
>> +	"ramdisk_addr_r=0x02d00000\0"
>>  
>>  #if CONFIG_IS_ENABLED(CMD_MMC)
>>  	#define BOOT_TARGET_MMC(func) \

Bumping from 36MB to 42MB seems address the issue at hand, although
would it make sense to bump even higher... or is that starting to hit
other limitations?

Thanks everyone!

live well,
  vagrant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 227 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20230324/da390208/attachment.sig>