The conflict between PHYSICAL_START and MIN_KERNEL_LOAD_ADDR

[Chris Koch]

My apologies, I didn't realize x86@ wasn't a mailing list earlier.
Adding the list.

On Thu, Dec 14, 2023 at 6:37 PM Chris Koch <chrisko at google.com> wrote:
>
> Hi all,
>
> we ran into some memory clobbering on kernel boot, and there seems to
> be a bit of a conflict between the documentation and reality, and
> CONFIG_PHYSICAL_START (default 0x1 000 000) and MIN_KERNEL_LOAD_ADDR
> (default 0x100 000).
>
> First observation: bzimage's pref_address is listed as "preferred" in
> the boot documentation for relocatable kernels, but what is not
> mentioned is that if the kernel is loaded below pref_address, it will
> relocate itself there [1]. The default CONFIG_PHYSICAL_START seems to
> be 0x1000000, and a review of bootloaders shows that in every case but
> GRUB a kernel will get loaded below that, and then relocate itself
> there. (ipxe 0x10000 or 0x100000 depending on config, kexec-tools
> 0x100000, kexec_file 0x100000)
>
> This means in all default cases except GRUB, the kernel will relocate
> itself and decompress in an area of memory it was not intended for,
> and which may hold reserved data in the memory map. (This is what
> happened in our case. It was clobbering reserved stuff.)
>
> This suggests the following:
> 1) This pref_address behavior should be documented in boot.rst.
> Something like "A relocatable kernel will relocate itself to
> pref_address (which is equal to CONFIG_PHYSICAL_START) if it was
> loaded below that address."
>
> 2) Documentation for PHYSICAL_START should be updated. It states
>
> > If kernel is a not relocatable (CONFIG_RELOCATABLE=n) then
> > bzImage will decompress itself to above physical address and
> > run from there. Otherwise, bzImage will run from the address where
> > it has been loaded by the boot loader and will ignore above physical
> > address.
>
> 3) Some patches should be sent to ipxe, kexec-tools and kexec_file to
> respect pref_address as a minimum address, not just a preferred
> address.
>
> 4) The default values of MIN_KEXEC_LOAD_ADDR and CONFIG_PHYSICAL_START
> should align to reduce the chances of this happening for users who
> aren't messing with these values. Given what current boot loaders do,
> should CONFIG_PHYSICAL_START be lowered to 0x100000, at least for
> relocatable kernels? This might be the most controversial of the
> changes.
>
> 5) Should PHYSICAL_START be ignored in relocatable kernels like its
> documentation suggests? Is the relocation above that unnecessary?
>
> Regardless of which change we make for the future, existing boot
> loaders & kernels conflict in their behavior for relocatable kernels.
> What is most likely to fix this issue for users & prevent breakage of
> existing workflows?
>
> If I see agreement on 1 to 3, I will start sending patches. Would love
> to hear opinions on 4 & 5.
>
> Thanks,
> Chris
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/x86/boot/compressed/head_64.S?h=linux-6.6.y#n333