[PATCH] kexec: return -ENOEXEC from image probe functions on mismatch

[Mukesh Pilaniya]

Hi Pratyush,
On 14/08/26 7:43 pm, Pratyush Yadav wrote:
> On Thu, Aug 13 2026, Mukesh Pilaniya wrote:
> 
>> Several kexec_file_load() image probe functions return -EINVAL when
>> they do not recognize the image format.  A probe function that rejects
>> an image should return -ENOEXEC to indicate that the image is not a
>> recognized executable format.  -EINVAL implies a problem with the
>> syscall parameters, not with image recognition.
>>
>> kexec_image_probe_default() iterates through registered loaders and
>> returns the last probe's error code to the caller.  That error
>> propagates as the kexec_file_load() return value to userspace.
>> Returning -EINVAL from a probe when no loader matches is semantically
>> incorrect and misleads userspace about the nature of the failure.
>>
>> Return -ENOEXEC from all probe functions and their helpers when the
>> image format is not recognized.
> 
> Sounds fine in principle but can you please also share what the real
> problem you face is and how changing these return codes helps? These
> error codes are uAPI and while we _can_ change them as long as we don't
> break something, there should be a clear motivation for doing so.
> 
> [...]
> 
While debugging a misleading error on s390x where kexec -s reported
"syscall kexec_file_load not available" instead of the actual EINVAL
from a kernel command line that exceeded the architecture limit, we
traced the problem to the kexec-tools userspace utility treating
EINVAL the same as ENOSYS and ENOEXEC -- as a signal to silently fall
back to kexec_load().

kexec-tools supports two syscalls: kexec_file_load() and the older
kexec_load(). With -a (the default), it tries kexec_file_load()
first and falls back to kexec_load() when the syscall is not
implemented (ENOSYS) or the kernel does not have a loader for the
image format. With -s, it uses kexec_file_load() only with no
fallback.

When the kernel returns -EINVAL it means something went wrong while
loading the image, not that the syscall is missing or the image
format is unrecognized. kexec-tools should not fall back to the
older syscall in that case. However, some kernel probe functions
currently return -EINVAL when the image header does not match,
instead of returning -ENOEXEC. Keeping EINVAL in the fallback set
to accommodate these probes has the side effect of also hiding
genuine loading errors like an oversized command line.

kexec-tools should only fall back when kexec_file_load() is not
implemented or does not have a matching loader -- not when something
goes wrong during load.

The fix on the kexec-tools side is to remove EINVAL from the fallback
set, but that requires the kernel to be clean first -- probe functions
must return -ENOEXEC when they do not recognize an image format, not -EINVAL.

kexec-tools patch:
https://lore.kernel.org/all/20260814075329.30203-1-mpilaniy@redhat.com/

A review of all kexec_file_ops.probe implementations found that arm64
image_probe(), riscv image_probe(), and loongarch efi_kexec_probe()
return -EINVAL where they should return -ENOEXEC. x86 bzImage64_probe()
and s390 s390_elf_probe() already use -ENOEXEC correctly.