kexec failure with Xen 4.19-rc4 and 4.20-dev on linux host

[Juergen Gross]

On 05.08.24 08:01, Jan Beulich wrote:
> On 04.08.2024 15:17, A Kundu wrote:
>> On 8/2/24 13:25, Jan Beulich wrote:
>>   > On 02.08.2024 09:28, A Kundu wrote:
>>   >> On 8/2/24 09:06, Baoquan He wrote:
>>   >>> On 07/31/24 at 06:34pm, A Kundu wrote:
>>   >>>> I am experiencing issues using kexec to load Xen 4.17(debian's apt
>> version),
>>   >>>> Xen 4.19-rc4 (compiled from source) and 4.20-dev (compiled from
>> source) on a
>>   >>>> debian host.
>>   >>> You should CC this to XEN dev list so that XEN dev knows this and may
>>   >>> provide help. Not everyone is interested in and knows XEN.
>>   >>>
>>   >>>> System information:
>>   >>>> $ uname -a
>>   >>>> Linux host 6.9.10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1
>> (2024-07-19)
>>   >>>> x86_64 GNU/Linux
>>   >>>>
>>   >>>> $ kexec --version # compiled from source tarball with ./configure
>> --with-xen
>>   >>>> kexec-tools 2.0.29
>>   >>>>
>>   >>>> Steps to reproduce:
>>   >>>>
>>   >>>> 1. Set variables:
>>   >>>>
>>   >>>> XEN_HYPERVISOR="/boot/xen.gz"
>>   >>>> XEN_CMD="dom0_mem=6G dom0_max_vcpus=6 dom0_vcpus_pin cpufreq=xen"
>>   >>>>
>>   >>>> 2. Attempt to load Xen 4.19-rc4:
>>   >>>>
>>   >>>> # kexec -l "$XEN_HYPERVISOR" --command-line="$XEN_CMD"
>>   >>>> Could not find a free area of memory of 0x3b6001 bytes...
>>   >>>> elf_exec_build_load_relocatable: ELF exec load failed
>>   >>>>
>>   >>>> 3. Attempt to load Xen 4.20-dev:
>>   >>>>
>>   >>>> # kexec -l "$XEN_HYPERVISOR" --command-line="$XEN_CMD"
>>   >>>> Could not find a free area of memory of 0x3f8001 bytes...
>>   >>>> elf_exec_build_load_relocatable: ELF exec load failed
>>   >>>>
>>   >>>> 4. Attempt to load Xen 4.17 (from debian rrepositories):
>>   >>>> # kexec -l /boot/xen-4.17-amd64.gz --command-line="$XEN_CMD"
>>   >>>> Could not find a free area of memory of 0x3b4001 bytes...
>>   >>>> elf_exec_build_load_relocatable: ELF exec load failed
>>   >
>>   > And with all of them saying effectively the same, did you verify you
>>   > actually have a sufficiently large area reserved? The obvious
>>   > place for you to look at is Xen's boot log (obtained via serial
>>   > console or "xl dmesg" immediately after booting the system). If you
>>   > find everything as expected there, ...
>>   >
>>   >>>> If you need any further information to investigate this problem,
>>   >>>> please let me know.
>>   >
>>   > ... please provide that boot log.
>>
>> I have also followed up on your suggestion to check the Xen boot log
>> using "xl dmesg", but unfortunately, I received the following error:
>>
>> xencall: error: Could not obtain handle on privileged command interface:
>> No such file or directory
>> libxl: error: libxl.c:102:libxl_ctx_alloc: cannot open libxc handle: No
>> such file or directory
>> cannot init xl context
>>
>> This indicates that Xen did not boot successfully, so there are no logs
>> available.
> 
> The fact that you have Dom0 up makes clear that Xen booted okay(ish). The
> fact that you get "No such file or directory" from xencall suggests you
> either didn't load the xen-privcmd driver (normally arrangements are made
> by distros for this to happen automatically), or you didn't even build it.

The messages seen don't indicate that Xen booted okay(ish). I get the same
messages when having booted the Linux kernel on bare metal without Xen.

> 
>>   > And with all of them saying effectively the same, did you verify you
>>   > actually have a sufficiently large area reserved? The obvious
>>   > place for you to look at is Xen's boot log (obtained via serial
>>   > console or "xl dmesg" immediately after booting the system). If you
>>   > find everything as expected there, ...
>>   >
>>
>> In an attempt to resolve the memory allocation issue, I have tried the
>> following:
>>
>> Added a crashkernel=<size>@<offset> parameter to the host kernel command
>> line to reserve a dedicated memory region for kexec, and attempted to
>> load Xen into that area.
> 
> That was a remote guess of mine. This command line option is meaningless
> when running under Xen. The reservation needs to be done in Xen.

Just one thing to add here: what do you want to accomplish by kexec-ing into
Xen? You need to specify a dom0 kernel and probably the dom0's initrd, too.

Or do you have configured your dom0 to work without an initrd? Even in this
case you'd need to pass on the dom0 to Xen via the kexec command.


Juergen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xB0DE9DD628BF132F.asc
Type: application/pgp-keys
Size: 3683 bytes
Desc: OpenPGP public key
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20240805/ce3605e9/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature.asc
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20240805/ce3605e9/attachment-0001.sig>