[PATCH v13 00/48] arm64: Support for Arm CCA in KVM
Gavin Shan
gshan at redhat.com
Wed Mar 25 17:48:41 PDT 2026
Hi Suzuki,
On 3/25/26 8:16 PM, Suzuki K Poulose wrote:
> On 25/03/2026 06:37, Gavin Shan wrote:
>> On 3/21/26 2:45 AM, Steven Price wrote:
[...]
>>
>> In upstream TF-A repository [1], I don't see the config option 'RMM_V1_COMPAT'.
>> would it be something else?
>>
>> [1] git at github.com:ARM-software/arm-trusted-firmware.git (branch: master)
>>
>
> suzuki at ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
> Makefile: RMM_V1_COMPAT \
> Makefile: RMM_V1_COMPAT \
> docs/getting_started/build-options.rst:- ``RMM_V1_COMPAT``: Boolean flag to enable support for RMM v1.x compatibility
> include/services/rmmd_svc.h:#if RMM_V1_COMPAT
> include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
> make_helpers/defaults.mk:RMM_V1_COMPAT := 1
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
> suzuki at ewhatever:trusted-firmware-a$ git log --oneline -1
> 8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge changes from topic "qti_lemans_evk" into integration
> suzuki at ewhatever:trusted-firmware-a$ git remote get-url origin
> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
>
Thanks for the details. It turned out that I used the wrong TF-A repository. In
the proposed repository, I'm able to see the option 'RMM_V1_COMPAT' and the EL3-RMM
interface compatible issue disappears. However, there are more issues popped up.
I build everything manually where the host is emulated by QEMU instead of shrinkwrap
and FVP model. It's used to work well before. Maybe it's time to switch to shinkwrap
and FVP model since device assignment (DA) isn't supported by an emulated host
by QEMU and shrinkwrap and FVP model seems the only option. I need to learn how
to do that later.
There are two issues I can see with the following combinations. Details are provided
like below.
QEMU: https://git.qemu.org/git/qemu.git (branch: stable-9.2)
TF-RMM: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git (branch: topics/rmm-v2.0-poc)
EDK2: git at github.com:tianocore/edk2.git (tag: edk2-stable202411)
TF-A: https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git (branch: master)
HOST: https://git.gitlab.arm.com/linux-arm/linux-cca.git (branch: cca-host/v13)
BUILDROOT: https://github.com/buildroot/buildroot (branch: master)
KVMTOOL: https://gitlab.arm.com/linux-arm/kvmtool-cca (branch: cca/v11)
GUEST: https://github.com/torvalds/linux.git (branch: master)
(1) The emulated host is started by the following command lines.
sudo /home/gshan/sandbox/cca/host/qemu/build/qemu-system-aarch64 \
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off \
-cpu max,x-rme=on -m 8G -smp 8 \
-serial mon:stdio -monitor none -nographic -nodefaults \
-bios /home/gshan/sandbox/cca/host/tf-a/flash.bin \
-kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/Image \
-initrd /home/gshan/sandbox/cca/host/buildroot/output/images/rootfs.cpio.xz \
-device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
-device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \
-device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3 \
-device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4 \
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0 \
-fsdev local,security_model=none,path=/home/gshan/sandbox/cca/guest,id=shr0 \
-netdev tap,id=tap1,script=/etc/qemu-ifup-gshan,downscript=/etc/qemu-ifdown-gshan \
-device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
(2) Issue-1: TF-RMM complains about the root complex list is invalid. This error is
raised in TF-RMM::setup_root_complex_list() where the error code is still set to
0 (SUCCESS) in this failing case. The TF-RMM initialization is terminated early,
but TF-A still thinks the initialization has been completely done.
INFO: BL31: Initializing RMM
INFO: RMM init start.
RMM EL3 compat memory reservation enabled.
Dynamic VA pool base address: 0xc0000000
Reserved 20 pages. Remaining: 3615 pages
Reserve mem: 20 pages at PA: 0x401f2000 (alignment 0x1000)
Static Low VA initialized. xlat tables allocated: 20 used: 7
Reserved 514 pages. Remaining: 3101 pages
Reserve mem: 514 pages at PA: 0x40206000 (alignment 0x1000)
Dynamic Low VA initialized. xlat tables allocated: 514 used: 514
Invalid: Root Complex list <<<<< ERROR
INFO: RMM init end.
(3) Issue-2: The host kernel gets stuck in rmi_check_version() where SMC_RMI_VERSION
is issued to TF-A, but it can't be forwarded to TF-RMM because its initialization
isn't completely done (issue-1).
[ 37.438253] Unpacking initramfs...
[ 37.563460] kvm [1]: nv: 570 coarse grained trap handlers
[ 37.581139] kvm [1]: nv: 664 fine grained trap handlers
<... system becomes stuck here ...>
So my workaround is to skip fetching root complex list from the EL3-RMM manifest data
in TF-RMM::setup_root_complex_list() since it's not provided for the qemu platform by
TF-A. With this workaround, the host can boot up into shell prompt and the guest can
be started by kvmtool.
host$ uname -r
7.0.0-rc1-gavin-gd62aa44b2590
host$ lkvm run --realm -c 2 -m 256 \
-k /mnt/linux/arch/arm64/boot/Image \
-i /mnt/buildroot/output/images/rootfs.cpio.xz
-p earlycon=uart,mmio,0x101000000
Info: # lkvm run -k /mnt/linux/arch/arm64/boot/Image -m 256 -c 2 --name guest-163
Info: Enabling Guest memfd for confidential guest
Warning: The maximum recommended amount of VCPUs is 1
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
[ 0.000000] Linux version 7.0.0-rc2-gavin-g0031c06807cf (gshan at nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC) 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-64.el10) #2 SMP PREEMPT Wed Mar 25 20:28:05 EDT 2026
[ 0.000000] KASLR enabled
:
[ 267.578060] Freeing initrd memory: 4728K
[ 267.921865] Warning: unable to open an initial console.
[ 270.327960] Freeing unused kernel memory: 1792K
[ 270.669368] Run /init as init process
Thanks,
Gavin
More information about the linux-arm-kernel
mailing list