[PATCH v13 00/48] arm64: Support for Arm CCA in KVM
Suzuki K Poulose
suzuki.poulose at arm.com
Thu Mar 26 04:22:50 PDT 2026
Hi Gavin,
On 26/03/2026 00:48, Gavin Shan wrote:
> Hi Suzuki,
>
> On 3/25/26 8:16 PM, Suzuki K Poulose wrote:
>> On 25/03/2026 06:37, Gavin Shan wrote:
>>> On 3/21/26 2:45 AM, Steven Price wrote:
>
> [...]
>
>>>
>>> In upstream TF-A repository [1], I don't see the config option
>>> 'RMM_V1_COMPAT'.
>>> would it be something else?
>>>
>>> [1] git at github.com:ARM-software/arm-trusted-firmware.git (branch:
>>> master)
>>>
>>
>> suzuki at ewhatever:trusted-firmware-a$ git grep RMM_V1_COMPAT
>> Makefile: RMM_V1_COMPAT \
>> Makefile: RMM_V1_COMPAT \
>> docs/getting_started/build-options.rst:- ``RMM_V1_COMPAT``: Boolean
>> flag to enable support for RMM v1.x compatibility
>> include/services/rmmd_svc.h:#if RMM_V1_COMPAT
>> include/services/rmmd_svc.h:#endif /* RMM_V1_COMPAT */
>> make_helpers/defaults.mk:RMM_V1_COMPAT := 1
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if !RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_main.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> services/std_svc/rmmd/rmmd_rmm_lfa.c:#if RMM_V1_COMPAT
>> suzuki at ewhatever:trusted-firmware-a$ git log --oneline -1
>> 8dae0862c (HEAD, origin/master, origin/integration, origin/HEAD) Merge
>> changes from topic "qti_lemans_evk" into integration
>> suzuki at ewhatever:trusted-firmware-a$ git remote get-url origin
>> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
>>
>
> Thanks for the details. It turned out that I used the wrong TF-A
> repository. In
> the proposed repository, I'm able to see the option 'RMM_V1_COMPAT' and
> the EL3-RMM
> interface compatible issue disappears. However, there are more issues
> popped up.
>
> I build everything manually where the host is emulated by QEMU instead
> of shrinkwrap
> and FVP model. It's used to work well before. Maybe it's time to switch
> to shinkwrap
> and FVP model since device assignment (DA) isn't supported by an
> emulated host
> by QEMU and shrinkwrap and FVP model seems the only option. I need to
> learn how
> to do that later.
Thanks for the update. Yes, QEMU TF-RMM support is in progress, @Mathieu
Poirier is looking into it
>
> There are two issues I can see with the following combinations. Details
> are provided
> like below.
>
> QEMU: https://git.qemu.org/git/
> qemu.git (branch: stable-9.2)
> TF-RMM: https://git.trustedfirmware.org/TF-RMM/tf-
> rmm.git (branch: topics/rmm-v2.0-poc)
> EDK2: git at github.com:tianocore/
> edk2.git (tag: edk2-stable202411)
> TF-A: https://git.trustedfirmware.org/TF-A/trusted-firmware-
> a.git (branch: master)
> HOST: https://git.gitlab.arm.com/linux-arm/linux-
> cca.git (branch: cca-host/v13)
> BUILDROOT: https://github.com/buildroot/
> buildroot (branch: master)
> KVMTOOL: https://gitlab.arm.com/linux-arm/kvmtool-
> cca (branch: cca/v11)
> GUEST: https://github.com/torvalds/
> linux.git (branch: master)
>
> (1) The emulated host is started by the following command lines.
>
> sudo /home/gshan/sandbox/cca/host/qemu/build/qemu-system-
> aarch64 \
> -M virt,virtualization=on,secure=on,gic-
> version=3,acpi=off \
> -cpu max,x-rme=on -m 8G -smp
> 8 \
> -serial mon:stdio -monitor none -nographic -
> nodefaults \
> -bios /home/gshan/sandbox/cca/host/tf-a/
> flash.bin \
> -kernel /home/gshan/sandbox/cca/host/linux/arch/arm64/boot/
> Image \
> -initrd /home/gshan/sandbox/cca/host/buildroot/output/images/
> rootfs.cpio.xz \
> -device pcie-root-
> port,bus=pcie.0,chassis=1,id=pcie.1 \
> -device pcie-root-
> port,bus=pcie.0,chassis=2,id=pcie.2 \
> -device pcie-root-
> port,bus=pcie.0,chassis=3,id=pcie.3 \
> -device pcie-root-
> port,bus=pcie.0,chassis=4,id=pcie.4 \
> -device virtio-9p-
> device,fsdev=shr0,mount_tag=shr0 \
> -fsdev local,security_model=none,path=/home/gshan/sandbox/cca/
> guest,id=shr0 \
> -netdev tap,id=tap1,script=/etc/qemu-ifup-gshan,downscript=/etc/
> qemu-ifdown-gshan \
> -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:f1
>
> (2) Issue-1: TF-RMM complains about the root complex list is invalid.
> This error is
> raised in TF-RMM::setup_root_complex_list() where the error code is
> still set to
> 0 (SUCCESS) in this failing case. The TF-RMM initialization is
> terminated early,
> but TF-A still thinks the initialization has been completely done.
>
> INFO: BL31: Initializing RMM
> INFO: RMM init start.
> RMM EL3 compat memory reservation enabled.
> Dynamic VA pool base address: 0xc0000000
> Reserved 20 pages. Remaining: 3615 pages
> Reserve mem: 20 pages at PA: 0x401f2000 (alignment 0x1000)
> Static Low VA initialized. xlat tables allocated: 20 used: 7
> Reserved 514 pages. Remaining: 3101 pages
> Reserve mem: 514 pages at PA: 0x40206000 (alignment 0x1000)
> Dynamic Low VA initialized. xlat tables allocated: 514 used: 514
> Invalid: Root Complex list
> <<<<< ERROR
> INFO: RMM init end.
>
> (3) Issue-2: The host kernel gets stuck in rmi_check_version() where
> SMC_RMI_VERSION
> is issued to TF-A, but it can't be forwarded to TF-RMM because its
> initialization
> isn't completely done (issue-1).
>
> [ 37.438253] Unpacking initramfs...
> [ 37.563460] kvm [1]: nv: 570 coarse grained trap handlers
> [ 37.581139] kvm [1]: nv: 664 fine grained trap handlers
> <... system becomes stuck here ...>
>
> So my workaround is to skip fetching root complex list from the EL3-RMM
> manifest data
> in TF-RMM::setup_root_complex_list() since it's not provided for the
> qemu platform by
^^ This may have to do with the RMM<->TF-A Manifest changes
> TF-A. With this workaround, the host can boot up into shell prompt and
> the guest can
> be started by kvmtool.
>
> host$ uname -r
> 7.0.0-rc1-gavin-gd62aa44b2590
> host$ lkvm run --realm -c 2 -m 256 \
> -k /mnt/linux/arch/arm64/boot/Image \
> -i /mnt/buildroot/output/images/rootfs.cpio.xz
> -p earlycon=uart,mmio,0x101000000
> Info: # lkvm run -k /mnt/linux/arch/arm64/boot/Image -m 256 -c 2 --
> name guest-163
> Info: Enabling Guest memfd for confidential guest
> Warning: The maximum recommended amount of VCPUs is 1
> [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x000f0510]
> [ 0.000000] Linux version 7.0.0-rc2-gavin-g0031c06807cf
> (gshan at nvidia-grace-hopper-01.khw.eng.bos2.dc.redhat.com) (gcc (GCC)
> 14.3.1 20251022 (Red Hat 14.3.1-4), GNU ld version 2.41-64.el10) #2 SMP
> PREEMPT Wed Mar 25 20:28:05 EDT 2026
> [ 0.000000] KASLR enabled
> :
> [ 267.578060] Freeing initrd memory: 4728K
> [ 267.921865] Warning: unable to open an initial console.
> [ 270.327960] Freeing unused kernel memory: 1792K
> [ 270.669368] Run /init as init process
>
Cool, thanks!
Suzuki
> Thanks,
> Gavin
>
More information about the linux-arm-kernel
mailing list