[PATCH v14 29/44] arm64: RMI: Runtime faulting of memory
Gavin Shan
gshan at redhat.com
Fri Jul 3 16:36:13 PDT 2026
On 7/1/26 11:58 PM, Lorenzo Pieralisi wrote:
> On Sun, Jun 28, 2026 at 08:33:01PM +1000, Gavin Shan wrote:
>> On 6/27/26 2:44 AM, Lorenzo Pieralisi wrote:
>>> On Fri, Jun 26, 2026 at 09:43:03PM +1000, Gavin Shan wrote:
>>>> On 6/26/26 6:47 PM, Suzuki K Poulose wrote:
>>>>> On 26/06/2026 08:43, Gavin Shan wrote:
>>>>>> On 6/26/26 1:58 AM, Suzuki K Poulose wrote:
>>>>>>> On 25/06/2026 14:53, Gavin Shan wrote:
>>>>>>>> On 6/6/26 12:35 AM, Lorenzo Pieralisi wrote:
>>>>>>>>> On Fri, Jun 05, 2026 at 06:11:11PM +1000, Gavin Shan wrote:
>>>>>>>>>> On 6/5/26 5:28 PM, Lorenzo Pieralisi wrote:
>>>>>>>>>>> On Fri, Jun 05, 2026 at 04:23:15PM +1000, Gavin Shan wrote:
>>>>>>
>>>>>> [...]
>>>>>>
>>>>>>>>>>
>>>>>>>>>> I tried to rebase Jean's latest QEMU series [1] to upstream QEMU, and found
>>>>>>>>>> that memory slots backed by THP are broken. With THP disabled on the host and
>>>>>>>>>> other fixes (mentioned in my prevous replies) applied on the top of this (v14)
>>>>>>>>>> series, I'm able to boot a realm guest with rebased QEMU series [2], plus more
>>>>>>>>>> fxies on the top.
>>>>>>>>>>
>>>>>>>>>> [1] https://git.codelinaro.org/linaro/dcap/qemu.git (branch: cca/ latest)
>>>>>>>>>> [2] https://git.qemu.org/git/qemu.git (branch: cca/ gavin)
>>>>>>>>>>
>>>>>>>>>> Lorenzo, You may be saying there is someone making QEMU to support ARM/CCA?
>>>>>>>>>
>>>>>>>>> Mathieu and I are working on that yes and with Steven/Suzuki to fix the THP
>>>>>>>>> issues you pointed out above.
>>>>>>>>>
>>>>>>>>>> If so, I'm not sure if there is a QEMU repository for me to try?
>>>>>>>>>
>>>>>>>>> We should be able to submit patches by end of June - we shall let you know
>>>>>>>>> whether we can make something available earlier.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Not sure if there are other known issues in this series. It seems the stage2
>>>>>>>> page fault handling on the shared space isn't working well. In my test, the
>>>>>>>> vring (struct vring_desc) of virtio-net-pci is updated by the guest, and the
>>>>>>>> data isn't seen by QEMU, I'm suspecting if the host-page-frame-number is properly
>>>>>>>> resolved in the s2 page fault handler for shared (unprotected) space.
>>>>>>>>
>>>>>>>> - I rebased Jean's latest qemu branch to the upstream qemu;
>>>>>>>>
>>>>>>>> - On the host, which is emulated by qemu/tcg, the THP (transparent huge page) is
>>>>>>>> disabled.
>>>>>>>>
>>>>>>>> - On the guest, I can see the virtio vring (struct vring_desc) is updated. The
>>>>>>>> S1 page-table entry looks correct because the corresponding physical address
>>>>>>>> 0x10046880000 is a sane shared (unprotected) space address.
>>>>>>>>
>>>>>>>> [ 52.094143] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
>>>>>>>> [ 52.289746] virtqueue_add_desc_split: desc[0]@0xffff000006880000, [00000100b983f000 00000640 0002 0001]
>>>>>>>> [ 52.432150] PTE 0x00e8010046880707 at address 0xffff000006880000
>>>>>>>>
>>>>>>>> - On the host, the s2 page-table-entry is unmapped due to attribute transition (private -> shared).
>>>>>>>> A subsequent S2 page fault is raised against the adress and the s2 page-table-entry is built.
>>>>>>>>
>>>>>>>> [ 109.259077] ====> realm_unmap_shared_range: tracked_unprot_addr=0x10046880000
>>>>>>>> [ 109.260249] realm_unmap_shared_range: unmapped shared range at 0x10046880000
>>>>>>>> [ 109.317786] realm_unmap_shared_range: unmapped shared range at 0x10046880000
>>>>>>>> [ 109.629939] ====> kvm_handle_guest_abort: fault_ipa=0x10046880000, esr=0x92000007
>>>>>>>> [ 109.630245] realm_map_non_secure: ipa=0x10046880000, pfn=0xb8b59, size=0x1000, prot=0xf
>>>>>>>> [ 109.630331] realm_map_non_secure: ipa=0x10046880000, ipa_top=0x10046881000, flags=0x1e0001, range_desc=0xb8b59004
>>>>>>>
>>>>>>> Are you able to correlate the order of the transitions and the Guest
>>>>>>> access with RMM log ? We haven't seen this from our end. We are aware
>>>>>>> of permission fault issues with Unprotected IPA when backing the memslot
>>>>>>> with MAP_PRIVATE areas. But this looks different.
>>>>>>>
>>>>>>> Lorenzo, have you run into this ?
>>>>>>>
>>>>>>
>>>>>> It's hard to correlate the order since the logs are collected from two separate
>>>>>> consoles. For the write permission, I add code to the host where the permission
>>>>>> is always added for all s2 page faults in the shared space. Otherwise, qemu can
>>>>>> be killed by -EFAULT or similar error.
>>>>>
>>>>> This is the problem. We can't add WRITE permission by default. I believe
>>>>> you may have MAP_PRIVATE mapping and it has to be mapped as READ only
>>>>> and on a permission fault, we replace it with a writable page. By
>>>>> overriding the WRITE permission, you let the guest write to a page
>>>>> that may not be seen by the VMM.
>>>>>
>>>>> We identified this as a bug in the KVM driver in this series (reported
>>>>> by Lorenzo) and there is a corresponding tf-RMM change that is required
>>>>> to get this working. So, please could you wait until the next series
>>>>> when this will be addressed ? Or you could switch to using MAP_SHARED
>>>>> for the "shared" memory in the memslot.
>>>>>
>>>>
>>>> Exactly. the syntax for MAP_PRIVATE is broken if the write permission is
>>>> enforced for a read fault in the shared space. In my case, the host page can
>>>> be the zero page and eventually multiple s2 page-table entries (for multiple
>>>> unprotected or shared pages) point to the zero page. It's why clearing the
>>>> 3rd queue (Ctrl queue) also clears the first queue (Rx queue) in my case.
>>>>
>>>> Yes, this issue can be avoid by using a shared memory backend in qemu, something
>>>> like below. With this, I'm able to see virtio-net-pci starts to work...
>>>>
>>>> -object memory-backend-ram,id=mem0,size=2G,share=yes
>>>
>>> Yes, as Suzuki said that's what we have been fixing. QEmu patches
>>> will be on the mailing lists very shortly - the KVM/tf-RMM fixes
>>> to make MAP_PRIVATE work will be included in the next posting.
>>>
>>> Feel free to drop your QEmu command line so that I can give it
>>> a shot and check whether the fixes solve the problem you hit
>>> (I think so because that's precisely the kind of issue I got
>>> into when I started debugging THP/MAP_PRIVATE but it is better
>>> to check).
>>>
>>
>> The virtio-net-pci doesn't work with the following command lines. The guest
>> kernel image is built from upstream kernel (v7.1.rc7).
>>
>> qemu-system-aarch64 -enable-kvm -object rme-guest,id=rme0, \
>> -machine virt,gic-version=3,confidential-guest-support=rme0 \
>> -cpu host,pmu=off \
>> -smp maxcpus=2,cpus=2,sockets=1,clusters=1,cores=1,threads=2 \
>> -m 2G -object memory-backend-ram,id=mem0,size=2G \
>> -numa node,nodeid=0,cpus=0-1,memdev=mem0 \
>> -serial mon:stdio -monitor none -nographic -nodefaults \
>> -kernel /mnt/linux/arch/arm64/boot/Image \
>> -initrd /mnt/buildroot/output/images/rootfs.cpio.xz \
>> -append earlycon=pl011,mmio,0x10009000000 \
>> -device pcie-root-port,bus=pcie.0,chassis=1,id=pcie.1 \
>> -device pcie-root-port,bus=pcie.0,chassis=2,id=pcie.2 \
>> -device pcie-root-port,bus=pcie.0,chassis=3,id=pcie.3 \
>> -device pcie-root-port,bus=pcie.0,chassis=4,id=pcie.4 \
>> -netdev tap,id=tap1,vhost=on,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
>> -device virtio-net-pci,bus=pcie.2,netdev=tap1,mac=b8:3f:d2:1d:3e:c0
>
> Tested with private RAM backend, THP left enabled (and additional KVM and
> RMM patches that will be included in v15) looks like the net interface
> comes up cleanly.
>
> Probably it is best to sync up on the upcoming v15 posting that I am
> testing in the background to make sure you don't spend more time chasing
> it.
>
Thanks for your confirmation. For every respin of the arm/cca host series,
I try to give it a test to make sure nothing gets broken in my coverage.
For this specific revision (v14), more time was actually spent to rebase
Jean's qemu (branch: cca/latest) to upstream so that I can do the tests.
Lets wait for a while to see what we will have in v15 and the qemu series
you're going to post, and I will give it another try.
One unrelated question is 64KB granule size, it's not supported by upstream
TF-RMM "main" or "topics/rmm-v2.0-poc_2" branch, not sure about the plan
when this will be supported in TF-RMM. Since 64KB isn't supported by TF-RMM,
I was unable to test 64KB page size.
Upstream EDK2 doesn't support realm guest yet even some of related commits
have been merged. So I still use linaro's one at "https://git.codelinaro.org
/linaro/dcap/edk2 (branch: cca/latest).
Thanks,
Gavin
> Thanks,
> Lorenzo
>
>>
>> The virtio-net-pci starts to work with the shareable memory-backend.
>>
>> -object memory-backend-ram,id=mem0,size=2G,share=yes
>>
>> Note that THP is disabled on my host.
>>
>> root at host:~# cat /sys/kernel/mm/transparent_hugepage/enabled
>> always madvise [never]
>
More information about the linux-arm-kernel
mailing list