[PATCH v14 29/44] arm64: RMI: Runtime faulting of memory
Suzuki K Poulose
suzuki.poulose at arm.com
Fri Jun 26 02:04:56 PDT 2026
On 26/06/2026 09:47, Suzuki K Poulose wrote:
> On 26/06/2026 08:43, Gavin Shan wrote:
>> On 6/26/26 1:58 AM, Suzuki K Poulose wrote:
>>> On 25/06/2026 14:53, Gavin Shan wrote:
>>>> On 6/6/26 12:35 AM, Lorenzo Pieralisi wrote:
>>>>> On Fri, Jun 05, 2026 at 06:11:11PM +1000, Gavin Shan wrote:
>>>>>> On 6/5/26 5:28 PM, Lorenzo Pieralisi wrote:
>>>>>>> On Fri, Jun 05, 2026 at 04:23:15PM +1000, Gavin Shan wrote:
>>
>> [...]
>>
>>>>>>
>>>>>> I tried to rebase Jean's latest QEMU series [1] to upstream QEMU,
>>>>>> and found
>>>>>> that memory slots backed by THP are broken. With THP disabled on
>>>>>> the host and
>>>>>> other fixes (mentioned in my prevous replies) applied on the top
>>>>>> of this (v14)
>>>>>> series, I'm able to boot a realm guest with rebased QEMU series
>>>>>> [2], plus more
>>>>>> fxies on the top.
>>>>>>
>>>>>> [1] https://git.codelinaro.org/linaro/dcap/qemu.git (branch: cca/
>>>>>> latest)
>>>>>> [2] https://git.qemu.org/git/qemu.git (branch: cca/
>>>>>> gavin)
>>>>>>
>>>>>> Lorenzo, You may be saying there is someone making QEMU to support
>>>>>> ARM/CCA?
>>>>>
>>>>> Mathieu and I are working on that yes and with Steven/Suzuki to fix
>>>>> the THP
>>>>> issues you pointed out above.
>>>>>
>>>>>> If so, I'm not sure if there is a QEMU repository for me to try?
>>>>>
>>>>> We should be able to submit patches by end of June - we shall let
>>>>> you know
>>>>> whether we can make something available earlier.
>>>>>
>>>>
>>>> Not sure if there are other known issues in this series. It seems
>>>> the stage2
>>>> page fault handling on the shared space isn't working well. In my
>>>> test, the
>>>> vring (struct vring_desc) of virtio-net-pci is updated by the guest,
>>>> and the
>>>> data isn't seen by QEMU, I'm suspecting if the host-page-frame-
>>>> number is properly
>>>> resolved in the s2 page fault handler for shared (unprotected) space.
>>>>
>>>> - I rebased Jean's latest qemu branch to the upstream qemu;
>>>>
>>>> - On the host, which is emulated by qemu/tcg, the THP (transparent
>>>> huge page) is
>>>> disabled.
>>>>
>>>> - On the guest, I can see the virtio vring (struct vring_desc) is
>>>> updated. The
>>>> S1 page-table entry looks correct because the corresponding
>>>> physical address
>>>> 0x10046880000 is a sane shared (unprotected) space address.
>>>>
>>>> [ 52.094143] software IO TLB: Memory encryption is active and
>>>> system is using DMA bounce buffers
>>>> [ 52.289746] virtqueue_add_desc_split:
>>>> desc[0]@0xffff000006880000, [00000100b983f000 00000640 0002 0001]
>>>> [ 52.432150] PTE 0x00e8010046880707 at address 0xffff000006880000
>>>>
>>>> - On the host, the s2 page-table-entry is unmapped due to attribute
>>>> transition (private -> shared).
>>>> A subsequent S2 page fault is raised against the adress and the
>>>> s2 page-table-entry is built.
>>>>
>>>> [ 109.259077] ====> realm_unmap_shared_range:
>>>> tracked_unprot_addr=0x10046880000
>>>> [ 109.260249] realm_unmap_shared_range: unmapped shared range at
>>>> 0x10046880000
>>>> [ 109.317786] realm_unmap_shared_range: unmapped shared range at
>>>> 0x10046880000
>>>> [ 109.629939] ====> kvm_handle_guest_abort:
>>>> fault_ipa=0x10046880000, esr=0x92000007
>>>> [ 109.630245] realm_map_non_secure: ipa=0x10046880000,
>>>> pfn=0xb8b59, size=0x1000, prot=0xf
>>>> [ 109.630331] realm_map_non_secure: ipa=0x10046880000,
>>>> ipa_top=0x10046881000, flags=0x1e0001, range_desc=0xb8b59004
>>>
>>> Are you able to correlate the order of the transitions and the Guest
>>> access with RMM log ? We haven't seen this from our end. We are aware
>>> of permission fault issues with Unprotected IPA when backing the memslot
>>> with MAP_PRIVATE areas. But this looks different.
>>>
>>> Lorenzo, have you run into this ?
>>>
>>
>> It's hard to correlate the order since the logs are collected from two
>> separate
>> consoles. For the write permission, I add code to the host where the
>> permission
>> is always added for all s2 page faults in the shared space. Otherwise,
>> qemu can
>> be killed by -EFAULT or similar error.
>
> This is the problem. We can't add WRITE permission by default. I believe
> you may have MAP_PRIVATE mapping and it has to be mapped as READ only
> and on a permission fault, we replace it with a writable page. By
> overriding the WRITE permission, you let the guest write to a page
> that may not be seen by the VMM.
>
> We identified this as a bug in the KVM driver in this series (reported
> by Lorenzo) and there is a corresponding tf-RMM change that is required
> to get this working. So, please could you wait until the next series
> when this will be addressed ? Or you could switch to using MAP_SHARED
> for the "shared" memory in the memslot.
For the record, you need something like this :
--- a/arch/arm64/kvm/rmi.c
+++ b/arch/arm64/kvm/rmi.c
@@ -838,8 +838,17 @@ int realm_map_non_secure(struct realm *realm,
if (RMI_RETURN_STATUS(ret) == RMI_ERROR_RTT) {
/* Create missing RTTs and retry */
int level = RMI_RETURN_INDEX(ret);
+ int req_level = find_map_level(realm, ipa, ipa_top);
+
+ /*
+ * There already exists a mapping at the level.
May be
+ * we are relaxing a permission for the given
range ?
+ */
+ if (level >= req_level) {
+ realm_unmap_shared_range(kvm, ipa,
ipa_top, false);
+ continue;
+ }
- WARN_ON(level == KVM_PGTABLE_LAST_LEVEL);
ret = realm_create_rtt_levels(realm, ipa, level,
KVM_PGTABLE_LAST_LEVEL,
memcache);
Thanks
Suzuki
>
>
> Suzuki
>
>
>>
>> There are more findings after more experiments: this virtio-net-pci
>> device has 3
>> queues or vrings (Rx/Tx/Ctrl). The Rx/Tx/Ctrl queue are populated in
>> order one after
>> one. In the guest kernel, I intentionally write fixed data
>> (0x0123456789abcdef) to
>> the first 8 bytes of the queue when it gets populated, and stop the
>> guest at random
>> points to see if the data is gone. I found that the data written to
>> Rx/ Tx queue are
>> lost after Ctrl queue is allocated.
>>
>> The data written to Rx/Tx queue is lost if the guest stops (B). The
>> data written to
>> Rx/Tx queue isn't lost if the guest stops at (A). I can see the
>> pattern (0x0123...cdef)
>> by dumping the physcial memory through 'pmemsave' command in qemu.
>>
>> DMA allocation
>> ==============
>> dma_alloc_coherent
>> dma_alloc_attrs
>> dma_direct_alloc
>> __dma_direct_alloc_pages
>> dma_set_decrypted // (A) No data lost if
>> being stopped here for the Ctrl queue
>> memset(ret, 0, size) // (B) Data lost after
>> being stopped after memset() for the Ctrl queue
>>
>> The memset() on the Ctrl queue should trigger a stage2 page fault. It
>> seems the page
>> fault enforces the shared pages for Rx/Tx queue to be dropped? I need
>> to add more
>> debugging code and track it down.
>>
>>> Suzuki
>>>
>>>
>>>>
>>>> - On QEMU, the updated vring (struct vring_desc) at GPA 0x46880000
>>>> isn't seen. All the
>>>> data in that adress are zeros.
>>>>
>>>> ====> virtqueue_split_pop: vdev=<virtio-net>, sz=0x38,
>>>> queue_index=0x0, vq->vring.num=0x100
>>>> virtqueue_split_pop: last_avail_idx=0x0, head=0x0
>>>> address_space_read_cached_slow: cache at 0xffff1c036440, addr=0x0,
>>>> buf=0xffffeee34880, len=0x10
>>>> address_space_read_cached_slow: cache: ptr=0x0,
>>>> xlat=0x10046880000, len=0x1000, mrs=<realm-dma-region>, is_write=no
>>>> address_space_read_cached_slow: translated to mr=<mach-virt.ram>,
>>>> mr_addr=0x6880000, l=0x10
>>>> flatview_read_continue_step: mr=<mach-virt.ram>,
>>>> host=0xffff23e00000, mr_addr=0x6880000, ram_ptr=0xffff2a680000
>>>> virtqueue_split_pop: desc: 0000000000000000 - 00000000 - 00000000
>>>> - 00000000
>>>> qemu-system-aarch64: virtio: zero sized buffers are not allowed
>>>>
>>>>
>> Thanks,
>> Gavin
>>
>
More information about the linux-arm-kernel
mailing list