[Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?

Thu Jul 27 07:15:08 PDT 2023

Hi Bibo,

> From: bibo mao <maobibo at loongson.cn>
> Sent: Tuesday, July 25, 2023 7:24 AM
> To: Salil Mehta <salil.mehta at huawei.com>

[...]

> 在 2023/7/25 13:45, Salil Mehta 写道:
> > Hello,
> >
> >> From: bibo mao <maobibo at loongson.cn>
> >> Sent: Tuesday, July 25, 2023 2:14 AM
> >> To: Salil Mehta <salil.mehta at huawei.com>

[...]

> >> 在 2023/7/25 08:56, Salil Mehta 写道:
> >>>
> >>>> From: bibo mao <maobibo at loongson.cn>
> >>>> Sent: Tuesday, July 25, 2023 1:29 AM
> >>>> To: Salil Mehta <salil.mehta at huawei.com>
> >>>> Cc: Catalin Marinas <catalin.marinas at arm.com>; Jonathan Cameron
> >>>> <jonathan.cameron at huawei.com>; Marc Zyngier <maz at kernel.org>; Will Deacon
> >>>> <will at kernel.org>; christoffer.dall at arm.com; oliver.upton at linux.dev;
> >>>> mark.rutland at arm.com; pbonzini at redhat.com; Salil Mehta
> >>>> <salil.mehta at opnsrc.net>; andrew.jones at linux.dev; yuzenghui
> >>>> <yuzenghui at huawei.com>; kvmarm at lists.cs.columbia.edu; linux-arm-
> >>>> kernel at lists.infradead.org; linux-kernel at vger.kernel.org;
> >>>> kvm at vger.kernel.org; qemu-devel at nongnu.org; james.morse at arm.com;
> >>>> steven.price at arm.com; Suzuki K Poulose <suzuki.poulose at arm.com>; Jean-
> >>>> Philippe Brucker <jean-philippe at linaro.org>; kvmarm at lists.linux.dev; linux-coco at lists.linux.dev
> >>>> Subject: Re: [Question - ARM CCA] vCPU Hotplug Support in ARM Realm world might require ARM spec change?
> >>>>
> >>>> Is vcpu hotplug supported in arm virt-machine now?
> >>>
> >>> Not yet. We are working on it. Please check the RFCs being tested.
> >>>
> >>>
> >>> [1] Pre-RFC V2 Changes: Support of Virtual CPU Hotplug for ARMv8 Arch (WIP)
> >>>     https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>> [2] [RFC PATCH 00/32] ACPI/arm64: add support for virtual cpuhotplug
> >>>     https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2
> >>>
> >>>
> >>>> There is arm64 vcpu hotplug patch in qemu mailing list, however it is not merged.
> >>>> I do not know why it is not merged.
> >>>
> >>>
> >>> I think you are referring to patches [3], [4]? Please follow the discussion
> >>> for details.
> >>
> >> yeap, we reference the patch [3], [4] and benefit from them greatly -:)
> >
> >
> > I am glad that our current work is useful to more than one architecture and it
> > was one of the aim of our work as well but...
> >
> >> The patch for LoongArch vcpu hotplug link is:
> >> https://lore.kernel.org/qemu-devel/cover.1689837093.git.lixianglai@loongson.cn/T/#t
> >
> >
> > I quickly went through above patches and it looks like this patch-set is mostly
> > based on our latest patches which are at [1], [2] and not just at [3], [4]. As I
> > could see most of the functions which you have ported to your architecture are 
> > part of our Qemu repositories [2] which we have yet to push to community. As I
> > am working towards RFC V2 patches and which shall be floated soon. It does not
> > makes sense for you to duplicate the GED/ACPI changes which are common across
> > architectures and which have been derived from the ARM64 vCPU Hotplug support
> > original patches.
> >
> > This will create merge conflicts, will break large part of our original patch-set.
> >
> > Hence, I would request you to drop patches 1-4 from your patch-set or rebase
> > it over ARM64 original patches in a week or 2 week of time. This is to avoid
> > spoiling our previous years of hard work for the topic we have been persistently
> > making efforts as you can see through the code and our detailed presentations.
>
> Do you have a plan to post new vcpu hotplug patch soon ? If there is, we can
> postpone  our patch for reviewing and wait for your arm64 vcpu Hotplug patch,
> and hope that your patch can merge into qemu asap.

Yes, we are working on it and plan to post it soon in 2 weeks of time.

> 
> We always rebase on the mainline qemu version, rather than personal private tree -:)

Ok. I thought if you could take code from private branch then technically you
could rebase from it as well :)

Anyways, I will be soon posting the changes and it will have those ACPI/GED
changes as well which we can try to get accepted quickly. Many thanks!

Best wishes,
Salil

> Regards
> Bibo Mao
> >
> > Hope you will agree we all need to respect others efforts and time in this
> > mode of open-source collaboration.
> >
> > Rest assured I will help you in review of your architecture specific patch-set
> > as it is a work of mutual interest.
> >
> > Thanks for understanding!
> >
> >
> > Best Wishes,
> > Salil.
> >
> >> Regards
> >> Bibo Mao
> >>
> >>>
> >>>
> >>> [3] [PATCH RFC 00/22] Support of Virtual CPU Hotplug for ARMv8 Arch
> >>>     https://lore.kernel.org/all/20200613213629.21984-1-salil.mehta@huawei.com/
> >>> [4] [PATCH RFC 0/4] Changes to Support *Virtual* CPU Hotplug for ARM64
> >>>     https://lore.kernel.org/all/20200625133757.22332-1-salil.mehta@huawei.com/#r
> >>>
> >>>
> >>> In summary, there were some ARM64 Architecture constraints which were being
> >>> violated in the earlier patches of the kernel [4] so we had to re-think of the
> >>> kernel changes. The Qemu part mostly remains same with some new introductions
> >>> of Guest HVC/SMC hyper call exit handling in user space etc. for policy checks
> >>> in VMM/Qemu.
> >>>
> >>>
> >>> You can follow the KVMForum conference presentations [5], [6] delivered in the
> >>> year 2020 and 2023 to get hold of more details related to this.
> >>>
> >>>
> >>> [5] KVMForum 2023: Challenges Revisited in Supporting Virt CPU Hotplug on architectures that don't Support CPU Hotplug (like ARM64)
> >>>     https://kvm-forum.qemu.org/2023/talk/9SMPDQ/
> >>> [6] KVMForum 2020: Challenges in Supporting Virtual CPU Hotplug on SoC Based Systems (like ARM64)
> >>>     https://kvmforum2020.sched.com/event/eE4m
> >>>
> >>>
> >>>> I ask this question because we propose
> >>>> similar patch about LoongArch system in qemu mailing list, and kernel need not be
> >>>> modified for vcpu hotplug.
> >>>
> >>>
> >>> Could you please share the link of your patches so that we can have a look and
> >>> draw a comparison?
> >>>
> >>>
> >>> Thanks
> >>> Salil.
> >>>
> >>>>
> >>>> Regards
> >>>> Bibo, mao
> >>>>
> >>>> 在 2023/7/19 10:35, Salil Mehta 写道:
> >>>>> [Reposting it here from Linaro Open Discussion List for more eyes to look at]
> >>>>>
> >>>>> Hello,
> >>>>> I have recently started to dabble with ARM CCA stuff and check if our
> >>>>> recent changes to support vCPU Hotplug in ARM64 can work in the realm
> >>>>> world. I have realized that in the RMM specification[1] PSCI_CPU_ON
> >>>>> command(B5.3.3) does not handles the PSCI_DENIED return code(B5.4.2),
> >>>>> from the host. This might be required to support vCPU Hotplug feature
> >>>>> in the realm world in future. vCPU Hotplug is an important feature to
> >>>>> support kata-containers in realm world as it reduces the VM boot time
> >>>>> and facilitates dynamic adjustment of vCPUs (which I think should be
> >>>>> true even with Realm world as current implementation only makes use
> >>>>> of the PSCI_ON/OFF to realize the Hotplug look-like effect?)
> >>>>>
> >>>>>
> >>>>> As per our recent changes [2], [3] related to support vCPU Hotplug on
> >>>>> ARM64, we handle the guest exits due to SMC/HVC Hypercall in the
> >>>>> user-space i.e. VMM/Qemu. In realm world, REC Exits to host due to
> >>>>> PSCI_CPU_ON should undergo similar policy checks and I think,
> >>>>>
> >>>>> 1. Host should *deny* to online the target vCPUs which are NOT plugged
> >>>>> 2. This means target REC should be denied by host. Can host call
> >>>>>    RMI_PSCI_COMPETE in such s case?
> >>>>> 3. The *return* value (B5.3.3.1.3 Output values) should be PSCI_DENIED
> >>>>> 4. Failure condition (B5.3.3.2) should be amended with
> >>>>>    runnable pre: target_rec.flags.runnable == NOT_RUNNABLE (?)
> >>>>>             post: result == PSCI_DENIED (?)
> >>>>> 5. Change would also be required in the flow (D1.4 PSCI flows) depicting
> >>>>>    PSCI_CPU_ON flow (D1.4.1)
> >>>>>
> >>>>>
> >>>>> I do understand that ARM CCA support is in its infancy stage and
> >>>>> discussing about vCPU Hotplug in realm world seem to be a far-fetched
> >>>>> idea right now. But specification changes require lot of time and if
> >>>>> this change is really required then it should be further discussed
> >>>>> within ARM.
> >>>>>
> >>>>> Many thanks!
> >>>>>
> >>>>>
> >>>>> Bes regards
> >>>>> Salil
> >>>>>
> >>>>>
> >>>>> References:
> >>>>>
> >>>>> [1] https://developer.arm.com/documentation/den0137/latest/
> >>>>> [2] https://github.com/salil-mehta/qemu.git virt-cpuhp-armv8/rfc-v1-port11052023.dev-1
> >>>>> [3] https://git.gitlab.arm.com/linux-arm/linux-jm.git virtual_cpu_hotplug/rfc/v2