[patch 00/37] cpu/hotplug, x86: Reworked parallel CPU bringup
Juergen Gross
jgross at suse.com
Mon Apr 17 01:35:41 PDT 2023
On 15.04.23 01:44, Thomas Gleixner wrote:
> Hi!
>
> This is a complete rework of the parallel bringup patch series (V17)
>
> https://lore.kernel.org/lkml/20230328195758.1049469-1-usama.arif@bytedance.com
>
> to address the issues which were discovered in review:
>
> 1) The X86 microcode loader serialization requirement
>
> https://lore.kernel.org/lkml/87v8iirxun.ffs@tglx
>
> Microcode loading on HT enabled X86 CPUs requires that the microcode is
> loaded on the primary thread. The sibling thread(s) must be in
> quiescent state; either looping in a place which is aware of potential
> changes by the microcode update (see late loading) or in fully quiescent
> state, i.e. waiting for INIT/SIPI.
>
> This is required by hardware/firmware on Intel. Aside of that it's a
> vendor independent software correctness issue. Assume the following
> sequence:
>
> CPU1.0 CPU1.1
> CPUID($A)
> Load microcode.
> Changes CPUID($A, $B)
> CPUID($B)
>
> CPU1.1 makes a decision on $A and $B which might be inconsistent due
> to the microcode update.
>
> The solution for this is to bringup the primary threads first and after
> that the siblings. Loading microcode on the siblings is a NOOP on Intel
> and on AMD it is guaranteed to only modify thread local state.
>
> This ensures that the APs can load microcode before reaching the alive
> synchronization point w/o doing any further x86 specific
> synchronization between the core siblings.
>
> 2) The general design issues discussed in V16
>
> https://lore.kernel.org/lkml/87pm8y6yme.ffs@tglx
>
> The previous parallel bringup patches just glued this mechanism into
> the existing code without a deeper analysis of the synchronization
> mechanisms and without generalizing it so that the control logic is
> mostly in the core code and not made an architecture specific tinker
> space.
>
> Much of that had been pointed out 2 years ago in the discussions about
> the early versions of parallel bringup already.
>
>
> The series is based on:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip x86/apic
>
> and also available from git:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git hotplug
>
>
> Background
> ----------
>
> The reason why people are interested in parallel bringup is to shorten
> the (kexec) reboot time of cloud servers to reduce the downtime of the
> VM tenants. There are obviously other interesting use cases for this
> like VM startup time, embedded devices...
>
> The current fully serialized bringup does the following per AP:
>
> 1) Prepare callbacks (allocate, intialize, create threads)
> 2) Kick the AP alive (e.g. INIT/SIPI on x86)
> 3) Wait for the AP to report alive state
> 4) Let the AP continue through the atomic bringup
> 5) Let the AP run the threaded bringup to full online state
>
> There are two significant delays:
>
> #3 The time for an AP to report alive state in start_secondary() on x86
> has been measured in the range between 350us and 3.5ms depending on
> vendor and CPU type, BIOS microcode size etc.
>
> #4 The atomic bringup does the microcode update. This has been measured
> to take up to ~8ms on the primary threads depending on the microcode
> patch size to apply.
>
> On a two socket SKL server with 56 cores (112 threads) the boot CPU spends
> on current mainline about 800ms busy waiting for the APs to come up and
> apply microcode. That's more than 80% of the actual onlining procedure.
>
> By splitting the actual bringup mechanism into two parts this can be
> reduced to waiting for the first AP to report alive or if the system is
> large enough the first AP is already waiting when the boot CPU finished the
> wake-up of the last AP.
>
>
> The actual solution comes in several parts
> ------------------------------------------
>
> 1) [P 1-2] General cleanups (init annotations, kernel doc...)
>
> 2) [P 3] The obvious
>
> Avoid pointless delay calibration when TSC is synchronized across
> sockets. That removes a whopping 100ms delay for the first CPU of a
> socket. This is an improvement independent of parallel bringup and had
> been discussed two years ago already.
>
> 2) [P 3-6] Removal of the CPU0 hotplug hack.
>
> This was added 11 years ago with the promise to make this a real
> hardware mechanism, but that never materialized. As physical CPU
> hotplug is not really supported and the physical unplugging of CPU0
> never materialized there is no reason to keep this cruft around. It's
> just maintenance ballast for no value and the removal makes
> implementing the parallel bringup feature way simpler.
>
> 3) [P 7-16] Cleanup of the existing bringup mechanism:
>
> a) Code reorganisation so that the general hotplug specific code is
> in smpboot.c and not sprinkled all over the place
>
> b) Decouple MTRR/PAT initialization from smp_callout_mask to prepare
> for replacing that mask with a hotplug core code synchronization
> mechanism.
>
> c) Make TSC synchronization function call based so that the control CPU
> does not have to busy wait for nothing if synchronization is not
> required.
>
> d) Remove the smp_callin_mask synchronization point as its not longer
> required due to #3c.
>
> e) Rework the sparse_irq_lock held region in the core code so that the
> next polling synchronization point in the x86 code can be removed to.
>
> f) Due to #3e it's not longer required to spin wait for the AP to set
> it's online bit. Remove wait_cpu_online() and the XENPV
> counterpart. So the control CPU can directly wait for the online
> idle completion by the AP and free the control CPU up for other
> work.
>
> This reduces the synchronization points in the x86 code to one, which
> is the AP alive one. This synchronization will be moved to core
> infrastructure in the next section.
>
> 4) [P 17-27] Replace the disconnected CPU state tracking
>
> The extra CPU state tracking which is used by a few architectures is
> completely separate from the CPU hotplug core code.
>
> Replacing it by a variant integrated in the core hotplug machinery
> allows to reduce architecture specific code and provides a generic
> synchronization mechanism for (parallel) CPU bringup/teardown.
>
> - Convert x86 over and replace the AP alive synchronization on x86 with
> the core variant which removes the remaining x86 hotplug
> synchronization masks.
>
> - Convert the other architectures usage and remove the old interface
> and code.
>
> 5) [P 28-30] Split the bringup into two steps
>
> First step invokes the wakeup function on the BP, e.g. SIPI/STARTUP on
> x86. The second one waits on the BP for the AP to report alive and
> releases it for the complete onlining.
>
> As the hotplug state machine allows partial bringup this allows later
> to kick all APs alive in a first iteration and then bring them up
> completely one by one afterwards.
>
> 6) [P 31] Switch the primary thread detection to a cpumask
>
> This makes the parallel bringup a simple cpumask based mechanism
> without tons of conditionals and checks for primary threads.
>
> 7) [P 32] Implement the parallel bringup core code
>
> The parallel bringup looks like this:
>
> 1) Bring up the primary SMT threads to the CPUHP_KICK_AP_ALIVE step
> one by one
>
> 2) Bring up the primary SMT threads to the CPUHP_ONLINE step one by
> one
>
> 3) Bring up the secondary SMT threads to the CPUHP_KICK_AP_ALIVE
> step one by one
>
> 4) Bring up the secondary SMT threads to the CPUHP_ONLINE
> step one by one
>
> In case that SMT is not supported this is obviously reduced to step #1
> and #2.
>
> 8) [P 33-37] Prepare X86 for parallel bringup and enable it
>
>
> Caveats
> -------
>
> The non X86 changes have been all compile tested. Boot and runtime
> testing has only be done on a few real hardware platforms and qemu as
> available. That definitely needs some help from the people who have
> these systems at their fingertips.
>
>
> Results and analysis
> --------------------
>
> Here are numbers for a dual socket SKL 56 cores/ 112 threads machine. All
> numbers in milliseconds. The time measured is the time which the cpu_up()
> call takes for each CPU and phase. It's not exact as the system is already
> scheduling, handling interrupts and soft interrupts, which is obviously
> skewing the picture slightly.
>
> Baseline tip tree x86/apic branch.
>
> total avg/CPU min max
> total : 912.081 8.217 3.720 113.271
>
> The max of 100ms is due to the silly delay calibration for the second
> socket which takes 100ms and was eliminated first. Also the other initial
> cleanups and improvements take some time away.
>
> So the real baseline becomes:
>
> total avg/CPU min max
> total : 785.960 7.081 3.752 36.098
>
> The max here is on the first CPU of the second socket. 20ms of that is due
> to TSC synchronization and an extra 2ms to react on the SIPI.
>
> With parallel bootup enabled this becomes:
>
> total avg/CPU min max
> prepare: 39.108 0.352 0.238 0.883
> online : 45.166 0.407 0.170 20.357
> total : 84.274 0.759 0.408 21.240
>
> That's a factor ~9.3 reduction on average.
>
> Looking at the 27 primary threads of socket 0 then this becomes even more
> interesting:
>
> total avg/CPU min max
> total : 325.764 12.065 11.981 14.125
>
> versus:
> total avg/CPU min max
> prepare: 8.945 0.331 0.238 0.834
> online : 4.830 0.179 0.170 0.212
> total : 13.775 0.510 0.408 1.046
>
> So the reduction factor is ~23.5 here. That's mostly because the 20ms TSC
> sync is not skewing the picture.
>
> For all 55 primaries, i.e with the 20ms TSC sync extra for socket 1 this
> becomes:
>
> total avg/CPU min max
> total : 685.489 12.463 11.975 36.098
>
> versus:
>
> total avg/CPU min max
> prepare: 19.080 0.353 0.238 0.883
> online : 30.283 0.561 0.170 20.357
> total : 49.363 0.914 0.408 21.240
>
> The TSC sync reduces the win to a factor of ~13.8
>
> With 'tsc=reliable' on the command line the socket sync is disabled which
> brings it back to the socket 0 numbers:
>
> total avg/CPU min max
> prepare: 18.970 0.351 0.231 0.874
> online : 10.328 0.191 0.169 0.358
> total : 29.298 0.543 0.400 1.232
>
> Now looking at the secondary threads only:
>
> total avg/CPU min max
> total : 100.471 1.794 0.375 4.745
>
> versus:
> total avg/CPU min max
> prepare: 19.753 0.353 0.257 0.512
> online : 14.671 0.262 0.179 3.461
> total : 34.424 0.615 0.436 3.973
>
> Still a factor of ~3.
>
> The average on the secondaries for the serialized bringup is significantly
> lower than for the primaries because the SIPI response time is shorter and
> the microcode update takes no time.
>
> This varies wildly with the system, whether microcode in BIOS is already up
> to date, how big the microcode patch is and how long the INIT/SIPI response
> time is. On an AMD Zen3 machine INIT/SIPI response time is amazingly fast
> (350us), but then it lacks TSC_ADJUST and does a two millisecond TSC sync
> test for _every_ AP. All of this sucks...
>
>
> Possible further enhancements
> -----------------------------
>
> It's definitely worthwhile to look into reducing the cross socket TSC sync
> test time. It's probably safe enough to use 5ms or even 2ms instead of 20ms
> on systems with TSC_ADJUST and a few other 'TSC is sane' indicators. Moving
> it out of the hotplug path is eventually possible, but that needs some deep
> thoughts.
>
> Let's take the TSC sync out of the picture by adding 'tsc=reliable" to the
> kernel command line. So the bringup of 111 APs takes:
>
> total avg/CPU min max
> prepare: 38.936 0.351 0.231 0.874
> online : 25.231 0.227 0.169 3.465
> total : 64.167 0.578 0.400 4.339
>
> Some of the outliers are not necessarily in the state callbacks as the
> system is already scheduling and handles interrupts and soft
> interrupts. Haven't analyzed that yet in detail.
>
> In the prepare stage which runs on the control CPU the larger steps are:
>
> smpcfd:prepare 16us avg/CPU
> threads:prepare 98us avg/CPU
> workqueue:prepare 43us avg/CPU
> trace/RB:prepare 135us avg/CPU
>
> The trace ringbuffer initialization allocates 354 pages and 354 control
> structures one by one. That probably should allocate a large page and an
> array of control structures and work from there. I'm sure that would reduce
> this significantly. Steven?
>
> smpcfd does just a percpu allocation. No idea why that takes that long.
>
> Vs. threads and workqueues. David thought about spreading out the
> preparation work and do it really in parallel. That's a nice idea, but the
> threads and workqueue prepare steps are self serializing. The workqueue one
> has a global mutex and aside of that both steps create kernel threads which
> implicitely serialize on kthreadd. alloc_percpu(), which is used by
> smpcfd:prepare is also globally serialized.
>
> The rest of the prepare steps is pretty much in the single digit
> microseconds range.
>
> On the AP side it should be possible to move some of the initialization
> steps before the alive synchronization point, but that really needs a lot
> of analysis whether the functions are safe to invoke that early and outside
> of the cpu_hotplug_lock held region for the case of two stage parallel
> bringup; see below.
>
> The largest part is:
>
> identify_secondary_cpu() 99us avg/CPU
>
> Inside of identify_secondary_cpu() the largest offender:
>
> mcheck_init() 73us avg/CPU
>
> This part is definitly worth to be looked at whether it can be at least
> partially moved to the early startup code before the alive
> synchronization point. There's a lot of deep analysis required and
> ideally we just rewrite the whole CPUID evaluation trainwreck
> completely.
>
> The rest of the AP side is low single digit microseconds except of:
>
> perf/x86:starting 14us avg/CPU
>
> smpboot/threads:online 13us avg/CPU
> workqueue:online 17us avg/CPU
> mm/vmstat:online 17us avg/CPU
> sched:active 30us avg/CPU
>
> sched:active is special. Onlining the first secondary HT thread on the
> second socket creates a 3.2ms outlier which skews the whole picture. That's
> caused by enabling the static key sched_smt_present which patches the world
> and some more. For all other APs this is really in the 1us range. This
> definitely could be postponed during bootup like the scheduler domain
> rebuild is done after the bringup. But that's still fully serialized and
> single threaded and obviously could be done later in the context of async
> parallel init. It's unclear why this is different with the fully serialized
> bringup where it takes significantly less time, but that's something which
> needs to be investigated.
>
>
> Is truly parallel bringup feasible?
> -----------------------------------
>
> In theory yes, realistically no. Why?
>
> 1) The preparation phase
>
> Allocating memory, creating threads for the to be brought up CPU must
> obviously happen on an already online CPU.
>
> While it would be possible to bring up a subset of CPUs first and let
> them do the preparation steps for groups of still offline CPUs
> concurrently, the actual benefit of doing so is dubious.
>
> The prime example is kernel thread creation, which is implicitely
> serialized on kthreadd.
>
> A simple experiment shows that 4 concurrent workers on 4 different
> CPUs where each is creating 14 * 5 = 70 kernel threads are 5% slower
> than a single worker creating 4 * 14 * 5 = 280 threads.
>
> So we'd need to have multiple kthreadd instances to handle that,
> which would then serialize on tasklist lock and other things.
>
> That aside the preparation phase is also affected by the problem
> below.
>
> 2) Assumptions about hotplug serialization
>
> a) There are quite some assumptions about CPU bringup being fully
> serialized across state transitions. A lot of state callbacks rely
> on that and would require local locking.
>
> Adding that local locking is surely possible, but that has several
> downsides:
>
> - It adds complexity and makes it harder for developers to get
> this correct. The subtle bugs resulting out of that are going
> to be interesting
>
> - Fine grained locking has a charm, but only if the time spent
> for the actual work is larger than the time required for
> serialization and synchronization.
>
> Serializing a callback which takes less than a microsecond and
> then having a large number of CPUs contending on the lock will
> not make it any faster at all. That's a well known issue of
> parallelizing and neither made up nor kernel specific.
>
> b) Some operations definitely require to be protected by the
> cpu_hotplug_lock, especially those which affect cpumasks as the
> masks are guaranteed to be stable in a cpus_read_lock()'ed region.
>
> As this lock cannot be taken in atomic contexts, it's required
> that the control CPU holds the lock write locked across these
> state transitions. And no, we are not making this a spinlock just
> for that and we even can't.
>
> Just slapping a lock into the x86 specific part of the cpumask
> update function does not solve anything. The relevant patch in V17
> is completely useless as it only serializes the actual cpumask/map
> modifications, but all read side users are hosed if the update
> would be moved before the alive synchronization point, i.e. into a
> non hotplug lock protected region.
>
> Even if the hotplug lock would be held accross the whole parallel
> bringup operation then this would still expose all usage of these
> masks and maps in the actual hotplug state callbacks to concurrent
> modifications.
>
> And no, we are not going to expose an architecture specific raw
> spinlock to the hotplug state callbacks, especially not to those
> in generic code.
>
> c) Some cpu_read_lock()'ed regions also expect that there is no CPU
> state transition happening which would modify their local
> state. This would again require local serialization.
>
> 3) The amount of work and churn:
>
> - Analyze the per architecture low level startup functions plus their
> descendant functions and make them ready for concurrency if
> necessary.
>
> - Analyze ~300 hotplug state callbacks and their descendant functions
> and make them ready for concurrency if necessary.
>
> - Analyze all cpus_read_lock()'ed regions and address their
> requirements.
>
> - Rewrite the core code to handle the cpu_hotplug_lock requirements
> only in distinct phases of the state machine.
>
> - Rewrite the core code to handle state callback failure and the
> related rollback in the context of the new rules.
>
> - ...
>
> Even if some people are dedicated enough to do that, it's very
> questionable whether the resulting complexity is justified.
>
> We've spent a serious amount of time to sanitize hotplug and bring it
> into a state where it is correct. This also made it reasonably simple
> for developers to implement hotplug state callbacks without having to
> become hotplug experts.
>
> Breaking this completely up will result in a flood of hard to diagnose
> subtle issues for sure. Who is going to deal with them?
>
> The experience with this series so far does not make me comfortable
> about that thought in any way.
>
>
> Summary
> -------
>
> The obvious and low hanging fruits have to be solved first:
>
> - The CPUID evaluation and related setup mechanisms
>
> - The trace/ringbuffer oddity
>
> - The sched:active oddity for the first sibling on the second socket
>
> - Some other expensive things which I'm not seeing in my test setup due
> to lack of hardware or configuration.
>
> Anything else is pretty much wishful thinking in my opinion.
>
> To be clear. I'm not standing in the way if there is a proper solution,
> but that requires to respect the basic engineering rules:
>
> 1) Correctness first
> 2) Keep it maintainable
> 3) Keep it simple
>
> So far this stuff failed already at #1.
>
> I completely understand why this is important for cloud people, but
> the real question to ask here is what are the actual requirements.
>
> As far as I understand the main goal is to make a (kexec) reboot
> almost invisible to VM tenants.
>
> Now lets look at how this works:
>
> A) Freeze VMs and persist state
> B) kexec into the new kernel
> C) Restore VMs from persistant memory
> D) Thaw VMs
>
> So the key problem is how long it takes to get from #B to #C and finally
> to #D.
>
> As far as I understand #C takes a serious amount of time and cannot be
> parallelized for whatever reasons.
>
> At the same time the number of online CPUs required to restore the VMs
> state is less than the number of online CPUs required to actually
> operate them in #D.
>
> That means it would be good enough to return to userspace with a
> limited number of online CPUs as fast as possible. A certain amount of
> CPUs are going to be busy with restoring the VMs state, i.e. one CPU
> per VM. Some remaining non-busy CPU can bringup the rest of the system
> and the APs in order to be functional for #D, i.e the restore of VM
> operation.
>
> Trying to optimize this purely in kernel space by adding complexity of
> dubious value is simply bogus in my opinion.
>
> It's already possible today to limit the number of CPUs which are
> initially onlined and online the rest later from user space.
>
> There are two issue there:
>
> a) The death by MCE broadcast problem
>
> Quite some (contemporary) x86 CPU generations are affected by
> this:
>
> - MCE can be broadcasted to all CPUs and not only issued locally
> to the CPU which triggered it.
>
> - Any CPU which has CR4.MCE == 0, even if it sits in a wait
> for INIT/SIPI state, will cause an immediate shutdown of the
> machine if a broadcasted MCE is delivered.
>
> b) Do the parallel bringup via sysfs control knob
>
> The per CPU target state interface allows to do that today one
> by one, but it's akward and has quite some overhead.
>
> A knob to online the rest of the not yet onlined present CPUs
> with the benefit of the parallel bringup mechanism is
> missing.
>
> #a) That's a risk to take by the operator.
>
> Even the regular serialized bringup does not protect against this
> issue up to the point where all present CPUs have at least
> initialized CR4.
>
> Limiting the number of APs to online early via the kernel command
> line widens that window and increases the risk further by
> executing user space before all APs have CR4 initialized.
>
> But the same applies to a deferred online mechanism implemented in
> the kernel where some worker brings up the not yet online APs while
> the early online CPUs are already executing user space code.
>
> #b) Is a no brainer to implement on top of this.
>
>
> Conclusion
> ----------
>
> Adding the basic parallel bringup mechanism as provided by this series
> makes a lot of sense. Improving particular issues as pointed out in the
> analysis makes sense too.
>
> But trying to solve an application specific problem fully in the kernel
> with tons of complexity, without exploring straight forward and simple
> approaches first, does not make any sense at all.
>
> Thanks,
>
> tglx
>
> ---
> Documentation/admin-guide/kernel-parameters.txt | 20
> Documentation/core-api/cpu_hotplug.rst | 13
> arch/Kconfig | 23 +
> arch/arm/Kconfig | 1
> arch/arm/include/asm/smp.h | 2
> arch/arm/kernel/smp.c | 18
> arch/arm64/Kconfig | 1
> arch/arm64/include/asm/smp.h | 2
> arch/arm64/kernel/smp.c | 14
> arch/csky/Kconfig | 1
> arch/csky/include/asm/smp.h | 2
> arch/csky/kernel/smp.c | 8
> arch/mips/Kconfig | 1
> arch/mips/cavium-octeon/smp.c | 1
> arch/mips/include/asm/smp-ops.h | 1
> arch/mips/kernel/smp-bmips.c | 1
> arch/mips/kernel/smp-cps.c | 14
> arch/mips/kernel/smp.c | 8
> arch/mips/loongson64/smp.c | 1
> arch/parisc/Kconfig | 1
> arch/parisc/kernel/process.c | 4
> arch/parisc/kernel/smp.c | 7
> arch/riscv/Kconfig | 1
> arch/riscv/include/asm/smp.h | 2
> arch/riscv/kernel/cpu-hotplug.c | 14
> arch/x86/Kconfig | 45 --
> arch/x86/include/asm/apic.h | 5
> arch/x86/include/asm/cpu.h | 5
> arch/x86/include/asm/cpumask.h | 5
> arch/x86/include/asm/processor.h | 1
> arch/x86/include/asm/realmode.h | 3
> arch/x86/include/asm/sev-common.h | 3
> arch/x86/include/asm/smp.h | 26 -
> arch/x86/include/asm/topology.h | 23 -
> arch/x86/include/asm/tsc.h | 2
> arch/x86/kernel/acpi/sleep.c | 9
> arch/x86/kernel/apic/apic.c | 22 -
> arch/x86/kernel/callthunks.c | 4
> arch/x86/kernel/cpu/amd.c | 2
> arch/x86/kernel/cpu/cacheinfo.c | 21
> arch/x86/kernel/cpu/common.c | 50 --
> arch/x86/kernel/cpu/topology.c | 3
> arch/x86/kernel/head_32.S | 14
> arch/x86/kernel/head_64.S | 121 +++++
> arch/x86/kernel/sev.c | 2
> arch/x86/kernel/smp.c | 3
> arch/x86/kernel/smpboot.c | 508 ++++++++----------------
> arch/x86/kernel/topology.c | 98 ----
> arch/x86/kernel/tsc.c | 20
> arch/x86/kernel/tsc_sync.c | 36 -
> arch/x86/power/cpu.c | 37 -
> arch/x86/realmode/init.c | 3
> arch/x86/realmode/rm/trampoline_64.S | 27 +
> arch/x86/xen/enlighten_hvm.c | 11
> arch/x86/xen/smp_hvm.c | 16
> arch/x86/xen/smp_pv.c | 56 +-
> drivers/acpi/processor_idle.c | 4
> include/linux/cpu.h | 4
> include/linux/cpuhotplug.h | 17
> kernel/cpu.c | 397 +++++++++++++++++-
> kernel/smp.c | 2
> kernel/smpboot.c | 163 -------
> 62 files changed, 953 insertions(+), 976 deletions(-)
>
>
Tested with a Xen PV dom0 on an 8 cpu system, no issues found.
Tested-by: Juergen Gross <jgross at suse.com>
Juergen
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0xB0DE9DD628BF132F.asc
Type: application/pgp-keys
Size: 3098 bytes
Desc: OpenPGP public key
URL: <http://lists.infradead.org/pipermail/linux-riscv/attachments/20230417/35363111/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 495 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/linux-riscv/attachments/20230417/35363111/attachment.sig>
More information about the linux-riscv
mailing list