[PATCH v1 0/8] x86, sched, mm: Speed up CPU hotplug on large systems
Pasha Tatashin
pasha.tatashin at soleen.com
Thu Oct 15 08:33:23 PDT 2026
For live update we require a fast CPU online and offline capability.
However, sequentially offlining and onlining all application processors
on modern high-core-count x86 servers exposes a series of hardware and
algorithmic bottlenecks across arch/x86, the scheduler topology builder,
deadline bandwidth accounting, and vmstat.
This series addresses these bottlenecks, reducing the time to offline and
online all AP CPUs by 95.2% (21.0x faster) on a 2-socket, 480-CPU Intel
Xeon 6985P-C and by 59.8% (2.5x faster) on a 2-socket, 512-CPU AMD EPYC
9B45.
Benchmark workload (mean of 10 runs per step):
# echo 1 > /sys/kernel/rcu_expedited
# for i in {1..N}; do echo 0 > /sys/devices/system/cpu/cpu$i/online; done
# for i in {1..N}; do echo 1 > /sys/devices/system/cpu/cpu$i/online; done
# echo 0 > /sys/kernel/rcu_expedited
2-socket, 480-CPU Intel Xeon 6985P-C (N=479):
Offline: 40.40 s -> 3.09 s (-92.4%, 13.1x faster)
Online: 65.83 s -> 1.98 s (-97.0%, 33.2x faster)
Total: 106.23 s -> 5.07 s (-95.2%, 21.0x faster)
2-socket, 512-CPU AMD EPYC 9B45 (N=511):
Offline: 4.53 s -> 2.30 s (-49.2%, 2.0x faster)
Online: 3.88 s -> 1.08 s (-72.2%, 3.6x faster)
Total: 8.41 s -> 3.38 s (-59.8%, 2.5x faster)
(On AMD EPYC 9B45, each 8-core CCD has a 32 MB L3 where WBINVD takes
< 0.5 ms, offline uses acpi_idle rather than mwait_play_dead(), and
Intel CMCI is unused, so Patches 1-3 do not affect AMD, which enters
Patch 4 at 4.53 s offline / 3.92 s online / 8.45 s total. Without
rcu_expedited=1, final 10-run mean total time with all 8 patches
applied is 12.04 s on Intel and 15.99 s on AMD.)
1. x86 WBINVD, CMCI rediscovery, and MTRR rendezvous (Patches 1-4):
On processors with large shared last-level caches (such as the 504 MB
L3 per socket on Xeon 6985P-C), CPU hotplug is initially dominated by
full coherence-domain cache flushes (WBINVD), each taking ~85-115 ms
per CPU while stalling active cores:
- Patch 1 removes the obsolete WBINVD from trampoline_start, which was
added in the 1990s for Sequent NUMA-Q hardware (removed in commit
b5660ba76b41 ("x86, platforms: Remove NUMAQ")). Per the Intel SDM
and AMD APM, INIT preserves cache contents and leaves CR0.CD and
CR0.NW unchanged. Removing it cuts 479-CPU online time on Intel from
65.83 s to 11.14 s (-83.1%) and total time from 106.23 s to 51.60 s
(-51.4%).
- Patch 2 skips the redundant WBINVD before MWAIT in mwait_play_dead()
during runtime CPU offline (retaining it when cpuhp_tasks_frozen is
set for system sleep). Architectural MWAIT C-states maintain cache
coherency while the package remains powered and flush core-private
L1/L2 caches to the LLC on C-state entry, just like normal idle in
intel_idle. Skipping it for runtime offline cuts 479-CPU offline
time on Intel from 40.46 s to 11.42 s (-71.8%) and total time from
51.60 s to 22.50 s (-56.4%).
- Patch 3 bounds cmci_rediscover() on CPU teardown. Currently,
cmci_rediscover() broadcasts an IPI via on_each_cpu() to all online
CPUs, serializing them on cmci_discover_lock while reading up to 32
MCA MSRs. Because any disabled MCA bank is shared at most with an
SMT sibling, L2 cluster peer, or die/package peer, probing a single
online topology peer via smp_call_function_single() (or returning
immediately when no shared banks were orphaned) reduces 479-CPU
offline time on Intel from 11.42 s to 9.63 s (-15.7%) and total time
from 22.50 s to 20.74 s (-7.8%).
- Patch 4 skips the stop_machine_from_inactive_cpu() all-CPU
rendezvous (and the AP's cache disable/flush and MTRR reprogramming
in cache_cpu_init()) in cache_ap_online() when the AP's hardware
MTRR MSRs already match mtrr_state (which cache_aps_init() already
programmed during initial boot and warm reset preserves). This cuts
online time on Intel from 11.11 s to 8.69 s (-21.8%; total 20.74 s
-> 18.54 s, -10.6%) and on AMD from 3.92 s to 3.15 s (-19.6%; total
8.45 s -> 7.66 s, -9.3%).
2. Scheduler domain rebuilds, deadline accounting, and vmstat (Patches 5-8):
Once the x86 hardware bottlenecks are removed, CPU hotplug on both
Intel and AMD is dominated by O(N^2) and O(N^3) work inside
partition_sched_domains_locked(), build_sched_domains(), and
refresh_zone_stat_thresholds():
- Patch 5 avoids redundant O(N^3) group weight, SMT core-mask, and
asym_prefer_cpu recomputations in init_sched_groups_capacity() by
caching those fields on struct sched_group_capacity, reuses
already-initialized sgc instances in init_overlap_sched_group(), and
allocates sdd->sg, sdd->sgc, and d->sds lazily on demand. This cuts
total hotplug time on Intel from 18.54 s to 9.20 s (-50.4%) and on
AMD from 7.66 s to 5.29 s (-30.9%).
- Patch 6 eliminates three O(N^2) dl_server bandwidth walks per
hotplug step. Because fair_server is enabled on every runqueue by
default, rq_attach_root() performs O(N^2) remote extra_bw updates
when attaching def_root_domain during teardown and again when
attaching the new root_domain during build, followed by a nested
O(N^2) loop in dl_clear_root_domain(). Dropping the redundant
__dl_server_attach_root() calls in rq_attach_root() and converting
dl_clear_root_domain() to two linear O(N) passes cuts
dl_clear_root_domain() at ~500 CPUs from 1.8-5.5 ms to < 65 us,
reducing total hotplug time by 30.5% on Intel (9.20 s -> 6.39 s)
and 18.7% on AMD (5.29 s -> 4.30 s).
- Patch 7 eliminates per-CPU bitmap scans over already-covered groups
in build_sched_groups(), reuses temporary per-CPU domain pointer
arrays across rebuilds instead of calling alloc_percpu() and
free_percpu() on every hotplug step, reuses precomputed domain/child
span weights, and hoists LLC size calculations. This cuts total
hotplug time by 15.0% on Intel (6.39 s -> 5.43 s) and 19.5% on AMD
(4.30 s -> 3.46 s).
- Patch 8 avoids unconditionally zeroing and rewriting every online
CPU's per-CPU stat_threshold in refresh_zone_stat_thresholds() on
every hotplug step (since calculate_normal_threshold() only changes
at powers of two and is clamped once num_online_cpus() >= 64),
reducing refresh_zone_stat_thresholds() by ~88-91% on Intel and ~51%
on AMD, and bringing total hotplug time down to 5.07 s on Intel
(-6.6%) and 3.38 s on AMD (-2.3%).
Pasha Tatashin (8):
x86/realmode: Remove unnecessary WBINVD from AP startup trampoline
x86/smp: Remove unnecessary WBINVD from mwait_play_dead()
x86/mce/intel: Bound cmci_rediscover() to a single topology peer
x86/cacheinfo, mtrr: Skip stop_machine rendezvous in cache_ap_online()
when MTRRs match
sched/topology: Avoid redundant O(N^3) group capacity walks and unused
allocations in build_sched_domains()
sched/deadline, topology: Avoid O(N^2) dl_server bandwidth walks
during root_domain rebuilds
sched/topology: Avoid redundant percpu allocations and bitmap walks in
build_sched_domains()
mm/vmstat: Avoid redundant per-CPU stat_threshold writes in
refresh_zone_stat_thresholds()
arch/x86/include/asm/mtrr.h | 2 +
arch/x86/kernel/cpu/cacheinfo.c | 7 +
arch/x86/kernel/cpu/mce/intel.c | 57 ++++-
arch/x86/kernel/cpu/mtrr/generic.c | 49 ++++
arch/x86/kernel/smpboot.c | 16 +-
arch/x86/realmode/rm/trampoline_32.S | 2 -
arch/x86/realmode/rm/trampoline_64.S | 1 -
include/linux/cpuset.h | 6 +-
include/linux/sched/deadline.h | 1 -
kernel/sched/deadline.c | 89 ++++---
kernel/sched/sched.h | 10 +-
kernel/sched/topology.c | 339 ++++++++++++++++-----------
mm/vmstat.c | 58 ++---
13 files changed, 422 insertions(+), 215 deletions(-)
base-commit: cee9395acd8043be0644b25c34bfa86623f2b935
--
2.56.0.385.gd3acb90ef8-goog
More information about the kexec
mailing list