diff --git a/scheds/rust/scx_cake/DESIGN.md b/scheds/rust/scx_cake/DESIGN.md index 256c8774c9..a55bcb98da 100644 --- a/scheds/rust/scx_cake/DESIGN.md +++ b/scheds/rust/scx_cake/DESIGN.md @@ -163,7 +163,8 @@ Peek own head (republishing the qmask bit) and `WAKE_DSQ`. Own first unless the wake head wins by one geometry unit or `WAKE_DSQ` went unserved for `WAKE_STARVE_WALL_NS` (24 ms); then the other; then the staggered `qmask`-gated ring steal from `cpu+1` (multi-CCD: same-CCD → cache-tier → -all). Else refill prev's slice. +all — active when the host has >1 LLC and fits `STEAL_SPAN`, else the +plain ring). Else refill prev's slice. --- diff --git a/scheds/rust/scx_cake/README.md b/scheds/rust/scx_cake/README.md index 2b91c51884..4de29fe423 100644 --- a/scheds/rust/scx_cake/README.md +++ b/scheds/rust/scx_cake/README.md @@ -49,9 +49,10 @@ results stand: [`docs/PERFORMANCE.md`](./docs/PERFORMANCE.md). | **If evicted** | the exit reason lands in `dmesg` (grep `sched_ext`) | The binary takes `-v` and `-V` and nothing else — that is the point. -(`intf.h` carries no policy switches; the only cflag inputs are the build -host's CCD/CPU counts. Experiments are A/B'd as two git commits, never a -build flag — see [`DESIGN.md`](./DESIGN.md).) +(`intf.h` carries no policy switches and no cflag inputs; the loader +measures the host's topology at attach, so one binary fits any machine. +Experiments are A/B'd as two git commits, never a build flag — see +[`DESIGN.md`](./DESIGN.md).) ## How it works @@ -176,7 +177,7 @@ only learned state is one per-CPU three-bit handoff-confidence hint. | file | contents | |---|---| | `src/bpf/cake.bpf.c` | the scheduler — 8 callbacks, ~1.5k lines, about a third comments explaining the why | -| `src/bpf/intf.h` | the constant surface: build-host CPU/CCD counts, the `SLICE_NS` boot seed, frame-clock bands, and the IDs shared with the loader | +| `src/bpf/intf.h` | the constant surface: the `SLICE_NS` boot seed, frame-clock bands, and the IDs shared with the loader | | `src/main.rs` | the loader: attach, exit reporting, hardware probes, frame-clock publish, sink monitor | | [`STATE.md`](./STATE.md) | **start here** — current state, the experiment ledger, and the `§` rationale registry every source comment resolves to | | [`DESIGN.md`](./DESIGN.md) | the full design: every rule, constant, invariant | diff --git a/scheds/rust/scx_cake/STATE.md b/scheds/rust/scx_cake/STATE.md index 78fd1a3a5d..374090664d 100644 --- a/scheds/rust/scx_cake/STATE.md +++ b/scheds/rust/scx_cake/STATE.md @@ -43,6 +43,22 @@ live WoW confirm); all hashes cited below resolve via those local backup branche | 10 | **G25** steal-ring bitmask | **LANDED**, verifier accepted, attach smoke passed 2026-08-08 | wake latency on HD2 render roles + P4 bench screen. Prediction on record: ~0.04% of a core quiet, 4.7% invalidated — **expected frame effect near zero; a null confirms the pricing** | | 11 | **G23** per-line IRQ-sink detection + mask avoidance | built, smoke PASSED, **endpoint unmeasured** | HD2 ABBA during ACTIVE play (P2 needs live mouse input) on `Window & Input` / `main` / `renderer` mean wake + severe-frame screen, then P4 `--blocks 2` bench screen. Receipts + steps: `docs/REVIEW_G21_G23_2026-08-02.md` §resume | +**PORTABILITY FIX (2026-08-20): one binary now runs on any machine.** A 1.2.1 +package refused a 16-CPU host: `build.rs` baked the build machine's CPU and CCD +counts into the BPF. A binary built on a smaller machine refused to attach; a +binary built on a single-CCD machine lost CCD steal with no warning. + +The fix, commit `9e5d31f56`: +- `build.rs` no longer reads the build machine. No topology cflags remain. +- `cpu_steal_order` has a fixed size: `STEAL_SPAN`² = 128² u16 = 32 KB rodata. +- The `#if CAKE_NR_CCDS` forks are gone. One rodata flag, `steal_order_live`, + replaces them. The loader sets it when the host has >1 LLC and fits the span. + The verifier then deletes the dead branch on each host at attach. +- Hosts wider than `STEAL_SPAN` use the ring walk. Only `MAX_CPUS` (1024) can + refuse attach. +- New CLAUDE.md invariant: cake runs on all hardware; per-hardware fast paths + are rodata-gated, never build-host-shaped. + **EXP S.1-1ms (2026-08-19): ❌ FALSIFIED.** Hypothesis was: nightly's extreme-tail loss vs shipped 1.1.3 (mutex-handoff p999 −37% 6/6; `runs/adhoc_wallclock_113_vs_nightly_20260819`) is SLICE_NS geometry. Probe SLICE_NS 3000→1000 µs (`092ada871`), 6-slot ABCCBA @@ -96,30 +112,32 @@ y-cruncher, namd, kernel-defconfig, xz, prime. | HD2 live, ABBA, 2026-08-02 (G21) | `main` mean wake | 0.79 / 0.79 | **0.47 / 0.48** | **cake 2/2, −39.9%** | | same | `renderer` mean wake | 0.91 / 0.83 | **0.66 / 0.64** | cake 2/2, −25.3% | -**WoW wake read (2026-08-17, single-arm cake @ G27.1c build `db0b3636…`, 22 s live play — -attribution only, no native arm):** 29 roles; every high-n role p99 ≤ 2.6 µs except -**vkd3d_fence p99 31.45 / max 255 µs**, 99% of slow wakes targeting idle-shown CPU 13 = -the nvidia IRQ CPU (ISR shadow; cpuidle driver `none` on this host). Off-target fence -wakes: p50 2 / max 11 µs. Sink set empty at attach → §G30. **After G30 Phase B + sink -veto (`wow-cake-g30c`): fence p99 1.25 µs, 6 slow wakes, CPU 13 share 2/50.8k — tail -eliminated at the wake tier; frames unmeasured.** Artifacts: +**WoW wake read (2026-08-17, single-arm cake @ G27.1c build `db0b3636…`, 22 s live +play — attribution only, no native arm).** 29 roles. Every high-n role p99 ≤ 2.6 µs +except vkd3d_fence: **p99 31.45 / max 255 µs**. 99% of the slow wakes target +idle-shown CPU 13 — the nvidia IRQ CPU (ISR shadow; cpuidle driver `none` on this +host). Off-target fence wakes: p50 2 / max 11 µs. The sink set was empty at attach +→ §G30. **After G30 Phase B + sink veto (`wow-cake-g30c`): fence p99 1.25 µs, +6 slow wakes, CPU 13 share 2/50.8k.** Tail eliminated at the wake tier; frames +unmeasured. Artifacts: `~/Documents/Repo/scx_cake_bench/history/wake_latency/wow-cake-{g27tip,g30a,g30b,g30c,g35d,g36}*`. -**Five-domain wake sweep (G17 rotation, delays > 200 µs per 10k transitions):** input -294.6/93.2 → **39.0/47.8** (4.5×), FAudio 288.3/80.8 → **31.7/40.6** (~5×), renderer -124.0/87.0 → **46.7/47.0** (2.2×), vkd3d_queue 42.0/18.7 → **13.2/15.5** (1.5×) — cake 2/2 -on all four. Network (n~555) and IO (250× transition mismatch) unusable; `data-loop.0` is -SCHED_FIFO, so cake never schedules it. **Law: cake's advantage scales inversely with the -thread's burst** — G10–G20 optimised the renderer, the longest-burst thread, where the win -is smallest. +**Five-domain wake sweep (G17 rotation; delays > 200 µs per 10k transitions, +native → cake):** input 294.6/93.2 → **39.0/47.8** (4.5×), FAudio 288.3/80.8 → +**31.7/40.6** (~5×), renderer 124.0/87.0 → **46.7/47.0** (2.2×), vkd3d_queue +42.0/18.7 → **13.2/15.5** (1.5×) — cake 2/2 on all four. Network (n~555) and IO +(250× transition mismatch) unusable. `data-loop.0` is SCHED_FIFO, so cake never +schedules it. **Law: cake's advantage scales inversely with the thread's burst.** +G10–G20 optimised the renderer — the longest-burst thread, where the win is +smallest. --- ## Open gaps 1. **Cake loses the EASY scene** — 0.1% low −5.6%, 2/2 no overlap, on a scene where - native's own tail is already tight. Mechanism unknown; **highest-value target on the - board.** The G17 frame win and this loss both stand; neither generalises. + native's own tail is already tight. Mechanism unknown; **highest-value open + target.** The G17 frame win and this loss both stand; neither generalises. 2. **ccm-memcpy −14.9%** — pure CPU-share reallocation from cake's sleeper catch-up (per-usr-second efficiency equal). Zero-sum, no point fix; four falsified attempts. The lever is unlike-type SMT pairing (+73% memcpy pinned) with a **per-TASK** duty @@ -130,34 +148,37 @@ is smallest. 4. **futex-lock-pi −71.8%** — lock-pi pins its workers, so a pinned wake queues behind a busy occupant and no CPU may steal it. A sched_ext **semantic gap**; kernel lane (a wakeup-equivalent flag for PI re-activations). Accepted known loss. -5. **The renderer wake tail resists everything** — locality (G13), preempt (G14/G15), +5. **The renderer wake tail: all levers null so far** — locality (G13), preempt (G14/G15), notification (G16) all null; G15 confirmed falsified on input too. 28–32% of slow same-CPU wakes have `swapper` as occupant in **both** schedulers — an idle-exit floor neither one beats. 6. **`ops.enqueue` reaches 0.142% of a game's dispatches** (vs 23.6% saturated). Every enqueue-side routing decision is nearly inert on games; leverage is on the direct path in `cake_select_cpu`, which serves 99.86%. -7. **Third arm UNBLOCKED 2026-08-19** — shipped 1.1.3 (`/usr/bin/scx_cake`, sha256 - `56f9e886…`) attaches and runs clean on 7.1.8; the libbpf skeleton warning still - prints but is non-fatal. First read: quick wallclock ABBA vs nightly HEAD - (diagnostic tier, `scx_cake_bench_assets/runs/adhoc_wallclock_113_vs_nightly_20260819/`): - nightly pipe **−16.0%**, memcpy −3.9%, but sched-messaging **+70.5%** (4/4, no - overlap) — the many-to-many handoff shape regressed vs 1.1.3 and is unscored at - the sealed tier. n=10 confirm (suite 4, 5×ABBA): schbench-light req p99 **+72% - 10/10** (S sd 7 µs, N sd 459), mutex-handoff p999 **2.0× 10/10** (S sd 0.024 µs), - p99 **BIMODAL** (4/10 runs match 1.1.3 exactly, 6/10 at 1.2–1.9 µs), pipe −9.7% - win 10/10. Slice dose falsified (EXP S.1-1ms). **MODE SWITCH ROOT-CAUSED 2026-08-19 - (`mode_probe/` in the run dir): it is `cake_frame_slice_ns`.** Placement exonerated - (24 runs, pair co-resident on CPU 11 in BOTH modes, sinks 0%). Dose-response with an - injected 1 kHz sleep-majority crowd, `--verbose` clock log as ground truth: crowd - binds in one poll (950.8 Hz published) → p99 1.8–2.1; crowd off → 3-poll re-publish - + ~16 s floor climb → p99 decays to 0.90. A **374.9 Hz desktop voter holds slice at - 2 ms** on this host even quiet — nightly never runs at the 3 ms cap on a live - desktop. 1.1.3 has no frame clock (constant geometry) — hence its sd ≈ 0. Mechanism: - any fast desktop crowd tightens patience windows (shifts of `cake_frame_slice_ns`, - §S.2) for EVERY task; the same-CPU handoff tail pays for it. The registered fix is - already on the board: §G12 KEYSTONE re-base of `FRAME_*_SHIFT` windows onto the - occupant's OWN period decouples bystander tasks from the global clock. +7. **Third arm UNBLOCKED 2026-08-19.** Shipped 1.1.3 (`/usr/bin/scx_cake`, sha256 + `56f9e886…`) attaches and runs clean on 7.1.8. The libbpf skeleton warning still + prints but is non-fatal. + - Quick wallclock ABBA vs nightly HEAD (diagnostic tier, + `scx_cake_bench_assets/runs/adhoc_wallclock_113_vs_nightly_20260819/`): nightly + pipe **−16.0%**, memcpy −3.9%, sched-messaging **+70.5%** (4/4, no overlap). + The many-to-many handoff shape regressed vs 1.1.3; unscored at the sealed tier. + - n=10 confirm (suite 4, 5×ABBA): schbench-light req p99 **+72% 10/10** (S sd + 7 µs, N sd 459), mutex-handoff p999 **2.0× 10/10** (S sd 0.024 µs), p99 + **BIMODAL** (4/10 runs match 1.1.3 exactly, 6/10 at 1.2–1.9 µs), pipe −9.7% + win 10/10. Slice dose falsified (EXP S.1-1ms). + - **MODE SWITCH ROOT-CAUSED 2026-08-19** (`mode_probe/` in the run dir): it is + `cake_frame_slice_ns`. Placement exonerated (24 runs, pair co-resident on + CPU 11 in both modes, sinks 0%). Dose-response with an injected 1 kHz + sleep-majority crowd (`--verbose` clock log as ground truth): crowd on binds + in one poll (950.8 Hz published) → p99 1.8–2.1; crowd off → 3-poll re-publish + + ~16 s floor climb → p99 decays to 0.90. + - A 374.9 Hz desktop voter holds the slice at 2 ms on this host even quiet, so + nightly never runs at the 3 ms cap on a live desktop. 1.1.3 has no frame clock + (constant geometry) — hence its sd ≈ 0. + - Mechanism: a fast desktop crowd tightens the patience windows (shifts of + `cake_frame_slice_ns`, §S.2) for every task; the same-CPU handoff tail pays. + Registered fix: §G12 KEYSTONE re-bases the `FRAME_*_SHIFT` windows onto the + occupant's own period, which decouples bystander tasks from the global clock. 8. **Receipts audit** — 5 load-bearing claims (deletion-queue zeros, 1464 ns, SLICE_NS dose, §R.17 +28-36%, G17/G21 wins); list + rationale: `REVIEW_INDEPENDENT_2026-08-17.md` Addendum (git history). diff --git a/scheds/rust/scx_cake/build.rs b/scheds/rust/scx_cake/build.rs index 55717485ce..509c533fa7 100644 --- a/scheds/rust/scx_cake/build.rs +++ b/scheds/rust/scx_cake/build.rs @@ -3,88 +3,11 @@ // This software may be used and distributed according to the terms of the // GNU General Public License version 2. -use std::collections::BTreeSet; -use std::env; -use std::fs; -use std::path::Path; - -fn detected_cache_domains() -> usize { - if let Ok(forced) = env::var("SCX_CAKE_FORCE_NR_CCDS") { - return forced - .parse::() - .ok() - .filter(|nr| *nr > 0) - .expect("SCX_CAKE_FORCE_NR_CCDS must be a positive integer"); - } - - let mut domains = BTreeSet::new(); - let Ok(cpus) = fs::read_dir(Path::new("/sys/devices/system/cpu")) else { - return 1; - }; - for cpu in cpus.flatten() { - let name = cpu.file_name(); - let Some(name) = name.to_str() else { continue }; - if !name - .strip_prefix("cpu") - .is_some_and(|suffix| !suffix.is_empty() && suffix.bytes().all(|c| c.is_ascii_digit())) - { - continue; - } - let Ok(indices) = fs::read_dir(cpu.path().join("cache")) else { - continue; - }; - for index in indices.flatten() { - if fs::read_to_string(index.path().join("level")) - .ok() - .is_some_and(|level| level.trim() == "3") - { - if let Ok(shared) = fs::read_to_string(index.path().join("shared_cpu_list")) { - domains.insert(shared.trim().to_owned()); - } - } - } - } - domains.len().max(1) -} - -fn detected_possible_cpus() -> usize { - if let Ok(forced) = env::var("SCX_CAKE_FORCE_NR_CPUS") { - return forced - .parse::() - .ok() - .filter(|nr| *nr > 0) - .expect("SCX_CAKE_FORCE_NR_CPUS must be a positive integer"); - } - - let Ok(cpus) = fs::read_dir("/sys/devices/system/cpu") else { - return 1024; - }; - cpus.flatten() - .filter_map(|cpu| cpu.file_name().to_str().map(str::to_owned)) - .filter_map(|name| name.strip_prefix("cpu")?.parse::().ok()) - .max() - .map_or(1024, |max_id| max_id + 1) -} +// No build-host inputs: topology is measured by the loader into rodata at +// attach, so one binary fits any machine (STEAL_SPAN in intf.h bounds the +// steal matrix). fn main() { - let nr_ccds = detected_cache_domains(); - let nr_cpus = detected_possible_cpus(); - println!("cargo:rerun-if-env-changed=SCX_CAKE_FORCE_NR_CCDS"); - println!("cargo:rerun-if-env-changed=SCX_CAKE_FORCE_NR_CPUS"); - println!("cargo:rustc-check-cfg=cfg(cake_multi_ccd)"); - if nr_ccds > 1 { - println!("cargo:rustc-cfg=cake_multi_ccd"); - } - - let mut post = env::var("BPF_EXTRA_CFLAGS_POST_INCL").unwrap_or_default(); - if !post.is_empty() { - post.push(' '); - } - post.push_str(&format!( - "-DCAKE_NR_CCDS={nr_ccds} -DCAKE_NR_CPUS={nr_cpus}" - )); - env::set_var("BPF_EXTRA_CFLAGS_POST_INCL", post); - scx_cargo::BpfBuilder::new() .unwrap() .enable_intf("src/bpf/intf.h", "bpf_intf.rs") diff --git a/scheds/rust/scx_cake/src/bpf/cake.bpf.c b/scheds/rust/scx_cake/src/bpf/cake.bpf.c index 978f3eca35..f0ebc05dca 100644 --- a/scheds/rust/scx_cake/src/bpf/cake.bpf.c +++ b/scheds/rust/scx_cake/src/bpf/cake.bpf.c @@ -23,8 +23,8 @@ _Static_assert((MAX_CPUS & (MAX_CPUS - 1)) == 0, "MAX_CPUS must remain a power of two"); _Static_assert((RECIP_TABLE_SIZE & (RECIP_TABLE_SIZE - 1)) == 0, "reciprocal table must remain mask-indexable"); -_Static_assert(CAKE_NR_CPUS <= MAX_CPUS, - "build-host CPU span must fit Cake MAX_CPUS"); +_Static_assert(STEAL_SPAN <= MAX_CPUS, + "steal matrix span must fit Cake MAX_CPUS"); char _license[] SEC("license") = "GPL"; @@ -477,10 +477,12 @@ static __noinline int cake_nonsink_rebuild(u32 gen) return 0; } -#if CAKE_NR_CCDS > 1 -/* Loader-sorted: same CCD, same cache-capacity tier, then unrestricted. */ -const volatile u16 cpu_steal_order[CAKE_NR_CPUS * CAKE_NR_CPUS]; -#endif +/* Loader-sorted: same CCD, same cache-capacity tier, then unrestricted. + * Fixed span so one binary fits any host; live only when the loader saw + * multiple CCDs AND the host fits the matrix — rodata, so the verifier + * folds the dead branch away on every other machine. */ +const volatile u16 cpu_steal_order[STEAL_SPAN * STEAL_SPAN]; +const volatile u8 steal_order_live; /* * Reciprocal-weight table for division-free vtime charging: @@ -1271,25 +1273,25 @@ void BPF_STRUCT_OPS(cake_enqueue, struct task_struct *p, u64 enq_flags) static __noinline bool cake_ring_steal(u32 ucpu) { u32 nr = nr_cpu_span; + u32 cw = (u32)-1; /* which qmask word `m` holds; none yet */ + u64 m = 0; u32 i; -#if CAKE_NR_CCDS > 1 && CAKE_CCD_STEAL_POLICY > 0 - /* One precomputed locality order avoids verifier-multiplying scan loops. */ - if (ucpu >= CAKE_NR_CPUS) - return true; - for (i = 0; i < MAX_CPUS; i++) { - u32 idx; + if (CCD_STEAL_POLICY > 0 && steal_order_live && ucpu < STEAL_SPAN) { + /* One precomputed locality order avoids verifier-multiplying + * scan loops. */ + for (i = 0; i < STEAL_SPAN; i++) { + u32 idx; - if (i >= CAKE_NR_CPUS || i + 1 >= nr) - break; - idx = cpu_steal_order[ucpu * CAKE_NR_CPUS + i]; - if (cake_qmark_test(idx) && - cake_move_to_local((u64)idx)) - return true; + if (i + 1 >= nr) + break; + idx = cpu_steal_order[ucpu * STEAL_SPAN + i]; + if (cake_qmark_test(idx) && + cake_move_to_local((u64)idx)) + return true; + } + return false; } -#else - u32 cw = (u32)-1; /* which qmask word `m` holds; none yet */ - u64 m = 0; for (i = 1; i < MAX_CPUS; i++) { u32 idx = ucpu + i, wi; @@ -1308,7 +1310,6 @@ static __noinline bool cake_ring_steal(u32 ucpu) if (cake_move_to_local((u64)idx)) return true; } -#endif return false; } diff --git a/scheds/rust/scx_cake/src/bpf/intf.h b/scheds/rust/scx_cake/src/bpf/intf.h index 3b748d9ed2..b436292ddc 100644 --- a/scheds/rust/scx_cake/src/bpf/intf.h +++ b/scheds/rust/scx_cake/src/bpf/intf.h @@ -22,15 +22,10 @@ typedef signed long s64; #endif /* __VMLINUX_H__ */ /* - * Build-host topology, detected by build.rs. The ONLY definitions a cflag may - * set: policy is source-only so an A/B is two commits, not two flags (§S.6). + * No cflag sets topology — the loader measures the host into rodata at + * attach, so one binary fits any machine. Policy is source-only so an A/B + * is two commits, not two flags (§S.6). */ -#ifndef CAKE_NR_CCDS -#define CAKE_NR_CCDS 1 -#endif -#ifndef CAKE_NR_CPUS -#define CAKE_NR_CPUS 1024 -#endif /* Multi-CCD steal order: 0 off, 1 same-CCD first, 2 also group cache tiers. */ #define CAKE_CCD_STEAL_POLICY 2 @@ -77,8 +72,9 @@ enum consts { FRAME_BUCKET_SHIFT = 17, FRAME_BUCKETS = 512, - NR_CCDS = CAKE_NR_CCDS, - BUILD_NR_CPUS = CAKE_NR_CPUS, + /* Widest host the CCD steal matrix covers (u16² = 32 KB rodata); + * wider machines take the generic ring walk at runtime. */ + STEAL_SPAN = 128, CCD_STEAL_POLICY = CAKE_CCD_STEAL_POLICY, /* Fixed-point weight scaling: representation, not policy (§S.7). */ diff --git a/scheds/rust/scx_cake/src/main.rs b/scheds/rust/scx_cake/src/main.rs index b190e81c4a..ed868c7b71 100644 --- a/scheds/rust/scx_cake/src/main.rs +++ b/scheds/rust/scx_cake/src/main.rs @@ -10,7 +10,6 @@ pub use bpf_skel::*; pub mod bpf_intf; pub use bpf_intf::*; -#[cfg(cake_multi_ccd)] use std::collections::BTreeMap; use std::mem::MaybeUninit; use std::sync::atomic::AtomicBool; @@ -232,7 +231,9 @@ impl<'a> Scheduler<'a> { bss.cake_frame_slice_ns = bpf_intf::consts_SLICE_NS as u64; } - #[cfg(cake_multi_ccd)] + // Multi-CCD steal order is a runtime decision, never a build-host + // property: one binary must attach on any topology. Hosts wider than + // the fixed matrix span keep the generic ring walk. { let rodata = skel .maps @@ -240,51 +241,57 @@ impl<'a> Scheduler<'a> { .as_mut() .context("BPF rodata unavailable for cache topology")?; let order = &mut rodata.cpu_steal_order; - let nr_span = bpf_intf::consts_BUILD_NR_CPUS as usize; - anyhow::ensure!( - topo.all_cpus - .keys() - .next_back() - .is_none_or(|cpu| *cpu < nr_span), - "runtime CPU topology exceeds Cake's build-host CPU span" - ); + let span = bpf_intf::consts_STEAL_SPAN as usize; + let fits = topo + .all_cpus + .keys() + .next_back() + .is_none_or(|cpu| *cpu < span); + let multi_ccd = topo.all_llcs.len() > 1; + rodata.steal_order_live = u8::from(multi_ccd && fits); order.fill(0); - let llc_cache: BTreeMap = topo - .all_llcs - .iter() - .map(|(id, llc)| { - ( - *id, - llc.all_cpus - .values() - .map(|cpu| cpu.cache_size) - .max() - .unwrap_or(0), - ) - }) - .collect(); - let policy = bpf_intf::consts_CCD_STEAL_POLICY; - - for src in topo.all_cpus.values() { - let mut candidates: Vec<_> = topo - .all_cpus - .values() - .filter(|dst| dst.id != src.id) + if multi_ccd && !fits { + warn!(" ccd host wider than steal matrix ({span} CPUs); ring steal only"); + } + if multi_ccd && fits { + let llc_cache: BTreeMap = topo + .all_llcs + .iter() + .map(|(id, llc)| { + ( + *id, + llc.all_cpus + .values() + .map(|cpu| cpu.cache_size) + .max() + .unwrap_or(0), + ) + }) .collect(); - candidates.sort_by_key(|dst| { - let class = if dst.llc_id == src.llc_id { - 0 - } else if policy > 1 && llc_cache[&dst.llc_id] == llc_cache[&src.llc_id] { - 1 - } else { - 2 - }; - (class, (dst.id + nr_span - src.id) % nr_span) - }); - let base = src.id * nr_span; - for (slot, dst) in candidates.into_iter().enumerate() { - order[base + slot] = dst.id as u16; + let policy = bpf_intf::consts_CCD_STEAL_POLICY; + let nr_ids = *NR_CPU_IDS; + + for src in topo.all_cpus.values() { + let mut candidates: Vec<_> = topo + .all_cpus + .values() + .filter(|dst| dst.id != src.id) + .collect(); + candidates.sort_by_key(|dst| { + let class = if dst.llc_id == src.llc_id { + 0 + } else if policy > 1 && llc_cache[&dst.llc_id] == llc_cache[&src.llc_id] { + 1 + } else { + 2 + }; + (class, (dst.id + nr_ids - src.id) % nr_ids) + }); + let base = src.id * span; + for (slot, dst) in candidates.into_iter().enumerate() { + order[base + slot] = dst.id as u16; + } } } }