CI runner runbook (maintainers)¶

How aeolus1d's self-hosted GitLab runners are registered, kept alive, and recovered. Companion to Release process and to ci/build-image.sh (which documents the CI image rebuild; this page covers the runners themselves — the part that was previously tribal knowledge).

Runner inventory¶

Runner	ID	Tags	Host	Executor	Role
`aeolus-docker`	55362415	`aeolus-x86`, `docker` (+ untagged)	Xeon E5-2643 v0 (Sandy Bridge-EP, 2012), 4c/8t @ 3.3 GHz — no AVX2, no FMA, no AVX512	docker	Primary since 2026-08-20. Takes every untagged tier plus `gui-tests` (pinned via `tags: [aeolus-x86]`). 1.5x-2.0x slower than the laptop, and the factor depends on the job — measured, see below.
`aeolus-wsl2-docker`	53205253	`aeolus-wsl-linux`, `docker`	WSL2 Ubuntu 22.04, x86_64, `tigerlake` (i7-11850H, 16 threads — the maintainer's laptop)	docker, default image `python:3.10`	Bench host. The ONLY host for `wallclock-budget` and `perf-budget`, both of which assert numbers calibrated on this hardware. Was the primary until 2026-08-20.
mac-mini	53432781	`mini`	Apple-silicon mac mini (ARM)	docker	Secondary / overflow for architecture-neutral tiers, currently paused. Cannot run `gui-tests` (ARM-emu PySide6/Shiboken6 wheels SIGABRT at Qt collection / VTK render) and must never run wall-clock gates.

All three runners are project-scoped and effectively single-concurrent (concurrent = 1 in /etc/gitlab-runner/config.toml). The WSL2 runner is also shared with the proteus project, so a busy queue there delays aeolus1d pipelines — that is queue contention, not a failure; see When the runner dies mid-queue for how to tell the difference.

The offload bought queue availability, not speed — and that is the bigger win

Measured end-to-end on the two main pipelines either side of the move. 2776629638 (2026-08-20, everything on the laptop) against 2778531730 (2026-08-21, everything on aeolus-docker except regression-bitid):

	all-laptop	new box
Job compute (pipeline `duration`)	2891 s	4466 s — 1.54x slower
Wall clock, created -> finished	3 h 38 m	1 h 15 m
Time not spent running jobs	2 h 50 m	26 s

So the new box runs every job slower and still returns merge feedback ~2.9x sooner, because the laptop spent 2 h 50 m of that pipeline queued behind proteus for its one concurrency slot. Compute was never the bottleneck; the shared slot was.

Two consequences worth holding on to:

Do not evaluate a runner move on per-job ratios alone. By the 1.5x-2.0x table below, this move looks like a straight regression. By turnaround — the thing a maintainer actually waits on — it is a 2.9x improvement.
The bench host's contention is now a feature. wallclock-budget and perf-budget stay on the box that shares a slot with proteus, which is exactly the condition that produced the three false perf failures in the trap table. They run on tag and schedule only, so nothing blocks on them — but check /proc/loadavg before believing a red one, as CLAUDE.md requires.

Caveat, per this repo's own rule: one pipeline each. The laptop's queueing depends on whatever proteus was doing that afternoon, and the 2 h 50 m figure is one sample of a quantity with no reason to be stable. The direction is solid; the magnitude is not a calibrated number.

The two x86 boxes are NOT interchangeable for timing — 1.5x to 2.0x

There is no single ratio between these hosts. Measured across three jobs on the first main pipeline after the offload (2778531730, 2026-08-21) against the same jobs on the two main pipelines before it, when the laptop still ran everything:

Job	`aeolus-wsl2-docker`	`aeolus-docker`	Ratio
`unit`	422 s / 424 s	643 s	1.52x
`integration-slow`	1003 s / 1513 s	1767 s	1.17x-1.76x
`integration-tests`	381 s	752 s	1.97x

Quote the job with the number. "The new box is 1.55x" was recorded here for a day and is a unit-only figure presented as a property of the machine; integration-tests is half again worse than that. Scaling any other job by a ratio measured on unit will under-predict it.

The laptop's own spread is part of the picture. integration-slow read 1003 s and 1513 s on the same host on consecutive main pipelines — a 1.5x swing with no code change. Ratios computed against a single baseline reading inherit that noise, which is why the row above gives a range rather than a point. Take the best of several before trusting one.

First-run figures are neither. The first unit on the new box read 1579 s and was cancelled at 1820 s without finishing, and was briefly recorded here as "3.7x". That run paid a cold numba JIT cache on a fresh container plus a ~2.5 min image pull. All of these numbers are real; they measure different things. Expect roughly 2.4x the steady-state figure on the first job after a runner rebuild or a cache wipe.

The gap is not mysterious once you look at the silicon: the bench host is an i7-11850H (Tiger Lake, 2021) with AVX2, FMA3 and AVX512; the new box is a Xeon E5-2643 v0 (Sandy Bridge-EP, 2012) whose flags carry avx but neither avx2 nor fma. For vectorised numeric work that is a generational difference, not a tuning one.

So "x86_64" is not a sufficient description of a perf host. Both perf gates stay pinned to the laptop for this reason — perf-budget was briefly moved to aeolus-x86 on the assumption that "host-tolerant" meant "any x86 box". Its last green reading was 77.8 s against an 80.0 s ceiling, so it clears by 2.8 %; any of the ratios in the table above puts it red (1.52x gives ~118 s, 1.97x gives ~153 s). Note that is a projection, not a measurement — perf-budget has never run on the new box, and the table says a projection from another job is exactly what you should not trust. It is quoted here only to show that the sign of the answer does not depend on which ratio you pick. A red gate nobody can act on is worse than no gate: it is the setup in which someone refreshes a reference number to get green, which CLAUDE.md forbids outright.

Budget accordingly when moving any timed job, and re-measure before trusting a number from a host that did not produce the reference.

2026-08-28 — the first job this caught was not one of the perf gates. A timeout is a reference number too, and nobody thinks of it as one. tests/integration/test_no_dead_jit.py bounds its probe subprocess at 900 s, sized against a 198 s figure measured on the laptop on 2026-08-13 — eight days before the offload. Nothing re-measured it on the new box, and -n auto then had the probe competing with a worker per CPU for the same cores. Main pipeline 2797879100 failed on it twice, original and retry; the same sha ran the module serially in 259 s on an idle box. The repair was a serial job of its own (integration-dead-jit), not a bigger number. When auditing what a host move invalidated, read timeout= and pytest-timeout settings alongside the perf references.

That job's first run measured 606 s on aeolus-docker against 259 s on the laptop — 2.34x, outside the 1.17x–1.76x row above. The projection that this MR itself carried for a few hours said 230–350 s, from that row; it earned the "do not carry a ratio between jobs" warning directly beneath it. A cold full-surface numba compile is not the same mix of work as the tier it sits in. The 900 s timeout is now 67 % spent.

Corrected 2026-08-31. That 67 % is n = 1, and this page is the place a reader would take it from. The job has since read ten times and the distribution is bimodal, so the honest figure is the worst case, not that run and not a median: 70 % spent, 1.43x of headroom. The readings and the arithmetic live in .gitlab-ci.yml's integration-dead-jit comment, which owns them; this page states the consequence and points there rather than keeping a second copy. The lesson is the row's own: a timeout is a reference number, and a reference number quoted from one run is a guess with a unit on it.

Why two x86 tags and not one. aeolus-x86 means "any x86_64 Linux runner"; aeolus-wsl-linux means "the specific box the wall-clock reference was measured on". Collapsing them would make wallclock-budget eligible for aeolus-docker, where ref_wall_s describes nothing — a red gate that invites someone to refresh the reference number, which is the one repair CLAUDE.md forbids. The split is enforced by tests/unit/test_ci_runner_tag_is_external.py.

A job tagged with a string no runner advertises does not fail — it sits pending with runner: None forever, which reads as "still running" rather than broken. That has already cost this project one debugging session (see the guard test's docstring). When adding or renaming a runner tag, change the runner first, the pipeline second.

Why single-concurrency: the heavy tiers (integration, GUI under xvfb, wall-clock benches) contend for the same few CPUs. A past experiment with concurrent = 4 slowed individual tests ~2× (e.g. test_scrf_no_depletes 82 s alone → ≥180 s under load), which is fatal for anything timing-related. Keep it at 1 unless you are deliberately draining a backlog and no perf job is queued.

Single-runner SPOF — accepted risk & redundancy¶

Update 2026-08-20: the merge-gate SPOF moved rather than disappeared. It is now aeolus-docker, which carries gui-tests:

gui-tests — a hard merge gate (VTK/PySide6 under xvfb, serial -n 0; the ARM secondary SIGABRTs on emulated Qt/VTK wheels). Pinned to aeolus-x86. If aeolus-docker is down, merges block.
wallclock-budget — the host-specific perf gate (AEOLUS_BENCH_HOST=1; its ref_wall_s numbers are only valid on the WSL2 laptop). Pinned to aeolus-wsl-linux. Not a merge gate: it runs on tag + schedule only, so losing that box delays a release, not a merge.

The nightly schedule (added 2026-08-30)¶

0 2 * * * Europe/Stockholm on main, GitLab schedule id 4413025. Recorded here because a schedule is project configuration, not repository content — nothing in a clone reveals whether it exists, and until this one was created the if: $CI_PIPELINE_SOURCE == "schedule" rules in .gitlab-ci.yml had never fired even once. perf-budget and wallclock-budget had only ever run on tags.

A scheduled pipeline starts 17 of the 23 jobs, but the runner split keeps the measurement clean without any per-job configuration:

host	`run_untagged`	takes on a nightly
`aeolus-wsl2-docker` (laptop)	false	only `perf-budget`, `wallclock-budget`, `regression-bitid`
`aeolus-docker`	true	the other 14

regression-bitid is in stage test and the two perf gates in integration, so it cannot overlap them.

Two things to check on the first fires, neither visible from here. The laptop has to be awake at 02:00 — a schedule fires whether or not a runner is available, and the jobs simply queue. And both perf gates share the integration stage on that one runner, so whether they serialise depends on its concurrent setting in config.toml; if they overlap, each measures the other's load, which is the contention-read-as-regression failure that has already cost this project two false alarms (2026-08-08, 2026-08-12).

Verify the schedule still exists — it can be deactivated or retargeted by anyone with access, and no test would go red:

glab api "projects/:id/pipeline_schedules"

The second runner therefore buys queue capacity and a quiet laptop, not merge-gate redundancy — the gate simply has a new sole host. Making it real redundancy means putting aeolus-x86 on a second box too, at which point gui-tests has a substitute for the first time.

If the pinned host is down, any pipeline that needs its job blocks until it is recovered. Recovery — not hot standby — is the mitigation; it is fast (usually minutes) and fully scripted in When the runner dies mid-queue.

Redundancy that does exist. A GitHub Actions mirror (.github/workflows/ci.yml, GitHub-hosted ubuntu-latest) independently runs the architecture-neutral tiers — ruff, mypy-strict, docs, smoke, fast, unit, integration, gui — on every push / PR. So a total WSL2 outage still gets those tiers checked off-box, on different hardware and a different CI provider, which also guards against a GitLab-runner-specific environment bug. Caveats: the GitLab pipeline remains authoritative (the mirror is not wired as the merge gate), and the mirror cannot reproduce the two WSL2-pinned jobs above — host-specific wall-clock references and the VTK-under-xvfb GUI path that the ARM box cannot run.

Why not a second equal runner. The blocker is not cost but fidelity: wall-clock references are host-specific (a second host needs its own best-of-5 re-measurement before it can arm the gate — see AEOLUS_BENCH_HOST semantics), the ARM secondary SIGABRTs on GUI wheels, and single-concurrency is a deliberate timing-stability choice. So the accepted posture for a bus-factor-≈1 project is: one primary + the GitHub mirror for neutral tiers + a fast documented recovery, not a hot standby. Revisit if the project gains a second maintainer. (A native-ARM CI image is no longer a reason to revisit: cross-arch reproducibility was scoped out on 2026-07-31 — aeolus1d targets x86-64 only, see MATURITY.md R1.)

2026-08-20 — half of this paragraph was overtaken, half was vindicated

A second x86 runner (aeolus-docker) was added, for a reason the paragraph above does not consider: not redundancy, but getting CI load off the maintainer's laptop. So "one primary" is no longer the posture.

The fidelity argument held exactly as written, and is why the move was partial: gui-tests and perf-budget went across, wallclock-budget did not, because its reference has not been re-measured best-of-5 on the new box. That re-measurement is the tracked item in BACKLOG.md. Until it lands, the laptop is still required — for one tag-and-schedule-only job. (Annotated 2026-10-07: perf-budget did not stay across; it went back to aeolus-wsl-linux (see "AEOLUS_BENCH_HOST semantics" below), so the laptop is required for perf-budget and wallclock-budget, and for regression-bitid, as the table above says.)

Registering (or re-registering) a docker runner¶

You re-register when: the runner token was rotated/revoked, the config.toml was lost, the host was rebuilt, or GitLab shows the runner as permanently offline. The same recipe stands up an additional runner.

Pick the tag before you register — see the inventory for what each one means:

Role	`tag_list`	`run_untagged`
General x86 capacity + the `gui-tests` pin (`perf-budget` is not on this tag: it is pinned to `aeolus-wsl-linux`, corrected 2026-10-07)	`aeolus-x86,docker`	`true`
Wall-clock bench host (only after a best-of-5 re-measurement)	`aeolus-wsl-linux,docker`	`false`
ARM overflow	`mini`	`false`

Do not invent a near-miss of an existing tag. aeolus-linux next to aeolus-wsl-linux is one word from silently re-pointing the wall-clock gate, and a mistagged pin hangs rather than fails.

Install (once per host):

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner docker.io
sudo systemctl enable --now gitlab-runner docker

Mint a runner authentication token. The project access token (the one in the git remote URL / .env) has API scope, so you can mint a glrt-… token directly instead of clicking through the UI:

curl -s -X POST -H "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/user/runners" \
  -d "runner_type=project_type" \
  -d "project_id=<PROJECT_ID>" \
  -d "description=<RUNNER_NAME>" \
  -d "tag_list=aeolus-x86,docker" \
  -d "run_untagged=true" | python -m json.tool
# → copy the "token": "glrt-…" field

(UI alternative: Project → Settings → CI/CD → Runners → "New project runner".)

Register non-interactively:

sudo gitlab-runner register --non-interactive \
  --url "https://gitlab.com" \
  --token "glrt-…" \
  --executor docker \
  --docker-image "python:3.10" \
  --docker-pull-policy "if-not-present" \
  --docker-volumes "/cache"

Post-register checks in /etc/gitlab-runner/config.toml:
- concurrent = 1 (top of file — see rationale above).
- pull_policy = ["if-not-present"] — Docker-Hub TLS to auth.docker.io is unreliable from this WSL2 box (x509: certificate signed by unknown authority), so jobs must be able to run from locally cached images.
- Then sudo systemctl restart gitlab-runner and confirm the runner shows green under Project → Settings → CI/CD → Runners.
Rebuild the CI images locally (ci/build-image.sh). The runner uses locally built images, not registry pulls (same TLS issue), so after a distro rebuild the registry.gitlab.com/...:py310 / :py39 tags don't exist yet and every job would fall back to stock python:3.10 (symptoms: ruff: command not found, No module named mypy). Run the script on the runner host; pushing to the registry is optional.

Token rotation¶

The glrt-… runner authentication token lives only in /etc/gitlab-runner/config.toml (root-owned). To rotate: mint a new token (step 2), run gitlab-runner register with it (step 3), then remove the stale runner entry from config.toml and from the GitLab runners page. There is no in-place rotate for project runners — re-registration is the rotation.
The project access token (git remote / API) is independent; rotating it does not affect an already-registered runner, but it does break the minting recipe above and git push from the bot — update .env.
Never echo either token into job logs.

First-stand-up gotchas (will bite again on a fresh host)¶

register without sudo writes to the WRONG FILE, and the runner looks fine. Hit on the aeolus-docker stand-up, 2026-08-20. Run as a normal user, gitlab-runner register writes ~/.gitlab-runner/config.toml; the system service reads /etc/gitlab-runner/config.toml and never loads it. The runner is created in GitLab and picks up nothing, with no error anywhere.

What makes it hard to spot: the runners page shows a real "last contact" timestamp — from the registration handshake itself — so the runner does not look dead. The discriminator is whether last-contact advances. A polling runner's does; a registration-only contact is frozen at the moment it was created. "Never contacted" is a different fault; do not wait for it as the tell.

Fix: delete the runner in the UI, rm ~/.gitlab-runner/config.toml, and re-register under sudo. Then confirm Checking for jobs... lines in journalctl -u gitlab-runner -f before believing it works.

(This box has a second, unrelated instance of the same stray file: a runner named eats pointing at gitlab.scania.com that the system daemon does not serve. Check for one before assuming a clean host.) * Identity verification — gitlab.com free tier requires phone or payment verification (https://gitlab.com/-/identity_verification) before ANY pipeline dispatches jobs, even to self-hosted runners. Until cleared, pipelines instantly flip to failed with zero jobs and yaml_errors: null. This is account-level, one-time. * GIT_DEPTH: "0" (set globally in .gitlab-ci.yml) — GitLab's default shallow clone (~50 commits) breaks the perf-history-refs check, which resolves hashes 200+ commits deep. Don't "optimise" it back to a shallow clone. * xdist + VTK/numba — GUI tests run xvfb-run -a pytest -n 0: serial (xdist workers segfault on concurrent numba JIT-cache hits and forked VTK GL contexts) and under xvfb (the Qt offscreen platform is not enough for VTK's OpenGL2 renderer). * PyPI flakes — files.pythonhosted.org reads time out from this box; PIP_DEFAULT_TIMEOUT=60 / PIP_RETRIES=10 are set globally, and the baked CI image avoids most installs entirely. * Single default: block in .gitlab-ci.yml — a second top-level default: key silently clobbers the first (YAML last-key-wins), dropping image: + before_script:. The authoritative check is the job trace header (Using docker image …), not ci/lint.

The ARM mac-mini secondary (53432781)¶

Registered the same way (executor docker), tagged mini. Scope note (2026-07-31): aeolus1d targets x86-64 only — cross-arch reproducibility was scoped out (MATURITY.md R1), so this box is overflow capacity for architecture-neutral tiers and is deliberately not part of the reproducibility story. Constraints:

No gui-tests (pinned off via tags: [aeolus-wsl-linux] on the job) — ARM-emu PySide6 wheels abort. This is a standing limitation, not a tracked item to close.
No wallclock-budget / perf gates — wall-clock references are host-specific (see below).
Architecture-neutral tiers (lint, unit, docs) run fine; it exists to absorb queue pressure when the WSL2 runner is busy.

Update 2026-07: the mac-mini secondary is currently paused (see the note on the gui-tests job in .gitlab-ci.yml) — aeolus1d pipelines run on the WSL2 runner alone. Re-enable it on the GitLab runners page when overflow capacity is needed again. (Annotated 2026-10-07: "the WSL2 runner alone" is stale beside the inventory above. Since 2026-08-20 aeolus-docker is the primary runner and the WSL2 runner is the bench host; the mac-mini is the paused one.)

AEOLUS_BENCH_HOST semantics (perf gates)¶

Wall-clock is host-SPECIFIC: the c02006 reference (~192 s full case, best-of-5, stable to 0.1% on the WSL2 runner) would falsely trip anywhere else. The contract:

wallclock-budget is pinned to tags: [aeolus-wsl-linux] AND sets AEOLUS_BENCH_HOST: "1". The env var arms the strict 2× wall-band asserts in tests/benchmarks/; without it those tests skip, so a developer laptop or the mac-mini can still run the suite without spurious failures.
perf-budget (cProfile self-time, ~1.8× ceilings) is host-tolerant, and until 2026-08-20 carried no pin at all — but both jobs run only on schedule (nightly) and tag pipelines, keeping the bench host free for merge pipelines. A perf regression merged to main therefore surfaces on the next nightly, not on the merge.

Superseded the same day (annotated 2026-10-07): perf-budget is pinned to aeolus-wsl-linux, not aeolus-x86. The paragraph below is the first form of the pin and is kept as the record. The reasoning is in the 1.5x-2.0x section above ("both perf gates stay pinned to the laptop"), and .gitlab-ci.yml (perf-budget, tags: and the comment above it) is the source of truth; tests/unit/test_ci_runner_tag_is_external.py pins it.

Pinned 2026-08-20 to aeolus-x86. Host-tolerant is not host-free. Leaving the row untagged was harmless while one runner served the project and became a measurement hazard the moment a second one did: an untagged row alternates hosts between pipelines, so a number it prints cannot be attributed to a box. CLAUDE.md asks for the harness with every measurement; this pin is what makes "which box?" answerable. Its ~1.8× ceilings were calibrated on the WSL2 laptop and are not yet re-validated on aeolus-docker — the last green run came in 77.8 s against an 80.0 s ceiling, a 2.2 s margin that does not survive an unexamined hardware change. Tracked in BACKLOG.md. * If the bench host changes (new hardware, WSL2 rebuild), the reference numbers must be re-measured best-of-5 before re-arming the gate — do NOT carry ref_wall_s across hosts.

Pipeline cost: the two things that made every MR expensive¶

Both fixed 2026-08-24, both measured first. Recorded here because the symptom — "CI is slow" — has two independent causes and fixing one leaves the other.

One MR used to create two full pipelines¶

workflow:rules already dropped the branch pipeline when a merge request was open. It could not help with the first push, because the ordinary sequence is push the branch, then open the MR — and at push time no MR exists. So every MR got a branch-ref pipeline and a refs/merge-requests/N/head pipeline.

Measured on !464, a single edited Markdown file: 50.1 + 59.3 = 109 runner-minutes. Thirteen pipelines ran that day for four merge requests.

The fix is one rule — a branch push with no open MR now creates nothing:

- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS == null
  when: never

What it costs. Pushing a branch gives no CI feedback until an MR exists. That is the entire saving, and it is acceptable because main is protected — every change lands through an MR, so a pre-MR pipeline was never on the path to landing. Open the MR to get a pipeline; iterating on an open MR is unaffected, and auto_cancel: on_new_commit: interruptible still collapses stacked pushes.

BACKLOG.md's §4.8b proposed push.pushOption merge_request.create instead, so the MR would exist before the branch pipeline was evaluated — and recorded that the ordering is "not documented and not verified". This is the deterministic form of the same intent: no ordering assumption, no repo-local git config, nothing a fresh clone can miss.

A one-line Markdown edit used to run the whole test matrix¶

CLAUDE.md's doc-currency table makes a .md edit part of finishing nearly every change, so this is the highest-traffic path in the repo, not an edge case.

ci/changed_scope.py answers one question — is every changed path documentation? — and its module docstring carries the full reasoning. The three things worth knowing here:

It fails safe. A path nobody put in the allowlist does not match, so the answer is "no" and the full pipeline runs. A missing allowlist entry costs a wasted run, never a skipped gate. This is what rules:changes cannot do (it is OR-over-globs, so **/*.md fires on any MR that touches one Markdown file, code or not) and what a positive whitelist of non-doc paths gets backwards (an unlisted path drops the heavy jobs).
A stood-down job is green with a banner, not skipped. A GitLab "skipped" badge is indistinguishable from "not needed" — the same defect the skip census exists to fix. Here the job ran and said what it decided.
unit drops coverage, not tests. It owns every doc-currency guard, so on a docs-only change the whole tier runs uninstrumented. The first version narrowed to a hand-written -k list of seven guard names and that was wrong — a whitelist of test names fails open exactly as a whitelist of paths does. Measured on review, it missed 25 modules under tests/unit that read documentation, among them test_closures_reference_doc, test_maturity_claims_match_code, test_run_settings_doc_coverage and test_ci_docs_fast_path — the test pinning this very feature. test_ci_docs_fast_path now fails if -k or -m reappears there.
Generated pages are not documentation for this purpose. docs/icon-reference.md, examples-python.md, closures-reference.md and benchmarks-baseline.md are produced from code and each has a currency test. Editing one by hand is a claim about the code, so changed_scope.GENERATED_DOCS forces the full pipeline. Without it, editing icon-reference.md stood down gui-tests — the one job whose changes: rule lists that file precisely so the edit is checked.
docs, ruff, lock-verify, build-smoke and wheel-install-smoke run unconditionally; the last two because the documentation ships inside the wheel, and v0.3.1 shipped one with zero doc files.

The coupling is pinned by tests/unit/test_ci_docs_fast_path.py, which fails if a guard falls out of a job or lands on one that must never stand down.

Measured on the first real docs-only MR (!466 — two files, both under .loop-review/), which is what this section previously carried a projection for. The projection said ~58 min saved; the actual saving is ~104, because it counted only the tier standing down and missed that it compounds with the duplicate pipeline disappearing:

	!464 (before)	!466 (after)
Pipelines	2	1
Runner-minutes	109.4	5.7

That 5.7 was measured against a design since found unsafe, and the correction costs some of it back. unit now runs its full tier uninstrumented on a docs-only change (~490 s, from unit-py39's figure for the same selection without coverage) instead of a 55 s subset, so expect roughly 13 min rather than 5.7 — still ~8x better than the 109 it replaced, and this time nothing is silently unguarded. Re-measure on the next real docs-only MR and record it here, as this section already asks.

Per job, the guard shows in both directions: the stood-down tier returns in 11-17 s each (regression 15.2, integration-tests 16.3, integration-slow-mr 14.3, unit-py39 14.7, smoke 16.7, mypy-strict 10.9, sbom 12.3), while unit runs 55.4 s — the narrowed doc-guard subset, not zero — and the unconditional jobs run in full (wheel-install-smoke 75.6 s, docs 43.3 s, build-smoke 34.4 s, ruff 18.0 s).

When the runner dies mid-queue¶

Symptoms and the decision tree:

Jobs sit pending for a long time — first check whether the runner is merely busy: it is single-concurrent and shared with proteus. curl -s -H "PRIVATE-TOKEN: $TOKEN" "https://gitlab.com/api/v4/runners/53205253/jobs?status=running" shows what it's chewing on. If something is running, it's contention — wait.
Runner offline (red on the runners page, nothing running): on the WSL2 host check systemctl status gitlab-runner and docker ps; a Windows reboot stops the WSL2 VM and everything in it. wsl.exe into the distro and sudo systemctl start gitlab-runner docker. The systemd units are enabled, but WSL2 itself does not auto-boot after a Windows restart until something opens the distro.
Jobs stuck after the runner comes back — jobs that were running when the runner died eventually fail with a stuck/timeout state; jobs that were pending resume on their own. Retry the failed ones:

# list the casualties on a pipeline
curl -s -H "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/projects/<PROJECT_ID>/pipelines/<PIPELINE_ID>/jobs?scope[]=failed" \
  | python -c "import json,sys; [print(j['id'], j['name']) for j in json.load(sys.stdin)]"
# retry one
curl -s -X POST -H "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.com/api/v4/projects/<PROJECT_ID>/jobs/<JOB_ID>/retry"

or simply hit "Retry" on the pipeline page. For a wedged pipeline (shows running but no job is), cancel it and POST .../pipelines/<id>/retry. 4. Jobs fail instantly with no log — re-check identity verification (account-level holds re-trigger this failure mode) and that the job's tags: still match a live runner.

ci/build-image.sh — rebuilding the baked CI images (rerun after any pyproject.toml dependency change; the runner consumes the local tags).
ci/verify_locks.sh — hash-pinned lockfile verification job.
Release process — tag pipeline expectations.