ADR-0010 — Verification & Validation strategy: what "correct" means and how each claim is earned

Status: Accepted

Context

aeolus1d makes quantitative claims (shock-tube profiles, engine IMEP, light-off temperatures, transmission loss). A reviewer needs to know, per claim, how correctness was established — and the two questions are different:

• Verification — are the equations solved right? (a code-correctness question, answered without any experimental data)
• Validation — are they the right equations? (a physics question, answered against reality)

Three properties of the codebase shape the strategy: the feature surface is broad (many closures × schemes × drivers); several aftertreatment closures are lumped with rate constants calibrated to a reference (not first-principles); and there is no single closed-form exact solution for the coupled engine / EAT system. So "run it and eyeball the plot" is not a defensible standard, and a flat percentage screening gate hides which numbers are actually converged and which are fitted.

Decision

Verification (dataset-free, CI-gated).

• Order of accuracy is measured, not asserted: flux spatial order (Sod + smooth wave, test_smooth_advection.py), per-field running order for ρ, u and p (test_running_order_multivar.py, Richardson self-convergence), and temporal order for all seven integrators (test_temporal_order.py).
• Method of Manufactured Solutions covers the source closures, not just the flux: a manufactured energy source (test_mms_source.py) plus area variation, wall friction, and wall heat (test_mms_closures.py), each with the closure re-derived by hand + a load-bearing control so a wrong-signed / mis-coupled closure fails to converge.
• The catalyst reaction source is not covered by a spatial MMS, and this is a deliberate scoping, not an omission. It is stiff, operator-split chemistry (ADR-0001): unlike wall friction / wall heat / area variation — which are algebraic sources inside the spatial residual, and are MMS-verified above — the reaction term never enters that residual. It is a separate per-cell ODE sub-step, so its "discretisation" is the ODE integrator, and the thing to verify is that integrator, not a spatial order.

What that means concretely, because it is easy to over-claim:

- The shipped integrators (LSODA; Bader-Deuflhard / Bulirsch-Stoer) are
  **adaptive-order** extrapolators. "The order" is not a constant, so
  *accuracy* is the meaningful check. It is verified reference-free against a
  closed-form solution — a frozen-temperature reaction reduces the kernel's
  `(Y, E)` ODE to `dY/dt = -k·Y`, whose exact answer the extrapolator must
  reproduce (`tests/unit/test_chem_bs_neville_accuracy.py`; this is the test
  that caught the Neville-recurrence bug).
- The opt-in `chem_solver = "fixed_step"` path is plain backward Euler, whose
  order *is* a fixed number. Its observed order is measured as **1** on a
  nonlinear ODE with a closed-form solution, with a control that the estimate
  can reject second order (`tests/integration/test_chem_integrator_order.py`).
- `test_temporal_order.py` pins the seven **advection** time integrators on a
  chemistry-free MUSCL wave. It does **not** exercise chemistry, and must not
  be cited as evidence about the chemistry integrator. (It was, until
  2026-07-09.)

Splitting order — CLOSED 2026-07-31 (was a known gap: both operands were verified separately, their composition was not). tests/integration/ test_operator_splitting_order.py measures Strang ≈ 2.27 (gated to [1.7, 2.6]) and the Lie super-cycle ≈ 1.04–1.05 (gated to [0.8, 1.6]), with Lie ~300× less accurate than Strang at equal Δt.

Two preconditions make this measurable, and the file asserts rather than assumes them:

• The problem must not commute. A spatially-uniform linear decay under uniform advection has identically zero splitting error — pointwise chemistry commutes with translation — so it reads as pure noise however finely it is refined. Measuring the real rate needs Arrhenius kinetics, exothermic coupling, and a spatially non-uniform field.
• Δt must stay above the acoustic CFL step. The inner transport call sub-cycles at CFL, so below that threshold refining Δt refines the transport error too and the measured slope stops being the splitting order.
• Discrete conservation (mass / momentum / energy / species) is checked to machine precision in CI (test_discrete_balance.py), and metamorphic / property invariants guard the flux core (test_solver_metamorphic.py).
• A golden-snapshot regression net (tests/regression/) freezes canonical cases and fails on silent physics drift.

Validation (vs reality, honestly labelled).

• Calibration is distinguished from prediction in every artifact. A tuned constant reproducing one dataset is calibration; predictiveness is shown by blind application to a second condition (the predictive DD13 Watson/Hardenberg-Hase path, the a-priori Argonne Willans model, the EPW second-operating-point check). pass_criteria.toml headers carry the label.
• The reference type is stated — analytic, code-to-code, or experiment — and self-consistency checks (turbo, hybrid) are labelled as such, not as validation.
• Quoted numbers carry an uncertainty. Where mesh is the meaningful term, a Roache / ASME V&V-20 GCI study is run (not just available) and the result reported as u_val = √(u_num² + u_data²) via avl_compare_common.gci_mesh_study — e.g. DOC light-off T50 (converged, ±0.3 K), c02006 regen peak-T (converged, ±0.8 K), and the SCR NO-conversion (shown not converged, ±10 pts — reported, not quoted as validated). For a calibrated lumped closure GCI on u_num is misleadingly small — the total uncertainty is dominated by the fitted-rate term u_D — so those cases are labelled GCI-not-applicable — calibrated rather than given a false mesh band.
• No fake gates. Per-quantity, physically-justified tolerances; a known gap is reported (or bounded by a regression ceiling), never gated at an always-pass threshold.

The honest, dated summary of all of this lives in docs/scope-and-limitations.md; the per-number evidence in docs/validation-dossier.md.

Consequences

• A reader can tell, for any number, whether it is verified (converges at the formal order), validated (matches experiment with a stated u_val), or calibrated / demonstrative (fitted or code-to-code) — the labels are a non-negotiable part of every new validation suite.
• New suites follow the pattern: state the reference type, distinguish calibration from prediction, run GCI where mesh is the meaningful uncertainty, and gate per-quantity (no always-pass rows).
• The strategy is now a decision record, not tribal knowledge in the docs prose or in a maintainer's head.

References

• Verification: tests/integration/{test_mms_source,test_mms_closures,test_running_order_multivar,test_temporal_order,test_discrete_balance,test_smooth_advection}.py.
• Validation / UQ: src/aeolus1d/validation.py (GCI, V&V-20 u_num/u_data), examples/avl_cruise_m/compare/avl_compare_common.py (gci_mesh_study).
• Docs: docs/scope-and-limitations.md, docs/validation-dossier.md, docs/numerics.md.
• Related: ADR-0001 (why catalyst chemistry is integrator-verified, not spatially MMS'd), ADR-0006 (the golden net as the bit-identity guard).