Expand description
The validation harness: case files, oracle references, metrics and reports.
Guide: Accuracy gives every validation result, and Checking a claim traces a number to its source, its test and its validation.
A case (Case) is a TOML file under validation/cases/: what to fly, how, and which
metrics to compare against which reference, each with its own tolerance. A reference
(Reference) is a JSON file under validation/fixtures/, written by a generator script
under validation/oracles/, carrying the oracle’s name, the command that produced it and a
source for every value. Running a case gives a Comparison per metric, and the run gives a
Report in Markdown and JSON.
Five lessons from Loft, mistakes found in the project that came before hpr-sim, shape the rules:
- Loft lesson L75: an oracle’s inputs come from the case file, never from hpr’s own output.
- L76: a reference is never regenerated to make a comparison pass; it moves only when its generator runs, which is a separate, deliberate step.
- L77: every reference value carries provenance, so no number is a hand-written “stored result”.
- L78: the cases that must run are locked, so a suite that silently skips one fails instead of reporting green.
- L79: every metric a case reports has a tolerance; nothing is “ungated”.
The method is in the validation plan, and the design in the decision record ADR-015, the validation harness.
Re-exports§
pub use case::Case;pub use case::CaseLock;pub use case::DragMode;pub use case::Metric;pub use case::Tolerance;pub use case::committed_cases;pub use metrics::Measured;pub use metrics::Reference;pub use metrics::ReferenceValue;pub use report::Comparison;pub use report::Gap;pub use report::NotReproduced;pub use report::Report;pub use report::Source;pub use report::Verdict;pub use run::CaseRun;pub use run::ValidateError;pub use run::run_case;pub use run::run_lock;
Modules§
- case
- Case files: what to fly, and which metrics to compare against which reference.
- census
- The accuracy census: the numbers the committed reports hold hpr to, counted once, classed by what they were compared with and how fast the flight went, and held to the census last accepted.
- committed
- The committed reports, held to a run and to the accepted census: the check
cargo xtask validate --checkmakes. - flight_
metrics - What a flight metric measures, tool by tool and version by version, and what becomes of a metric whose event never happened (M2.2d1).
- metrics
- References and measured values: what an oracle said, and what hpr said.
- openrocket
- hpr held to OpenRocket’s own answers (M2.2): the tests, and one rule of OpenRocket’s
that
cargo xtask orkneeds to explain a gap,openrocket_fin_set_roll_kg_m2. - real_
flight - Real flights: hpr against the altitude logs of RocketPy’s example rockets, flown in the ERA5 weather of their day (M2.3b, decision ADR-082).
- report
- Reports: one line per metric, in Markdown for people and JSON for machines.
- rocketpy
- Reading RocketPy’s generator output into the harness’s shapes.
- run
- Running cases: fly what the case says, measure it, and compare against the reference.
- summary
- A run’s report in a line per case and one for the whole: what
cargo xtask validateprints.