Expand description
The validation harness: case files, oracle references, metrics and reports.
Accuracy gives every validation result. Checking a claim traces a number to its source, its test and its validation.
A case (Case) is a TOML file under validation/cases/. It says what to fly, how, and which
metrics to compare against which reference, each with its own tolerance. A reference
(Reference) is a JSON file under validation/fixtures/, written by a generator script
under validation/oracles/. It carries the oracle’s name, the command that produced it and a
source for every value. Running a case gives a Comparison per metric, and the run gives a
Report in Markdown and JSON.
Five lessons from Loft, mistakes found in the project that came before hpr-sim, shape the rules:
- An oracle’s inputs come from the case file, never from hpr’s own output.
- A reference is never regenerated to make a comparison pass. It moves only when its generator runs, which is a separate, deliberate step.
- Every reference value carries provenance, so no number is a hand-written “stored result”.
- The cases that must run are locked, so a suite that silently skips one fails instead of reporting green.
- Every metric a case reports has a tolerance; nothing is “ungated”.
The method is in the validation plan, and the design in the decision record on the validation harness.
Re-exports§
pub use case::Case;pub use case::CaseLock;pub use case::DragMode;pub use case::Metric;pub use case::Tolerance;pub use case::committed_cases;pub use metrics::Measured;pub use metrics::Reference;pub use metrics::ReferenceValue;pub use report::Comparison;pub use report::Gap;pub use report::NotReproduced;pub use report::Report;pub use report::Source;pub use report::Verdict;pub use run::CaseRun;pub use run::ValidateError;pub use run::run_case;pub use run::run_lock;
Modules§
- case
- Case files: what to fly, and which metrics to compare against which reference.
- census
- The accuracy census: the numbers the committed reports hold hpr to, counted once, classed by what they were compared with and how fast the flight went, and held to the census last accepted.
- committed
- The committed reports, held to a run and to the accepted census: the check
cargo xtask validate --checkmakes. - correction
- A correction of the bias the simulators share, flown on a public flight: every motor’s impulse scaled by one number and the drag by another, the rest of the flight as its report flies it.
- flight_
metrics - What a flight metric measures, tool by tool and version by version, and what becomes of a metric whose event never happened (M2.2d1).
- metrics
- References and measured values: what an oracle said, and what hpr said.
- openrocket
- hpr held to OpenRocket’s own answers (M2.2): the tests, and one rule of OpenRocket’s
that
cargo xtask orkneeds to explain a gap,openrocket_fin_set_roll_kg_m2. - predicted_
first - The public flights predicted before they were compared, each beside its log’s apogee and its error (the errors, decisions ADR-328 §1b and ADR-342).
- public_
predictions - Predictions merged before their errors: the public real flights that join the seven of
crate::real_flight, each flown from its record alone (M2.3d3, decisions ADR-328 §1a and ADR-334). - real_
flight - Real flights: hpr against the altitude logs of RocketPy’s example rockets, flown in the ERA5 weather of their day (M2.3b, decision ADR-082).
- report
- Reports: one line per metric, in Markdown for people and JSON for machines.
- rocketpy
- Reading RocketPy’s generator output into the harness’s shapes.
- run
- Running cases: fly what the case says, measure it, and compare against the reference.
- summary
- A run’s report in a line per case and one for the whole: what
cargo xtask validateprints. - tip_off
- HPR Sim’s tip-off phase, flown as RocketPy flew its own: the flights behind
validation/reports/tip-off-model.json(the tip-off phase, held to RocketPy’s; the tip-off decision record, ADR-357, decision 4).