Online data and the cache

This page covers hpr-net, the one crate that uses the network, and the cache that makes its answers work offline. It is for anyone who will pull weather, elevation or motor data into a flight. Today the crate holds the cache, the offline rule and an HTTP client (M5.1, the online layer), and six data sources: Open-Meteo’s weather (Launch-day weather), weather-balloon soundings from the University of Wyoming (Weather-balloon soundings), NOAA’s GFS and RAP forecasts (NOAA forecasts: GFS and RAP), Open-Meteo’s ground elevation (A launch site’s elevation), motor.fusionspace.co’s motor stock and prices, and ThrustCurve.org’s motor records and thrust curves (both on Motor stock and prices; hpr sim and hpr motors fetch fetch a curve the bundled catalog lacks, Motors from ThrustCurve.org). Its tests replay a small hand-written sample response and answers recorded from each, from a folder and from a test web server on the machine running the tests, never the live network. Those tests speak plain HTTP only: encrypted HTTPS was checked by hand, against Open-Meteo and then through the command line’s hpr weather against all four sources, not in CI.

What it promises

hpr’s simulation itself never touches the network (the architecture’s pure-core rule: no filesystem, network or clock). Anything fetched from the internet goes through a client, which puts a cache (a folder of saved responses) in front of a transport (the thing that actually fetches). The client runs in one of two modes:

• Online: if the cache holds a fresh copy, the client returns it without fetching. Otherwise it fetches, saves the response, and returns it. If the fetch fails and an old copy exists, it returns the old copy, marked stale, with the reason the fetch failed; with no old copy, the fetch’s error.
• Offline: it never calls the transport. It returns whatever the cache holds, marked stale if it is old, or an error saying the URL is not cached.

Each answer says how fresh it is (Fetched, Cached or Stale), when it was fetched, and the data source’s attribution, the credit line that its terms ask you to show with the data.

A data source can also give the client a check, its own parser (Client::fetch_checked). Then an answer the parser refuses, such as an error page sent as a success or a forecast whose hours are still empty, is never saved, so it can’t take a good copy’s place. Online, a good copy that has gone stale comes back instead, with the reason; with no copy, the fetch fails. A saved copy the check refuses counts as missing online, and is the error offline.

How long a copy stays fresh

Each source has a time to live (TTL): how many seconds a saved copy counts as fresh. A copy exactly one TTL old is no longer fresh: online it is fetched again, offline it comes back marked Stale. A copy dated later than the time asked (saved while the clock ran fast) is never fresh. The caller passes the time, in seconds since 1 January 1970 (Unix time); the crate reads no clock, so the tests can use small numbers.

Here is a worked example with a one-hour TTL (3,600 s), from the tests. Each scenario starts with an empty cache and fills it at 1,000 s:

Fetching over HTTP

The HTTP client is Http. It is behind hpr-net’s http cargo feature, so a program that depends on hpr-net alone and needs only the cache and saved responses builds no network code. The hpr library’s net feature turns HTTP on: with hpr = { ..., features = ["net"] } in Cargo.toml it is hpr::hpr_net::Http. Its API page has an example, compiled in CI, that fetches into the platform’s cache folder. Building with HTTP needs a C compiler, since the encryption library compiles some C and assembly.

A failed fetch (an error status, a timeout, a refused or dropped connection, a too-long answer) is handled as above: online, the client falls back to an old copy marked stale.

The time and size limits, and whether to use the environment’s proxy, are fields of HttpConfig. Start from HttpConfig::default(), change a field (for example config.timeout = Duration::from_secs(300)), and pass it to Http::with_config(config).

Because the computer’s certificate store is not read, a network that inspects encrypted traffic with its own certificate (common on company and school networks) makes every HTTPS fetch fail, and there is no setting yet to add a certificate.

Where the cache lives

The cache folder is whatever you pass. Cache::platform_dir gives the usual place for caches on each system, and the folder is created on the first save:

Setting the environment variable HPR_CACHE_DIR puts the cache in that folder instead, on every system; a relative path there is taken from the current folder. The folders come from environment variables alone: on Windows LOCALAPPDATA is read rather than asking the system, which gives the same folder unless the variable was changed. If HPR_CACHE_DIR is not set and HOME (macOS, Linux) or LOCALAPPDATA (Windows) is unset or empty, Cache::platform_dir returns nothing and you must name a folder.

Setting HPR_OFFLINE to anything but empty or 0 makes every hpr command answer from the cache alone, as each command’s --offline does: for a script, or a field laptop, that must not reach the network.

How it is checked

The tests in crates/hpr-net/tests/offline.rs replay a hand-written sample from a folder instead of using the network:

• The offline test gives the client a transport that fails the test if it is ever called, then asks for a URL before and after it is cached, fresh and thirty days stale.
• Offline, a corrupt entry is an error; online, it is fetched again and overwritten.
• The checked fetch is tested through Open-Meteo’s source (Launch-day weather): an answer with an hour’s values missing is not saved, and an earlier good copy comes back and stays saved. The soundings’ source is tested the same way (Weather-balloon soundings), and NOAA’s forecasts refuse, and don’t save, a file for another run or hour than the one asked for (NOAA forecasts: GFS and RAP). The elevation source refuses an answer with no heights, and a lookup made once works offline (A launch site’s elevation). The motor-stock source refuses, and doesn’t save, a motor’s page that holds another motor, and the ThrustCurve.org source a download of another motor’s file (Motor stock and prices).

The cache’s own tests check that a saved body reads back, and that another URL’s entry in the same file reads as a miss. They also check that the hash that names each file (FNV-1a, a standard 64-bit hash, so names stay the same on every platform) matches its published test values. The cache folder rules are checked for each system with a made-up environment, HPR_CACHE_DIR included; the XDG_CACHE_HOME case runs only on Unix test machines. The API reference for hpr_net has a runnable example.

The HTTP tests in crates/hpr-net/tests/http.rs start a small web server inside the test, on the computer’s own loopback address (127.0.0.1), which serves the same sample. Nothing leaves the machine, and the tests turn off any proxy the environment names. They check that:

• a fetch saves the sample in the cache, a second fetch inside the time to live sends no request, and offline mode answers from the cache without a request while the server is still running;
• a compressed answer arrives unpacked, and the request names hpr-sim and asks for gzip;
• a redirect is followed and saved under the address asked for;
• a 404 and a 304 are failed fetches and save nothing;
• an answer one byte over the limit is refused and one at the limit passes, and a megabyte of zeros that compresses to under 64 KiB is refused at a 64 KiB limit;
• a server that never answers is cut off at a 0.3 s limit (the test requires under 5 s; the server stalls for 10 s), and a timeout of Duration::MAX (the usual way to say “no limit”), which hpr treats as 30 days, does not crash;
• when the server hangs up without answering, an old copy comes back marked stale, with the reason.

What it leaves out

• No retries: a failed fetch is tried once. Plain, unencrypted http:// addresses are accepted; nothing forces HTTPS. The 60 s time limit covers the whole transfer, so a large answer on a slow connection can time out; raise it in HttpConfig.
• No test makes an encrypted (HTTPS) connection: the test server speaks plain HTTP, since a local encrypted server would need certificates made for the test. One manual run on a Mac fetched a real Open-Meteo forecast over HTTPS and refused a site with an expired certificate (ADR-118: M5.1b, HTTP over the cache), but that is not repeated in CI.
• Freshness comes only from the source’s TTL: a server’s cache headers are ignored, and the cache key is the URL alone.
• If saving a fetched body fails (a full disk, a read-only folder), the fetch fails too.
• Nothing prunes old entries, and nothing locks the cache. Each file is written whole under a temporary name and then renamed, so no file is ever half written. But two programs saving the same URL at once, or a crash between the two renames, can leave a body beside another fetch’s time. Temporary files a crash leaves behind are not cleaned up.