Expand description
A table of samples from a logger no other reader knows, read by what its header states: one header row naming the columns, the units in the names, then one row a sample. Or the same table read by a column mapping the caller gives, for a file whose header states too little or that has none.
Teams’ own flight computers, data-logging scripts and spreadsheet exports write such tables, each in its own words, such as:
Time (s),Acc (g),Alt AGL (ft),Temp (F)
time__us,baro__kpa,temp_c,acc_y__m_s2,gyro_y__rad_s
Time [s],Battery [V],Altitude [m],Accel X [m/s^2]No maker documents these layouts: the reader takes what the header says and no more. It reads
a column in a unit only where the header states the unit, and keeps every other column as the
file writes it (Quantity::Unrecognized), with a note saying why. A stated unit is the
file’s own statement, the first of the two checks ADR-258 asks of a unit; the
second is that the values bear it out (The values’ check, below), and a column whose values
contradict its unit is kept as written too. Nothing is guessed from a column’s values alone.
Recognition. Tried after every other layout, so it never takes a file another reader
knows. Within the file’s first 64 lines, past any blank lines, lines starting # and a
sep= line naming the separator, the first line is the header. Its separator is the one of
,, ;, a tab and | that splits it into the most cells, at least two, unless a sep= line
named it; no cell of it may be a number. One column must be the clock: a name with a time
word in it (time, t, ts, timestamp, uptime) and a unit of time stated. The next line
that isn’t blank must hold a number in the clock’s cell.
Units in a name. The unit is read from brackets at the name’s end, Alt AGL (ft) or
Altitude [m]: their whole content, spaces taken out, as [deg C], or else its first word,
the rest a qualifier, as (m ASL) or (° dec). With no brackets, it is read from the name’s
last one to four words joined by underscores, as Time_s, temp_c, acc_y__m_s2 or
pitch_rate_deg_per_s; a single-letter unit counts there only if it is s, m, g,
c, f or v, so that a name such as flag_a isn’t read as amperes. Units are compared
without regard to case. The units known, by what they measure (unit_spellings lists
every spelling):
| Measures | Units |
|---|---|
| time | s, ms, us (µs), and their words |
| length | m, ft (feet), km |
| speed | m/s (mps, m_s, meters_per_s), ft/s (fps, ft_per_s), km/h, mph, kn |
| acceleration | m/s2 (m/s², m/s^2, m_s2, mps2), ft/s2, g (gs), mg (thousandths of a g) |
| angle | deg (°), rad |
| rate of turn | deg/s (dps, deg_per_s), rad/s (rad_per_s) |
| temperature | degC (°C, C, celsius), degF (°F, F, fahrenheit), K |
| voltage | V, mV |
| current | A (in brackets only), mA (milliamps) |
| pressure | Pa, hPa, mbar, kPa, bar, atm, psi, inHg |
g is standard gravity, 9.806 65 m/s² (CGPM 1901); mg a thousandth of it, as accelerometer
logs write it (an accelerometer log never holds milligrams); inHg 3,386.389 Pa and psi
6,894.757 Pa (NIST SP 811, 2008, appendix B.8); atm 101,325 Pa; kn 1,852 m an hour;
mph 0.447 04 m/s.
What a column is. The unit says what the column measures. The name’s words say what it
may be: an altitude (alt…, …alt, altitude, height, elev, agl, msl, asl), a
pressure (press…, baro…, pression), a temperature (temp…), a voltage (volt…,
batt…, vbat…), a current (curr…, amp…), an acceleration (acc…, …acc, …accel…),
a rate of turn (gyro…, rate), a speed (vel…, speed…, vx, vy, vz, v), an angle
(lat, lon, lng, long, heading, angle, tilt, roll, pitch, yaw, course,
bearing, cog) or a time (time…, t, ts, uptime). A column is read in its unit when
the name’s words allow what the unit measures, or name nothing; it is kept as written when
they disagree, as vz_filtered_ms (a speed’s name, a time’s unit), or say a rate of turn over
an angle’s unit, as Roll rate (deg); when the name says an estimate’s error or a difference
(accuracy, error, err, std, stdev, stddev, sigma, var, variance,
uncertainty, diff, differential, dynamic, dyn, pitot, delta), as Alt error (m)
or a pitot tube’s Diff press (Pa); when it has a length or a time and the name says no
altitude or time, as z (m), which may be a height or a distance; and when its unit may stand
for one the reader doesn’t know (g, gs and mg, also grams and gauss; c, f, v, k
and a) and the name doesn’t say what it measures, as Mag X (G) or pyro_c. Then:
| The unit measures | The channel’s quantity |
|---|---|
| a length | Quantity::GnssAltitude if the name says gps or gnss and not agl; Quantity::AltitudeAboveGround or Quantity::AltitudeMsl if it says baro and agl, or baro and msl or asl; Quantity::InertialAltitude if it says inertial; otherwise Quantity::Altitude, an altitude whose sensor or zero the file doesn’t state |
| a time | Quantity::Duration |
| an angle | Quantity::Latitude or Quantity::Longitude if the name says so, otherwise Quantity::Angle |
| anything else | its own quantity: Quantity::Speed, Quantity::Acceleration, Quantity::AngularRate, Quantity::Temperature, Quantity::Voltage, Quantity::Current, Quantity::Pressure |
A column with no unit is kept as written. A column with no number in it, such as a flight
phase written as words, is no channel: if its name says state, phase, status, event
or mode, each change of its word is an Event at the row’s time, up to
MAX_EVENTS_PER_COLUMN a column, the rest counted in a note; otherwise it is left out, with
a note.
The values’ check. A column read in a unit is kept as written instead if any of its values, converted, lies outside what that quantity can be on a flight logger: a pressure below 0 or above 110 kPa, a little over the highest air pressure measured at sea level, 108.38 kPa (Agata, Siberia, 1968; WMO’s archive of weather extremes); a temperature below −100 °C or above 150 °C, past −40 °C to 125 °C, the automotive grade 1 range (AEC-Q100); a length beyond 150 km; a speed beyond 3,000 m/s, more than twice Mach 3.5, the simulator’s limit; an acceleration beyond 1,000 g; a rate of turn beyond 50,000 °/s; a voltage or a current beyond 100; a latitude beyond 90° or a longitude beyond 180°. Angles but those and times aren’t checked. The clock’s unit is checked by its step: the median step forward, converted, must lie from 0.1 ms to 60 s, from 10,000 samples a second to one a minute; a clock outside it, or that never steps forward, as a file of one row, is refused. Milliseconds headed as seconds pass when the logger writes more than about 17 rows a second, a step of 60 ms or less, and then the flight’s times come out a thousand times too long; so do microseconds headed as milliseconds. The check can only keep a column as written; it never changes a unit.
Rows. Lines end in CR, LF or both. A cell is trimmed of spaces, and of double quotes
around it (a quoted cell may hold the separator; "" in it is one quote). The file’s decimal
mark is settled once: a point, unless the separator isn’t a comma, no number below the header
is written with a point, and one is written with a comma before other than three digits, as
0,05, which no thousands separator is. A cell in the other mark, such as 1,250 in a file
of points or in one whose commas all stand before three digits, isn’t a number: a gap,
counted in a note, so a thousands separator is never read as a decimal mark. A row must have as many cells as the header names, and a number in the
clock’s cell: a row that doesn’t is refused on its line. Blank lines and lines starting #
are skipped. The clock may step back, as a logger’s clock does when it restarts; a note counts
the steps back, and every channel keeps the rows in the file’s order.
The record. Every column but the clock is a channel on the clock, in the file’s order; the
clock is the logger’s own, its zero unstated, so times are kept as the file writes them,
converted to seconds. The record has no primary track (FlightLog::time_s and
FlightLog::altitude_m are empty): which column is a barometric altitude above the pad isn’t
stated, so the readings that need one wait for M7.2 to take them from the channels.
A mapping can state it (Mapping::primary): that column, an
Quantity::AltitudeAboveGround, becomes the primary track on the clock, and stays a channel
too. Its clock must go forward, as the track’s must; a mapping is refused if it doesn’t, or if
the column holds no altitude it can read.
A column mapping. mapping returns what the reader takes from the header, as a
Mapping a person or a program can change; read_with reads a file by a mapping, with the
same checks. A mapping can name the columns of a file with no header row, state a unit the
header leaves out, or keep a column as written; a unit it states must measure what its
quantity does, or the mapping is refused (LogError::Mapping).
A mapping can also say what a header can’t, about how the logger writes its rows:
- A clock that turns over (
Mapping::clock_wrap): the count a logger’s counter runs over at, in the clock’s unit, such as 2³² for a 32-bit count of microseconds. Each value is taken modulo it, as a logger may write it past the turn (one team’s board prints its count 9 × 2³² high in the upper half of each turn, for a reason its files don’t state), and each step back of more than half a turn is a turn: from there on the turn is added, so the clock runs on. A restart that steps back so far is read as a turn too. A note counts the values written past the turn and the turns. - Rows written as lists (
RowForm::List):[1, 2.5, True], as a program printing its lists writes them. The brackets are dropped; a row without them is refused on its line. - Stamped rows (
RowForm::Stamped): a date and time, then:and the cells, as2022-10-13 14:05:09.125: 1005.71or2022-10-13 14:05:09.125: (0.1, -0.2, 9.8). The stamp is column 0, the clock, in seconds from 1970-01-01 00:00 in the stamp’s own zone, which such a file doesn’t state: only the differences of its times are durations. The cells are split at the separator, a tuple’s parentheses dropped. A line with no stamp is a mark, such as a payload’s=== MOTION DETECTED ===: an event named by its words, without the=,-and*around them, at the next row’s time, up toMAX_MARKS, the rest counted in a note. A line shaped as a stamp,dddd-dd-dd dd:dd:dd, with no valid date and time in it is refused, a leap second’s23:59:60among them. As anf64, a time since 1970 resolves to about 0.25 µs, so a finer fraction is rounded. - Columns left out (
ColumnMapping::skip): cells beside a table that aren’t part of it. They count in a row’s width, but make no channel; a note says how many are left out.
use hpr_flightdata::generic;
use hpr_flightdata::log::Quantity;
// An invented table: seconds, an altitude above ground in feet, a speed with no unit.
let text = "Time (s),Alt AGL (ft),Velocity\n0.0,0,0\n0.5,100,190\n1.0,180,140\n";
let log = generic::read(text)?;
let altitude = &log.channels[0];
assert_eq!(altitude.quantity, Quantity::Altitude);
assert_eq!(altitude.values[1], 100.0 * 0.3048);
assert_eq!(log.channels[1].quantity, Quantity::Unrecognized); // no unit stated
// The same file, its velocity in feet a second by the caller's word.
let mut mapping = generic::mapping(text)?;
mapping.columns[2].quantity = Some(Quantity::Speed);
mapping.columns[2].unit = Some("ft/s".to_owned());
let log = generic::read_with(text, &mapping)?;
assert_eq!(log.channels[1].values[1], 190.0 * 0.3048);A mapping is JSON a person can write. Every column has an entry in columns, a column left
out too; stamped rows need the stamp’s entry first, the clock, in s and with no turn; and a
turn is in the clock’s own unit, 4294967296 (2³²) for a 32-bit count of microseconds:
use hpr_flightdata::generic::{self, Mapping};
// An invented payload's text: a stamp and a pressure in hectopascals, a note beside it, and a
// line marking motion, an event at the next row's time.
let text = "2024-06-01 14:05:09.0: 1005.5, ok\n\
=== MOTION DETECTED ===\n\
2024-06-01 14:05:09.5: 1005.25, ok\n";
let mapping: Mapping = serde_json::from_str(
r#"{"header_line": null, "separator": ",", "decimal_comma": false, "clock": 0,
"rows": "stamped",
"columns": [
{"name": "stamp", "quantity": null, "unit": "s"},
{"name": "pressure", "quantity": "pressure", "unit": "hPa"},
{"name": "note", "quantity": null, "unit": null, "skip": true}]}"#,
)?;
let log = generic::read_with(text, &mapping)?;
assert_eq!(log.channels.len(), 1); // the note is left out
assert_eq!(log.channels[0].values, [100_550.0, 100_525.0]);
assert_eq!(log.channels[0].time_s[1] - log.channels[0].time_s[0], 0.5);
assert_eq!(log.events[0].name, "MOTION DETECTED");
// An invented 32-bit count of microseconds, in rows written as lists, turning over once.
let text = "[4294967000, 10.0]\n[4294967200, 10.5]\n[104, 11.0]\n";
let mapping: Mapping = serde_json::from_str(
r#"{"header_line": null, "separator": ",", "decimal_comma": false, "clock": 0,
"clock_wrap": 4294967296, "rows": "list",
"columns": [
{"name": "clock", "quantity": null, "unit": "us"},
{"name": "altitude", "quantity": "altitude_above_ground", "unit": "m"}]}"#,
)?;
let log = generic::read_with(text, &mapping)?;
let t = &log.channels[0].time_s;
assert!((t[2] - t[1] - 200e-6).abs() < 1e-9); // 104 µs is 200 µs past 4294967200 µsStructs§
- Column
Mapping - What one column holds.
- Mapping
- How a file’s columns are read: the header’s line, the separator, and what each column holds.
mappingreturns the one the header states; a caller may change it and read the file by it withread_with.
Enums§
- RowForm
- How a table’s rows are written (module docs, A column mapping).
Constants§
- MAX_
COLUMNS - The most columns a header or a mapping may name: the widest table among the project’s sample logs names 99.
- MAX_
EVENTS_ PER_ COLUMN - The most events one column of words gives; past it, its changes are counted in a note, so a hostile file whose word changes every row can’t fill memory with events.
- MAX_
MARKS - The most marks a file of stamped rows gives as events (
RowForm::Stamped); past it, they are counted in a note.
Functions§
- header_
reading mapping, with a note on each column the header leaves kept as written, saying why, asreadgives them.- mapping
- What the reader takes from the file’s header: its line, the separator and each column’s
quantity and unit, as
readwould read them. - read
- Reads a table by what its header states (module docs).
- read_
with - Reads a table by the caller’s column mapping, with the checks
readmakes. - recognizes
- Whether the text is in this layout, judged from its first lines, never the file’s name.
- unit_
spellings - Every spelling of a unit the reader knows, in lower case with spaces taken out (module docs,
Units in a name): what a
ColumnMapping::unitmay be.