Skip to main content

Module generic

Module generic 

Source
Expand description

A table of samples from a logger no other reader knows, read by what its header states: one header row naming the columns, the units in the names, then one row a sample. Or the same table read by a column mapping the caller gives, for a file whose header states too little or that has none.

Teams’ own flight computers, data-logging scripts and spreadsheet exports write such tables, each in its own words, such as:

Time (s),Acc (g),Alt AGL (ft),Temp (F)
time__us,baro__kpa,temp_c,acc_y__m_s2,gyro_y__rad_s
Time [s],Battery [V],Altitude [m],Accel X [m/s^2]

No maker documents these layouts: the reader takes what the header says and no more. It reads a column in a unit only where the header states the unit, and keeps every other column as the file writes it (Quantity::Unrecognized), with a note saying why. A stated unit is the file’s own statement, the first of the two checks ADR-258 asks of a unit; the second is that the values bear it out (The values’ check, below), and a column whose values contradict its unit is kept as written too. Nothing is guessed from a column’s values alone.

Recognition. Tried after every other layout, so it never takes a file another reader knows. Within the file’s first 64 lines, past any blank lines, lines starting # and a sep= line naming the separator, the first line is the header. Its separator is the one of ,, ;, a tab and | that splits it into the most cells, at least two, unless a sep= line named it; no cell of it may be a number. One column must be the clock: a name with a time word in it (time, t, ts, timestamp, uptime) and a unit of time stated. The next line that isn’t blank must hold a number in the clock’s cell.

Units in a name. The unit is read from brackets at the name’s end, Alt AGL (ft) or Altitude [m]: their whole content, spaces taken out, as [deg C], or else its first word, the rest a qualifier, as (m ASL) or (° dec). With no brackets, it is read from the name’s last one to four words joined by underscores, as Time_s, temp_c, acc_y__m_s2 or pitch_rate_deg_per_s; a single-letter unit counts there only if it is s, m, g, c, f or v, so that a name such as flag_a isn’t read as amperes. Units are compared without regard to case. The units known, by what they measure (unit_spellings lists every spelling):

MeasuresUnits
times, ms, us (µs), and their words
lengthm, ft (feet), km
speedm/s (mps, m_s, meters_per_s), ft/s (fps, ft_per_s), km/h, mph, kn
accelerationm/s2 (m/s², m/s^2, m_s2, mps2), ft/s2, g (gs), mg (thousandths of a g)
angledeg (°), rad
rate of turndeg/s (dps, deg_per_s), rad/s (rad_per_s)
temperaturedegC (°C, C, celsius), degF (°F, F, fahrenheit), K
voltageV, mV
currentA (in brackets only), mA (milliamps)
pressurePa, hPa, mbar, kPa, bar, atm, psi, inHg

g is standard gravity, 9.806 65 m/s² (CGPM 1901); mg a thousandth of it, as accelerometer logs write it (an accelerometer log never holds milligrams); inHg 3,386.389 Pa and psi 6,894.757 Pa (NIST SP 811, 2008, appendix B.8); atm 101,325 Pa; kn 1,852 m an hour; mph 0.447 04 m/s.

What a column is. The unit says what the column measures. The name’s words say what it may be: an altitude (alt…, …alt, altitude, height, elev, agl, msl, asl), a pressure (press…, baro…, pression), a temperature (temp…), a voltage (volt…, batt…, vbat…), a current (curr…, amp…), an acceleration (acc…, …acc, …accel…), a rate of turn (gyro…, rate), a speed (vel…, speed…, vx, vy, vz, v), an angle (lat, lon, lng, long, heading, angle, tilt, roll, pitch, yaw, course, bearing, cog) or a time (time…, t, ts, uptime). A column is read in its unit when the name’s words allow what the unit measures, or name nothing; it is kept as written when they disagree, as vz_filtered_ms (a speed’s name, a time’s unit), or say a rate of turn over an angle’s unit, as Roll rate (deg); when the name says an estimate’s error or a difference (accuracy, error, err, std, stdev, stddev, sigma, var, variance, uncertainty, diff, differential, dynamic, dyn, pitot, delta), as Alt error (m) or a pitot tube’s Diff press (Pa); when it has a length or a time and the name says no altitude or time, as z (m), which may be a height or a distance; and when its unit may stand for one the reader doesn’t know (g, gs and mg, also grams and gauss; c, f, v, k and a) and the name doesn’t say what it measures, as Mag X (G) or pyro_c. Then:

The unit measuresThe channel’s quantity
a lengthQuantity::GnssAltitude if the name says gps or gnss and not agl; Quantity::AltitudeAboveGround or Quantity::AltitudeMsl if it says baro and agl, or baro and msl or asl; Quantity::InertialAltitude if it says inertial; otherwise Quantity::Altitude, an altitude whose sensor or zero the file doesn’t state
a timeQuantity::Duration
an angleQuantity::Latitude or Quantity::Longitude if the name says so, otherwise Quantity::Angle
anything elseits own quantity: Quantity::Speed, Quantity::Acceleration, Quantity::AngularRate, Quantity::Temperature, Quantity::Voltage, Quantity::Current, Quantity::Pressure

A column with no unit is kept as written. A column with no number in it, such as a flight phase written as words, is no channel: if its name says state, phase, status, event or mode, each change of its word is an Event at the row’s time, up to MAX_EVENTS_PER_COLUMN a column, the rest counted in a note; otherwise it is left out, with a note.

The values’ check. A column read in a unit is kept as written instead if any of its values, converted, lies outside what that quantity can be on a flight logger: a pressure below 0 or above 110 kPa, a little over the highest air pressure measured at sea level, 108.38 kPa (Agata, Siberia, 1968; WMO’s archive of weather extremes); a temperature below −100 °C or above 150 °C, past −40 °C to 125 °C, the automotive grade 1 range (AEC-Q100); a length beyond 150 km; a speed beyond 3,000 m/s, more than twice Mach 3.5, the simulator’s limit; an acceleration beyond 1,000 g; a rate of turn beyond 50,000 °/s; a voltage or a current beyond 100; a latitude beyond 90° or a longitude beyond 180°. Angles but those and times aren’t checked. The clock’s unit is checked by its step: the median step forward, converted, must lie from 0.1 ms to 60 s, from 10,000 samples a second to one a minute; a clock outside it, or that never steps forward, as a file of one row, is refused. Milliseconds headed as seconds pass when the logger writes more than about 17 rows a second, a step of 60 ms or less, and then the flight’s times come out a thousand times too long; so do microseconds headed as milliseconds. The check can only keep a column as written; it never changes a unit.

Rows. Lines end in CR, LF or both. A cell is trimmed of spaces, and of double quotes around it (a quoted cell may hold the separator; "" in it is one quote). The file’s decimal mark is settled once: a point, unless the separator isn’t a comma, no number below the header is written with a point, and one is written with a comma before other than three digits, as 0,05, which no thousands separator is. A cell in the other mark, such as 1,250 in a file of points or in one whose commas all stand before three digits, isn’t a number: a gap, counted in a note, so a thousands separator is never read as a decimal mark. A row must have as many cells as the header names, and a number in the clock’s cell: a row that doesn’t is refused on its line. Blank lines and lines starting # are skipped. The clock may step back, as a logger’s clock does when it restarts; a note counts the steps back, and every channel keeps the rows in the file’s order.

The record. Every column but the clock is a channel on the clock, in the file’s order; the clock is the logger’s own, its zero unstated, so times are kept as the file writes them, converted to seconds. The record has no primary track (FlightLog::time_s and FlightLog::altitude_m are empty): which column is a barometric altitude above the pad isn’t stated, so the readings that need one wait for M7.2 to take them from the channels. A mapping can state it (Mapping::primary): that column, an Quantity::AltitudeAboveGround, becomes the primary track on the clock, and stays a channel too. Its clock must go forward, as the track’s must; a mapping is refused if it doesn’t, or if the column holds no altitude it can read.

A column mapping. mapping returns what the reader takes from the header, as a Mapping a person or a program can change; read_with reads a file by a mapping, with the same checks. A mapping can name the columns of a file with no header row, state a unit the header leaves out, or keep a column as written; a unit it states must measure what its quantity does, or the mapping is refused (LogError::Mapping).

A mapping can also say what a header can’t, about how the logger writes its rows:

  • A clock that turns over (Mapping::clock_wrap): the count a logger’s counter runs over at, in the clock’s unit, such as 2³² for a 32-bit count of microseconds. Each value is taken modulo it, as a logger may write it past the turn (one team’s board prints its count 9 × 2³² high in the upper half of each turn, for a reason its files don’t state), and each step back of more than half a turn is a turn: from there on the turn is added, so the clock runs on. A restart that steps back so far is read as a turn too. A note counts the values written past the turn and the turns.
  • Rows written as lists (RowForm::List): [1, 2.5, True], as a program printing its lists writes them. The brackets are dropped; a row without them is refused on its line.
  • Stamped rows (RowForm::Stamped): a date and time, then : and the cells, as 2022-10-13 14:05:09.125: 1005.71 or 2022-10-13 14:05:09.125: (0.1, -0.2, 9.8). The stamp is column 0, the clock, in seconds from 1970-01-01 00:00 in the stamp’s own zone, which such a file doesn’t state: only the differences of its times are durations. The cells are split at the separator, a tuple’s parentheses dropped. A line with no stamp is a mark, such as a payload’s === MOTION DETECTED ===: an event named by its words, without the =, - and * around them, at the next row’s time, up to MAX_MARKS, the rest counted in a note. A line shaped as a stamp, dddd-dd-dd dd:dd:dd, with no valid date and time in it is refused, a leap second’s 23:59:60 among them. As an f64, a time since 1970 resolves to about 0.25 µs, so a finer fraction is rounded.
  • Columns left out (ColumnMapping::skip): cells beside a table that aren’t part of it. They count in a row’s width, but make no channel; a note says how many are left out.
use hpr_flightdata::generic;
use hpr_flightdata::log::Quantity;

// An invented table: seconds, an altitude above ground in feet, a speed with no unit.
let text = "Time (s),Alt AGL (ft),Velocity\n0.0,0,0\n0.5,100,190\n1.0,180,140\n";
let log = generic::read(text)?;
let altitude = &log.channels[0];
assert_eq!(altitude.quantity, Quantity::Altitude);
assert_eq!(altitude.values[1], 100.0 * 0.3048);
assert_eq!(log.channels[1].quantity, Quantity::Unrecognized); // no unit stated

// The same file, its velocity in feet a second by the caller's word.
let mut mapping = generic::mapping(text)?;
mapping.columns[2].quantity = Some(Quantity::Speed);
mapping.columns[2].unit = Some("ft/s".to_owned());
let log = generic::read_with(text, &mapping)?;
assert_eq!(log.channels[1].values[1], 190.0 * 0.3048);

A mapping is JSON a person can write. Every column has an entry in columns, a column left out too; stamped rows need the stamp’s entry first, the clock, in s and with no turn; and a turn is in the clock’s own unit, 4294967296 (2³²) for a 32-bit count of microseconds:

use hpr_flightdata::generic::{self, Mapping};

// An invented payload's text: a stamp and a pressure in hectopascals, a note beside it, and a
// line marking motion, an event at the next row's time.
let text = "2024-06-01 14:05:09.0: 1005.5, ok\n\
            === MOTION DETECTED ===\n\
            2024-06-01 14:05:09.5: 1005.25, ok\n";
let mapping: Mapping = serde_json::from_str(
    r#"{"header_line": null, "separator": ",", "decimal_comma": false, "clock": 0,
        "rows": "stamped",
        "columns": [
            {"name": "stamp", "quantity": null, "unit": "s"},
            {"name": "pressure", "quantity": "pressure", "unit": "hPa"},
            {"name": "note", "quantity": null, "unit": null, "skip": true}]}"#,
)?;
let log = generic::read_with(text, &mapping)?;
assert_eq!(log.channels.len(), 1); // the note is left out
assert_eq!(log.channels[0].values, [100_550.0, 100_525.0]);
assert_eq!(log.channels[0].time_s[1] - log.channels[0].time_s[0], 0.5);
assert_eq!(log.events[0].name, "MOTION DETECTED");

// An invented 32-bit count of microseconds, in rows written as lists, turning over once.
let text = "[4294967000, 10.0]\n[4294967200, 10.5]\n[104, 11.0]\n";
let mapping: Mapping = serde_json::from_str(
    r#"{"header_line": null, "separator": ",", "decimal_comma": false, "clock": 0,
        "clock_wrap": 4294967296, "rows": "list",
        "columns": [
            {"name": "clock", "quantity": null, "unit": "us"},
            {"name": "altitude", "quantity": "altitude_above_ground", "unit": "m"}]}"#,
)?;
let log = generic::read_with(text, &mapping)?;
let t = &log.channels[0].time_s;
assert!((t[2] - t[1] - 200e-6).abs() < 1e-9); // 104 µs is 200 µs past 4294967200 µs

Structs§

ColumnMapping
What one column holds.
Mapping
How a file’s columns are read: the header’s line, the separator, and what each column holds. mapping returns the one the header states; a caller may change it and read the file by it with read_with.

Enums§

RowForm
How a table’s rows are written (module docs, A column mapping).

Constants§

MAX_COLUMNS
The most columns a header or a mapping may name: the widest table among the project’s sample logs names 99.
MAX_EVENTS_PER_COLUMN
The most events one column of words gives; past it, its changes are counted in a note, so a hostile file whose word changes every row can’t fill memory with events.
MAX_MARKS
The most marks a file of stamped rows gives as events (RowForm::Stamped); past it, they are counted in a note.

Functions§

header_reading
mapping, with a note on each column the header leaves kept as written, saying why, as read gives them.
mapping
What the reader takes from the file’s header: its line, the separator and each column’s quantity and unit, as read would read them.
read
Reads a table by what its header states (module docs).
read_with
Reads a table by the caller’s column mapping, with the checks read makes.
recognizes
Whether the text is in this layout, judged from its first lines, never the file’s name.
unit_spellings
Every spelling of a unit the reader knows, in lower case with spaces taken out (module docs, Units in a name): what a ColumnMapping::unit may be.