Skip to main content

hpr_flightdata/
raven_fipa.rs

1//! The Featherweight Raven's raw downloads, `.FIPa`: the altimeter's flash memory as the
2//! Featherweight Interface Program (FIP) saves it, before any export.
3//!
4//! The file opens with text lines, each starting `#`, up to a `#den` line; then the memory, in
5//! frames of a `:`, 256 data bytes and a checksum byte, the sum of the 256 modulo 256; then
6//! `#DONE` and a line end:
7//!
8//! ```text
9//! #Component=Raven
10//! #DateTime=1/2/2020 3:04:05 PM         (invented values throughout)
11//! #FlightNum=7
12//! #FlightState=#7  O  5  1000  …        (tab-separated: the settings block's words)
13//! #BinaryFormat 1.0
14//! #R 3.0                                (the firmware's version)
15//! #Num 25
16//! #den 20                               (a tick is Num/den ms: 1.25 ms)
17//! :<256 bytes><checksum>:<256 bytes><checksum>…#DONE
18//! ```
19//!
20//! **Blocks.** Each frame holds four blocks of 32 big-endian 16-bit words. Unused memory reads
21//! `0xFFFF`; the flight ends at the first block of nothing else, and every block after it must be
22//! unused too. Block 5 is the settings block, the words `#FlightState` lists after its first two
23//! fields. Each other block covers 40 ticks, 0.05 s: blocks 0 to 4 are the 0.25 s before the Raven
24//! detected the launch, and the blocks after the settings block the flight, one tick later in each
25//! block's pattern. A block is ten triplets `(A, B, S)`, then two words, at ticks `4k` to `4k + 3`
26//! within it (`4k + 1` to `4k + 4` after the settings block):
27//!
28//! | Bits | Sample | Tick |
29//! |---|---|---|
30//! | `A >> 5`, 11 bits | the axial accelerometer | `4k` |
31//! | `(A & 0x1f) << 5 \| B >> 11`, 10 bits | the lateral accelerometer | `4k + 1` |
32//! | `B & 0x7ff`, 11 bits | the axial accelerometer | `4k + 2` |
33//! | `S` | the `k`th slow channel | `4k + 3` |
34//!
35//! so the axial accelerometer runs at 400 Hz, the lateral at 200 Hz and the slow channels at 20
36//! Hz, as the blocks lay them out and as the FIP's export times them; Featherweight's manual says
37//! the Raven records the data its deployment logic uses at 20 Hz (*Raven User's Manual, updated
38//! for the Raven4*, 2020-08-15). Time zero is the first sample before the launch, as in the FIP's
39//! export.
40//!
41//! **Values.** Each into the FIP's export's column, by the export's own arithmetic, then SI as
42//! [`crate::raven`] reads that column:
43//!
44//! | Column | From the word | Raven's unit |
45//! |---|---|---|
46//! | `Axial Accel (Gs)` | `(trunc(X1 (a − Z1) / 1000) + F3) / 100` | g |
47//! | `Lateral Accel (Gs)` | `trunc(X2 (l − Z2) / 1000) / 100` | g |
48//! | `Current Draw (A)` | slow 0, signed, / 1000 | A |
49//! | `Volts Pyro Apogee (V)`, `… Main`, `… 3rd`, `… 4th` | slow 1 to 4, signed, / 1000 | V |
50//! | `Volts Battery (V)` | slow 6, signed, / 1000 | V |
51//! | `Temperature (F)` | slow 7, signed (no sample file holds a negative one to show it), / 100 | °F |
52//! | `Baro (Atm)` | slow 8, unsigned, / 25,000 | atm |
53//! | `Velocity (Accel-Ft/Sec)` | slow 9, signed, / 10 | ft/s |
54//!
55//! `trunc` rounds toward zero, and `X1 / 1000` is worked in double precision, as the FIP does:
56//! exact integers differ from its export by a hundredth of a g where the product is a whole number
57//! that double precision leaves just under it, as on 11 of the sample files ([ADR-259][adr-259]).
58//! `X1`, `Z1`, `F3`, `X2` and `Z2` are the settings block's: word 0 the axial offset `F3` (read
59//! signed, though no sample file's is negative), words 23–24 the signed 32-bit axial gain `X1`, word 25 the axial zero `Z1`,
60//! words 26–27 the lateral gain `X2`, word 28 the lateral zero `Z2`. A header whose `#Axis` is 1
61//! (a single-axis accelerometer; 2 otherwise) keeps the lateral bits as written, as `lateral
62//! counts`, since the Raven calibrates no lateral axis. Slow channel 5, which the FIP
63//! doesn't export, and the two words after each block's triplets are kept as written, numbered by
64//! their place, as [`Quantity::Unrecognized`]. The export stops `Velocity (Accel-Ft/Sec)` at its
65//! first sample at or below zero after it has gone positive, its apogee; the record keeps every
66//! sample the Raven wrote.
67//!
68//! **The record.** The primary track is empty, with a note: the Raven records a pressure, not an
69//! altitude; the export's `[Altitude (Baro-Ft-AGL)]` and `[Altitude (Baro-Ft-ASL)]` are the FIP's
70//! conversions, and working an altitude out of the pressure is a reading, not this reader's to make
71//! (milestone [M7.2], the readings, takes it). The settings block's word 1, the ground's elevation in feet, which the FIP's export
72//! takes from the altitude above sea level to give it above the pad, is stated as the launch
73//! site's elevation; `#FlightNum` is the flight's number and `#R` the firmware.
74//!
75//! **Checked against the FIP.** The layout and the arithmetic were worked out from raw files and the
76//! FIP's exports of the same flights; wherever the project's sample logs hold both, `cargo xtask
77//! logs` compares every sample of every channel the export holds with the raw file's: 55 of the 57
78//! sample files have an export beside them, and all 55 match it.
79//!
80//! **Refused**, naming the byte: a header without its `#Num` and `#den` ticks, or a
81//! `#BinaryFormat` other than 1.0; a frame whose checksum fails; anything after the frames but
82//! `#DONE` and a line end; a settings block other than `#FlightState`'s words, or a
83//! `#FlightState` listing none; an `#Axis` other
84//! than 1 or 2; fewer than the five
85//! blocks before the launch and the settings block; and a used block after an unused one.
86//!
87//! [M7.2]: https://hpr.fusionspace.co/decisions-and-roadmap.html#m7-2
88//! [adr-259]: https://github.com/nrdptel/fusionspace-eridanus/blob/main/docs/decisions/0259-raw-downloads-against-their-makers-exports-first.md
89
90use hpr_core::gravity::STANDARD_GRAVITY_MPS2;
91
92use crate::error::LogError;
93use crate::log::{Channel, FlightLog, LogFormat, Quantity, Stated};
94use crate::perfectflite::FOOT_M;
95
96/// The format's name in errors.
97const FORMAT: &str = "Raven .FIPa";
98
99/// How the file opens.
100const OPENING: &[u8] = b"#Component=Raven";
101
102/// The most bytes the text header may run to.
103const MAX_HEADER: usize = 4096;
104
105/// A frame: the `:`, 256 data bytes and the checksum byte.
106const FRAME: usize = 258;
107
108/// The words in a block.
109const WORDS: usize = 32;
110
111/// The settings block's place among the blocks.
112const SETTINGS: usize = 5;
113
114/// Ticks in a block.
115const BLOCK_TICKS: u64 = 40;
116
117/// The slow channels, by their place in a block's triplets: the FIP's export's names, or `None`
118/// for the one it doesn't export.
119const SLOW: [Option<&str>; 10] = [
120    Some("Current Draw (A)"),
121    Some("Volts Pyro Apogee (V)"),
122    Some("Volts Pyro Main (V)"),
123    Some("Volts Pyro 3rd (V)"),
124    Some("Volts Pyro 4th (V)"),
125    None,
126    Some("Volts Battery (V)"),
127    Some("Temperature (F)"),
128    Some("Baro (Atm)"),
129    Some("Velocity (Accel-Ft/Sec)"),
130];
131
132/// Pascals in a standard atmosphere, exactly (ISO 2533).
133const ATM_PA: f64 = 101_325.0;
134
135/// The Raven's pressure word's count in an atmosphere.
136const BARO_COUNTS_PER_ATM: f64 = 25_000.0;
137
138/// Whether the bytes are a `.FIPa`: they open `#Component=Raven`, and a line starting `#den`,
139/// within the first [`MAX_HEADER`] bytes, is followed by a frame's `:`.
140pub(crate) fn recognizes(bytes: &[u8]) -> bool {
141    bytes.starts_with(OPENING) && header_end(bytes).is_some()
142}
143
144/// Where the text header ends: the byte after the `#den` line's end, which must be a frame's `:`.
145fn header_end(bytes: &[u8]) -> Option<usize> {
146    let head = &bytes[..bytes.len().min(MAX_HEADER)];
147    let mut start = 0;
148    while start < head.len() {
149        let end = head[start..]
150            .iter()
151            .position(|&b| b == b'\n')
152            .map(|at| start + at + 1)?;
153        if head[start..].starts_with(b"#den") {
154            return (bytes.get(end) == Some(&b':')).then_some(end);
155        }
156        if head[start] != b'#' {
157            return None;
158        }
159        start = end;
160    }
161    None
162}
163
164/// The header's lines, `#Key=Value`, `#Key<tab>Value` or `#Key Value`, as key and value.
165fn header_fields(header: &str) -> Vec<(&str, &str)> {
166    header
167        .lines()
168        .filter_map(|line| {
169            let line = line.trim_end_matches('\r').strip_prefix('#')?;
170            let split = line.find(['=', '\t', ' '])?;
171            Some((line[..split].trim(), line[split + 1..].trim()))
172        })
173        .collect()
174}
175
176/// A word read as a signed 16-bit integer.
177fn signed(word: u16) -> i64 {
178    i64::from(word as i16)
179}
180
181/// The settings block's calibration.
182struct Calibration {
183    /// The axial accelerometer's offset, hundredths of a g.
184    f3: i64,
185    /// The ground's elevation, ft above sea level.
186    f4: i64,
187    /// The axial accelerometer's gain, thousandths of a hundredth of a g a count.
188    x1: i64,
189    /// The axial accelerometer's zero, counts.
190    z1: i64,
191    /// The lateral accelerometer's gain.
192    x2: i64,
193    /// The lateral accelerometer's zero, counts.
194    z2: i64,
195}
196
197impl Calibration {
198    fn from_block(block: &[u16; WORDS]) -> Self {
199        let pair = |hi: u16, lo: u16| i64::from(((u32::from(hi) << 16) | u32::from(lo)) as i32);
200        Self {
201            f3: signed(block[0]),
202            f4: signed(block[1]),
203            x1: pair(block[23], block[24]),
204            z1: i64::from(block[25]),
205            x2: pair(block[26], block[27]),
206            z2: i64::from(block[28]),
207        }
208    }
209
210    /// The axial acceleration, g, from an 11-bit count.
211    fn axial_g(&self, count: u16) -> f64 {
212        (scaled(self.x1, i64::from(count) - self.z1) + self.f3) as f64 / 100.0
213    }
214
215    /// The lateral acceleration, g, from a 10-bit count.
216    fn lateral_g(&self, count: u16) -> f64 {
217        scaled(self.x2, i64::from(count) - self.z2) as f64 / 100.0
218    }
219}
220
221/// A count times a gain in thousandths, hundredths of a g: `trunc((gain / 1000) × counts)`, in
222/// double precision, as the FIP's export works it. Exact integer arithmetic, `trunc(gain × counts /
223/// 1000)`, differs from the export by a hundredth of a g where the product is a whole number that
224/// rounding in double precision leaves just under it: on 11 of the 55 sample files with the
225/// FIP's export beside them (ADR-259). Both operands are below 2³², so the
226/// conversions are exact.
227#[allow(
228    clippy::cast_possible_truncation,
229    reason = "operands below 2³² keep the product inside an i64"
230)]
231fn scaled(gain: i64, counts: i64) -> i64 {
232    ((gain as f64 / 1000.0) * counts as f64).trunc() as i64
233}
234
235/// One channel as it fills: its sample ticks and values in SI.
236struct Filling {
237    column: String,
238    quantity: Quantity,
239    ticks: Vec<u64>,
240    values: Vec<f64>,
241}
242
243impl Filling {
244    fn new(column: &str, quantity: Quantity) -> Self {
245        Self {
246            column: column.to_owned(),
247            quantity,
248            ticks: Vec::new(),
249            values: Vec::new(),
250        }
251    }
252
253    fn push(&mut self, tick: u64, value: f64) {
254        self.ticks.push(tick);
255        self.values.push(value);
256    }
257}
258
259/// Reads a Raven `.FIPa` file.
260pub fn read(bytes: &[u8]) -> Result<FlightLog, LogError> {
261    if !bytes.starts_with(OPENING) {
262        return Err(LogError::NotThisFormat {
263            format: FORMAT,
264            message: "the file doesn't open `#Component=Raven`".to_owned(),
265        });
266    }
267    let Some(body) = header_end(bytes) else {
268        return Err(binary(
269            0,
270            format!(
271                "the text header has no `#den` line followed by a frame within its first \
272                 {MAX_HEADER} bytes"
273            ),
274        ));
275    };
276    let header = String::from_utf8_lossy(&bytes[..body]);
277    let fields = header_fields(&header);
278    let field = |key: &str| {
279        fields
280            .iter()
281            .find(|(k, _)| k.eq_ignore_ascii_case(key))
282            .map(|(_, v)| *v)
283    };
284    if let Some(version) = field("BinaryFormat")
285        && version != "1.0"
286    {
287        return Err(binary(
288            0,
289            format!("the header's `#BinaryFormat` is {version:?}; this reader knows 1.0"),
290        ));
291    }
292    let tick_part = |key: &str| {
293        field(key)
294            .and_then(|value| value.parse::<u32>().ok())
295            .filter(|&value| value > 0)
296            .ok_or_else(|| binary(0, format!("the header has no positive whole `#{key}`")))
297    };
298    let (num, den) = (tick_part("Num")?, tick_part("den")?);
299    // A tick, s: `#Num` / `#den` milliseconds.
300    let tick_s = f64::from(num) / f64::from(den) / 1000.0;
301
302    // The frames, then `#DONE`.
303    let mut blocks: Vec<[u16; WORDS]> = Vec::new();
304    let mut at = body;
305    while bytes.get(at) == Some(&b':') {
306        let Some(frame) = bytes.get(at + 1..at + FRAME) else {
307            return Err(binary(at, "the last frame is cut short".to_owned()));
308        };
309        let (data, checksum) = frame.split_at(256);
310        let sum = data.iter().fold(0u8, |sum, &b| sum.wrapping_add(b));
311        if sum != checksum[0] {
312            return Err(binary(
313                at,
314                format!(
315                    "the frame's checksum is {:#04x}, its bytes sum to {sum:#04x}",
316                    checksum[0]
317                ),
318            ));
319        }
320        let (pairs, _) = data.as_chunks::<2>();
321        for block in pairs.as_chunks::<WORDS>().0 {
322            blocks.push(block.map(u16::from_be_bytes));
323        }
324        at += FRAME;
325    }
326    let trailer = &bytes[at..];
327    let trailer_ok = trailer
328        .strip_prefix(b"#DONE")
329        .is_some_and(|rest| matches!(rest, b"" | b"\n" | b"\r\n"));
330    if !trailer_ok {
331        return Err(binary(
332            at,
333            "the frames are followed by something other than `#DONE` and a line end".to_owned(),
334        ));
335    }
336    if blocks.len() <= SETTINGS {
337        return Err(LogError::NoData { format: FORMAT });
338    }
339
340    let settings = blocks[SETTINGS];
341    if let Some(state) = field("FlightState") {
342        let listed: Vec<&str> = state.split('\t').skip(2).collect();
343        // Each listed word as the settings block holds it, signed or not: a value outside 16 bits
344        // is no word of it.
345        #[allow(
346            clippy::cast_possible_truncation,
347            clippy::cast_sign_loss,
348            reason = "a value in i16's or u16's range keeps its 16 bits: the word, signed or not"
349        )]
350        let agrees = !listed.is_empty()
351            && listed.len() <= WORDS
352            && listed.iter().zip(settings).all(|(listed, word)| {
353                listed.trim().parse::<i64>().is_ok_and(|value| {
354                    (i64::from(i16::MIN)..=i64::from(u16::MAX)).contains(&value)
355                        && value as u16 == word
356                })
357            });
358        if !agrees {
359            return Err(binary(
360                body + FRAME * (SETTINGS / 4) + 1 + 2 * WORDS * (SETTINGS % 4),
361                "the settings block's words aren't the ones `#FlightState` lists".to_owned(),
362            ));
363        }
364    }
365    let calibration = Calibration::from_block(&settings);
366    // `#Axis` is the number of the accelerometer's axes the Raven records: 2 in all but one of the
367    // sample files, 1 in that one, whose lateral zero, 4 counts, is far from the two-axis files'
368    // (near 510 counts, the middle of a 10-bit count).
369    let two_axes = match field("Axis") {
370        None | Some("2") => true,
371        Some("1") => false,
372        Some(other) => {
373            return Err(binary(
374                0,
375                format!("the header's `#Axis` is {other:?}; this reader knows 1 and 2"),
376            ));
377        }
378    };
379
380    let mut axial = Filling::new("Axial Accel (Gs)", Quantity::Acceleration);
381    let mut lateral = if two_axes {
382        Filling::new("Lateral Accel (Gs)", Quantity::Acceleration)
383    } else {
384        Filling::new("lateral counts", Quantity::Unrecognized)
385    };
386    let mut slow: Vec<Filling> = SLOW
387        .iter()
388        .enumerate()
389        .map(|(k, name)| match name {
390            Some(name) => Filling::new(name, quantity_of(name)),
391            None => Filling::new(&format!("slow channel {k}"), Quantity::Unrecognized),
392        })
393        .collect();
394    let mut tails = [
395        Filling::new("block word 30", Quantity::Unrecognized),
396        Filling::new("block word 31", Quantity::Unrecognized),
397    ];
398    let mut data_blocks = 0u64;
399    let mut ended = None;
400    for (index, block) in blocks.iter().enumerate() {
401        if index == SETTINGS {
402            continue;
403        }
404        if block.iter().all(|&word| word == 0xffff) {
405            ended = Some(index);
406            break;
407        }
408        let shift = u64::from(index > SETTINGS);
409        let first = data_blocks * BLOCK_TICKS;
410        data_blocks += 1;
411        for (k, &[a, b, s]) in block[..30].as_chunks::<3>().0.iter().enumerate() {
412            let base = first + 4 * k as u64 + shift;
413            axial.push(base, calibration.axial_g(a >> 5) * STANDARD_GRAVITY_MPS2);
414            let count = ((a & 0x1f) << 5) | (b >> 11);
415            lateral.push(
416                base + 1,
417                if two_axes {
418                    calibration.lateral_g(count) * STANDARD_GRAVITY_MPS2
419                } else {
420                    f64::from(count)
421                },
422            );
423            axial.push(
424                base + 2,
425                calibration.axial_g(b & 0x7ff) * STANDARD_GRAVITY_MPS2,
426            );
427            slow[k].push(base + 3, slow_si(k, s));
428        }
429        let last = first + BLOCK_TICKS - 1 + shift;
430        tails[0].push(last, f64::from(block[30]));
431        tails[1].push(last, f64::from(block[31]));
432    }
433    if let Some(stop) = ended
434        && let Some(used) = (stop + 1..blocks.len())
435            .filter(|&index| index != SETTINGS)
436            .find(|&index| blocks[index].iter().any(|&word| word != 0xffff))
437    {
438        return Err(binary(
439            body + FRAME * (used / 4),
440            format!("block {used} holds data after block {stop}, which is unused"),
441        ));
442    }
443    if data_blocks <= SETTINGS as u64 {
444        return Err(LogError::NoData { format: FORMAT });
445    }
446
447    let mut notes = vec![
448        "the record's primary track is empty: the Raven records a pressure, not an altitude; the \
449         FIP's export works its altitudes out of the pressure"
450            .to_owned(),
451        format!(
452            "read from the Raven's memory: {data_blocks} blocks of 0.05 s, the first {SETTINGS} \
453             before the launch; a tick is {num}/{den} ms"
454        ),
455        "`slow channel 5` and `block word 30` and `31` are kept as written: the FIP's export \
456         holds neither"
457            .to_owned(),
458    ];
459    if !two_axes {
460        notes.push(
461            "the header's `#Axis` is 1: the Raven records no lateral accelerometer it calibrates, \
462             so the lateral bits are kept as written, as `lateral counts`"
463                .to_owned(),
464        );
465    }
466    if let Some(stop) = ended {
467        notes.push(format!(
468            "the memory is unused from block {stop} of {}",
469            blocks.len()
470        ));
471    }
472    let date = field("DateTime").map(str::to_owned);
473    if let Some(date) = date {
474        notes.push(format!(
475            "the header's `#DateTime` is {date:?}, as written: the file doesn't say which clock \
476             wrote it"
477        ));
478    }
479
480    let mut channels = Vec::new();
481    for filling in [axial, lateral].into_iter().chain(slow).chain(tails) {
482        channels.push(Channel {
483            column: filling.column,
484            quantity: filling.quantity,
485            time_s: filling
486                .ticks
487                .iter()
488                .map(|&tick| tick as f64 * tick_s)
489                .collect(),
490            values: filling.values,
491        });
492    }
493
494    Ok(FlightLog {
495        format: LogFormat::RavenFipa,
496        logger: "Featherweight Raven".to_owned(),
497        serial_number: None,
498        firmware: field("R").map(str::to_owned),
499        flight_number: field("FlightNum").and_then(|n| n.parse().ok()),
500        stated: Stated {
501            ground_elevation_msl_m: Some(calibration.f4 as f64 * FOOT_M),
502            ..Stated::default()
503        },
504        time_s: Vec::new(),
505        altitude_m: Vec::new(),
506        vertical_speed_m_s: None,
507        temperature_k: None,
508        battery_v: None,
509        channels,
510        events: Vec::new(),
511        notes,
512    })
513}
514
515/// What a slow channel measures, by its export name.
516fn quantity_of(name: &str) -> Quantity {
517    match name {
518        "Current Draw (A)" => Quantity::Current,
519        "Temperature (F)" => Quantity::Temperature,
520        "Baro (Atm)" => Quantity::Pressure,
521        "Velocity (Accel-Ft/Sec)" => Quantity::Speed,
522        _ => Quantity::Voltage,
523    }
524}
525
526/// The `k`th slow channel's word in SI.
527fn slow_si(k: usize, word: u16) -> f64 {
528    match k {
529        0..=4 | 6 => signed(word) as f64 / 1000.0,
530        // As `crate::raven` converts the export's column, so that the two agree to the bit.
531        7 => (signed(word) as f64 / 100.0 - 32.0) * 5.0 / 9.0 + 273.15,
532        8 => f64::from(word) / BARO_COUNTS_PER_ATM * ATM_PA,
533        9 => signed(word) as f64 / 10.0 * FOOT_M,
534        _ => f64::from(word),
535    }
536}
537
538/// A refusal at a byte of the file.
539fn binary(offset: usize, message: String) -> LogError {
540    LogError::Binary {
541        format: FORMAT,
542        offset,
543        message,
544    }
545}