Skip to main content

hpr_flightdata/
packets.rs

1//! One team's radio telemetry packets, `255:…:254`: the flight computer's own log of them, the
2//! packets a receiver heard, and the same packets in an RTF document a spreadsheet saved as CSV.
3//!
4//! A university team's flight computer writes each reading as a packet framed by `255` and `254`,
5//! and sends the packets over its radio. The team published, for one flight, four files of them:
6//! the flight computer's own log, the text a ground receiver wrote of what it heard, and two
7//! exports of the log, cut to the flight, saved as RTF documents and then as CSV. No document
8//! describes the packets, so this reader takes them from the files, and the sample logs' survey
9//! (`cargo xtask logs`) holds the reading to the files: every packet the receiver or an export
10//! holds that the onboard log also holds, by its stage, kind and clock, matches the log's field
11//! for field ([ADR-274][adr-274]).
12//!
13//! **A packet** is fields separated by colons, or by commas in the exports:
14//!
15//! ```text
16//! 255:T:11:PP:100150:950.12:21.50:3.25:254
17//! 255:T:11:AA:100152:9.70:0.40:-0.80:254
18//! 255:T:11:CC:100160:1.25:-0.50:14.00:2.50:254
19//! 255:N:88:GG:100200:1.00:2.00:1234.56:9876.54:8.00:250.00:90.00:0.50:254
20//! ```
21//!
22//! | Field | Read as |
23//! |---|---|
24//! | `255` | the packet's start |
25//! | `T` | the path it came by, one capital letter: `T` from the flight computer, `N` relayed by a second transmitter, which also sends the GPS packets |
26//! | `11` | two digits: the flight computer's stage of the flight, `00` to `55` in the samples, or `88` on a GPS packet |
27//! | `PP` | its kind, two capital letters |
28//! | `100150` | the clock, ms, a whole number: the flight computer's, or on stage `88` the relaying transmitter's own |
29//! | the values | numbers, as many as the kind's first packet holds |
30//! | `254` | the packet's end |
31//!
32//! The path, the stages' meanings and the GPS packets are inferred from the files: no document
33//! confirms them. Each kind is a set of channels on its own clock, named by the kind and the
34//! value's place from 1, such as `PP 1`, and the kind's stage field is its own channel, `PP stage`.
35//! A kind whose first packet is on stage `88`, the GPS packets in the samples, runs on the relaying
36//! transmitter's own clock, whose offset from the flight computer's nothing states: its times are
37//! gaps (`NaN`) and its clock is kept as
38//! written, ms, in a channel of its own, `GG clock`. A packet of the kind on another stage than its
39//! first's, or the reverse, is broken. The clock is read
40//! as milliseconds, the survey holding it to the flight: the time from liftoff (the first packet,
41//! after the pad's stage `00`, whose pressure is more than 1 hPa below the median of the 5 s ending
42//! 1 s before it) to the least pressure is 1.055 times √(2h/g), h the standard atmosphere's height
43//! of the two pressures, where the check allows a factor of 2. `PP 1` is read as a pressure in
44//! hectopascals, [`Quantity::Pressure`], the survey holding its pad reading between 500 and 1,100
45//! hPa, where every launch site's is; the rest is kept as written, [`Quantity::Unrecognized`], as
46//! nothing states its unit. Among them, `PP 3` climbs with the flight, but where it reads above 200 it is 1.11 to 1.24 times
47//! the standard atmosphere's height of the pressure above the pad's in meters, by the survey, so no
48//! unit is settled for it.
49//!
50//! **Lines that aren't packets.** A `Stage` line, `Stage` and a number's word from `One` to `Nine`,
51//! such as `Stage Two`, is an event at the clock of the next packet; one heard again is a repeat. The onboard log opens with a whole number and `Gain[0]: …` lines, its filter's
52//! gains, which a note counts. Any other line is broken, a packet the radio cut or ran into the next:
53//! the receiver's text holds many. A broken line is dropped, and a note counts them; so is a
54//! packet heard again, the same stage, kind, clock and values as written as one already read, by
55//! either path.
56//! A packet whose kind has another count of values than its first packet is broken.
57//!
58//! **The exports** ([`LogFormat::RadioPacketRtf`]) are an RTF document whose lines a spreadsheet
59//! split at the colons and saved as CSV, so each line has commas after it, the document's markup
60//! sits in the first cells, and a `\` ends a line where the document broke it. The reader takes each
61//! line on its own: the commas after it dropped, then the control words before a packet, such as
62//! `\f0\fs24 \cf0 `, a `\` ending the line, and the `}` closing the document at its end. A line
63//! of markup alone is dropped.
64//!
65//! **What refuses the file.** No packet at all ([`LogError::Syntax`]); more than [`MAX_KINDS`]
66//! kinds, more than [`MAX_VALUES`] values in a packet, a clock of 2³² ms, 49.7 days, or more, or a
67//! `PP 1` too large for its conversion to pascals.
68//!
69//! The record has no barometric track, as no packet states an altitude above the pad in a settled
70//! unit: [`FlightLog::time_s`] and [`FlightLog::altitude_m`] are empty, and every value is a channel.
71//!
72//! [adr-274]: https://github.com/nrdptel/fusionspace-eridanus/blob/main/docs/decisions/0274-the-rest-of-the-importers-planned-teensy-logger-first.md
73
74use std::collections::HashSet;
75
76use crate::error::LogError;
77use crate::log::{Channel, Event, FlightLog, LogFormat, Quantity, Stated};
78use crate::table::{finite, numbered_lines, short};
79
80/// The packets' text form's name in errors.
81const TEXT: &str = "radio packets";
82/// The exports' name in errors.
83const RTF: &str = "radio packets in RTF";
84
85/// How many of a file's first lines are searched for a packet.
86const RECOGNITION_LINES: usize = 100;
87
88/// The most kinds a file may hold, eight times the sample files' four.
89pub const MAX_KINDS: usize = 32;
90
91/// The most values a packet may hold, four times the sample files' eight.
92pub const MAX_VALUES: usize = 32;
93
94/// The largest clock, ms: a 32-bit count of milliseconds.
95const CLOCK_LIMIT_MS: u64 = u32::MAX as u64;
96
97/// The stage field of the relaying transmitter's packets, the GPS packets in the samples.
98const RELAYED_STAGE: &str = "88";
99
100/// Hectopascals to pascals.
101const HECTOPASCAL_PA: f64 = 100.0;
102
103/// A whole packet's fields.
104#[derive(Debug, Clone, PartialEq)]
105struct Packet<'a> {
106    stage: &'a str,
107    kind: &'a str,
108    clock_ms: u64,
109    /// The values as written, for telling a packet heard again.
110    written: Vec<&'a str>,
111    values: Vec<f64>,
112}
113
114/// Whether a field is a number as the packets write one: an optional `-`, digits, and optionally
115/// a point and more digits.
116fn is_number(field: &str) -> bool {
117    let digits = field.strip_prefix('-').unwrap_or(field);
118    let (whole, fraction) = digits.split_once('.').unwrap_or((digits, "0"));
119    !whole.is_empty()
120        && !fraction.is_empty()
121        && whole.bytes().all(|b| b.is_ascii_digit())
122        && fraction.bytes().all(|b| b.is_ascii_digit())
123}
124
125/// A line as a whole packet with fields separated by `separator`, or `None`.
126fn packet(line: &str, separator: char) -> Option<Packet<'_>> {
127    let fields: Vec<&str> = line.split(separator).collect();
128    let [start, source, stage, kind, clock, values @ .., end] = fields.as_slice() else {
129        return None;
130    };
131    let capitals = |field: &str, count: usize| {
132        field.len() == count && field.bytes().all(|b| b.is_ascii_uppercase())
133    };
134    let whole_clock =
135        !clock.is_empty() && clock.len() <= 10 && clock.bytes().all(|b| b.is_ascii_digit());
136    if *start != "255"
137        || *end != "254"
138        || !capitals(source, 1)
139        || stage.len() != 2
140        || !stage.bytes().all(|b| b.is_ascii_digit())
141        || !capitals(kind, 2)
142        || !whole_clock
143        || values.is_empty()
144        || !values.iter().all(|value| is_number(value))
145    {
146        return None;
147    }
148    let clock_ms = clock.parse::<u64>().ok()?;
149    // A number of hundreds of digits parses as infinite: no packet writes one, so the line is
150    // broken.
151    let parsed = values
152        .iter()
153        .map(|value| value.parse::<f64>().ok().filter(|v| v.is_finite()))
154        .collect::<Option<Vec<f64>>>()?;
155    Some(Packet {
156        stage,
157        kind,
158        clock_ms,
159        written: values.to_vec(),
160        values: parsed,
161    })
162}
163
164/// Whether a line is one of the onboard log's filter gains, `Gain[0]: …`.
165fn is_gain(line: &str) -> bool {
166    line.starts_with("Gain[")
167}
168
169/// The words a stage's mark may number it by, as the onboard log writes them.
170const STAGE_WORDS: [&str; 9] = [
171    "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine",
172];
173
174/// Whether a line is a stage's mark, `Stage` and a number's word, such as `Stage Two`; a word
175/// the radio garbled, such as `Nwo`, isn't one.
176fn stage_mark(line: &str) -> Option<&str> {
177    let word = line.strip_prefix("Stage ")?;
178    STAGE_WORDS.contains(&word).then_some(line)
179}
180
181/// Whether the text is the packets' text form: its first line that isn't blank is a packet, a whole
182/// number or a filter gain, and a packet separated by colons is within its first
183/// [`RECOGNITION_LINES`] lines.
184pub(crate) fn recognizes_text(text: &str) -> bool {
185    let mut lines = numbered_lines(text)
186        .take(RECOGNITION_LINES)
187        .map(|(_, line)| line.trim())
188        .filter(|line| !line.is_empty())
189        .peekable();
190    let Some(first) = lines.peek().copied() else {
191        return false;
192    };
193    let opens = packet(first, ':').is_some()
194        || (!first.is_empty() && first.len() <= 10 && first.bytes().all(|b| b.is_ascii_digit()))
195        || is_gain(first);
196    opens && lines.any(|line| packet(line, ':').is_some())
197}
198
199/// Whether the text is the exports' form: an RTF document with a packet separated by commas, its
200/// markup taken off as [`rtf_line`] does, within its first [`RECOGNITION_LINES`] lines.
201pub(crate) fn recognizes_rtf(text: &str) -> bool {
202    crate::eggtimer::recognizes_rtf(text)
203        && numbered_lines(text)
204            .take(RECOGNITION_LINES)
205            .any(|(_, line)| matches!(rtf_line(line, false), RtfLine::Text(text) if packet(text, ',').is_some()))
206}
207
208/// An export's line, its markup taken off.
209#[derive(Debug, PartialEq)]
210enum RtfLine<'a> {
211    /// The document's markup alone, or nothing.
212    Markup,
213    /// The line's text.
214    Text(&'a str),
215}
216
217/// An export's line with the spreadsheet's commas after it, the control words before its text, a
218/// `\` ending it and, on the document's `last` line, the closing `}` taken off.
219fn rtf_line(line: &str, last: bool) -> RtfLine<'_> {
220    let mut text = line.trim_end_matches(',');
221    if last {
222        text = text.strip_suffix('}').unwrap_or(text);
223    }
224    text = text.strip_suffix('\\').unwrap_or(text);
225    // Control words: `\`, letters, an optional signed number and an optional space.
226    while let Some(rest) = text.strip_prefix('\\') {
227        let letters = rest.bytes().take_while(u8::is_ascii_alphabetic).count();
228        if letters == 0 {
229            break;
230        }
231        let rest = &rest[letters..];
232        let signed = rest.strip_prefix('-').unwrap_or(rest);
233        let digits = signed.bytes().take_while(u8::is_ascii_digit).count();
234        let rest = &signed[digits..];
235        text = rest.strip_prefix(' ').unwrap_or(rest);
236    }
237    if text.is_empty() || text.starts_with('{') || text.starts_with('\\') {
238        RtfLine::Markup
239    } else {
240        RtfLine::Text(text)
241    }
242}
243
244/// One kind's packets as they are read.
245struct Kind<'a> {
246    name: &'a str,
247    count: usize,
248    /// Whether the kind runs on the relaying transmitter's own clock, as stage `88`'s packets do.
249    own_clock: bool,
250    /// Each packet's clock as written, ms, kept for a kind on its own clock.
251    clocks: Vec<f64>,
252    time_s: Vec<f64>,
253    stages: Vec<f64>,
254    values: Vec<Vec<f64>>,
255}
256
257/// The packets read so far, with the counts the notes give.
258#[derive(Default)]
259struct Reading<'a> {
260    kinds: Vec<Kind<'a>>,
261    heard: HashSet<(&'a str, &'a str, u64, Vec<&'a str>)>,
262    events: Vec<Event>,
263    pending_marks: Vec<String>,
264    marks_seen: HashSet<String>,
265    broken: usize,
266    first_broken: Option<usize>,
267    repeated: usize,
268    repeated_marks: usize,
269    gains: usize,
270}
271
272impl<'a> Reading<'a> {
273    /// Takes a packet found on line `line`.
274    fn take(
275        &mut self,
276        packet: Packet<'a>,
277        format: &'static str,
278        line: usize,
279    ) -> Result<(), LogError> {
280        if packet.written.len() > MAX_VALUES {
281            return Err(syntax(
282                format,
283                line,
284                format!("a packet holds more than {MAX_VALUES} values"),
285            ));
286        }
287        if packet.clock_ms > CLOCK_LIMIT_MS {
288            return Err(syntax(
289                format,
290                line,
291                "a packet's clock is 2³² ms or more".to_owned(),
292            ));
293        }
294        let index = match self.kinds.iter().position(|kind| kind.name == packet.kind) {
295            Some(index) => index,
296            None => {
297                if self.kinds.len() >= MAX_KINDS {
298                    return Err(syntax(
299                        format,
300                        line,
301                        format!("the file holds more than {MAX_KINDS} kinds of packet"),
302                    ));
303                }
304                self.kinds.push(Kind {
305                    name: packet.kind,
306                    count: packet.values.len(),
307                    own_clock: packet.stage == RELAYED_STAGE,
308                    clocks: Vec::new(),
309                    time_s: Vec::new(),
310                    stages: Vec::new(),
311                    values: vec![Vec::new(); packet.values.len()],
312                });
313                self.kinds.len() - 1
314            }
315        };
316        let kind = &self.kinds[index];
317        if kind.count != packet.values.len() || kind.own_clock != (packet.stage == RELAYED_STAGE) {
318            self.broke(line);
319            return Ok(());
320        }
321        if !self.heard.insert((
322            packet.stage,
323            packet.kind,
324            packet.clock_ms,
325            packet.written.clone(),
326        )) {
327            self.repeated += 1;
328            return Ok(());
329        }
330        let kind = &mut self.kinds[index];
331        if kind.own_clock {
332            // Another clock than the flight computer's, its offset unknown: no time on the log's
333            // axis.
334            kind.time_s.push(f64::NAN);
335            kind.clocks.push(packet.clock_ms as f64);
336        } else {
337            // The clock's milliseconds as seconds: a shrinking conversion, which can't overflow.
338            let time_s = packet.clock_ms as f64 / 1000.0;
339            for name in self.pending_marks.drain(..) {
340                self.events.push(Event { time_s, name });
341            }
342            kind.time_s.push(time_s);
343        }
344        kind.stages.push(f64::from(stage_number(packet.stage)));
345        for (place, (column, value)) in kind.values.iter_mut().zip(&packet.values).enumerate() {
346            let value = if packet.kind == "PP" && place == 0 {
347                finite(value * HECTOPASCAL_PA, format, line, "PP 1")?
348            } else {
349                *value
350            };
351            column.push(value);
352        }
353        Ok(())
354    }
355
356    /// Takes a stage's mark; one already taken is a repeat.
357    fn mark(&mut self, mark: &str) {
358        if self.marks_seen.insert(mark.to_owned()) {
359            self.pending_marks.push(mark.to_owned());
360        } else {
361            self.repeated_marks += 1;
362        }
363    }
364
365    /// Counts a broken line.
366    fn broke(&mut self, line: usize) {
367        self.broken += 1;
368        self.first_broken.get_or_insert(line);
369    }
370
371    /// The record, or a refusal if it holds no packet.
372    fn finish(self, format: LogFormat, name: &'static str) -> Result<FlightLog, LogError> {
373        if self.kinds.is_empty() {
374            return Err(syntax(
375                name,
376                1,
377                "the file holds no whole packet, `255`, its fields and `254`".to_owned(),
378            ));
379        }
380        let mut notes = Vec::new();
381        if self.gains > 0 {
382            notes.push(format!(
383                "{} `Gain[…]` lines, the flight computer's filter gains, kept out",
384                self.gains
385            ));
386        }
387        if let Some(line) = self.first_broken {
388            notes.push(format!(
389                "{} broken lines dropped. They are packets the radio cut or ran into the next, or \
390                 with another count of values than their kind's first. The first broken line \
391                 is line {line}",
392                self.broken
393            ));
394        }
395        if self.repeated > 0 {
396            notes.push(format!("{} packets heard again dropped", self.repeated));
397        }
398        if self.repeated_marks > 0 {
399            notes.push(format!(
400                "{} `Stage` lines heard again dropped",
401                self.repeated_marks
402            ));
403        }
404        for name in self.pending_marks {
405            notes.push(format!(
406                "the mark `{}` at the file's end, no packet after it, kept out",
407                short(&name)
408            ));
409        }
410        let mut channels = Vec::new();
411        for kind in self.kinds {
412            channels.push(Channel {
413                column: format!("{} stage", kind.name),
414                quantity: Quantity::Unrecognized,
415                time_s: kind.time_s.clone(),
416                values: kind.stages,
417            });
418            if kind.own_clock {
419                channels.push(Channel {
420                    column: format!("{} clock", kind.name),
421                    quantity: Quantity::Unrecognized,
422                    time_s: kind.time_s.clone(),
423                    values: kind.clocks,
424                });
425            }
426            for (place, values) in kind.values.into_iter().enumerate() {
427                let quantity = if kind.name == "PP" && place == 0 {
428                    Quantity::Pressure
429                } else {
430                    Quantity::Unrecognized
431                };
432                channels.push(Channel {
433                    column: format!("{} {}", kind.name, place + 1),
434                    quantity,
435                    time_s: kind.time_s.clone(),
436                    values,
437                });
438            }
439        }
440        Ok(FlightLog {
441            format,
442            logger: "radio telemetry packets".to_owned(),
443            serial_number: None,
444            firmware: None,
445            flight_number: None,
446            stated: Stated::default(),
447            time_s: Vec::new(),
448            altitude_m: Vec::new(),
449            vertical_speed_m_s: None,
450            temperature_k: None,
451            battery_v: None,
452            channels,
453            events: self.events,
454            notes,
455        })
456    }
457}
458
459/// A stage's two digits as a number.
460fn stage_number(stage: &str) -> u8 {
461    stage.bytes().fold(0, |sum, b| sum * 10 + (b - b'0'))
462}
463
464fn syntax(format: &'static str, line: usize, message: String) -> LogError {
465    LogError::Syntax {
466        format,
467        line,
468        message,
469    }
470}
471
472/// Reads the packets' text form, [`LogFormat::RadioPacketText`]: the onboard log, or the text a
473/// receiver wrote of what it heard.
474///
475/// # Errors
476///
477/// [`LogError::Syntax`] for a file the module's documentation says is refused.
478pub fn read(text: &str) -> Result<FlightLog, LogError> {
479    let mut reading = Reading::default();
480    let mut first = true;
481    for (number, line) in numbered_lines(text) {
482        let line = line.trim();
483        if line.is_empty() {
484            continue;
485        }
486        let opening = std::mem::take(&mut first);
487        if let Some(packet) = packet(line, ':') {
488            reading.take(packet, TEXT, number)?;
489        } else if let Some(mark) = stage_mark(line) {
490            reading.mark(mark);
491        } else if is_gain(line) {
492            reading.gains += 1;
493        } else if opening && line.len() <= 10 && line.bytes().all(|b| b.is_ascii_digit()) {
494            // The onboard log's first line, a whole number that no packet explains.
495        } else {
496            reading.broke(number);
497        }
498    }
499    reading.finish(LogFormat::RadioPacketText, TEXT)
500}
501
502/// Reads the exports' form, [`LogFormat::RadioPacketRtf`].
503///
504/// # Errors
505///
506/// [`LogError::Syntax`] for a file the module's documentation says is refused.
507pub fn read_rtf(text: &str) -> Result<FlightLog, LogError> {
508    let lines: Vec<(usize, &str)> = numbered_lines(text).collect();
509    let last = lines
510        .iter()
511        .rposition(|(_, line)| !line.trim().is_empty())
512        .unwrap_or(0);
513    let mut reading = Reading::default();
514    for (index, (number, line)) in lines.iter().enumerate() {
515        let RtfLine::Text(line) = rtf_line(line.trim(), index == last) else {
516            continue;
517        };
518        if let Some(packet) = packet(line, ',') {
519            reading.take(packet, RTF, *number)?;
520        } else if let Some(mark) = stage_mark(line) {
521            reading.mark(mark);
522        } else {
523            reading.broke(*number);
524        }
525    }
526    reading.finish(LogFormat::RadioPacketRtf, RTF)
527}