hpr_flightdata/packets.rs
1//! One team's radio telemetry packets, `255:…:254`: the flight computer's own log of them, the
2//! packets a receiver heard, and the same packets in an RTF document a spreadsheet saved as CSV.
3//!
4//! A university team's flight computer writes each reading as a packet framed by `255` and `254`,
5//! and sends the packets over its radio. The team published, for one flight, four files of them:
6//! the flight computer's own log, the text a ground receiver wrote of what it heard, and two
7//! exports of the log, cut to the flight, saved as RTF documents and then as CSV. No document
8//! describes the packets, so this reader takes them from the files, and the sample logs' survey
9//! (`cargo xtask logs`) holds the reading to the files: every packet the receiver or an export
10//! holds that the onboard log also holds, by its stage, kind and clock, matches the log's field
11//! for field ([ADR-274][adr-274]).
12//!
13//! **A packet** is fields separated by colons, or by commas in the exports:
14//!
15//! ```text
16//! 255:T:11:PP:100150:950.12:21.50:3.25:254
17//! 255:T:11:AA:100152:9.70:0.40:-0.80:254
18//! 255:T:11:CC:100160:1.25:-0.50:14.00:2.50:254
19//! 255:N:88:GG:100200:1.00:2.00:1234.56:9876.54:8.00:250.00:90.00:0.50:254
20//! ```
21//!
22//! | Field | Read as |
23//! |---|---|
24//! | `255` | the packet's start |
25//! | `T` | the path it came by, one capital letter: `T` from the flight computer, `N` relayed by a second transmitter, which also sends the GPS packets |
26//! | `11` | two digits: the flight computer's stage of the flight, `00` to `55` in the samples, or `88` on a GPS packet |
27//! | `PP` | its kind, two capital letters |
28//! | `100150` | the clock, ms, a whole number: the flight computer's, or on stage `88` the relaying transmitter's own |
29//! | the values | numbers, as many as the kind's first packet holds |
30//! | `254` | the packet's end |
31//!
32//! The path, the stages' meanings and the GPS packets are inferred from the files: no document
33//! confirms them. Each kind is a set of channels on its own clock, named by the kind and the
34//! value's place from 1, such as `PP 1`, and the kind's stage field is its own channel, `PP stage`.
35//! A kind whose first packet is on stage `88`, the GPS packets in the samples, runs on the relaying
36//! transmitter's own clock, whose offset from the flight computer's nothing states: its times are
37//! gaps (`NaN`) and its clock is kept as
38//! written, ms, in a channel of its own, `GG clock`. A packet of the kind on another stage than its
39//! first's, or the reverse, is broken. The clock is read
40//! as milliseconds, the survey holding it to the flight: the time from liftoff (the first packet,
41//! after the pad's stage `00`, whose pressure is more than 1 hPa below the median of the 5 s ending
42//! 1 s before it) to the least pressure is 1.055 times √(2h/g), h the standard atmosphere's height
43//! of the two pressures, where the check allows a factor of 2. `PP 1` is read as a pressure in
44//! hectopascals, [`Quantity::Pressure`], the survey holding its pad reading between 500 and 1,100
45//! hPa, where every launch site's is; the rest is kept as written, [`Quantity::Unrecognized`], as
46//! nothing states its unit. Among them, `PP 3` climbs with the flight, but where it reads above 200 it is 1.11 to 1.24 times
47//! the standard atmosphere's height of the pressure above the pad's in meters, by the survey, so no
48//! unit is settled for it.
49//!
50//! **Lines that aren't packets.** A `Stage` line, `Stage` and a number's word from `One` to `Nine`,
51//! such as `Stage Two`, is an event at the clock of the next packet; one heard again is a repeat. The onboard log opens with a whole number and `Gain[0]: …` lines, its filter's
52//! gains, which a note counts. Any other line is broken, a packet the radio cut or ran into the next:
53//! the receiver's text holds many. A broken line is dropped, and a note counts them; so is a
54//! packet heard again, the same stage, kind, clock and values as written as one already read, by
55//! either path.
56//! A packet whose kind has another count of values than its first packet is broken.
57//!
58//! **The exports** ([`LogFormat::RadioPacketRtf`]) are an RTF document whose lines a spreadsheet
59//! split at the colons and saved as CSV, so each line has commas after it, the document's markup
60//! sits in the first cells, and a `\` ends a line where the document broke it. The reader takes each
61//! line on its own: the commas after it dropped, then the control words before a packet, such as
62//! `\f0\fs24 \cf0 `, a `\` ending the line, and the `}` closing the document at its end. A line
63//! of markup alone is dropped.
64//!
65//! **What refuses the file.** No packet at all ([`LogError::Syntax`]); more than [`MAX_KINDS`]
66//! kinds, more than [`MAX_VALUES`] values in a packet, a clock of 2³² ms, 49.7 days, or more, or a
67//! `PP 1` too large for its conversion to pascals.
68//!
69//! The record has no barometric track, as no packet states an altitude above the pad in a settled
70//! unit: [`FlightLog::time_s`] and [`FlightLog::altitude_m`] are empty, and every value is a channel.
71//!
72//! [adr-274]: https://github.com/nrdptel/fusionspace-eridanus/blob/main/docs/decisions/0274-the-rest-of-the-importers-planned-teensy-logger-first.md
73
74use std::collections::HashSet;
75
76use crate::error::LogError;
77use crate::log::{Channel, Event, FlightLog, LogFormat, Quantity, Stated};
78use crate::table::{finite, numbered_lines, short};
79
80/// The packets' text form's name in errors.
81const TEXT: &str = "radio packets";
82/// The exports' name in errors.
83const RTF: &str = "radio packets in RTF";
84
85/// How many of a file's first lines are searched for a packet.
86const RECOGNITION_LINES: usize = 100;
87
88/// The most kinds a file may hold, eight times the sample files' four.
89pub const MAX_KINDS: usize = 32;
90
91/// The most values a packet may hold, four times the sample files' eight.
92pub const MAX_VALUES: usize = 32;
93
94/// The largest clock, ms: a 32-bit count of milliseconds.
95const CLOCK_LIMIT_MS: u64 = u32::MAX as u64;
96
97/// The stage field of the relaying transmitter's packets, the GPS packets in the samples.
98const RELAYED_STAGE: &str = "88";
99
100/// Hectopascals to pascals.
101const HECTOPASCAL_PA: f64 = 100.0;
102
103/// A whole packet's fields.
104#[derive(Debug, Clone, PartialEq)]
105struct Packet<'a> {
106 stage: &'a str,
107 kind: &'a str,
108 clock_ms: u64,
109 /// The values as written, for telling a packet heard again.
110 written: Vec<&'a str>,
111 values: Vec<f64>,
112}
113
114/// Whether a field is a number as the packets write one: an optional `-`, digits, and optionally
115/// a point and more digits.
116fn is_number(field: &str) -> bool {
117 let digits = field.strip_prefix('-').unwrap_or(field);
118 let (whole, fraction) = digits.split_once('.').unwrap_or((digits, "0"));
119 !whole.is_empty()
120 && !fraction.is_empty()
121 && whole.bytes().all(|b| b.is_ascii_digit())
122 && fraction.bytes().all(|b| b.is_ascii_digit())
123}
124
125/// A line as a whole packet with fields separated by `separator`, or `None`.
126fn packet(line: &str, separator: char) -> Option<Packet<'_>> {
127 let fields: Vec<&str> = line.split(separator).collect();
128 let [start, source, stage, kind, clock, values @ .., end] = fields.as_slice() else {
129 return None;
130 };
131 let capitals = |field: &str, count: usize| {
132 field.len() == count && field.bytes().all(|b| b.is_ascii_uppercase())
133 };
134 let whole_clock =
135 !clock.is_empty() && clock.len() <= 10 && clock.bytes().all(|b| b.is_ascii_digit());
136 if *start != "255"
137 || *end != "254"
138 || !capitals(source, 1)
139 || stage.len() != 2
140 || !stage.bytes().all(|b| b.is_ascii_digit())
141 || !capitals(kind, 2)
142 || !whole_clock
143 || values.is_empty()
144 || !values.iter().all(|value| is_number(value))
145 {
146 return None;
147 }
148 let clock_ms = clock.parse::<u64>().ok()?;
149 // A number of hundreds of digits parses as infinite: no packet writes one, so the line is
150 // broken.
151 let parsed = values
152 .iter()
153 .map(|value| value.parse::<f64>().ok().filter(|v| v.is_finite()))
154 .collect::<Option<Vec<f64>>>()?;
155 Some(Packet {
156 stage,
157 kind,
158 clock_ms,
159 written: values.to_vec(),
160 values: parsed,
161 })
162}
163
164/// Whether a line is one of the onboard log's filter gains, `Gain[0]: …`.
165fn is_gain(line: &str) -> bool {
166 line.starts_with("Gain[")
167}
168
169/// The words a stage's mark may number it by, as the onboard log writes them.
170const STAGE_WORDS: [&str; 9] = [
171 "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine",
172];
173
174/// Whether a line is a stage's mark, `Stage` and a number's word, such as `Stage Two`; a word
175/// the radio garbled, such as `Nwo`, isn't one.
176fn stage_mark(line: &str) -> Option<&str> {
177 let word = line.strip_prefix("Stage ")?;
178 STAGE_WORDS.contains(&word).then_some(line)
179}
180
181/// Whether the text is the packets' text form: its first line that isn't blank is a packet, a whole
182/// number or a filter gain, and a packet separated by colons is within its first
183/// [`RECOGNITION_LINES`] lines.
184pub(crate) fn recognizes_text(text: &str) -> bool {
185 let mut lines = numbered_lines(text)
186 .take(RECOGNITION_LINES)
187 .map(|(_, line)| line.trim())
188 .filter(|line| !line.is_empty())
189 .peekable();
190 let Some(first) = lines.peek().copied() else {
191 return false;
192 };
193 let opens = packet(first, ':').is_some()
194 || (!first.is_empty() && first.len() <= 10 && first.bytes().all(|b| b.is_ascii_digit()))
195 || is_gain(first);
196 opens && lines.any(|line| packet(line, ':').is_some())
197}
198
199/// Whether the text is the exports' form: an RTF document with a packet separated by commas, its
200/// markup taken off as [`rtf_line`] does, within its first [`RECOGNITION_LINES`] lines.
201pub(crate) fn recognizes_rtf(text: &str) -> bool {
202 crate::eggtimer::recognizes_rtf(text)
203 && numbered_lines(text)
204 .take(RECOGNITION_LINES)
205 .any(|(_, line)| matches!(rtf_line(line, false), RtfLine::Text(text) if packet(text, ',').is_some()))
206}
207
208/// An export's line, its markup taken off.
209#[derive(Debug, PartialEq)]
210enum RtfLine<'a> {
211 /// The document's markup alone, or nothing.
212 Markup,
213 /// The line's text.
214 Text(&'a str),
215}
216
217/// An export's line with the spreadsheet's commas after it, the control words before its text, a
218/// `\` ending it and, on the document's `last` line, the closing `}` taken off.
219fn rtf_line(line: &str, last: bool) -> RtfLine<'_> {
220 let mut text = line.trim_end_matches(',');
221 if last {
222 text = text.strip_suffix('}').unwrap_or(text);
223 }
224 text = text.strip_suffix('\\').unwrap_or(text);
225 // Control words: `\`, letters, an optional signed number and an optional space.
226 while let Some(rest) = text.strip_prefix('\\') {
227 let letters = rest.bytes().take_while(u8::is_ascii_alphabetic).count();
228 if letters == 0 {
229 break;
230 }
231 let rest = &rest[letters..];
232 let signed = rest.strip_prefix('-').unwrap_or(rest);
233 let digits = signed.bytes().take_while(u8::is_ascii_digit).count();
234 let rest = &signed[digits..];
235 text = rest.strip_prefix(' ').unwrap_or(rest);
236 }
237 if text.is_empty() || text.starts_with('{') || text.starts_with('\\') {
238 RtfLine::Markup
239 } else {
240 RtfLine::Text(text)
241 }
242}
243
244/// One kind's packets as they are read.
245struct Kind<'a> {
246 name: &'a str,
247 count: usize,
248 /// Whether the kind runs on the relaying transmitter's own clock, as stage `88`'s packets do.
249 own_clock: bool,
250 /// Each packet's clock as written, ms, kept for a kind on its own clock.
251 clocks: Vec<f64>,
252 time_s: Vec<f64>,
253 stages: Vec<f64>,
254 values: Vec<Vec<f64>>,
255}
256
257/// The packets read so far, with the counts the notes give.
258#[derive(Default)]
259struct Reading<'a> {
260 kinds: Vec<Kind<'a>>,
261 heard: HashSet<(&'a str, &'a str, u64, Vec<&'a str>)>,
262 events: Vec<Event>,
263 pending_marks: Vec<String>,
264 marks_seen: HashSet<String>,
265 broken: usize,
266 first_broken: Option<usize>,
267 repeated: usize,
268 repeated_marks: usize,
269 gains: usize,
270}
271
272impl<'a> Reading<'a> {
273 /// Takes a packet found on line `line`.
274 fn take(
275 &mut self,
276 packet: Packet<'a>,
277 format: &'static str,
278 line: usize,
279 ) -> Result<(), LogError> {
280 if packet.written.len() > MAX_VALUES {
281 return Err(syntax(
282 format,
283 line,
284 format!("a packet holds more than {MAX_VALUES} values"),
285 ));
286 }
287 if packet.clock_ms > CLOCK_LIMIT_MS {
288 return Err(syntax(
289 format,
290 line,
291 "a packet's clock is 2³² ms or more".to_owned(),
292 ));
293 }
294 let index = match self.kinds.iter().position(|kind| kind.name == packet.kind) {
295 Some(index) => index,
296 None => {
297 if self.kinds.len() >= MAX_KINDS {
298 return Err(syntax(
299 format,
300 line,
301 format!("the file holds more than {MAX_KINDS} kinds of packet"),
302 ));
303 }
304 self.kinds.push(Kind {
305 name: packet.kind,
306 count: packet.values.len(),
307 own_clock: packet.stage == RELAYED_STAGE,
308 clocks: Vec::new(),
309 time_s: Vec::new(),
310 stages: Vec::new(),
311 values: vec![Vec::new(); packet.values.len()],
312 });
313 self.kinds.len() - 1
314 }
315 };
316 let kind = &self.kinds[index];
317 if kind.count != packet.values.len() || kind.own_clock != (packet.stage == RELAYED_STAGE) {
318 self.broke(line);
319 return Ok(());
320 }
321 if !self.heard.insert((
322 packet.stage,
323 packet.kind,
324 packet.clock_ms,
325 packet.written.clone(),
326 )) {
327 self.repeated += 1;
328 return Ok(());
329 }
330 let kind = &mut self.kinds[index];
331 if kind.own_clock {
332 // Another clock than the flight computer's, its offset unknown: no time on the log's
333 // axis.
334 kind.time_s.push(f64::NAN);
335 kind.clocks.push(packet.clock_ms as f64);
336 } else {
337 // The clock's milliseconds as seconds: a shrinking conversion, which can't overflow.
338 let time_s = packet.clock_ms as f64 / 1000.0;
339 for name in self.pending_marks.drain(..) {
340 self.events.push(Event { time_s, name });
341 }
342 kind.time_s.push(time_s);
343 }
344 kind.stages.push(f64::from(stage_number(packet.stage)));
345 for (place, (column, value)) in kind.values.iter_mut().zip(&packet.values).enumerate() {
346 let value = if packet.kind == "PP" && place == 0 {
347 finite(value * HECTOPASCAL_PA, format, line, "PP 1")?
348 } else {
349 *value
350 };
351 column.push(value);
352 }
353 Ok(())
354 }
355
356 /// Takes a stage's mark; one already taken is a repeat.
357 fn mark(&mut self, mark: &str) {
358 if self.marks_seen.insert(mark.to_owned()) {
359 self.pending_marks.push(mark.to_owned());
360 } else {
361 self.repeated_marks += 1;
362 }
363 }
364
365 /// Counts a broken line.
366 fn broke(&mut self, line: usize) {
367 self.broken += 1;
368 self.first_broken.get_or_insert(line);
369 }
370
371 /// The record, or a refusal if it holds no packet.
372 fn finish(self, format: LogFormat, name: &'static str) -> Result<FlightLog, LogError> {
373 if self.kinds.is_empty() {
374 return Err(syntax(
375 name,
376 1,
377 "the file holds no whole packet, `255`, its fields and `254`".to_owned(),
378 ));
379 }
380 let mut notes = Vec::new();
381 if self.gains > 0 {
382 notes.push(format!(
383 "{} `Gain[…]` lines, the flight computer's filter gains, kept out",
384 self.gains
385 ));
386 }
387 if let Some(line) = self.first_broken {
388 notes.push(format!(
389 "{} broken lines dropped. They are packets the radio cut or ran into the next, or \
390 with another count of values than their kind's first. The first broken line \
391 is line {line}",
392 self.broken
393 ));
394 }
395 if self.repeated > 0 {
396 notes.push(format!("{} packets heard again dropped", self.repeated));
397 }
398 if self.repeated_marks > 0 {
399 notes.push(format!(
400 "{} `Stage` lines heard again dropped",
401 self.repeated_marks
402 ));
403 }
404 for name in self.pending_marks {
405 notes.push(format!(
406 "the mark `{}` at the file's end, no packet after it, kept out",
407 short(&name)
408 ));
409 }
410 let mut channels = Vec::new();
411 for kind in self.kinds {
412 channels.push(Channel {
413 column: format!("{} stage", kind.name),
414 quantity: Quantity::Unrecognized,
415 time_s: kind.time_s.clone(),
416 values: kind.stages,
417 });
418 if kind.own_clock {
419 channels.push(Channel {
420 column: format!("{} clock", kind.name),
421 quantity: Quantity::Unrecognized,
422 time_s: kind.time_s.clone(),
423 values: kind.clocks,
424 });
425 }
426 for (place, values) in kind.values.into_iter().enumerate() {
427 let quantity = if kind.name == "PP" && place == 0 {
428 Quantity::Pressure
429 } else {
430 Quantity::Unrecognized
431 };
432 channels.push(Channel {
433 column: format!("{} {}", kind.name, place + 1),
434 quantity,
435 time_s: kind.time_s.clone(),
436 values,
437 });
438 }
439 }
440 Ok(FlightLog {
441 format,
442 logger: "radio telemetry packets".to_owned(),
443 serial_number: None,
444 firmware: None,
445 flight_number: None,
446 stated: Stated::default(),
447 time_s: Vec::new(),
448 altitude_m: Vec::new(),
449 vertical_speed_m_s: None,
450 temperature_k: None,
451 battery_v: None,
452 channels,
453 events: self.events,
454 notes,
455 })
456 }
457}
458
459/// A stage's two digits as a number.
460fn stage_number(stage: &str) -> u8 {
461 stage.bytes().fold(0, |sum, b| sum * 10 + (b - b'0'))
462}
463
464fn syntax(format: &'static str, line: usize, message: String) -> LogError {
465 LogError::Syntax {
466 format,
467 line,
468 message,
469 }
470}
471
472/// Reads the packets' text form, [`LogFormat::RadioPacketText`]: the onboard log, or the text a
473/// receiver wrote of what it heard.
474///
475/// # Errors
476///
477/// [`LogError::Syntax`] for a file the module's documentation says is refused.
478pub fn read(text: &str) -> Result<FlightLog, LogError> {
479 let mut reading = Reading::default();
480 let mut first = true;
481 for (number, line) in numbered_lines(text) {
482 let line = line.trim();
483 if line.is_empty() {
484 continue;
485 }
486 let opening = std::mem::take(&mut first);
487 if let Some(packet) = packet(line, ':') {
488 reading.take(packet, TEXT, number)?;
489 } else if let Some(mark) = stage_mark(line) {
490 reading.mark(mark);
491 } else if is_gain(line) {
492 reading.gains += 1;
493 } else if opening && line.len() <= 10 && line.bytes().all(|b| b.is_ascii_digit()) {
494 // The onboard log's first line, a whole number that no packet explains.
495 } else {
496 reading.broke(number);
497 }
498 }
499 reading.finish(LogFormat::RadioPacketText, TEXT)
500}
501
502/// Reads the exports' form, [`LogFormat::RadioPacketRtf`].
503///
504/// # Errors
505///
506/// [`LogError::Syntax`] for a file the module's documentation says is refused.
507pub fn read_rtf(text: &str) -> Result<FlightLog, LogError> {
508 let lines: Vec<(usize, &str)> = numbered_lines(text).collect();
509 let last = lines
510 .iter()
511 .rposition(|(_, line)| !line.trim().is_empty())
512 .unwrap_or(0);
513 let mut reading = Reading::default();
514 for (index, (number, line)) in lines.iter().enumerate() {
515 let RtfLine::Text(line) = rtf_line(line.trim(), index == last) else {
516 continue;
517 };
518 if let Some(packet) = packet(line, ',') {
519 reading.take(packet, RTF, *number)?;
520 } else if let Some(mark) = stage_mark(line) {
521 reading.mark(mark);
522 } else {
523 reading.broke(*number);
524 }
525 }
526 reading.finish(LogFormat::RadioPacketRtf, RTF)
527}