Skip to main content

hpr_flightdata/
rtx.rs

1//! Missile Works' RTx/GPS tracker: the `.rtx` file its base station's software saves when it
2//! downloads the tracker's fixes, and the CSV and text files the same software exports from it.
3//!
4//! The RTx, built around a u-blox 7 receiver, keeps its fixes at 1 Hz, "~68 minutes per flight",
5//! and sends them to a base station "at 1-Hz intervals" (*RTx/GPS User Manual*, firmware 1.31,
6//! Missile Works, 2016-11-26, pp2, 14); Missile Works' mDACS software saves a download "with the
7//! .rtx file extension" and exports it "to Google Earth KML, Excel CSV, or a simple text format"
8//! (*mDACS User Manual*, revision 1.71, May 2024, p16), the KML read by [`crate::kml`]. Neither
9//! manual describes the files; what follows was learned from the 5 `.rtx` files among the
10//! project's sample logs and the 13 exports beside them, with no other program's source read.
11//!
12//! **The `.rtx` file** is a .NET `BinaryFormatter` stream, the layout Microsoft publishes as
13//! MS-NRBF (*[MS-NRBF]: .NET Remoting: Binary Format Data Structure*, revision 13.0, 2024-11-19),
14//! of one `List<Int32>`: a serialization header (§2.6.1), the list's class record,
15//! `SystemClassWithMembersAndTypes` (§2.3.2.3) with its three members `_items`, `_size` and
16//! `_version`, a `MemberReference` (§2.5.3) to the items, the two counts, an
17//! `ArraySinglePrimitive` (§2.4.3.3) of 32-bit integers, and `MessageEnd` (§2.6.3). Every sample
18//! file holds the same 186 bytes up to `_size`, a list of 65,536 slots, and nothing after the end
19//! record. The reader takes that shape and no other: a stream that differs is refused at the first
20//! byte that does.
21//!
22//! **The words.** Each slot holds a 16-bit value, but not as written: the value times 387,420,489
23//! (3¹⁸), modulo 10⁹. The reader multiplies by 513,180,409, that multiplier's inverse modulo 10⁹,
24//! to read it back. Three things in the files show it: the slot that holds the time of day steps
25//! by exactly 387,420,489, modulo 10⁹, from one record to the next a second later; every one of
26//! the 264,336 slots in use in the 5 files lies from 0 to 10⁹ − 1 and reads back as a value below
27//! 65,536, which a slot drawn at random from that range would do about once in 15,000; and the values read back
28//! equal the exports' figures, field by field, on every row (`cargo xtask logs` compares them). A
29//! slot outside 0 to 10⁹ − 1, or one that reads back past 65,535, is refused.
30//!
31//! **The records.** `_size` counts the slots in use, 16 a record, oldest first; the slots after
32//! them are 0. The list's 65,536 slots hold 4,096 records, 68.3 minutes at 1 Hz, the manual's
33//! flight. Each record is one fix, its 16 words, numbered from 0:
34//!
35//! | Words | Field | As read |
36//! |---|---|---|
37//! | 0, 1 | `UTC Time`, `HHMMSS` | word 1 × 65,536 + word 0 |
38//! | 2 | `Latitude`'s degrees and whole minutes, `DDMM` | |
39//! | 3, 4 | its minutes' five decimals, after a leading 1 | word 4 × 65,536 + word 3 − 100,000 |
40//! | 5 | `Longitude`'s degrees and whole minutes, `DDDMM` | |
41//! | 6, 7 | its minutes' five decimals, after a leading 1 | word 7 × 65,536 + word 6 − 100,000 |
42//! | 8 | `Altitude`, m | |
43//! | 9 | `Velocity` | kept as written |
44//! | 10 | `VDOP` | kept as written |
45//! | 11 | the hemispheres: `N` or `S` in its low byte, `E` or `W` in its high | |
46//! | 12 | `NavStatus` in its low byte, `Sats` in its high | |
47//! | 13 | `SOG` | kept as written |
48//! | 14 | `COG`, degrees | |
49//! | 15 | a word the exports leave out | kept as written |
50//!
51//! The leading 1 keeps a minute's decimals' leading zeros, as `193418` holds `.93418`.
52//!
53//! **Units and the second checks.** The altitude is in whole meters: the RTx's manual says its GPS
54//! altitude "is always expressed as Mean Sea Level" (p13), the KML export beside each sample file
55//! writes the word itself, as meters, on every point, and the CSV and text exports write the word ×
56//! 3.280 84 ft rounded down on every row. A word can't fall below 0: in 4 of the 5 sample files,
57//! from pads near sea level, it reads 0 for most of the fixes, as a receiver's altitude clamped at
58//! 0 would, and a note counts those 0s. The survey (`cargo xtask logs`) holds the exports' feet and
59//! meters to the words, and reports, without checking, the climb from the pad to the highest fix
60//! against the barometric altimeter flown beside the tracker: 0.98 to 1.15 times it in 4 flights.
61//! `COG` is a course in degrees, as a GPS receiver's course over ground is: it lies from 0 to 359
62//! in the sample files, and in each the median difference either way from the bearing between the
63//! fixes a second either side of it, while the tracker moves, is 1.7° to 6.9°. `Velocity` and `SOG`
64//! are kept as written: `SOG` runs 3.5 to 4.0 times the speed between fixes in meters a second, the
65//! median in each file, as kilometers an hour would, one of the two speed units the RTx's display
66//! offers (p12), but no document says which unit the file holds, and feet a second would give 3.3;
67//! `Velocity` reads, while the rocket climbs, far more than the climb rate in meters a second, and
68//! while it falls, about the fall rate itself, a pattern no document explains. `VDOP`, `NavStatus`
69//! and `Sats` are numbers, and word 15, which runs from 814 to 2,272 in the sample files, is kept
70//! as written.
71//!
72//! **The clock** is the GPS's time of day, UTC, in whole seconds; the file states no date. The
73//! log's zero is the first record's time; each time takes the day that puts it nearest the one
74//! before it, so a time more than 12 hours before the one before is read as the next day's, and
75//! one more than 12 hours after it, as a repeated pair written across midnight would be, as the
76//! day before's; a log that truly skips more than 12 hours would be read a day off. Records keep the file's order, so a time may go back: in 3 of the 5
77//! sample files a pair of records repeats the pair before it a second back, as `t`, `t + 1`, `t`,
78//! `t + 1`. A note counts how often times go back. A time that isn't a time of day, such as the
79//! record in one sample file whose word 1 reads 65,282 where every other reads 2, as a garbled
80//! radio packet's might, is a gap, counted in a note.
81//!
82//! **Positions.** A latitude's or longitude's minutes past 59, decimals that don't start with the
83//! leading 1, a hemisphere byte other than its two letters, or a latitude past 90° or longitude
84//! past 180° make that coordinate a gap, counted in a note; so do both, where both read exactly 0,
85//! as a receiver writes before it has a fix. No sample file has either.
86//!
87//! **The exports.** The CSV opens `UTC Time,Latitude,Longitude,Altitude,Velocity,VDOP,NavStatus,
88//! Sats,SOG,COG`; the text file, tab-separated, `UTC`, `Lat`, an empty name, `Lon`, an empty name,
89//! `Alt`, `Vel`, `VDOP`, `Nav`, `Sats`, `SOG`, `COG`, its rows ten cells, as the CSV's. Both write
90//! the time as `HHMMSS`, a position either as `DDMM.mmmmm` and its hemisphere's letter, such as
91//! `3259.01234N`, or, in 5 of the 9 sample exports, in signed degrees cut toward zero at the
92//! sixth decimal and a degree sign, such as `-106.975720°`, and the altitude in whole feet, read as 0.3048 m each. Each export
93//! holds every record of its `.rtx` but the last, in the same order (`cargo xtask logs`). A row of
94//! other than ten cells, or a cell that isn't its field's number, is refused, naming the line; a
95//! time or a position out of range is a gap, as in the `.rtx`.
96//!
97//! **The record.** No barometer: the primary track is empty, with a note, and every field is a
98//! channel on the clock above, as [`crate::fwgps`] reads a GPS tracker's. The logger is `Missile
99//! Works RTx`; the files state no serial number, firmware, flight number or date.
100//!
101//! **Refused**, the `.rtx` naming the byte: a stream that isn't the list's shape above; a `_size`
102//! that isn't a whole number of records or that passes the list's slots; a slot in use that isn't
103//! a word, as above; a slot past `_size` that isn't 0; and a file with no record whose time is a
104//! time of day.
105
106use crate::error::LogError;
107use crate::log::{Channel, FlightLog, LogFormat, Quantity, Stated};
108use crate::table::{Notes, short};
109
110/// The `.rtx` format's name in errors.
111const RAW: &str = "RTx .rtx";
112
113/// The CSV export's name in errors.
114const CSV: &str = "RTx CSV";
115
116/// The text export's name in errors.
117const TEXT: &str = "RTx text";
118
119/// The logger, as the record names it, and as [`crate::kml`] names the RTx's KML.
120const LOGGER: &str = "Missile Works RTx";
121
122/// MS-NRBF's `SerializationHeaderRecord` (§2.6.1): record type 0, `RootId` 1, `HeaderId` −1,
123/// version 1.0.
124const SERIALIZATION_HEADER: &[u8] = &[
125    0x00, 0x01, 0x00, 0x00, 0x00, 0xff, 0xff, 0xff, 0xff, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
126    0x00,
127];
128
129/// `SystemClassWithMembersAndTypes` (§2.3.2.3, record type 4) for object 1, then its name's
130/// length, 126, as a one-byte `LengthPrefixedString` length (§2.1.1.6).
131const CLASS_RECORD: &[u8] = &[0x04, 0x01, 0x00, 0x00, 0x00, 0x7e];
132
133/// The list's class, as the stream names it.
134const CLASS_NAME: &[u8] = b"System.Collections.Generic.List`1[[System.Int32, mscorlib, \
135Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]";
136
137/// The class's three members, their names, their `BinaryTypeEnumeration`s (§2.1.2.2: 7, a
138/// primitive array, then two primitives) and their primitive types (§2.1.2.3: 8, `Int32`, each),
139/// then a `MemberReference` (§2.5.3, record type 9) to object 2, the items.
140const MEMBERS: &[u8] = b"\x03\x00\x00\x00\x06_items\x05_size\x08_version\x07\x00\x00\x08\x08\x08\
141\x09\x02\x00\x00\x00";
142
143/// `ArraySinglePrimitive` (§2.4.3.3, record type 15) for object 2.
144const ARRAY_RECORD: &[u8] = &[0x0f, 0x02, 0x00, 0x00, 0x00];
145
146/// The array's primitive type, `Int32` (§2.1.2.3).
147const INT32: u8 = 8;
148
149/// `MessageEnd` (§2.6.3, record type 11).
150const MESSAGE_END: u8 = 0x0b;
151
152/// The words in a record.
153const RECORD_WORDS: usize = 16;
154
155/// The modulus of the slots' mapping (module docs).
156const MODULUS: u64 = 1_000_000_000;
157
158/// What a word is multiplied by, modulo [`MODULUS`], to make its slot: 3¹⁸.
159const MULTIPLIER: u64 = 387_420_489;
160
161/// [`MULTIPLIER`]'s inverse modulo [`MODULUS`]: a slot times it, modulo [`MODULUS`], is its word.
162const INVERSE: u64 = 513_180_409;
163
164/// The minutes' decimals carry a leading 1, so a decimals word pair reads from this to twice it,
165/// less one.
166const DECIMALS_LEAD: u32 = 100_000;
167
168/// A day, s.
169const DAY_S: f64 = 86_400.0;
170
171/// A time this far before the one before it is the next day's, and this far after it the day
172/// before's, s.
173const NEXT_DAY_S: f64 = DAY_S / 2.0;
174
175/// A foot, m (international foot).
176const FOOT_M: f64 = 0.3048;
177
178/// The CSV export's header.
179const CSV_HEADER: [&str; 10] = [
180    "UTC Time",
181    "Latitude",
182    "Longitude",
183    "Altitude",
184    "Velocity",
185    "VDOP",
186    "NavStatus",
187    "Sats",
188    "SOG",
189    "COG",
190];
191
192/// The text export's header, with its two empty names.
193const TEXT_HEADER: [&str; 12] = [
194    "UTC", "Lat", "", "Lon", "", "Alt", "Vel", "VDOP", "Nav", "Sats", "SOG", "COG",
195];
196
197/// The `.rtx`'s altitude step, m.
198pub const ALTITUDE_STEP_M: f64 = 1.0;
199
200/// The exports' altitude step, m.
201pub const EXPORT_ALTITUDE_STEP_M: f64 = FOOT_M;
202
203/// Whether the bytes open as an `.rtx` file: MS-NRBF's serialization header, then the class record
204/// of a `List<Int32>` as the RTx's software writes it.
205pub fn recognizes_raw(bytes: &[u8]) -> bool {
206    bytes
207        .strip_prefix(SERIALIZATION_HEADER)
208        .and_then(|rest| rest.strip_prefix(CLASS_RECORD))
209        .is_some_and(|rest| rest.starts_with(CLASS_NAME))
210}
211
212/// Which export a line is the header of, if either.
213fn export_header(line: &str) -> Option<LogFormat> {
214    if line.split(',').map(str::trim).eq(CSV_HEADER) {
215        Some(LogFormat::RtxCsv)
216    } else if line.split('\t').map(str::trim).eq(TEXT_HEADER) {
217        Some(LogFormat::RtxText)
218    } else {
219        None
220    }
221}
222
223/// Which RTx export a file's text is, if either: its first line that isn't blank is the CSV's or
224/// the text file's header, the names as written.
225pub fn export_format(text: &str) -> Option<LogFormat> {
226    text.strip_prefix('\u{feff}')
227        .unwrap_or(text)
228        .split(['\r', '\n'])
229        .find(|line| !line.trim().is_empty())
230        .and_then(export_header)
231}
232
233/// A slot's word: the slot times [`INVERSE`] modulo [`MODULUS`]; `None` for a slot outside 0 to
234/// 10⁹ − 1, or one that reads back past 65,535.
235fn word(slot: i32) -> Option<u16> {
236    let slot = u64::try_from(slot).ok().filter(|&s| s < MODULUS)?;
237    u16::try_from(slot * INVERSE % MODULUS).ok()
238}
239
240/// A slot for a word, as the RTx's software writes it: the inverse of [`word`]. For tests.
241#[doc(hidden)]
242pub fn slot(word: u16) -> i32 {
243    #[allow(
244        clippy::cast_possible_truncation,
245        reason = "a remainder modulo 10⁹ is within an `i32`"
246    )]
247    {
248        (u64::from(word) * MULTIPLIER % MODULUS) as i32
249    }
250}
251
252/// An `HHMMSS` time as seconds after midnight; `None` if it isn't a time of day. A second of 60
253/// is a leap second's.
254fn time_of_day(hhmmss: u64) -> Option<f64> {
255    let (hours, minutes, seconds) = (hhmmss / 10_000, hhmmss / 100 % 100, hhmmss % 100);
256    (hours < 24 && minutes < 60 && seconds <= 60).then(|| {
257        #[allow(
258            clippy::cast_precision_loss,
259            reason = "under 86,401 s, exact in an `f64`"
260        )]
261        {
262            (hours * 3_600 + minutes * 60 + seconds) as f64
263        }
264    })
265}
266
267/// A coordinate from its `DDMM` or `DDDMM` word, its decimals word pair and its hemisphere's byte;
268/// `None` if any part is out of range (module docs).
269fn coordinate(
270    degrees_minutes: u16,
271    decimals: u32,
272    hemisphere: u8,
273    letters: [u8; 2],
274    most: f64,
275) -> Option<f64> {
276    let decimals = decimals.checked_sub(DECIMALS_LEAD)?;
277    if decimals >= DECIMALS_LEAD || degrees_minutes % 100 >= 60 {
278        return None;
279    }
280    let minutes = f64::from(degrees_minutes % 100) + f64::from(decimals) / 1e5;
281    let value = f64::from(degrees_minutes / 100) + minutes / 60.0;
282    let sign = if hemisphere == letters[0] {
283        1.0
284    } else if hemisphere == letters[1] {
285        -1.0
286    } else {
287        return None;
288    };
289    (value <= most).then_some(sign * value)
290}
291
292/// What a set of fixes' times and positions came to, for the notes.
293#[derive(Default)]
294struct Counts {
295    bad_times: usize,
296    bad_time_example: Option<u64>,
297    bad_latitudes: usize,
298    bad_longitudes: usize,
299    no_fix: usize,
300    next_days: usize,
301    days_before: usize,
302}
303
304/// One fix's time and position, before the clock is made.
305struct Fix {
306    hhmmss: u64,
307    latitude: Option<f64>,
308    longitude: Option<f64>,
309}
310
311/// The clock: each fix's time from the first that is a time of day, `NaN` for one that isn't; and
312/// the zero's `HHMMSS`. Each time takes the day that puts it nearest the one before it: a time more
313/// than 12 hours before the one before is the next day's, and one more than 12 hours after it the
314/// day before's, as a repeated pair of records written across midnight is.
315fn clock(fixes: &[Fix], counts: &mut Counts) -> Option<(Vec<f64>, u64)> {
316    let mut zero: Option<(f64, u64)> = None;
317    let mut day = 0.0;
318    let mut last: Option<f64> = None;
319    let mut time_s = Vec::with_capacity(fixes.len());
320    for fix in fixes {
321        let Some(of_day) = time_of_day(fix.hhmmss) else {
322            counts.bad_times += 1;
323            counts.bad_time_example.get_or_insert(fix.hhmmss);
324            time_s.push(f64::NAN);
325            continue;
326        };
327        if last.is_some_and(|last| of_day + day < last - NEXT_DAY_S) {
328            day += DAY_S;
329            counts.next_days += 1;
330        } else if last.is_some_and(|last| of_day + day > last + NEXT_DAY_S) {
331            day -= DAY_S;
332            counts.days_before += 1;
333        }
334        let at = of_day + day;
335        last = Some(at);
336        let (zero_s, _) = *zero.get_or_insert((at, fix.hhmmss));
337        time_s.push(at - zero_s);
338    }
339    zero.map(|(_, hhmmss)| (time_s, hhmmss))
340}
341
342/// The notes on the clock, the times and the positions.
343fn clock_notes(time_s: &[f64], zero: u64, counts: &Counts, quoted: &str) -> Vec<String> {
344    let mut notes = vec![format!(
345        "the log's clock is the GPS's time of day, UTC, `{quoted}`, in whole seconds. Its zero is \
346         the first record's, {:02}:{:02}:{:02}. The file states no date",
347        zero / 10_000,
348        zero / 100 % 100,
349        zero % 100
350    )];
351    let known: Vec<f64> = time_s.iter().copied().filter(|t| !t.is_nan()).collect();
352    let backs = known.windows(2).filter(|pair| pair[1] < pair[0]).count();
353    let largest = known
354        .windows(2)
355        .map(|pair| pair[0] - pair[1])
356        .fold(0.0, f64::max);
357    let mut sorted = known.clone();
358    sorted.sort_by(f64::total_cmp);
359    let repeats = sorted.windows(2).filter(|pair| pair[0] == pair[1]).count();
360    if backs > 0 {
361        notes.push(format!(
362            "times go back {backs} times from one record to the next, by up to {largest:.0} s. \
363             {repeats} records repeat a time already written. Every record is kept, in the \
364             file's order"
365        ));
366    }
367    if counts.next_days > 0 {
368        notes.push(format!(
369            "times fall more than 12 hours behind the one before {} times, each read as the next \
370             day's",
371            counts.next_days
372        ));
373    }
374    if counts.days_before > 0 {
375        notes.push(format!(
376            "times run more than 12 hours ahead of the one before {} times, each read as the day \
377             before's",
378            counts.days_before
379        ));
380    }
381    if counts.bad_times > 0 {
382        notes.push(format!(
383            "{} records hold a time that isn't a time of day, such as {}: their times are gaps",
384            counts.bad_times,
385            counts.bad_time_example.unwrap_or_default()
386        ));
387    }
388    for (count, what) in [
389        (counts.bad_latitudes, "latitudes"),
390        (counts.bad_longitudes, "longitudes"),
391    ] {
392        if count > 0 {
393            notes.push(format!(
394                "{count} {what} are out of range, so they are gaps. Out of range means minutes \
395                 past 59, decimals without their leading 1, or a hemisphere byte other than its \
396                 two letters. It also means a value beyond ±90° or ±180°"
397            ));
398        }
399    }
400    if counts.no_fix > 0 {
401        notes.push(format!(
402            "{} records have the latitude and longitude both 0, as a receiver writes before it \
403             has a fix: their positions are gaps",
404            counts.no_fix
405        ));
406    }
407    notes
408}
409
410/// A position read: a gap for a coordinate out of range, both for (0, 0).
411fn position(
412    latitude: Option<f64>,
413    longitude: Option<f64>,
414    counts: &mut Counts,
415) -> (Option<f64>, Option<f64>) {
416    counts.bad_latitudes += usize::from(latitude.is_none());
417    counts.bad_longitudes += usize::from(longitude.is_none());
418    if latitude == Some(0.0) && longitude == Some(0.0) {
419        counts.no_fix += 1;
420        (None, None)
421    } else {
422        (latitude, longitude)
423    }
424}
425
426/// The note that the log has no barometer.
427const NO_BAROMETER: &str = "the log has no barometer, so the record's altitude track is empty: \
428                            an RTx's file holds GPS fixes only, and every field is a channel";
429
430/// A channel on the clock.
431fn channel(column: &str, quantity: Quantity, time_s: &[f64], values: Vec<f64>) -> Channel {
432    Channel {
433        column: column.to_owned(),
434        quantity,
435        time_s: time_s.to_vec(),
436        values,
437    }
438}
439
440/// An error naming a byte of an `.rtx` file.
441fn fault(offset: usize, message: impl Into<String>) -> LogError {
442    LogError::Binary {
443        format: RAW,
444        offset,
445        message: message.into(),
446    }
447}
448
449/// A little-endian 32-bit integer at a byte; the caller has checked the length.
450fn int_at(bytes: &[u8], at: usize) -> i32 {
451    let mut four = [0; 4];
452    four.copy_from_slice(&bytes[at..at + 4]);
453    i32::from_le_bytes(four)
454}
455
456/// The bytes a stream must hold from a byte on, or the offset of the first that differs.
457fn expect(bytes: &[u8], at: usize, wanted: &[u8], what: &str) -> Result<usize, LogError> {
458    match wanted
459        .iter()
460        .enumerate()
461        .find(|&(k, b)| bytes.get(at + k) != Some(b))
462    {
463        Some((k, _)) => Err(fault(
464            at + k,
465            format!("the stream isn't the RTx's list here: {what}"),
466        )),
467        None => Ok(at + wanted.len()),
468    }
469}
470
471/// Reads an RTx `.rtx` file.
472///
473/// # Errors
474///
475/// [`LogError::NotThisFormat`] if the bytes don't open as an `.rtx`'s; [`LogError::Binary`] if
476/// the stream isn't laid out as the module docs say, or a slot isn't a word; [`LogError::NoData`]
477/// if the list holds no record, or no record whose time is a time of day.
478pub fn read_raw(bytes: &[u8]) -> Result<FlightLog, LogError> {
479    if !recognizes_raw(bytes) {
480        return Err(LogError::NotThisFormat {
481            format: RAW,
482            message: "the file doesn't open as an MS-NRBF stream of the RTx's `List<Int32>`"
483                .to_owned(),
484        });
485    }
486    let mut at = expect(bytes, 0, SERIALIZATION_HEADER, "the serialization header")?;
487    at = expect(bytes, at, CLASS_RECORD, "the list's class record")?;
488    at = expect(bytes, at, CLASS_NAME, "the list's class name")?;
489    at = expect(bytes, at, MEMBERS, "the list's members")?;
490    if bytes.len() < at + 8 {
491        return Err(fault(bytes.len(), "the file ends inside the list's counts"));
492    }
493    let size_at = at;
494    let size = int_at(bytes, at);
495    at = expect(bytes, at + 8, ARRAY_RECORD, "the items' array record")?;
496    if bytes.len() < at + 5 {
497        return Err(fault(
498            bytes.len(),
499            "the file ends inside the items' array record",
500        ));
501    }
502    let length_at = at;
503    let length = usize::try_from(int_at(bytes, at))
504        .map_err(|_| fault(length_at, "the items' array has a negative length"))?;
505    at = expect(bytes, at + 4, &[INT32], "the items' type, `Int32`")?;
506    let data_at = at;
507    let end = length
508        .checked_mul(4)
509        .and_then(|n| n.checked_add(data_at))
510        .filter(|&end| end < bytes.len())
511        .ok_or_else(|| {
512            fault(
513                length_at,
514                format!("the items' {length} slots run past the file's end"),
515            )
516        })?;
517    if bytes[end] != MESSAGE_END || bytes.len() != end + 1 {
518        return Err(fault(
519            end,
520            "the stream doesn't end with `MessageEnd` straight after the items",
521        ));
522    }
523    let size = usize::try_from(size)
524        .ok()
525        .filter(|&s| s % RECORD_WORDS == 0 && s <= length)
526        .ok_or_else(|| {
527            fault(
528                size_at,
529                format!(
530                    "the list's `_size`, {size}, isn't a whole number of {RECORD_WORDS}-word \
531                     records within its {length} slots"
532                ),
533            )
534        })?;
535    let slot_at = |k: usize| data_at + 4 * k;
536    if let Some(k) = (size..length).find(|&k| int_at(bytes, slot_at(k)) != 0) {
537        return Err(fault(
538            slot_at(k),
539            format!("slot {k}, past the list's `_size` of {size}, isn't 0"),
540        ));
541    }
542    let mut records = Vec::with_capacity(size / RECORD_WORDS);
543    for r in 0..size / RECORD_WORDS {
544        let mut record = [0u16; RECORD_WORDS];
545        for (w, value) in record.iter_mut().enumerate() {
546            let k = r * RECORD_WORDS + w;
547            let raw = int_at(bytes, slot_at(k));
548            *value = word(raw).ok_or_else(|| {
549                fault(
550                    slot_at(k),
551                    format!(
552                        "slot {k}, {raw}, isn't a 16-bit word times {MULTIPLIER} modulo \
553                         {MODULUS}"
554                    ),
555                )
556            })?;
557        }
558        records.push(record);
559    }
560    if records.is_empty() {
561        return Err(LogError::NoData { format: RAW });
562    }
563
564    let mut counts = Counts::default();
565    let fixes: Vec<Fix> = records
566        .iter()
567        .map(|w| {
568            let [north, east] = w[11].to_le_bytes();
569            let pair = |lo: u16, hi: u16| u32::from(hi) << 16 | u32::from(lo);
570            Fix {
571                hhmmss: u64::from(pair(w[0], w[1])),
572                latitude: coordinate(w[2], pair(w[3], w[4]), north, *b"NS", 90.0),
573                longitude: coordinate(w[5], pair(w[6], w[7]), east, *b"EW", 180.0),
574            }
575        })
576        .collect();
577    let (time_s, zero) = clock(&fixes, &mut counts).ok_or(LogError::NoData { format: RAW })?;
578    let (latitudes, longitudes): (Vec<f64>, Vec<f64>) = fixes
579        .iter()
580        .map(|fix| {
581            let (lat, lon) = position(fix.latitude, fix.longitude, &mut counts);
582            (lat.unwrap_or(f64::NAN), lon.unwrap_or(f64::NAN))
583        })
584        .unzip();
585    let field = |w: usize| -> Vec<f64> { records.iter().map(|r| f64::from(r[w])).collect() };
586    let byte = |w: usize, high: bool| -> Vec<f64> {
587        records
588            .iter()
589            .map(|r| f64::from(if high { r[w] >> 8 } else { r[w] & 0xff }))
590            .collect()
591    };
592    let channels = vec![
593        channel("Latitude", Quantity::Latitude, &time_s, latitudes),
594        channel("Longitude", Quantity::Longitude, &time_s, longitudes),
595        channel("Altitude", Quantity::GnssAltitude, &time_s, field(8)),
596        channel("Velocity", Quantity::Unrecognized, &time_s, field(9)),
597        channel("VDOP", Quantity::Number, &time_s, field(10)),
598        channel("NavStatus", Quantity::Number, &time_s, byte(12, false)),
599        channel("Sats", Quantity::Number, &time_s, byte(12, true)),
600        channel("SOG", Quantity::Unrecognized, &time_s, field(13)),
601        channel("COG", Quantity::Angle, &time_s, field(14)),
602        channel("word 15", Quantity::Unrecognized, &time_s, field(15)),
603    ];
604    let mut notes = vec![
605        NO_BAROMETER.to_owned(),
606        format!(
607            "each slot is its 16-bit word times {MULTIPLIER} modulo {MODULUS}, read back by \
608             multiplying by {INVERSE}; the list holds {} records of {RECORD_WORDS} words",
609            records.len()
610        ),
611    ];
612    notes.extend(clock_notes(&time_s, zero, &counts, "UTC Time"));
613    let at_zero = records.iter().filter(|r| r[8] == 0).count();
614    notes.extend(units_notes("Altitude", "Velocity", true, at_zero));
615    notes.push(
616        "`word 15` kept as written, a word the exports leave out: no document names it".to_owned(),
617    );
618    Ok(log(LogFormat::RtxRaw, records.len(), channels, notes))
619}
620
621/// The notes on the fields' units, by the file's column names: the altitude's, the velocity's
622/// and the course's; `raw` for the `.rtx`, whose altitude is in meters, not an export's feet; and
623/// the count of altitudes at 0, the least a word holds, each perhaps a lower one clamped.
624fn units_notes(altitude: &str, velocity: &str, raw: bool, at_zero: usize) -> Vec<String> {
625    let mut notes = vec![
626        if raw {
627            format!(
628                "`{altitude}` read in meters, from the manual and the exports. The values are \
629                 whole meters, above mean sea level as the RTx's manual states its GPS altitude. \
630                 The KML export writes them the same way, and the CSV and text exports convert \
631                 them to feet. A word can't fall below 0"
632            )
633        } else {
634            format!(
635                "`{altitude}` read in feet, from the `.rtx` files beside the sample exports. The \
636                 values are whole feet, the `.rtx`'s meters × 3.280 84 rounded down, read as 0.3048 m each"
637            )
638        },
639        format!(
640            "`{velocity}` and `SOG` kept as written, in units no document states. `SOG` runs \
641             about 3.6 times the speed between fixes, as kilometers an hour would. The \
642             velocity runs far more than the climb rate while it climbs"
643        ),
644        "`COG` read in degrees, from the positions. It is the course over ground, within a few \
645         degrees of the bearing between the fixes either side while the tracker moves"
646            .to_owned(),
647    ];
648    if at_zero > 0 {
649        notes.push(format!(
650            "{at_zero} `{altitude}` values read exactly 0, the least the file can hold. Each \
651             stands for 0 m or lower, as an altitude clamped at 0 would"
652        ));
653    }
654    notes
655}
656
657/// The record every RTx file reads into.
658fn log(format: LogFormat, samples: usize, channels: Vec<Channel>, notes: Vec<String>) -> FlightLog {
659    FlightLog {
660        format,
661        logger: LOGGER.to_owned(),
662        serial_number: None,
663        firmware: None,
664        flight_number: None,
665        stated: Stated {
666            samples: Some(samples),
667            ..Stated::default()
668        },
669        time_s: Vec::new(),
670        altitude_m: Vec::new(),
671        vertical_speed_m_s: None,
672        temperature_k: None,
673        battery_v: None,
674        channels,
675        events: Vec::new(),
676        notes,
677    }
678}
679
680/// An export's coordinate cell: `DDMM.mmmmm` and its hemisphere's letter, or signed degrees and a
681/// degree sign. `Err` for a cell that is neither; `Ok(None)` for one out of range.
682fn coordinate_cell(cell: &str, letters: [u8; 2], most: f64) -> Result<Option<f64>, ()> {
683    if let Some(degrees) = cell.strip_suffix('°') {
684        let value: f64 = degrees.trim().parse().map_err(|_| ())?;
685        if !value.is_finite() {
686            return Err(());
687        }
688        return Ok((value.abs() <= most).then_some(value));
689    }
690    let hemisphere = *cell.as_bytes().last().ok_or(())?;
691    if !letters.contains(&hemisphere) {
692        return Err(());
693    }
694    let number = &cell[..cell.len() - 1];
695    let (whole, decimals) = number.split_once('.').ok_or(())?;
696    if whole.is_empty()
697        || decimals.len() != 5
698        || !whole
699            .bytes()
700            .chain(decimals.bytes())
701            .all(|b| b.is_ascii_digit())
702    {
703        return Err(());
704    }
705    // All digits, so the only failure is a number past a word's 65,535: out of range, a gap.
706    let Ok(degrees_minutes) = whole.parse::<u16>() else {
707        return Ok(None);
708    };
709    let decimals: u32 = decimals.parse().map_err(|_| ())?;
710    Ok(coordinate(
711        degrees_minutes,
712        decimals + DECIMALS_LEAD,
713        hemisphere,
714        letters,
715        most,
716    ))
717}
718
719/// Reads an RTx CSV or text export.
720///
721/// # Errors
722///
723/// [`LogError::NotThisFormat`] if the first line that isn't blank isn't either export's header;
724/// [`LogError::Syntax`] for a row of other than ten cells or a cell that isn't its field's number;
725/// [`LogError::NoData`] if no row holds a time of day.
726pub fn read_export(text: &str) -> Result<FlightLog, LogError> {
727    let mut lines =
728        crate::table::numbered_lines(text).skip_while(|(_, line)| line.trim().is_empty());
729    let Some(format) = lines.next().and_then(|(_, line)| export_header(line)) else {
730        return Err(LogError::NotThisFormat {
731            format: CSV,
732            message: "the first line that isn't blank isn't the RTx CSV's or text file's header"
733                .to_owned(),
734        });
735    };
736    let (name, separator, names) = match format {
737        LogFormat::RtxCsv => (CSV, ',', CSV_HEADER),
738        _ => (
739            TEXT,
740            '\t',
741            [
742                "UTC", "Lat", "Lon", "Alt", "Vel", "VDOP", "Nav", "Sats", "SOG", "COG",
743            ],
744        ),
745    };
746    let mut fixes = Vec::new();
747    let mut rows: Vec<[f64; 7]> = Vec::new();
748    for (line_number, line) in lines {
749        if line.trim().is_empty() {
750            continue;
751        }
752        let syntax = |message: String| LogError::Syntax {
753            format: name,
754            line: line_number,
755            message,
756        };
757        let cells: Vec<&str> = line.split(separator).map(str::trim).collect();
758        let [time, latitude, longitude, rest @ ..] = &cells[..] else {
759            return Err(syntax(format!("{} cells, not 10", cells.len())));
760        };
761        if cells.len() != 10 {
762            return Err(syntax(format!("{} cells, not 10", cells.len())));
763        }
764        let not_a =
765            |k: usize, cell: &str| syntax(format!("`{}` {} isn't one", names[k], short(cell)));
766        if time.is_empty() || time.len() > 10 || !time.bytes().all(|b| b.is_ascii_digit()) {
767            return Err(not_a(0, time));
768        }
769        let hhmmss: u64 = time.parse().map_err(|_| not_a(0, time))?;
770        let latitude = coordinate_cell(latitude, *b"NS", 90.0).map_err(|()| not_a(1, latitude))?;
771        let longitude =
772            coordinate_cell(longitude, *b"EW", 180.0).map_err(|()| not_a(2, longitude))?;
773        let mut values = [0.0; 7];
774        for (k, (value, cell)) in values.iter_mut().zip(rest).enumerate() {
775            *value = crate::table::number(cell, false).ok_or_else(|| not_a(k + 3, cell))?;
776        }
777        fixes.push(Fix {
778            hhmmss,
779            latitude,
780            longitude,
781        });
782        rows.push(values);
783    }
784    if fixes.is_empty() {
785        return Err(LogError::NoData { format: name });
786    }
787    let mut counts = Counts::default();
788    let (time_s, zero) = clock(&fixes, &mut counts).ok_or(LogError::NoData { format: name })?;
789    let (latitudes, longitudes): (Vec<f64>, Vec<f64>) = fixes
790        .iter()
791        .map(|fix| {
792            let (lat, lon) = position(fix.latitude, fix.longitude, &mut counts);
793            (lat.unwrap_or(f64::NAN), lon.unwrap_or(f64::NAN))
794        })
795        .unzip();
796    let column = |k: usize, scale: f64| -> Vec<f64> { rows.iter().map(|r| r[k] * scale).collect() };
797    let channels = vec![
798        channel(names[1], Quantity::Latitude, &time_s, latitudes),
799        channel(names[2], Quantity::Longitude, &time_s, longitudes),
800        channel(names[3], Quantity::GnssAltitude, &time_s, column(0, FOOT_M)),
801        channel(names[4], Quantity::Unrecognized, &time_s, column(1, 1.0)),
802        channel(names[5], Quantity::Number, &time_s, column(2, 1.0)),
803        channel(names[6], Quantity::Number, &time_s, column(3, 1.0)),
804        channel(names[7], Quantity::Number, &time_s, column(4, 1.0)),
805        channel(names[8], Quantity::Unrecognized, &time_s, column(5, 1.0)),
806        channel(names[9], Quantity::Angle, &time_s, column(6, 1.0)),
807    ];
808    let mut notes = Notes::default();
809    notes.push(NO_BAROMETER.to_owned());
810    for note in clock_notes(&time_s, zero, &counts, names[0]) {
811        notes.push(note);
812    }
813    let at_zero = rows.iter().filter(|r| r[0] == 0.0).count();
814    for note in units_notes(names[3], names[4], false, at_zero) {
815        notes.push(note);
816    }
817    notes.push(
818        "the software writes every record of its `.rtx` file but the last: the `.rtx` holds one \
819         more fix"
820            .to_owned(),
821    );
822    Ok(log(format, fixes.len(), channels, notes.into_vec()))
823}
824
825#[cfg(test)]
826mod tests {
827    use super::*;
828
829    #[test]
830    fn the_inverse_undoes_the_multiplier() {
831        assert_eq!(MULTIPLIER * INVERSE % MODULUS, 1);
832        assert_eq!(MULTIPLIER, 3u64.pow(18));
833        for value in [0u16, 1, 2, 255, 22_350, 65_535] {
834            assert_eq!(word(slot(value)), Some(value));
835        }
836        // A slot one off a word's reads back far past 65,535.
837        assert_eq!(word(slot(2) + 1), None);
838        assert_eq!(word(-1), None);
839        assert_eq!(word(1_000_000_000), None);
840    }
841
842    #[test]
843    fn times_of_day_are_read_strictly() {
844        assert_eq!(time_of_day(170_903), Some(61_743.0));
845        assert_eq!(time_of_day(0), Some(0.0));
846        assert_eq!(time_of_day(235_960), Some(86_400.0));
847        assert_eq!(time_of_day(240_000), None);
848        assert_eq!(time_of_day(126_000), None);
849        assert_eq!(time_of_day(120_061), None);
850        assert_eq!(time_of_day(4_278_331_365), None);
851    }
852
853    #[test]
854    fn a_coordinate_takes_its_leading_one_and_hemisphere() {
855        let north = coordinate(2755, 193_418, b'N', *b"NS", 90.0).unwrap_or(f64::NAN);
856        assert!((north - (27.0 + 55.934_18 / 60.0)).abs() < 1e-12);
857        let west = coordinate(8042, 100_589, b'W', *b"EW", 180.0).unwrap_or(f64::NAN);
858        assert!((west + (80.0 + 42.005_89 / 60.0)).abs() < 1e-12);
859        assert_eq!(coordinate(2760, 100_000, b'N', *b"NS", 90.0), None);
860        assert_eq!(coordinate(2755, 99_999, b'N', *b"NS", 90.0), None);
861        assert_eq!(coordinate(2755, 200_000, b'N', *b"NS", 90.0), None);
862        assert_eq!(coordinate(2755, 100_000, b'X', *b"NS", 90.0), None);
863        assert_eq!(coordinate(9100, 100_000, b'N', *b"NS", 90.0), None);
864    }
865}