Geodesy: the ellipsoid, coordinates, distance and bearing
HPR Sim models the Earth as a flattened ellipsoid, and this page shows how it converts coordinates and measures distance and bearing between places.
In short
- What it models: the Earth’s shape, as the WGS 84 ellipsoid (a sphere slightly flattened at the poles). It converts between latitude, longitude and height and Earth-centered x, y, z. It gives the local east, north and up directions, and the distance and bearing between two places.
- Sources: the NGA’s WGS 84 standard, NGA.STND.0036 (2014). C. F. F. Karney, Geodesics on
an ellipsoid of revolution (2011), appendix B. Karney, Algorithms for geodesics (2013),
through the
geographiclib-rscrate. - How well it is validated: for the conversions, the derived ellipsoid values reproduce the standard’s Table 3.5 to its printed digits. Unit tests make random round trips from −10 km (−6.2 mi) to +1000 km (621 mi). They return latitude within 1e-14 rad and height within 2e-8 m. The conversions are not compared with another library, a simulator or a real flight. For distance and bearing, all 500,000 lines of Karney’s published test set match within 15 nanometers (nm, billionths of a meter). Distance is within 11.18 nm, the far point within 14.02 nm, and the bearings within 15 nm of sideways miss (below). CI checks every 500th line and the 21 mirror lines. The whole set is checked where it has been downloaded, and has been measured on macOS.
- What it leaves out: height above sea level, which needs the geoid, up to about 100 m (328 ft) from
the ellipsoid (Frames). HPR Sim has no geoid model.
A flight takes that difference at the site as an input. Nothing in a flight uses distance and
bearing yet. The landing distance
hprprints is measured on a flat map from the pad’s east and north offsets, and there is no command for geodesics.
Sources
Code: hpr_core::geodesy and hpr_core::geodesic. Conventions: Frames.
- [NGA] NGA.STND.0036_1.0.0_WGS84, Department of Defense World Geodetic System 1984, Its
Definition and Relationships with Local Geodetic Systems, July 8, 2014. Pinned as
wgs84-nga-stnd-0036. US government work. - [Karney] C. F. F. Karney, Geodesics on an ellipsoid of revolution, arXiv:1102.1215v1,
2011, appendix B. Pinned as
karney-2011-geodesics. - Karney2013 C. F. F. Karney, Algorithms for geodesics, J. Geodesy 87 (2013) 43–55,
arXiv:1109.4448v2. Pinned as
karney-2013-algorithms-for-geodesics. - GeodTest C. F. F. Karney, Test set for geodesics, doi:10.5281/zenodo.32156, CC0.
Pinned as
karney-geodtest.
Ellipsoid and coordinates
The ellipsoid, coordinate conversions and the local axes.
WGS 84 ellipsoid
-
The defining parameters ([NGA] Table 3.1) are and .
-
The derived values are , and .
-
These reproduce [NGA] Table 3.5 to its printed digits (
wgs84_derived_geometry_matches_table_3_5):quantity value (m, or ratio) 6,356,752.3142 m (20,855,486.595 ft) 6.694379990141e-3 5.2185400842339e5 m
Geodetic to ECEF ([NGA] eqs. 4-14, 4-15)
The code writes as .
ECEF to geodetic ([Karney] appendix B)
This is Vermeille’s closed form, which Karney extended to cover points near the Earth’s center.
Setup. Let , and . The geodetic latitude follows from the largest real root of (B1):
Solving for .
If , with the real cube root and given the sign of ( if ):
If :
Solving for . The last line is (B5). When , the code takes , which avoids cancellation.
Latitude, height and longitude. The first line is (B2); the next two are (B3).
is the argument, computed with atan2; on the axis.
The closed form fails only in the equatorial plane within (42.7 km (27 mi)) of the center. There
the paper needs its limiting forms (B6)–(B7). The code returns CoreError::Domain instead.
Measured accuracy (property tests, 256 cases each per run):
- geodetic → ECEF → geodetic: latitude within 1e-14 rad and height within 2e-8 m, from −10 km (−6.2 mi) to +1000 km (621 mi);
- ECEF → geodetic → ECEF: within 1e-7 m per 6,400 km (3,977 mi) of radius, out to 46,000 km (28,583 mi).
Local ENU axes
ecef_from_enu_rotation(φ, λ) has columns , and (Frames). A test checks that
equals the normalized gradient of at the foot point, i.e. the ellipsoid
normal.
Distance and bearing: geodesics
How far is the landing from the pad, and in which direction? On a flat map that is Pythagoras. On the Earth it is a geodesic, the shortest path over the ellipsoid’s surface. Its length is the distance, and its direction where it leaves is the bearing.
Geodesists call a bearing an azimuth. Here the two mean the
same. hpr_core::geodesic solves the two classic problems
(API reference):
- The inverse (
Ellipsoid::geodesic_inverse): from two places, the distance and the azimuths (leaving the first) and (arriving at the second). - The direct (
Ellipsoid::geodesic_direct): from a place, a bearing and a distance, the latitude and longitude you reach and the azimuth there.
Azimuths are clockwise from true north, in radians from to , so due west is . For the 0°
to 360° a compass uses, take .to_degrees().rem_euclid(360.0). is the direction of travel
on arrival, so the bearing back to the start is . Over a long path it differs from
because meridians converge. Heights are ignored: the path runs on the ellipsoid’s
surface, and the direct problem’s end has no height until you give it one.
Method. The code is Karney’s GeographicLib, as georust’s geographiclib-rs 0.2.7 (MIT) ports
it. The simulator uses the crate rather than a port of its own, so the code is Karney’s line for
line (the geodesics decision record). Karney2013 maps the ellipsoid onto an auxiliary sphere,
where a geodesic is a great circle (the sphere’s shortest path). Karney2013 corrects distance and
longitude with series in the flattening to sixth order (§2). The inverse finds by Newton’s
method (§4), from a starting guess (§5).
Karney states that round-off stays under 15 nm in both problems on WGS 84 (§7, page 10). He adds that up to a flattening of 1/150 the series’ truncation is smaller still (page 9). Past 1/150 the series lose accuracy, so the simulator refuses such an ellipsoid; WGS 84’s flattening is 1/298. The familiar haversine formula treats the Earth as a sphere, which the Earth is not. Vincenty’s ellipsoidal method is less accurate than Karney’s, and its inverse sometimes fails to converge (§7).
Worked example. A pad at 32.9904° N, 106.9750° W and a landing at 33.0000° N, 106.9680° W:
| quantity | value |
|---|---|
| distance | 1,249.614 m (4,099.78 ft) |
| bearing from the pad | 31.567° |
| azimuth on arrival | 31.571° |
At this range a flat map gives the same distance if it uses the ellipsoid’s curvature at the
middle latitude: 1,249.614 m (4,099.78 ft) again, to under a millimeter. The other way round,
2 km (1.2 mi) from the same pad on a bearing of 60° reaches 32.999415° N, 106.956466° W, heading
60.010°. The geodesic matters over long paths, where a flat map’s error grows. A unit test,
the_guides_worked_example, holds these numbers.
Validation. GeodTest is Karney’s set of 500,000 WGS 84 geodesics. They were worked separately from the code, with the series carried to thirtieth order and high-precision arithmetic. Each end is known to 1e-18°, so they check this code’s method and its rounding.
They come in nine kinds of 50,000 or 100,000 lines each, listed in the
geodesics report.
The kinds are random pairs, nearly antipodal pairs (almost opposite sides of the Earth), short
paths and paths near the poles. The rest run nearly along a meridian or the equator, or near a
vertex, the point where a geodesic runs due east or west. crates/hpr-core/tests/geodtest.rs
solves every line both ways and measures five errors, each held to Karney’s 15 nm on every line:
| error | what it is | largest |
|---|---|---|
| inverse distance | the computed against the set’s | 11.18 nm |
| inverse landing | solve the direct problem with the inverse’s own and , and measure how far it lands from the second place | 11.26 nm |
| inverse azimuths × | each azimuth’s error times the reduced length , how far the far end moves sideways per radian the start’s bearing turns: the sideways miss the error stands for | 8.49 nm |
| direct end point | the computed end against the set’s, in Earth-centered coordinates | 14.02 nm |
| direct heading × | the angle between the computed and the set’s direction of travel at the end, as directions in space, times the Earth’s radius | 13.99 nm |
The inverse’s azimuths can’t be checked on the 50,000 “between vertices” lines. There is at most 1e-13 m, so any azimuth error reads as no miss. Their distance and landing are checked.
Mirror lines: two paths of the same length. Suppose the second place’s latitude is exactly
the first’s negated () and the two azimuths differ (). Then two geodesics of the
same length join the places, one the mirror of the other. The second geodesic has and swapped
(GeographicLib’s GeodSolve manual, Multiple solutions). Either answer is right. Where
, as on the between-vertices lines, the geodesic is unique.
The set has 21 mirror lines once its numbers are read as f64 (the 64-bit floating-point numbers
the simulator computes in). They are all nearly antipodal, with under a centimeter, so their azimuths are nearly
undetermined. On these lines the test scores the simulator’s answer against whichever pair it is
nearer to. That error is within 15 nm on all 21. On 4 the nearer pair is the swapped one.
CI checks every 500th line (1,000 of them) and the 21 mirror lines, which are committed. The
whole set is checked where cargo xtask refs fetch has downloaded it, against the committed
report. The whole set has been measured only on macOS, by the debug build that wrote the report’s
table. A release build there moves three cells by up to 1.83 nm. On any other build the test
holds the 15 nm bound without comparing the table.
What it leaves out. Heights: two places at 3,000 m (9,843 ft) are as far apart as the same places at
sea level. Only WGS 84 is measured. On any other ellipsoid up to a flattening of 1/150, the
accuracy is Karney’s claim, not something this project has measured. A distance of many trips
round the Earth carries its own rounding: one step of f64 in the distance. That is at least
14.9 nm past 67,109 km (41,700 mi). Nothing in a flight uses geodesics yet, and hpr has no command for them.