Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Geodesy: the ellipsoid, coordinates, distance and bearing

HPR Sim models the Earth as a flattened ellipsoid, and this page shows how it converts coordinates and measures distance and bearing between places.

In short

  • What it models: the Earth’s shape, as the WGS 84 ellipsoid (a sphere slightly flattened at the poles). It converts between latitude, longitude and height and Earth-centered x, y, z. It gives the local east, north and up directions, and the distance and bearing between two places.
  • Sources: the NGA’s WGS 84 standard, NGA.STND.0036 (2014). C. F. F. Karney, Geodesics on an ellipsoid of revolution (2011), appendix B. Karney, Algorithms for geodesics (2013), through the geographiclib-rs crate.
  • How well it is validated: for the conversions, the derived ellipsoid values reproduce the standard’s Table 3.5 to its printed digits. Unit tests make random round trips from −10 km (−6.2 mi) to +1000 km (621 mi). They return latitude within 1e-14 rad and height within 2e-8 m. The conversions are not compared with another library, a simulator or a real flight. For distance and bearing, all 500,000 lines of Karney’s published test set match within 15 nanometers (nm, billionths of a meter). Distance is within 11.18 nm, the far point within 14.02 nm, and the bearings within 15 nm of sideways miss (below). CI checks every 500th line and the 21 mirror lines. The whole set is checked where it has been downloaded, and has been measured on macOS.
  • What it leaves out: height above sea level, which needs the geoid, up to about 100 m (328 ft) from the ellipsoid (Frames). HPR Sim has no geoid model. A flight takes that difference at the site as an input. Nothing in a flight uses distance and bearing yet. The landing distance hpr prints is measured on a flat map from the pad’s east and north offsets, and there is no command for geodesics.

Sources

Code: hpr_core::geodesy and hpr_core::geodesic. Conventions: Frames.

  • [NGA] NGA.STND.0036_1.0.0_WGS84, Department of Defense World Geodetic System 1984, Its Definition and Relationships with Local Geodetic Systems, July 8, 2014. Pinned as wgs84-nga-stnd-0036. US government work.
  • [Karney] C. F. F. Karney, Geodesics on an ellipsoid of revolution, arXiv:1102.1215v1, 2011, appendix B. Pinned as karney-2011-geodesics.
  • Karney2013 C. F. F. Karney, Algorithms for geodesics, J. Geodesy 87 (2013) 43–55, arXiv:1109.4448v2. Pinned as karney-2013-algorithms-for-geodesics.
  • GeodTest C. F. F. Karney, Test set for geodesics, doi:10.5281/zenodo.32156, CC0. Pinned as karney-geodtest.

Ellipsoid and coordinates

The ellipsoid, coordinate conversions and the local axes.

WGS 84 ellipsoid

  • The defining parameters ([NGA] Table 3.1) are a=6378137.0m and 1/f=298.257223563.

  • The derived values are b=a(1−f), e2=f(2−f) and E=ae.

  • These reproduce [NGA] Table 3.5 to its printed digits (wgs84_derived_geometry_matches_table_3_5):

    quantityvalue (m, or ratio)
    b6,356,752.3142 m (20,855,486.595 ft)
    e26.694379990141e-3
    E5.2185400842339e5 m

Geodetic to ECEF ([NGA] eqs. 4-14, 4-15)

N=a1−e2sin2φ,X=(N+h)cosφcosλ,Y=(N+h)cosφsinλ,Z=(b2a2N+h)sinφ

The code writes b2/a2 as 1−e2.

ECEF to geodetic ([Karney] appendix B)

This is Vermeille’s closed form, which Karney extended to cover points near the Earth’s center.

Setup. Let R=X2+Y2, x=R/a and y=1−e2Z/a. The geodetic latitude follows from the largest real root κ of (B1):

κ4+2e2κ3−(x2+y2−e4)κ2−2e2y2κ−e4y2=0

Solving for u.

r=x2+y2−e46,S=e4x2y24,d=S(S+2r3)

If d≥0, with T the real cube root and d given the sign of S+r3 (u=0 if T=0):

T=(S+r3±d)1/3,u=r+T+r2T

If d<0:

ψ=ph(−S−r3+i−d),u=r(1+2cosψ3)

Solving for κ. The last line is (B5). When u<0, the code takes v+u=e4y2/(v−u), which avoids cancellation.

v=u2+e4y2,w=((v+u)−y2)e22v,κ=v+u(v+u)+w2+w

Latitude, height and longitude. The first line is (B2); the next two are (B3).

φ=ph(Rκ+e2+iZκ),h=(1−1−e2κ)D2+Z2,D=κRκ+e2,λ=ph(X+iY)

ph is the argument, computed with atan2; λ=0 on the axis.

The closed form fails only in the equatorial plane within ae2 (42.7 km (27 mi)) of the center. There the paper needs its limiting forms (B6)–(B7). The code returns CoreError::Domain instead.

Measured accuracy (property tests, 256 cases each per run):

  • geodetic → ECEF → geodetic: latitude within 1e-14 rad and height within 2e-8 m, from −10 km (−6.2 mi) to +1000 km (621 mi);
  • ECEF → geodetic → ECEF: within 1e-7 m per 6,400 km (3,977 mi) of radius, out to 46,000 km (28,583 mi).

Local ENU axes

ecef_from_enu_rotation(φ, λ) has columns 𝐞ˆ, 𝐧ˆ and 𝐮ˆ (Frames). A test checks that 𝐮ˆ equals the normalized gradient of x2/a2+y2/a2+z2/b2 at the foot point, i.e. the ellipsoid normal.

Distance and bearing: geodesics

How far is the landing from the pad, and in which direction? On a flat map that is Pythagoras. On the Earth it is a geodesic, the shortest path over the ellipsoid’s surface. Its length is the distance, and its direction where it leaves is the bearing.

Geodesists call a bearing an azimuth. Here the two mean the same. hpr_core::geodesic solves the two classic problems (API reference):

  • The inverse (Ellipsoid::geodesic_inverse): from two places, the distance s12 and the azimuths α1 (leaving the first) and α2 (arriving at the second).
  • The direct (Ellipsoid::geodesic_direct): from a place, a bearing α1 and a distance, the latitude and longitude you reach and the azimuth α2 there.

Azimuths are clockwise from true north, in radians from −π to π, so due west is −π/2. For the 0° to 360° a compass uses, take .to_degrees().rem_euclid(360.0). α2 is the direction of travel on arrival, so the bearing back to the start is α2±π. Over a long path it differs from α1±π because meridians converge. Heights are ignored: the path runs on the ellipsoid’s surface, and the direct problem’s end has no height until you give it one.

Method. The code is Karney’s GeographicLib, as georust’s geographiclib-rs 0.2.7 (MIT) ports it. The simulator uses the crate rather than a port of its own, so the code is Karney’s line for line (the geodesics decision record). Karney2013 maps the ellipsoid onto an auxiliary sphere, where a geodesic is a great circle (the sphere’s shortest path). Karney2013 corrects distance and longitude with series in the flattening to sixth order (§2). The inverse finds α1 by Newton’s method (§4), from a starting guess (§5).

Karney states that round-off stays under 15 nm in both problems on WGS 84 (§7, page 10). He adds that up to a flattening of 1/150 the series’ truncation is smaller still (page 9). Past 1/150 the series lose accuracy, so the simulator refuses such an ellipsoid; WGS 84’s flattening is 1/298. The familiar haversine formula treats the Earth as a sphere, which the Earth is not. Vincenty’s ellipsoidal method is less accurate than Karney’s, and its inverse sometimes fails to converge (§7).

Worked example. A pad at 32.9904° N, 106.9750° W and a landing at 33.0000° N, 106.9680° W:

quantityvalue
distance s121,249.614 m (4,099.78 ft)
bearing from the pad α131.567°
azimuth on arrival α231.571°

At this range a flat map gives the same distance if it uses the ellipsoid’s curvature at the middle latitude: 1,249.614 m (4,099.78 ft) again, to under a millimeter. The other way round, 2 km (1.2 mi) from the same pad on a bearing of 60° reaches 32.999415° N, 106.956466° W, heading 60.010°. The geodesic matters over long paths, where a flat map’s error grows. A unit test, the_guides_worked_example, holds these numbers.

Validation. GeodTest is Karney’s set of 500,000 WGS 84 geodesics. They were worked separately from the code, with the series carried to thirtieth order and high-precision arithmetic. Each end is known to 1e-18°, so they check this code’s method and its rounding.

They come in nine kinds of 50,000 or 100,000 lines each, listed in the geodesics report. The kinds are random pairs, nearly antipodal pairs (almost opposite sides of the Earth), short paths and paths near the poles. The rest run nearly along a meridian or the equator, or near a vertex, the point where a geodesic runs due east or west. crates/hpr-core/tests/geodtest.rs solves every line both ways and measures five errors, each held to Karney’s 15 nm on every line:

errorwhat it islargest
inverse distancethe computed s12 against the set’s11.18 nm
inverse landingsolve the direct problem with the inverse’s own α1 and s12, and measure how far it lands from the second place11.26 nm
inverse azimuths × m12each azimuth’s error times the reduced length m12, how far the far end moves sideways per radian the start’s bearing turns: the sideways miss the error stands for8.49 nm
direct end pointthe computed end against the set’s, in Earth-centered coordinates14.02 nm
direct heading × athe angle between the computed and the set’s direction of travel at the end, as directions in space, times the Earth’s radius a13.99 nm

The inverse’s azimuths can’t be checked on the 50,000 “between vertices” lines. There m12 is at most 1e-13 m, so any azimuth error reads as no miss. Their distance and landing are checked.

Mirror lines: two paths of the same length. Suppose the second place’s latitude is exactly the first’s negated (φ2=−φ1) and the two azimuths differ (α1≠α2). Then two geodesics of the same length join the places, one the mirror of the other. The second geodesic has α1 and α2 swapped (GeographicLib’s GeodSolve manual, Multiple solutions). Either answer is right. Where α1=α2, as on the between-vertices lines, the geodesic is unique.

The set has 21 mirror lines once its numbers are read as f64 (the 64-bit floating-point numbers the simulator computes in). They are all nearly antipodal, with m12 under a centimeter, so their azimuths are nearly undetermined. On these lines the test scores the simulator’s answer against whichever pair it is nearer to. That error is within 15 nm on all 21. On 4 the nearer pair is the swapped one.

CI checks every 500th line (1,000 of them) and the 21 mirror lines, which are committed. The whole set is checked where cargo xtask refs fetch has downloaded it, against the committed report. The whole set has been measured only on macOS, by the debug build that wrote the report’s table. A release build there moves three cells by up to 1.83 nm. On any other build the test holds the 15 nm bound without comparing the table.

What it leaves out. Heights: two places at 3,000 m (9,843 ft) are as far apart as the same places at sea level. Only WGS 84 is measured. On any other ellipsoid up to a flattening of 1/150, the accuracy is Karney’s claim, not something this project has measured. A distance of many trips round the Earth carries its own rounding: one step of f64 in the distance. That is at least 14.9 nm past 67,109 km (41,700 mi). Nothing in a flight uses geodesics yet, and hpr has no command for them.

FusionSpace HPRVersion 0.1.0Issued 2026-11-21In preparationAbout HPR Sim