Spatial Epi Public Health GIS

Validating Isochrones Against Observed Travel Times

An isochrone is a model output presented as a fact about the world, and almost nobody checks it. This guide, part of Drive-Time Isochrone Generation, covers assembling observed travel times, scoring the model against them, and calibrating the impedance so the next isochrone is right for a stated reason rather than by default.

Problem Context & Constraints

Routing engines compute travel time from an impedance model: a speed assigned to each road segment, usually from its OpenStreetMap classification, sometimes adjusted by a global factor. That model is a guess about local driving conditions, and its errors are systematic rather than random — a rural highway tagged at 90 km/h that is actually driven at 70 will make every isochrone crossing it too large, everywhere, always.

Three observation sources are practical for validation. Ambulance or non-emergency transport records carry origin, destination and elapsed time and are usually already held by the agency. Staff travel claims are coarser and widely available. Probe data from a commercial provider is the most complete and the least likely to be affordable. All three have selection issues — emergency journeys are driven differently, claims round to the nearest convenient number — and all three are enormously better than no validation.

The constraint that shapes the work is that observations arrive as origin-destination-duration triples, not as isochrones. Validation therefore compares modelled and observed durations for the same journeys, and the isochrone is validated indirectly through the impedance that produces it.

Modelled Against Observed, Before Calibration A scatter of modelled travel time against observed travel time for several hundred journeys, with a line of equality. Urban journeys scatter closely around the line. Rural journeys sit consistently below it, meaning the model predicts shorter times than were observed, by a median of about eighteen percent. The bias is systematic and one-directional, which is the signature of an impedance error rather than of noise. The rural cloud sits below the line, everywhere 0 30 60 modelled (min) equality urban journeys rural journeys — modelled 18% short 0 30 60 observed travel time (min) A one-directional bias is calibratable; scatter around the line is not

Prerequisites

  • At least a few hundred observed journeys with origin, destination, start time and elapsed time
  • The routing engine and impedance profile currently in production
  • python 3.11, pandas 2.2.2, geopandas 1.0.1, numpy 1.26.4
  • A rural/urban classification for stratifying the error

The three observation sources differ in what they cost and in how they are biased, and picking one is picking a bias to adjust for:

Three Sources of Observed Travel Times Three practical sources of observed journeys. Ambulance and transport records are usually already held, give exact durations, and are driven faster than the public drives. Staff travel claims are widely available, round to convenient numbers, and reflect ordinary driving. Commercial probe data is the most complete and the most expensive, and is biased toward vehicles carrying telematics. Each needs a stated adjustment rather than a caveat. Every source is biased; only an unstated bias is a problem Transport records already held, exact times bias: driven faster adjust downward, or restrict to non-emergency journeys Staff travel claims widely available, coarse bias: rounded durations use for bias, not for spread Commercial probe data complete, expensive bias: telematics fleet check the vehicle mix before treating it as the population

Step-by-Step Solution

# Score modelled travel times against observations and fit a calibration factor.
# Pinned: pandas==2.2.2, numpy==1.26.4, geopandas==1.0.1
import logging
import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("iso.validate")


def clean_observations(obs: pd.DataFrame, min_min: float = 3.0,
                       max_min: float = 120.0) -> pd.DataFrame:
    """Drop journeys that cannot be validated.

    Very short trips are dominated by parking and access time the router does not
    model; very long ones usually include a stop. Both inflate the apparent error
    without saying anything about the impedance."""
    n0 = len(obs)
    d = obs.loc[obs["observed_min"].between(min_min, max_min)].copy()
    d = d.loc[d["origin_geom"].notna() & d["dest_geom"].notna()]
    log.info("kept %d of %d observations after cleaning", len(d), n0)
    return d


def score(obs: pd.DataFrame, stratum: str = "rurality") -> pd.DataFrame:
    """Bias and spread by stratum. Bias is the calibratable part."""
    d = obs.copy()
    d["ratio"] = d["modelled_min"] / d["observed_min"]
    d["err_min"] = d["modelled_min"] - d["observed_min"]
    g = d.groupby(stratum).agg(
        n=("ratio", "size"),
        median_ratio=("ratio", "median"),
        mae_min=("err_min", lambda s: float(np.mean(np.abs(s)))),
        p90_abs_err=("err_min", lambda s: float(np.percentile(np.abs(s), 90))),
    )
    log.info("validation by %s:\n%s", stratum, g.to_string())
    return g


def calibration_factors(obs: pd.DataFrame, by: str = "road_class") -> pd.Series:
    """Multiplicative speed adjustment per class that removes the median bias.

    Fitting per road class rather than globally is what makes the calibration
    transferable: a single global factor that fixes rural journeys will make
    urban ones worse."""
    d = obs.copy()
    d["ratio"] = d["modelled_min"] / d["observed_min"]
    f = d.groupby(by)["ratio"].median().rename("speed_multiplier")
    log.info("speed multipliers:\n%s", f.to_string())
    return f


def holdout_check(obs: pd.DataFrame, factors: pd.Series, by: str = "road_class",
                  frac: float = 0.3, seed: int = 42) -> dict:
    """Apply the factors to a held-out subset. Fitting and scoring on the same
    journeys will always look good and means nothing."""
    rng = np.random.default_rng(seed)
    mask = rng.random(len(obs)) < frac
    test = obs.loc[mask].copy()
    test["adj_modelled"] = test["modelled_min"] / test[by].map(factors).fillna(1.0)
    before = float(np.median(test["modelled_min"] / test["observed_min"]))
    after = float(np.median(test["adj_modelled"] / test["observed_min"]))
    log.info("holdout median ratio: %.3f before calibration, %.3f after", before, after)
    return {"n_test": int(mask.sum()), "before": before, "after": after}

Fitting per road class and validating on held-out journeys are the two steps that separate a calibration from a fudge factor. A single global multiplier tuned on all the data will always improve the fit and will usually degrade the urban isochrones it was not meant to touch.

Validation & Edge Cases

1. Stratify by time of day as well as by geography. Congestion is the largest source of within-day variation, and a model calibrated on off-peak ambulance journeys will overstate peak-hour access. Where the observations support it, fit separate factors for peak and off-peak and generate two isochrone sets:

INFO kept 812 of 1,046 observations after cleaning
INFO validation by rurality:
          n  median_ratio  mae_min  p90_abs_err
rural   331         0.821      6.4         13.1
urban   481         0.978      2.2          5.4
INFO holdout median ratio: 0.874 before calibration, 0.996 after

2. Remember what the observations are. Ambulance journeys are driven faster than the public drives; staff claims are rounded; probe data is biased toward vehicles with telematics. Each needs a stated adjustment or, at minimum, a stated caveat, and mixing sources without a source column makes the bias uninterpretable.

3. Do not calibrate away a network error. A systematic overestimate on one corridor may be a missing road closure or a mis-tagged one-way rather than a speed problem, and a multiplier will paper over it while leaving the routes wrong. Inspect the worst-scoring journeys individually before fitting anything.

4. Re-validate after any network refresh. A new OpenStreetMap extract changes both the geometry and the tagging, so the calibration is specific to the extract it was fitted against and should be recorded with its vintage.

5. Report the residual error, not only the calibrated bias. After calibration the median ratio is near one and the spread is unchanged. That spread — a p90 absolute error of thirteen minutes in rural areas — is the honest precision of the isochrone, and it belongs beside any threshold-based access claim.

6. Validate the isochrone polygons, not only the durations. Calibrating the impedance fixes the travel times; it does not confirm that the polygon built from them encloses the right area. Take a handful of validated journeys whose observed duration falls just inside and just outside the threshold, and check that their origins fall inside and outside the generated isochrone respectively. A systematic failure here points at the polygon-building step — the hull or buffer choice — rather than at the impedance.

7. Keep a small permanent validation set. Reserving fifty journeys that are never used for fitting, and re-scoring them after every network refresh or profile change, turns validation from a project into a monitor. A step change in their scores is the earliest available signal that something in the routing stack has moved.

8. Report the calibration’s own uncertainty. The per-class multipliers are estimated from a finite sample, so they have standard errors, and a class represented by twelve journeys should not be given a confidently different multiplier from the default. Where a class is thinly observed, shrink its multiplier toward one and say so.

After calibration the bias is gone and the spread remains, and the spread is the number that constrains any threshold claim:

Residual Error After Calibration Ninetieth-percentile absolute error by stratum after calibration. Urban off-peak is 4 minutes, urban peak 7, rural off-peak 11 and rural peak 16. A thirty-minute catchment computed on rural peak journeys therefore has a boundary uncertain by more than half a standard appointment slot, which is what a threshold-based access claim has to acknowledge. Calibration removes the bias, not the spread Urban, off-peak 4 min Urban, peak 7 min Rural, off-peak 11 min Rural, peak 16 min 90th-percentile absolute error — publish it beside any 30-minute catchment claim

Compliance Notes

  • Record the observation source and its selection, since the calibration inherits whatever bias the source has.
  • Version the impedance profile together with the network extract it was fitted against, and treat a change to either as a new configuration.
  • Publish the residual error by stratum with any isochrone-derived figure, because a 30-minute catchment with a 13-minute rural error is a different claim from one with a 3-minute error.
  • Treat journey records as sensitive. Ambulance and transport records are patient data, and a validation dataset built from them falls under the same controls as any other case-level file.