Spatial Epi Public Health GIS

Validating an Areal Interpolation With Held-Out Points

Interpolation methods are usually chosen on reputation — dasymetric beats population-weighted beats areal — and almost never measured on the data at hand. When point locations exist for any period, the error can be measured exactly rather than assumed. This guide, part of Areal Interpolation & Boundary Harmonization, builds that validation and turns it into a publish-or-refuse decision.

Problem Context & Constraints

The validation works by simulation against ground truth. Take a period for which point-level locations are held, aggregate them to the source geography, run the interpolation to the target geography, and compare against the counts obtained by aggregating the same points directly to the target. The difference is the interpolation error, exactly, with no assumptions.

Two constraints limit what this can tell you. The truth period must resemble the periods the interpolation will actually be applied to; validating on a summer quarter and applying to a winter one imports whatever seasonal geography differs between them. And the error measured is the error of the crosswalk on this population, so it does not transfer to a different study area or a different vintage.

Within those limits the diagnostic is decisive, because it answers the question that actually matters: not “which method is theoretically better” but “how wrong will the numbers I am about to publish be”.

The Validation Loop Held-out case points feed two paths. On the upper path they are aggregated to the source geography and then interpolated to the target geography, producing estimated target counts. On the lower path they are aggregated directly to the target geography, producing observed target counts. The two are compared cell by cell, and the difference is the interpolation error with no assumptions involved. Two paths from the same points; the gap is the error Held-out points one full period aggregate to source ZCTA counts interpolate estimated tract counts aggregate to target observed tract counts error per tract No assumptions enter the lower path, which is what makes it truth rather than a second estimate

Prerequisites

  • Point locations for at least one full reporting period, at rooftop or street-interpolated precision per Geocoding Quality & Address Standardization
  • The source and target geographies and the crosswalk under evaluation
  • python 3.11, geopandas 1.0.1, pandas 2.2.2, numpy 1.26.4
  • Enough cases that per-target counts are not dominated by noise — a few hundred targets with a median count above about five

Step-by-Step Solution

# Score an areal interpolation against directly aggregated held-out points.
# Pinned: geopandas==1.0.1, pandas==2.2.2, numpy==1.26.4
import logging
import numpy as np
import pandas as pd
import geopandas as gpd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("interp.validate")


def observed_target_counts(points: gpd.GeoDataFrame, target: gpd.GeoDataFrame,
                           tgt_id: str = "geoid") -> pd.Series:
    j = gpd.sjoin(points.to_crs(target.crs), target[[tgt_id, "geometry"]],
                  how="inner", predicate="within")
    return j.groupby(tgt_id).size().rename("observed")


def score_interpolation(estimated: pd.Series, observed: pd.Series) -> dict:
    """Absolute and relative error, plus the two summaries that decide publication."""
    df = pd.concat([estimated.rename("est"), observed.rename("obs")], axis=1).fillna(0.0)
    df["err"] = df["est"] - df["obs"]
    df["abs_err"] = df["err"].abs()

    total = df["obs"].sum()
    # MAPE is unusable when many targets hold zero cases, so report a total-normalised
    # error alongside it: the share of all cases that ended up in the wrong target.
    misplaced = df["abs_err"].sum() / (2 * total) if total else np.nan
    res = {
        "n_targets": int(len(df)),
        "mae": float(df["abs_err"].mean()),
        "rmse": float(np.sqrt((df["err"] ** 2).mean())),
        "share_misplaced": float(misplaced),
        "max_abs_err": float(df["abs_err"].max()),
        "worst_target": df["abs_err"].idxmax(),
        "corr": float(df["est"].corr(df["obs"])),
    }
    log.info("interp score: MAE %.2f, RMSE %.2f, %.1f%% of cases misplaced, r=%.3f",
             res["mae"], res["rmse"], 100 * res["share_misplaced"], res["corr"])
    return res


def compare_methods(points, source, target, crosswalks: dict, tgt_id="geoid") -> pd.DataFrame:
    """Score several candidate crosswalks against the same truth."""
    obs = observed_target_counts(points, target, tgt_id)
    src_counts = (gpd.sjoin(points.to_crs(source.crs), source[["src_id", "geometry"]],
                            how="inner", predicate="within")
                  .groupby("src_id").size().rename("cases").reset_index())
    rows = []
    for name, cw in crosswalks.items():
        m = cw.merge(src_counts, on="src_id", how="left")
        m["cases"] = m["cases"].fillna(0.0) * m["frac"]
        est = m.groupby(tgt_id)["cases"].sum()
        rows.append({"method": name, **score_interpolation(est, obs)})
    return pd.DataFrame(rows).sort_values("share_misplaced")

share_misplaced is the number to lead with. It is the fraction of all cases that the interpolation put in the wrong target zone, it is bounded between zero and one, it is interpretable without knowing the case volume, and unlike a percentage error it does not explode on zero-count targets.

Validation & Edge Cases

1. Compare against the null, not only against each other. Include a crosswalk that assigns every source zone’s cases entirely to its largest-overlap target. If a sophisticated method beats that by two percentage points of misplacement, the sophistication is not buying much:

INFO interp score: MAE 2.41, RMSE 4.02, 18.9% of cases misplaced, r=0.912   [area weighting]
INFO interp score: MAE 1.28, RMSE 2.11, 9.4% of cases misplaced, r=0.968    [population weighting]
INFO interp score: MAE 1.09, RMSE 1.84, 7.8% of cases misplaced, r=0.975    [dasymetric]
INFO interp score: MAE 3.66, RMSE 6.20, 27.4% of cases misplaced, r=0.848   [largest-overlap null]

2. Look at the worst target, not just the average. An overall 8% misplacement can conceal one tract whose count is wrong by a factor of three, and that tract will be the one somebody asks about. Report max_abs_err and identify the target.

3. Validate on the statistic, not only on the counts. If the interpolated counts feed a hotspot detection, run the detection on both the estimated and observed tract counts and compare the flagged sets directly. Agreement on counts does not guarantee agreement on which tracts are flagged, because the flagging is a threshold on a derived quantity.

4. Beware validating on the period that built the weights. If the population layer used for weighting was derived from the same period’s data, the validation is partly circular. Use an independent population source, and say which.

Measured Misplacement, Not Assumed Ranking Share of cases assigned to the wrong target zone by four approaches on the same held-out period. The largest-overlap null misplaces 27.4 percent, area weighting 18.9 percent, population weighting 9.4 percent and dasymetric 7.8 percent. The gap between the null and area weighting is larger than the gap between population weighting and dasymetric, so most of the benefit comes from the first step up, not the last. Most of the benefit is in the first step up Largest-overlap null 27.4% Area weighting 18.9% Population weighting 9.4% Dasymetric 7.8% 0% 15% 30% share of cases assigned to the wrong target zone If your best method cannot beat the null by much, publish on the source geography

5. Set the refusal threshold before running. A defensible rule is that a transfer misplacing more than a stated share of cases — ten percent is a common choice — is not published at target resolution. Setting it in advance prevents the threshold from being adjusted to accommodate a result.

6. Validate the statistic, not only the counts, when the analysis is a map. Two crosswalks can misplace a similar share of cases and produce very different maps, because what matters for a choropleth is whether the misplacement is spatially systematic. Compute the rank correlation between estimated and observed target values, and map the signed error: a random scatter is tolerable, a coherent north-south gradient in the error is not.

7. Repeat the validation on a second period before trusting it. A single held-out period can be unrepresentative for reasons that have nothing to do with the crosswalk — an unusual outbreak, a reporting change, a seasonal shift in where cases arise. Two periods agreeing is much stronger evidence than one, and the second costs only the routing already built.

8. Report the validation even when it is favourable. A stated misplacement share of four percent is a claim a reviewer can check and a reader can weigh. Its absence is usually read, correctly, as meaning the check was not run.

9. Use the validation to choose the publication geography, not only the method. If no crosswalk clears the threshold at tract level but all of them clear it at ZCTA level, the finding is that the data supports ZCTA-level publication. That is a more useful answer than a marginal tract-level map, and the validation is what makes it defensible rather than a retreat.

The summary share hides the property that matters most for a mapped result, so plot the error as well as reporting it:

Signed Error Mapped, Not Just Summarised Two error maps for the same overall misplacement share. In the first, the signed error scatters randomly across the study area, which is tolerable because it will not bias a spatial statistic. In the second, the errors form a coherent gradient with positive error in the north and negative in the south, so the interpolation has introduced a spatial trend that was not in the data. The summary statistic is identical in both. Same 9% misplacement, two very different consequences random error — tolerable systematic gradient — not Map the signed error — a coherent pattern is a finding the summary statistic cannot show

Compliance Notes

  • Publish the measured misplacement, not the method name. “Population-weighted interpolation” tells a reader nothing about accuracy; “9.4% of cases assigned to a different tract in validation” tells them everything they need.
  • State the validation period and its representativeness. An error measured on one quarter is evidence about similar quarters and nothing more.
  • Record the refusal threshold and whether it was met, including for transfers that were performed and then withheld.
  • Treat the held-out point set as restricted data. It is the same case-level data the interpolation exists to avoid publishing, and the validation must run inside the same controls described in Privacy-Preserving Spatial Analytics.