Spatial Epi Public Health GIS

Measuring Geocoder Match-Rate Bias in Surveillance Data

A geocoded case file with a high overall match rate can still be badly biased, because the records that failed are not a random sample of the records that were sent. This guide, part of Geocoding Quality & Address Standardization, gives a reproducible procedure for detecting differential non-match, estimating how far it moves a published rate, and reporting the result so a reader can judge it.

Problem Context & Constraints

The naive check — compute the match rate, compare it to a threshold, proceed — fails because it summarizes a distribution by its mean. Match failure is driven by properties of the address and of the reference layer, and both correlate with characteristics that also drive incidence. New construction is under-represented in address-range files; rural routes lack a matchable street name; multi-unit buildings produce ambiguous parses; and addresses transcribed from a fax at a small clinic are lower quality than those arriving through an electronic feed. Each of those is associated with population characteristics that matter epidemiologically.

The consequence is not a loss of precision but a shift in the estimate. If matching succeeds for 94% of urban cases and 72% of rural ones, then an analysis restricted to matched records has quietly re-weighted the study population toward urban residents. Any rate computed from it is a rate for a population that does not exist, and the direction of the error depends on the direction of the true urban–rural incidence difference — which is usually the very thing the analysis was commissioned to measure.

This cannot be fixed by geocoding harder. It is a measurement problem, and the measurement has three parts: establish the strata that matter, compute the match rate within each, and quantify the effect of the differential on the specific statistic being published.

What a 22-Point Match Differential Does to a Rate Ratio A true rural-to-urban incidence ratio of 1.35 is compared with the ratio computed from matched records only. Because rural cases match at 72 percent while urban cases match at 94 percent, the rural numerator loses proportionally more cases than the urban one, and the observed ratio falls to 1.03 — close enough to one that the analysis would report no rural excess at all. The denominators were unaffected, because they come from the census rather than from the geocoder. The differential does not add noise — it moves the estimate no difference 1.0 1.0 1.4 1.35 true ratio 1.03 matched records only reported as “no rural excess” rural incidence relative to urban

The diagnostic has one prerequisite that decides whether it is possible at all, and it is worth stating before the code:

Where the Stratifier Has to Come From Two candidate stratifiers compared. A geography derived from the geocoded coordinate covers only the matched records, so unmatched records have no stratum and drop out of the table, hiding the differential entirely. A geography recorded at intake covers every submitted record, matched or not, and is the only kind that can support the diagnostic. A stratifier from the geocoder cannot see the records it lost county from the geocoded point 40,417 matched records — have a stratum 1,376 unmatched — no stratum, dropped the differential becomes invisible by construction, not by luck county recorded at intake 40,417 matched records — have a stratum 1,376 unmatched — also have a stratum the differential is measurable this is the only usable option Assert stratum coverage equals the submitted row count before computing anything

Prerequisites

Fix the environment and the inputs so the diagnostic is reproducible:

  • python 3.11
  • pandas 2.2.2, numpy 1.26.4
  • geopandas 1.0.1 for the stratum join
  • A geocoded file produced as in the parent guide, carrying match_type, match_rank and every input row including unmatched ones
  • A stratifying variable available for unmatched records. This is the constraint that determines whether the diagnostic is possible at all: if the only geography you have comes from the geocoder, you cannot stratify the records that failed it. The reporting ZIP code, county of residence as recorded at intake, or the reporting facility are the usual candidates, because they exist independently of the match.

Step-by-Step Solution

The procedure computes the match rate per stratum, tests whether the variation is larger than sampling noise, and then estimates the distortion under an inverse-probability reweighting.

# Detect and quantify differential geocoding non-match.
# Pinned: pandas==2.2.2, numpy==1.26.4, scipy==1.13.1
import logging
import numpy as np
import pandas as pd
from scipy.stats import chi2_contingency

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("geocode.bias")


def match_rate_by_stratum(df: pd.DataFrame, stratum: str, rank_max: int = 1) -> pd.DataFrame:
    """Match rate within each stratum, at a stated precision cut-off.

    rank_max=1 means "rooftop or address-range interpolated" — the tract-safe
    tiers. Running this at rank_max=9 (any match at all) reports a different and
    usually much flatter picture, which is why the cut-off is an argument."""
    d = df.copy()
    d["matched"] = d["match_rank"] <= rank_max
    out = (d.groupby(stratum)
             .agg(n=("matched", "size"), matched=("matched", "sum"))
             .assign(rate=lambda x: x["matched"] / x["n"]))
    out["se"] = np.sqrt(out["rate"] * (1 - out["rate"]) / out["n"])
    return out.sort_values("rate")


def differential_test(df: pd.DataFrame, stratum: str, rank_max: int = 1):
    """Chi-square test of independence between stratum and match success.

    A significant result means the non-match is differential. A non-significant
    result on a small file means the test lacked power, NOT that matching was
    even — so the effect size is reported alongside."""
    d = df.copy()
    d["matched"] = d["match_rank"] <= rank_max
    tab = pd.crosstab(d[stratum], d["matched"])
    chi2, p, dof, _ = chi2_contingency(tab)
    n = tab.to_numpy().sum()
    cramers_v = np.sqrt(chi2 / (n * (min(tab.shape) - 1)))
    log.info("differential non-match: chi2=%.1f dof=%d p=%.2e Cramer's V=%.3f",
             chi2, dof, p, cramers_v)
    return {"chi2": chi2, "p": p, "dof": dof, "cramers_v": cramers_v}


def reweighted_counts(df: pd.DataFrame, stratum: str, rank_max: int = 1) -> pd.DataFrame:
    """Inverse-probability weights that restore the stratum composition.

    Each matched record stands in for 1 / (stratum match rate) cases. This is the
    simplest defensible correction and it assumes non-match is ignorable WITHIN a
    stratum — an assumption that must be stated, not buried."""
    rates = match_rate_by_stratum(df, stratum, rank_max)["rate"]
    d = df.loc[df["match_rank"] <= rank_max].copy()
    d["ipw"] = d[stratum].map(1.0 / rates)
    log.info("weights range %.2f to %.2f across %d strata",
             d["ipw"].min(), d["ipw"].max(), rates.size)
    return d


def distortion_report(df: pd.DataFrame, stratum: str, rank_max: int = 1) -> dict:
    """Naive vs reweighted stratum shares — the number to publish."""
    naive = (df.loc[df["match_rank"] <= rank_max, stratum]
               .value_counts(normalize=True).rename("matched_share"))
    truth = df[stratum].value_counts(normalize=True).rename("submitted_share")
    cmp = pd.concat([truth, naive], axis=1)
    cmp["shift_pp"] = 100 * (cmp["matched_share"] - cmp["submitted_share"])
    worst = cmp["shift_pp"].abs().max()
    log.info("largest composition shift: %.1f percentage points", worst)
    return {"table": cmp, "max_shift_pp": float(worst)}

The three functions answer three separate questions and should be reported together. match_rate_by_stratum says where the failure is concentrated. differential_test says whether the pattern is distinguishable from chance, with Cramér’s V giving the effect size so a non-significant result on a small file is not mistaken for evenness. distortion_report translates the differential into the quantity a reader can act on: how far the analysed sample’s composition has drifted from the submitted sample’s, in percentage points.

Validation & Edge Cases

1. Check the stratifier is independent of the geocoder. The most common error in this diagnostic is stratifying by a geography derived from the match — county from the geocoded point, say. Unmatched records then have no stratum, they fall out of the table, and the differential becomes invisible by construction. Assert coverage explicitly:

INFO strata coverage: 42,007 of 42,007 records have a stratum value
INFO differential non-match: chi2=1184.3 dof=4 p=3.7e-255 Cramer's V=0.168
INFO largest composition shift: 6.4 percentage points

2. Watch the precision cut-off. Running the same file at rank_max=9 — counting any match, including ZIP centroids — often shows a nearly flat rate across strata, because the fallback tiers absorb exactly the records that failed the precise ones. That flatness is not reassurance; it is the differential reappearing as a precision differential rather than a match differential. Report both cut-offs.

3. Small strata destabilise the weights. A stratum with twelve records and a 25% match rate produces a weight of 4.0 attached to three cases, and those three cases will dominate any map they appear in. Collapse strata below a minimum count — 50 is a reasonable floor — and record the collapse rather than letting a weight of 12 through.

4. The correction has an assumption, and it is testable in one direction. Inverse-probability weighting assumes that within a stratum, matched and unmatched cases are exchangeable. That is often untrue: within the rural stratum, the unmatched cases are disproportionately the most remote. If an independent variable is available on unmatched records — age, diagnosis, reporting facility — compare its distribution between matched and unmatched inside each stratum. A large difference means the assumption fails and the weighting understates the bias.

The Composition Shift Is the Number to Publish Paired bars for five strata comparing each stratum's share of the submitted file with its share of the matched subset. Urban established rises from thirty-one percent submitted to thirty-five percent matched, suburban from twenty-six to twenty-eight, urban multi-family holds at eighteen, rural established falls from eighteen to fourteen, and recently built falls from seven to five. The largest single shift is 4.2 percentage points, and the direction is consistent: the analysed sample is more urban than the population it is meant to describe. Submitted composition against matched composition submitted matched 0 20% 40% urban est. +4.2 pp suburban +1.9 pp multi-family 0.0 pp rural est. −4.1 pp recently built −2.0 pp Every shift points the same way — that consistency is what makes it a bias rather than noise

5. A flat table is a result worth publishing. If the match rate really is even across strata, say so and give the numbers. It is the only way a reader can distinguish “we checked and it was fine” from “we did not check”, and those two are currently indistinguishable in most published surveillance work.

The weights the correction produces are worth inspecting directly, because their spread is the differential restated in units a reader can judge:

Inverse-Probability Weights by Stratum Weights derived from each stratum's match rate. Urban established records carry a weight of 1.06, suburban 1.10, urban multi-family 1.20, rural established 1.39 and recently built 1.64. A rural case therefore stands in for 1.39 cases in the corrected estimate. The spread of the weights is a direct summary of how uneven the matching was. Each matched record stands in for this many cases Urban, established 1.06 Suburban, established 1.10 Urban multi-family 1.20 Rural, established 1.39 Recently built 1.64 A weight above about 1.5 means the stratum is being reconstructed rather than observed

Compliance Notes

  • The diagnostic itself is disclosive if published at fine geography. A stratum table at ZIP level with counts under the disclosure threshold is a release like any other and has to clear the same controls described in Privacy-Preserving Spatial Analytics. Report the table at the coarsest stratification that still demonstrates the differential.
  • Record the stratifier and its source. “Rurality” computed three different ways gives three different tables, and the difference between an RUCA-based and a population-density-based definition is large enough to change the conclusion.
  • State the weighting assumption in the output, not the appendix. If reweighted figures are published, the assumption that non-match is ignorable within stratum belongs in the same paragraph as the number, because a reader who does not know it will treat the corrected estimate as unbiased rather than less biased.
  • Version the diagnostic with the geocoding run. Re-running the geocoder against a newer reference layer changes both the match rate and its differential, so the bias report is only valid for the run signature it was computed from.