Spatial Epi Public Health GIS

Spatial Data Quality Monitoring

This guide is part of the Surveillance Pipeline Automation & Spatial Data Quality section. It covers the checks that sit between stages and decide whether a run’s output is trustworthy — the part of automation that turns silent failure into loud failure.

Concept & Epidemiological Alignment

Data quality monitoring for spatial surveillance has a specific target: detecting that the geography of the data has changed for a reason other than disease. A case count that shifts because transmission shifted is the signal. A case count that shifts because a geocoder update moved a thousand addresses, a boundary file gained a vintage, or an upstream system started sending ZIP centroids instead of street addresses is noise that looks exactly like signal to every downstream statistic.

Four kinds of check cover almost everything that goes wrong.

Validity checks assert structural properties that must hold: geometries are valid, coordinates fall inside the jurisdiction, identifiers resolve against the current boundary vintage, required fields are present. These are binary and cheap, and they catch the gross failures.

Distribution checks compare this run’s data to recent history: the share of cases geocoded at each match level, the fraction falling in each county, the distribution of distances between consecutive vintages of the same address. They catch the subtle failures — a feed that is well-formed but different.

Reconciliation checks count the same quantity two ways and require agreement: cases by tract summed to county must equal cases by county, and both must equal the record count less the explicitly excluded. They catch join errors, which are the most common source of wrong surveillance output.

Coverage checks confirm that every area that should have data has data, or an explicit reason it does not. They catch the failure mode where a jurisdiction’s feed silently stops.

The epidemiological discipline required is to set thresholds from the history of the data rather than from intuition, and to treat an alarm from a quality check with the same seriousness as an alarm from a cluster statistic. Both are claims that something changed.

A useful way to frame the whole topic is that every quality check is a test of an assumption the analysis makes silently. A cluster statistic assumes the geocoder behaved the same way last week as this week; a rate map assumes numerator and denominator refer to the same areas; a trend assumes the reporting jurisdictions did not change. Writing those assumptions down, one per check, is usually the fastest way to discover which ones nobody was testing.

Check Families and What Each One Catches

Family Example assertion Catches Should it halt?
Validity All geometries valid, all points inside the state Corrupt uploads, coordinate swaps Yes
Schema Expected columns, types and ranges present Upstream format changes Yes
Distribution Match-type shares within historical band Geocoder regressions, feed changes Warn, halt if extreme
Reconciliation Tract sum equals county sum equals record count Join errors, dropped records Yes
Coverage Every reporting jurisdiction present A silent feed outage Warn
Stability Re-geocoded addresses moved less than a tolerance Reference data changes Warn
Where Each Family of Check Sits A horizontal pipeline of four stages: ingest, geocode, aggregate and publish. Between ingest and geocode sit schema and validity checks, catching malformed or structurally broken input. Between geocode and aggregate sit distribution and stability checks, catching a geocoder or reference data change that moves cases without errors. Between aggregate and publish sit reconciliation and coverage checks, catching dropped records, failed joins and missing jurisdictions. Each check is marked as halting or warning, with reconciliation drawn as the strongest single gate. Each boundary between stages has its own characteristic failure Ingest Geocode Aggregate Publish Schema and validity malformed input, swapped coordinates halts Distribution, stability geocoder or reference change, no error raised warns, halts if extreme Reconciliation, coverage dropped records, failed joins, missing feeds halts before publication Reconciliation catches more real defects than any other single check because it is indifferent to where the error came from

Spatial Data Prerequisites

  • A jurisdiction polygon in the pipeline’s working coordinate reference system, for containment checks.
  • The current boundary vintage for every geographic level the pipeline aggregates to, with a nesting relationship that makes reconciliation possible.
  • A history of each monitored metric — at least several months of nightly values — from which thresholds are derived.
  • Stable record keys, so the same record can be compared across runs and across geocoder vintages.
  • An explicit exclusions ledger, recording every record deliberately dropped and why, so reconciliation can account for it.

Production Implementation

The routines below implement one check from each family and a common result structure, so a run accumulates a list of outcomes that feeds both the halt decision and the scorecard.

# Spatial data quality checks with a common result structure.
# Pinned: geopandas==0.14.4, shapely==2.0.4, pandas==2.2.2, numpy==1.26.4, scipy==1.13.1
import logging
from dataclasses import dataclass, asdict

import numpy as np
import pandas as pd
import geopandas as gpd
from scipy.stats import chi2_contingency

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("dq.checks")


@dataclass
class CheckResult:
    name: str
    family: str
    passed: bool
    severity: str           # "halt" | "warn"
    metric: float
    threshold: float
    detail: str             # counts only — never a record or a coordinate


def check_validity(gdf: gpd.GeoDataFrame, jurisdiction: gpd.GeoDataFrame) -> list[CheckResult]:
    invalid = int((~gdf.geometry.is_valid).sum())
    outside = int((~gdf.geometry.within(jurisdiction.union_all())).sum())
    empty = int(gdf.geometry.is_empty.sum())
    return [
        CheckResult("geometry_valid", "validity", invalid == 0, "halt",
                    invalid, 0, f"{invalid} invalid of {len(gdf)}"),
        CheckResult("inside_jurisdiction", "validity", outside <= 0.001 * len(gdf), "halt",
                    outside, 0.001 * len(gdf), f"{outside} outside of {len(gdf)}"),
        CheckResult("no_empty_geometry", "validity", empty == 0, "halt",
                    empty, 0, f"{empty} empty"),
    ]


def check_match_distribution(today: pd.Series, history: pd.DataFrame,
                             alpha: float = 0.001) -> CheckResult:
    """Has the mix of geocode match types shifted beyond historical variation?"""
    base = history.sum()
    table = pd.concat([today.reindex(base.index).fillna(0), base], axis=1).T.values
    chi2, p, *_ = chi2_contingency(table + 0.5)
    return CheckResult("match_type_distribution", "distribution", p >= alpha, "warn",
                       float(p), alpha,
                       f"chi2={chi2:.1f} p={p:.2e} over {int(today.sum())} records")


def check_reconciliation(fine: pd.DataFrame, coarse: pd.DataFrame, n_records: int,
                         n_excluded: int, fine_parent: str = "county_geoid",
                         value: str = "cases") -> CheckResult:
    """Fine-level sums must equal coarse-level totals must equal records less exclusions."""
    rolled = fine.groupby(fine_parent)[value].sum()
    coarse_s = coarse.set_index(fine_parent)[value]
    diff = (rolled.reindex(coarse_s.index).fillna(0) - coarse_s).abs().sum()
    expected = n_records - n_excluded
    total_gap = abs(int(coarse_s.sum()) - expected)
    ok = diff == 0 and total_gap == 0
    return CheckResult("count_reconciliation", "reconciliation", ok, "halt",
                       float(diff + total_gap), 0,
                       f"fine-vs-coarse gap={int(diff)} coarse-vs-records gap={total_gap}")


def check_coverage(present: set, expected: set) -> CheckResult:
    missing = sorted(expected - present)
    return CheckResult("jurisdiction_coverage", "coverage", not missing, "warn",
                       len(missing), 0, f"{len(missing)} expected jurisdictions absent")


def decide(results: list[CheckResult]) -> dict:
    """Halt if any halting check failed; otherwise continue with warnings recorded."""
    halts = [r for r in results if not r.passed and r.severity == "halt"]
    warns = [r for r in results if not r.passed and r.severity == "warn"]
    for r in results:
        (log.error if (not r.passed and r.severity == "halt") else
         log.warning if not r.passed else log.info)(
            "%-26s %-14s %s  %s", r.name, r.family,
            "PASS" if r.passed else "FAIL", r.detail)
    return {"halt": bool(halts), "n_halt": len(halts), "n_warn": len(warns),
            "results": [asdict(r) for r in results]}

The common result structure is what makes the checks useful beyond the halt decision. Every check returns a name, a family, a metric, a threshold and a count-only detail, so the same list drives the pipeline’s control flow, populates the provenance row, feeds the scorecard, and supplies a content-free alert payload — without any check having to know about any of those consumers.

The distribution check uses a contingency test against pooled history rather than a fixed band, which adapts to the size of the day’s batch: a small batch is allowed more variation than a large one, as it should be. The significance level is deliberately strict, because a distribution check that fires weekly is a distribution check nobody reads.

Parameter Selection & Tuning

Derive every threshold from history. Compute the metric over past runs, choose a percentile that would have fired rarely, and record what it would have caught. A threshold set from intuition either fires constantly or never.

Separate halting from warning deliberately. Validity, schema and reconciliation halt because their failure means the output is wrong. Distribution, stability and coverage warn because their failure means the output may be different, and a halted surveillance pipeline provides no surveillance.

Tolerance for containment. A strict “every point inside the jurisdiction” rule fails on legitimate border addresses geocoded a few metres outside. A tiny fractional tolerance, or a small buffer on the jurisdiction polygon, handles that without masking a coordinate swap, which moves points by hundreds of kilometres.

History window for distribution checks. Too short and seasonal variation trips the check; too long and a genuine slow change is absorbed into the baseline. Several months, excluding known incidents, is usually right, and should be re-derived periodically.

Reconciliation tolerance: zero. Counts either reconcile or they do not. A tolerance on reconciliation converts the strongest check into a weak one.

Edge Cases & Failure Modes

Checks that never fail. A check that has passed for a year is either protecting against something that no longer happens or is miscalibrated. Periodically inject a known fault to confirm each check still fires.

Thresholds that drift with the data. Recomputing thresholds from a rolling window lets a slow degradation redefine normal. Anchor at least one threshold to a fixed reference period.

Reconciliation without an exclusions ledger. If records are dropped for good reasons without being recorded, reconciliation fails every night and gets disabled. The ledger is what makes zero tolerance workable.

Coverage checks on jurisdictions with genuinely zero cases. A small county with no cases this week is not a feed outage. Coverage should check that the feed reported, not that it reported cases.

Distribution checks swamped by an outbreak. A genuine large outbreak in one county shifts the county distribution and trips the check. That is acceptable — the warning prompts a look — but the check must not halt, or it would suppress surveillance precisely when it matters most.

Detail strings that leak. A helpful check message quoting the invalid geometry’s coordinates is a disclosure. Details carry counts only.

Proving the Checks Still Work A periodic fault-injection run in which a synthetic defect is introduced for each check family and the expected check is confirmed to fail. Swapped coordinates should trip containment, a dropped county should trip reconciliation and coverage, a batch of centroid-level geocodes should trip the distribution check, and a boundary file with a self-intersection should trip validity. Checks that fail to fire on their injected fault are flagged as miscalibrated. A note records that a quality check which has never failed has never been shown to work. A check that has never failed has never been shown to work injected defect check that must fire latitude and longitude swapped inside_jurisdiction fired one county's records dropped count_reconciliation fired 30% of addresses as ZIP centroids match_type_distribution fired self-intersecting boundary polygon geometry_valid did not fire The last row is a finding: that check was only run on case points, never on boundaries Run the injection suite on a schedule, separately from production

The drifting-threshold failure in that list is the slowest to appear and the hardest to spot afterwards, so it helps to see both kinds of baseline:

A Rolling Baseline Lets Degradation Redefine Normal A quality metric declining slowly over a year. A threshold recomputed from a rolling window follows the decline down and never fires, because each month's baseline already includes the previous month's degradation. A threshold anchored to a fixed reference period stays level, and the metric crosses it after several months, raising a warning. Keeping at least one anchored threshold alongside the rolling ones is what catches slow degradation. Anchor at least one threshold to a fixed reference period quality metric rolling threshold: never fires anchored threshold warning raised here Jan Dec monthly geocode match rate over a year of slow degradation

Compliance & Audit Controls

Every check result belongs in the run’s provenance: name, family, metric, threshold, outcome and count-only detail. That record is the evidence that a published product passed its gates, and its absence is indistinguishable from the gates never having run.

Thresholds are configuration with a history of their own. Record each threshold, the data it was derived from, the date it was set and what it would have caught historically; a threshold changed to make a failing check pass is a decision that needs to be visible to a reviewer.

The exclusions ledger is simultaneously a quality control and a disclosure-relevant artifact, because it lists records by reason. Keep it inside the protected environment, key it by record identifier, and expose only its aggregate counts to the scorecard.

Production Implementation Checklist

Frequently Asked Questions

Which single check should I implement first? Count reconciliation across geographic levels. It is indifferent to the cause of an error — a failed join, a dropped partition, a boundary mismatch — and catches all of them by the same mechanism. It is also the check most likely to reveal errors that have been happening unnoticed.

How do I stop quality alerts from being ignored? Derive thresholds from history so they fire rarely, route warnings to a scorecard rather than to a pager, and reserve interruptions for halting failures. A quality system that alerts nightly will be muted within weeks, after which it provides no protection at all.

Should a distribution shift halt the pipeline? Rarely. A shift can be a data problem or a genuine epidemiological change, and halting on it would suppress surveillance during an outbreak. Warn, surface it prominently, and halt only on shifts so extreme that no epidemiological explanation is plausible.

How often should thresholds be recomputed? Periodically, on a fixed cadence, and never automatically in response to a failure. Recomputing whenever a check fails lets degradation redefine normal. Record each recomputation with the data window it used.

What about checks on the boundary files themselves? They need the same validity and stability checks as case data, run whenever a new vintage is registered. A self-intersecting polygon in a boundary file corrupts every aggregation that uses it, and it enters through exactly the step least likely to be tested.