Spatial Epi Public Health GIS

Building a Data Quality Scorecard for Surveillance Geography

This guide, part of Spatial Data Quality Monitoring, turns the stream of check results into a surface that programme managers, epidemiologists and reporting partners actually look at.

Problem Context & Constraints

A pipeline with good checks and no scorecard has quality information that only the engineers see. The epidemiologist interpreting a cluster alert does not know that the county it sits in had a 61% rooftop match rate this month against a state average of 88%. The programme manager does not know that one laboratory’s reports now arrive four days late. The reporting partner does not know their feed has been truncating street numbers since March.

A scorecard fixes that by making quality a published property of the data rather than a private property of the pipeline. It also changes behaviour: a jurisdiction that can see its own match rate next to its neighbours’ tends to fix its address capture.

Three constraints shape it. It must be built from provenance, not recomputed, so it reports exactly what the pipeline measured and cannot disagree with the gate decisions. It must be per-jurisdiction, because a state-level average hides the county whose feed has failed. And it must be safe to share, which means aggregate rates only — no small counts, no record-level detail — since its whole purpose is to leave the engineering team.

One Row Per Jurisdiction, One Column Per Dimension A scorecard grid with five jurisdictions as rows and four quality dimensions as columns: geocode quality, completeness of required fields, timeliness of reporting, and reconciliation status. Each cell is graded into one of three levels. Most cells are at the good level. One county shows poor geocode quality and poor timeliness together, which points to a single upstream reporting system rather than two unrelated problems. The layout makes that co-occurrence visible at a glance, which a state-level average would conceal entirely. A state average would hide the one county that matters geocode completeness timeliness reconciled Alder County good 94% good 99% good 1.2 d yes Birch County good 91% good 98% fair 2.6 d yes Cedar County poor 61% good 97% poor 4.4 d yes Dogwood County good 89% fair 93% good 1.4 d yes Cedar's two poor cells together point at one reporting system, not two problems Show every dimension side by side; co-occurrence is the diagnostic

Prerequisites

  • python 3.11
  • pandas 2.2.2, numpy 1.26.4
  • Provenance rows with per-run check results, as produced by the monitoring and orchestration guides
  • Per-jurisdiction summaries of match type, required-field completeness and report latency
  • The disclosure policy’s minimum count, since rates on tiny denominators must be withheld

Step-by-Step Solution

Aggregate per-jurisdiction metrics over a reporting period from provenance, grade each against thresholds derived from state-wide history, withhold rates on small denominators, and emit a table and a trend.

# Per-jurisdiction data quality scorecard built from pipeline provenance.
# Pinned: pandas==2.2.2, numpy==1.26.4
import logging

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("dq.scorecard")

MIN_DENOMINATOR = 20        # rates below this many records are withheld, not shown


def jurisdiction_metrics(runs: pd.DataFrame, period_start: str, period_end: str) -> pd.DataFrame:
    """Aggregate nightly per-jurisdiction counts into period rates.

    runs: run_id, as_of, jurisdiction, n_records, n_rooftop_parcel, n_required_complete,
          latency_days_median, reconciled (bool)
    """
    r = runs[(runs["as_of"] >= period_start) & (runs["as_of"] <= period_end)]
    g = r.groupby("jurisdiction").agg(
        n=("n_records", "sum"),
        good_geo=("n_rooftop_parcel", "sum"),
        complete=("n_required_complete", "sum"),
        latency=("latency_days_median", "median"),
        reconciled_share=("reconciled", "mean"),
        nights=("run_id", "nunique"))
    g["geocode_rate"] = g["good_geo"] / g["n"]
    g["completeness_rate"] = g["complete"] / g["n"]
    small = g["n"] < MIN_DENOMINATOR
    g.loc[small, ["geocode_rate", "completeness_rate"]] = np.nan
    log.info("scorecard period %s..%s: %d jurisdictions, %d withheld for small n",
             period_start, period_end, len(g), int(small.sum()))
    return g.reset_index()


def grade(value: float, good: float, fair: float, higher_is_better: bool = True) -> str:
    if pd.isna(value):
        return "withheld"
    if higher_is_better:
        return "good" if value >= good else "fair" if value >= fair else "poor"
    return "good" if value <= good else "fair" if value <= fair else "poor"


def thresholds_from_history(history: pd.DataFrame) -> dict:
    """Grade boundaries at state-wide historical percentiles, not round numbers."""
    return {
        "geocode": (history["geocode_rate"].quantile(0.25),
                    history["geocode_rate"].quantile(0.05)),
        "completeness": (history["completeness_rate"].quantile(0.25),
                         history["completeness_rate"].quantile(0.05)),
        "latency": (history["latency"].quantile(0.75),
                    history["latency"].quantile(0.95)),
    }


def build_scorecard(metrics: pd.DataFrame, th: dict) -> pd.DataFrame:
    sc = metrics[["jurisdiction", "n", "nights"]].copy()
    sc["geocode"] = [grade(v, *th["geocode"]) for v in metrics["geocode_rate"]]
    sc["completeness"] = [grade(v, *th["completeness"]) for v in metrics["completeness_rate"]]
    sc["timeliness"] = [grade(v, *th["latency"], higher_is_better=False)
                        for v in metrics["latency"]]
    sc["reconciled"] = np.where(metrics["reconciled_share"] == 1.0, "good", "poor")
    sc["geocode_rate"] = metrics["geocode_rate"].round(3)
    sc["latency_days"] = metrics["latency"].round(1)
    # Co-occurring poor grades usually share one upstream cause.
    dims = ["geocode", "completeness", "timeliness", "reconciled"]
    sc["n_poor"] = (sc[dims] == "poor").sum(axis=1)
    flagged = sc[sc["n_poor"] >= 2]["jurisdiction"].tolist()
    if flagged:
        log.warning("jurisdictions with two or more poor dimensions: %s", flagged)
    # Bin record counts before sharing: the scorecard leaves the engineering team.
    sc["n"] = pd.cut(sc["n"], [0, 50, 200, 1000, 10_000, np.inf],
                     labels=["<50", "50-199", "200-999", "1k-10k", "10k+"])
    return sc.sort_values(["n_poor", "jurisdiction"], ascending=[False, True])

Building the scorecard from provenance rather than recomputing it is the design constraint that keeps it trustworthy. The numbers on the scorecard are the numbers the pipeline’s gates saw, so a manager reading “Cedar County reconciled every night” knows that the gate passed rather than that someone ran a separate query that might disagree with it.

Binning the record counts before the scorecard is shared is a small disclosure control with a real purpose. A jurisdiction with 23 records in a month is, for a rare condition, a statement about 23 people; a bin of “fewer than 50” answers the question a quality reader actually has — is this rate based on enough data to believe — without publishing the count.

The Trend Line Is What Gets a Feed Fixed A monthly trend of the rooftop or parcel geocode rate for one county, alongside the state median as a reference line. The county tracks the state closely for seven months, then drops sharply in the month its main hospital changed its reporting system, and stays low for four months until a data-exchange correction restores it. The annotation records that the drop was visible on the scorecard in the first month, which is what prompted the reporting partner to investigate their address capture. Visible in month one, fixed in month five state median hospital reporting system changed exchange corrected Jan Jul Dec Cedar County share of rooftop or parcel geocodes, by month

Validation & Edge Cases

Thresholds set as round numbers. An 85% cut-off might sit in the middle of normal variation for one state and far below it for another. Derive the grade boundaries from state-wide history, as the code does.

Jurisdictions with very few records. A county with eight records in a month has a match rate that swings from 50% to 100% by chance. Withhold rather than grade; a “poor” grade on eight records is noise that will be read as a finding.

A dimension graded on a different period. Timeliness measured over the last week and geocode quality over the last month produce a row whose cells describe different times. Use one period for every dimension on a given scorecard.

Grades that never change. A scorecard where every cell is good every month is either describing an unusually clean system or has thresholds too loose to discriminate. Check that the lowest-performing jurisdiction is at least occasionally graded fair.

Gaming the measure. Once a scorecard is shared, a partner under pressure may improve the grade without improving the data — for instance by filling a missing field with a placeholder that passes the completeness check. Pair each completeness measure with a plausibility check on the values, and treat a sudden jump to perfect completeness as worth a look rather than a celebration.

Scorecards compared across different periods. A partner comparing this month’s grade with a screenshot from last year may be comparing against different thresholds, because the grade boundaries are re-derived as history accumulates. Print the thresholds on the scorecard itself and keep the historical versions, so a change in grade can be attributed either to the data or to the boundaries, never ambiguously to both.

Dimensions that matter unequally. Geocode quality at rooftop level barely affects a county-level report and decides a tract-level cluster analysis. Rather than weighting dimensions into a single overall score, which hides this, show the dimensions separately and state beside each which products depend on it.

Sharing with partners who cannot fix anything. A scorecard sent to a jurisdiction without an explanation of what each dimension depends on produces concern and no action. Pair each dimension with the upstream step it reflects.

INFO dq.scorecard scorecard period 2026-08-01..2026-08-31: 58 jurisdictions, 4 withheld for small n
WARNING dq.scorecard jurisdictions with two or more poor dimensions: ['Cedar']

The small-denominator rule is visible in the raw rates, where the counties with the fewest records swing the most from month to month:

Small Denominators Produce Noise That Looks Like Findings Monthly geocode match rates for many jurisdictions plotted against the number of records each contributed. Jurisdictions with many records cluster tightly around the state rate. Jurisdictions with few records scatter from very low to perfect by chance alone, forming a funnel shape. A vertical line marks the minimum denominator below which rates are withheld from the scorecard, removing the part of the funnel where a poor grade would be noise rather than a finding. Withhold the rates in the wide end of the funnel minimum denominator withheld graded records contributed in the month match rate

Compliance Notes

Record the reporting period, the grade thresholds and the history they were derived from, the minimum denominator for showing a rate, the bins used for record counts, and the provenance runs aggregated. The scorecard is intended to be shared outside the engineering team, so it contains rates and bins only; the underlying per-run metrics stay with the provenance store.

Two uses of the scorecard carry governance weight. When a published product is questioned, the scorecard for its period is the fastest evidence of whether the data behind it were sound. And when a reporting partner’s data quality falls, the scorecard is a neutral, consistently computed basis for the conversation — more persuasive than an anecdote, and less likely to be received as an accusation, because every partner sees the same measures applied to everyone.