Spatial Epi Public Health GIS

Calibrating Alert Thresholds Against Historical Signals

This guide, part of Cluster Detection Power & Benchmarking, turns a program’s own history into the evidence for where its alert threshold should sit.

Problem Context & Constraints

Every automated cluster detection system ends in a threshold. A prospective space-time scan raises an alert when a cluster’s recurrence interval exceeds some number of days; a CUSUM raises one when the cumulative sum crosses a decision limit; a Local Moran’s I workflow raises one when an adjusted p-value falls below a cut-off. That number decides two things the program cares about directly: how often an epidemiologist is interrupted, and how many real events go unnoticed until someone else reports them.

Thresholds are usually inherited — a recurrence interval of 365 days because the software documentation suggested it, a p-value of 0.05 because it is conventional. Neither reflects the program’s data, its disease mix or its capacity to investigate. Synthetic benchmarks help, but they only measure performance on outbreaks someone imagined. The program’s own history measures performance on outbreaks that happened.

Calibrating against history needs three things. A replay — the detection method run day by day over past data, using only the data that would have been available on each day. Labels — which past alerts corresponded to real events, taken from investigation records, outbreak reports and line lists. And a capacity constraint — the number of alerts per week the team can actually investigate, which sets the ceiling the threshold must respect.

Replay, Label, Calibrate Four stages left to right. Replay runs the detection method day by day over past years using only data available on each day, producing a list of candidate signals with their scores. Label matches each signal to investigation records to mark it as a real event or a false alarm, and lists real events the method never signalled. Sweep evaluates many thresholds, counting alerts per week and events detected with their delay. Choose picks the threshold that detects the most events within the team's weekly investigation capacity. History is the benchmark the program already owns 1. Replay day by day as-of data only keep every score 2. Label match to investigations real or false alarm list missed events 3. Sweep many thresholds alerts per week events found, delay 4. Choose most events found within capacity document the trade recalibrate yearly or whenever the data feed or case definitions change

Prerequisites

  • python 3.11
  • numpy 1.26.4, pandas 2.2.2
  • At least two, ideally five, years of surveillance data with report dates, so the replay can reconstruct what was known each day
  • Investigation records or an outbreak log with dates, locations and a determination of whether each event was real
  • The detection method already implemented, as in Running a Prospective Space-Time Scan for Early Warning
  • An agreed weekly investigation capacity from the epidemiology team

Step-by-Step Solution

Replay once with no threshold, keeping every candidate signal and its score. The threshold sweep then runs on that table in seconds, rather than re-running detection for every threshold.

# Calibrate a cluster alert threshold against labelled historical signals.
# Pinned: numpy==1.26.4, pandas==2.2.2
import logging

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("calibrate")

MATCH_DAYS = 14        # a signal matches an event if within this many days of onset
MATCH_KM = 25.0        # and within this distance of the event centroid


def replay(detect, cases: pd.DataFrame, start: str, end: str) -> pd.DataFrame:
    """Run detection as of each day, using only cases reported by that day."""
    out = []
    for day in pd.date_range(start, end, freq="D"):
        known = cases[cases["report_date"] <= day]
        for sig in detect(known, as_of=day):
            out.append({"as_of": day, **sig})   # sig has cluster_id, x, y, score
    signals = pd.DataFrame(out)
    log.info("replayed %s to %s: %d candidate signals", start, end, len(signals))
    return signals


def label(signals: pd.DataFrame, events: pd.DataFrame) -> tuple[pd.DataFrame, pd.DataFrame]:
    """Mark each signal real or false and record detection delay per event."""
    s = signals.copy()
    s["event_id"] = pd.NA
    for _, ev in events.iterrows():
        near = (np.hypot(s["x"] - ev["x"], s["y"] - ev["y"]) / 1000 <= MATCH_KM)
        timely = (s["as_of"] - ev["onset"]).dt.days.between(-3, MATCH_DAYS)
        s.loc[near & timely & s["event_id"].isna(), "event_id"] = ev["event_id"]
    s["real"] = s["event_id"].notna()
    return s, events


def sweep(labelled: pd.DataFrame, events: pd.DataFrame, thresholds, weeks: float) -> pd.DataFrame:
    """For each threshold: alerts per week, events found, median delay."""
    rows = []
    for t in thresholds:
        fired = labelled[labelled["score"] >= t]
        first = fired[fired["real"]].groupby("event_id")["as_of"].min()
        delay = (first - events.set_index("event_id").loc[first.index, "report_first"]).dt.days
        rows.append({
            "threshold": t,
            "alerts_per_week": fired["cluster_id"].nunique() / weeks,
            "events_found": len(first),
            "events_total": len(events),
            "median_delay_days": float(delay.median()) if len(delay) else np.nan,
            "ppv": fired["real"].mean() if len(fired) else np.nan,
        })
    return pd.DataFrame(rows)


def choose(table: pd.DataFrame, capacity_per_week: float) -> pd.Series:
    """Most events found within capacity; ties broken by shorter delay."""
    ok = table[table["alerts_per_week"] <= capacity_per_week]
    if ok.empty:
        raise RuntimeError("no threshold fits capacity; review method or capacity")
    best = ok.sort_values(["events_found", "median_delay_days"],
                          ascending=[False, True]).iloc[0]
    log.info("chosen threshold %.3g: %.1f alerts/week, %d/%d events, median delay %.0f days",
             best.threshold, best.alerts_per_week, best.events_found,
             best.events_total, best.median_delay_days)
    return best

The replay must be honest about timing. Using final, cleaned data rather than the data as it stood on each day makes every threshold look better than it would have performed, because late-reported cases fill in outbreaks that were thin at the time. If historical report dates are unavailable, the replay is at best an upper bound and should be labelled that way.

Detection delay is measured from the first report of the event through any channel — a clinician call, a laboratory notification — not from onset. That is the delay that matters: an automated alert that fires after the phone call has added nothing.

Where Capacity Meets Detection A chart with threshold strictness on the horizontal axis. Alerts per week falls steeply as the threshold tightens. Events found falls slowly at first and then steeply. A horizontal capacity line at four alerts per week crosses the alerts curve at a moderate threshold; that threshold is chosen, keeping most historical events detected. Looser thresholds exceed capacity, and stricter thresholds lose events without much workload benefit. Choose the loosest threshold the team can staff threshold strictness alerts per week events found capacity: 4 per week chosen stricter than this loses events; looser than this outruns the team

Validation & Edge Cases

Incomplete event labels. Investigation records capture the events someone noticed. An event nobody noticed does not appear, so a signal that matched it is counted as a false alarm. Review a sample of “false” signals manually each calibration cycle; the ones that turn out to be real are both label corrections and evidence of the system’s value.

Matching windows that are too generous. A 50 km, 30-day match window will match almost any signal to some event and inflate the positive predictive value. Set the windows from the method’s spatial and temporal resolution, and report how many matches depend on the outer edge of the window.

Few events. A program with eight labelled outbreaks in five years cannot calibrate a threshold precisely; the events-found curve moves in steps of one eighth. Combine historical calibration with synthetic power estimates from Simulating Outbreaks to Measure Detection Power, and present the history as a check rather than the sole basis.

Repeated alerts for one event. A prospective scan re-signals the same cluster on consecutive days. Count alerts as the team experiences them — one new alert per cluster, with follow-ups suppressed — or the workload figure will be several times the real burden.

Score definitions that drift between versions. A recurrence interval computed by one release of the scan software and a p-value from another are not on the same scale, and even the same method changes scores when its baseline window or maximum cluster size changes. Treat the threshold as belonging to one exact configuration. When any parameter changes, the replay has to be run again; carrying the old number across silently is how a program ends up paging twice as often as it planned, or half as often, without noticing either.

Seasonality in capacity. Weekly capacity is not constant; respiratory season, summer staffing and emergency deployments all change it. Where the method allows, calibrate seasonal thresholds, or at least check that the chosen threshold stays within capacity in the busiest historical month.

Data feed changes. A new laboratory interface, a changed case definition or an electronic case reporting rollout changes the baseline. Thresholds calibrated before the change may alert far more or far less after it. Recalibrate on post-change data as soon as enough has accumulated, and flag the interim period in the protocol.

INFO calibrate replayed 2021-01-01 to 2025-12-31: 18342 candidate signals
INFO calibrate chosen threshold 180: 3.6 alerts/week, 23/29 events, median delay 2 days
What the Chosen Threshold Would Have Done A stacked bar of twenty-nine historical events at the chosen threshold. Fifteen would have been detected before the first report through any other channel, eight would have been detected after it, and six would have been missed. The report lists the six missed events by name so the team can judge whether any of them would change the threshold decision. 29 historical events at the chosen threshold 15 found before first report 8 found after 6 missed the missed events are listed by name in the calibration report "found before first report" is the number that shows the system's added value

Compliance Notes

Store the replay configuration, the as-of data snapshot identifiers, the event list and matching rules, the full threshold sweep table, the capacity figure and who agreed it, and the chosen threshold with the date it took effect.

The calibration report is a governance document as much as a technical one. It states, in numbers, which past events the program has accepted it would miss in exchange for a manageable workload, and it should be signed off by the surveillance program lead rather than set silently in a configuration file. Event lists derived from investigation records contain sensitive details; keep them under the same access controls as the records themselves, and share only aggregate calibration results.