Spatial Epi Public Health GIS

Running a Prospective Space-Time Scan for Early Warning

A prospective scan is a scheduled job that runs every day, sees mostly the same data it saw yesterday, and must decide whether anything has changed enough to warrant a phone call. This guide, part of Space-Time Cluster Detection, sets out a working daily configuration and the three details that separate a usable early-warning system from an alert generator nobody reads.

Problem Context & Constraints

Three constraints shape every prospective deployment.

The scan runs on incomplete data, always. Cases from the last few days have not finished being reported, so the most recent window — the one the scan most needs to be sensitive in — has an artificial deficit. Ignoring this is the single most common reason a prospective system fails to detect a real outbreak until a week after the epidemiologists already knew.

The scan repeats on overlapping data. Today’s analysis shares 27 of its 28 days with yesterday’s, so consecutive results are strongly dependent and the alarm sequence is autocorrelated. This is why the operating characteristic that matters is the expected number of alarms per year, not the per-run significance level.

And the scan must produce something an epidemiologist can act on. A cluster of nine areas over eleven days with a likelihood ratio of 14.7 is not an instruction. The alert has to name the areas, the window, the observed and expected counts, and the recurrence interval, and it has to say whether the same cluster alarmed yesterday.

Prerequisites

  • Case records with a stable area identifier and one consistent date type, per the parent guide’s prerequisites
  • An estimated reporting-delay distribution for the surveillance stream
  • python 3.11, pandas 2.2.2, numpy 1.26.4, scipy 1.13.1, geopandas 1.0.1
  • A persistent store for run outputs — the alarm history is part of the method, not a log
One Day of a Prospective Surveillance Run Cases arrive and are trimmed at the reporting-completeness threshold. The trimmed series feeds the cylinder generator, which produces only windows ending at the scan day. The scan computes the most likely cluster and its recurrence interval. That result is compared against yesterday's alarm state to decide whether this is a new alarm, a continuing one or a resolved one, and only new or escalating alarms produce a notification. Every run is persisted whether or not it alarmed. The comparison with yesterday is what makes it a system Ingest cases to today Trim to completeness drop the last 5 days Open windows only cylinder generation Scan + permute recurrence interval Compare to yesterday new / continuing / resolved Notify on new or on escalation Persist every run alarm or not Without the third box the scan is retrospective; without the fifth it will alarm every day for a month and without the last one the system's false-alarm rate can never be measured

Alarm identity across days is what turns a sequence of results into a system, and the four states it has to distinguish are worth naming:

The Four Alarm States Four states a daily scan result can take relative to the previous day. New: no alarm yesterday, or an alarm whose areas overlap this one by less than half, so a notification is sent. Continuing: the same cluster at a similar strength, so no notification. Escalating: the same cluster at more than three times yesterday's recurrence interval, so a notification is sent. Resolved: yesterday alarmed and today does not, which is recorded but not notified. Only two of the four states send a message New no alarm yesterday, or area overlap under 0.5 notify Continuing same cluster, similar strength record only Escalating same cluster, recurrence interval more than 3× notify Resolved alarmed yesterday, not today record only Without state tracking, a growing cluster notifies every morning until nobody reads it

Step-by-Step Solution

# Daily prospective scan with completeness trimming and alarm-state tracking.
# Pinned: pandas==2.2.2, numpy==1.26.4, scipy==1.13.1
import hashlib
import json
import logging
import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("scan.daily")


def completeness_cutoff(delays: pd.Series, target: float = 0.90) -> int:
    """Days back from today at which `target` of cases have been reported.

    `delays` is the observed distribution of (report_date - onset_date) in days,
    estimated from closed periods. Scanning past this point measures reporting,
    not incidence."""
    q = int(np.ceil(np.quantile(delays.to_numpy(), target)))
    log.info("reporting %.0f%% complete after %d days; trimming the tail", 100 * target, q)
    return q


def run_daily_scan(cases: pd.DataFrame, scan_day: int, config: dict,
                   scanner) -> dict:
    """One day's scan. `scanner` is the scan() function from the parent guide."""
    cut = scan_day - config["completeness_days"]
    trimmed = cases.loc[cases["day"] <= cut].copy()
    log.info("scan day %d: using data to day %d (%d cases)", scan_day, cut, len(trimmed))

    cyls = config["build_cylinders"](end_day=cut)
    result = scanner(trimmed, cyls, n_perm=config["n_perm"], seed=config["seed"])
    result["scan_day"] = scan_day
    result["data_through"] = cut
    result["config_sha256"] = hashlib.sha256(
        json.dumps({k: v for k, v in config.items() if not callable(v)},
                   sort_keys=True).encode()).hexdigest()[:16]
    return result


def alarm_state(today: dict, yesterday: dict | None, threshold: float) -> dict:
    """Classify the alarm as new, continuing, escalating or resolved.

    Two clusters are 'the same' when their area sets overlap by more than half.
    Without this, a cluster that gains one area each day generates a new alarm
    every day and the recipients stop reading them within a week."""
    alarmed = today["recurrence_scans"] >= threshold
    if not alarmed:
        return {"state": "no_alarm", "notify": False}
    if yesterday is None or not yesterday.get("alarmed"):
        return {"state": "new", "notify": True}

    a, b = set(today["cluster"]["areas"]), set(yesterday["cluster"]["areas"])
    jaccard = len(a & b) / len(a | b) if a | b else 0.0
    if jaccard < 0.5:
        return {"state": "new", "notify": True, "jaccard": jaccard}
    escalated = today["recurrence_scans"] >= 3 * yesterday["recurrence_scans"]
    return {"state": "escalating" if escalated else "continuing",
            "notify": escalated, "jaccard": jaccard}


def format_alert(result: dict, area_names: dict, obs: int, exp: float) -> str:
    """The text a human reads. Numbers first, statistic last."""
    c = result["cluster"]
    days = c["end_day"] - c["start_day"] + 1
    return (
        f"{obs} cases observed against {exp:.1f} expected "
        f"({obs / exp:.1f}x) in {len(c['areas'])} areas over the last {days} days.\n"
        f"Areas: {', '.join(sorted(area_names[a] for a in c['areas']))}\n"
        f"Recurrence interval: 1 in {result['recurrence_scans']:.0f} daily scans.\n"
        f"Data complete through day {result['data_through']}; "
        f"config {result['config_sha256']}."
    )

The alarm_state function is what most implementations omit and most operational failures come from. Without cluster-identity matching, a slowly growing cluster produces a fresh “new cluster detected” notification every morning, and the recipients learn within a fortnight that the system’s alerts do not mean anything.

Validation & Edge Cases

1. Measure the alarm rate on a quiet period before going live. Run the configuration over a historical stretch with no known outbreak and count alarms. A daily system alarming more than about once a month on quiet data will not be trusted, regardless of its sensitivity:

INFO reporting 90% complete after 5 days; trimming the tail
INFO scan day 412: using data to day 407 (18,204 cases)
INFO generated 84,120 prospective cylinders
INFO most likely cluster: 6 areas, 9-day window, LLR 11.84, p=0.0030, recurrence interval 333 scans
INFO backtest over 365 quiet days: 11 alarms at RI>=100, 2 at RI>=365

2. Backtest against known outbreaks, and record the detection delay. Sensitivity for a prospective system is measured in days, not in power. Take three historical outbreaks, replay the data as it would have arrived, and record how many days after true onset the system would have alarmed.

3. Confirm the trimming is applied before cylinder generation, not after. Generating cylinders that end at today and then filtering the data leaves windows whose expected counts assume complete reporting. The order in the code above is load-bearing.

4. Watch for a cluster that never resolves. A cluster continuing for more than a few weeks is usually a baseline problem. Add a maximum continuation before the system forces a re-baseline and says so.

6. Handle the case where the scan cannot run. A daily job will occasionally fail — the feed is late, the geography service is down, the cluster is out of memory. The failure must be visible, because a silent gap in a surveillance system is indistinguishable from a quiet period. Emit a heartbeat on every run, alarming or not, and alert on its absence rather than on its content.

7. Keep the notification short and the detail linked. An alert that opens with three paragraphs of methodology will not be read at seven in the morning. Lead with the counts and the areas, give the recurrence interval in one line, and link to the full result including the configuration hash for anyone who needs it. The structured record behind the alert should carry everything; the message should carry what determines whether someone picks up the phone.

8. Re-run yesterday when late data arrives. Records arriving after their reporting day change the counts the scan already evaluated. Re-running the previous few days on each cycle, and recording that the re-run happened, keeps the alarm history consistent with the data as it now stands.

Detection delay measured by replay is the number that establishes whether the system is worth operating, and it varies more by outbreak than by configuration:

Replay Detection Delay on Three Historical Outbreaks Detection delay measured by replaying three closed outbreaks through the configured system. A point-source gastrointestinal outbreak was detected 4 days after onset. A vector-borne cluster took 11 days. A slow respiratory increase took 23 days. The spread between them is wider than the spread across any configuration change, which is why sensitivity must be reported per outbreak type rather than as one number. Detection delay depends on the outbreak more than on the settings Point-source GI 4 days Vector-borne 11 days Slow respiratory rise 23 days Report per outbreak type; a single “median delay” conceals which kinds the system serves

Compliance Notes

  • Persist every run, including the ones that did not alarm, with the configuration hash. The false-alarm rate is a property of the system that can only be measured from the complete series.
  • Version the alert text. When the alert format changes, previously issued alerts must remain interpretable, so store the structured result alongside the rendered text.
  • Treat the area list as restricted until the cluster is confirmed. A named list of small areas with an active signal is disclosive, and distribution lists should reflect that.
  • Record who was notified and when. An early-warning system’s value in review is the timeline it can produce, and that timeline has to include the notification, not only the detection.