Space-Time Cluster Detection

A purely spatial method looks at one snapshot and asks where the excess is. A space-time method asks where and when, and in a surveillance setting the second question is the one that triggers a response. This guide is part of Disease Clustering & Spatial Statistical Modeling, and it covers prospective and retrospective space-time scanning, small-area aberration detection, and the multiple-testing problem that repeated scanning creates and that purely spatial work never has to face.

Concept & Epidemiological Alignment

Adding time changes the question in three ways that matter operationally.

First, the cluster becomes a cylinder rather than a circle: a spatial extent paired with a time window. The scan evaluates many combinations of both, and the most likely cluster is the cylinder with the largest likelihood ratio against the null of constant risk. That is the same machinery as the purely spatial scan covered in Spatial Scan Statistics Configuration, with one more dimension to search.

Second, prospective and retrospective analysis are different methods, not the same method run at different times. A retrospective scan searches all time windows within a closed study period and asks whether any excess occurred. A prospective scan searches only windows that are still open — windows ending at today — because a cluster that ended six weeks ago is not actionable. Restricting the window set changes the null distribution, and using retrospective p-values in a prospective setting overstates significance badly.

Third, repetition is the dominant statistical problem. A surveillance system scanning daily performs 365 analyses a year on overlapping data. At a nominal 0.05 threshold that is eighteen expected false alarms annually before any real signal, and the alarms will not be independent because consecutive days share almost all their data. This is why prospective surveillance reports recurrence intervals rather than p-values, and why the recurrence interval is the single most important number in a prospective configuration.

Method Selection

The permutation scan deserves particular attention in surveillance work because it needs no denominator. It conditions on the observed spatial and temporal margins and asks whether cases are more concentrated in space-time than that conditioning implies. That makes it robust to a population denominator that is stale or unavailable, which is common in the early weeks of a response — and it also means it cannot detect a cluster that is purely spatial, because the spatial margin is conditioned away.

Spatial Data Prerequisites

• A date on every case that means the same thing. Onset date, specimen date and report date produce different clusters, and mixing them produces an artefact at every reporting-pattern change. Pick one, and handle its missingness explicitly.
• A stable areal geography across the whole period, harmonized per Areal Interpolation & Boundary Harmonization if a vintage boundary changed mid-series.
• Consistent geocoding across the period. A geocoder upgrade mid-series shifts cases between areas and the scan reads it as an emerging cluster; see Detecting Coordinate Drift Between Geocoding Vintages.
• A reporting-delay distribution. Recent periods are always incomplete, and a prospective scan run on incomplete recent data will under-detect exactly when detection matters.

Production Implementation

# Prospective space-time permutation scan with recurrence-interval reporting.
# Pinned: python 3.11, numpy==1.26.4, pandas==2.2.2, geopandas==1.0.1, scipy==1.13.1
import logging
import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("scan.spacetime")


def build_cylinders(coords: np.ndarray, area_ids: np.ndarray,
                    max_radius_m: float, max_days: int, end_day: int,
                    min_days: int = 1) -> list[dict]:
    """Every (spatial disc, open time window) pair to be evaluated.

    Only windows ending at `end_day` are generated: this is what makes the scan
    prospective. Generating closed windows here and filtering later is a common
    bug that leaves the null distribution retrospective."""
    from scipy.spatial import cKDTree
    tree = cKDTree(coords)
    cylinders = []
    for i in range(len(coords)):
        neigh = tree.query_ball_point(coords[i], max_radius_m)
        neigh = sorted(neigh, key=lambda j: np.linalg.norm(coords[j] - coords[i]))
        for k in range(2, len(neigh) + 1):          # grow the disc one unit at a time
            members = frozenset(area_ids[j] for j in neigh[:k])
            for w in range(min_days, max_days + 1):
                cylinders.append({"centre": area_ids[i], "areas": members,
                                  "start_day": end_day - w + 1, "end_day": end_day})
    log.info("generated %d prospective cylinders", len(cylinders))
    return cylinders


def permutation_statistic(cases: pd.DataFrame, cyl: dict) -> float:
    """Space-time permutation test statistic for one cylinder.

    Expected count conditions on both margins: the area's share of all cases and
    the window's share of all days. No population denominator is used."""
    n = len(cases)
    in_area = cases["area_id"].isin(cyl["areas"])
    in_time = cases["day"].between(cyl["start_day"], cyl["end_day"])
    obs = int((in_area & in_time).sum())
    exp = in_area.sum() * in_time.sum() / n if n else 0.0
    if obs <= exp or exp <= 0:
        return 0.0
    rest_obs, rest_exp = n - obs, n - exp
    return obs * np.log(obs / exp) + rest_obs * np.log(rest_obs / rest_exp)


def scan(cases: pd.DataFrame, cylinders: list[dict], n_perm: int = 999,
         seed: int = 42) -> dict:
    """Most likely cluster and its recurrence interval."""
    rng = np.random.default_rng(seed)
    stats = np.array([permutation_statistic(cases, c) for c in cylinders])
    best = int(stats.argmax())

    null_max = np.empty(n_perm)
    for b in range(n_perm):
        shuffled = cases.copy()
        shuffled["day"] = rng.permutation(cases["day"].to_numpy())
        null_max[b] = max(permutation_statistic(shuffled, c) for c in cylinders)

    rank = int((null_max >= stats[best]).sum())
    p = (rank + 1) / (n_perm + 1)
    # Recurrence interval: how often a signal this strong would occur by chance,
    # in units of the scan's own period. This is what a daily system reports.
    recurrence = 1.0 / p
    log.info("most likely cluster: %d areas, %d-day window, LLR %.2f, p=%.4f, "
             "recurrence interval %.0f scans",
             len(cylinders[best]["areas"]),
             cylinders[best]["end_day"] - cylinders[best]["start_day"] + 1,
             stats[best], p, recurrence)
    return {"cluster": cylinders[best], "llr": float(stats[best]),
            "p": float(p), "recurrence_scans": float(recurrence)}

The recurrence interval is the number that belongs in an alert. “Recurrence interval 365 scans” on a daily system means an alarm this strong is expected about once a year by chance, which an epidemiologist can act on. “p = 0.0027” invites the reader to compare it to 0.05 and conclude, wrongly, that it is highly unusual.

Parameter Selection & Tuning

• Maximum temporal window should be set from the plausible duration of the outbreak being watched for, and it interacts with the maximum spatial extent: a wide, long cylinder will win on likelihood while describing nothing actionable. Bound both.
• Minimum window length guards against single-day artefacts, especially where reporting is batched weekly.
• Recurrence-interval threshold replaces the significance threshold. A common operating point for a daily system is 100 scans for investigation and 365 for escalation, and both should be set from response capacity rather than convention.
• Baseline length for the permutation margins should cover at least one full seasonal cycle where the condition is seasonal, or the scan will flag every seasonal rise.

Edge Cases & Failure Modes

Reporting-delay artefacts. The most recent days are always incomplete, so a scan run on them sees a deficit, not an excess, and prospective sensitivity collapses in the final window. Either scan only up to a completeness threshold or apply a nowcast; do not scan raw counts to today.

Recurrent alarms on the same area. A system that flags the same census tract weekly is not detecting a persistent outbreak; it is usually detecting a stable baseline difference that the model has not accounted for. Persistent alarms are a specification problem.

Holiday and weekday effects. Specimen dates cluster on weekdays and collapse on holidays. Without a day-of-week adjustment, the scan will find a “cluster” every Tuesday in whichever area has the largest laboratory.

Boundary effects in time. Cases near the start of the study period have truncated windows, exactly as spatial edge cases have truncated neighbourhoods. Discard the first max_days of the series from evaluation rather than reporting weak clusters there.

Compliance & Audit Controls

• Log the full configuration per scan run: maximum radius, maximum and minimum window, end day, completeness threshold, permutation count and seed. A prospective system’s alerts are only reproducible if each day’s configuration is retrievable.
• Persist every scan’s result, not only the ones that alarmed. The sequence of non-alarms is what establishes the system’s false-alarm rate in practice, and it cannot be reconstructed later.
• Record the recurrence-interval threshold and any change to it. Lowering the threshold mid-season changes the alarm rate and must not be invisible in the series.
• Treat cluster membership as disclosive. A named list of areas in an active cluster, combined with a small population, can identify individuals; apply the same controls as any other release, per Privacy-Preserving Spatial Analytics.

From Signal to Response

A detection system that produces statistically sound alerts nobody can act on has not helped, and the gap between the two is usually organisational rather than methodological. Three things close it.

A named owner per alert stream. An alarm routed to a distribution list is an alarm routed to nobody. Each configured system — the daily permutation scan, the weekly Poisson scan, the per-area aberration detector — needs a person who receives its output and is expected to triage it, and that expectation needs to survive staff turnover.

A written triage protocol with a time budget. The first question on receiving an alert is almost never epidemiological: it is whether the signal is a data artefact. A protocol that starts with the reporting-completeness check, the boundary-vintage check and the recent-configuration-change check resolves a substantial share of alerts in minutes, and it does so consistently rather than depending on who is on duty. Only once those are cleared does the clinical question begin.

A feedback loop into the configuration. Every investigated alert produces a verdict — real signal, data artefact, known baseline shift — and those verdicts are the only empirical evidence about the system’s positive predictive value. Recording them turns a year of operation into a calibration dataset, and a system that has been running for two years without recording verdicts has thrown that away.

The same discipline applies to the absence of alerts. A quiet quarter can mean the population is healthy, or that a feed stopped delivering, or that a configuration change silently raised the effective threshold. A weekly heartbeat check confirming that the pipeline ran, consumed the expected number of records and produced a result — alarming or not — is cheap and catches the class of failure where surveillance stops without anybody noticing. Pair it with a simple count of records processed per run, plotted over time, because a step change in that series is almost always an upstream problem rather than an epidemiological one.

Reporting-delay behaviour differs sharply between surveillance streams, and it is the input that most constrains how early a prospective system can detect anything:

Implementation Checklist

• One date type chosen for the whole series, with missingness handled explicitly
• Reporting-delay distribution estimated and a completeness threshold applied
• Prospective window generation restricted to windows ending at the scan day
• Maximum spatial extent and maximum temporal window both bounded and justified
• Day-of-week and holiday effects adjusted or explicitly ruled out
• Recurrence-interval thresholds set from response capacity and recorded
• Permutation seed pinned and configuration hashed per run
• Every run persisted, including non-alarms, with its configuration
• Boundary vintage and geocoding vintage constant across the scanned period

FAQ

Why report a recurrence interval instead of a p-value? Because a prospective system runs repeatedly on overlapping data, so a p-value from one run does not describe the system’s behaviour over a season. A recurrence interval states how often a signal this strong is expected by chance in the units the system actually operates in.

Can I run a retrospective scan every week and treat it as prospective surveillance? No. A retrospective scan searches closed windows too, so its most likely cluster may be an excess that ended months ago, and its null distribution assumes a single analysis of a closed period.

Does the permutation scan need a population denominator? No, and that is its main operational advantage. It conditions on the observed spatial and temporal margins instead, which also means it cannot detect a purely spatial excess.

How do I keep a stable baseline when reporting practices change? You cannot, entirely. Record known reporting changes as annotations on the series, exclude the transition period from baseline estimation, and expect the scan to flag the change itself if you do not.

Choosing Between the Scan and the Detector

The two families in this topic answer different questions and the choice between them is usually decided by the shape of the surveillance programme rather than by statistics.

Reach for a scan statistic when the geography of an excess is unknown in advance, when a cluster might straddle administrative boundaries, and when the areas are individually too small to support their own baselines. The scan searches over shapes, so it can find a cluster that no single area’s data would reveal, and it needs no per-area model.

Reach for a per-area detector when the areas are large enough to carry stable baselines, when the question is about a known place — a facility catchment, a school district — and when the value of the system lies in continuity rather than discovery. A detector tracks a specific area’s level over time and can say when it changed, which a scan cannot.

Most mature programmes run both, and the useful discipline is to keep their alert streams separate rather than merging them into one queue. They have different false-alarm rates, different detection delays and different triage steps, and a merged stream makes all three unmeasurable. Where a signal appears in both, that agreement is itself informative — it is the closest thing prospective surveillance has to independent confirmation, since the two methods use the data in different ways.

The one combination to avoid is running a scan whose areas are the output of a detector’s alarm — selecting the areas to scan on the basis of the same data being scanned invalidates the permutation null entirely.

Related Topics

• Disease Clustering & Spatial Statistical Modeling — the parent section and the purely spatial methods this one extends.
• Spatial Scan Statistics Configuration — the spatial machinery, including maximum cluster size, that carries over directly.
• Bayesian Disease Mapping & Rate Smoothing — the complementary approach when the question is level rather than change.
• Getis-Ord Gi* Hotspot Detection — the single-snapshot alternative, and the one that cannot see emergence.
• Areal Interpolation & Boundary Harmonization — required whenever a boundary vintage changes inside the scanned period.