Spatial Epi Public Health GIS

Cluster Detection Power & Benchmarking

This guide is part of the Disease Clustering & Spatial Statistical Modeling section. It covers the question every other topic in the section leaves implicit: given this jurisdiction’s geography, population and case counts, how good is a chosen detection method at finding the outbreaks a program cares about?

Concept & Epidemiological Alignment

A detection method’s published properties — its type I error, its asymptotic power — describe its behaviour in general. A surveillance program needs something more specific: its behaviour on this map, with these denominators, for outbreaks of this plausible size and shape. A scan statistic that is powerful for compact circular clusters in a dense urban grid may have almost no power for an elongated cluster along a rural river valley; a local Moran’s I that performs well with regular tracts may flag noise in a county map dominated by a few very large, sparsely populated units.

Benchmarking answers those questions by simulation. Generate many synthetic case datasets on the real geography: some under the null, with cases distributed in proportion to population; some with a planted outbreak of known location, size and excess risk. Run the method on each. Measure how often it signals under the null (false alarms), how often it detects the planted outbreak (power), and how well its flagged region overlaps the true one (localisation).

These three numbers are what a program should use to choose a method and set its threshold, and they are rarely computed. Programs usually inherit a method from a guideline and a threshold from convention, and discover its real behaviour only through the pattern of alerts it produces — which is a slow, expensive and unrepresentative benchmark.

The epidemiological alignment is in the planted outbreaks. They must look like the outbreaks the program actually needs to find: foodborne clusters concentrated around a venue, vector-borne clusters along habitat, environmental clusters downwind of a source, each with realistic excess risk. A benchmark of perfect circles with a fivefold excess tells you nothing about a real outbreak with a 50% excess spread along a highway.

Benchmarking also changes the conversation about thresholds. A program choosing between a p-value threshold of 0.01 and 0.05 is, without a benchmark, choosing between two numbers it cannot evaluate. With one, it is choosing between two points on a curve: at 0.05, perhaps three false alarms a month and detection of a 1.5-fold outbreak in 70% of urban counties; at 0.01, one false alarm a month and detection in 45%. That is a decision a program manager can make, weighing staff time against missed outbreaks, and it is the decision the threshold was always implicitly making.

The same logic applies to comparing geographies. Running a method on tracts rather than counties buys spatial precision and costs power, because each tract’s expected count is small. A benchmark on both geographies quantifies the trade — how much localisation improves and how much power falls — so the choice of unit becomes an evidence-based decision rather than a habit inherited from the last analyst.

Metrics That Matter

Metric Definition Why a program cares
False alarm rate Share of null simulations with any signal Staff time spent on nothing
Power Share of outbreak simulations where the outbreak is detected Outbreaks missed
Sensitivity (areal) Share of true outbreak areas inside the flagged region How much of the outbreak is found
Positive predictive value Share of flagged areas that are truly in the outbreak How much of the alert is noise
Timeliness Days from outbreak onset to first signal Whether detection is early enough to act
The Benchmarking Loop A loop that starts from the real geography and population. Two streams of synthetic case datasets are generated: null datasets with cases in proportion to population, and outbreak datasets with a planted excess of known location, size and shape. The detection method runs on every dataset. Null runs yield the false alarm rate; outbreak runs yield power, areal sensitivity and positive predictive value by comparing the flagged region with the planted one. Repeating across outbreak scenarios produces a performance profile for the method on this jurisdiction. Plant outbreaks you know, and count what the method finds real geography and population null datasets cases ∝ population outbreak datasets planted, known truth false alarm rate any signal under the null power · sensitivity · PPV flagged region vs planted repeat across outbreak shapes, sizes and locations to build a performance profile

Spatial Data Prerequisites

  • The real analysis geography — tracts, counties or a grid — with its actual boundary vintage.
  • Population denominators for every unit, as used in production.
  • A realistic case total, typically the count over the detection window the method will be run on.
  • Outbreak scenarios drawn from the program’s history or plausible hazards: location type, spatial extent, shape, and excess relative risk.
  • The exact method configuration used in production: weights, window sizes, thresholds, permutation counts. Benchmarking a different configuration benchmarks nothing.

Production Implementation

The harness below generates null and outbreak datasets on real areas, runs any detection function, and computes the metrics in the table above.

# Simulation benchmark harness for spatial cluster detection methods.
# Pinned: numpy==1.26.4, pandas==2.2.2, geopandas==0.14.4
import logging

import numpy as np
import pandas as pd
import geopandas as gpd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("benchmark")

SEED = 20260918


def simulate_counts(pop: np.ndarray, total: int, rr: np.ndarray, rng) -> np.ndarray:
    """Multinomial allocation of a fixed total in proportion to pop * relative risk."""
    p = pop * rr
    return rng.multinomial(total, p / p.sum())


def plant_outbreak(areas: gpd.GeoDataFrame, center_id: str, radius_m: float,
                   excess_rr: float) -> tuple[np.ndarray, set]:
    """Relative risk vector with a compact outbreak around one area's centroid."""
    c = areas.set_index("geoid").geometry.centroid
    d = c.distance(c.loc[center_id]).to_numpy()
    inside = d <= radius_m
    rr = np.where(inside, excess_rr, 1.0)
    return rr, set(areas["geoid"].to_numpy()[inside])


def score(flagged: set, truth: set) -> dict:
    if not truth:
        return {"signal": bool(flagged)}
    tp = len(flagged & truth)
    return {"signal": bool(flagged), "detected": tp > 0,
            "sensitivity": tp / len(truth),
            "ppv": (tp / len(flagged)) if flagged else np.nan}


def benchmark(areas: gpd.GeoDataFrame, detect, total: int, scenarios: list[dict],
              n_null: int = 500, n_per_scenario: int = 200, seed: int = SEED) -> pd.DataFrame:
    """detect(areas_with_counts) -> set of flagged geoids, configured exactly as in production."""
    rng = np.random.default_rng(seed)
    areas = areas.sort_values("geoid").reset_index(drop=True)
    pop = areas["population"].to_numpy(float)
    rows = []
    for _ in range(n_null):
        a = areas.assign(cases=simulate_counts(pop, total, np.ones_like(pop), rng))
        rows.append({"scenario": "null", **score(detect(a), set())})
    for sc in scenarios:
        rr, truth = plant_outbreak(areas, sc["center"], sc["radius_m"], sc["rr"])
        for _ in range(n_per_scenario):
            a = areas.assign(cases=simulate_counts(pop, total, rr, rng))
            rows.append({"scenario": sc["name"], **score(detect(a), truth)})
    df = pd.DataFrame(rows)
    summary = df.groupby("scenario").agg(
        runs=("signal", "size"), signal_rate=("signal", "mean"),
        power=("detected", "mean"), sensitivity=("sensitivity", "mean"),
        ppv=("ppv", "mean"))
    log.info("benchmark summary:\n%s", summary.round(3).to_string())
    return summary

Fixing the case total and allocating it multinomially matches how surveillance data behave: the number of cases in a window is observed, and the question is where they fall. Simulating Poisson counts per area instead lets the total vary, which changes the method’s behaviour for reasons unrelated to spatial pattern.

The detection function is passed in unchanged from production, which is the point. A benchmark run with a different weights matrix, a different permutation count or a different threshold measures a method nobody uses.

Power Depends on the Outbreak You Are Looking For Power plotted against the excess relative risk of a planted outbreak for two methods and two outbreak shapes. For compact circular outbreaks, the circular scan reaches high power at a lower excess than local Moran's I. For elongated outbreaks along a corridor, the ordering reverses, because the circular scan window captures the corridor poorly while local statistics flag its individual units. A method's ranking is therefore specific to the outbreak shape a program expects, which is what benchmarking reveals. The better method depends on the outbreak's shape scan, compact outbreak Moran, compact scan, corridor Moran, corridor 1.0 2.0 3.0 excess relative risk inside the planted outbreak power

Parameter Selection & Tuning

Number of simulations. Five hundred null runs estimate a false alarm rate near 5% to within about two percentage points; two hundred runs per outbreak scenario estimate power to within about seven. Increase both for a published comparison.

Scenario design. Draw scenarios from past outbreaks where possible — their locations, extents and excesses. Where history is thin, span plausible ranges: small and large, urban and rural, compact and elongated, modest and strong excess.

Outbreak placement. Place planted outbreaks across the geography rather than at one convenient location. Power in a dense city and power in a sparse rural county differ enormously, and an average across placements hides the rural blind spot.

Case total. Use the total the method will actually see in its window. Power depends strongly on it; a method that works on a year of cases may fail on a week.

Threshold sweeps. Run the benchmark at several thresholds to trace the trade-off between false alarms and power, which is the curve a program actually chooses from.

Relative risk versus absolute excess. Power is driven by the number of excess cases, not the relative risk alone. A 1.5-fold excess in a county of a million produces hundreds of extra cases; the same excess in a county of five thousand produces two or three. Parameterise scenarios by both, and report power against expected excess cases as well as against relative risk, since the former generalises across places and the latter does not.

Comparing methods fairly. When benchmarking two methods, calibrate both to the same false alarm rate before comparing power. A method with a looser default threshold will appear more powerful simply because it signals more often; equal false alarm rates are the only fair basis for comparison.

Edge Cases & Failure Modes

Benchmarking an idealised geography. Synthetic grids of equal squares make every method look good. Use the real boundaries, with their islands, slivers and extremes of size.

Excess risks that are unrealistically large. A fivefold excess is detected by anything. Benchmarks should concentrate on the modest excesses — 1.3 to 2 — that real outbreaks often present, where methods differ.

Ignoring timeliness. A method with high power that detects outbreaks only after they have peaked is of little use. Where the method runs prospectively, add a temporal dimension and measure days to detection.

Overfitting the threshold to the benchmark. Tuning a threshold to maximise power on one set of scenarios and then reporting that power is optimistic. Tune on one set, evaluate on another.

Outbreaks that straddle boundaries. Real outbreaks ignore administrative lines, and a planted outbreak split between two counties is harder to detect for methods that treat units independently. Include scenarios centred on boundaries as well as inside units, and on state lines where the method’s geography stops.

Scoring detection too generously. Counting a run as a detection whenever any flagged area touches the planted outbreak rewards methods that flag large regions. Pair power with positive predictive value, or require a minimum overlap, so a method that flags half the state is not scored as perfectly sensitive.

Stochastic methods inside the benchmark. Permutation-based methods add their own randomness to each simulated run. Fix the method’s permutation seed per run, derived from the run index, so the benchmark is reproducible end to end and two methods are compared on identical simulated datasets rather than on independent draws.

Heterogeneous reporting. Real case data carry reporting differences between counties that simulated data do not. A benchmark that ignores them overstates power where reporting is weak; where possible, simulate under-ascertainment in the counties known to have it.

Where the Method Is Blind A map of a state with each county shaded by the detection power achieved when the planted outbreak is centred there, at a fixed modest excess risk. Urban counties achieve high power because their larger populations produce more excess cases. A band of sparse rural counties achieves very low power, because the same relative excess produces only a handful of extra cases. The map shows a program where its surveillance is effectively blind, which an average power figure would conceal. An average power figure hides the counties where you cannot see 0.91 0.14 0.11 0.09 0.83 power when a 1.5× outbreak is centred in each county; darker is higher

Compliance & Audit Controls

Record the geography and population vintage, the case total, every scenario’s location, extent, shape and excess, the simulation counts and seed, the exact production configuration benchmarked, and the resulting metrics per scenario. Where the benchmark informed the choice of method or threshold, record that decision and the metrics that supported it.

A benchmark is evidence about a program’s capability, and it can be uncomfortable: it may show that the program cannot detect a moderate outbreak in half its counties. That finding belongs in program documentation and planning, not in a drawer, because it is exactly the information needed to justify more sensitive methods, longer windows, or investment in case ascertainment in the blind areas.

Production Implementation Checklist

Frequently Asked Questions

Why not rely on a method’s published power? Published power is computed on idealised geographies and outbreak shapes. Power on your jurisdiction depends on its population distribution, its unit sizes and the outbreaks you care about, and can differ dramatically from the published figure.

How realistic do the planted outbreaks need to be? Realistic enough to resemble what you need to find: plausible extent, shape and — most importantly — excess risk. Benchmarks with very strong excesses make every method look perfect and distinguish nothing.

Should the benchmark include reporting delays? For prospective methods, yes. A cluster that becomes detectable only after delayed reports arrive is detected later than the benchmark on complete data suggests, and timeliness is part of performance.

How often should benchmarking be repeated? When the geography, population or method configuration changes materially, and periodically as the case volume changes, since power depends strongly on the number of cases in the detection window.

What if the benchmark shows the method is blind somewhere? Report it. Consider longer windows, aggregated geography, a different method for that region, or improved ascertainment. Pretending the blind spot does not exist is the only wrong response.