Spatial Epi Public Health GIS

Controlling False Alarms in Daily Prospective Surveillance

A surveillance system that alarms too often is worse than none, because it consumes the attention that a real signal would have received. This guide, part of Space-Time Cluster Detection, covers why the nominal significance level of a daily scan says almost nothing about its alarm rate, how to measure the real rate, and how to choose an operating threshold from what the response team can absorb.

Problem Context & Constraints

Three effects compound in a repeated prospective system, and each of them makes the naive calculation optimistic in a different way.

Repetition. A daily system performs 365 tests a year. At a nominal α of 0.05, the expected number of false alarms is about 18 — and that figure assumes independence, which is the second problem.

Dependence. Consecutive scans share almost all of their data, so alarms come in runs rather than singly. A system with 18 expected false alarms per year does not produce 18 separate events; it produces perhaps five episodes, each lasting several days. Episodes are what a response team experiences, and the episode count is what should be measured.

Multiplicity within a run. The scan itself already searches thousands of cylinders and reports the maximum, so its p-value is adjusted for that search. What it is not adjusted for is the repetition across days, and no per-run adjustment can be, because the adjustment depends on how long the system will run.

The response to all three is to abandon per-run significance entirely in favour of a recurrence interval, and then to verify the recurrence interval empirically rather than trusting its derivation.

Alarms Arrive in Episodes, Not Singly A calendar strip of 365 quiet days with alarm days marked. Eighteen alarm days occur, but they fall into five clusters of consecutive or near-consecutive days rather than being spread evenly. A response team experiences five interruptions, not eighteen, and a system tuned on the day count will be tuned to the wrong quantity. 18 alarm days, 5 interruptions 1 2 3 4 5 Jan Dec What the theory counts 18 false-positive days per year from 365 tests at α = 0.05 What the team experiences 5 interruptions per year tune on this number Counting days overstates the burden threefold and understates each episode's cost

Prerequisites

  • A configured prospective scan, per Running a Prospective Space-Time Scan for Early Warning
  • At least one year of historical data believed to contain no true cluster, or a mechanism to simulate quiet series from the baseline
  • A stated response cost per investigation, in hours
  • python 3.11, numpy 1.26.4, pandas 2.2.2

The gap between the theoretical false-alarm count and the experienced one comes from dependence, and its size is worth seeing:

Alarm Days Against Investigation Episodes Alarm days and episodes per year at four recurrence-interval thresholds. At a threshold of 50 there are 31.2 alarm days but only 9.4 episodes. At 100, 18.0 days and 5.1 episodes. At 200, 9.6 days and 2.8 episodes. At 365, 4.8 days and 1.5 episodes. The ratio holds near three to one throughout, because consecutive scans share almost all their data. Three alarm days per investigation, at every threshold alarm days episodes RI 50 RI 100 RI 200 RI 365

Step-by-Step Solution

# Measure the empirical alarm rate of a prospective configuration.
# Pinned: numpy==1.26.4, pandas==2.2.2
import logging
import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("scan.alarmrate")


def episodes(alarm_days: np.ndarray, gap: int = 7) -> list[tuple[int, int]]:
    """Collapse alarm days into episodes separated by at least `gap` quiet days.

    `gap` should match how a response team actually works: two alarms four days
    apart on the same areas are one investigation, not two."""
    if len(alarm_days) == 0:
        return []
    d = np.sort(alarm_days)
    breaks = np.where(np.diff(d) > gap)[0]
    starts = np.concatenate([[0], breaks + 1])
    ends = np.concatenate([breaks, [len(d) - 1]])
    return [(int(d[s]), int(d[e])) for s, e in zip(starts, ends)]


def alarm_profile(results: pd.DataFrame, thresholds=(50, 100, 200, 365, 730),
                  gap: int = 7, days: int | None = None) -> pd.DataFrame:
    """Alarm days and episodes per year at a range of recurrence thresholds."""
    days = days or int(results["scan_day"].nunique())
    rows = []
    for th in thresholds:
        hits = results.loc[results["recurrence_scans"] >= th, "scan_day"].to_numpy()
        eps = episodes(hits, gap)
        rows.append({
            "threshold": th,
            "alarm_days_per_year": 365.0 * len(hits) / days,
            "episodes_per_year": 365.0 * len(eps) / days,
            "median_episode_days": float(np.median([e - s + 1 for s, e in eps])) if eps else 0.0,
        })
        log.info("RI>=%4d: %.1f alarm days/yr, %.1f episodes/yr, median episode %.0f days",
                 th, rows[-1]["alarm_days_per_year"], rows[-1]["episodes_per_year"],
                 rows[-1]["median_episode_days"])
    return pd.DataFrame(rows)


def choose_threshold(profile: pd.DataFrame, budget_investigations_per_year: float) -> int:
    """Lowest threshold whose episode rate fits the response budget.

    Lower thresholds are more sensitive, so the correct choice is the smallest
    threshold the team can actually absorb — not the largest one that looks safe."""
    ok = profile.loc[profile["episodes_per_year"] <= budget_investigations_per_year]
    if ok.empty:
        raise ValueError("no threshold in the sweep fits the budget; widen the sweep "
                         "or reconsider the configuration")
    th = int(ok["threshold"].min())
    log.info("selected recurrence threshold %d (%.1f episodes/yr within budget %.1f)",
             th, float(ok.loc[ok['threshold'] == th, 'episodes_per_year'].iloc[0]),
             budget_investigations_per_year)
    return th

Selecting the lowest threshold within budget, rather than a comfortable high one, is the point. Sensitivity is the reason the system exists; the threshold exists only to keep the burden survivable, so it should be pushed as low as the burden allows and no lower.

Validation & Edge Cases

1. Confirm the quiet period was actually quiet. A year chosen as a baseline that contained an unrecognised outbreak will inflate the measured alarm rate and drive the threshold too high. Cross-check against the outbreak log before using it.

2. Simulate if history is short. Where fewer than two years of clean history exist, generate quiet series by resampling case dates within areas under the baseline model and run the full configuration on each. A hundred simulated years gives a much more stable episode rate than one observed one:

INFO RI>=  50: 31.2 alarm days/yr, 9.4 episodes/yr, median episode 3 days
INFO RI>= 100: 18.0 alarm days/yr, 5.1 episodes/yr, median episode 3 days
INFO RI>= 200:  9.6 alarm days/yr, 2.8 episodes/yr, median episode 3 days
INFO RI>= 365:  4.8 alarm days/yr, 1.5 episodes/yr, median episode 2 days
INFO selected recurrence threshold 100 (5.1 episodes/yr within budget 6.0)

3. Re-measure after any configuration change. Window length, maximum spatial extent and completeness trimming all move the alarm rate, so the profile is a property of the whole configuration and not of the threshold alone.

4. Do not raise the threshold in response to a busy month. A run of alarms during an active season is the system working. Raising the threshold mid-season silently changes the series and destroys the ability to compare across years; if the burden is genuinely unsustainable, triage the alarms rather than suppressing them.

5. Report sensitivity alongside the alarm rate. A threshold chosen purely on burden is half a decision. Pair each candidate threshold with the detection delays from the replay described in Tuning the Temporal Window for Space-Time Scans, so the trade is explicit.

6. Separate the alarm rate by stratum. A system covering a whole state may have an acceptable aggregate alarm rate while alarming constantly in one metropolitan area whose baseline model fits poorly. Breaking the episode count down by region turns “the system is noisy” into “the model is misspecified in these three counties”, which is actionable.

7. Track the positive predictive value, not only the alarm rate. Once verdicts are being recorded, the share of alarms that turned out to be real is the number that determines whether the system retains its audience. A configuration with six episodes a year of which four are real is far more valuable than one with three of which none are, and the alarm rate alone cannot distinguish them.

8. Publish the operating characteristics with the system, not on request. Anyone receiving alerts should be able to see, without asking, how often the system alarms, how often those alarms have been real, and how quickly it has detected past events. A single page carrying those three numbers, refreshed automatically, does more for the system’s credibility than any individual alert.

The operating characteristics that belong on a public status page are three numbers, refreshed automatically:

The Three Numbers That Make a System Credible Three operating characteristics displayed together: alarm episodes per year at the current threshold, the share of past alarms that turned out to be real, and the median detection delay measured by replay on historical outbreaks. Together they let a recipient judge how much weight to place on any individual alert, and none of them can be computed without persisting every run and every verdict. Publish these three, refreshed automatically 5.1 episodes per year at the current threshold 38% alarms that were real from recorded verdicts 9 d median detection delay replayed on past outbreaks None of the three is computable without persisting every run and every verdict

Compliance Notes

  • Record the threshold, the measured episode rate and the budget it was chosen against. All three are needed to explain a later decision not to investigate.
  • Log every threshold change with its date and rationale, because an alarm-rate change is otherwise indistinguishable from an epidemiological change.
  • Persist non-alarming runs. The empirical alarm rate cannot be measured from the alarms alone.
  • Report the false-alarm rate in any evaluation of the system, alongside detection delay. A system evaluated only on the outbreaks it caught has not been evaluated.