Spatial Epi Public Health GIS

Scheduling a Nightly Geocode and Cluster Run

This guide, part of Orchestrating Spatial Surveillance Pipelines, builds the specific job most health departments want first: geocode yesterday’s new case reports overnight, run a prospective space-time scan, and have any signal on a desk by the morning meeting.

Problem Context & Constraints

The naive schedule runs at midnight and fails most nights. Case reports arrive from laboratories and providers on their own timetable, which for many feeds means a batch that lands somewhere between 2 a.m. and 5 a.m., with stragglers the following day. A job triggered by the clock rather than by data arrival processes an incomplete day and reports an artificially low count — which a prospective scan interprets as a decline.

Three constraints shape a working schedule.

Completeness versus latency. Waiting for every record maximises completeness and delays the signal past the point of usefulness. The resolution is not to pick one but to make the reporting delay explicit: run at a fixed offset, treat the most recent days as provisional, and let the scan model the reporting delay rather than pretending it does not exist.

Geocoding is the slow stage and the rate-limited one. A backlog of several thousand addresses against a rate-limited service can take longer than the window between data arrival and the morning meeting. Caching by normalised address, and geocoding only new addresses, turns most nights into a few hundred calls.

Alert fatigue is the real failure mode. A prospective scan run nightly across several disease groups will produce signals. If every one of them pages an epidemiologist, the system is muted within a month. The threshold and the routing matter as much as the statistic.

Where to Put the Trigger A timeline across one night showing the cumulative arrival of case reports. Very few have arrived by midnight, the bulk lands between two and five in the morning, and a small tail continues through the following day. A trigger at midnight would process a fraction of the day. A trigger at five thirty captures the great majority while leaving time before the morning meeting. The remaining tail is handled by treating the most recent days as provisional rather than by waiting for it. Trigger on arrival, not on the clock alone midnight trigger: 8% of the day 05:30 trigger: 92% 00:00 03:00 08:00 next day cumulative share of one day's case reports received The remaining tail is modelled as reporting delay, not waited for

Prerequisites

  • python 3.11
  • pandas 2.2.2, geopandas 0.14.4, numpy 1.26.4
  • The orchestration primitives from the parent guide
  • A geocoder with a result cache keyed by normalised address and geocoder vintage
  • A prospective scan implementation, as in Running a Prospective Space-Time Scan for Early Warning
  • Historical arrival timestamps, to derive the trigger offset empirically

Step-by-Step Solution

Derive the trigger offset from arrival history, geocode only uncached addresses, run the scan on a window that respects reporting delay, and route signals by severity.

# Nightly geocode-and-scan job with an arrival-derived trigger and cached geocoding.
# Pinned: pandas==2.2.2, geopandas==0.14.4, numpy==1.26.4
import logging

import numpy as np
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("pipeline.nightly")

TARGET_COMPLETENESS = 0.90
PROVISIONAL_DAYS = 3


def derive_trigger_offset(arrivals: pd.DataFrame, target=TARGET_COMPLETENESS) -> float:
    """Hours after midnight at which the target share of a day has historically landed.

    arrivals: event_date, received_utc  (one row per case report, many days)
    """
    a = arrivals.copy()
    a["received_utc"] = pd.to_datetime(a["received_utc"])
    a["event_date"] = pd.to_datetime(a["event_date"])
    a["lag_h"] = (a["received_utc"] - (a["event_date"] + pd.Timedelta(days=1))
                  ).dt.total_seconds() / 3600.0
    by_day = a.groupby("event_date")["lag_h"].quantile(target)
    offset = float(np.nanmedian(by_day))
    log.info("trigger offset for %.0f%% completeness: %.1f h after the event day ends "
             "(derived from %d days)", 100 * target, offset, by_day.notna().sum())
    return offset


def addresses_needing_geocode(cases: pd.DataFrame, cache: pd.DataFrame,
                              geocoder_vintage: str) -> pd.DataFrame:
    """Only addresses absent from the cache for THIS geocoder vintage are sent out."""
    c = cases.copy()
    c["addr_key"] = c["address_normalised"].str.upper().str.strip()
    have = set(cache.loc[cache["geocoder_vintage"] == geocoder_vintage, "addr_key"])
    todo = c[~c["addr_key"].isin(have)].drop_duplicates("addr_key")
    log.info("geocode batch: %d case rows, %d distinct addresses, %d need the service "
             "(%.1f%% cache hit)", len(c), c["addr_key"].nunique(), len(todo),
             100.0 * (1 - len(todo) / max(c["addr_key"].nunique(), 1)))
    return todo


def scan_window(as_of: pd.Timestamp, baseline_days: int = 365,
                provisional_days: int = PROVISIONAL_DAYS) -> dict:
    """Study window ending before the provisional tail, with its baseline."""
    end = as_of - pd.Timedelta(days=provisional_days)
    return {"baseline_start": end - pd.Timedelta(days=baseline_days),
            "study_end": end,
            "provisional_from": end + pd.Timedelta(days=1),
            "provisional_days": provisional_days}


def route_signals(signals: pd.DataFrame, page_threshold: float = 0.001,
                  review_threshold: float = 0.01) -> pd.DataFrame:
    """Severity routing. Only the strongest signals interrupt a person."""
    s = signals.copy()
    s["routing"] = np.select(
        [s["p_value"] <= page_threshold, s["p_value"] <= review_threshold],
        ["page", "review_queue"], default="log_only")
    counts = s["routing"].value_counts().to_dict()
    log.info("signals routed: %s (page<=%.4f review<=%.3f)",
             counts, page_threshold, review_threshold)
    if counts.get("page", 0) > 3:
        log.warning("%d paging signals in one night — check for a data artefact "
                    "before escalating", counts["page"])
    return s

Deriving the trigger from arrival history rather than choosing a time is what makes the schedule survive a change in an upstream reporting system. When a laboratory switches to an earlier batch, the empirical offset moves and the job can move with it; a hard-coded 5:30 silently starts running an hour later than it needs to, or worse, an hour too early.

The provisional tail is the honest treatment of reporting delay. The scan’s study window ends three days before the run date, so the most recent days — which are known to be incomplete — do not enter the statistic and produce a spurious decline. Those days are still reported, clearly labelled provisional, because an epidemiologist wants to see them even knowing they will rise.

Three Routes, Only One of Which Wakes Anyone Signals from the nightly scan are routed by strength into three destinations. A small number of very strong signals page an on-call epidemiologist and should occur only a few times a year. A larger number enter a review queue looked at during working hours. The remainder are logged for trend review and never notify anyone. Beneath each route is the approximate frequency that keeps the system credible, and a warning that several paging signals on one night usually indicates a data artefact rather than several outbreaks. The routing threshold is an operational decision, not a statistical one Page strongest signals only a few times a year more than three in one night means check the data first Review queue seen in working hours a handful a month where most genuine early signals are actually caught Log only retained for trend review most nights notifies nobody, and that is what protects the top route A system that pages every night is a system nobody reads by the second month Set the paging threshold from how often you can afford to be interrupted, then check what it would have caught historically before adopting it

Validation & Edge Cases

A feed that arrives empty. An upstream system that returns a zero-row file on failure produces a day with no cases, which the scan reads as a dramatic decline. Assert the row count against the recent distribution and halt rather than analysing it.

Duplicate reports across nights. Laboratories resubmit. Deduplicate on a stable record key before counting, and log how many duplicates were removed; a sudden change in that number is itself a signal about the feed.

A run that overruns into the working day. The geocoding backlog after a holiday weekend can be several times a normal night, and a scan over an enlarged window costs more too. Measure the wall clock of every stage and alert when the total approaches the time available before the morning meeting, because a run that is still executing when people need its output has failed operationally even though every stage will eventually succeed.

Retrospective corrections arriving as new records. Some feeds resubmit a corrected record with a new identifier rather than updating the original, so a correction appears as an additional case. Deduplicating on the record key alone will not catch it. Where the feed provides a supersedes reference, honour it; where it does not, reconcile the daily count against the source system’s own total and investigate any night where the two diverge.

Weekend and holiday patterns. Reporting drops at weekends and collapses around holidays, and a scan that does not model day of week will signal every Tuesday. Either include a day-of-week term or run on a weekly grain.

Geocoder vintage change. A reference data update changes coordinates for previously matched addresses. The cache key includes the vintage, so the change is visible as a mass of cache misses rather than as silently shifted points — but that means a vintage upgrade is an expensive night, and it should be scheduled deliberately.

A scan that signals on the provisional tail. If the study window accidentally includes provisional days, the scan sees a decline and may signal on an adjacent area by contrast. Assert that the study window ends before the provisional boundary.

INFO pipeline.nightly trigger offset for 90% completeness: 5.4 h after the event day ends (derived from 412 days)
INFO pipeline.nightly geocode batch: 1184 case rows, 1102 distinct addresses, 96 need the service (91.3% cache hit)
INFO pipeline.nightly signals routed: {'log_only': 217, 'review_queue': 6, 'page': 1} (page<=0.0010 review<=0.010)

The provisional tail is easier to reason about when drawn against the scan’s study window and baseline:

The Scan Stops Before the Provisional Days A timeline ending at the run date. The long baseline period on the left provides expected counts. The study window follows and is where the prospective scan searches for an excess. The last three days before the run date form the provisional tail, known to be incomplete because of reporting delay, and they are excluded from the study window so that their artificially low counts cannot register as a decline or make neighbouring areas look relatively high. They are still reported, labelled provisional. Incomplete days are shown, never scanned baseline: expected counts study window provisional 3 days run date scanned for excess 365 days assert: study window ends before the provisional boundary otherwise an incomplete day reads as a decline provisional days still appear in the report, clearly marked

Compliance Notes

Record the trigger offset and the arrival history it was derived from, the geocoder vintage and cache hit rate, the deduplication rule and counts, the scan window boundaries including the provisional cut, the routing thresholds, and the signals produced at each route.

Alerts from this job reach people’s phones, which puts them outside the protected environment by definition. A paging message must carry the run identifier, the disease group, the geographic level and the signal strength — never a case count small enough to be identifying, never an address, never a facility name. The recipient opens the protected system to see the detail, which is a small friction at 3 a.m. and the only defensible design.