Spatial Epi Public Health GIS

Surveillance Pipeline Automation & Spatial Data Quality

A spatial surveillance pipeline is the difference between an analysis someone ran once and a system a health department can rely on every Monday morning. This section covers the engineering that makes the methods elsewhere on this site repeatable on a schedule: orchestration, storage, data quality gates, testing, and publication.

The failure that motivates all of it is quiet. A geocoder’s reference data updates, a boundary file gains a new vintage, an upstream feed changes a column name — and the nightly run completes successfully with subtly wrong output that nobody notices until a cluster alert fires in the wrong county. Automation without assertions does not reduce risk; it industrialises it. Everything below is organised around making failures loud and making successful runs reconstructable.

A Scheduled Spatial Surveillance Pipeline A left-to-right pipeline in five stages: ingest case records and reference geography, geocode and standardise addresses, run the analytical methods such as clustering and access metrics, apply disclosure control, and publish map products and tables. Between every pair of stages sits a quality gate that can halt the run. Beneath the whole pipeline runs a provenance and configuration store recording input vintages, code versions, parameters and run identifiers, and above it runs an alerting channel that carries failures without carrying protected data. Every arrow is a gate that can stop the run — that is what makes the schedule safe Alerting channel — failure reason, run id, stage, never a record or an address an alert that quotes the offending row is a disclosure incident 1 · Ingest cases, boundaries, denominators 2 · Geocode standardise, match, flag quality 3 · Analyse clusters, access, exposure joins 4 · Suppress disclosure control before anything leaves 5 · Publish tiles, tables, reports gate gate gate gate row counts · geometry validity · match rate · count reconciliation · small-cell rules Provenance store — one row per run input dataset vintages · code commit · library versions · parameters · random seeds · gate results Any run reconstructable from its provenance row alone which is the property that makes a scheduled analysis defensible

The pipeline is a chain of pure transformations with gates between them. Each stage reads inputs it did not create, writes outputs it fully owns, and records what it did. That shape is what makes a failed run restartable from the stage that failed rather than from the beginning, and what makes a question six months later — why did the 14 March run flag that tract — answerable.

Two organisational facts shape every one of these controls. First, the person who wrote the pipeline is frequently not the person operating it at 6 a.m. when a run fails, so the failure has to explain itself without requiring the author. Second, a surveillance pipeline outlives the project that funded it; the code will be read in three years by someone with no memory of the decisions in it, and the record left behind is the only thing that will tell them why a threshold is 0.82 rather than 0.9.

Data Governance & Compliance Architecture

Automation changes the governance problem rather than removing it. A manual analysis has a person between the data and the output; a scheduled one does not, so the controls must be in the code.

The alerting channel is the most common leak. A pipeline that emails an exception traceback has emailed whatever was in the variables — an address, a record identifier, a small-cell count. Alerts must carry a run identifier, a stage, a failure class and a count, and nothing else. Diagnostic detail stays inside the protected environment, retrievable by someone with access using the run identifier.

Disclosure control belongs in the pipeline, not after it. The suppression stage must run before any artifact leaves the protected boundary, and it must fail closed: if the suppression step errors, nothing publishes. Wiring it as a post-processing step on published output means a failure publishes raw counts. The rules themselves come from Privacy-Preserving Spatial Analytics, encoded as the machine-readable policy described in Compliance Mapping Frameworks.

Retention has to be scheduled too. A pipeline that writes an intermediate per run accumulates thousands of copies of the most disclosive tables in the system. Intermediates need a lifecycle policy with the same authority as the source data’s, and the policy needs to be enforced by the storage layer rather than by a cleanup script someone might disable.

Access to run outputs is not the same as access to the pipeline. Analysts reading published tables, operators restarting failed runs and developers changing the code are three different privilege levels, and collapsing them into one service account is how an automation system becomes the weakest link in an otherwise careful architecture.

Where the pipeline boundary sits

The single most consequential architectural decision is where the protected boundary is drawn, and it should be drawn as late as possible. A pipeline that de-identifies at ingestion looks safer and is usually less safe in practice, because every stage after it that genuinely needs an address — geocoding, exposure assignment, catchment allocation — must then be done somewhere else, by hand, outside the audited system. The pattern that holds up is a fully protected pipeline with one narrow, heavily instrumented egress: the suppression stage. Everything upstream of it runs on real data under access control; nothing downstream of it has ever seen a coordinate.

That boundary also determines what testing is possible. Code running inside the protected environment cannot be exercised by developers who lack access to it, which is why the synthetic fixtures described later are not a convenience but a requirement — they are the only way the address-handling stages get tested by the people who maintain them.

Spatial Data Preparation

Scheduled pipelines have preparation problems that one-off analyses do not, all arising from the fact that the inputs change underneath them.

Pin every reference dataset by vintage. Boundary files, geocoder reference data, denominators and crosswalks all update. A pipeline that reads “the current tract file” produces results that change for reasons unrelated to the data it is analysing. Pin the vintage in configuration, upgrade deliberately, and record which vintage each run used — the mechanics are in Versioning Boundary Files Across Vintages.

Assert the schema at the boundary. An upstream feed that renames a column, changes a date format or switches a coordinate order will not raise an error in most spatial code — it will produce plausible garbage. A schema assertion at ingestion that checks column names, types, coordinate ranges and row-count plausibility catches nearly all of it.

Enforce the coordinate reference system on entry. Convert everything to the pipeline’s working system at ingestion and assert it thereafter, following Coordinate Reference Systems for Public Health. A pipeline that carries mixed systems will eventually join two of them.

Validate geometry before it reaches an algorithm. Self-intersecting polygons and unclosed rings enter from upstream editing, and a spatial weights build or an overlay will either fail loudly or, worse, silently produce a slightly wrong result.

Failure modes that only appear on a schedule

Four failure classes are specific to automation and do not occur in interactive analysis.

Silent upstream substitution. A feed that fails and returns an empty file, or yesterday’s file, looks like a successful run with fewer cases. A row-count plausibility check against the recent distribution catches it; a null check does not, because the file is well-formed.

Partial writes. A run killed mid-write leaves a truncated output that the next stage happily reads. Writing to a temporary location and moving atomically on completion is the fix, and it is the single most valuable habit in pipeline engineering.

Clock and time-zone drift. A pipeline that defines “yesterday” in local time and a source system that stamps records in coordinated universal time will disagree for one hour every day and for a whole day twice a year. Define every window in one explicit time zone and store it with the run.

Slowly accumulating state. A cache that is never invalidated, a geocoder result table that keeps old matches, a deduplication key that drifts — none of these fail on any single run, and all of them degrade the output over months. They are found only by re-running an old date and comparing, which is a check worth scheduling in its own right.

Core Methods Overview

The five topics in this section correspond to the five things a scheduled pipeline must get right, in the order they bite.

Orchestrating Spatial Surveillance Pipelines covers the control flow: scheduling, dependencies between stages, idempotency so a re-run is safe, restartability so a failure does not cost the whole night, and backfills that do not corrupt the present. The central design property is that every stage should be safe to run twice.

Cloud-Native Geospatial Storage for Surveillance covers where the data lives: partitioned columnar archives that support time-sliced queries without a database, cloud-optimized rasters that can be read by window, and the vintage management that keeps a multi-year archive queryable. It extends the format reasoning in Spatial Data Types & Formats from a single file to an evolving corpus.

Spatial Data Quality Monitoring covers the gates themselves: geometry validity assertions, drift detection on the distribution of case geography, reconciliation of counts across geographic levels, and the scorecard that makes quality visible to people who do not read logs.

Testing Geospatial Analysis Code covers correctness before deployment: property-based tests for transformations whose correct output is hard to write down, synthetic case data with known answers, pinned library versions so a dependency upgrade cannot silently change a projection, and regression tests on cluster detection output.

Publishing Surveillance Map Products covers the last mile: vector tiles built from already-suppressed data, read-only interfaces for partner agencies, automated static exports for recurring reports, and the standards metadata that lets a recipient know what they received.

Threshold Tuning & Validation

A pipeline’s thresholds are as consequential as any statistical parameter, and they need the same discipline.

Gate thresholds must be derived from history, not guessed. A rule that fails the run when the geocoding match rate drops below 85% should be set from the observed distribution of match rates over previous runs, not from a round number. Compute the historical distribution, set the threshold at a percentile that would have fired rarely, and record what it would have caught.

Distinguish a warning from a halt. Not every anomaly should stop a surveillance pipeline, because a halted pipeline provides no surveillance at all. Health-critical gates — disclosure control, count reconciliation — halt. Quality gates — a match rate slightly low, a boundary file a month old — warn and continue, with the warning surfaced in the scorecard.

Reconciliation is the strongest single gate. Counting the same quantity two ways and requiring agreement catches more real defects than any other check. Cases summed by tract must equal cases summed by county must equal the raw record count minus the explicitly excluded. When the three disagree, something in the geography joined wrong, and the discrepancy names roughly where.

Validate the whole pipeline, not each stage. Stage tests pass while the composition fails, usually at a join. An end-to-end test on synthetic data with a known answer, run on every deployment, catches the composition errors that unit tests structurally cannot.

Re-validate after every dependency change. A projection library upgrade can shift coordinates by metres, which is invisible in a unit test and visible in a cluster boundary. Pin versions, upgrade deliberately, and run the regression suite as part of the upgrade rather than after it.

Setting a Gate Threshold From History A distribution of nightly geocoding match rates observed over two years of runs. The bulk of runs sit in a narrow band at the high end. A warn threshold is placed at the low percentile where roughly one run a month would have triggered, and a halt threshold is placed further out where only a genuine upstream failure would reach. The chart notes that a round number such as eighty-five per cent would have sat in the middle of the normal range for this pipeline, firing constantly and training operators to ignore it. A threshold that fires every night trains people to ignore it halt: genuine upstream failure warn: about one run a month a round 85% would sit here inside the normal range low high nightly geocoding match rate, two years of runs Record what each candidate threshold would have caught before you adopt it

What a run costs and what it is worth

Automation has a running cost that is easy to under-budget. A nightly geocode of a full case backlog, a state-wide exposure join and a prospective scan over several disease groups is hours of compute and a non-trivial storage bill, and it recurs. The budgeting question is not whether the pipeline is cheaper than the manual process — it usually is not, once maintenance is counted — but what the schedule buys. It buys three things a manual process cannot: a detection latency measured in a day rather than a reporting cycle, an audit trail that exists by construction, and a guarantee that the same method was applied to every week rather than whichever weeks somebody had time for.

Those benefits argue for a specific scope discipline. Run on a schedule the analyses whose value depends on timeliness or on consistency across periods — aberration detection, weekly access recalculation, quality scorecards. Leave on demand the analyses that are investigative, where a human is formulating a question each time. A pipeline that schedules exploratory work produces output nobody reads and costs the same as output somebody does.

Migrating an existing manual analysis

Most surveillance pipelines begin as a notebook that worked. The migration that succeeds moves in a fixed order: first make the notebook reproducible on the same inputs, pinning versions and removing interactive state; then split it into stages with explicit inputs and outputs; then add the schema and reconciliation assertions, which will fail immediately and reveal the informal fixes that were happening by hand; then schedule it; and only then optimise. Reordering those steps — scheduling first, assertions later — is how an organisation ends up with an automated system it does not trust and cannot stop using.

The step that surprises people is the third one. A notebook that has run monthly for two years will fail its first reconciliation assertion, because the analyst has been quietly correcting a mismatch each month without recording it. That discovery is the main value of the migration, and it arrives before any of the automation does.

The migration order described above is worth fixing in a diagram, because reordering it is the most common way an automation project goes wrong:

From Notebook to Pipeline, in Order Five steps left to right. First make the notebook reproducible on the same inputs by pinning versions and removing interactive state. Second split it into stages with explicit inputs and outputs. Third add schema and reconciliation assertions, which typically fail immediately and reveal informal monthly fixes. Fourth schedule it. Fifth optimise. A note under the third step records that its first failure is the main value of the migration, and a warning notes that scheduling before asserting produces an automated system nobody trusts. Assert before you schedule; optimise last 1 · Reproduce pin versions, drop hidden state 2 · Split stages with explicit inputs and outputs 3 · Assert schema and reconciliation 4 · Schedule only once it fails loudly 5 · Optimise with tests protecting it first failure reveals the monthly hand-fix the main value of the migration Scheduling before asserting automates a process nobody has verified

Operationalization & Output Standards

One run, one identifier, one provenance row. Every artifact the pipeline produces carries the run identifier that made it. That single convention answers most operational questions: which inputs produced this map, which code version, what were the parameters, did the gates pass.

Outputs are immutable. A run writes new artifacts rather than overwriting yesterday’s. Overwriting destroys the ability to compare, to roll back, and to explain what a recipient saw last week.

Publish with standards metadata. A map product leaving the agency needs its lineage, its spatial and temporal extent, its constraints on use, and its contact — the ISO 19115 content described in Documenting a Release With ISO 19115 Metadata. Without it an interagency recipient cannot tell a suppressed product from an unsuppressed one.

Serialize for the consumer. GeoParquet for analysts, cloud-optimized GeoTIFF for rasters, vector tiles for web maps, and a plain CSV with a data dictionary for the people who will open it in a spreadsheet regardless of what you provide.

Production Implementation Checklist

Frequently Asked Questions

Do I need a workflow orchestrator, or will cron do? Cron is adequate for a single job with no dependencies and no restart requirement. The moment one stage depends on another, or a failure should skip downstream work rather than run it on stale inputs, an orchestrator earns its complexity. The deciding question is not scale but whether a partial failure needs to be handled correctly.

How do I test spatial code when the correct answer is a geometry? By testing properties rather than values. A reprojection round-trip should return the original coordinates within tolerance; an areal interpolation should conserve totals; a buffer should contain its input. Those assertions are writable even when the expected output geometry is not, and they catch the errors that matter.

Should the pipeline run on protected data or on a de-identified copy? On protected data, inside the protected environment, with de-identification as a pipeline stage rather than a precondition. Building the pipeline against a de-identified copy means the geocoding and exposure joins — the stages that require real addresses — are never exercised by the automation.

What belongs in provenance? Enough to rebuild the run: input dataset identifiers and vintages, the code commit, library versions, every parameter, every random seed, and the gate results. If a reviewer with that row and access to the archive cannot reproduce the output, the provenance is incomplete.

How much should a pipeline do before a human looks at it? Everything up to publication, and nothing past it for a product that will be read as an official statement. A fully automated release is appropriate for internal dashboards and inappropriate for a public cluster alert, where the cost of an automated false positive is measured in community alarm rather than in a rerun.