Spatial Epi Public Health GIS

Alerting on Pipeline Failures Without Leaking Protected Data

This guide, part of Orchestrating Spatial Surveillance Pipelines, addresses a disclosure route that almost every automated surveillance system has and almost no threat model covers: the alerting channel.

Problem Context & Constraints

An unhandled exception in a geocoding stage produces a traceback containing local variables, and those variables contain addresses. A validation error from a suppression stage prints the cell that failed, and that cell is a count of one. A monitoring system that ingests application logs then stores both outside the protected boundary, indexes them, and shows them to whoever can read the dashboard.

This is a disclosure incident with no attacker in it. It happens because the default behaviour of every logging and alerting stack is to be as informative as possible, and because the people configuring it are thinking about operability rather than confidentiality.

The constraint that resolves it is a strict separation between the fact of a failure and its detail. The fact — run identifier, stage, failure class, counts — is not protected and can travel anywhere an operator needs it. The detail — which record, which address, which cell — stays inside the protected environment and is retrieved by someone with access, using the run identifier as the key.

Two secondary constraints matter. The alert must still be actionable: “stage failed” with no class of failure sends the operator to the logs, which defeats the purpose. And the redaction must be structural rather than a filter, because a regular expression that strips things that look like addresses will miss some and will give false confidence about the rest.

The Fact Leaves, the Detail Stays A pipeline failure inside the protected environment splits into two paths. The detail record, containing the offending values, the traceback and the record identifiers, is written to protected storage keyed by run identifier and never crosses the boundary. The alert payload, containing only the run identifier, the stage name, a failure class, counts and a link back into the protected system, crosses the boundary to reach the operator's phone and the monitoring dashboard. A dashed line marks the protected boundary, and a note states that anything crossing it has been published. Anything that crosses the boundary has been published Protected environment Failure occurs in a stage locals hold addresses and record keys Detail record, keyed by run id traceback · offending values record identifiers · failing cells never leaves protected boundary Operator and dashboard Alert payload run id · stage · failure class affected count · elapsed time link into the protected system enough to decide what to do the operator opens the link and authenticates to see the detail

Prerequisites

  • python 3.11
  • pandas 2.2.2
  • A protected store for detail records, keyed by run identifier
  • An alerting transport that accepts structured fields rather than a free-text body
  • A defined failure taxonomy — the small set of classes an operator can act on
  • Agreement on which counts are safe to transmit, from the disclosure policy

Step-by-Step Solution

Raise structured exceptions that carry a class and a safe summary separately from their detail, write the detail inside the boundary, and emit only the safe summary.

# Structured failure handling: safe summaries out, detail inside the boundary.
# Pinned: pandas==2.2.2
import json
import logging
import os
import traceback
from dataclasses import dataclass, field

import pandas as pd

log = logging.getLogger("pipeline.alert")

# The operator can act on exactly these. A class outside the taxonomy is itself
# a defect: it means a failure mode nobody has decided how to respond to.
FAILURE_CLASSES = {
    "upstream_missing", "upstream_implausible", "schema_mismatch",
    "geometry_invalid", "reconciliation_failed", "suppression_failed",
    "service_unavailable", "quota_exhausted", "timeout", "internal_error",
}
# Counts below this are not transmitted even as counts, per the disclosure policy.
MIN_SAFE_COUNT = 11


@dataclass
class StageFailure(Exception):
    """A failure that knows what is safe to say about itself."""
    failure_class: str
    stage: str
    affected: int = 0
    detail: dict = field(default_factory=dict)   # NEVER transmitted

    def __post_init__(self):
        if self.failure_class not in FAILURE_CLASSES:
            raise ValueError(f"undeclared failure class: {self.failure_class!r}")

    def safe_summary(self, run_id: str) -> dict:
        """The payload that may cross the boundary. Nothing else does."""
        return {
            "run_id": run_id,
            "stage": self.stage,
            "failure_class": self.failure_class,
            # Small counts are themselves disclosive; report a band instead.
            "affected": (int(self.affected) if self.affected >= MIN_SAFE_COUNT
                         else f"<{MIN_SAFE_COUNT}"),
            "detail_ref": f"run://{run_id}/failures/{self.stage}",
        }


def record_failure(exc: BaseException, run_id: str, stage: str,
                   protected_dir: str) -> dict:
    """Write full detail inside the boundary; return only the safe summary."""
    if isinstance(exc, StageFailure):
        failure, summary = exc, exc.safe_summary(run_id)
    else:
        # An unexpected exception is internal_error; its message may contain
        # anything, so it is never used in the summary.
        failure = StageFailure(failure_class="internal_error", stage=stage,
                               detail={"message": str(exc)})
        summary = failure.safe_summary(run_id)

    os.makedirs(protected_dir, exist_ok=True)
    path = os.path.join(protected_dir, f"{run_id}.{stage}.failure.json")
    with open(path, "w", encoding="utf-8") as fh:
        json.dump({"summary": summary,
                   "detail": failure.detail,
                   "traceback": traceback.format_exc()}, fh, default=str)
    log.error("stage=%s class=%s run=%s detail_written=%s",
              summary["stage"], summary["failure_class"], run_id, path)
    return summary


def emit_alert(summary: dict, send_fn) -> None:
    """Transmit the summary. The transport never sees anything else."""
    allowed = {"run_id", "stage", "failure_class", "affected", "detail_ref"}
    extra = set(summary) - allowed
    if extra:
        # Fail rather than transmit an unexpected field that somebody added.
        raise ValueError(f"alert payload contains undeclared fields: {sorted(extra)}")
    send_fn(summary)
    log.info("alert emitted run=%s stage=%s class=%s",
             summary["run_id"], summary["stage"], summary["failure_class"])

The whitelist in emit_alert is the structural control. Redaction by pattern matching fails eventually because somebody adds a helpful field; a whitelist fails safely, because the unexpected field raises rather than transmits. It is the difference between a policy that depends on everyone remembering and one that depends on the code.

Banding small counts matters more than it first appears. “Suppression failed for 3 cells” is a statement about three cells containing small numbers, and in a sparse disease that is close to a statement about three individuals. Reporting “fewer than 11” costs the operator nothing and removes the question entirely.

A Failure Class Is a Decision, Not a Description A table mapping failure classes to operator actions. Upstream missing means wait and retry later. Upstream implausible means contact the data provider and do not re-run. Schema mismatch means the pipeline needs a code change and must not be retried. Reconciliation failed means investigate before any publication. Suppression failed means nothing publishes and the disclosure officer is informed. Service unavailable and quota exhausted mean retry with backoff. Internal error means escalate to the developer on call. The point of the taxonomy is that every class has exactly one correct response. If a class does not imply an action, it is not a class failure class what the operator does upstream_missing wait, retry later, no escalation upstream_implausible contact the provider, do not re-run schema_mismatch code change needed, retry is futile reconciliation_failed investigate before anything publishes suppression_failed nothing publishes; inform disclosure officer quota_exhausted retry with backoff, review the budget A class nobody has an action for becomes a class nobody responds to

Validation & Edge Cases

Third-party libraries logging at debug level. A spatial or HTTP library configured verbosely will log request bodies containing addresses. Set third-party logger levels explicitly rather than inheriting the root level, and review what a debug-enabled run actually emits.

Log aggregation configured after the fact. An agent shipping application logs to an external platform exports everything the application ever logged. Decide what the application logs before deciding where the logs go, and confirm the destination is inside the compliance boundary.

Exception messages constructed from data. A validation library that raises “value ‘JOHN SMITH, 14 ELM ST’ failed pattern” has put the value in the message, and the message goes everywhere. Wrap such libraries so their exceptions are caught and re-raised as a declared class with the value in the detail field.

Metrics that are themselves disclosive. A monitoring dashboard that graphs cases per tract per night is a live small-area count series outside the protected boundary. Operational metrics should be pipeline metrics — rows processed, stage duration, match rate, gate outcomes — and not epidemiological ones. Where a count must be graphed for operational reasons, aggregate it to a level where no cell can be small, and review the dashboard against the same disclosure policy that governs a published map.

On-call handover without access. A content-free alert is only workable if the person receiving it can actually reach the detail. An operator rota that includes people without access to the protected system will, predictably, lead to detail being copied out to them. Either grant the access or route those classes to someone who has it, and record which classes each rota member can action.

Alert transports with retention. A paging service stores message history. Content that seemed transient is retained on a third-party system for years. Treat the alert payload as permanently published.

Screenshots and pasted logs. The most common leak is human: an operator pastes a traceback into a chat channel to ask for help. A content-free alert makes that harmless, which is another reason the redaction has to be at the source rather than at the transport.

ERROR pipeline.alert stage=suppress class=suppression_failed run=20260918-4f2a1c8e detail_written=/protected/failures/20260918-4f2a1c8e.suppress.failure.json
INFO pipeline.alert alert emitted run=20260918-4f2a1c8e stage=suppress class=suppression_failed

The difference between the two payloads is easiest to judge side by side, reading each as if it had already been forwarded to someone outside the agency:

Read Each Alert as if It Were Already Forwarded Two alert payloads for the same failure. The default payload is a traceback whose local variables include a normalised address, a record key and a count of three in a suppressed cell, all of which leave the protected boundary with the alert. The structured payload contains only the run identifier, the stage name, the failure class, an affected count reported as fewer than eleven, and a link back into the protected system. Both let an operator decide what to do; only one is safe to receive on a phone. Both are actionable; only one is safe outside the boundary Default traceback ValueError in geocode_batch addr = 14 ELM ST APT 3 record_key = R-00418822 cell count = 3 three disclosures in one message Structured payload run_id: 20260918-4f2a stage: geocode class: upstream_implausible affected: fewer than 11 detail: run://20260918-4f2a/… nothing identifying leaves the operator authenticates to the protected system to see the detail

Compliance Notes

Record the failure taxonomy and the action associated with each class, the alert payload schema with its whitelist, the minimum safe count, the location and retention of detail records, the logger configuration for third-party libraries, and the destination and retention of the alerting transport.

Treat the alerting channel as an export route in the system’s data flow documentation, because that is what it is. An agency that maps its disclosure paths through published products and access requests, and omits the monitoring stack, has an unmapped route that carries data every night. Reviewing a week of real alert payloads against the whitelist is a cheap and unusually effective audit — it finds both the helpful extra field somebody added and the library that is logging more than anyone intended.