Spatial Epi Public Health GIS

Serving a Read-Only Spatial API for Partner Agencies

This guide, part of Publishing Surveillance Map Products, designs the interface a health department offers to other agencies that want its surveillance aggregates programmatically — and closes the query patterns that would let them recover what suppression withheld.

Problem Context & Constraints

Partner agencies need machine access: a neighbouring county tracking cross-border spread, a state programme compiling regional figures, a federal partner building a national view. Emailing spreadsheets does not scale and produces inconsistent versions. An interface is the right answer, and a flexible one is the dangerous one.

The danger is differencing. Suppression is applied to a specific set of published cells. An interface that accepts arbitrary geographies or filters lets a caller request combinations the suppression stage never evaluated. Requesting a county total, then every tract in it but one, recovers the withheld tract by subtraction. Requesting a region with and without a single block group recovers that block group. Requesting overlapping time windows recovers a single suppressed week.

No individual response in those sequences is disclosive; the combination is. That means per-response checks cannot prevent the leak, and the protection has to be structural: the interface serves only the cells that were published, exactly as they were published.

Three further constraints apply. Read-only means read-only — no writes, no uploads, no parameters that reach a query engine as text. Logging must record who asked for what without recording what they got, so an access review is possible without the log becoming a second copy of the data. And rate limits exist to make systematic reconstruction impractical, not merely to protect the server.

Differencing Needs Two Requests the Suppression Never Saw On the left, a flexible interface accepts a request for a county total and a second request for the same county excluding one tract. Subtracting the two responses recovers the excluded tract's count, which the published table had suppressed. On the right, a fixed-cell interface accepts only requests for published cells as published, so the second request, an ad hoc geography, is rejected, and the suppressed tract remains withheld. Neither individual response on the left was disclosive; only the pair was. Each response is safe; the pair is not Flexible interface request 1: county total → 214 request 2: county minus tract T → 211 tract T recovered: 3 (was suppressed) no per-response check can see this Fixed-cell interface request 1: county total → 214 request 2: ad hoc geography → rejected tract T: suppressed, as published only evaluated cells can be requested The protection is structural: the interface cannot express the second query

Prerequisites

  • python 3.11
  • fastapi 0.111.0, pydantic 2.7.4, pandas 2.2.2, pyarrow 16.1.0
  • The published store: suppressed aggregates keyed by geography level, area identifier and period
  • An identity provider issuing per-partner credentials
  • A data-sharing agreement per partner specifying which levels and periods they may access

Step-by-Step Solution

Expose endpoints whose parameters are enumerations of published cells, validate every request against the partner’s agreement, log the request and not the response, and rate-limit per partner.

# Read-only partner interface over published cells only.
# Pinned: fastapi==0.111.0, pydantic==2.7.4, pandas==2.2.2, pyarrow==16.1.0
import logging
import time
from collections import defaultdict, deque
from enum import Enum

import pandas as pd
from fastapi import Depends, FastAPI, HTTPException, Request
from pydantic import BaseModel

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("publish.api")

PUBLISHED = pd.read_parquet("/publish/current/aggregates.parquet")  # suppressed store only


class Level(str, Enum):          # the ONLY geographies that can be asked for
    state = "state"
    county = "county"
    tract = "tract"


AGREEMENTS = {                    # from data-sharing agreements, per partner
    "partner-neighbour-county": {"levels": {"county", "tract"}, "min_period": "2024-01"},
    "partner-state-programme": {"levels": {"state", "county"}, "min_period": "2020-01"},
}
RATE = {"per_minute": 30, "per_day": 2000}
_hits: dict[str, deque] = defaultdict(deque)


class Cell(BaseModel):
    level: Level
    geoid: str
    period: str
    rate_class: int | None
    rate: float | None
    suppressed: bool
    provisional: bool


def partner(request: Request) -> str:
    pid = request.headers.get("x-partner-id")    # set by the identity gateway
    if pid not in AGREEMENTS:
        raise HTTPException(403, "unknown partner")
    now = time.time()
    q = _hits[pid]
    while q and now - q[0] > 86_400:
        q.popleft()
    last_min = sum(1 for t in q if now - t < 60)
    if last_min >= RATE["per_minute"] or len(q) >= RATE["per_day"]:
        raise HTTPException(429, "rate limit")
    q.append(now)
    return pid


app = FastAPI(title="Surveillance aggregates (read-only)")


@app.get("/v1/{level}/{period}", response_model=list[Cell])
def cells(level: Level, period: str, pid: str = Depends(partner)):
    ag = AGREEMENTS[pid]
    if level.value not in ag["levels"] or period < ag["min_period"]:
        raise HTTPException(403, "outside data-sharing agreement")
    sel = PUBLISHED[(PUBLISHED["level"] == level.value) & (PUBLISHED["period"] == period)]
    if sel.empty:
        raise HTTPException(404, "no published cells for that level and period")
    # Suppressed cells are returned as suppressed, with no value, exactly as published.
    sel = sel.assign(rate=sel["rate"].where(~sel["suppressed"]))
    # Log WHO asked for WHAT — never the response contents.
    log.info("partner=%s level=%s period=%s cells=%d", pid, level.value, period, len(sel))
    return sel[["level", "geoid", "period", "rate_class", "rate", "suppressed",
                "provisional"]].to_dict("records")

The enumerated Level type is the structural control. The interface has no parameter through which a caller can describe a geography, a filter or an exclusion; it can only name a published level and a published period and receive the published cells for that combination. The differencing sequence in the diagram cannot be expressed in the request language, which is a far stronger guarantee than trying to detect it in the request log.

Returning suppressed cells explicitly as suppressed, rather than omitting them, is also deliberate. An interface that silently drops suppressed rows invites the caller to infer them from totals; one that returns every published cell with its suppression flag makes the complete published picture unambiguous and gives the caller nothing further to infer.

Four Checks Before Any Cell Is Returned A request passes four checks in sequence. The identity gateway confirms which partner is calling. The agreement check confirms the requested level and period fall inside that partner's data-sharing agreement. The rate limiter confirms the partner is within per-minute and per-day limits. Finally the lookup returns only published cells for the requested level and period, with suppressed cells marked. A log entry records the partner, level, period and cell count, but never the returned values. Identity, agreement, rate, then published cells only Identity which partner Agreement level and period Rate limit per minute, per day Published cells for level and period suppressed cells marked, no values Access log: partner · level · period · cell count never the values returned — the log must not become a second copy Any failed check returns before the published store is read

Validation & Edge Cases

Complementary suppression across levels. Fixed cells prevent ad hoc differencing, but the published set itself must already be safe across levels — a county total and all its published tracts must not allow recovery of a suppressed tract. That is a property of the release, reviewed before publication, not something the interface can repair.

Provisional periods. A partner caching this week’s provisional counts will hold stale numbers next week. Return the provisional flag and a revision timestamp so partners know when to refresh.

Agreements that change. A partner whose agreement is narrowed must lose access immediately. Load agreements from a managed source at request time rather than from a file baked into the deployment.

Rate limits in memory across replicas. The in-memory limiter above is per process; behind several replicas a partner gets several times the limit. Use a shared store for counters in any multi-instance deployment.

Partners who also receive other products. A federal partner receiving the interface feed, the public web map and a quarterly spreadsheet can combine all three. If those products use different suppression passes — a different run, a different threshold — their union may allow recovery that none allows alone. Serve every channel from the same published store and the same run, so the partner receives one consistent release through several doors rather than several releases.

Versioning the interface. A change to the published geographies or to the fields returned breaks partner code that parses the response. Version the path, keep the previous version serving for a stated deprecation period, and announce changes through the same channel used for data-sharing agreements rather than letting partners discover them as errors.

Bulk export requests. Partners will ask for a full-history download to avoid paging through periods. Providing one is reasonable if it is exactly the published cells for every published period, generated from the same store and reviewed as the same release. It is not reasonable if it is produced by a separate query written for the occasion, which is how one-off extracts escape the suppression review.

Errors that reveal existence. Returning “cell suppressed” for a specific tract request tells the caller the tract has a small nonzero count if suppression applies only to small nonzero counts. Return suppressed cells uniformly within a full level response, which the design above does, rather than answering single-cell questions.

INFO publish.api partner=partner-neighbour-county level=tract period=2026-08 cells=1842

The replica point in that list is a common deployment oversight, and its effect on a per-partner limit is multiplicative:

Per-Process Limits Multiply Behind a Load Balancer A partner's requests spread by a load balancer across four replicas of the interface. When each replica keeps its own in-memory counter, each allows the full per-minute limit, so the partner's effective limit is four times what the agreement specifies. When all replicas read and increment a shared counter, the partner is limited to the agreed rate regardless of how requests are distributed. Four replicas with private counters allow four times the limit Private counters 30/min 30/min 30/min 30/min effective limit: 120/min four times the agreement Shared counter one store: 30/min effective limit: 30/min the agreement's limit holds only if every replica counts against the same total

Compliance Notes

Record each partner’s agreement — levels, periods, purpose, expiry — and tie the interface configuration to it. Retain the access log for the period the governing policy requires, and review it periodically for patterns that suggest reconstruction attempts even though the interface cannot express them: many requests across adjacent periods, or systematic sweeps of every level.

The interface serves the same published cells as the web map and downloadable tables, which means the disclosure review of the release as a whole covers it. What the interface adds is a need to confirm, at deployment, that it is connected only to the published store — a configuration test that queries for an internal column and expects failure is a cheap way to prove the service cannot see the analytic archive.