Spatial Epi Public Health GIS

Versioning Boundary Files Across Vintages

This guide, part of Cloud-Native Geospatial Storage for Surveillance, covers the archive whose identity is a version rather than a date: the boundary, denominator and crosswalk layers every analysis joins to.

Problem Context & Constraints

Reference geography is not static. Census tracts are redrawn each decade and revised in between; ZIP code areas change continuously; service areas expand; state reporting regions are reorganised. An analysis that joined to “the tract file” is reproducible only if that phrase still resolves to the same bytes, and in most agencies it does not.

The failure is quiet and recurrent. Someone downloads a newer boundary file into the same path, the nightly pipeline picks it up, and county totals shift slightly because a few blocks moved. Nothing errors. The shift appears in a trend as a small step, and it is attributed to the data.

Three constraints define a workable store. Vintages are append-only — a file is never replaced, only superseded, because the previous file is the only thing that can reproduce prior output. Each run binds one vintage explicitly, recorded in provenance, so the question “which boundaries did the March run use” always has an answer. And crosswalks between vintages are themselves versioned artifacts, because a crosswalk is a modelling product with assumptions, not a lookup table.

Vintages Accumulate; Runs Point at One Three boundary vintages stored side by side and never overwritten, each with its own identifier and effective period. Two pipeline runs are shown below, each drawing an arrow to exactly one vintage: an older run bound to the earlier vintage and the current run bound to the latest. Between the vintages sit crosswalk artifacts, themselves versioned, which allow counts on one vintage to be restated on another. A note records that replacing a vintage in place would break the older run's reproducibility without any error being raised. A superseded vintage is still load-bearing tracts@2010 effective 2012–2021 superseded, retained tracts@2020 effective 2022–2024 superseded, retained tracts@2020r3 effective 2025 onward current xwalk xwalk run 20210914-a3f1 still reproducible today run 20260918-4f2a bound to the current vintage Overwriting the left-hand file breaks the left-hand run, silently and permanently

Prerequisites

  • python 3.11
  • geopandas 0.14.4, pandas 2.2.2, pyarrow 16.1.0
  • Object or file storage supporting immutable writes, ideally with object-lock or a write-once policy
  • A vintage naming convention with an effective period per vintage
  • The run context from Orchestrating Spatial Surveillance Pipelines

Step-by-Step Solution

Register vintages with a content hash, resolve a vintage for each run from its logical date or an explicit pin, and assert on read that the resolved vintage matches what provenance recorded.

# Append-only boundary vintage store with a resolver and drift assertions.
# Pinned: geopandas==0.14.4, pandas==2.2.2, pyarrow==16.1.0
import hashlib
import logging
import os

import geopandas as gpd
import pandas as pd

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("storage.vintage")

REGISTRY = "vintages.parquet"


def _content_hash(path: str, chunk: int = 1 << 20) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()


def register_vintage(root: str, layer: str, vintage: str, path: str,
                     effective_from: str, effective_to: str | None,
                     source: str) -> dict:
    """Add a vintage. Refuses to replace an existing one."""
    reg_path = os.path.join(root, REGISTRY)
    reg = pd.read_parquet(reg_path) if os.path.exists(reg_path) else pd.DataFrame(
        columns=["layer", "vintage", "path", "sha256", "effective_from",
                 "effective_to", "source", "registered_utc"])
    clash = reg[(reg["layer"] == layer) & (reg["vintage"] == vintage)]
    if len(clash):
        raise FileExistsError(
            f"{layer}@{vintage} is already registered with sha256 "
            f"{clash.iloc[0]['sha256'][:16]} — vintages are append-only; "
            "publish a new vintage instead of replacing this one")
    row = {"layer": layer, "vintage": vintage, "path": path,
           "sha256": _content_hash(path), "effective_from": effective_from,
           "effective_to": effective_to, "source": source,
           "registered_utc": pd.Timestamp.utcnow().isoformat()}
    out = pd.concat([reg, pd.DataFrame([row])], ignore_index=True)
    tmp = reg_path + ".tmp"
    out.to_parquet(tmp, index=False)
    os.replace(tmp, reg_path)
    log.info("registered %s@%s sha256=%s effective %s..%s",
             layer, vintage, row["sha256"][:16], effective_from, effective_to or "open")
    return row


def resolve_vintage(root: str, layer: str, as_of: pd.Timestamp,
                    pin: str | None = None) -> dict:
    """The vintage a run should use: an explicit pin, else the one effective then."""
    reg = pd.read_parquet(os.path.join(root, REGISTRY))
    lay = reg[reg["layer"] == layer]
    if pin:
        sel = lay[lay["vintage"] == pin]
        if sel.empty:
            raise KeyError(f"pinned vintage {layer}@{pin} is not registered")
        log.info("resolved %s to pinned vintage %s", layer, pin)
        return sel.iloc[0].to_dict()
    eff = lay[(pd.to_datetime(lay["effective_from"]) <= as_of)
              & (lay["effective_to"].isna()
                 | (pd.to_datetime(lay["effective_to"]) >= as_of))]
    if eff.empty:
        raise KeyError(f"no {layer} vintage effective at {as_of.date()}")
    row = eff.sort_values("effective_from").iloc[-1].to_dict()
    log.info("resolved %s at %s to vintage %s", layer, as_of.date(), row["vintage"])
    return row


def load_vintage(row: dict) -> gpd.GeoDataFrame:
    """Load and assert the file has not changed since it was registered."""
    actual = _content_hash(row["path"])
    if actual != row["sha256"]:
        raise AssertionError(
            f"{row['layer']}@{row['vintage']} content changed: registered "
            f"{row['sha256'][:16]}, found {actual[:16]} — an append-only store "
            "has been modified in place")
    gdf = gpd.read_parquet(row["path"])
    log.info("loaded %s@%s features=%d crs=%s", row["layer"], row["vintage"],
             len(gdf), gdf.crs.to_string() if gdf.crs else "none")
    return gdf

The content hash assertion is what turns a convention into a control. Append-only is a policy that someone will eventually violate with a well-intentioned correction, and the hash check converts that from a silent change in every downstream result into an immediate, specific failure naming the layer and vintage.

Resolving by effective period rather than by “latest” is the other half. A run reprocessing 2019 asks for the tract layer effective in 2019 and receives it automatically, which makes historically correct backfills the default behaviour rather than something the operator has to remember.

A Crosswalk Is a Model, Not a Lookup Two boundary vintages over the same area, with the newer one splitting one old area into two and merging two others into one. A crosswalk allocates counts between them using population weights. The split is handled by apportioning the old count across the two new areas, which introduces uncertainty because the true within-area distribution is unknown. The merge is exact, because summing is lossless. A note records that a crosswalk therefore has a direction: aggregating upward is exact and splitting downward is an estimate with error. Merging is exact; splitting is an estimate older vintage area P: 240 Q: 60 R: 90 split merge newer vintage P1: 142 ± e P2: 98 ± e QR: 150 exact Therefore store the crosswalk as a versioned artifact with its weighting method, and record that split allocations carry error that merges do not

Validation & Edge Cases

A vintage registered without an effective period. The resolver then cannot pick it automatically and every run must pin. Require the period at registration.

Overlapping effective periods. Two vintages effective at the same date make resolution ambiguous. Assert non-overlap at registration time rather than discovering it during a backfill.

A corrected release of the same vintage. Agencies do reissue files with fixes. That is a new vintage — a revision suffix — not a replacement, and analyses that used the original remain reproducible.

Coordinate reference system differing between vintages. A newer release may ship in a different system. Normalise on registration and record both the source system and the stored one.

Identifier churn without geometry change. Some releases renumber areas whose boundaries did not move. A crosswalk is still required, and a naive join on identifier silently drops or misassigns rows.

Consumers that cache a vintage locally. A partner agency or an analyst who downloaded a boundary file keeps using it after it is superseded, and their figures stop matching yours with no visible reason. Publish the vintage identifier and content hash alongside every boundary file you distribute, so a mismatch between two reports can be diagnosed in minutes by comparing a hash rather than in days by comparing maps.

Denominators released on a different cycle. Population estimates are revised annually while boundaries change once a decade, so a rate combines two layers with independent vintage histories. Register denominators in the same store with their own effective periods, and record both vintages for every rate a run produces; a rate whose numerator geography and denominator vintage disagree is a common and invisible source of drift in trend reports.

Crosswalk used in the wrong direction. Aggregating fine areas up is exact; splitting coarse areas down is an estimate. A crosswalk table that does not record which direction it was built for will be used both ways.

INFO storage.vintage registered tracts@2020r3 sha256=9c41ab77e0d3f215 effective 2025-01-01..open
INFO storage.vintage resolved tracts at 2019-06-30 to vintage 2010
ERROR storage.vintage tracts@2020 content changed: registered 4f81c2a90b6d7e33, found a7702ce41b98d550 — an append-only store has been modified in place

The overlapping-period check in that list is simplest to state as a timeline in which every date resolves to exactly one vintage:

Every Date Must Resolve to Exactly One Vintage Two sets of boundary vintage effective periods on a timeline. In the valid set, three vintages follow one another with no gap and no overlap, so any date resolves to exactly one. In the invalid set, a revised vintage was registered with a start date before the previous vintage's end date, creating an overlap where two vintages are both effective, so automatic resolution is ambiguous. Registration must reject the overlap rather than leaving the resolver to pick one arbitrarily. Reject overlapping periods at registration, not at resolution valid 2010 2020 2020r3 invalid 2010 2020 2020r3, starts too early ambiguous dates 2012 2022 2026 effective periods of three tract vintages

Compliance Notes

Record for every vintage: the layer, the vintage identifier, the source and its citation, the content hash, the effective period, the stored coordinate reference system and the registration timestamp. Record for every run: which vintage of each layer it resolved, and whether the resolution was automatic or pinned.

The vintage store should be configured write-once at the storage layer where the platform supports it, because the policy is exactly the kind that erodes under time pressure. Retention here runs the opposite way from protected data: boundary vintages are not disclosive and must be kept indefinitely, since deleting a superseded vintage destroys the reproducibility of every analysis that used it. Budgeting for that permanence is cheap — reference geography is small relative to the case archive — and failing to plan for it is how an agency discovers it cannot reproduce a figure it published four years ago.