Spatial Epi Public Health GIS

Documenting a Release With ISO 19115 Metadata

This guide, part of Publishing Surveillance Map Products, generates standards-based metadata for every published product directly from the pipeline’s provenance, so that documentation cannot drift from what was actually done.

Problem Context & Constraints

Metadata written by hand has two failure modes. It is written once and never updated, so the published description of a weekly product describes the week it was first documented. Or it is not written at all, because it is tedious, and the product ships with a filename and nothing else.

Both leave a recipient unable to answer questions that determine whether the data are usable: what geography vintage is this, which weeks are provisional, were small cells suppressed and at what threshold, what coordinate system are the geometries in, who do I contact. An interagency recipient who cannot answer those will either not use the product or, worse, use it wrongly.

ISO 19115 is the international standard for geographic metadata, and its structure maps closely onto what a surveillance pipeline already records. Identification maps to the product; extent to the boundary set and the reporting period; lineage to the run’s stages and inputs; constraints to the disclosure policy and data-sharing terms; data quality to the gate results. The constraint that makes this practical is to generate the metadata from the provenance row rather than author it, so that every release is documented and every document is correct for its release.

Provenance Already Contains the Metadata Two columns connected by mapping lines. On the left, fields from a pipeline run's provenance record: product identifier and title, boundary vintage and coordinate system, reporting period and provisional date, input datasets and stages, disclosure policy and threshold, and gate results. On the right, the corresponding ISO 19115 sections: identification information, spatial extent and reference system, temporal extent, lineage with sources and process steps, constraints on use, and data quality. Every metadata section is filled from something the pipeline already records, so nothing needs to be written by hand. Every section is filled from something the pipeline already recorded Run provenance ISO 19115 section product id, title, revision identification boundary vintage, CRS spatial extent, reference system period, provisional from temporal extent inputs, stages, commit lineage: sources, process steps disclosure policy, threshold constraints on use gate results data quality

Prerequisites

  • python 3.11
  • lxml 5.2.2 for XML generation, pandas 2.2.2
  • The run’s provenance record with inputs, stages, parameters and gate results
  • The product configuration: identifier, title, abstract, contact, keywords
  • The disclosure policy reference and any data-sharing constraints

Step-by-Step Solution

Build a metadata document from provenance and product configuration, with lineage expressed as a sequence of process steps, and write it beside the product with the same run identifier.

# Generate ISO 19115-style metadata from run provenance.
# Pinned: lxml==5.2.2, pandas==2.2.2
import json
import logging
from datetime import datetime, timezone

from lxml import etree

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("publish.metadata")

NS = {"mdb": "http://standards.iso.org/iso/19115/-3/mdb/2.0",
      "cit": "http://standards.iso.org/iso/19115/-3/cit/2.0",
      "gco": "http://standards.iso.org/iso/19115/-3/gco/1.0",
      "mri": "http://standards.iso.org/iso/19115/-3/mri/1.0",
      "gex": "http://standards.iso.org/iso/19115/-3/gex/1.0",
      "mrl": "http://standards.iso.org/iso/19115/-3/mrl/2.0",
      "mco": "http://standards.iso.org/iso/19115/-3/mco/1.0",
      "mdq": "http://standards.iso.org/iso/19157/-2/mdq/1.0"}


def _el(parent, tag, text=None):
    ns, name = tag.split(":")
    e = etree.SubElement(parent, etree.QName(NS[ns], name))
    if text is not None:
        cs = etree.SubElement(e, etree.QName(NS["gco"], "CharacterString"))
        cs.text = str(text)
    return e


def build_metadata(product: dict, prov: dict) -> etree._Element:
    root = etree.Element(etree.QName(NS["mdb"], "MD_Metadata"), nsmap=NS)
    _el(root, "mdb:metadataIdentifier", f"{product['product_id']}:{prov['run_id']}")
    _el(root, "mdb:dateInfo", datetime.now(timezone.utc).isoformat())

    ident = _el(root, "mdb:identificationInfo")
    di = _el(ident, "mri:MD_DataIdentification")
    cit = _el(_el(di, "mri:citation"), "cit:CI_Citation")
    _el(cit, "cit:title", product["title"])
    _el(cit, "cit:edition", f"rev {product['revision']} run {prov['run_id']}")
    _el(di, "mri:abstract", product["abstract"])
    _el(di, "mri:purpose", product.get("purpose", "Public health surveillance"))
    _el(di, "mri:pointOfContact", product["contact"])

    ext = _el(_el(di, "mri:extent"), "gex:EX_Extent")
    _el(ext, "gex:description",
        f"{prov['geography_level']} areas, boundary vintage {prov['boundary_vintage']}")
    _el(ext, "gex:temporalElement",
        f"{prov['period_start']} to {prov['period_end']}; provisional from "
        f"{prov['provisional_from']}")
    _el(root, "mdb:referenceSystemInfo", prov["crs"])

    lin = _el(_el(root, "mdb:resourceLineage"), "mrl:LI_Lineage")
    _el(lin, "mrl:statement", product["lineage_statement"])
    for src in prov["inputs"]:
        s = _el(lin, "mrl:source")
        _el(s, "mrl:description", f"{src['name']} vintage {src['vintage']} "
                                  f"sha256 {src['sha256'][:16]}")
    for step in prov["stages"]:
        ps = _el(lin, "mrl:processStep")
        _el(ps, "mrl:description", f"{step['name']}: {step['summary']}")

    con = _el(_el(root, "mdb:resourceConstraints"), "mco:MD_Constraints")
    _el(con, "mco:useLimitation",
        f"Cells with counts below {prov['suppression_threshold']} are suppressed "
        f"under {prov['disclosure_policy']}. Complementary suppression applied. "
        "Do not attempt to recover suppressed values.")
    for c in product.get("use_constraints", []):
        _el(con, "mco:useLimitation", c)

    dq = _el(_el(root, "mdb:dataQualityInfo"), "mdq:DQ_DataQuality")
    for g in prov["gates"]:
        _el(dq, "mdq:report",
            f"{g['name']} ({g['family']}): {'pass' if g['passed'] else 'FAIL'} — {g['detail']}")
    return root


def write_metadata(product: dict, prov: dict, out_path: str) -> None:
    doc = build_metadata(product, prov)
    failed = [g["name"] for g in prov["gates"] if not g["passed"] and g["severity"] == "halt"]
    if failed:
        raise RuntimeError(f"refusing to document a release whose halting gates failed: {failed}")
    etree.ElementTree(doc).write(out_path, pretty_print=True, xml_declaration=True,
                                 encoding="UTF-8")
    log.info("metadata written %s for %s run %s (%d sources, %d steps, %d gates)",
             out_path, product["product_id"], prov["run_id"], len(prov["inputs"]),
             len(prov["stages"]), len(prov["gates"]))

Refusing to write metadata for a release whose halting gates failed is a small guard with a useful side effect. Metadata generation happens at the end of publication, so if it refuses, the product has no documentation and the promotion step — which should require metadata — cannot proceed. It turns a missed gate into a visible, blocking failure at the last possible point.

The lineage section carries a content hash for each input rather than only its name. Two releases whose lineage names the same boundary file but different hashes used different geometry, and a recipient reconciling figures between releases can see that without contacting anyone.

What the Recipient Can Now Answer A list of questions a receiving agency asks about a surveillance product, each paired with the metadata section that answers it. Which boundaries: spatial extent with vintage. Which weeks are final: temporal extent with provisional date. Were small cells hidden and at what threshold: constraints on use. What coordinate system are the geometries in: reference system. Did the quality checks pass: data quality reports. Who do I ask: point of contact. Without the metadata each question requires an email and a wait. Six questions a recipient no longer has to email about recipient's question answered by Which boundaries are these? extent: level and vintage Which weeks are final? temporal extent: provisional from Were small cells hidden? constraints: threshold and policy What coordinate system? reference system Did the checks pass? data quality reports Who do I ask? point of contact Generated per release, so the answers are correct for the file in hand

Validation & Edge Cases

Schema validation. Generated metadata should be validated against the standard’s schema as part of the build. A document that does not validate may be silently ignored by catalogue software, which defeats its purpose.

Contact information that goes stale. A named individual leaves; a role mailbox does not. Use role-based contacts in the product configuration.

Lineage that exposes internals. Process step descriptions should say what was done — “geocoded to rooftop where possible; unassignable records excluded” — without naming internal hostnames, storage paths or credentials. Review the stage summary templates once, since they repeat in every release.

Provisional status omitted. A recipient who cannot tell final weeks from provisional ones will compare them. The temporal extent must always state the provisional boundary, even when it is “none”.

Keywords and discoverability. Catalogue software indexes metadata by keyword and subject category, and a document with none is effectively invisible in a partner’s data catalogue. Draw keywords from a controlled vocabulary in the product configuration — condition, geography level, theme — rather than generating them from free text, so the same product is findable the same way every week.

Multiple formats for one release. A release often ships as tiles, a table and a static map. Each needs its own identification section, but all should share one lineage and one constraints section and reference each other as related resources, so a recipient holding the table can find the map it corresponds to and confirm both came from the same run.

Corrections and supersession. When a release is corrected, the new metadata should reference the superseded release’s identifier and state what changed. Without that link, a catalogue holds two documents for the same period with no indication of which is current.

Metadata detached from the product. Write the document beside the product with the same run identifier in its filename, and reference it from the product itself where the format allows — a raster tag, a GeoParquet key-value entry — so the two can be reassociated if separated.

INFO publish.metadata metadata written /publish/runs/20260918-4f2a/tracts.xml for ILI-TRACT-WEEKLY run 20260918-4f2a (4 sources, 6 steps, 11 gates)

The related-resources and supersession links in that list turn a pile of documents into a navigable record of what was released and what replaced it:

Link the Products of One Release, and the Release It Replaced Three metadata documents for one weekly release, describing the tile set, the downloadable table and the static map, share one lineage and one constraints section and reference one another as related resources. A correction release is shown beside them, whose metadata references the superseded release's identifier and states what changed. A catalogue reading these links can show a user holding the table which map it corresponds to, and whether a newer version exists. Related-resource and supersession links make the catalogue navigable tiles metadata run 0911-a2 table metadata run 0911-a2 map metadata run 0911-a2 shared lineage and constraints related resources correction release run 0913-c7 supersedes 0911-a2 states what changed

Compliance Notes

Generated metadata is itself a published artifact and falls under the same review as the product. Its constraints section is where the agency’s disclosure commitments become a statement to recipients — the suppression threshold, the policy that sets it, and an explicit request not to attempt recovery of suppressed values. That statement is not a technical control, but in many data-sharing arrangements it is the clause that makes a recovery attempt a breach of terms rather than merely a clever analysis.

Record in the product registry, for every release, the metadata document’s identifier and a hash of its content. When a question arises later about what a recipient was told — the provisional boundary, the threshold, the boundary vintage — the registry answers it precisely, and the answer is the document they actually received rather than a reconstruction.