Spatial Epi Public Health GIS

GeoParquet vs GeoPackage for Surveillance Archives

An archival format decision looks like an engineering preference and behaves like a data-governance one: it determines which metadata survives, whether a query five years from now can read a subset without loading everything, and whether the file can be opened at all by whatever tooling exists then. This guide, part of Spatial Data Types & Formats, compares the two serious candidates for surveillance archives.

Problem Context & Constraints

Both formats are open specifications with multiple independent implementations, both preserve a CRS properly, and both handle mixed geometry types. They differ on four axes that matter for an archive.

Type fidelity. GeoPackage is SQLite, so column types are SQLite’s: five storage classes with dynamic typing. GeoParquet is Apache Parquet, with a rich logical type system including proper dates, timestamps with timezone, decimals and nested structures. For surveillance data carrying dates, categorical codes and nullable integers, that difference is substantial — a nullable integer count round-trips exactly in Parquet and becomes a float in many GeoPackage workflows.

Partial reads. Parquet is columnar and supports predicate pushdown and column projection, so reading one year and three columns from a twenty-year archive touches a fraction of the bytes. GeoPackage requires an index and a query, which works, but the file is a single object that must be present in full.

Partitioning. A Parquet dataset is naturally a directory partitioned by year or county, so a new period is a new file rather than a rewrite. A GeoPackage grows, and a growing SQLite file is a concurrency and backup liability.

Editability and tooling breadth. GeoPackage is a working format: desktop GIS opens it, edits it and writes it back. GeoParquet is read-mostly, and while support is now broad it is younger.

Where the Two Formats Actually Differ Five criteria compared. Type fidelity favours GeoParquet strongly because SQLite's dynamic typing loses nullable integers and proper timestamps. Partial reads favour GeoParquet, which supports column projection and predicate pushdown. Partitioning favours GeoParquet, since a dataset is a directory rather than one growing file. Editability favours GeoPackage, which desktop GIS can open and write. Archive size favours GeoParquet by roughly a factor of two and a half after compression. Four of five criteria point one way; the fifth is why both exist GeoPackage GeoParquet Type fidelity dynamic typing; ints drift logical types preserved Partial reads whole file must be present column + predicate pushdown Partitioning one growing file directory partitioned by year Editability desktop GIS reads and writes read-mostly Archive size 540 MB 210 MB

Prerequisites

  • python 3.11, geopandas 1.0.1, pyarrow 16.1.0, shapely 2.0.6
  • A declared archival CRS, chosen per Choosing an Equal-Area Projection for Rate Mapping where areas matter
  • A schema for the archive — column names, types and nullability — written down before the first write

The operational difference between the two formats shows up most clearly in the question an archive is actually asked:

Read Time for a Three-Year Slice of a Twenty-Year Archive Time to answer a three-year, four-column question from a twenty-year archive. A single GeoPackage must open the whole file and takes about 41 seconds. An unpartitioned GeoParquet file uses column projection and takes 12 seconds. A GeoParquet dataset partitioned by year touches only three files and four columns and takes 1.8 seconds. The difference is not compression; it is how much of the archive has to be read at all. Answering a 3-year, 4-column question GeoPackage 41 s — whole file opened GeoParquet, one file 12 s — columns only GeoParquet, by year 1.8 s — three files, four columns Partitioning is what makes a twenty-year archive queryable rather than merely stored

Step-by-Step Solution

# Write a partitioned GeoParquet surveillance archive with a checked schema.
# Pinned: geopandas==1.0.1, pyarrow==16.1.0, pandas==2.2.2
import logging
import pandas as pd
import geopandas as gpd
import pyarrow as pa

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("format.archive")

# The schema is the contract. Declaring it means a column whose type drifts
# between years fails on write rather than being discovered on read in 2031.
ARCHIVE_SCHEMA = {
    "geoid": "string",
    "period": "date32[day][pyarrow]",
    "observed": "int32",
    "expected": "float64",
    "suppressed": "bool",
    "method_version": "string",
}


def check_schema(gdf: gpd.GeoDataFrame) -> None:
    missing = set(ARCHIVE_SCHEMA) - set(gdf.columns)
    if missing:
        raise ValueError(f"archive schema requires {sorted(missing)}")
    for col, want in ARCHIVE_SCHEMA.items():
        got = str(gdf[col].dtype)
        if got != want:
            raise TypeError(f"column {col!r} is {got}, archive requires {want}. "
                            "Cast explicitly rather than letting the writer guess.")
    log.info("schema check passed on %d rows", len(gdf))


def write_partition(gdf: gpd.GeoDataFrame, root: str, year: int,
                    archive_crs: str = "EPSG:5070") -> str:
    """One file per year. A new period never rewrites an existing one."""
    check_schema(gdf)
    if gdf.crs is None:
        raise ValueError("refusing to archive geometry with no CRS")
    out = gdf.to_crs(archive_crs)
    path = f"{root}/year={year}/part-0.parquet"
    out.to_parquet(path, compression="zstd", write_covering_bbox=True, index=False)
    log.info("wrote %s (%d rows, CRS %s)", path, len(out), archive_crs)
    return path


def read_slice(root: str, years: list[int], columns: list[str]) -> gpd.GeoDataFrame:
    """Read only the years and columns needed.

    This is the operational payoff: a twenty-year archive answers a three-year,
    four-column question without decompressing the other seventeen years."""
    cols = list(dict.fromkeys(columns + ["geometry"]))
    gdf = gpd.read_parquet(root, columns=cols,
                           filters=[("year", "in", years)])
    log.info("read %d rows from %d year(s), %d column(s)", len(gdf), len(years), len(cols))
    return gdf

write_covering_bbox=True is worth calling out: it stores a bounding-box column that lets a reader do spatial predicate pushdown, so a county-level question does not read the state. Without it, GeoParquet’s spatial querying is no better than a full scan.

Validation & Edge Cases

1. Round-trip the schema before committing to a format. Write a representative frame, read it back, and compare dtypes exactly. This is where GeoPackage’s dynamic typing shows up:

INFO schema check passed on 18,204 rows
INFO wrote archive/year=2026/part-0.parquet (18204 rows, CRS EPSG:5070)
INFO read 54,612 rows from 3 year(s), 5 column(s)
WARNING gpkg round trip: 'observed' int32 -> int64, 'suppressed' bool -> int64, 'period' date32 -> object

2. Keep a GeoPackage export for distribution, not for archive. The two roles are different. Archive in GeoParquet and generate a GeoPackage on demand for partners whose tooling needs it, recording which archive version each export came from.

3. Do not partition too finely. A partition per county per month produces tens of thousands of tiny files and reads slower than one file per year. Year, or year and state, is usually right.

4. Store the CRS in the file, not in a README. Both formats carry it properly; the failure is a pipeline that writes without setting it, which is why the code above refuses geometry with no CRS.

5. Plan for the format outliving its library. Both are open specifications with independent implementations, which is the property that matters. Verify that at least two independent readers can open the archive, and record the specification version in the archive metadata.

6. Decide the geometry encoding explicitly. GeoParquet supports well-known binary and, in newer versions, native nested coordinate encodings. The choice affects which readers can open the file and how efficiently a bounding-box filter runs, and it is easy to leave to a library default that changes between versions. Pin the encoding in the write call and record it with the schema version.

7. Keep an integrity manifest. An archive spanning a decade will be copied between storage systems several times, and silent corruption in one partition is invisible until somebody reads it. A manifest of per-file checksums, verified on a schedule, turns that from a discovery into an alert.

8. Test the read path, not only the write path. An archive is written once and read for years, so the check that matters is that a fresh environment with only the pinned dependencies can open a partition, resolve its CRS and reproduce a known summary statistic. Running that check quarterly is inexpensive and is the only thing that actually establishes the archive is still usable.

9. Write the archive from a single code path. Where several pipelines contribute to one archive, each will drift toward its own conventions for nullability, string encoding and column order, and the schema check will start failing for reasons nobody wants to debug. A single writer module that every pipeline calls is a small amount of shared code that keeps the archive coherent for its whole life.

10. Decide the retention rule before the archive is large. An archive with no stated retention policy grows until somebody deletes something under pressure, which is the worst moment to be making that decision. Write the rule down at the same time as the schema, and make it specific about which partitions are permanent and which may be aggregated after a stated period.

The layout itself carries much of the governance, so it is worth drawing:

Archive Layout on Disk An archive laid out as a directory partitioned by year, each partition holding one Parquet file, alongside a schema document, an integrity manifest of per-file checksums, and a README recording the archival coordinate reference and the retention rule. Adding a year adds a directory and a manifest line; nothing existing is rewritten, so a published figure computed from an earlier partition stays reproducible. Adding a year adds a directory, and rewrites nothing surveillance-archive/ year=2024/ part-0.parquet year=2025/ part-0.parquet year=2026/ added this release … seventeen more untouched schema.json types + version manifest.csv per-file checksums README archival CRS + retention rule Everything a reader in 2036 needs sits beside the data, not in a wiki

Compliance Notes

  • Version the archive, never overwrite a period. A corrected year is a new file with a new version, and the superseded one stays retrievable, since published figures were computed from it.
  • Record the schema and its version in the archive metadata, so a reader in five years knows what the columns meant.
  • Keep the archival CRS fixed for the life of the archive. Changing it mid-series makes areas incomparable across years even though every file is individually correct.
  • Treat the archive as restricted where it holds unsuppressed counts, and generate suppressed derivatives for distribution rather than suppressing in place.