Spatial Epi Public Health GIS

Handling ACS Margins of Error in Small-Area Rates

This guide, part of Small-Area Population Denominators, covers what to do once the denominator of a rate is a survey estimate: carry its uncertainty into the rate, judge whether the result is stable enough to publish, and fix it by aggregation when it is not.

Problem Context & Constraints

The standard interval for a small-area rate treats the case count as a Poisson variable and the denominator as a fixed, known number. When the denominator is a decennial census count, that is close to true. When it is a survey estimate, it is not: the denominator has its own sampling error, published as a margin of error, and at tract level for a demographic subgroup that error can be a quarter or a half of the estimate.

Ignoring it has a predictable effect. The rate’s interval is too narrow, and it is too narrow by the most in exactly the areas where the denominator is least reliable — small tracts, small subgroups, rural areas. A rate map built on those intervals shows the least certain areas as though they were as well measured as the rest, and any ranking or cluster analysis built on the map inherits the false precision.

Three constraints shape the fix. The two errors combine, but not additively: for a ratio, relative errors add in quadrature, so a rate with a 20% relative error from the numerator and a 30% relative error from the denominator has roughly a 36% relative error overall. Published margins are at 90% confidence, so they must be converted before combination. And the correct response to an unstable rate is usually aggregation, not a caveat — a rate with a 60% relative error cannot be rescued by a footnote.

The Interval That Ignores the Denominator Is Too Narrow Two confidence intervals for the same tract rate. The narrow interval treats the survey denominator as exact and reflects only the Poisson uncertainty of the case count. The wider interval combines the numerator's relative error with the denominator's relative error in quadrature and is substantially wider. For a small subgroup in a small tract the difference is large enough that the narrow interval excludes the county rate while the honest interval includes it, so a finding of difference disappears. Adding the denominator's error can erase an apparent difference county rate denominator as exact excludes county with denominator error includes county relative errors of numerator and denominator add in quadrature for a ratio the smaller the tract and subgroup, the larger the denominator's share

Prerequisites

  • python 3.11
  • pandas 2.2.2, numpy 1.26.4, scipy 1.13.1
  • Case counts per area and stratum, pooled over the same years as the survey product
  • Survey estimates and 90% margins of error for the same areas and strata
  • A declared reliability threshold on the combined relative error
  • An aggregation geography to fall back to — block-group clusters, regionalised units, or county

Step-by-Step Solution

Compute each component’s relative error, combine them, build the interval on the log scale, flag unreliable rates, and aggregate those until they meet the threshold.

# Rates with combined numerator (Poisson) and denominator (survey) uncertainty.
# Pinned: pandas==2.2.2, numpy==1.26.4, scipy==1.13.1
import logging

import numpy as np
import pandas as pd
from scipy import stats

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("rates.moe")

Z90 = 1.645
RSE_MAX = 0.30          # combined relative standard error above this is unreliable
PER = 100_000


def rate_with_denominator_error(df: pd.DataFrame, cases: str = "cases",
                                pop: str = "pop_est", moe: str = "pop_moe",
                                conf: float = 0.95) -> pd.DataFrame:
    """Rate, combined relative standard error, and a log-scale interval."""
    out = df.copy()
    c = out[cases].astype(float)
    p = out[pop].astype(float)
    out["rate"] = np.where(p > 0, c / p * PER, np.nan)
    # Poisson relative SE of the count; undefined at zero cases.
    rse_num = np.where(c > 0, 1.0 / np.sqrt(c), np.inf)
    # Survey relative SE of the denominator, from a 90% margin.
    rse_den = np.where(p > 0, (out[moe].astype(float) / Z90) / p, np.inf)
    out["rse_numerator"] = rse_num
    out["rse_denominator"] = rse_den
    out["rse_rate"] = np.sqrt(rse_num ** 2 + rse_den ** 2)
    z = stats.norm.ppf(0.5 + conf / 2.0)
    # Log-scale interval keeps the lower bound positive and is better behaved.
    with np.errstate(divide="ignore", invalid="ignore"):
        out["lo"] = out["rate"] * np.exp(-z * out["rse_rate"])
        out["hi"] = out["rate"] * np.exp(z * out["rse_rate"])
    out["reliable"] = out["rse_rate"] <= RSE_MAX
    share_den = np.nanmedian(np.where(np.isfinite(out["rse_rate"]),
                                      (rse_den ** 2) / (out["rse_rate"] ** 2), np.nan))
    log.info("rates: %d areas, %d reliable (rse<=%.2f); denominator contributes a "
             "median %.0f%% of the rate variance", len(out), int(out["reliable"].sum()),
             RSE_MAX, 100 * share_den)
    return out


def aggregate_unreliable(rates: pd.DataFrame, parent: str,
                         cases: str = "cases", pop: str = "pop_est",
                         moe: str = "pop_moe") -> pd.DataFrame:
    """Roll unreliable areas up to their parent unit and recompute."""
    bad = rates[~rates["reliable"]]
    if bad.empty:
        return rates.assign(level="original")
    g = bad.groupby(parent, as_index=False).agg(
        cases=(cases, "sum"), pop_est=(pop, "sum"),
        pop_moe=(moe, lambda m: float(np.sqrt((m.astype(float) ** 2).sum()))))
    rolled = rate_with_denominator_error(g, cases="cases", pop="pop_est", moe="pop_moe")
    rolled["level"] = "aggregated"
    log.info("aggregated %d unreliable areas into %d %s units; %d now reliable",
             len(bad), len(rolled), parent, int(rolled["reliable"].sum()))
    good = rates[rates["reliable"]].assign(level="original")
    return pd.concat([good, rolled], ignore_index=True)

The log line reporting the denominator’s median share of the rate variance is the number worth reading first. When it is small, the survey error is a minor refinement and the conventional Poisson interval was nearly right. When it is large — often above half for subgroup rates at tract level — the conventional interval was substantially wrong, and every downstream use of those rates needs to know.

Aggregating unreliable areas to a parent unit, rather than suppressing them, keeps the information in the published product. The aggregated rate is less spatially specific and much more trustworthy, and it answers the question a reader actually has about that part of the map rather than leaving a hole.

When the Denominator Dominates the Uncertainty The share of a rate's total variance contributed by the survey denominator, plotted against the population of the stratum. For a whole county the denominator's share is negligible and the Poisson interval is essentially correct. For all-ages tract populations the share is modest. For a single age-sex stratum within a tract it rises steeply, and for small subgroups the denominator contributes most of the uncertainty. Three example strata are marked along the curve. For small subgroups the denominator is most of the uncertainty one age-sex stratum, one tract all ages, one tract whole county small large population of the stratum the rate is computed for denominator share of variance

Validation & Edge Cases

Zero cases. The Poisson relative error is undefined at a zero count and the log interval collapses. Use an exact Poisson interval for the numerator in that case, or aggregate; do not report a zero rate with a zero-width interval.

Margins that exceed the estimate. A survey estimate of 40 with a margin of 55 has a relative error well above one, and any rate built on it is meaningless. The reliability flag catches it; the aggregation step should run before such a rate reaches any output.

Derived denominators. A denominator computed by summing several survey cells — ages 65 and over from five published age groups — needs its margin combined in quadrature from the components, not taken from any single cell.

Covariance between numerator and denominator. When both come from the same survey, their errors are correlated and quadrature overstates the combined error slightly. For surveillance numerators from a separate reporting system, the two are independent and quadrature is correct.

Period mismatch. A five-year survey denominator paired with a one-year case count produces a rate for no real period. Pool cases over the same five years.

INFO rates.moe rates: 1842 areas, 1204 reliable (rse<=0.30); denominator contributes a median 58% of the rate variance
INFO rates.moe aggregated 638 unreliable areas into 212 regionalised units; 201 now reliable

Aggregation that still fails. Some regionalised units remain unreliable even after aggregation, typically because the subgroup is genuinely rare throughout the region. Those need a coarser level again or suppression under the disclosure rules; the aggregation should iterate until every unit either meets the threshold or reaches the coarsest permitted level, and the remaining failures should be counted and reported rather than quietly dropped.

Maps that mix levels. After aggregation, a single map shows tracts in some places and regionalised units in others. That is correct, and it needs a visual cue — a heavier boundary around aggregated units, a legend note — so a reader does not assume a regionalised rate applies uniformly to every tract inside it.

Aggregate Until Reliable, Then Stop A three-step sequence. In the first step, twelve tracts are shown with five of them flagged unreliable. In the second step, the unreliable tracts are merged with neighbours into three regionalised units, two of which now meet the threshold. In the third step, the one remaining unreliable unit is merged again into a larger unit that meets it. Reliable tracts are never merged, so the map keeps the finest geography that the data can support everywhere. Keep the finest geography the data can support, area by area 1 · flag 12 tracts 5 unreliable 2 · merge with neighbours 7 tracts + 3 units 1 still unreliable 3 · merge again 7 tracts + 2 units all reliable reliable tracts are never merged, so precision is only given up where it is missing

Compliance Notes

Record the reliability threshold, the conversion from published margins, the combination rule, the interval method, the aggregation geography and iteration rule, and the counts of areas that were reliable, aggregated, or still unreliable after aggregation.

State in any published product that intervals include the survey denominator’s sampling error, because that is not the default a reader will assume. The aggregation step also serves disclosure control: a rate stable enough to publish is usually built on enough cases and population to be safe as well. Where the two rules disagree — stable but disclosive, or safe but unstable — apply whichever is stricter, and record which rule decided each area.