Batch Routing & Error Handling for Healthcare Access Analytics

This guide sits within Healthcare Access & Network Analysis Automation and covers the fault-tolerance layer that keeps large origin-destination (OD) routing jobs deterministic, auditable, and statistically complete. Calculating travel impedance between patient populations and clinical facilities is the foundational input to resource allocation and equity modeling; when a job spans tens of thousands of OD pairs, a single unhandled timeout, misaligned coordinate pair, or disconnected network node can propagate sampling bias through every downstream accessibility index and capacity forecast. The job of this layer is to ensure that the routing matrix handed to those models is either complete or explicitly, traceably incomplete — never silently wrong.

Concept & Epidemiological Alignment

Batch routing error handling is not a generic resilience concern bolted onto a request loop. In a public-health context, the spatial distribution of failures is itself an epidemiological signal. If timeouts and unroutable pairs cluster in rural census tracts — exactly the populations most sensitive to access deficits — then dropping failed pairs and renormalizing biases the accessibility surface toward urban over-coverage. The statistic this layer protects is coverage completeness: the fraction of intended OD pairs that resolve to a valid, calibrated travel time, broken down by population stratum.

Use a hardened batch architecture (rather than a naive request loop with a blanket try/except) whenever any of the following hold: the OD matrix exceeds a few thousand pairs; the routing engine is a shared or rate-limited service (hosted OSRM, Valhalla, or a commercial Matrix API); the outputs feed regulated reporting; or failures are expected to correlate with geography. For small, one-shot exploratory runs against a local engine, the overhead is rarely justified. The key assumptions that must hold for the downstream models to remain valid are: (1) every emitted travel time derives from a topologically sound, projected-CRS path; (2) every failure is classified and accounted for, never silently discarded; and (3) the failure set is verified to be spatially random before results are promoted.

Failure-Mode Decision Table

Each failure is triaged into exactly one branch before any retry budget is spent, so transient faults loop through backoff while permanent faults go straight to fallback handling and never consume retry attempts.

Spatial Data Prerequisites

The router only sees clean inputs because a validation gate runs first. The required preconditions are concrete:

• Geometry type: point pairs (origin and destination), each a valid 2D coordinate.
• CRS: all OD geometries projected into a single distance-preserving coordinate reference system before network ingestion. Unprojected WGS84 lat/lon pairs introduce geodetic distortion that compounds across thousands of shortest-path calculations and silently degrades metric accuracy — enforce a canonical projection per the rules in Coordinate Reference Systems for Public Health.
• Schema: strict validation via pandera or pydantic to reject malformed coordinates, null timestamps, or mismatched EPSG codes prior to execution, consistent with the conventions in Spatial Data Types & Formats.
• Snapping: origins and destinations snapped to the nearest routable network edge within a configurable tolerance (typically 10–20 meters), with points falling in non-routable zones filtered out via spatial joins against land-use or hydrological polygons.
• Precision: coordinate precision aligned to the Precision Standards in Epi Mapping so that snapping tolerance and de-identification jitter do not interact to move a point across an edge boundary.

This gate guarantees that downstream Drive-Time Isochrone Generation and Facility Capacity Allocation Models receive topologically sound, geodetically aligned inputs.

Concurrency Architecture & Rate-Limit Management

Commercial routing APIs and open-source engines (OSRM, Valhalla) enforce strict rate limits, connection quotas, and payload-size restrictions. Synchronous batch execution will inevitably trigger HTTP 429 responses or socket exhaustion. Adopt an asynchronous request pool with bounded concurrency, typically managed via asyncio.Semaphore capped at 50–100 concurrent tasks. Configure connection pooling, socket read/write timeouts, and keep-alive headers to minimize TLS handshake overhead. Engine-specific timeout tuning — thread-pool sizing, payload chunking, and connection reuse without saturating upstream infrastructure — is covered in depth in Handling API Timeouts in Batch OSM Routing.

Deterministic Error Classification & Circuit Breakers

Transient failures (HTTP 502/503/504, DNS drops, temporary rate limits) must be strictly isolated from permanent failures (invalid coordinates, disconnected graph components, HTTP 400/404 responses). Implement exponential backoff with randomized jitter, capping retries at 3–5 attempts. Libraries like tenacity provide production-ready decorators that encapsulate retry logic while preserving idempotent request payloads and deterministic backoff intervals.

Each failed OD calculation is triaged into exactly one branch before any retry budget is spent. Transient faults re-enter the backoff loop; permanent faults are quarantined for fallback handling so they never consume retry attempts:

When the failure rate for a batch exceeds a predefined threshold (e.g., >5%), trigger a circuit breaker: halt execution, flush pending requests, and emit an alert to the operations queue. All request metadata — HTTP status, payload hash, retry count, latency, and CRS metadata — must be written to an append-only audit log. This pattern satisfies HIPAA/GDPR data-handling requirements by maintaining a verifiable chain of custody for every OD calculation.

The circuit breaker moves between three states based on the rolling failure rate, isolating a degraded routing engine before failures cascade:

Production Implementation

The following pipeline combines schema validation, bounded async concurrency, exponential backoff with jitter, deterministic error classification, and append-only audit logging. Inputs are sorted by a stable ID before dispatch so that re-runs produce byte-identical audit logs.

# Pinned: aiohttp==3.9.5  tenacity==8.5.0  pandera==0.20.4  pandas==2.2.2
import asyncio
import hashlib
import json
import logging
from datetime import datetime, timezone
from typing import Any

import aiohttp
import pandas as pd
import pandera as pa
from pandera.typing import DataFrame, Series
from tenacity import (
    retry, stop_after_attempt, wait_exponential_jitter, retry_if_exception_type,
)


# --- 1. Strict input validation gate -----------------------------------------
class ODPointSchema(pa.DataFrameModel):
    origin_id: Series[str]
    dest_id: Series[str]
    origin_x: Series[float]              # projected metres, NOT lat/lon
    origin_y: Series[float]
    dest_x: Series[float]
    dest_y: Series[float]
    epsg: Series[int] = pa.Field(eq=3857)  # enforce a single projected CRS

    class Config:
        strict = True                   # reject unexpected columns
        coerce = False                  # never silently cast bad types


# --- 2. Append-only audit logger ---------------------------------------------
audit = logging.getLogger("routing_audit")
audit.setLevel(logging.INFO)
_handler = logging.FileHandler("routing_audit.log", mode="a")  # append-only
_handler.setFormatter(logging.Formatter("%(message)s"))
audit.addHandler(_handler)


def _log(event: str, **fields: Any) -> None:
    """Emit one structured, deterministic JSON record per event."""
    record = {"event": event, "ts": datetime.now(timezone.utc).isoformat(), **fields}
    audit.info(json.dumps(record, sort_keys=True))


# --- 3. Permanent vs transient classification --------------------------------
class PermanentRoutingError(Exception):
    """4xx, NoRoute, or disconnected graph — never retried."""


def classify(status: int, body: dict) -> None:
    if status in (400, 404, 422) or body.get("code") == "NoRoute":
        raise PermanentRoutingError(f"status={status} code={body.get('code')}")


# --- 4. Transient-only retry: jitter prevents thundering-herd ----------------
@retry(
    stop=stop_after_attempt(4),
    wait=wait_exponential_jitter(initial=2, max=30),
    retry=retry_if_exception_type((aiohttp.ClientError, asyncio.TimeoutError)),
    reraise=True,
)
async def fetch_route(session: aiohttp.ClientSession, origin: str, dest: str,
                      api_url: str) -> dict:
    payload_hash = hashlib.sha256(f"{origin};{dest}".encode()).hexdigest()
    url = f"{api_url}/route/v1/driving/{origin};{dest}?overview=false&steps=false"
    async with session.get(url) as resp:
        body = await resp.json()
        classify(resp.status, body)     # permanent errors raise BEFORE retry
        resp.raise_for_status()         # remaining 5xx -> transient retry
        _log("success", hash=payload_hash, status=resp.status)
        return body


# --- 5. Bounded-concurrency batch with circuit breaker -----------------------
async def execute_batch_routing(od_df: pd.DataFrame, api_url: str,
                                max_concurrency: int = 75,
                                breaker_threshold: float = 0.05
                                ) -> tuple[list, list]:
    ODPointSchema.validate(od_df)                    # fail fast on bad input
    od_df = od_df.sort_values("origin_id").reset_index(drop=True)  # deterministic
    config_hash = hashlib.sha256(
        json.dumps({"conc": max_concurrency, "thr": breaker_threshold,
                    "n": len(od_df)}, sort_keys=True).encode()).hexdigest()
    _log("batch_start", config_hash=config_hash, n_pairs=len(od_df))

    sem = asyncio.Semaphore(max_concurrency)
    results: list[tuple[str, dict]] = []
    failures: list[tuple[str, str]] = []
    timeout = aiohttp.ClientTimeout(total=20, connect=5)

    async with aiohttp.ClientSession(timeout=timeout) as session:
        async def bounded(row) -> None:
            pair = f"{row.origin_id}->{row.dest_id}"
            o = f"{row.origin_x},{row.origin_y}"
            d = f"{row.dest_x},{row.dest_y}"
            async with sem:
                try:
                    results.append((pair, await fetch_route(session, o, d, api_url)))
                except PermanentRoutingError as e:
                    failures.append((pair, f"permanent:{e}"))
                    _log("permanent_failure", pair=pair, detail=str(e))
                except Exception as e:                # transient, retries spent
                    failures.append((pair, f"exhausted:{type(e).__name__}"))
                    _log("exhausted_failure", pair=pair, error=type(e).__name__)

        await asyncio.gather(*(bounded(r) for r in od_df.itertuples()))

    rate = len(failures) / max(len(od_df), 1)
    _log("batch_end", config_hash=config_hash, ok=len(results),
         failed=len(failures), failure_rate=round(rate, 4))
    if rate > breaker_threshold:                     # circuit breaker
        raise RuntimeError(
            f"Circuit breaker tripped: {rate:.1%} > {breaker_threshold:.0%}")
    return results, failures

Topological Validation & Network Integrity

Rural and peri-urban networks frequently exhibit structural anomalies: unconnected cul-de-sacs, missing turn restrictions, misclassified private roads, or seasonal closures. Shortest-path algorithms fail silently if the routing graph contains isolated subgraphs or invalid edge weights. Pre-validate the network with graph traversal (BFS/DFS, or connected-component labelling) to confirm full connectivity between major facility nodes and census tracts. When routing fails due to a topological gap, the fallback strategy is one of: expand the search radius, substitute a straight-line Euclidean distance with a calibrated impedance factor, or flag the OD pair for manual topology review. Whichever path is taken is recorded so the affected pairs can be excluded from precision-sensitive metrics.

Parameter Selection & Tuning

The behaviour of this layer is governed by a small set of parameters that should be chosen deliberately and logged, not left at library defaults:

• Concurrency cap (max_concurrency): start at the engine’s documented sustained throughput minus headroom; back off automatically when 429 rates rise. For a single hosted OSRM instance, 50–100 is typical; a contracted Matrix API may tolerate far less.
• Retry budget (stop_after_attempt): 3–5. Beyond 5, a transient-looking fault is usually a misclassified permanent one — investigate rather than retry harder.
• Backoff schedule: exponential with jitter, initial=2 s, max=30 s. Jitter is mandatory on shared infrastructure to avoid synchronized retry storms.
• Circuit-breaker threshold: 5% is a sane default for production, but set it from the coverage-completeness tolerance the downstream model requires. An equity index that is sensitive to rural under-sampling may demand a 1–2% threshold.
• Snapping tolerance: 10–20 m; widen only with explicit logging, because aggressive snapping can re-route a point onto the wrong functional class of road.

Edge Cases & Failure Modes

• Spatially clustered failures. If exhausted/permanent failures concentrate in a region, the cause is almost always a localized topology gap, not load. Test the spatial randomness of the failure set (a quick join of failed pairs to tracts, then a Moran’s I on the failure indicator) before renormalizing — the technique mirrors Global & Local Moran’s I Implementation. Clustered failures must be patched, not dropped.
• Transboundary CRS drift. OD pairs straddling UTM-zone or state-plane boundaries silently distort if half the batch is in one projection. Enforce a single project-wide CRS at the validation gate; never let the router infer it.
• Zero-distance / self-pairs. Origin equal to destination produces a degenerate 0-time route that some engines reject as NoRoute. Filter self-pairs in preprocessing.
• Memory at scale (N > 50k). A full N×M matrix held in memory exhausts the worker. Stream results to columnar storage in chunks and partition the OD set by origin tile; never accumulate the whole matrix in a Python list for very large jobs.
• Idempotency on re-run. Because retries can partially complete a batch, every write keys on the OD pair ID so a re-run overwrites rather than duplicates. The stable sort plus pair-keyed audit records make re-runs reproducible.

Compliance & Audit Controls

• Deterministic execution. Inputs are sorted by origin_id before dispatch and all audit records are serialized with sort_keys=True, so an identical input set yields a byte-identical audit log — a prerequisite for regulatory reproducibility.
• Configuration provenance. The concurrency, threshold, and batch size are hashed into a config_hash (SHA-256) emitted at batch start and end; pipeline manifests record every parameter, CRS transformation, and fallback rule under version control.
• PHI separation. Maintain strict separation between routing payloads and protected health information. Hash patient identifiers before any transmission, store routing results in encrypted, access-controlled data lakes, and enforce role-based access controls (RBAC) for audit-log retrieval. De-identification obligations follow the Compliance Mapping Frameworks for the relevant jurisdiction.
• Output schema. Emit a stable, documented column set — origin_id, dest_id, travel_seconds, path_source (network | euclidean_fallback), epsg, config_hash, computed_at — accompanied by ISO 19115 lineage metadata describing the engine version, network snapshot date, and calibration reference.
• Chain of custody. Every OD calculation appears in the append-only log with its HTTP status, payload hash, retry count, and disposition, enabling rapid rollback when an upstream provider changes API schemas or pricing tiers.

Statistical Validation & Calibration

Routing outputs must undergo post-execution validation before ingestion into public-health models. Compare calculated drive times against ground-truth GPS traces or historical EMS dispatch logs to quantify systematic bias, then apply linear regression or quantile matching to calibrate the engine’s impedance factors against observed travel behaviour. Confirm that the failure distribution is spatially random; clustered failures signal localized network degradation or missing turn restrictions that require manual graph patching. Only after passing spatial-autocorrelation tests (Moran’s I on residuals) and bias thresholds should routing outputs be promoted to the access metrics consumed by Facility Capacity Allocation Models and downstream equity dashboards.

Related Topics

• Healthcare Access & Network Analysis Automation — the parent guide this routing layer feeds.
• Handling API Timeouts in Batch OSM Routing — engine-specific timeout tuning and payload partitioning.
• Drive-Time Isochrone Generation — the catchment-construction stage that consumes routed travel times.
• Facility Capacity Allocation Models — capacity-constrained access metrics built on the routing matrix.
• Coordinate Reference Systems for Public Health — CRS enforcement rules referenced by the validation gate.