Spatial Epi Public Health GIS

Handling API Timeouts in Batch OSM Routing

This guide solves one specific failure: keeping a large origin-destination (OD) travel-time job complete and reproducible when an OSM routing engine returns 504 Gateway Timeout (or simply drops the connection) partway through the batch. It is part of Batch Routing & Error Handling, within Healthcare Access & Network Analysis Automation.

Problem Context & Constraints

The naive approach — loop over every facility-patient pair, fire one request, append the result — fails at agency scale for a reason that is easy to miss: a timeout is not a failure of the coordinates, it is a failure of the call. A single dropped chunk leaves a hole in the travel-time matrix, and any downstream accessibility metric (a Two-Step Floating Catchment Area index, a provider-to-population ratio, an isochrone coverage count) silently treats that hole as “no access” rather than “not measured.” That is spatial sampling bias introduced by an HTTP error, and it is invisible unless you instrument for it.

Three constraints make this scenario distinct from generic retry advice:

  • Timeouts are correlated with payload shape, not load alone. OSM-derived engines (OSRM, Valhalla) run graph traversals whose cost scales non-linearly with coordinate count, edge density, and turn restrictions. The OSRM table service computes an N×N duration matrix, so a 100-coordinate chunk is ~10,000 cells of work. Doubling the chunk roughly quadruples engine time and pushes you past the server’s own request budget — you get a 504 that a retry of the same chunk will reproduce deterministically.
  • Recoverable and terminal failures look similar over HTTP. A 504 (engine overran) and a 429 (rate limited) are worth retrying; a 400 (malformed geometry) or 404 (unsnappable coordinate) will never succeed and must not be retried, or the job stalls behind a poison chunk.
  • Public-health pipelines must be auditable. Every retry, every dropped chunk, and every final status has to be reconstructable for peer review and regulatory reporting — without persisting raw patient coordinates in the log.

So the default for-loop is replaced by three cooperating mechanisms: payload partitioning that keeps each request under the engine’s timeout budget, a classifier that routes each response to retry / terminal / success, and an audit trail keyed on a payload hash rather than identifiers.

The decision logic that routes each engine response to one of three terminal outcomes — backed-off retry, hard failure, or a validated result — is the core of the error-handling layer:

Routing Response Classification & Circuit Breaker A decision flowchart. A routing response enters a status-code classifier. Network timeout, 429, 502 or 504 responses are recoverable: they pass to a circuit-breaker check that compares consecutive failures against a threshold. Below the threshold, the request is retried with exponential backoff and jitter and re-enters the classifier; at or above the threshold the circuit opens and the chunk is flagged for fallback. Status 400, 404 or an invalid geometry is a terminal failure that logs and flags the chunk for manual review. A 200 response with a valid durations matrix passes spatial validation and is appended to the results set. Engine response status + body classify status code timeout · 429 502 · 504 400 · 404 bad geometry 200 OK circuit breaker fails < threshold? yes backoff + jitter re-issue request retry no terminal failure log + flag chunk spatial validation plausible durations? fallback queue manual / secondary append result validated matrix review queue non-recoverable Every response resolves to one outcome — retry, fallback, append, or review — so no chunk is silently dropped

A single OD chunk flows through the retry layer, which classifies failures and re-issues recoverable requests with backoff before returning a validated matrix:

Batch Routing Retry Sequence A sequence diagram with three participants — Pipeline, Retry layer and Routing engine — shown as lifelines. The Pipeline submits an origin-destination chunk to the Retry layer. The Retry layer sends a Table API request (attempt 1) to the Routing engine, which returns a 504 Gateway Timeout. After exponential backoff with jitter, the Retry layer sends attempt 2, the engine returns 200 OK with a durations matrix, and the Retry layer returns a validated travel-time matrix to the Pipeline. Pipeline Retry layer Routing engine submit OD chunk Table API request (attempt 1) 504 Gateway Timeout exp. backoff + jitter Table API request (attempt 2) 200 OK · durations matrix validated travel-time matrix Retry sequence — a timed-out routing call is retried with backoff until a valid matrix returns

Prerequisites

Pin the stack so retries and hashes are reproducible across runs and reviewers:

  • python 3.11
  • geopandas 0.14.4 (pulls shapely 2.0.x, pyproj 3.6.x)
  • requests 2.32.x
  • tenacity 8.5.0 — the backoff/retry policy engine
  • A reachable OSRM table endpoint (self-hosted osrm-routed or a managed instance). The same pattern adapts to Valhalla’s sources_to_targets by changing the URL and the response key.

Input state assumed by the code below:

  • Origins and destinations as point GeoDataFrames with a defined CRS. Identifiers must be stable, sorted keys (so a re-run hashes identically) and must not be raw patient IDs.
  • CRS: OSRM and the OSM tile pipeline expect EPSG:4326 in (longitude, latitude) order. Project distance work to a local metric CRS separately; align WGS84 and UTM exactly as covered in How to Align WGS84 and UTM for County Health Data before any Euclidean fallback. Engine-bound coordinates stay in EPSG:4326.
  • Coordinate precision fixed at 6 decimals (~0.11 m at the equator) so payload hashes are deterministic and no excess precision leaks.

Step-by-Step Solution

The pipeline below partitions OD pairs into bounded chunks, drives each request through a tenacity retry policy that distinguishes recoverable from terminal failures, and writes an audit line per attempt keyed on a payload hash rather than coordinates.

# python 3.11
# geopandas==0.14.4  shapely==2.0.4  pyproj==3.6.1
# requests==2.32.3   tenacity==8.5.0
import hashlib
import json
import logging
import geopandas as gpd
import requests
from tenacity import (
    retry, stop_after_attempt, wait_exponential,
    retry_if_exception_type, retry_if_result,
)
from requests.exceptions import Timeout, ConnectionError

# Structured, audit-ready logging. JSON lines parse cleanly downstream.
logging.basicConfig(
    level=logging.INFO,
    format='{"ts":"%(asctime)s","level":"%(levelname)s","msg":%(message)s}',
    handlers=[logging.FileHandler("routing_audit.log")],
)
logger = logging.getLogger("osm_batch_router")

RECOVERABLE = {429, 502, 503, 504}      # retry these
TERMINAL = {400, 404, 422}              # never retry — poison chunk

def payload_hash(coords):
    """Deterministic 12-char hash for audit trails — no raw coordinates stored."""
    blob = json.dumps(coords, sort_keys=True).encode()
    return hashlib.sha256(blob).hexdigest()[:12]

def _is_recoverable(response):
    """tenacity predicate: True for an HTTP response that is worth retrying."""
    return getattr(response, "status_code", None) in RECOVERABLE

@retry(
    retry=(retry_if_exception_type((Timeout, ConnectionError))
           | retry_if_result(_is_recoverable)),
    wait=wait_exponential(multiplier=1, min=2, max=30),  # 2,4,8,16,30s + jitter
    stop=stop_after_attempt(5),
    reraise=True,
)
def fetch_route_batch(engine_url, coords_chunk, timeout=15):
    """POST one coordinate chunk to the OSRM Table API; classify the response."""
    h = payload_hash(coords_chunk)
    coord_str = ";".join(f"{lon},{lat}" for lon, lat in coords_chunk)
    url = f"{engine_url}/table/v1/driving/{coord_str}"
    try:
        resp = requests.get(url, params={"annotations": "duration"}, timeout=timeout)
    except (Timeout, ConnectionError) as e:
        logger.warning(json.dumps({"event": "network_error", "hash": h, "detail": str(e)}))
        raise  # tenacity retries on the exception type

    if resp.status_code == 200:
        logger.info(json.dumps({"event": "success", "hash": h,
                                "secs": round(resp.elapsed.total_seconds(), 2)}))
        return resp.json()
    if _is_recoverable(resp):
        logger.warning(json.dumps({"event": "recoverable", "hash": h,
                                   "status": resp.status_code}))
        return resp                       # retry_if_result triggers backoff
    # terminal: log and raise so the caller can quarantine this chunk
    logger.error(json.dumps({"event": "terminal", "hash": h,
                             "status": resp.status_code, "body": resp.text[:120]}))
    resp.raise_for_status()

def chunk_coords(gdf, chunk_size=40):
    """Project to EPSG:4326 and emit bounded [lon, lat] chunks.

    chunk_size bounds the N×N table cost so a single request stays inside the
    engine's timeout budget. 40 coords -> a 40x40 (1,600-cell) matrix.
    """
    gdf = gdf.to_crs("EPSG:4326")
    coords = [[round(g.x, 6), round(g.y, 6)] for g in gdf.geometry]
    for i in range(0, len(coords), chunk_size):
        yield coords[i:i + chunk_size]

def run_batch_routing(engine_url, origins_path, destinations_path):
    origins = gpd.read_file(origins_path)
    destinations = gpd.read_file(destinations_path)
    combined = gpd.GeoDataFrame(
        # stable, sorted concatenation -> identical hashes across re-runs
        origins.assign(role="o")._append(destinations.assign(role="d"))
    ).reset_index(drop=True)

    results, quarantine = [], []
    for i, chunk in enumerate(chunk_coords(combined)):
        try:
            results.append({"chunk": i, "data": fetch_route_batch(engine_url, chunk)})
        except Exception as e:
            # exhausted retries OR terminal status -> never imputed, always flagged
            logger.critical(json.dumps({"event": "chunk_failed", "chunk": i,
                                        "detail": str(e)}))
            quarantine.append(i)
    return results, quarantine

The contract is deliberate: a recoverable status backs off and re-issues; a terminal status raises immediately so the chunk lands in quarantine instead of stalling the run; a success returns the matrix. Nothing is imputed, and every branch emits one audit line.

Validation & Edge Cases

A returned 200 is not yet a valid result. Validate the matrix before it reaches any accessibility metric, and watch for these three failure modes specifically.

1. null durations inside a 200 response. OSRM returns 200 with null cells when a coordinate cannot be snapped to the road graph (off-network, in water, wrong hemisphere from a lon/lat swap). These are silent holes:

def validate_durations(matrix, max_secs=24 * 3600):
    """Flag null / non-positive / implausible cells. Returns (ok, problems)."""
    problems = []
    for r, row in enumerate(matrix.get("durations", [])):
        for c, val in enumerate(row):
            if val is None:
                problems.append((r, c, "null_unsnappable"))
            elif r != c and (val <= 0 or val > max_secs):
                problems.append((r, c, f"implausible:{val}"))
    return (len(problems) == 0, problems)
WARNING {"event":"validation","hash":"a1b2c3d4e5f6","null_cells":3,"sample":[[0,7,"null_unsnappable"]]}

A lon/lat swap is the usual culprit behind a wall of nulls — confirm EPSG:4326 ordering is (lon, lat), the opposite of most desktop GIS display order.

2. The poison chunk that exhausts every retry. If one chunk repeatedly times out because it is simply too large, retrying it five times wastes ~60 s of backoff and still fails. The diagnostic signature is a recoverable line at the same hash on every attempt:

WARNING {"event":"recoverable","hash":"9f8e7d6c5b4a","status":504}   # attempt 1
WARNING {"event":"recoverable","hash":"9f8e7d6c5b4a","status":504}   # attempt 5
CRITICAL {"event":"chunk_failed","chunk":12,"detail":"RetryError[...504...]"}

The fix is structural, not more retries: halve chunk_size for the quarantined chunk and re-run it, because the engine cost scales with the square of the coordinate count.

3. Coverage drift between runs. Compare the count of valid OD cells against the expected count. A run that is “successful” but covers fewer pairs than the previous one signals an upstream engine or data regression:

expected = len(combined) ** 2
got = sum(len(r["data"]["durations"]) ** 2 for r in results)
if got < expected:
    logger.warning(json.dumps({"event": "coverage_gap",
                               "missing_cells": expected - got,
                               "quarantined_chunks": quarantine}))

Missing routes must be carried forward as explicit nulls — never zero, never imputed — so the gap propagates honestly into the accessibility surface instead of masquerading as inaccessibility.

Compliance Notes

For regulatory defensibility, three artifacts must be logged for every run:

  • Payload hash, not coordinates. The SHA-256 payload_hash lets a reviewer prove which input produced which result without persisting patient-derived points — satisfying HIPAA minimum-necessary and GDPR data-minimization. Six-decimal rounding caps positional precision in any retained artifact.
  • Per-chunk retry count, final status, and quarantine list. Store these in a version-controlled metadata table alongside the engine version and chunk_size. The quarantine list is the auditable record of what was not measured, which is the difference between a defensible analysis and silent bias.
  • Deterministic ordering. The sorted origin+destination concatenation guarantees identical hashes on re-run, so a reviewer re-executing the pipeline reproduces the same audit log — a precondition for peer review and interagency handoff.

FAQ

Should I retry a 400 or 404 from the routing engine?

No. A 400 (malformed request / invalid geometry) and 404 (unsnappable coordinate or unknown service) are terminal — they will fail identically on every attempt. The code above keeps them out of the retry set and raises immediately so the chunk is quarantined rather than stalling the batch behind a poison request.

Why exponential backoff with jitter rather than a fixed delay?

Fixed delays synchronize many workers into a thundering herd that re-saturates a shared engine the instant it recovers. Exponential growth (2, 4, 8, 16, 30s) plus tenacity’s jitter spreads retries out, giving a self-hosted OSRM instance room to drain its queue before the next wave.

How do I pick chunk_size?

Start from the engine cost model: the table service computes an N×N matrix, so cost grows with the square of the coordinate count. If chunks time out, halve chunk_size rather than raising the HTTP timeout — a smaller matrix finishes inside the engine’s own budget. 40 coordinates (1,600 cells) is a safe default for a modestly provisioned OSRM instance.

What about coordinates that return null durations inside a 200?

Treat them as measurement gaps, not zeros. The validate_durations check flags null cells; carry them forward as explicit nulls into the accessibility metric so they never read as “no access.” A wall of nulls almost always means a (lat, lon) / (lon, lat) swap — OSRM expects (lon, lat).