Spatial Epi Public Health GIS

Avoiding Overfitting in Land-Use Regression Variable Selection

This guide, part of Land-Use Regression for Air Pollution Exposure, addresses the failure that makes most published land-use regression models unusable outside their own campaign: a selection procedure run once, on all the data, and then validated as though it had been fixed in advance.

Problem Context & Constraints

A land-use regression campaign produces perhaps sixty measured sites. The candidate predictor set, generated by crossing eight land-use variables with six buffer radii and two or three transformations, easily runs past two hundred columns. Under those conditions, a forward-selection procedure will find a four-term model explaining 80% of the variance in pure noise — not occasionally, but reliably, because with two hundred candidates the best few are selected precisely for fitting the residuals of this particular sixty-site sample.

The trap is that the model then validates, if validation is done wrong. Selecting the terms on all sixty sites and then running five-fold cross-validation on the selected model reports the error of a model that already saw every held-out point during selection. The reported number is optimistic by an amount that grows with the size of the candidate set, and it is the number that ends up in the paper.

Three constraints define the fix. Selection is part of the model, so it must happen inside every fold — nested cross-validation, where the inner loop selects and the outer loop scores. A baseline is needed, because a fit of 0.6 means nothing until you know what a permuted outcome would produce with the same candidate set. And stability must be measured: if ten bootstrap resamples of the campaign select ten different four-term models, the specific terms are an artifact even when the predictive performance is real.

Flat Versus Nested Cross-Validation Two validation designs side by side. In the flat design, variable selection runs once on the full set of sites and the resulting fixed model is then scored by five-fold cross-validation, so every held-out site influenced the selection and the reported error is optimistic. In the nested design, the outer loop holds out a fold, the inner loop performs the entire selection using only the remaining sites, and the selected model is then scored on the untouched outer fold. A note records that the nested design produces a different selected model in each outer fold, and that this variability is itself the information you need. Selection is part of the model, so it belongs inside the fold Flat: selection first, then score select terms using ALL 60 sites 5-fold CV on the fixed model every held-out site already shaped the model reported error is optimistic by an unknown amount Nested: select inside every fold outer fold held out and untouched inner loop repeats the whole selection a different model is selected in each outer fold and that disagreement is the stability signal The nested number is the one to publish; the flat number is the one that fails on new data Nested validation costs one extra loop and removes the largest source of optimism in the field

Prerequisites

  • python 3.11
  • numpy 1.26.4, pandas 2.2.2, scikit-learn 1.5.0
  • A candidate predictor table built as in Building Buffer Predictors for a Land-Use Regression Model
  • A selection function that takes a training subset and returns selected terms — the supervised forward selection from the parent guide
  • Measured concentrations for every campaign site, on the transformation the model will use

Step-by-Step Solution

Run three diagnostics together. Nested cross-validation gives the honest error; a permutation baseline gives the null it must beat; a bootstrap selection frequency table gives the stability of the chosen terms.

# Nested CV, permutation baseline and selection stability for a land-use regression.
# Pinned: numpy==1.26.4, pandas==2.2.2, scikit-learn==1.5.0
import logging
from collections import Counter

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("lur.overfit")

SEED = 20260918


def nested_cv(X: pd.DataFrame, y: np.ndarray, select_fn, n_splits: int = 10) -> dict:
    """Outer loop scores; inner loop performs the ENTIRE selection."""
    kf = KFold(n_splits=n_splits, shuffle=True, random_state=SEED)
    obs, pred, per_fold_terms = [], [], []
    for train_i, test_i in kf.split(X):
        terms = select_fn(X.iloc[train_i], y[train_i])          # selection inside
        if not terms:
            pred.extend([y[train_i].mean()] * len(test_i))
        else:
            m = LinearRegression().fit(X.iloc[train_i][terms], y[train_i])
            pred.extend(m.predict(X.iloc[test_i][terms]))
        obs.extend(y[test_i])
        per_fold_terms.append(tuple(terms))
    obs, pred = np.asarray(obs), np.asarray(pred)
    ss_res = float(((obs - pred) ** 2).sum())
    ss_tot = float(((obs - obs.mean()) ** 2).sum())
    out = {"rmse": float(np.sqrt(((obs - pred) ** 2).mean())),
           "bias": float((obs - pred).mean()),
           "r2": 1.0 - ss_res / ss_tot,
           "distinct_models": len(set(per_fold_terms)),
           "n_splits": n_splits}
    log.info("nested rmse=%.4f r2=%.3f distinct_models_across_folds=%d/%d",
             out["rmse"], out["r2"], out["distinct_models"], n_splits)
    return out


def permutation_baseline(X: pd.DataFrame, y: np.ndarray, select_fn,
                         n_perm: int = 200) -> dict:
    """What does the same pipeline achieve when the outcome carries no signal?"""
    rng = np.random.default_rng(SEED)
    scores = []
    for _ in range(n_perm):
        yp = rng.permutation(y)
        terms = select_fn(X, yp)
        if not terms:
            scores.append(0.0)
            continue
        m = LinearRegression().fit(X[terms], yp)
        scores.append(float(m.score(X[terms], yp)))
    scores = np.asarray(scores)
    out = {"null_r2_median": float(np.median(scores)),
           "null_r2_p95": float(np.percentile(scores, 95)),
           "n_perm": n_perm}
    log.info("permutation null in-sample r2: median=%.3f p95=%.3f (n=%d)",
             out["null_r2_median"], out["null_r2_p95"], n_perm)
    return out


def selection_stability(X: pd.DataFrame, y: np.ndarray, select_fn,
                        n_boot: int = 500) -> pd.DataFrame:
    """How often does each predictor survive a bootstrap resample of the campaign?"""
    rng = np.random.default_rng(SEED)
    n = len(y)
    counts = Counter()
    for _ in range(n_boot):
        idx = rng.integers(0, n, n)
        for t in select_fn(X.iloc[idx], y[idx]):
            counts[t] += 1
    df = (pd.DataFrame({"predictor": list(counts), "frequency": list(counts.values())})
          .assign(frequency=lambda d: d["frequency"] / n_boot)
          .sort_values("frequency", ascending=False)
          .reset_index(drop=True))
    log.info("stability: %d predictors selected in >50%% of resamples",
             int((df["frequency"] > 0.5).sum()))
    return df

The permutation baseline is the diagnostic most often skipped and the most informative when the candidate set is large. If a pipeline applied to a shuffled outcome routinely reaches an in-sample fit of 0.55, then a real model reporting 0.62 has demonstrated almost nothing. The gap between the observed nested score and the permutation null is the model’s actual evidence, and reporting both is what makes the claim checkable.

The stability table changes how the final model is described. A predictor selected in 90% of bootstrap resamples is a finding about the city. One selected in 30%, which is common among members of a correlated buffer ladder, is a coin flip between near-identical candidates — the family is real, the specific radius is not, and the write-up should say so rather than presenting the chosen radius as a discovered scale.

Reading the Selection Stability Table A horizontal bar chart of bootstrap selection frequency for eight candidate predictors. Two predictors are selected in more than eighty per cent of resamples and are labelled stable findings. Four predictors from the same road-length buffer ladder each sit between twenty and forty per cent, summing to a high family total, and are labelled interchangeable: the family is real but the specific radius is arbitrary. Two predictors sit near five per cent and are labelled noise. A threshold line at fifty per cent separates terms that may be reported individually from those that should be described at family level. A ladder that splits its votes is one finding, not four industrial_1000 popdens_3000 road_major_50 road_major_100 road_major_300 road_major_25 green_100 water_3000 50% threshold stable stable one family, votes split four ways noise share of 500 bootstrap resamples in which the predictor was selected Report the road-length family and its plausible scale range, not the single radius that won

Validation & Edge Cases

A nested score far below the flat score. This is the normal result, not a bug. A flat fit of 0.78 alongside a nested fit of 0.41 tells you the selection was doing most of the work. Publish the nested number.

Folds that select wildly different models. Ten outer folds producing ten distinct term sets means the campaign cannot identify a unique model. Either reduce the candidate set on physical grounds, or accept a fixed model specified in advance from prior literature and fitted without selection.

A candidate set that grew during the analysis. Adding a new buffer radius or a new land-cover class after seeing the first results resets every guarantee the nested design provides, because the addition was informed by the outcome. If the candidate set must change, the honest description is that the earlier analysis was exploratory and the nested validation now applies to the enlarged set from the beginning. Record the version of the candidate list that each reported statistic was computed against.

Sites removed for being poorly predicted. Dropping the two monitors the model fits worst raises every statistic on the page and is indefensible unless the removal has an independent justification — a documented instrument fault, a relocation mid-campaign, a construction site next to the inlet. Removals must be decided from field records, listed explicitly, and the analysis reported both with and without them so a reader can see what the exclusion bought.

A permutation null that is not small. With two hundred candidates and sixty sites, a null median in-sample fit above 0.4 is ordinary. If the real model’s nested score does not clearly exceed the 95th percentile of the null, there is no evidence of a usable relationship at all.

Bootstrap resamples that break the fit. Resampling with replacement duplicates sites and can produce singular design matrices at high term counts. Catch the exception, count it, and report the number of failed resamples; a large count is itself evidence of instability.

Spatial structure ignored by random folds. Nested cross-validation with random folds still benefits from spatial proximity. For a campaign spread over a metropolitan area, add a spatially blocked outer loop as described in Cross-Validating an Exposure Surface With Held-Out Monitors.

INFO lur.overfit nested rmse=4.8812 r2=0.412 distinct_models_across_folds=7/10
INFO lur.overfit permutation null in-sample r2: median=0.361 p95=0.508 (n=200)
INFO lur.overfit stability: 2 predictors selected in >50% of resamples

Put side by side, the three diagnostics answer three different questions, and a model should only be reported when it passes all of them:

The Evidence Is the Gap Above the Null A distribution of in-sample fit obtained by running the full selection pipeline on permuted outcomes two hundred times, centred near thirty-six per cent with a ninety-fifth percentile near fifty-one per cent. The real model's flat in-sample fit sits at seventy-eight per cent and its nested cross-validated fit at forty-one per cent. The nested score lies inside the upper tail of the null distribution, so the apparent seventy-eight per cent is mostly selection, and the model has not demonstrated a usable relationship. Compare the honest score to what noise alone achieves permutation null null p95 nested 0.41 flat 0.78 0.2 0.5 0.8 explained variance · the nested score sits inside the null's upper tail verdict: no evidence of a usable relationship at this site count

Compliance Notes

The record for a land-use regression must distinguish what was fixed in advance from what was chosen by the data. Store the full candidate list as it stood before any fitting, the selection rules and their thresholds, the nested design and its seed, the nested error and fit, the permutation null distribution summary, and the bootstrap stability table.

State the flat and nested numbers together in any write-up, with the nested one as the headline. A study that quotes only the flat statistic has not necessarily done anything wrong internally, but a reader cannot tell it apart from one that has — and the correction is usually large enough to change what the exposure model is fit to be used for.