Picture a nightly dbt run with 60 models. Model 41 fails at 3am: a transient BigQuery error, an upstream hiccup, whatever. Airflow does what Airflow does: it retries the task. And your pipeline re-runs all 60 models from scratch, including the 40 that had already succeeded.

On a serious BigQuery project that’s not just slow; it’s money. Every retried model burns slots again, and your SLA quietly slips while dbt rebuilds tables that were perfectly fine.

This is the problem I solved in production, on a stack running dbt on Airflow with KubernetesPodOperator. The result: retry runtimes cut substantially across staging and warehouse tasks. Here’s the pattern.

dbt already knows what failed

dbt writes two artifacts on every invocation:

  • manifest.json: the compiled project, every model, test and dependency edge.
  • run_results.json: the status of each node from the last run: success, error, skipped.

Combine them with state selection and dbt can re-run only failures and everything downstream of them:

dbt run \
  --select result:error+ \
  --state /artifacts/last_run

That + suffix matters: a failed model’s children were skipped, so they need to run too. result:error+ picks up both.

The catch: Kubernetes pods are amnesiacs

On KubernetesPodOperator, every task attempt is a fresh pod. The artifacts from attempt 1 die with the container, so attempt 2 has no run_results.json to select against. State selection is useless without state.

The fix is boring and effective: persist the artifacts to GCS at the end of every run, success or failure, keyed by DAG run so retries can find them.

#!/usr/bin/env bash
# entrypoint.sh runs inside the dbt pod
set -uo pipefail

STATE_URI="gs://my-dbt-state/${DAG_ID}/${RUN_ID}"

if gsutil -q stat "${STATE_URI}/run_results.json"; then
  # A previous attempt left state behind → surgical retry
  mkdir -p /artifacts/last_run
  gsutil cp "${STATE_URI}/manifest.json" "${STATE_URI}/run_results.json" /artifacts/last_run/
  dbt run --select result:error+ --state /artifacts/last_run
else
  # First attempt → normal full run
  dbt run
fi
rc=$?

# Always persist artifacts, even (especially) on failure
gsutil cp target/manifest.json target/run_results.json "${STATE_URI}/" || true
exit $rc

The Airflow side stays a plain KubernetesPodOperator with retries set; all the intelligence lives in the entrypoint. Airflow decides when to retry; dbt decides what to re-run.

The details that bite

Scope state to the DAG run, not the DAG. If you key the GCS path on the DAG alone, tomorrow’s run will “retry” against yesterday’s failures. The RUN_ID in the path guarantees each scheduled run starts clean.

Persist on failure, not just success. The whole point is that the failed run’s run_results.json drives the retry. An if-success guard around the upload defeats the mechanism. Hence the || true: the upload must never mask the real exit code.

Same selector universe on both paths. If your full run is dbt run --select staging, the retry must be --select staging,result:error+ (intersection), otherwise state selection can resurrect nodes outside the task’s scope.

Watch for schema drift between attempts. --state compares against the manifest of the failed run. If your project changed between attempts (rare mid-night, common mid-day), state:modified semantics can surprise you. For retry purposes, comparing to the same run’s manifest is exactly what you want.

Was it worth it?

Yes, measurably. Across our staging and warehouse Airflow tasks, retry runtimes dropped substantially depending on where in the DAG the failure happened. The later the failure, the bigger the win: a failure at model 55 of 60 now re-runs 5 models instead of 60.

There’s a second-order benefit too: retries became cheap enough to be aggressive. We could raise retries on flaky external dependencies without worrying about the BigQuery bill, which turned several 3am pages into next-morning log entries.

That’s the quiet goal of most data platform work: not more dashboards, but fewer pages.