The most interesting data engineering happening right now sits at the seam with machine learning: pipelines where the “data product” at the end is a model decision, and where a silent data bug doesn’t break a dashboard; it breaks a lending decision for a real person.

I built airflow-batch-credit-scoring to work through that seam properly. It’s the second half of a two-repo platform: credit-scoring-dbt turns raw mobile-money transactions into a credit_features table, and the Airflow project re-scores the full customer base from it every night.

list_chunks ──▶ score_chunk (mapped, ×N) ──▶ drift_gate (PSI) ──▶ publish

Three design decisions carry the whole thing.

1. The model is code, and the version lands next to every score

The scoring model is a versioned logistic model whose coefficients live in the repository, which means a coefficient change is a pull request, with a reviewer and a diff, not a mystery binary swapped in a bucket. And every score row is written with its model_version, so any historical decision can be traced back to the exact model that produced it.

For anything touching credit, that auditability isn’t a nice-to-have. When someone asks “why was this customer scored 412 in March?”, the answer must be reconstructible.

2. Chunked scoring, because “score everyone” must scale and retry

The DAG lists the customer base, splits it into chunks, and scores each chunk in a dynamically mapped task with capped parallelism. Familiar data-engineering mechanics applied to an ML workload: a failed chunk retries alone, parallelism doesn’t stampede the warehouse, and scoring stays read-only until the final atomic publish step.

3. The drift gate: PSI as a circuit breaker

This is the part I’d defend in any architecture review. Before publishing, the DAG computes the Population Stability Index between tonight’s score distribution and yesterday’s:

PSI = Σ (actual% − expected%) × ln(actual% / expected%)   over score bins

PSI is the standard credit-risk measure of distribution shift. The rule-of-thumb thresholds: under 0.10, stable; 0.10–0.25, worth a look; above 0.25, something changed materially. In this pipeline, PSI > 0.25 doesn’t send a Slack warning to be ignored at 9am: it fails the run and withholds publication. Yesterday’s scores stay live.

Why so blunt? Think about what actually causes a sudden distribution shift in a nightly batch: almost never “the population changed overnight,” almost always an upstream feature broke: a source table half-loaded, a dbt model that started emitting nulls, a currency column that changed units. The scores are wrong, they look plausible individually, and only the distribution gives the game away. A drift gate is a data-quality test that operates on the model’s output instead of the model’s input, and it catches the failures the input tests can’t.

The subtle case is the first run, when there’s no yesterday to compare against. The gate has to handle an empty baseline explicitly (pass, and record) rather than crash or, worse, silently skip forever.

Test the seam, not just the sides

The pytest suite is where DE and ML practice genuinely merge: score bounds and monotonicity (riskier behavior must produce a lower score, a property test on the model), band mapping, PSI behavior on identical / slightly shifted / heavily drifted distributions, the empty-baseline first run, and plain old DAG structure tests.

ML people test models. Data engineers test pipelines. In production the failures live between the two: a perfect model fed by a broken feature is a broken system. Pipelines that score people need circuit breakers, and the circuit breaker is a data engineering artifact.

That’s the direction I’m pushing my work: not “data engineering or AI”, but pipelines where the AI is a first-class, versioned, gated, testable component, boring in exactly the way production systems should be.