Every Airflow repo I’ve seen goes through the same life cycle. The first DAG is clean. The second is a copy of the first with two lines changed. By DAG twenty, the dags/ folder is a graveyard of near-identical Python files where fixing one bug means fixing it twenty times, and hoping you found all twenty.

This is DAG proliferation, and the cure is old-fashioned factoring: if the DAGs differ only in data (schedule, tasks, dependencies), then the DAGs should be data. I built airflow-dynamic-dag-factory as a working, tested implementation of that idea: one Python entrypoint, every pipeline is a YAML file.

The shape of it

configs/*.yml  ──▶  dag_factory (validate + build)  ──▶  Airflow scheduler

Adding a pipeline is dropping a file in configs/:

dag_id: my_pipeline
schedule: "@daily"
start_date: "2026-01-01"
tasks:
  extract:
    operator: python
    callable: dag_factory.callables.extract
  load:
    operator: bash
    bash_command: "dbt run"
    depends_on: [extract]

No Python. No import boilerplate. No opportunity to copy-paste a bug.

Validation is the actual product

A naive DAG factory is ten lines: loop over YAML files, call DAG(**config). The naive version is also how you end up debugging a KeyError in the scheduler logs at deploy time.

The factory validates every config before building anything:

  • Schema: required fields present, types correct.
  • Unknown keys: a typo like scheddule: is a hard error, not a silently ignored field. This one check catches more real mistakes than all the others combined.
  • Operator params: a bash task without bash_command, or a python task whose dotted-path callable doesn’t resolve, fails fast with a message naming the file and the task.
  • Dependency cycles: depends_on edges are checked for cycles before the DAG object exists. Airflow would eventually catch this too, but “eventually, in the scheduler” is the worst place to learn about it.

Fail-fast validation moves errors from runtime in production to import time in CI. That’s the entire value proposition, and it’s why the validation code is bigger than the building code.

Keep the operator whitelist small

The factory supports three operators: bash, python (dotted-path callable) and empty. That’s deliberate.

The temptation is to expose every Airflow operator through YAML, and then you’ve reinvented Python, badly, in YAML. The factory covers the repetitive 80% of pipelines. The genuinely complex DAG (dynamic task mapping, branching, custom sensors) keeps its escape hatch: write it as a normal Python file next to the factory. Config-as-code is a default, not a prison.

Test the factory, not the DAGs

Because pipelines are data, testing becomes pleasant:

def test_unknown_key_rejected(tmp_path):
    write_config(tmp_path, "bad.yml", {"dag_id": "x", "scheddule": "@daily"})
    with pytest.raises(ConfigError, match="unknown key"):
        load_configs(tmp_path)

def test_cycle_detected(tmp_path):
    cfg = two_tasks_depending_on_each_other()
    with pytest.raises(ConfigError, match="cycle"):
        build_dag(cfg)

One pytest suite covers every current and future pipeline, because they all flow through the same code path. In CI, the suite plus a “load all real configs” smoke test means a broken YAML can’t reach the scheduler.

When it isn’t worth it

One caveat, honestly: a DAG factory earns its keep when you have many similar pipelines: per-client ELT, per-table syncs, per-country ingestion. If you have four DAGs and they’re all genuinely different, the factory is architecture astronautics; just write the Python.

But if you’ve ever fixed the same bug in five DAG files in one afternoon, you already know which side of that line you’re on.