In 2011, 5.8% of Senegalese adults had a financial account. In the 2024 Global Findex round, that number is 76.5%, a 70-point jump in thirteen years, driven overwhelmingly by mobile money. That single statistic is why I work in fintech data.

It’s also the output of a pipeline I built end-to-end: west-africa-financial-inclusion, a real-data ELT project tracking financial inclusion across the 8 WAEMU countries. No synthetic data, no CSV downloaded once and frozen: the pipeline hits the live World Bank API every run.

World Bank API ──▶ ingestion (retry, BOM quirk, idempotent load) ──▶ DuckDB ──▶ dbt (staging + marts + 17 tests)
        └── Airflow: one mapped extract task per indicator

Four indicators: Findex account ownership, inflation, mobile subscriptions, GDP per capita. About 650 rows per run. Small data, but every lesson below scales.

Why real data, not synthetic

Most portfolio pipelines run on generated data, and it shows: generated data is clean. It never has an encoding quirk, a missing country, or a survey that skipped a year. Real data has all three, and handling them is the actual job of a data engineer. Three examples from this one small pipeline:

The API returns a BOM. The World Bank JSON endpoint prepends a UTF-8 byte order mark that breaks response.json(). You find this at row zero, minute one, and never with synthetic data. The ingestion client strips it explicitly and carries a retry-with-backoff wrapper for the API’s occasional 5xx moods.

Findex is a survey, not a sensor. Account-ownership data exists only for survey years (2011, 2014, 2017, 2021, 2024), and not every country joined every round: Côte d’Ivoire’s first data point is 2014. The staging models must distinguish “no data collected” from “zero”, which is a modeling decision, not a cleaning step.

Guinea-Bissau isn’t covered by Findex at all. Eight WAEMU countries, seven with data. The lazy fix is to drop the country and present a clean 7-row mart. The pipeline instead surfaces the gap: Guinea-Bissau appears in the country dimension with explicit nulls, because a data product that hides its blind spots is lying to its users.

The boring virtues: idempotency and mapped tasks

The load is idempotent (delete-then-insert per indicator and run scope), so re-running an Airflow task never duplicates rows. This is table stakes, and it’s precisely what an append-only quick script gets wrong.

On the orchestration side, Airflow dynamic task mapping creates one extract task per indicator. Four indicators today, one line of config to add a fifth, and a failed indicator retries alone instead of re-pulling everything (a miniature of the stateful-retry principle).

dbt on DuckDB: the 17 tests are the point

Downstream, dbt builds staging models and marts on DuckDB, a real columnar warehouse that runs in CI for free. The project carries 17 dbt tests on live data: uniqueness of (country, indicator, year), accepted country codes, non-null constraints where data must exist, and range checks on percentages.

Tests on synthetic data verify your assumptions. Tests on live data verify the world: when the World Bank restates a series or a new Findex round lands, a test failing is the pipeline telling me the data changed before a user does. That’s not a portfolio trick; that’s the entire reason dbt tests exist.

The payoff

The final mart reads like a story. Senegal +70.6 points. Togo +47.2. Mali +46.5. And Niger, still at 14.8%, a reminder that the mobile-money revolution is real but uneven.

A pipeline on synthetic data demonstrates that you can write code. A pipeline on live data demonstrates that you can handle reality (encoding bugs, missing surveys, absent countries) and still ship numbers you’d defend in a meeting. If you’re building a portfolio project, pick a real API. The mess is the curriculum.