Data Engineering Foundations

Real portfolio pieces built during the course, not a certificate for its own sake. Each one is work you can show.

• Source to target mappingA row per target column: where it comes from, the transformation applied, the type, whether nulls are allowed, and what happens when the source sends something unexpected.
• Schema design with decisionsThe target tables you designed, with the grain of each stated in one sentence, the keys, the partitioning choice, and a short written reason for every decision you could have made another way.
• ETL pipeline definitionOne pipeline written out end to end: schedule, dependencies, extract, load strategy, transformations, the re-run behaviour, and the SQL that makes it idempotent.
• Data quality check suiteA set of checks written as code with thresholds, severities and an owner for each, covering freshness, volume, uniqueness, nulls, referential integrity and a business rule.
• Pipeline runbookThe page someone reads at 3am: what this pipeline does, what it depends on, every known failure and the exact recovery, how to backfill safely, and who to tell.
• Restatement noteThe note finance can forward without editing: which figure is correct and which was presented, why it moved, which days were rewritten and by how much, whether it is expected to move again, and the one control that would have shown it before the review rather than after it.