Data Engineering Course

Lesson 1 • Introduction to Data Architecture Patterns

Surveys foundational architectural styles including data warehouses, data lakes, and lakehouses. Connects architectural choices to business requirements and scalability goals.

Lesson 2 • Core Data Concepts and Terminology

Introduces structured, semi-structured, and unstructured data types and key storage abstractions. Ensures shared vocabulary before diving into architecture and tooling.

Lesson 3 • Data Engineering Lifecycle Overview

Maps the end-to-end journey of data from source to consumption. Provides a mental model that frames every technical topic covered later in the course.

Lesson 1 • Advanced SQL for Data Engineering

Introduces window functions, CTEs, and set operations for complex analytical transformations. Prepares learners to write production-grade SQL used in pipelines and reporting layers.

Lesson 2 • Dimensional Modeling for Analytics

Applies star and snowflake schema patterns to design analytics-ready data models. Directly enables the data warehouse and transformation work covered in later chapters.

Lesson 3 • Core SQL Querying Techniques

Teaches SELECT, filtering, aggregation, and multi-table joins for data retrieval. These skills underpin every transformation and analytical workload in the course.

Lesson 1 • Python Essentials for Data Work

Reviews Python syntax, data structures, and control flow relevant to data manipulation tasks. Establishes the programming foundation required for pipeline scripting throughout the course.

Lesson 2 • File and Data Format Handling

Teaches reading and writing CSV, JSON, Parquet, and Avro files using Python libraries. Directly supports ingestion and serialisation tasks in pipeline development chapters.

Lesson 3 • Automation and Scripting Best Practices

Introduces logging, configuration management, and packaging for production-ready Python scripts. Prepares learners to write maintainable code that integrates into orchestrated pipelines.

Lesson 4 • Data Manipulation with Pandas

Covers DataFrame operations, merging, reshaping, and cleaning using the Pandas library. Provides the transformation toolkit used in batch processing and exploratory data work.

Lesson 1 • Extracting Data from Relational Sources

Covers CDC, timestamp-based extraction, and query-based pulls from relational databases. Builds on SQL skills to implement reliable, low-impact extraction strategies.

Lesson 2 • Pipeline Design Principles

Introduces modularity, idempotency, error handling, and observability as pipeline design pillars. Establishes engineering standards applied throughout all subsequent pipeline chapters.

Lesson 3 • Ingestion Patterns and Source Types

Surveys common data sources including databases, APIs, files, and event streams. Frames the ingestion design decisions that drive pipeline architecture choices.

Lesson 4 • Ingesting from APIs and File Sources

Teaches structured approaches to consuming REST APIs, FTP sources, and cloud storage files. Extends Python API skills into production ingestion pipeline patterns.

Lesson 1 • Table Formats and Open Storage Standards

Introduces open table formats that enable ACID transactions and schema evolution on object storage. Prepares learners for lakehouse implementations covered in advanced chapters.

Lesson 2 • Cloud Data Warehouse Architecture

Examines columnar storage, MPP query engines, and separation of compute and storage. Connects warehouse internals to query performance and cost optimisation decisions.

Lesson 3 • Data Modeling in the Warehouse

Applies dimensional and vault modeling techniques within cloud warehouse environments. Builds on relational modeling skills to produce analytics-ready warehouse layers.

Lesson 4 • Cloud Object Storage Fundamentals

Explains object storage concepts, bucket organization, partitioning, and access control. Provides the storage foundation underlying data lakes and lakehouse architectures.

Lesson 1 • Advanced dbt Patterns

Covers macros, Jinja templating, packages, and incremental strategies for complex use cases. Extends core dbt skills to handle large-scale and dynamic transformation requirements.

Lesson 2 • Testing and Documentation in dbt

Implements schema tests, custom tests, and auto-generated documentation within dbt projects. Enforces data quality and maintainability standards in transformation pipelines.

Lesson 3 • Transformation Layer Architecture

Defines staging, intermediate, and mart transformation layers and their responsibilities. Establishes the layered design pattern that structures all dbt project work in this chapter.

Lesson 1 • Pipeline Monitoring and Observability

Establishes logging, metrics, and lineage tracking practices for orchestrated pipeline visibility. Connects observability tooling to the data quality and governance themes of the course.

Lesson 2 • Modern Orchestration Alternatives

Surveys Prefect, Dagster, and event-driven orchestration as alternatives to Airflow. Equips learners to evaluate orchestration tools against project requirements and team constraints.

Lesson 3 • Error Handling and Retry Strategies

Implements retries, SLAs, alerting, and failure callbacks within orchestrated workflows. Ensures pipelines meet reliability standards required in production data environments.

Lesson 4 • Orchestration Concepts and Patterns

Introduces DAG-based scheduling, dependency resolution, and orchestration vs. execution separation. Frames the orchestration layer that ties ingestion, storage, and transformation together.

Lesson 1 • Streaming Data Storage and Serving

Addresses writing streaming outputs to data lakes, warehouses, and real-time serving stores. Connects streaming pipelines to the storage and consumption layers built in earlier chapters.

Lesson 2 • Streaming Pipeline Reliability and Scaling

Covers backpressure, schema registry, dead-letter topics, and horizontal scaling for streaming systems. Ensures learners can operate streaming pipelines reliably at production scale.

Lesson 3 • Stream Processing with Apache Flink

Teaches stateful stream processing, windowing, and event-time handling using Apache Flink. Builds on Kafka knowledge to implement real-time transformation and aggregation logic.

Lesson 4 • Streaming Fundamentals and Use Cases

Contrasts streaming with batch processing and identifies use cases where real-time data is required. Establishes the conceptual foundation for all streaming architecture and tooling decisions.