Data Engineering Course

Lesson 1 • The Data Engineering Landscape

Defines data engineering scope, distinguishing it from data science and analytics. Establishes vocabulary used throughout the course.

Lesson 2 • Core Data Infrastructure Components

Introduces databases, data warehouses, data lakes, and lakehouses as distinct infrastructure patterns. Prepares learners to select appropriate storage architectures.

Lesson 3 • Data Pipelines and Workflow Concepts

Explains what a data pipeline is, its stages, and how orchestration ties stages together. Grounds learners in pipeline thinking before hands-on implementation.

Lesson 1 • Dimensional Modeling for Analytics

Covers star and snowflake schemas, fact tables, and dimension tables for analytical workloads. Bridges relational theory to warehouse design in later chapters.

Lesson 2 • Advanced SQL Techniques

Teaches window functions, CTEs, and set operations for complex analytical queries. Enables learners to handle real-world reporting and transformation logic.

Lesson 3 • SQL Performance and Optimisation

Examines query execution plans, partitioning, and indexing strategies to improve query speed. Prepares learners to write production-grade SQL at scale.

Lesson 4 • Relational Data Modeling Principles

Introduces normalisation forms and entity-relationship modelling for transactional systems. Provides the schema design foundation needed before analytical modelling.

Lesson 1 • Streaming Ingestion Fundamentals

Introduces event streaming concepts, producers, consumers, and topics for real-time ingestion. Lays groundwork for the streaming processing chapter that follows.

Lesson 2 • Batch Ingestion Techniques

Covers full-load, incremental, and change-data-capture (CDC) patterns for batch extraction. Teaches learners to balance completeness with ingestion cost.

Lesson 3 • API and Web Data Ingestion

Demonstrates extracting data from REST APIs, handling pagination, authentication, and rate limits. Connects to real-world SaaS and third-party data integration scenarios.

Lesson 4 • Understanding Source System Patterns

Surveys OLTP databases, APIs, event streams, and flat files as common data sources. Sets context for choosing the right ingestion strategy per source type.

Lesson 1 • Cloud Data Warehouse Architecture

Examines separation of compute and storage, MPP query engines, and concurrency scaling. Provides the architectural context for all warehouse design decisions.

Lesson 2 • Data Lake and Object Storage Design

Covers folder hierarchy, file format selection, and partition pruning in object storage environments. Prepares learners to manage raw and curated zones in a data lake.

Lesson 3 • Warehouse Schema Design Patterns

Applies dimensional modelling from Chapter 2 to physical warehouse schema design with partitioning and clustering. Bridges theory to production warehouse implementation.

Lesson 4 • Table Formats and Open Standards

Introduces open table formats that add ACID transactions and schema evolution to data lakes. Enables learners to implement lakehouse patterns on cloud object storage.

Lesson 1 • Python-Based Data Transformation

Introduces pandas and PySpark for programmatic transformation of large datasets. Expands learners' toolset beyond SQL for complex or non-relational transformations.

Lesson 2 • Transformation with SQL-Based Tools

Teaches modular SQL transformation using layered modeling conventions such as staging, intermediate, and mart layers. Connects SQL skills from Chapter 2 to production workflows.

Lesson 3 • Data Cleaning and Quality Enforcement

Covers deduplication, null handling, type casting, and constraint validation during transformation. Directly improves the reliability of downstream analytical outputs.

Lesson 4 • ETL vs. ELT Paradigms

Compares extract-transform-load and extract-load-transform architectures and their trade-offs. Guides learners in selecting the right paradigm for a given platform.

Lesson 1 • Building and Configuring DAGs

Covers authoring DAGs with operators, sensors, and hooks for common data tasks. Translates orchestration concepts into working pipeline code.

Lesson 2 • Orchestration Concepts and Patterns

Defines DAGs, task dependencies, triggers, and scheduling intervals as orchestration primitives. Establishes the mental model for all hands-on orchestration work.

Lesson 3 • Pipeline Monitoring and Observability

Establishes logging, metrics collection, and alerting strategies for orchestrated pipelines. Prepares learners to maintain pipeline health in production.

Lesson 4 • Scheduling and Dependency Management

Teaches cron-based scheduling, cross-DAG dependencies, and data-aware scheduling patterns. Enables learners to coordinate complex multi-pipeline workflows.

Lesson 1 • Stream Processing Operations

Teaches filtering, mapping, aggregation, and joining on unbounded event streams. Enables learners to implement core transformation logic in a streaming context.

Lesson 2 • Streaming Architecture Fundamentals

Contrasts streaming with batch processing and introduces Lambda and Kappa architecture patterns. Provides the design vocabulary for all streaming implementation work.

Lesson 3 • Streaming Pipeline Integration and Output

Covers sinking processed streams to databases, warehouses, and downstream topics. Connects streaming outputs to the broader data platform built in earlier chapters.

Lesson 4 • Stateful Streaming and Fault Tolerance

Addresses state stores, checkpointing, and watermarks for reliable stateful stream processing. Ensures learners can build streaming pipelines that survive failures.

Lesson 1 • Regulatory Compliance and Data Privacy

Addresses data residency, retention obligations, consent management, and the right to erasure. Prepares learners to build platforms that meet privacy and compliance requirements.

Lesson 2 • Data Catalog and Metadata Management

Covers data discovery, lineage tracking, and business glossary management in a data catalog. Enables teams to find, understand, and trust data assets.

Lesson 3 • Access Control and Data Security

Implements role-based access control, column masking, and row-level security on data assets. Protects sensitive data while enabling appropriate analytical access.

Lesson 4 • Data Contracts and Expectations

Introduces schema contracts, column-level expectations, and automated quality gates in pipelines. Prevents bad data from propagating to downstream consumers.