Big Data Engineer Course

Lesson 1 • Data Engineer Role and Responsibilities

Defines the data engineer's scope versus data scientists and analysts. Clarifies ownership of pipelines, reliability, and data quality.

Lesson 2 • Big Data Architecture Patterns

Introduces Lambda and Kappa architectures and their trade-offs. Provides the structural context for all pipeline designs covered later.

Lesson 3 • Defining Big Data and Its Dimensions

Covers the 5 Vs framework and real-world data growth drivers. Establishes vocabulary used throughout the entire course.

Lesson 1 • Networking Concepts for Data Engineers

Explains TCP/IP, DNS, ports, and firewalls in the context of distributed systems. Enables students to troubleshoot connectivity issues in cluster environments.

Lesson 2 • Cloud Infrastructure Fundamentals

Introduces compute, storage, and networking primitives across major cloud providers. Students can deploy and configure virtual machines (VMs) and object storage buckets.

Lesson 3 • Containerization with Docker

Teaches building and running containers for reproducible data engineering environments. Containers are used in labs starting in the next chapter.

Lesson 1 • HDFS Architecture and Internals

Explains NameNode, DataNode roles, block replication, and fault tolerance. Provides the storage foundation required before introducing processing frameworks.

Lesson 2 • Data Formats for Distributed Storage

Compares row-based and columnar formats including Avro, Parquet, and ORC. Format choice directly impacts query performance in later processing chapters.

Lesson 3 • Cloud Object Storage Integration

Demonstrates using cloud object storage as a drop-in HDFS replacement. Prepares students for cloud-native pipeline architectures introduced later.

Lesson 4 • Data Lake Design Principles

Introduces zone-based data lake architecture: raw, curated, and consumption layers. Students design storage layouts that support downstream analytics pipelines.

Lesson 1 • Spark Performance Tuning

Addresses partitioning, caching, serialization, and memory management for production jobs. Students reduce job runtimes and resource costs significantly.

Lesson 2 • Spark Architecture and Execution Model

Explains driver, executor, DAG scheduler, and task execution. Understanding internals is prerequisite to writing efficient Spark code.

Lesson 3 • RDDs, DataFrames, and Datasets

Covers the three Spark abstraction layers and when to use each. Builds from low-level RDD operations to high-level DataFrame transformations.

Lesson 4 • Spark SQL and Data Manipulation

Teaches SQL queries, window functions, and complex aggregations on DataFrames. Enables analysts-turned-engineers to leverage existing SQL skills in Spark.

Lesson 1 • Batch Ingestion with Apache Sqoop and Alternatives

Covers bulk relational database extraction using Sqoop and modern alternatives. Completes the ingestion toolkit for both streaming and batch source systems.

Lesson 2 • Producing and Consuming Messages

Covers producer acknowledgment settings, consumer offset management, and exactly-once semantics. Students write reliable producers and consumers in Python and Java.

Lesson 3 • Change Data Capture Techniques

Introduces CDC using database transaction logs to capture row-level changes. Enables near-real-time synchronization between operational databases and the data lake.

Lesson 4 • Apache Kafka Architecture

Explains brokers, topics, partitions, consumer groups, and replication. This architecture underpins all streaming pipelines built in subsequent chapters.

Lesson 1 • Streaming Pipeline Reliability and Monitoring

Addresses backpressure, consumer lag, failure recovery, and alerting for streaming jobs. Ensures students can operate streaming pipelines in production.

Lesson 2 • Apache Flink Architecture and APIs

Covers Flink's JobManager, TaskManager, DataStream API, and Table API. Students write stateful streaming jobs with exactly-once guarantees.

Lesson 3 • Stream Processing Fundamentals

Defines event time, processing time, watermarks, and windowing semantics. These concepts are prerequisite to writing correct streaming applications.

Lesson 4 • Spark Structured Streaming

Teaches the micro-batch and continuous processing modes of Spark Structured Streaming. Leverages existing Spark DataFrame skills for streaming workloads.

Lesson 1 • Cloud Data Warehouse Platforms

Compares MPP cloud warehouse architectures and their storage-compute separation models. Students load, transform, and query large datasets using warehouse SQL.

Lesson 2 • Apache Iceberg and Hudi Formats

Covers Iceberg's hidden partitioning and Hudi's upsert capabilities as alternatives to Delta Lake. Students select the right open table format for their use case.

Lesson 3 • Delta Lake and Table Formats

Introduces ACID transactions, time travel, and schema enforcement in Delta Lake. Bridges the gap between raw data lake storage and warehouse-grade reliability.

Lesson 4 • Data Warehouse Modeling Fundamentals

Covers star schema, snowflake schema, slowly changing dimensions, and fact table design. Provides the modeling vocabulary used in warehouse and lakehouse implementations.

Lesson 1 • Apache Airflow Architecture and DAGs

Covers Airflow's scheduler, executor, metadata database, and DAG authoring. Establishes the orchestration foundation for all pipeline automation in this chapter.

Lesson 2 • CI/CD and Infrastructure as Code for Pipelines

Applies Git workflows, automated testing, and Terraform to pipeline deployment. Students deliver changes to production safely and repeatably.

Lesson 3 • Advanced DAG Design Patterns

Teaches dynamic DAG generation, branching, and cross-DAG dependencies. Enables students to build flexible, reusable orchestration patterns.

Lesson 4 • Data Quality and Pipeline Testing

Implements data quality checks using Great Expectations and dbt tests within pipelines. Prevents bad data from propagating to downstream consumers.