Data Engineering with Databricks Course

Lesson 1 • Cluster Configuration and Management

Teaches cluster creation, sizing, autoscaling, and termination policies. Students gain control over compute costs and performance from the start.

Lesson 2 • Databricks Architecture Overview

Covers the control plane, data plane, and cluster architecture underpinning Databricks. Grounds all subsequent platform usage in a clear mental model.

Lesson 3 • Workspace Navigation and Setup

Introduces the Databricks UI, folder structure, and user settings. Enables students to organize projects and manage personal environments effectively.

Lesson 4 • Databricks CLI and REST API Basics

Introduces programmatic access to the platform via CLI and REST API. Prepares students for automation tasks covered in later chapters.

Lesson 1 • Transformations and Actions

Distinguishes narrow vs. wide transformations and explains action triggers. Students learn to reason about shuffle costs and execution boundaries.

Lesson 2 • Spark Distributed Execution Model

Explains RDDs, DAGs, stages, and tasks within Spark's execution engine. Provides the conceptual foundation for writing efficient distributed code.

Lesson 3 • DataFrame API Fundamentals

Covers DataFrame creation, schema inference, and core transformations. Students apply these operations as the primary data manipulation interface.

Lesson 4 • Spark Performance Fundamentals

Introduces the Spark UI, execution plans, and basic tuning levers. Equips students to diagnose and address common performance bottlenecks.

Lesson 1 • Delta Lake Storage Format

Explains Parquet files, the transaction log, and checkpoint files that form Delta Lake. Students understand why Delta provides reliability over raw file formats.

Lesson 2 • Creating and Managing Delta Tables

Covers DDL for creating, altering, and dropping Delta tables. Establishes the table management skills required for all pipeline work ahead.

Lesson 3 • ACID Transactions and Concurrency

Teaches optimistic concurrency control, isolation levels, and conflict resolution in Delta. Students can design pipelines that safely handle concurrent writes.

Lesson 4 • Delta Lake Optimization Techniques

Covers OPTIMIZE, Z-ordering, and file compaction to improve query performance. Students apply these techniques to production-grade Delta tables.

Lesson 1 • ELT Design Patterns

Presents medallion architecture and incremental load strategies for structured ELT. Students design multi-hop pipelines that separate raw, cleansed, and curated layers.

Lesson 2 • Batch Ingestion from Common Sources

Teaches reading from cloud object storage, JDBC databases, and file formats. Provides the ingestion foundation for all pipeline patterns in this chapter.

Lesson 3 • Data Quality Enforcement at Ingestion

Teaches constraint-based validation, quarantine patterns, and expectation frameworks. Ensures data quality is enforced before data reaches downstream consumers.

Lesson 4 • Structured Streaming Fundamentals

Covers streaming DataFrames, triggers, and output modes for continuous data processing. Connects streaming concepts to the Delta Lake sink used in production.

Lesson 1 • Pipeline Configuration and Deployment

Teaches pipeline settings, cluster policies, and deployment via UI and API. Students configure production-ready pipelines with appropriate compute and storage settings.

Lesson 2 • Monitoring and Troubleshooting DLT

Explores the event log, pipeline UI metrics, and common failure patterns. Students diagnose and resolve pipeline issues using built-in observability tools.

Lesson 3 • Data Quality with Expectations

Covers EXPECT, EXPECT OR DROP, and EXPECT OR FAIL constraints in DLT. Students embed quality rules directly into pipeline definitions for automated enforcement.

Lesson 4 • Orchestrating Pipelines with Databricks Workflows

Integrates DLT pipelines into multi-task Databricks Workflows with dependencies and scheduling. Students build end-to-end orchestrated data products.

Lesson 1 • Identity and Access Management

Covers users, groups, service principals, and privilege grants in Unity Catalog. Students implement least-privilege access models for data assets.

Lesson 2 • Unity Catalog Architecture

Explains the metastore, catalog, schema, and table hierarchy in Unity Catalog. Provides the structural understanding needed to design governed namespaces.

Lesson 3 • Sharing Data with Delta Sharing

Introduces the open Delta Sharing protocol for cross-platform and cross-organization data sharing. Students publish and consume shared datasets securely.

Lesson 4 • Auditing and Compliance

Teaches audit log access, query history, and compliance reporting patterns. Students configure monitoring to satisfy enterprise audit requirements.

Lesson 1 • Cost-Based Optimization and Statistics

Covers ANALYZE TABLE, column statistics, and the cost-based optimizer's join reordering. Students collect and leverage statistics to improve plan quality.

Lesson 2 • Memory Management and Spill

Covers Spark memory regions, executor sizing, and spill detection and mitigation. Students configure memory settings to eliminate costly disk spill.

Lesson 3 • Photon Engine and Vectorized Execution

Introduces Databricks Photon engine and its vectorized execution benefits. Students identify workloads that benefit most from Photon and enable it appropriately.

Lesson 4 • Partitioning and Shuffling Strategies

Teaches optimal partition counts, repartition vs. coalesce, and shuffle tuning. Students eliminate data skew and reduce shuffle overhead in complex pipelines.

Lesson 1 • Cost Management and FinOps

Covers DBU consumption analysis, cluster right-sizing, and spot instance strategies. Students reduce compute costs without sacrificing pipeline reliability.

Lesson 2 • CI/CD for Databricks Projects

Teaches Databricks Asset Bundles, deployment automation, and environment promotion. Students implement a full CI/CD pipeline from development to production.

Lesson 3 • Security Hardening and Best Practices

Teaches network isolation, secret management, and secure credential handling. Students apply defense-in-depth principles to protect data assets in production.

Lesson 4 • Testing Data Pipelines

Covers unit testing with pytest, integration testing strategies, and test data management. Students build a test suite that validates pipeline logic before deployment.