Databricks ML in Action Course

Lesson 1 • Notebooks and Collaborative Development

Introduces notebook creation, magic commands, and co-authoring features. Establishes reproducible development habits used in all later ML experiments.

Lesson 2 • Workspace Architecture and Navigation

Covers the Databricks control plane, data plane, and workspace UI. Grounds students in the environment they will use throughout every subsequent chapter.

Lesson 3 • Data Access and Storage Integration

Explains mount points, Unity Catalog, and credential management for cloud storage. Enables secure, consistent data access required for ML pipelines.

Lesson 4 • Databricks CLI and REST API Basics

Introduces programmatic workspace control via CLI and REST endpoints. Prepares students to automate tasks and integrate Databricks into external toolchains.

Lesson 1 • Delta Live Tables for Pipeline Orchestration

Introduces declarative pipeline definitions, expectations, and lineage tracking. Automates multi-stage data quality enforcement before features reach model training.

Lesson 2 • Data Transformation with Spark SQL

Covers aggregations, window functions, and complex type handling in Spark SQL. Transforms raw data into structured features consumed by ML models.

Lesson 3 • Apache Spark Core Concepts

Covers RDDs, DataFrames, and the Spark execution model. Provides the computational foundation for all data processing tasks in later ML chapters.

Lesson 4 • Delta Lake for ML Workloads

Introduces ACID transactions, time travel, and Delta table optimisation. Ensures data reliability and reproducibility across iterative ML experiments.

Lesson 1 • Eliminating Training-Serving Skew

Addresses consistency between offline training data and online serving features. Students implement lookup-based training sets that mirror production inference paths.

Lesson 2 • Databricks Feature Store Architecture

Explains Feature Store tables, metadata, and the offline-online split. Students understand how features are stored, versioned, and retrieved for training and serving.

Lesson 3 • Writing and Reading Feature Tables

Teaches the write and read APIs for batch and streaming feature updates. Connects ingestion pipelines from Chapter 2 to downstream model training workflows.

Lesson 4 • Feature Engineering Fundamentals

Covers encoding, scaling, imputation, and interaction terms using Spark. Establishes the feature vocabulary applied throughout the Feature Store and model training chapters.

Lesson 1 • Training Deep Learning Models

Introduces TensorFlow and PyTorch training loops integrated with MLflow logging. Prepares students for distributed deep learning covered in the advanced training chapter.

Lesson 2 • Hyperparameter Tuning with Hyperopt

Teaches Bayesian optimisation and parallel search using Hyperopt and SparkTrials. Students systematically improve model performance beyond manual grid search.

Lesson 3 • MLflow Tracking Fundamentals

Introduces experiments, runs, parameters, metrics, and tags in MLflow. Establishes the logging discipline that underpins reproducible ML throughout the course.

Lesson 4 • Training Scikit-learn and Spark ML Models

Covers pipeline construction, cross-validation, and MLflow autologging for both frameworks. Students train and log baseline models that serve as benchmarks for advanced techniques.

Lesson 1 • Registering and Versioning Models

Teaches programmatic and UI-based model registration from MLflow runs. Students establish a consistent versioning discipline for all trained models.

Lesson 2 • Stage Transitions and Approval Workflows

Covers Staging, Production, and Archived transitions with webhook-based notifications. Implements a controlled promotion process that mirrors enterprise MLOps standards.

Lesson 3 • Unity Catalog Model Registry

Explains the Unity Catalog-backed Model Registry for cross-workspace governance. Students migrate from workspace-level to catalog-level model management for enterprise scale.

Lesson 4 • Model Signatures and Input Examples

Introduces MLflow model signatures, input examples, and schema enforcement. Prevents runtime type errors and documents expected model interfaces for downstream consumers.

Lesson 1 • Distributed Hyperparameter Tuning at Scale

Extends Hyperopt SparkTrials and introduces Optuna for large-scale parallel search. Students tune deep learning models across hundreds of concurrent trials efficiently.

Lesson 2 • TorchDistributor for PyTorch Training

Introduces TorchDistributor as the native Databricks API for distributed PyTorch. Students launch multi-GPU and multi-node PyTorch jobs without manual cluster configuration.

Lesson 3 • Distributed Training Concepts

Covers data parallelism, model parallelism, and gradient synchronization strategies. Provides the theoretical grounding needed before implementing distributed frameworks.

Lesson 4 • Horovod for Distributed Deep Learning

Teaches Horovod initialization, rank-based data sharding, and HorovodRunner on Databricks. Students convert single-node training scripts to multi-node distributed jobs.

Lesson 1 • Databricks Model Serving Architecture

Explains serverless model serving, endpoint types, and traffic routing. Orients students to the serving layer before configuring and testing live endpoints.

Lesson 2 • External Deployment Targets

Covers exporting models to ONNX, Docker containers, and cloud-native serving platforms. Prepares students to deploy Databricks-trained models outside the Databricks environment.

Lesson 3 • Custom Inference Logic with PyFunc

Teaches the MLflow PyFunc flavor for wrapping arbitrary Python logic as a deployable model. Students add preprocessing, postprocessing, and ensemble logic to serving endpoints.

Lesson 4 • Deploying MLflow Models to Endpoints

Covers endpoint creation, model version pinning, and REST API invocation. Students deploy registered models from Chapter 5 and validate responses end-to-end.

Lesson 1 • Audit, Lineage, and Compliance

Covers Unity Catalog lineage, MLflow audit logs, and access governance for regulated environments. Students document model provenance to satisfy internal and external audit requirements.

Lesson 2 • Lakehouse Monitoring for ML

Introduces Databricks Lakehouse Monitoring for automated metric computation on inference tables. Provides out-of-the-box dashboards for model quality and data health.

Lesson 3 • MLOps Principles and Maturity Levels

Defines MLOps maturity from manual experimentation to fully automated pipelines. Frames the governance and automation goals students implement in subsequent sections.

Lesson 4 • Data and Model Drift Detection

Covers statistical tests for covariate shift, label drift, and prediction drift. Students implement drift alerts that trigger retraining pipelines automatically.