Databricks training

Lesson 1 • Navigating the Databricks Workspace

Covers the workspace UI, resource organization, and user roles. Learners gain hands-on familiarity with the environment before writing any code.

Lesson 2 • Connecting to Data Sources

Demonstrates mounting cloud storage and connecting to external databases. Establishes data access patterns required for all data engineering and analytics tasks.

Lesson 3 • What Is the Lakehouse Architecture

Defines the lakehouse paradigm and contrasts it with data warehouses and data lakes. Establishes the architectural context for all subsequent Databricks work.

Lesson 4 • Working with Databricks Notebooks

Introduces notebook creation, magic commands, and multi-language support. Notebooks are the primary development interface used in every subsequent chapter.

Lesson 1 • Reading and Writing Data Formats

Covers reading and writing Parquet, JSON, CSV, and Delta formats with options. Proper format handling is foundational for all ingestion and output pipelines.

Lesson 2 • DataFrames and the DataFrame API

Covers DataFrame creation, schema inference, and core transformations. DataFrames are the primary abstraction used throughout all data processing chapters.

Lesson 3 • Spark Performance Basics

Introduces partitioning, caching, and broadcast joins for performance tuning. These techniques are applied in advanced optimization chapters later in the course.

Lesson 4 • Spark Architecture and Execution Model

Explains drivers, executors, DAGs, and lazy evaluation. Understanding execution flow is critical for writing efficient Spark code in later chapters.

Lesson 1 • DML Operations on Delta Tables

Teaches INSERT, UPDATE, DELETE, and MERGE operations on Delta tables. Enables learners to implement upsert patterns essential for incremental data pipelines.

Lesson 2 • Optimizing Delta Tables

Covers OPTIMIZE, ZORDER, and VACUUM commands for performance and storage management. These operations are critical for maintaining production-grade Delta tables.

Lesson 3 • Time Travel and Data Versioning

Demonstrates querying historical table versions and restoring prior states. Time travel supports auditing and error recovery in production data pipelines.

Lesson 4 • Creating and Managing Delta Tables

Covers DDL operations, table properties, and managed vs. external tables. These skills are required for building structured data layers in later pipeline chapters.

Lesson 1 • Auto Loader for Incremental Ingestion

Introduces Auto Loader's file notification and directory listing modes for scalable ingestion. Auto Loader is the recommended Databricks pattern for continuous file-based ingestion.

Lesson 2 • Batch Ingestion Patterns

Covers full and incremental batch loading strategies using Spark and Delta. Establishes the ingestion patterns that underpin all pipeline architectures in this chapter.

Lesson 3 • Building Transformation Logic

Covers multi-step transformation patterns including cleansing, enrichment, and deduplication. Transformation logic is the core value-add layer in any ETL pipeline.

Lesson 4 • Delta Live Tables for Pipeline Orchestration

Introduces Delta Live Tables (DLT) for declarative pipeline definition and quality enforcement. DLT simplifies pipeline management and integrates with the medallion architecture.

Lesson 1 • Medallion Architecture Principles

Defines Bronze, Silver, and Gold layer responsibilities and data contracts. Provides the structural framework applied in every subsequent modeling and pipeline section.

Lesson 2 • Designing the Bronze Layer

Covers raw data landing, schema-on-read, and preserving source fidelity. A well-designed Bronze layer ensures full auditability and reprocessing capability.

Lesson 3 • Constructing Gold Layer Aggregates

Covers building pre-aggregated, denormalized tables optimized for BI and reporting. Gold tables reduce query complexity and improve dashboard performance.

Lesson 4 • Building the Silver Layer

Teaches cleansing, type casting, deduplication, and conforming data to a unified schema. The Silver layer is the primary source for analytical and ML workloads.

Lesson 1 • Query Optimization for Analytics

Teaches query plan analysis, caching strategies, and result caching for SQL workloads. Optimized queries reduce warehouse costs and improve user experience.

Lesson 2 • Building Dashboards and Visualizations

Covers creating charts, dashboards, and alerts within Databricks SQL. Dashboards enable business stakeholders to consume data without writing code.

Lesson 3 • Unity Catalog for Data Governance

Introduces Unity Catalog's three-level namespace, access control, and data lineage. Governance is foundational for secure, compliant analytics across teams.

Lesson 4 • Advanced SQL Techniques in Databricks

Teaches window functions, CTEs, and higher-order functions for complex analytical queries. These techniques enable sophisticated analysis directly within Databricks SQL.

Lesson 1 • MLflow Model Registry

Introduces model registration, stage transitions, and version management in the Model Registry. The registry provides governance and traceability for production models.

Lesson 2 • Model Serving and Batch Inference

Covers real-time model serving endpoints and batch inference with Spark. Deploying models in both modes ensures flexibility for diverse business requirements.

Lesson 3 • Training Models at Scale with Spark

Covers distributed training using pandas UDFs, Spark ML, and Hyperopt for tuning. Distributed training reduces time-to-insight on large datasets.

Lesson 4 • Experiment Tracking with MLflow

Teaches logging parameters, metrics, and artifacts with MLflow Tracking. Systematic experiment tracking enables reproducibility and informed model selection.

Lesson 1 • Monitoring, Logging, and Cost Management

Covers cluster event logs, job metrics, audit logs, and DBU cost tracking. Observability and cost control are essential for sustainable production operations.

Lesson 2 • Scheduling and Triggering Workflows

Teaches cron-based scheduling, file-arrival triggers, and external API triggers. Flexible triggering ensures pipelines run at the right time with minimal manual intervention.

Lesson 3 • Infrastructure as Code with Terraform

Introduces provisioning Databricks resources using the Terraform provider. Infrastructure as code ensures reproducible, auditable environment setup across workspaces.

Lesson 4 • Databricks Jobs and Task Orchestration

Covers creating multi-task jobs, setting dependencies, and configuring compute for each task. Jobs are the primary mechanism for running production workloads on a schedule.