Azure Data Lake Course

Lesson 1 • Setting Up Your First Data Lake

Guides creation of an ADL Storage Gen2 account with correct configuration. Reinforces portal and CLI skills while introducing storage account settings.

Lesson 2 • Azure Portal and CLI Basics

Teaches navigation of the Azure Portal and Azure CLI for resource management. Provides hands-on access skills required throughout the course.

Lesson 3 • Cloud Data Storage Fundamentals

Covers distributed storage concepts and cloud data management principles. Establishes the baseline needed to understand Azure Data Lake's design rationale.

Lesson 1 • Lifecycle Management Policies

Teaches automated tiering and deletion rules to control storage costs. Links lifecycle policies to data retention and compliance requirements.

Lesson 2 • Metadata and Tagging

Introduces blob metadata, index tags, and custom properties for discoverability. Supports data catalog integration covered in later chapters.

Lesson 3 • Containers, Directories, and Files

Explains the container-directory-file hierarchy in ADL Storage Gen2. Connects namespace design to downstream query performance and governance.

Lesson 4 • File Formats for Big Data

Compares CSV, JSON, Parquet, ORC, and Avro for ADL workloads. Guides format selection based on compression, schema evolution, and query speed.

Lesson 1 • Azure Identity and Authentication

Covers Azure Active Directory identities, service principals, and managed identities. Establishes authentication foundations required for all access control configurations.

Lesson 2 • Network Security and Firewalls

Configures virtual network service endpoints, private endpoints, and firewall rules. Restricts ADL access to trusted networks and Azure services.

Lesson 3 • Role-Based Access Control for ADL

Explains built-in and custom RBAC roles scoped to storage accounts and containers. Demonstrates least-privilege assignment for analytics teams.

Lesson 4 • POSIX ACLs on Hierarchical Namespace

Teaches fine-grained file and directory permissions using POSIX-style ACLs. Connects ACL inheritance to secure multi-team data lake designs.

Lesson 1 • Ingestion with Azure IoT Hub

Routes IoT device telemetry to ADL via IoT Hub message routing rules. Demonstrates time-series data landing patterns for sensor workloads.

Lesson 2 • Third-Party and On-Premises Ingestion

Covers ADF self-hosted integration runtime and partner connectors for hybrid ingestion. Addresses common connectivity challenges for on-premises source systems.

Lesson 3 • Batch Ingestion with ADF

Uses Azure Data Factory pipelines and copy activities to load data into ADL. Builds on security knowledge to configure linked services with managed identities.

Lesson 4 • Ingestion Monitoring and Error Handling

Implements pipeline monitoring, alerting, and retry logic for reliable data ingestion. Connects to Azure Monitor and Log Analytics for operational visibility.

Lesson 1 • SQL-Based Transformation with Synapse SQL

Applies T-SQL CETAS and external tables to transform ADL files without data movement. Connects SQL skills to serverless query patterns introduced later.

Lesson 2 • Medallion Architecture Design

Introduces bronze, silver, and gold layer patterns for data lake organization. Provides the structural framework applied in all transformation sections.

Lesson 3 • Orchestrating Multi-Step Pipelines

Chains ingestion and transformation steps into end-to-end ADF or Synapse pipelines. Implements dependency management and conditional execution logic.

Lesson 4 • Transformation with Azure Synapse Pipelines

Leverages Synapse Analytics mapping data flows for code-free transformations. Demonstrates integration with ADL Gen2 as both source and sink.

Lesson 1 • Serverless SQL Queries on ADL

Uses Synapse serverless SQL pool to query Parquet, CSV, and JSON files in ADL. Demonstrates OPENROWSET syntax and external table creation for ad hoc analysis.

Lesson 2 • Querying with Azure Databricks SQL

Runs SQL analytics on ADL-backed Delta tables using Databricks SQL warehouses. Connects Delta Lake concepts introduced in transformation to query optimization.

Lesson 3 • Connecting BI Tools to ADL

Integrates Power BI and other BI tools with Synapse and Databricks endpoints. Enables self-service analytics on curated ADL datasets for business users.

Lesson 1 • Compliance and Retention Policies

Configures immutability policies, legal holds, and retention locks on ADL containers. Addresses regulatory data retention and deletion requirements functionally.

Lesson 2 • Microsoft Purview for ADL

Registers ADL Storage Gen2 as a Purview data source and runs automated scans. Establishes the catalog foundation for all governance activities in this chapter.

Lesson 3 • Data Classification and Sensitivity Labels

Applies built-in and custom classification rules to tag sensitive ADL data. Links classification results to access control and compliance reporting.

Lesson 4 • Data Lineage Tracking

Captures end-to-end lineage from ingestion through transformation to consumption. Uses Purview lineage graphs to support impact analysis and auditing.

Lesson 1 • Cost Analysis and Budgeting

Uses Azure Cost Management to analyze ADL storage and transaction costs. Sets budgets and alerts to prevent unexpected spending in production environments.

Lesson 2 • Disaster Recovery and High Availability

Configures geo-redundant replication and failover for ADL storage accounts. Designs recovery point and recovery time objectives for critical data lake workloads.

Lesson 3 • Azure Monitor and Diagnostics for ADL

Enables diagnostic settings to stream ADL metrics and logs to Log Analytics. Builds the observability foundation for all monitoring tasks in this chapter.

Lesson 4 • Storage Optimization Techniques

Reduces costs through compaction, tiering automation, and redundant data removal. Connects lifecycle policies from Chapter 2 to cost-driven optimization decisions.