Cloud Data Engineer Course

Lesson 1 • Cloud Service Models and Deployment Types

Covers IaaS, PaaS, and SaaS distinctions alongside public, private, and hybrid deployments. Grounds all subsequent cloud data service decisions in architectural context.

Lesson 2 • Core Cloud Infrastructure Components

Introduces compute, storage, networking, and identity primitives used across major cloud platforms. Provides the vocabulary needed to design and discuss data infrastructure.

Lesson 3 • Cloud Security and Compliance Fundamentals

Covers encryption at rest and in transit, key management, and regulatory compliance frameworks for data. Establishes security habits applied throughout the course.

Lesson 1 • Choosing the Right Storage for Each Use Case

Applies a decision framework to match storage systems to latency, throughput, and cost requirements. Synthesises all storage options into practical selection guidance.

Lesson 2 • NoSQL and Key-Value Store Options

Covers document, wide-column, key-value, and graph database models available as cloud services. Guides selection based on access patterns and consistency requirements.

Lesson 3 • Cloud Relational Database Services

Examines managed relational database offerings, high-availability configurations, and read replicas. Connects SQL fundamentals to cloud-managed deployment patterns.

Lesson 4 • Data Warehousing as a Cloud Service

Introduces columnar storage, MPP architecture, and cloud data warehouse provisioning and scaling. Prepares students for analytical query workloads covered in later chapters.

Lesson 1 • Batch Ingestion Patterns and Tools

Covers scheduled file transfers, bulk API ingestion, and incremental load strategies for batch pipelines. Establishes the most common ingestion pattern before introducing streaming.

Lesson 2 • Change Data Capture Techniques

Explains log-based, trigger-based, and timestamp-based CDC methods for capturing database changes. Enables near-real-time replication without full table scans.

Lesson 3 • Streaming Ingestion Fundamentals

Introduces event streaming concepts, producers, consumers, topics, and partitions for real-time data flow. Bridges batch ingestion knowledge to continuous data processing.

Lesson 1 • Transformation Workflow Orchestration

Applies orchestration tools to schedule, monitor, and version transformation jobs at scale. Extends pipeline orchestration fundamentals to complex multi-step transformation workflows.

Lesson 2 • Data Modelling for Analytics

Introduces star schema, snowflake schema, and data vault modelling for analytical workloads. Ensures transformation outputs are structured for efficient downstream querying.

Lesson 3 • SQL-Based Transformation Patterns

Covers window functions, CTEs, incremental models, and query optimisation for large-scale SQL transformations. Connects SQL skills to cloud data warehouse and lake query engines.

Lesson 4 • Distributed Batch Processing with Spark

Teaches RDD and DataFrame APIs, partitioning, shuffling, and job optimisation in Apache Spark. Establishes the primary distributed processing engine used throughout the course.

Lesson 1 • Windowing and Time Semantics

Covers tumbling, sliding, and session windows alongside event time, processing time, and watermarks. Enables accurate aggregations over out-of-order and late-arriving event streams.

Lesson 2 • Streaming Pipeline Design Patterns

Applies lambda, kappa, and streaming-first architectures to real-world data engineering scenarios. Connects framework knowledge to end-to-end pipeline design decisions.

Lesson 3 • Stateful Stream Processing

Explains keyed state, operator state, checkpointing, and state backends for fault-tolerant streaming jobs. Enables complex event detection and running aggregations across unbounded streams.

Lesson 1 • Data Contracts and Schema Management

Defines data contracts between producers and consumers and covers schema registry usage and evolution. Prevents breaking changes from propagating across dependent data systems.

Lesson 2 • Data Quality Validation Frameworks

Covers schema validation, null checks, referential integrity, and statistical profiling for data quality. Integrates quality gates directly into ingestion and transformation pipelines.

Lesson 3 • Data Lineage and Cataloguing

Explains column-level lineage, metadata cataloguing, and data discovery tools for governance compliance. Enables teams to trace data origins and understand downstream impact of changes.

Lesson 4 • Pipeline Observability and Monitoring

Introduces metrics, logs, traces, and SLA tracking for end-to-end pipeline health monitoring. Provides the operational visibility needed to maintain production data systems reliably.

Lesson 1 • Designing for Multi-Cloud and Hybrid Environments

Covers data portability, cross-cloud networking, and vendor-neutral tooling for multi-cloud data platforms. Prepares students to design platforms that avoid single-vendor lock-in.

Lesson 2 • Scalability and Reliability Patterns

Addresses horizontal scaling, fault tolerance, idempotency, and disaster recovery for data platform components. Ensures platform designs meet production reliability and availability requirements.

Lesson 3 • Platform Cost Optimisation Strategies

Applies rightsizing, spot usage, storage tiering, and query cost controls to reduce platform spend. Connects architectural decisions directly to measurable cost outcomes.

Lesson 4 • Modern Data Platform Reference Architectures

Examines medallion, data mesh, and lakehouse architectures as blueprints for cloud data platforms. Provides architectural vocabulary and patterns for the design exercises that follow.

Lesson 1 • Capacity Planning and Performance Management

Applies load testing, resource forecasting, and performance benchmarking to data platform capacity decisions. Ensures platforms scale proactively rather than reactively under growing data volumes.

Lesson 2 • DataOps Culture and Continuous Improvement

Introduces DataOps principles, feedback loops, pipeline versioning, and team collaboration practices. Embeds a continuous improvement mindset into day-to-day data engineering operations.

Lesson 3 • Environment Management and Configuration

Covers dev, staging, and production environment parity, secrets management, and environment-specific configs. Prevents configuration drift and credential exposure across deployment environments.

Lesson 4 • CI/CD Pipelines for Data Engineering

Builds automated testing, linting, deployment, and rollback workflows for data pipeline code. Applies software engineering delivery practices to the data engineering lifecycle.