PySpark Course

Lesson 1 • Apache Spark Architecture Overview

Explains Spark's driver-executor model, cluster managers, and DAG execution engine. Provides the mental model students need to reason about PySpark job behaviour.

Lesson 2 • Big Data Concepts and Challenges

Covers volume, velocity, and variety as drivers of distributed processing needs. Establishes why traditional tools fail at scale, motivating the PySpark approach.

Lesson 3 • SparkSession and SparkContext Basics

Introduces SparkSession as the unified entry point for all Spark functionality. Students create sessions, set configurations, and understand context hierarchy.

Lesson 4 • Setting Up the PySpark Environment

Guides installation of Java, Python, and PySpark locally and via cloud notebooks. Students verify their setup by running a minimal Spark context.

Lesson 1 • Creating RDDs from Various Sources

Demonstrates parallelising Python collections and reading from files and external systems. Students practice creating RDDs with controlled partition counts.

Lesson 2 • Actions and RDD Persistence

Explains how actions trigger execution and how caching avoids redundant recomputation. Students choose appropriate storage levels for iterative workloads.

Lesson 3 • Transformations: map, filter, and flatMap

Covers the core lazy transformations that reshape RDD contents element by element. Students build transformation chains and inspect the resulting DAG.

Lesson 4 • Key-Value RDDs and Pair Operations

Introduces pair RDDs and shuffle-based operations like groupByKey and reduceByKey. Students learn to minimise shuffles for better performance.

Lesson 1 • Spark SQL: Querying with SQL Syntax

Registers DataFrames as temporary views and queries them with standard SQL. Students combine SQL and DataFrame API calls in the same pipeline.

Lesson 2 • Aggregations and Grouping

Applies groupBy, agg, and window functions to summarise data at various granularities. Students produce business-level metrics from raw event data.

Lesson 3 • Creating DataFrames from Multiple Sources

Covers reading CSV, JSON, Parquet, and JDBC sources into DataFrames. Students configure reader options and handle malformed records gracefully.

Lesson 4 • DataFrame Transformations and Column Operations

Teaches selecting, filtering, renaming, casting, and deriving columns using the Column API. Students build reusable transformation pipelines.

Lesson 1 • Complex Types: Arrays, Maps, and Structs

Manipulates nested and semi-structured data using explode, array functions, and struct access. Students flatten and reshape JSON-like columns for tabular analysis.

Lesson 2 • Deduplication and Data Validation

Removes duplicate records and enforces business rules through assertion-style checks. Students build validation layers that surface data quality issues early.

Lesson 3 • String Manipulation and Regex

Uses built-in string functions and regex patterns to clean and standardise text fields. Students normalise inconsistent formats such as phone numbers and categories.

Lesson 4 • Date and Timestamp Processing

Parses, formats, and computes date arithmetic using Spark's date functions. Students extract time-based features for downstream analytics.

Lesson 1 • Advanced Aggregation Patterns

Applies multi-level grouping, conditional aggregation, and approximate functions at scale. Students balance accuracy and performance for large aggregation workloads.

Lesson 2 • Window Functions: Ranking and Ordering

Introduces window specifications and ranking functions like row_number, rank, and dense_rank. Students assign ordered ranks within partitions for leaderboard-style analytics.

Lesson 3 • Window Functions: Running Totals and Lag/Lead

Computes cumulative sums, moving averages, and period-over-period comparisons using frame boundaries. Students build time-series metrics without self-joins.

Lesson 4 • Join Types and Strategies

Covers inner, left, right, full outer, semi, and anti joins with practical examples. Students select join types based on business logic and data cardinality.

Lesson 1 • Writing DataFrames with Save Modes

Explains overwrite, append, ignore, and error save modes and their production implications. Students implement idempotent write patterns for reliable pipelines.

Lesson 2 • Connecting to Cloud and Distributed Storage

Configures PySpark to read from and write to object storage and distributed file systems. Students authenticate securely and tune I/O settings for cloud environments.

Lesson 3 • File Formats: Parquet, ORC, Avro, and Delta

Compares columnar and row-based formats on read performance, compression, and schema evolution. Students choose formats based on workload access patterns.

Lesson 4 • Partitioning and Bucketing Strategies

Applies write-time partitioning and bucketing to reduce scan costs on large datasets. Students design partition schemes aligned with common query predicates.

Lesson 1 • Caching and Broadcast Variables

Applies DataFrame caching and broadcast variables to eliminate redundant computation and network transfer. Students measure cache hit rates and eviction behaviour.

Lesson 2 • Understanding the Catalyst Optimiser

Traces how Catalyst transforms logical plans into optimised physical plans through rule-based and cost-based passes. Students read explain output to verify optimisations.

Lesson 3 • Memory Management and Garbage Collection

Explains executor memory regions, off-heap storage, and JVM GC impact on Spark jobs. Students tune memory fractions to reduce spill and GC pauses.

Lesson 4 • Partitioning for Parallelism

Aligns partition count with cluster cores to maximise parallelism without excessive overhead. Students apply adaptive query execution to automate partition sizing.

Lesson 1 • Logging, Monitoring, and Error Handling

Implements structured logging, exception handling, and metric emission within PySpark jobs. Students build observable pipelines that surface failures and performance regressions.

Lesson 2 • Testing PySpark Code

Applies unit and integration testing patterns using pytest and small in-memory DataFrames. Students build test suites that catch regressions before production deployment.

Lesson 3 • Workflow Orchestration with Airflow

Schedules and monitors PySpark jobs using DAG-based orchestration with dependency management. Students build a multi-step pipeline DAG with retry and alerting logic.

Lesson 4 • Pipeline Architecture and Design Patterns

Introduces medallion architecture, modular pipeline design, and separation of ingestion, transformation, and serving layers. Students map business requirements to pipeline stages.