Python for Data Engineers Course

Lesson 1 • Control Flow and Iteration

Apply conditionals, loops, and comprehensions to process sequences of data records. Directly maps to row-level transformation logic used in ETL pipelines.

Lesson 2 • Functions and Error Handling

Define reusable functions with arguments, defaults, and return values, then handle exceptions gracefully. Enables modular, fault-tolerant pipeline code.

Lesson 3 • Core Python Syntax and Data Types

Cover variables, operators, and built-in types including strings, integers, floats, and booleans. Provides the syntactic baseline for writing any data pipeline logic.

Lesson 4 • Collections: Lists, Tuples, Dicts, Sets

Explore mutable and immutable collection types and their performance trade-offs. These structures underpin in-memory data manipulation throughout the course.

Lesson 1 • Binary Formats: Parquet and Avro

Read and write columnar Parquet and row-based Avro files using Python libraries. These formats are standard in modern data lake architectures.

Lesson 2 • Working with Text and CSV Files

Open, read, and write text files using context managers, then parse CSV with the standard library. Covers the most common raw data format in batch pipelines.

Lesson 3 • File System and Path Operations

Navigate directories, glob files, and manage paths using pathlib and os modules. Automates file discovery steps common in batch ingestion workflows.

Lesson 1 • DataFrames and Series Fundamentals

Create DataFrames from dicts, CSV, and JSON, then inspect structure and data types. Forms the foundation for all Pandas-based transformation work.

Lesson 2 • Filtering, Sorting, and Selecting

Apply boolean masks, loc/iloc selectors, and sort operations to isolate relevant data subsets. Directly supports data quality filtering in pipeline stages.

Lesson 3 • Aggregation, Grouping, and Pivoting

Summarise data with groupby, agg, and pivot operations to produce analytical outputs. Enables metric computation and reporting within pipeline workflows.

Lesson 4 • Data Cleaning and Type Coercion

Handle missing values, duplicates, and incorrect types to produce clean, consistent datasets. Cleaning is the most time-intensive step in real-world data pipelines.

Lesson 1 • Working with NoSQL Databases

Connect to document and key-value stores, perform CRUD operations, and map results to Python objects. Extends pipeline connectivity beyond relational systems.

Lesson 2 • Relational Databases with SQLAlchemy

Establish connections, reflect schemas, and execute raw SQL using SQLAlchemy's core API. Provides a database-agnostic access layer for pipeline code.

Lesson 3 • Connection Pooling and Best Practices

Configure connection pools, manage credentials securely, and avoid common anti-patterns. Ensures production-grade reliability and security in database access code.

Lesson 1 • Pipeline Logging and Observability

Instrument pipelines with structured logging, metrics, and alerting to enable operational monitoring. Transforms scripts into observable, production-ready processes.

Lesson 2 • Transformation Logic and Business Rules

Implement field mappings, derived columns, and validation rules as pure Python functions. Encapsulates business logic in testable, reusable transformation units.

Lesson 3 • Loading Data to Target Systems

Write transformed data to databases, data warehouses, and object storage using Python connectors. Completes the pipeline by delivering clean data to its destination.

Lesson 4 • ETL Architecture and Design Patterns

Understand pipeline stages, data flow, and common design patterns like ELT and medallion architecture. Sets the conceptual framework for all pipeline implementation work.

Lesson 1 • Infrastructure as Code for Data Pipelines

Define cloud resources programmatically using Python-based IaC tools to automate environment setup. Reduces manual provisioning and ensures reproducible pipeline infrastructure.

Lesson 2 • Event-Driven Ingestion with Message Queues

Produce and consume messages from queues and streaming topics to trigger pipeline execution. Connects Python pipelines to event-driven and real-time data architectures.

Lesson 3 • Interacting with Managed Data Warehouses

Connect to cloud data warehouses via Python connectors, execute queries, and load data efficiently. Enables Python pipelines to serve analytical workloads at scale.

Lesson 1 • Error Handling and Retry Strategies

Configure retries, timeouts, and SLA alerts to make orchestrated pipelines resilient to transient failures. Directly reduces manual intervention in production environments.

Lesson 2 • Dynamic DAGs and Parameterisation

Generate DAGs programmatically and pass runtime parameters to tasks for flexible pipeline execution. Enables scalable orchestration of many similar pipeline variants.

Lesson 3 • Orchestration Concepts and Tool Landscape

Understand DAGs, task dependencies, and the role of orchestrators in data engineering. Provides the conceptual foundation before implementing orchestration code.

Lesson 4 • Building DAGs with Apache Airflow

Define DAGs, operators, and task dependencies in Airflow using Python to schedule pipeline runs. Airflow is the most widely deployed orchestrator in data engineering.

Lesson 1 • Code Quality and Packaging

Enforce style with linters, type-check with mypy, and package pipeline code as installable Python modules. Raises code maintainability to professional software engineering standards.

Lesson 2 • Unit and Integration Testing for Pipelines

Write pytest-based unit tests for transformation functions and integration tests for database interactions. Testing prevents regressions and validates pipeline correctness.

Lesson 3 • Parallel and Concurrent Processing

Apply multiprocessing, threading, and async patterns to parallelise I/O-bound and CPU-bound pipeline tasks. Reduces end-to-end pipeline runtime significantly.

Lesson 4 • Profiling and Optimising Python Code

Identify bottlenecks using profiling tools and apply vectorisation, chunking, and caching to improve throughput. Performance optimisation is critical for large-scale data pipelines.