Big Data Analysis Course

Lesson 1 • Data Types and Formats

Covers structured, semi-structured, and unstructured data with common file formats. Prepares students to select appropriate storage and processing strategies.

Lesson 2 • Defining Big Data Concepts

Introduces the 5 Vs framework and distinguishes big data from traditional data. Anchors all subsequent technical topics in shared terminology.

Lesson 3 • Business Drivers and Use Cases

Examines industry motivations for big data adoption and common analytical goals. Connects technical concepts to measurable business outcomes.

Lesson 1 • Distributed File System Principles

Explains replication, fault tolerance, and block storage in distributed systems. Establishes the storage foundation required for all processing frameworks ahead.

Lesson 2 • Hadoop Distributed File System

Covers HDFS architecture, configuration, and core operations for storing large datasets. Directly enables hands-on work with Hadoop-based processing in later chapters.

Lesson 3 • NoSQL Database Fundamentals

Surveys key-value, document, column-family, and graph NoSQL models. Equips students to match storage technology to data structure and query requirements.

Lesson 4 • Object Storage for Big Data

Introduces cloud-native object storage as a scalable alternative to HDFS. Prepares students to architect hybrid and cloud-first data lake solutions.

Lesson 1 • MapReduce Programming Model

Explains the map, shuffle, and reduce phases with data flow diagrams. Provides the conceptual core needed before writing any distributed processing code.

Lesson 2 • MapReduce Performance Tuning

Covers memory settings, speculative execution, and task parallelism for faster jobs. Connects theoretical knowledge to measurable performance improvements.

Lesson 3 • Writing MapReduce Jobs

Guides students through coding mapper and reducer functions with practical examples. Builds the programming foundation for all subsequent processing frameworks.

Lesson 1 • Distributed SQL Query Engines

Introduces MPP query engines that execute SQL directly on distributed storage. Expands students' toolkit beyond Hive for interactive and ad hoc analytics.

Lesson 2 • Schema Design for Analytics

Teaches star and snowflake schemas, denormalisation, and schema-on-read patterns. Prepares students to design data warehouse structures on big data platforms.

Lesson 3 • Hive Architecture and Data Model

Explains Hive metastore, tables, and HiveQL as a SQL abstraction over HDFS. Enables students to leverage existing SQL skills in a distributed context.

Lesson 4 • Query Optimisation in Hive

Covers execution plans, vectorisation, and cost-based optimisation for Hive queries. Directly reduces query latency in analytical pipelines built in this chapter.

Lesson 1 • Spark Architecture and Execution Model

Explains driver, executors, DAG scheduler, and task execution in Spark clusters. Provides the architectural understanding needed to write and tune efficient Spark jobs.

Lesson 2 • DataFrames and Spark SQL

Introduces the DataFrame API and Spark SQL for structured data processing at scale. Connects SQL knowledge from Chapter 4 to Spark's optimised execution engine.

Lesson 3 • Advanced Spark Operations

Covers joins, aggregations, window functions, and user-defined functions in Spark. Enables students to handle complex analytical transformations in production pipelines.

Lesson 4 • RDDs and Core Transformations

Covers Resilient Distributed Datasets, lazy evaluation, and key transformation operations. Builds the foundational Spark programming model before higher-level APIs.

Lesson 1 • Message Queuing with Kafka

Covers Kafka topics, partitions, producers, and consumers as a streaming backbone. Provides the ingestion layer required by all downstream stream processors.

Lesson 2 • Apache Spark Structured Streaming

Teaches micro-batch and continuous streaming with Spark's unified DataFrame API. Leverages Spark skills from batch chapters and extends them to streaming workloads.

Lesson 3 • Apache Flink Stream Processing

Introduces Flink's native streaming model, state backends, and event-time processing. Provides an alternative to Spark for low-latency, stateful streaming applications.

Lesson 4 • Real-Time Pipeline Design Patterns

Presents Lambda, Kappa, and CQRS architectures for combining batch and streaming. Equips students to architect end-to-end real-time data systems.

Lesson 1 • Data Quality and Validation

Introduces data profiling, schema validation, and automated quality checks in pipelines. Ensures analytical outputs are reliable and trustworthy for downstream consumers.

Lesson 2 • Workflow Orchestration with Airflow

Teaches DAG authoring, scheduling, and dependency management in Apache Airflow. Provides the orchestration layer that coordinates all pipeline components.

Lesson 3 • Data Ingestion Strategies

Covers batch ingestion, change data capture, and event-driven ingestion patterns. Establishes the entry point for all data entering the pipelines built in this chapter.

Lesson 4 • Pipeline Reliability and Monitoring

Covers idempotency, retry logic, dead-letter queues, and observability for pipelines. Prepares students to maintain production-grade data infrastructure.

Lesson 1 • MLOps for Big Data Pipelines

Presents continuous training, feature stores, and experiment tracking for scalable ML operations. Enables students to maintain and improve models in production environments.

Lesson 2 • Model Deployment and Serving

Covers batch scoring, real-time inference, and model registry patterns for production ML. Bridges the gap between model training and operational business value.

Lesson 3 • Distributed Machine Learning Concepts

Explains data parallelism, model parallelism, and distributed gradient descent. Establishes the theoretical foundation for all ML frameworks covered in this chapter.

Lesson 4 • Graph Analytics at Scale

Introduces graph data models and distributed graph processing for network analysis. Extends analytical capabilities to relationship-centric big data problems.