Apache Hive: Design, Query, and Optimize Big Data Course

Lesson 1 • Hive Architecture and Components

Explains Hive's core components: metastore, driver, compiler, and execution engine. Connects architecture to query lifecycle understanding.

Lesson 2 • Big Data Fundamentals and Challenges

Covers volume, velocity, and variety challenges driving big data adoption. Establishes context for why distributed processing tools like Hive exist.

Lesson 3 • Setting Up a Hive Environment

Guides installation and configuration of a local or sandbox Hive environment. Provides hands-on readiness for all subsequent chapters.

Lesson 1 • Primitive and Complex Data Types

Covers numeric, string, boolean, and date primitives alongside ARRAY, MAP, and STRUCT complex types. Correct type selection affects storage and query accuracy.

Lesson 2 • HiveQL Syntax and Query Structure

Introduces HiveQL as a SQL dialect with Hive specific extensions. Covers SELECT, FROM, WHERE, and ORDER BY clauses as building blocks.

Lesson 3 • Aggregation and Grouping

Teaches GROUP BY, HAVING, and aggregate functions for summarising datasets. Aggregation is foundational to analytical query patterns used throughout the course.

Lesson 4 • Subqueries and Common Table Expressions

Introduces inline views, subqueries, and WITH clause CTEs for modular query design. These patterns appear in complex analytical queries in later chapters.

Lesson 1 • Partitioning Tables

Teaches static and dynamic partitioning to reduce full-table scans. Partitioning is the primary technique for improving query performance on large tables.

Lesson 2 • Creating and Managing Databases

Covers CREATE DATABASE, DROP DATABASE, and database properties. Proper database organisation supports multi-team data governance.

Lesson 3 • Table Storage Formats

Compares TextFile, SequenceFile, ORC, and Parquet storage formats. Format choice directly impacts query speed and storage efficiency.

Lesson 4 • Managed vs. External Tables

Distinguishes managed tables (Hive-owned data) from external tables (user-owned data). Choosing correctly prevents accidental data loss.

Lesson 1 • Loading Data from Files

Covers LOAD DATA LOCAL and LOAD DATA from HDFS paths. File-based loading is the most common initial ingestion pattern in Hive workflows.

Lesson 2 • ACID Transactions and CRUD Operations

Explains Hive ACID support enabling UPDATE and DELETE on ORC tables. ACID operations require specific configuration and table properties.

Lesson 3 • INSERT and INSERT OVERWRITE Patterns

Teaches INSERT INTO and INSERT OVERWRITE for query-driven data population. These patterns support ETL pipelines built entirely within HiveQL.

Lesson 1 • Skew Joins and Data Skew Handling

Addresses join skew caused by high-frequency key values that overload reducers. Skew join optimisation distributes hot keys across multiple tasks.

Lesson 2 • Join Types and Syntax

Covers INNER, LEFT, RIGHT, FULL OUTER, and CROSS joins with HiveQL syntax. Understanding join semantics prevents incorrect result sets.

Lesson 3 • UNION, INTERSECT, and EXCEPT

Teaches set operations for combining and comparing result sets across queries. These operations support deduplication and differential analysis patterns.

Lesson 4 • Map-Side and Broadcast Joins

Explains map-side joins using the MAPJOIN hint for small-table optimisation. Broadcast joins eliminate shuffle overhead in large-to-small table joins.

Lesson 1 • Understanding the Query Execution Plan

Uses EXPLAIN and EXPLAIN EXTENDED to interpret MapReduce and Tez execution plans. Reading plans is the prerequisite skill for all optimisation decisions.

Lesson 2 • Tez Execution Engine Tuning

Configures Tez DAG execution for reduced latency compared to MapReduce. Tez is the recommended engine for interactive and iterative Hive workloads.

Lesson 3 • Partition Pruning and Predicate Pushdown

Ensures queries scan only relevant partitions and push filters as early as possible. These techniques reduce I/O, which is the dominant cost in Hive queries.

Lesson 4 • File Format and Compression Optimisation

Selects ORC and Parquet with appropriate compression codecs to minimise scan time. Compression reduces both storage cost and network I/O during query execution.

Lesson 1 • User-Defined Table Functions

Builds UDTFs that emit multiple rows from a single input row using the UDTF interface. UDTFs support exploding nested structures and custom row generation.

Lesson 2 • User-Defined Aggregate Functions

Implements UDAFs using the UDAF and GenericUDAF interfaces for custom aggregations. UDAFs operate across reducer groups and require buffer management.

Lesson 3 • UDF Development Fundamentals

Covers the UDF base class, evaluate method, and JAR packaging for simple scalar functions. UDFs fill gaps where built-in functions are insufficient.

Lesson 1 • Metastore Management and High Availability

Configures a remote metastore with a relational backend and high-availability setup. Metastore reliability is critical for uninterrupted Hive service in production.

Lesson 2 • Authorisation Models: SQL Standards and Ranger

Compares Hive SQL standard authorisation with Apache Ranger policy-based control. Fine-grained authorisation enforces least-privilege access at column level.

Lesson 3 • Authentication and Kerberos Integration

Configures Kerberos-based authentication for HiveServer2 and client connections. Authentication prevents unauthorised access to sensitive data assets.

Lesson 4 • Data Masking and Encryption

Applies dynamic data masking policies and HDFS encryption zones to protect sensitive columns. Masking enables analytics on sensitive data without exposing raw values.