Data Engineering with Big Data and AWS

Master Distributed Data Processing with Real-World Tools and Projects

Course Overview

This advanced training program is designed for learners who want to delve deeply into Big Data engineering using technologies such as Hadoop, Spark, Kafka, and AWS. You’ll build the skills required to process, manage, and analyze large-scale data systems, while working on mini projects after each topic and a capstone project at the end of every module.

Module 1: Advanced Hadoop & Spark Programming with PySpark + Introduction to Kafka

Duration: 40 Hours

Core Concepts Covered

HDFS (Hadoop Distributed File System)
- Understand why HDFS is critical in Big Data ecosystems
- Compare HDFS vs traditional file systems
- Explore HDFS architecture, components, and high availability setups
- Learn the roles of NameNode, DataNode, Secondary NameNode, Checkpoint Node & Backup Node
- Understand Data Replication and Rack Awareness
- Get hands-on with HDFS commands & file operations

Distributed Processing with MapReduce
- Dive into the MapReduce architecture and workflow
- Understand key phases: Mapping, Shuffling, Reducing
- Set up your local development environment
- Learn how distributed processing works behind the scenes