Master the core concepts and primitives used in data engineering around ETL. Abstractions and tools such as Airflow and Airbyte are built on top of the core concepts and primitives taught in this topic. 

• Python virtual environments 
• ETL with interactive Jupyter Notebooks
• Data extraction patterns from APIs (full vs incremental) 
• Data transformation using dataframes (Pandas) 
• Data loading patterns to files (CSV, Parquet) and databases (SQLAlchemy, PostgreSQL) 
• Functional Programming with Python 
• Modular programming using Python modules
• Object Oriented Programming with Python
• Logging with Python (PostgreSQL) 
• Unit and integration testing with Python 
• Code linting with Python
• Metadata config pipeline (YAML) 
• Metadata logging to database 
• Cron scheduling 

Continue down the path of mastering core concepts and primitives used in data engineering. In this topic, we learn the ELT pattern, a fairly recent addition to data engineering that was born from the recent explosion of cloud adoption. 

• Data extraction patterns from databases (full vs incremental vs CDC) 
• Data loading patterns to databases (overwrite vs insert vs upsert vs merge) 
• SQL  Transformations in databases (PostgreSQL)
• Jinja for SQL templating (Jinja, Python, SQLAlchemy)  
• Directed acyclic graphs (DAGs) with Python 
• SQL Common Table Expressions (CTEs) 
• SQL Window Functions 
• Modularising ELT pipelines 
• Unit testing ELT pipelines 
• Logging for ELT pipelines (PostgreSQL) 
• Metadata config for ELT pipelines (YAML) 

Master the concepts to containerize, build, and deploy ETL pipelines into a production environment hosted on the cloud. Enable code versioning and team collaboration best practices through Git. 

• Git:
  • Git version control system 
  • Git workflow (add, commit, push, merge, pull) 
  • Git branching 
  • Github pull requests 
• Docker:
  • Computer vs Virtual Machine vs Docker 
  • Docker image and container 
  • Docker commands 
  • Dockerfile 
  • Docker volumes 
  • Docker compose
  • Docker repository 
  • Containerize an ETL pipeline 
• Cloud (AWS): 
  • Identity and Access Management (IAM) – Users, Policies, Groups, and Roles 
  • Relational Database Service (RDS)
  • Simple Storage Service (S3) 
  • AWS CLI and Boto3 
  • Elastic Container Registry (ECR) 
  • Elastic Container Service (ECS) 
  • Deploy and schedule an ETL pipeline on ECS

Create an end-to-end ETL pipeline using extract, load, and transform patterns covered in earlier topics. 

• Extract data from a data source of your choosing, load data into a data store of your choosing, and transform data to support a use-case of your choosing. 
• Apply metadata logging, metadata configuration, unit and integration testing, and cron scheduling for your pipeline. 
• Use git and github to apply code versioning, git branching, and pull requests. 
• Containerize your ETL pipeline using docker, and deploy the pipeline on the cloud.

Most businesses generate and store data in multiple systems such as Customer Relationship Management (CRM) systems, Order Management Systems (OMS), Accounting systems, Marketing platforms, and many more. Handcrafting the Extract and Load logic for each system is a tedious process that can be automated using data integration tools such as Airbyte. 

• Airbyte sources, destinations, and connections 
• Airbyte extract patterns (full, incremental, CDC) 
• Airbyte load patterns (overwrite, insert, upsert, merge) 
• Octavia CLI 
• Airbyte API 
• Airbyte Custom Connectors 
• Deploying airbyte on AWS EC2
• End-to-end ELT pipeline with Airbyte on AWS

As businesses scale, their data volumes increase until it is no longer viable to process data on a single compute instance. To solve scale issues, we look at technologies like Snowflake which are designed to perform analytical processing on large volumes of data. 

We streamline our Transform pipeline development process using dbt, which popularized a subfield of data engineering known as Analytics Engineering. 

• OLAP vs OLTP 
• Snowflake architecture 
• Snowflake RBAC 
• Loading data into Snowflake 
• Parsing JSON with Snowflake 
• Snowflake micro-partitions and clustering
• dbt project 
• dbt commands (run, test, build, list)
• Writing and running a dbt model
• dbt seeds, tests, and macros
• dbt docs
• dbt in production (profiles, targets, and deploy on AWS)

Data engineering does not exist in a vacuum. As data engineers, we transform and model data for our end user consumption to power use-cases such as machine learning, business intelligence, and analytics. To model the data with software engineering principles of modularity and reusability, we apply data modelling techniques such as dimensional modelling. To enable end users to slice and dice models that we produce, we provide a layer on top of the data warehouse known as the semantic layer. 

• Normalization vs Denormalization
• Data modelling concepts: 
  • Dimensional modelling by Ralph Kimball (also known as Star Schema) 
  • Data warehouse modelling by Bill Inmon 
  • Data vault modelling by Dan Linstedt 
  • One Big Table (OBT) 
• Applied dimensional modelling using dbt: 
  • Fact and dimension tables
  • dbt snapshots and Slowly Changing Dimensions (SCD) 
  • Transactional fact table 
  • Snapshot fact table 
  • Accumulating snapshot fact table 
  • Factless fact table 
  • Incremental fact table load 
• Semantic modelling: 
  • Semantic modelling concepts and tools 
  • Semantic modelling and metrics using Preset 
  • Preset chart
  • Preset dashboard

Spark is a distributed data processing system capable of processing large volumes of data. Databricks provides an ecosystem of tooling to enable Spark to run for multiple use-cases such as data engineering, stream processing, and machine learning. 

We discover how Spark uses the separation of storage and compute to enable scale, learn about the delta file format, use Spark for data engineering, and apply data quality tests using Great Expectations. 

• Big data processing architectures
• Spark internals and core concepts 
• Spark reading and writing 
• Spark SQL 
• Spark DataFrame 
• Spark joins, group by, and aggregation 
• Spark UDF 
• Spark query plan and optimization 
• Spark partition keys  
• ACID file formats (delta file format) 
• Data orchestration using Databricks Workflows
• Manage the Databricks workspace using API and CLI 
• Data quality testing with Great Expectations

Create an end-to-end ETL pipeline capable of processing large volumes of data. 

• Extract data from a data source of your choosing, load data into a data store of your choosing, and transform data to support a use-case of your choosing. 
• Use Airbyte to perform data integration, an abstraction layer over low-level Extract and Load patterns. 
• Transform data using compute engines from the Data Warehousing paradigm (Snowflake), or the Data Lakehouse paradigm (Databricks). 
• Apply transformation logic using traditional ETL patterns, or Analytics Engineering patterns using dbt. 
• Apply metadata logging, metadata configuration, unit and integration testing, and cron scheduling for your pipeline. 
• Use git and github to apply code versioning, git branching, and pull requests. 
• Deploy your end-to-end ETL pipeline on the cloud.

Data orchestration enables data engineers to stitch together different parts of ETL into a single cohesive pipeline. Data orchestration makes it easy to trigger, schedule, monitor, and configure alerts for the pipelines. Data orchestrators like Airflow come with plugins or providers to connect existing tools in your data stack like Airbyte, dbt, Snowflake and Databricks, so that you can easily orchestrate steps between them. 

• Data orchestration architecture and patterns 
• Airflow DAGs, Operators and Tasks
• Airflow schedule 
• Airflow patterns for catchup, idempotence, backfill, and branching
• Airflow variables, connections, hooks, and providers 
• Airflow cross communication (XComs) 
• Airflow sensors
• Airflow Dynamic Task Mapping, and taskflow
• Trigger Rules
• Watcher Pattern
• Deploy airflow locally, and on AWS using EC2 or MWAA 
• Extending the Airflow Docker image
• Airflow providers for Airbyte, dbt, Databricks, Snowflake, and Slack alerts

Enable real-time insights from fast moving data. Learn the core concepts and primitives of stream processing using Kafka, and deploy kafka topics on Confluent Cloud. Integrate real-time events into Clickhouse, a real-time database, and perform data transformation in Clickhouse. Define and test Clickhouse materialized views using dbt. 

• Streaming concepts 
• Kafka key concepts
• Kafka broker and topics  
• Kafka CLI
• Creating a Python Kafka producer
• Creating a Python Kafka consumer
• Stream analytics using ksqlDB
• Deploy kafka on Confluent Cloud 
• Real-time databases with Clickhouse 
• Clickhouse architecture and internals 
• Use Kafka Connect to integrate data into Clickhouse
• Tables, views, and materialized views on Clickhouse
• dbt to test and deploy Clickhouse objects 
• Real-time dashboard with Preset, Clickhouse, and Kafka

As the data engineering team grows, so does the code complexity. To provide assurance that data engineers are doing the right things, automated code integration pipelines can be used to test and verify a data engineer’s code changes in a separate branch-based environment. After code has been validated, code can be automatically built and released into various deployment environments such as staging and production. 

• Principles of DataOps 
• Continuous integration pipelines: 
  • Unit testing 
  • Code linting tests 
  • Data quality testing 
  • Branch-based testing environments 
  • CI pipelines for dbt 
  • CI pipelines for Python ETL 
• Continuous deployment pipelines: 
  • Containerize and build 
  • Deployment environments 
  • Deploy using Infrastructure as Code (IaC)

Showcase all the skills and technologies you have learnt throughout the bootcamp to future employers. Implement either a lambda or kappa architecture with ETL pipelines capable of processing large volumes of data.

• Extract data from a data source of your choosing, load data into a data store of your choosing, and transform data to support a use-case of your choosing. 
• Apply unit testing, data quality testing, and monitoring over your ETL pipelines 
• Use git and github to apply code versioning, git branching, and pull requests 
• Implement CI pipelines to automatically test your code in a branch-based environment 
• Implement CD pipelines to automatically deploy your end-to-end ETL pipeline on the cloud 
• Present your capstone projects at the Demo Day to data engineering experts working in startups or large companies