課程介紹

Apache Airflow is an open-source  platform to programmatically author, schedule and monitor workflows. If you have many ETL(s) to manage, Airflow is a must-have.

In this course you are going to learn everything you need to start using Apache Airflow

…[更多細節]

哪些人適合這堂課？

• People being curious about data engineering.
• People who want to learn basic and advanced concepts about Apache Airflow.
• People who like hands-on approach.

學習目標

• Create plugins to add functionalities to Apache Airflow.
• Using Docker with Airflow and different executors
• Master core functionalities such as DAGs, Operators, Tasks, Workflows, etc
• Understand and apply advanced concepts of Apache Airflow such as XCOMs, Branching and SubDAGs.
• The difference between Sequential, Local and Celery Executors, how do they work and how can you use them.
• Use Apache Airflow in a Big Data ecosystem with Hive, PostgreSQL, Elasticsearch etc.
• Install and configure Apache Airflow
• Think, answer and implement solutions using Airflow to real data processing problems

課程介紹

Apache Airflow is a platform created by community to programmatically author, schedule and monitor workflows.

It is scalable, dynamic, extensible and modulable.

Without any doubts, mastering Airflow is becoming a must-have and an attractive skill f

…[更多細節]

哪些人適合這堂課？

• Data Engineers
• Inspiring Data Engineers
• DevOps
• Software Engineers
• Data Scientists

學習目標

• Coding Production Grade Data pipelines by Mastering Airflow through Hands-on Examples
• How to Follow Best Practices with Apache Airflow
• How to Scale Airflow with the Local, Celery and Kubernetes Wxecutors
• How to Set Up Monitoring with Elasticsearch and Grafana
• How to Secure Airflow with authentication, crypto and the RBAC UI
• Core and Advanced Concepts with Pros and Limitations
• Mastering DAGs with timezones, unit testing, backfill and catchup
• Organising the DAG folder and keep things clean

課程介紹

“Apache Airflow is a platform created by community to programmatically author, schedule and monitor workflows.”

Airflow is going to change the way of scheduling data pipelines and that is why it has become the Top-level project of Apache.

What’s in

…[更多細節]

哪些人適合這堂課？

• Data engineers who want to deploy their pipelines using Airflow in their company’s projects.
• Engineers who want to switch their career from conventional schedulers (Oozie, Luigi etc)
• Experienced techies who want to add a hot & demanding technology in their resume.

學習目標

• Learn Full In & out of Apache Airflow with proper HANDS-ON examples from scratch.
• Start with the implementation of Airflow core nomenclature – DAG, Operators, Tasks, Executors, Cfg file, UI views etc.
• ADVANCE Airflow concepts, the explanation to which is not very clear even in Airflow’s Official Documentation.
• XComs, Hooks, Pools, SubDAGs, Variables, Connections, Plugins, Adhoc queries, Branching, Sensors and many more….
• Exclusive features – Data Profiling, Charts, Trigger rules, airflowignore file, Zombies, Undeads, LatestOnly operator.
• Create data pipeline of Real-Time case studies using Apache Airflow.
• Codes and Data-sets are available in resources tab. This will save your typing efforts.
• Learn Best practices / Do’s & Don’ts to follow in Real-Time Airflow Projects.

課程介紹

Struggling to set up Airflow on AWS EKS?

You are at the right place!

With more than 15,000 students, I got many feedbacks about how difficult it is to configure Airflow on AWS with the official Helm chart.

Guess what? You are about to learn everyt

…[更多細節]

哪些人適合這堂課？

• Data engineers
• Software Engineers
• DevOps

學習目標

• How to Set Up a Production Ready Architecture for Airflow on AWS EKS From A-Z
• How to deploy DAGs from Git (public and private)
• How to Create CI/CD Pipelines with AWS CodePipeline Deploy DAGs
• How to Share DAGs and Store Logs with AWS EFS
• How to Enable Remote Logging with AWS S3 in EKS
• How to Test your DAGs in CI/CD pipelines
• How to Store Sensitive Data in AWS Secret Manager

課程介紹

Hi there, my name is Alexandra Abbas. I’m an Apache Airflow Contributor and a Google Cloud Certified Data Engineer & Architect with over 3 years experience as a Data Engineer.

Are you struggling to learn Apache Airflow on your own? In this course I

…[更多細節]

哪些人適合這堂課？

• Data Engineers
• Data Scientists
• Python Developers Interested in Data Engineering
• Data Analysts with Python Programming Knowledge

學習目標

• Core and Advanced Concepts in Airflow through Real-World Examples
• Architecture Components of Apache Airflow
• How to Set Up Connections to External Resources
• How to Load and Analyse Data in a Data Warehouse using Airflow
• How to Schedule PySpark jobs using Apache Airflow
• How to Extend Airflow with Custom Operators and Sensors
• How to Test Airflow DAGs and Operators
• How to Deploy Airflow Instances with Different Executors
• How to Set Up Error Tracking and Monitoring

課程介紹

Apache Airflow is an open-source  platform to programmatically author, schedule and monitor workflows.

Cloud Composer  is a fully managed workflow orchestration service that empowers you to author, schedule, and monitor pipelines that span across cl

…[更多細節]

哪些人適合這堂課？

• People interested in Data warehousing, Big data, Data engineering
• People interested in Automated tools for task workflow scheduling
• Student interested to know about Airflow
• Professional to wish to explore as how Apache Airflow can be used in Task scheduling and building Data pipelines

學習目標

• Understand automation of Task workflows through Airflow
• Airflow Architecture – On Premise (local install), Cloud, single node, multiple node
• How to use connection functionality to connect to different systems to automate data pipelines
• What is Google cloud Big query and briefly how it can be used in Dataware housing as well as in Airflow DAG
• Master core functionalities such as DAGs, Operators, Tasks through hands on demonstrations
• Understand advanced functionalities like XCOM, Branching, Subdags through hands on demonstrations
• Get an overview understanding on SLAs, Kubernetes executor functionality in Apache Airflow
• The source files of Python DAG programs (9 .py files) used in demonstration are available for download towards practice for students

課程介紹

Apache Airflow has more than 700 Operators and 70 tools it can interact with.

It’s huge! ?

Operators are tasks in your data pipeline. They are important as they correspond to the actions of your data pipeline, to the different steps to produce the

…[更多細節]

哪些人適合這堂課？

• Data engineers
• Data scientist
• Business Analysts
• Software Engineers

學習目標

• Master Apache Airflow Operators
• How to version your DAGs
• How to create DAG dependencies efficiently
• How to trigger your DAGs on specific days
• Create Incredible Data Pipelines by truly understanding Airflow operators

課程介紹

Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow is one of the best open source orchestrators and it is used widely because it is simplicity, scalability and extensibility.

Main goal

…[更多細節]

哪些人適合這堂課？

• Curious on Big Data Technologies

學習目標

• Gain complete understanding of the apache airflow
• Learn different types of Executors and it’s working principles
• Own a scalable distributed airflow setup which could be shared by multi teams in your organisation

課程介紹

Why Am I conducting this course ?
When I started configuring airflow in my organisation, I spent many weeks on writing Docker Compose files for each airflow component.

Airflow community provides a single docker compose file which installs all the co

…[更多細節]

哪些人適合這堂課？

• Devops

學習目標

• Apache Airflow 2.0 Architectures and Enhancements
• Docker and Docker Swarm commands
• Setting up airflow 2.0.1 using Docker Compose and Docker Swarm

課程介紹

Airflow is an Apache project, used to programmatically author, schedule and monitor workflows. It was originally developed by Airbnb and then later open-sourced. Today, many technology companies use Airflow to orchestrate their workflows.

This co

…[更多細節]

哪些人適合這堂課？

• Data scientists and data engineers who would like to start writing workflows in Airflow v2 quickly

學習目標

• Understand the overall Airflow system
• Learn about DAGs, operators, sensors, connections, hooks, etc.
• Get an introduction into advanced topics, like XCOMs, conditional execution, etc.
• Do a high level review of the Apache Airflow code base
• Get the ability to unblock yourself if you run into any issues running Airflow workflows