Apache airflow for beginners | Executors Architecture

Apache airflow for beginners | Executors Architecture | Comparison

Рет қаралды 18,170

Күн бұрын

#apacheairflow #airflowforbeginners #apacheairflowarchitecture #datascience #datapipelines
Read article here maxcotec.com/2021/11/apache-a...
Second video on series Apache airflow for beginners, focusing on the high-level architecture of airflow. We'll be discussing and comparing the architectures of 4 major executors namely Sequential, Local, Celery and Kubernetes Executors.
By the end of this video, you'll know how airflow works under-the-hood and which executor suits for your use-case.
If you're new to airflow, be sure to watch the first part of the series here
• Apache airflow for beg...
Watch next video here:
Run Airflow 2 via Docker -
• Run Airflow 2.0 via Do...
Run Airflow on Kubernetes
• Airflow with Kubernete...
👍 If this video was helpful to you, please don't forget to thumbs-up,
⌨️ leave a comment and
🔗 share with your friends.
⭐ Support our work by hitting that subscribe button.
🔔 hit bell icon to stay alert on more upcoming interesting stuff.

Пікірлер: 17

@maxcoteclearning Жыл бұрын

Run airflow on multinode kubernetes cluster - tutorial out now kzfaq.info/get/bejne/iNeDjMtkuq_Deac.html

@PedroCarneiroJr1971 2 жыл бұрын

At 5:12 you'll need to reedit the audio because there's an overlapping audio on the top of the main one.

@PedroCarneiroJr1971 2 жыл бұрын

Very clarifying video! Thanks!

@shaikhmushfikurrahman4233 Жыл бұрын

Great explanations! Appreciate your effort and works!

@kkmahapatra8538 Жыл бұрын

Brilliant explanation on point . Thanks 👍

@essakhan1575 2 жыл бұрын

Great video. Good simplification.

@vijayannallasami376 2 жыл бұрын

Excellent, thanks bro!

@lingxu9697 2 жыл бұрын

Great Video! Thanks

@uneq9589 2 жыл бұрын

Short and crisp content!

@shakib.polock Жыл бұрын

nice and precise explanation.

@manivenkata Жыл бұрын

Very precise explanation

@shivkrishnajaiswal8394 2 жыл бұрын

Good video

@jatingrover5557 3 жыл бұрын

Thanks for sharing knowledge. It is great video indeed. Can you post some content for single node and multi node concept of Airflow. What all components run on same machine/node and what all run on different machine/node. How different components work in conjunction with each other for Airflow execution .

@maxcoteclearning 3 жыл бұрын

Glad you like it. And thank you for being our first subscriber. Sequential and Local Executors are good example of how airflow manages running tasks on same machine. Whereas Celery and Kubernetes executors are normally utilised for multi node architecture. We will look into this in-depth on upcoming videos about how Kubernets Executors work with minikube and on AWS. meanwhile this may help you further limitlessdatascience.wordpress.com/2019/10/10/apache-airflow-architecture .

@maxcoteclearning Жыл бұрын

@Jatin Run airflow on multinode kubernetes cluster - tutorial is out now kzfaq.info/get/bejne/iNeDjMtkuq_Deac.html

@NathanPrice-qr2wn Жыл бұрын

With a Celery Executor could you have the workers behind a HorizontalPodAutoscaler to not always have a fixed number of workers or does something about Airflow prevent that kind of dynamic scaling?