What is Data Pipeline | How to design Data Pipeline ?

What is Data Pipeline | How to design Data Pipeline ? - ETL vs Data pipeline (2024)

Рет қаралды 1,037,455

3 жыл бұрын

What is Data Pipeline | How to design Data Pipeline? - ETL vs Data pipeline
#datapipeline
📢📢 Subscribe to my FREE newsletter "Normal I.T. Guy" to know more about what I am thinking, learning and doing in my career & life !!
CLICK THE LINK BELOW 👇👇👇
anshultiwari.substack.com
**Do check out our popular playlists**
1) Latest technology tutorial (2020) -
• What is a Data Vault ?...
2) Google Cloud Platform Beginner Series (2020) -
• Google Cloud Platform ...
3) Crunching Data Series (2020) - • Learn - Data Engineeri...
Hi Friends, I am Anshul Tiwari, and welcome to our youtube channel I.T. k Funde.
More about this video -
In this video, we will understand what is a data pipeline with the help of a real-life example. Data pipelines are designed to move data from one point to another. In this video, we will cover below topics -
1 - What is a Data Pipeline?
2 - Why you need a data pipeline?
3 - Basic design of a data pipeline
4 - Types of Data Pipeline - Batch, Streaming, Lambda architecture
5 - Advanced data pipeline design
We will also learn about various products that can be used in a data pipeline - SAP BODS, Mongo DB, Apache Kafka, Big Query, MDM, Teradata, SAP Business Objects, Tableau.
Credits & Resources -
Wikipedia
www.alooma.com/blog/what-is-a...
*********************FOLLOW US ON*************************
Facebook - / itkfunde
Linkedin - / ansh9685
Twitter - / ansh9685
Blog - blogs.itkfunde.com/
Instagram - / itkfunde
****************************************************************
*****************About This Channel************************
Friends ITkFUNDE channel wants to bring I.T related knowledge, information, career advice, and much more to every individual regardless of whether he or she belongs to I.T or not. This channel is for everyone interested in learning something new!

Пікірлер: 1 200

@sivanagarajugamidi 3 жыл бұрын

Master piece tutorial for data engineering

@ITkFunde 3 жыл бұрын

Thanks Siva

@indexima6517 3 жыл бұрын

hey! don't hesitate to follow us and to take a look at our videos which deal with the same topics :)

@vijayjayaram606 3 жыл бұрын

@@indexima6517 I guess the videos on ur channel deals with more on, what do we do after receiving the data, analytics if I understand correctly. Here, its more of pumping the data from one place to a common place, and make it available for interested people down the lane

@gautamdeusa 2 жыл бұрын

@@ITkFunde It's truly one of the finest and easiest video to follow and relate. Many thanks. Will check other videos.

@lwhieldon1 2 жыл бұрын

Thank you for breaking down concepts that are difficult to understand!

@nathancarranza9860 3 жыл бұрын

Something I’ve noticed is that Indians are good teachers and give great illustrations. Good work. Greetings from the US.

@ITkFunde 3 жыл бұрын

Thanks Nathan for making me feel even more proud of being an Indian thank you for the compliment means a lot brother 🙏😊

@fsfernandes20 3 жыл бұрын

Yes Indians like to make difficult concept easy

@nathancarranza9860 3 жыл бұрын

I don’t use my real name online, but I do give real compliments.

@AutitsicDysexlia 3 жыл бұрын

@@nathancarranza9860 Plot Twist: His real name was not Nathan. It was always Vladimir Putin.

@visionxx8656 3 жыл бұрын

Can't believe Putin is from US

@altamashjawad6691 2 жыл бұрын

Loved this video, probably the best explanation on advanced data pipeline out there. If in your next videos, maybe create a playlist which can show each of the section of this pipeline in detail with little examples using Python or any language etc. Just an idea, brilliant work!

@ericdasse8174 2 жыл бұрын

That was great! As a data engineer in the making, this is the first time I have understood the concept of data pipelines so clearly. Thank you very much

@lukmanaliyu7386 2 жыл бұрын

Hello Eric, I'd love to know how it's going for you at the moment with the DE track

@MrBignate12345 3 жыл бұрын

Please continue to create videos like these! So easy to understand. Love your visual teaching style and the examples you give.

@ITkFunde 3 жыл бұрын

Thank you MrBignate...The aim is to simplify these techie jargons for everyone to correlate and enjoy learning.

@sy-vf4js 3 жыл бұрын

And again, another easy-to-digest video. Thumbs up!

@ITkFunde 3 жыл бұрын

Thank you 🙏🙏☺️

@squarehead6c1 3 жыл бұрын

Great intro, just what I needed. I learned the distinction between ETL and general pipe lines, and Kafka's place in the architecture.

@ITkFunde 3 жыл бұрын

Thanks Ronnie☺️

@MrBignate12345 3 жыл бұрын

Would love to learn more about how to choose the right frameworks/technologies for data pipelines and data warehouses/lakes for differing requirements. It would be nice to see a playlist of you designing or comparing solutions for an analytic stack.

@ITkFunde 3 жыл бұрын

Thanks MrBignate I have created various playlists one of which is " Crunching Data Series "...I will surely make more videos on similar topic. It is because of encouragement from audience like you which helps me move forward so thanks and really grateful for your positive feedback.

@sid1r 3 жыл бұрын

Thank you so much for a great and easy to understand data pipeline introduction. I love how you focus on the concepts and not jargons, as it allows for people to understand the essence of data pipeline.

@lcsxwtian 3 жыл бұрын

Simply one of the best videos on data pipeline on KZfaq. Deserves so much more attention.

@ci2484 3 жыл бұрын

This was extremely helpful in my new job working in data. Thank you for your approachable and easy to understand teaching style!

@ITkFunde 3 жыл бұрын

Thanks a lot 🙏

@jananisri6214 2 жыл бұрын

One of the best tutorials in youtube so far which gives an overview of data engineering process and that too within 10 minutes. Really appreciate your effort and time you put into making this video. Thank you so much. Please keep doing more such tutorials.

@aitemiryeskenov355 3 жыл бұрын

Wonderful explanation. Can't tell you how much I appreciate brilliant tutorials like this one.

@jamesmcmurtry5351 3 жыл бұрын

Great visual layout. Would love to see this applied to an ELT model with Snowflake and it's advantages/disadvantages. Possibly a suggestion on ML complementary tools like Looker and Kraken.

@prabur4027 2 жыл бұрын

This would be the Best start for the Data Engineers.. A clear precise and short pictorial representation of Data Pipeline (Basics). Best video so far I had seen.. 😊 Thanks.. Much Appreciated.. 👍

@ITkFunde 2 жыл бұрын

Thanks Prabu 👍☺️🙏

@vivekjoshi3769 2 жыл бұрын

Do data analysts also use data pipeline creation in their jobs ? Or are they expected to know it ? Asking as some companies write knowledge of ETL in JDs.

@prabur4027 2 жыл бұрын

@@vivekjoshi3769 knowing any of the ETL tools would help in constructing the pipelines and they can visualize data flow from source to target.. Yes mostly it is used..

@alexanderulloaopazo6275 3 жыл бұрын

Thank you! I had read a lot of papers about Data Pipeline, but I couldn't get the main idea. However, your video was so easy to understand!! Now I have a better picture of the complete process. Thanks again.

@ITkFunde 3 жыл бұрын

Thank you Alexander !!!

@chocochipbananasplit 3 жыл бұрын

I got more out of your video than reading 5 articles on the matter! Your content is great!

@bigglesharrumpher4139 Жыл бұрын

Great video - it seems while technology has advanced, the concepts of batch loads and real-time data is actually decades old. Back in early 2000's we controlled all ETL and real-time loads with Unix or DOS or SQL scripts that provided return codes for success/failure which triggered alert emails, and we had KPI's for Data quality, backing-out jobs for failed loads, and many other control systems. It just seems there are more 'out-of-the-box' software to handle these now as opposed to custom-built solutions. Great presentation!

@Thekingslayer-ig5se 11 ай бұрын

Nice

@rajguru1998 3 жыл бұрын

Finally understood the pipeline in 10 mints... thank u

@mitchelleleeuw2266 Жыл бұрын

☺️I’m new in Data Engineering and man you created a clear picture of what I’ve been learning and trying to understand 🙂love this… definitely subscribing 🤩

@mardiidking4030 Жыл бұрын

This topic is so complex as a beginner, but I understand this explanation so well. I didn't even have to go back in the video or rewatch it to understand. This is beautiful.

@ITkFunde Жыл бұрын

Thank you so much for your kind words and support 🙏🙏♥♥

@kenford3738 3 жыл бұрын

Great job explaining the difference between Data Pipelines and ETL.

@ITkFunde 3 жыл бұрын

Thanks Ken 🙏☺️

@chimmenegaius-assor3767 2 жыл бұрын

Great explanation for introduction to data pipelines. Thanks for clarifying the distinction between ETL and data Pipelines.

@ramakrishnachimmani7273 3 жыл бұрын

Thank you. The best way of explanation. I was looking for this kind of video for long time. As a traditional ETL developer, I questioned my self, why people are using a term called 'Data pipeline' though we have ETL process and what is the exact difference between them. Thanks again.

@ITkFunde 3 жыл бұрын

Thanks Rama for your positive feedback !!

@juliansihite1289 Жыл бұрын

This guy really explain everything clearly and simple! Good job brother, keep sharing and contributing! You're a great teacher :)

@ITkFunde Жыл бұрын

Thanks Julian 😊❤️🙏

@brentcos9370 3 жыл бұрын

Very informative, especially for a non-computer science guy like myself. Thanks!

@ITkFunde 3 жыл бұрын

Thanks Brent that is the essence of this channel - Making I.T. interesting for everyone.

@JoeASalter 3 жыл бұрын

This is such a clear and useful explanation. Thank you!

@tektauron 3 жыл бұрын

GREAT explanation, wow. Thank you so much dude! This was incredibly useful. I've subscribed!

@dhritimanbnrj 3 жыл бұрын

best productive 10 minutes of my life.

@ITkFunde 3 жыл бұрын

Thanks Dhritiman for this super comment you made my day 🙏☺️

@K0n5tant 3 жыл бұрын

Your way of explaining these concepts is excellent, thank you!

@ITkFunde 3 жыл бұрын

Thanks a lot

@haydarissa9371 2 жыл бұрын

Very elegant way to explain data pipelining and ETL approach. I appreciate the examples given especially the master data management. Well done.

@komalsharma3856 2 жыл бұрын

You beautifully explained the basics & placements of various technologies in an architectural diagram. Very useful. Pls make more videos clarifying the basics

@mikebrooks4182 3 жыл бұрын

Thanks for a great overview of how the Lambda architecture can expedite the delivery of data to data consumers. For future videos, it would be helpful to map this to the roles, responsibilities, and skill requirements needed to manage this environment.

@ITkFunde 3 жыл бұрын

Thanks Mike for suggestion will try to add this

@e2xsq Жыл бұрын

Excellent high level overview Anshul, I appreciate that you differentiated between batch data and real time data with the Lambda Architecture as it seems most applicable to modern organizations. Your explanation of dashboards as consumers was also very realistic. Your video helped me better understand the general steps in the process. +1 Subscriber.

@ITkFunde Жыл бұрын

Thanks Matthew for supporting ❤️

@nitinbobde3170 3 жыл бұрын

Crystal clear explanation. Simple but effective way of explaining concepts. Excellent teacher. You rock.

@gurpreetsmatharu 3 жыл бұрын

Very well explained session on Data Pipeline and comparison with traditional ETL. Thanks so much!

@kalyanchakri5258 3 жыл бұрын

Love your way of teaching in a simple understandable concepts. Im mad of you..!

@ITkFunde 3 жыл бұрын

Thanks Kalyan for your feeback it helps a lot..

@JibrilLamai 3 жыл бұрын

This is a very good explanation and the best I have seen so far in my quest to understand this concept. Thank you very much. Now I can confidently visualize and explain the same concept with ease and a great understanding of it.

@ITkFunde 3 жыл бұрын

Thanks Jibril glad it helped 🙏☺️

@gilbertsibomana3081 3 жыл бұрын

Thanks for the video, I like how everything is clear.

@jessicafb5398 3 жыл бұрын

This was very helpful, thank you!

@Gridblue Жыл бұрын

Thank you for the video, I learnt what data lake hydration projects are, my previous company had no proper KT, I struggled to grasp what I was doing. This was very nicely explained and cleared the doubts that I had.

@ITkFunde Жыл бұрын

Thanks♥️

@knorth2386 3 жыл бұрын

Hi Anshul, your video was helpful. I have experience with ETL but didn't know that it was a specific type of data pipeline. Thanks for showing the different type of systems and technologies used for the concept visual that you explained with.

@ITkFunde 3 жыл бұрын

Thank you Kyle coming from an experienced guy means a lot. Hoping for continued support !!

@GoldJSilverio 2 жыл бұрын

Yes I must say this is very concise and how he names the commercial vendors as examples really augments the value further.

@johnsonmaurice98 3 жыл бұрын

Thanks for posting this. You are a gifted teacher. I am an experienced ETL Data Ops professional and I am prepping for a an interview for a junior Data Engineering position and I feel much more confident having studied your video !!!

@theososanna5096 Жыл бұрын

There will be another tech company IPOing soonish. Keep an eye out for shares, and a open career position.

@vox-populi-vox-dei- Жыл бұрын

I think, you'ver covered it all as part of Enterprise Data Architecture. This is from the technology view and bit of functional view as well. Good job!

@ITkFunde Жыл бұрын

Thanks

@hussamcheema 3 жыл бұрын

Excellent Explanation. Keep making more videos regarding Data Engineering, AI, and Data Science.

@ITkFunde 3 жыл бұрын

Thanks a lot mate for your feedback and suggestion!!

@arsaikia02 2 жыл бұрын

Hello Anshul Sir, I do not have any data engineering background, but I am very strong in visualization. You have aided me in finding the missing blocks about data processing and its treatment for final consumption. Keep doing your good job. Thank you again.

@edsonsabino 2 жыл бұрын

Great! The part that I liked the most was the one in wich he explained the difference between ETL and data pipeline

@KolawoleAdekoya Жыл бұрын

Simplified and clear explanation of the concepts. Great diction and presentation. Well done!

@ITkFunde Жыл бұрын

thanks Kolawale

@ravinduabeygunasekara833 2 жыл бұрын

This is superb!. I am very strange to Data Engineering, and this video gave me a super insight! Keep up the good work

@ITkFunde 2 жыл бұрын

Thanks Ravindu ☺️

@TheSocialDrone 3 жыл бұрын

An excellent and yet simple explanation of the topic! Thanks a lot for creating and uploading!

@hasmilaomar5562 2 жыл бұрын

It is good that u explain the concept of data pipeline by referring to water pipeline. So much easier to understand and remember. Thank you for your video!!

@maelherbert321 3 жыл бұрын

Really content. Bravo from France 👏👏👏

@ITkFunde 3 жыл бұрын

Merci Mael 😊

@Manoj419419 3 жыл бұрын

Great explanation and examples used. Thanks a ton !!

@ITkFunde 3 жыл бұрын

Thanks Manny

@djmenon21 Жыл бұрын

Superb video

@puneettiwari2251 3 жыл бұрын

Great

@sourabhsuri8812 3 жыл бұрын

Thank you so much brother, for clarifying some of the concepts.. Truly appreciate it. Can you suggest - Which way is the Tech Heading now - Data Warehouse Vs. Data Lake? Are DWH a thing of past?

@ITkFunde 3 жыл бұрын

Thanks Sourabh, DWH is here to stay its not going anywhere. Today data world has become enormously huge and there is space for DWH and DL to co exist also Datalake can not solve all business problem. There is a hybrid approach coming up wherein you have your DWH on top of your Datalake

@vivek1joshi 2 жыл бұрын

Data Mesh

@obiradaniel7391 2 жыл бұрын

Thank you very much, very elaborate and concise, this import for everyone in the technical data cycle, data engineer, analyst, administrator and data scientist.

@Nmmmmmmy 3 жыл бұрын

Thanks for the video - found the comparison to an actual water pipeline very intuitive. Really enjoyed the video !

@girishsoragavi7973 3 жыл бұрын

Wonderful 👍 this is exactly what I thought the data pipeline would look like and you hit the nail on the head there! Spot on!

@ITkFunde 3 жыл бұрын

Thanks Girish

@sam007sd 3 жыл бұрын

Very well explained with a perfect example in the end.

@issamfakhari3152 3 жыл бұрын

Great explication!!!!

@KESHA1982 Жыл бұрын

So glad, that I've found your channel on KZfaq. Thanks a lot!

@quocle5977 Жыл бұрын

As someone new to Data and AI sales, this was very useful to understand the flow of data and the tools used during each process. Awesome video!

@ITkFunde Жыл бұрын

Thanks Quoc ☺️

@nu7493 Жыл бұрын

Great job man! Very straight to the point and very informative. Thank you so much!

@jackieyou445 3 жыл бұрын

There is a lot of value in explaining technology in lay man terms like this. Thanks very much!

@sunils2111 2 жыл бұрын

Badhiya sir !!

@sharmilanadgir5030 2 жыл бұрын

Thank you for this simple and clear explanation of data pipeline. Now I have a clear picture of how data flows from consumer to producer

@arunachalampalani4321 11 ай бұрын

Couldn't have asked for more. Very well explained, Thank you mate.

@empressbelless3232 2 жыл бұрын

This is meant to be a compliment. I appreciate how articulate your English is with each word you speak! Easy to listen to!

@caunteya 2 жыл бұрын

Gazab hai bhai... Dhanyawad 🙏

@janbazahmedkhan8023 Жыл бұрын

bhai itna lucid style mein kisi ne nai smjhaya. great work!!

@rajivjani8594 Жыл бұрын

Superb! In 10 minutes, you have put such a clear picture of data pipeline in my mind that I will never forget! Many thanks for your time and sharing this valuable piece!

@ramakambhampati5094 Жыл бұрын

You are a real "Data Pipeline Spiderman".... fantastic instructor..please share more videos....thanks

@ITkFunde Жыл бұрын

Thanks Rama ☺️☺️

@jayanth1376 3 жыл бұрын

👌👌👌

@formulaRoot 2 жыл бұрын

Beautiful! Thanks for this!

@thenuchellappen4688 3 жыл бұрын

Explained in a manner that even a beginner can understand the clear cut concept.. I truly appreciate ur effort.Thanks. Do make more videos.

@ITkFunde 3 жыл бұрын

Thanks Thenu 🙏☺️

@learnnorwegian207 2 жыл бұрын

great video

@commonman1271 2 жыл бұрын

Excellent explaination

@DanishAnsari-hw7so 2 жыл бұрын

Such an awesome explanation, short, crisp and to the point. Great!

@georgesmith9178 Жыл бұрын

That was incredible. Thank you so much for this crisp, yet encompassing explanation of a data pipeline. I feel I totally understand it after watching for only 10 min.

@TheGhumanz 2 жыл бұрын

Very neat and simplified approach. Cleared my doubts about the need for data pipeline vs ETL. Thanks for sharing!

@essboogy 2 жыл бұрын

This is excellent. Really interesting and easy to follow. I am just starting training with IBM to be a Data Engineer. Leaving healthcare for good!

@ITkFunde 2 жыл бұрын

Thanks a lot ☺️☺️🙏

@tulajagautam4284 2 жыл бұрын

Amazing!!

@EMPELZIP 3 жыл бұрын

Great vid!

@himanshukadam9163 2 жыл бұрын

Awesome mate

@gajini1001 2 жыл бұрын

Simple, easy introduction. Thanks for your patience to explain the concepts.

@AbelAkeni 2 жыл бұрын

Succinct, presented with clarity! Beginners, get in here! 👏🏾👏🏾👏🏾

@meemanikandan 3 жыл бұрын

Good one to understand Data Pipeline. Thanks!

@ITkFunde 3 жыл бұрын

Thanks🙏

@vigneshvicky3481 3 жыл бұрын

Ua very clear wid d subject. No hurry, no messy... Content delivery is ultimate... Clarified all my misunderstandings ... kudos 😍😍

@Alebobb 2 жыл бұрын

That was just great, thanks

@tataigh12 2 жыл бұрын

Very nice.....

@0xggbrnr 3 жыл бұрын

Yay!

@technoarunav Жыл бұрын

WoW....Your lecture on Data pipeline is so simple and lucid to understand unlike other youtubers. I loved the way you have explained the concepts....Please do more videos on Azure and AWS...Count on me as a new subscriber has been added!!

@elifylmaz7940 11 ай бұрын

Wov, I think I just watched one of the best explanation video in my life. You did an amazing job! The structure you explain the details and use cases, the examples you give in real world applications made a lot of sense to me. Thank you so much!

@ITkFunde 11 ай бұрын

Thanks Elif for your kind words means a lot ☺️🙏