How Data Engineering Works

Рет қаралды 413,923

Күн бұрын

So, the sole purpose of data engineering is to take data from the source and save it to make it available for analysis. Sounds simple, but it’s the matter of the system that works under the hood.
Watch our video to find out more about data engineering:
00:00 Basic Data Infrastructure
02:07 ETL Pipeline
04:34 Data Warehouse
06:07 Data scientists and data engineers
08:05 Big Data
09:45 Data streaming
11:47 Distributed Computing
Sources:
[1] sre.google/sre-book/eliminati...
[2] www.statista.com/statistics/4...
[3] www.forbes.com/sites/bernardm...
To learn more check our articles:
1) What is Data Engineer: Role Description, Responsibilities, Skills, and Background www.altexsoft.com/blog/what-i...
2) Explaining the Data Pipeline, Data Warehouse, and Data Engineer Role www.altexsoft.com/blog/datasc...
3) Big Data Engineer: www.altexsoft.com/blog/big-da...
4) What is ETL Developer www.altexsoft.com/blog/datasc...
Learn more about AltexSoft: www.altexsoft.com/
Follow us on LinkedIn: / altexsoft
Follow us on Facebook: / altexsoft
Follow us on Twitter: / altexsoft
Follow us on Instagram: / altexsoftcom
Music in video by Kevin MacLeod (incompetech.filmmusic.io): Sneaky Snitch, Sheep May Safely Graze - BWV 208, Wholesome, Disco Sting, Krampuss Workshop, Thinking Music, Whiskey on the Mississippi, Hidden Agenda, Egmont Overture, Sheep May Safely Graze, Holiday Weasel, Brittle Rille, March of the spoons, Hidden Agenda, Fuzzball Parade, Gothamlicious, Light Sting.
License incompetech.filmmusic.io/stan...

Пікірлер: 545

@DistortedV12 3 жыл бұрын

This video deserves an Emmy award

@jesska.parker 3 жыл бұрын

Second that

@TheInternetFan 3 жыл бұрын

agreed

@Weallgoingtodiesomeday 2 жыл бұрын

I approve

@musicaansen5449 2 жыл бұрын

Seriously true...!!

@karthikanand1403 2 жыл бұрын

absolutely!

@siddharthsingh5031 3 жыл бұрын

I am a Data Engineer and now I know what I do :D

@codingwombat 3 жыл бұрын

You.. Whuuuuuuuuuuut?.. XD

@nandhithanathan432 2 жыл бұрын

What should someone learn or know before entering into Data Engineering job.In what platform should they prepare for that ??

@avnishdeo5165 2 жыл бұрын

me too

@tektauron 2 жыл бұрын

LOL XD

@abcrx32j 2 жыл бұрын

Only non programmers would be surprised by this comment

@Krousedp 2 жыл бұрын

After 8 years of trying to explain to my parents what I do, this does it in 14 minutes. Thanks!

@MrBoomBoom225 2 жыл бұрын

I can imagine how the initial conversation went with your parents “No dad I’m not help desk, I do things to move data”

@malcorub 2 жыл бұрын

My parents are super old school Mexican in their 80s. All they know is "I work with computers" 🤣 It's super hard to explain when they don't even know how to use smart phones or computers.

@victoriwuoha3081 2 жыл бұрын

Lol.... I can imagine.

@nitronesstalk4783 2 жыл бұрын

It’s frustrating job? does it pay well? Pls answer back, adding more details will even help.

@diwasyt 2 жыл бұрын

@@MrBoomBoom225 Dad: That's what I'm saying. You move data from one desk to another desk😂😂

@krishnacism 2 жыл бұрын

I have never seen such a beautiful video on Data Engineering topic. It covered everything what a 400 pages book can't do. Appreciate your efforts

@rqn2274 Жыл бұрын

I'm working as a data engineer for 15 years and this is the best explanation of my work I have ever seen :)

@rajageetha1936 10 ай бұрын

Hi sir iam new to this field... Can you explain me your project

@itsorcus649 10 ай бұрын

@@rajageetha1936 lmao how can he explain all that in one comment

@sreeyeshb 2 жыл бұрын

This video is the single malt of all youtube videos explaining data engineering. This deserves an award for sure.

@gmr409 7 ай бұрын

One of the best technical explanation videos I’ve ever seen in 25 years in the business. Bravo, well done.

@AltexSoft 7 ай бұрын

Thanks =)

@moverecursus1337 7 ай бұрын

On January of this year (2023) after I took a time off university I watched this video, loved so much and started studying this field with everyday from morning to night watching, reading, coding and adventuring on cloud. And I can say, after 9 Months, I got my first job on tech as Data Engineer. Thank You AltexSoft ❣ Without this I would be nothing 🙏

@AltexSoft 7 ай бұрын

Thanks for sharing. We are glad that our video helped you in your journey

@user-fk9zm4mj5b 7 ай бұрын

First of all, congratulations for your steady obsession which led you to your job.🎉🎉 Can you give us the rough roadmap of what you did during those precious 9 months ? Again, congratulations ❤

@moverecursus1337 6 ай бұрын

@@user-fk9zm4mj5bIt's a long story, I recommend taking a look at Data Engineering Zoomcamp, it will start January 15th and it talks about everything a Data Engineer need (free, on KZfaq)

@mahmoudfadaly8074 Ай бұрын

This might be the best video on the internet explaining data engineer role

@generativeresearch Жыл бұрын

This team who made this video truly deserves an award.

@livyaharikachowdary644 Жыл бұрын

I am a Data Engineer and I cant tell you how much sense this video makes. Felt like we, Data Engineers got validation for the work we do! Next time some asks what do you do, I am going to send them this video link. Thank you :) Also special call for creativity and pun

@goldenmilktea4992 3 жыл бұрын

Now I have a better understanding of data engineering! Thanks for the video!

@music4u98 3 жыл бұрын

These videos are like movies. Well scripted, picturized and to the point . Loved watching them thoroughly. I like the background music change as the topic of content changes

@everenjohn 3 жыл бұрын

Hi AltexSoft! I comment here not because I watched this video, but just to thank you for the amazing content! You've helped me a lot in understanding many things about software engineering in a much interesting manner compared to lecturers. I'm glad I found this channel regardless of the number of views because you making amazing content!

@JuanCoolDad Жыл бұрын

I'm trying to transition from Data Analyst to Data Engineer. I already understand some of the concepts, but this video took it to the next level! Awesome content! 👌

@enesdogan1801 Жыл бұрын

Any news?

@enesdogan1801 Жыл бұрын

Any news?

@sakinasaifee3118 3 жыл бұрын

One of the best videos I came across, with great explanation and animation. Enjoyed watching it! Thanks!

@julesmules0901 2 жыл бұрын

I think I just fell in love with that video. It was truly mind-blowing. Not only did you guys manage to illustrate the different relationships between data systems and data discipline on a hight level but the way you visualised it made it so easy to comprehend that even people new to the subject would get it. Amazing! With topics like Data Engineering the true difficulty lies in simplifying things. And just just nailed that

@shatandv 3 жыл бұрын

Amazing. A really good high-level explanation of Data Engineering!

@jacobjulag-ay5639 3 жыл бұрын

This is an amazing video, summarized everything I have been researching in just one video. Thank you!

@future_teknokrat7585 2 жыл бұрын

Video so clean and clear, I had to view it several times. I normally have to do that for stuff I don't quite catch. I understood everything here on the first go and just have to see it again and again. This was the perfect combination of audio and visual intake with just enough jargon.

@priteshraka6408 3 жыл бұрын

Cant believe you cleared so many doubts at once in 15 min video! This was amazing work.

@ginnycheah5493 2 жыл бұрын

Concise and perfect! Soothing voice too! This video really deserves an award.

@user-vk6kk9wt9m Жыл бұрын

Great video! Very easy to understand explanations yet goes into enough depth for the general audience to understand the scope and complexity of data engineering. The graphics are great too, further helps with understanding the concept. Great work, thank you!

@Eric-fv8ft 2 жыл бұрын

Amazing explanation!! Now I know what's the difference and the relationship between all these terms and professions! Thank you so much for the content producer!

@shivnandanigarg9315 2 жыл бұрын

This is exactly how, lectures should be. This is ideal. Nothing can be enough when I want to appreciate it. Thanks

@dravennow8948 2 жыл бұрын

great video! really appreciate your work. the visualization looks pretty clear. can you tell us what's software are you using for the visualization?

@terran008 8 ай бұрын

This video is amazing, so informative, I don’t remember the last time I watch such a short video with such valuable and clear content on data

@caonhantran3531 2 жыл бұрын

The most meaningful explanation about Data Engineering I've ever seen.

@swathik3117 2 жыл бұрын

I was searching for this kinda info almost since 3weeks, was so confused about the Big data DE tools how they fit into the system. Million thanks to you! :)

@Seeker_Winterfell Жыл бұрын

After watching this video , I feel like I have more experience in Data engineering now. Thank you

@jithinjawahar3744 2 жыл бұрын

So simple yet so profoundly explained. Thanks for your efforts

@ariesykes1432 2 жыл бұрын

Really interesting video! It definitely relates to what I am working on currently. I work in IT recruiting. I'm always looking for a data engineers for my jobs and this video helps me understand a little more about what data engineers have to face in their field. Thank you!

@bernardogalvao85 Жыл бұрын

This video is amazing. I finally understand the place Spark and Kafka have. The more I learn about Data Engineering the more fascinated I get. Your video definitely contributed to that!

@sonicbroom8522 2 жыл бұрын

This is honestly lightyears ahead of other videos on what DE is. Giant Kudos to you

@vaibhavgadewar5391 2 жыл бұрын

Never seen such impactful short video on the Data Engineering concepts which clearly show the role of each component and how they get added in progression with evolving concept.. Masterpiece.. Hats off to you guys for this simplicity 👏

@hormengyoong2054 Жыл бұрын

This is the best video that I have watched covering all aspects of data engineering in a simple, funny and engaging manner. It is a great work 👏 👍 👌

@denidota6393 Жыл бұрын

I am an ETL Developer, the more I learn about data, the more I feel that my knowledges very less . Thank you for amazing videos !

@supriyodatta3116 2 жыл бұрын

A wonderful way to explain data engineering! Provides a complete overview! Wow!

@divyaaggarwal7824 Жыл бұрын

What a lovely way to explain the concept of Data Engineering ! Kudos to the team 👏

@TehUnbreakable 2 жыл бұрын

Such a great video ! Clear explanation with a good storytelling. Thank you!

@bnee4313 2 жыл бұрын

In such a short time, video touches on every aspect of Data engineering. Great job

@cidrisonly 2 жыл бұрын

In the middle of the night when you fetch your phone to make sure your decision on the Data Engineering boot camp is right, such videos are very reassuring. I don’t know how much I should appreciate this channel for this video in particular. You have earned a sincere Subscriber now.

@shubhamdandekar20 Жыл бұрын

I have never seen any video explain Data Engineering this good. Very easy to understand.

@whymeiy 6 ай бұрын

The most concise, accessible, and beautiful video I've ever seen describe my job. Captured why data engineering matters, took us through a history of how pipelines have evolved over time, and illustrated the complexity we deal with without actually bogging down the viewer with said complexity. Absolutely wonderful! Thank you for this masterpiece! 💜

@syed_hussaini Жыл бұрын

I didn't know how data scientists & data engineers work together & also the usage of data lake and data warehouse before watching this video. A big thanks from your new subscriber.

@KarenLeos91 3 жыл бұрын

Im in love with this video, thank you for the thorough explanation

@calmrelaxationmusic8856 2 жыл бұрын

I have seen multiple videos on DE on youtube but this one sums up everything so well! All of you new folks need to check out this video. This deserves more traction.

@edsonsabino 2 жыл бұрын

The video is so instructive and fun at the same time

@SamrudhVNayak 2 жыл бұрын

This video is just wow!! Simplified but captures the entire essence of Data Engineering.

@tinarajan9886 Жыл бұрын

This is the most brilliant explanation I have watched of data engineering. The most worthwhile 15 mins of my life today 😅

@AltexSoft Жыл бұрын

Thanks, we'll continue to do our best ❤️

@daniellange7190 2 жыл бұрын

This video is so well produced and informative! Thanks!

@54peace 2 жыл бұрын

this video is Such a GOLD❤❤❤ no tutorial videos explained like this widely & beautifully. now everything makes sence to me about my study. no more watching 'day in life as a data scientist' videos hoping to grasp some knowledge. i ma start working now. Thanks ❤❤❤

@ellouzesami5600 2 жыл бұрын

great vid. helped massively in preparation for an interview. keep up the good work!

@thibaultvialar2346 Жыл бұрын

This video is amazing! Simple and yet so precise. Thank you so much for your great work!

@RZing 3 ай бұрын

By far the best video to describe what data engineer is in a simple way,thank you!

@ShaileshKumar-lx6ov 2 жыл бұрын

The best video about Data Engineering ever. Kudos to the team who developed this.

@mosesojonuba416 4 ай бұрын

Have been reading a lot about data Engineering but couldn't understand and place together all the concepts and terminologies.. But after watching this video animation and explanation, I can boldly say that i now understand the concept of data engineering. Thanks you

@zma314125 2 жыл бұрын

Thank you guys for explaining Data Engineering in a way that I actually understood it.

@meowy2k Жыл бұрын

this is THE BEST DE summary i've seen so far

@shankz1328 6 ай бұрын

This is a phenomenal video. Very well explained and designed. A big thanks to you.

@robinalvarezp 2 жыл бұрын

This is the most clear video about Data and Analytics than I ever seen. Congratulations AltexSoft.

@lopella 2 ай бұрын

This is fabulous. So much good information summed up in a short package. Very easy to understand. Thank you!

@durgaprasada226 2 жыл бұрын

This is awesome. I hope all good things happen to you. I'm starting on big data and now i have better idea of what's going one.

@mohamedkhalaf6791 3 ай бұрын

i don't think there is a better explanation than that. i am thinking about learning it and it made me love it.

@sole008 2 жыл бұрын

wow best video so far about this topic and i spend the day looking at articles and videos of this subject

@adaw.5180 Жыл бұрын

This video is awesome! It explains the key elements in data engineering in such a simple way while catching all the main points.

@ganapatihegde2320 2 жыл бұрын

Absolute beauty! Sheer brilliance to cover this in quarter of an hour.

@RoryBarrett 2 жыл бұрын

This video has explained this incredibly well. I'm sharing this with my entire team.

@arefeshghi 2 жыл бұрын

An exceptionally well-made video! The only 'slightly' confusing part was the part describing Kafka.

@ossamahasni7739 2 жыл бұрын

This is absolutely one of the best videos I have seen in my life. PERFECTION

@farazqadir324 Жыл бұрын

probably the best video on the internet about data specific roles and their respective operations

@arigato1901 3 ай бұрын

This is probably the best video I have seen to simply describe Data engineering! 👍

@jamwithsaksham 2 жыл бұрын

Thank you KZfaq for recommending this ❤️One of the best video on explanation of Data engineering 🔥

@darylarenas7503 3 ай бұрын

Well explained and simplified!

@AminKhan11 2 жыл бұрын

Amazing Video, much efforts had taken to make this video having simple understanding of big data concepts

@anikulapoidris8860 3 ай бұрын

One of the best video I’ve ever watched on Data topics

@jacquessteyn78 2 жыл бұрын

Wow - the absolute best explanation I've seen. Thank you for sharing!!

@satabdisatapathy433 3 жыл бұрын

This is all that I needed and was Searching for since a week ...thanks a ton... I usually never comment.. but this was a life saver!

@alejandrogomez8766 2 жыл бұрын

Wow! this is the best way I have found to understand data engineering. Thank you so much.

@abhinendrsharma8137 2 жыл бұрын

Bruh ! Loved this beautiful and simplified illustration of the Data Flow ☺️ Great Work !! ✌️✌️

@et.sachin 10 ай бұрын

This is by far the best video explaining the word of data science.

@elyelena1002 2 жыл бұрын

This video is incredible. thank you for making it. Perfect explanation.

@vinitmahale 2 жыл бұрын

Omg! This is absolute gold! Thank you so much! I'm not an engineer but I understood how Data Engineering works now in under 15mins! I can't thank you enough for this video.

@amadousidibe2931 9 ай бұрын

Best ever , simple and clear

@CuriousDragonite Жыл бұрын

Thank You, Thank You, Thank You! Never seen such a perfectly crafted video talking about Data Engineering.

@AltexSoft Жыл бұрын

Thanks for your kind words, glad u liked our video

@venkatm.energy Жыл бұрын

What a fabulous video! Kudos on the great animation work

@alen740926 3 жыл бұрын

very insightful and give you a brief history of data engineering/architecture.

@saikatnextd 11 ай бұрын

Awesomeness is an understatement // you have covered such a vast topic in such a crisp and concise manner......

@AltexSoft 11 ай бұрын

Thanks =) Glad you enjoyed it

@ambhat3953 Жыл бұрын

One of the very few videos i have watched fully before going into comments section.

@siyavashfilom9120 2 жыл бұрын

Insightful, concise, and attractive. Many thanks!

@stathiskapnidis9389 5 ай бұрын

you guys should give your animator a raise. Great video

@Shradha_tech 2 жыл бұрын

What an amazing explanation. Thank you so much for this video

@Icelander00 Жыл бұрын

It was so clear and best ever explanation of DA,DE & DS

@prasannakumarrg9253 2 жыл бұрын

Well described, great visuals, really impressed. Thank you for your efforts !!

@Riku155 Жыл бұрын

I love the animation! The explanation is well explained too! Thank you!

@himanshupandey4038 Жыл бұрын

This is one of the best videos on Data Engineering I was confused with all the other resources just using big buzz words and not explaining the core of the job Thank you for the simple explanation

@AltexSoft Жыл бұрын

Glad it helped you ♥