What Tools Should Data Engineers Know In 2024 - 100 Days Of Data Engineering

Рет қаралды 23,201

Ай бұрын

What tools should a data engineer know?
Honestly this video is more of a list of tools that goes far beyond what most data engineers know but I wanted to create a video that shared a list of data engineering tools for the 100 days of data engineering video.
So here it is!
Also, if you're looking to check out some tools that I am advising for you can look at
Estuary for data ingestion
bit.ly/3Ed1RLe
And
Mage for workflow orchestration
www.mage.ai/
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Пікірлер: 33

@avikchatterjee7854 Ай бұрын

As an absolute clueless beginner, have to say you are doing an incredible job introducing this subject. No fancy jargon, just pure simple English for anyone to understand.

@SeattleDataGuy Ай бұрын

Thank you! I am so glad to hear that, any thoughts on what the next few videos should be?

@avikchatterjee7854 Ай бұрын

@@SeattleDataGuy I really feel you are taking the channel in the right direction. You have cleared most of the general broader explanatory and introductory stuff and you are now creating in depth tutorials, how to actually use stuff; just keep on with that. Also maybe include some real life case studies and incidents similar to the style of that video where Hussein Nasser delved into that Uber shifting from Postgress to MySQL. And DE, being the tool/skill intensive and ever rapidly changing stream it is, you will never be short of content tbf atleast for the next 5 yrs.

@Machiavelli698 20 күн бұрын

The sound at @15:57 is hilarious. But great video thank you.

@SeattleDataGuy Ай бұрын

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

@user-sd1hf1kj3i Ай бұрын

can you create a dedicated vedios on datawarehouse concepts,it will be helpfull for me!

@user-ww1rr5sb7w Ай бұрын

This video perfectly explains where to start with and where most of the focus should be.

@SeattleDataGuy Ай бұрын

I am glad you found it helpful!

@user-ww1rr5sb7w Ай бұрын

@@SeattleDataGuy Thanks Ben. I'm already on my 100 day challenge. Will mention you once I complete it. You are an inspiration!! ✨. Thank you. Can you please make an updated video on how to search for Data engineering jobs and what roles are good for beginners/ entry level

@Boredom_Incarnate Ай бұрын

I'm not sure where else to ask this so here's my shot in the dark; I've been a Data Engineer for 2.5 years now and things are getting bad at work (typical management change issues, favoritism at work causing bad decisions etc) and trying to find another job even just at my own level is depressing. I already got my Azure Data Fundamentals certification and am working towards my DP-203 (Azure Data Engineering) but it doesn't seem to be doing anything for my call-back rate on applications. I feel like the big issue besides years of experience is that my current employer is very narrow-minded with their data so besides Data Factory and T-SQL we don't use much, so I can't list things like Synapse or Databricks on my resume in good faith. All this to ask; How do you get claimable experience with some of these things if your current employer doesn't use them?

@danielsoto9876 19 күн бұрын

Maybe working on personal projects or looking around and seeing if you can contribute to open source stuff

@francisco_carvalho_ Ай бұрын

I love this content. Thank you so much

@SeattleDataGuy Ай бұрын

Glad you enjoy it! Any other content you'd like to see!

@karszn Ай бұрын

Can you make a video on how dbt is used to make a software engineering type of transformations? Thank you and everything regarding it

@SeattleDataGuy Ай бұрын

Thanks for the comment. I am curious, what do you mean by software engineering type?

@karszn Ай бұрын

@@SeattleDataGuy I see many companies moving towards Dbt for ELT style of DE. From what I’ve seen online I know they follow software engineering principles to perform the transformations. What I couldn’t find was a good video on how they perform the end to end project . I’m reading this book called Data Engineering with Dbt by Roberto zagni to understand more on DBT. If you could make a video on that it would be great . I understand your explanations much better . Thank you again :)

@SeattleDataGuy Ай бұрын

@@karszn Sounds like an interesting video! I often see dbt at mid-sized companies, i feel like larger companies tend to use more enterprise level tools

@karszn Ай бұрын

@@SeattleDataGuy yes . Same and thank you again for your videos and content . Appreciate it

@firstshield9507 Ай бұрын

I just started a job as trainee in data engineering, I just know Python (Numpy, Pandas, Matplotlib) and Advanced SQL and some concepts like ETL/ELT, EDA etc. I dont know anything about cloud, is it a good time to get a cloud certification as AZ-900 or it is more recommended to learn Spark before? (my company provide a voucher of 100% in Azure certifications)

@sandeepkumarv7366 Ай бұрын

Learning Spark on top of a cloud is the way!

@omkarjadhav8482 Ай бұрын

Can you share your LinkedIn please?

@franzuaandrez5591 Күн бұрын

what do you guys think about pentaho data integration ?

@christiansanchez9880 Ай бұрын

I've liked Prefect over Airflow

@tommydang7682 Ай бұрын

This is super helpful!!!

@SeattleDataGuy Ай бұрын

Glad you think so! Let me know if there is any other content you'd like to see!

@DANG3R0USS Ай бұрын

🎉🎉🎉🎉🎉

@reddixiecrat Ай бұрын

Are you planning on covering Palantir anytime soon?

@sushrafshaikh7996 Ай бұрын

He is using indian meme in his video...,🤣

@richardtvtv Ай бұрын

Ok at this point would it be easier to list skills Data Engineers shouldn't be learning now 👀👀

@relaxinggospelmusic2421 Ай бұрын

😗

@nadlev123 Ай бұрын

You started nice, but then your moved to name dropping if all the latest buzz words in the industry. Snowflake, DBT, azure, etc. are not tools, those are just different type of technologies you might be using as part of your work