Fastest way to Start Your Data Engineer Journey in 2024 - 100 Days Of Data Engineering Crash Course

Рет қаралды 67,767

Күн бұрын

How do you become a data engineer.
There are lots of skills you need to learn that range from Python, SQL, Data warehousing, etc.
Can you learn all of that in 100 days? The short answer is no.
But if you are looking to become a data engineer in 2024, this video will help you set a baseline! Also check out the checklist below.
100 Days Of Data Engineering Checklist
docs.google.com/spreadsheets/...
If you enjoyed this video, check out some of my other top videos.
Becoming A Data Consultant In 2024
• The Ultimate Guide To ...
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
0:00 Intro
2:19 How To Use The Data Engineering Checklist
3:59 Kicking Off Your 100 Days Of Data Engineering
4:12 Days 1-10 Of Data Engineering The Basics
6:02 Days 11-40 Deeper Dive Into Python, SQL, Data Warehousing
7:57 Days 41-50 Building A Mini Data Project
8:52 Days 51-70 Survey Of Data Engineering Tools
10:32 Days 71-100 Data Engineering Project
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Пікірлер: 90

@SeattleDataGuy 4 ай бұрын

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

@yuen3660 4 ай бұрын

inv is invalid :

@Fresh_Career_Compass 4 ай бұрын

Also upload videos on same format on Data Analyst and Data Scientist I have 300 days to learn 🕶️

@moradabbouazaz365 4 ай бұрын

This is really funny because this year my company offered me the posibility to enter the Data Department (I'm a Software Dev.). I'm kind of fed up with coding everyday and for now I'm learning Elastic Search, Apache Nifi and Airflow, then I'll head to Spark and AWS. This video is pretty intereseting to serve as a guidebook. Thanks for stressing the part of "What have you learned so far?"

@ishaan7287 4 ай бұрын

How is the work life balance in this career?

@JKyadav111 4 ай бұрын

What do you think about ai replacing software and data jobs?

@Someone-yl8sb Ай бұрын

@@JKyadav111 About as good of a chance that AI will replace doctors and engineers

@gobrax 4 ай бұрын

This couldn't have come at a better time for me, and a free sheets document to track all of this? So valuable, thanks!

@SeattleDataGuy 4 ай бұрын

glad you're finding it helpful!

@HI-bw8fe 27 күн бұрын

Thanks for the 3m plan. Exactly 👍🏼 just what I need!

@jameshizon4861 4 ай бұрын

One potential key topic is how to bounce back from difficult client/project that did not go as successful as desired (especially in data consulting if you have had that experience). For me, because of this experience, it made me lose confidence during a client screening to try & explain this experience & re-frame it as positive.

@SeattleDataGuy 4 ай бұрын

Hmm, this does sound like a solid video topic! Also I hope you feel confident again!!

@jameshizon4861 4 ай бұрын

@@SeattleDataGuy thanks. U the best :)

@yeaaayeaa1153 Ай бұрын

looking into this now, solid background in data primarily SSIS, informatica, Alteryx with a good amount of "BI" work. Thanks for this guide, we use Azure in our current work environment but mostly only at the corporate level. I'm going to challenge myself to follow this guide so I can transition into being a data engineer!

@arulnidhi6482 4 ай бұрын

Thanks for sharing brother! I'm just going to stick with this plan.

@SeattleDataGuy 4 ай бұрын

You can do it!

@tenzinnamdhak Ай бұрын

Will be following your map for the next 6 months as I am into 9-5 routine.

@travishawks7702 4 ай бұрын

Thanks for the video. Another video idea would be one for finding / getting your first DE job. It's a bit of a struggle finding entry DE positions

@SeattleDataGuy 4 ай бұрын

That would be a great video, i can add it to the line-up

@EndlessLearner 3 ай бұрын

Like number 1,000🎉 Great video! I’m going to start this challenge, you gave us a lot good information😎

@SeattleDataGuy 3 ай бұрын

glad you liked it!

@Yavin4 4 ай бұрын

Data Engineering means different things to different companies. For example, you don't list SAP as a core competency, but some companies do. WIth that said, isn't it better to pick a lane and stay in it? I.E. one or two of the cloud providers, AWS or Azure or GCP, and master data engineering, data analysis, machine learning, AI services, etc.

@ChildOfYahweh13 4 ай бұрын

I have a question? I am interested in becoming a data engineer but I want to ensure I go about things diligently. I see a lot of boot camps where I live in the UK and also the option to go to university. I’m open to both but what would be better for a learning time frame of 3 years?

@mintlatte1376 3 ай бұрын

Would pet projects be enough to show some knowledge of Spark/Databricks or Azure? I feel like without previous real work experience it's impossible to even land an interview for any data engineering related position.

@evelyns5218 3 ай бұрын

Hi Ben, I recently got laid off and looking into following your guide here. My background is in google analytics and Looker but we didn’t use too much SQL. Debating between Data Engineering or Cloud Engineering. I do love to build and create new things. Do you have any advice? I need to find a job quick and concerned data engineer could be saturated. What do you think?

@brainydel 4 ай бұрын

Thanks alot for the roadmap man! I appreciate. I just have a question for the cloud services like GCP, AWS courses should I take them all or just pick one. Also many are suggesting GCP. What's your take

@darshantawte7435 4 ай бұрын

Start with AWS. My suggestion would be to be book an AWS associate exam (Solutions Architect Associate ) 6months/1year from now and start preparing for different services through udemy course by stephaane Marekk. I did the same and cleared the above exam in 5 months (won't recommend to do it though as it takes a lot of effort in preparing , i did this only because my company was paying for the certification which i had to pass anyhow).

@brainydel 4 ай бұрын

@@darshantawte7435 Thanks alot

@SeattleDataGuy 4 ай бұрын

This was more meant to give you an overview, AWS is a solid place to start!

@goadyjerry967 4 ай бұрын

Hello what course , would you recommend for study ?

@pineapplepyramid5798 Ай бұрын

I’m up for the challenger

@davidkennedy4811 4 ай бұрын

Starts at 3:58

@SeattleDataGuy 4 ай бұрын

Yeah I didn't realize how long my intro was!

@willianferreira6613 4 ай бұрын

Would you recommend any cloud platform course like acloudguru or cloud academy?

@kol4321 4 ай бұрын

Hey this is awesome. Thank you!

@SeattleDataGuy 4 ай бұрын

you're welcome!

@teonaloving7654 3 ай бұрын

Thank you

@SeattleDataGuy 3 ай бұрын

You're welcome!

@tqbchannel3296 4 ай бұрын

Hi! I'm newbie in this field. Now, I'm following your roadmap in the video posted 2 years ago. Is this new roadmap better? Should I change my current roadmap to this video?

@SeattleDataGuy 4 ай бұрын

if you've gone through most of the prior roadmap, i'd finish that. I just wanted to create a timeline vs. the old roadmap that doesn't have a clear time constraint

@i_youtube_ 2 ай бұрын

Should I learn a lot about database administration or just writing SQL?

@daniell.7026 3 ай бұрын

I have a video idea for you. What would be an example of the minimum project portfolio to begin applying for jobs without previous engineering experience? Like 5 legit projects that include ____ & _____ & ____? I have no idea what's a realistic amount of coding to be considered for a coding entry level job....

@StartDataLate 4 ай бұрын

Can you tell more about the project 11:51 "transcript some videos, podcast and then summarize it with chatgpt"

@SeattleDataGuy 4 ай бұрын

It wasn't my project but they set up a bot to scrape podcast and videos and then used chatgpt to shorten it.

@jagannathishere 4 ай бұрын

can we get into data engineering with no experience? I'm good with Python and SQL.

@SeattleDataGuy 4 ай бұрын

its always a challenge, but often I see people go from data analyst to data engineer a lot.

@muhammadyasir7671 4 ай бұрын

@seattledataguy i have started the datacamps data engineer track. completed about 6 courses in there. what should i do . start this one or continue with that one

@TheShuchika 2 ай бұрын

Which datacamp have you started attending?

@zakariaelmoumnaoui4912 4 ай бұрын

Is data integration a part of data engineering or what exactly?

@agentsmidt3209 3 ай бұрын

Yes it is. It is part of the "ecosystem" i.e. one of the things you might be required to do.

@technicaltech8141 2 ай бұрын

@SeattleDataGuy Can AI effect totally to job role of data engineer

@xinyatang9633 4 ай бұрын

Thanks for the video!! I have a question: what is the book's name at 7:03 ?? Thank you!

@vikramadityapandey8966 3 ай бұрын

The Data Warehouse ETL Toolkit: Practical Techniques for Extracting, Cleaning, Conforming, and Delivering Data

@kiranyeluru7392 4 ай бұрын

Great awesome...best wishes from kiranu😢

@SeattleDataGuy 4 ай бұрын

Thank you but why the tears

@monicakherajani5840 4 ай бұрын

Amaaaazzziiinnngggg

@SeattleDataGuy 4 ай бұрын

Glad you're finding it helpful!

@Erinla_ 4 ай бұрын

is Data Integration engineer the same as Data Enineer

@agentsmidt3209 3 ай бұрын

I think it comes down to semantics. Some companies might call it ETL Engineer but it is essentially the same thing. Think of it this way, companies have pain points and they usually search for someone with a specific skill set to fix that specific problem (or a set of problems). It's like companies who are looking for a "python developer", "MySQL Developer", "PostGres Admin" etc.... but when you read the job description, the skills requirements listed are basically a Data Admin, Data Engineer, or even Data Analyst. Just know in the IT industry in general there is a lot of crossover in skills and duties. A Data Engineer is a "Software Engineer", "Database Admin", "Data Analyst" (some times), " Data Cloud Engineer/Cloud Engineer" etc...just read the job description and determine if it something you can do. Do not get hung up on job titles.

@pradhyumansinghmandloi8240 4 ай бұрын

But you told in your vedio that DE will end in a decade....so why this roadmap?

@SeattleDataGuy 4 ай бұрын

I don't recall saying DE will end, but it will change. When i started everyone was on hadoop, now everyone wants to be on snowflake, databricks, spark, etc.

@fung4629 4 ай бұрын

is data engineer a recession proof or layoff proof job?really worry if I should learn this under economy downturn😢

@SeattleDataGuy 4 ай бұрын

I don't think any job is recession proof. It's a reliable job but you never know when a company decides to cut teams

@Kamal-ju6qx 4 ай бұрын

just curious.can you land on data engineering intern with this 100 days knowledge.

@SeattleDataGuy 4 ай бұрын

It likely depends on your background. Perhaps, but you'd have to focus on making sure you could pass an interview

@Kamal-ju6qx 4 ай бұрын

@@SeattleDataGuy thanks I'm actually a Computer systems engineering student currently in 4th year have basic knowledge on Databases and cloud computing I'm hoping to enter Data engineering field in the future

@brainydel 4 ай бұрын

The first sql video uses SQL Server management studio but I use Ubuntu. Is It a must I learn it or install windows VM?

@SeattleDataGuy 4 ай бұрын

You could also use a different database like mysql

@danielpolanskyi2812 4 ай бұрын

can't access that spreadsheet :(

@SeattleDataGuy 4 ай бұрын

Were you able to access it? I have tested it and it should work

@supermarkethobo9567 4 ай бұрын

is there a way to learn cloud tools without spending a ton of my own money?

@SeattleDataGuy 4 ай бұрын

I believe most cloud solutions offer a free tier or trial. For example snowflake does like 30 days free or aws has its free tier - aws.amazon.com/free/

@supermarkethobo9567 4 ай бұрын

@@SeattleDataGuy hmm okay thanks I'll probably make a new account if my project takes too long 🤣

@ericxls93 4 ай бұрын

Fabric is the future…

@omarhassan6764 4 ай бұрын

Luffy reference at 2:04 🐐validates the entire video!! Really appreciate the list breakdown, Benjamin!

@SeattleDataGuy 4 ай бұрын

hahha glad you liked it!

@diegowargenosky8010 4 ай бұрын

I am doing the IBM Data Engeneering course from.Coursera. I hope to be ready to create a project and get hired for a first job after that.

@SeattleDataGuy 4 ай бұрын

GOod luck!

@mostofamojlish8255 4 ай бұрын

But it’s hard to crack data interview now

@SeattleDataGuy 4 ай бұрын

It can be difficult, but i believe you can do it!

@jagannathishere 4 ай бұрын

the discord invite in the Google sheets doc is invalid :(

@feveredpanda 4 ай бұрын

the link in the pinned comment works.

@SeattleDataGuy 4 ай бұрын

let me add it

@fabmeyer_ch 4 ай бұрын

Which is Kim's book? kzfaq.info/get/bejne/b6yGlsuI3Jmpkok.htmlsi=JkVtysrM5iKPSNr4&t=423

@SeattleDataGuy 4 ай бұрын

there are a few references in the excel link but you can find the book here 160592857366.free.fr/joe/ebooks/ShareData/The%20Data%20Warehouse%20Toolkit.pdf