God Tier Data Engineering Roadmap 2024 with End-To-End Projects

No video

God Tier Data Engineering Roadmap 2024 with End-To-End Projects

Рет қаралды 150,472

Darshil Parmar

Күн бұрын

Пікірлер: 499

@DarshilParmar 8 ай бұрын

Let me know your thoughts in the comments and don't forget to hit LIKE and SUBSCRIBE :)

@sohamshah4666 7 ай бұрын

why you join Microsoft? you have your freelancing and your own one!?

@Coolthoom 7 ай бұрын

@darshilparmar Thank you so much for your videos. I'm trying to purchase your 3 courses through the combo plan link but the page isn't loading for some reason. The individual courses load properly but the combo one doesn't. Kindly advise further.

@sivasankardevendla4108 7 ай бұрын

@@CoolthoomDid you purchase Courses?

@BShabnam 7 ай бұрын

Can you please provide me your discord link, the link which mentioned is showing as expired

@logsofzany 5 ай бұрын

Bro can you please mention the duration for each of your courses?

@samarjeetsinghkheda9460 6 ай бұрын

1) Book:- fundamentals of data engineering 2) Data warehouse (olap vs oltp, dimension table, ETL, ER-modelling, dimension modelling) 3) Snowflake (data warehouse) 4) Book:- The data warehouse ETL toolkit 5) Data processing (bath processing, real-time data processing) 6) Apache spark - real time data processing (core architecture, high level API, functions available) 7) Data spark (Pyspark) 8) Apache Kafka 9) Apache Airflow (workflow) 10) cloud 11) open table format 12) data observability tool 13) docker, Kubernetes

@alansmailov6147 8 ай бұрын

watched till the end and already committed for 4 month, just learned basics of ci/cd in azure devops. hope getting a decent job in new year. massive thanks to u Darshil and all the community u building here! happy new year from Kazakhstan, Almaty

@LOFI_WORLD_SONG 7 ай бұрын

Hello can you tell me what are challenging and hard in data engineering also. I am in devops should I continue or should I switch to data engineer. As I don't want coding !!!

@alansmailov6147 7 ай бұрын

guys i did it, landed my first job as Azure Data Engineer, huge thanks to Darshil

@alansmailov6147 7 ай бұрын

@@LOFI_WORLD_SONGhonestly i don’t know😅

@MohamedSahith87 4 ай бұрын

@@alansmailov6147 where i can learn data engineering?

@ChristianeRhelyJoselleA.Bacani Ай бұрын

@@LOFI_WORLD_SONG if you don`t want coding as sad I said it but Data Engineering is a hard coding jobs you will automate the task as much as possible and the core concepts is data pipelines and databases so you need to code as much as possible too so yeah if you don`t want coding data engineering is not for you they are software engineers also that specialized in backend

@ahmetalperenyildirim6184 7 ай бұрын

Darshil, thank you! I'm following you for 2 years and now working as Data Analyst. I want be a DE! Great topics quite clear, thank you for sharing!

@khushbukeshari5577 6 ай бұрын

Hi Can you please help me on this? Can you provide me the roadmap how to move to as a data analyst? What are ths courses are required?

@phucluong6341 6 ай бұрын

00:04 Data engineering is a fast-growing field with high demand and numerous tools available. 02:24 Acquire computer science fundamentals and foundational data engineering skills. 04:41 SQL is the universal data language for communicating with databases. 06:58 Learning approach: Combine video learning with reading a book for effective learning. 09:11 Learning about data warehouses includes foundation and tools. 11:33 Data engineering involves bringing various data formats together and processing them efficiently. 13:52 Learning Data Engineering and Cloud Platforms 16:09 Open table format and data observability tools are key trends in data engineering. 18:10 Stay consistent and learn in public 20:15 Encouragement to engage with the video content

@hritikapal683 8 ай бұрын

Already followed your 2023 roadmap and learnt everything mentioned just building projects these days thankyou Darshil watched this to get the knowledge of modern data engineering tools quality content as always!

@DarshilParmar 8 ай бұрын

Thank you

@dawateden-yh4yy 8 ай бұрын

@@DarshilParmar MAKE COMPLETE COURSE AS SHASHANK WEBSITE GROW DATA SKILLS

@prof.mangabhai 8 ай бұрын

Hey hritika , how did you build your project, any good KZfaq channels or resources you used?

@mrweirdo163 7 ай бұрын

I am gonna become a Data Engineer in 2024 💪

@mr.chintuji 7 ай бұрын

Which coding language u prefer

@TM-op1xs 7 ай бұрын

Me too

@rishabhjain1381 6 ай бұрын

Me too

@viraleditzofficial789 5 ай бұрын

Mee too

@teamxipratikop9247 4 ай бұрын

Python sir @@mr.chintuji

@adib_shaikh 8 ай бұрын

As per your roadmap that you mentioned in 2023, i just completed the advance postgresSQL , now i sharpening my skills with python, thanks for guiding us❤️ hope I'll become Data engineer , with pursuing your roadmap in 2024😊

@kimathinewton855 8 ай бұрын

Same case here

@payalpokale214 7 ай бұрын

Hey Darshil,I am working as data engineer since 9 years almost now, couldn’t stop myself to go till last, good content man, almost have touched based all the tools & technologies are mentioned. Yeah it’s like big sea one should keep learning and growing. Keep up the good work and keep creating good content.

@newagene8276 7 ай бұрын

what would you suggest a person wanting to switch from swe to data engineering

@ankurghosh2387 4 ай бұрын

hey can I connect with you for guidance?

@faruqolasupo2277 7 ай бұрын

Thanks for sharing this roadmap - watched to the end with ❤ from Nigeria

@peterobumneme3911 4 ай бұрын

Good day. I am also a Nigerian pathing on the journey of Data Engineering. can we link up??

@sabyasachichoudhury2717 7 ай бұрын

1. Positive Mindset 2. Stop Distractions 3. First 5 videos CS50 By havard university 4. Python 5. SQL 6. Raed basics of data engineering 7. Data warehouse - snowflake 8. Data processing - Batch processing - Apache spark Streaming Data - apache kalfka 9.Workflow management tool -Apache work flow 10. AWS

@malikmudassarawan 7 ай бұрын

Consistency is the key and to have consistency you need proper motivation. Mostly we loose motivation along the way

@Shahnawaz_Ansari_ 8 ай бұрын

Great video and resources mentioned Appreciated 👏

@DarshilParmar 8 ай бұрын

Glad it was helpful!

@tamarainwang6362 8 ай бұрын

Getting my first data engineering role in 2024. Thank you!!

@njorogekamau3820 2 ай бұрын

I'm giving myself at least one year to learn data engineering without any expectations. I hope to revisit this comment again next year having landed a data engineering role.

@AHSAN88077 Ай бұрын

Me too on the same path.

@Vincent45678 Ай бұрын

@@AHSAN88077 Same

@grant5941 7 ай бұрын

Thank you, Darshil! I watched until the end! I already started Harvard Cs50 last week so I'm on my way. Thank you for this roadmap.

@kavyamuralidharan6878 8 ай бұрын

Very clear and driving roadmap video, thanks for the guidance!😃

@AryanRaghuvanshi-tm9el 8 ай бұрын

I have watched your end to end project videos previously, really amazing content you have been providing, I will check out your courses.. keep uploading videos like these ..

@DarshilParmar 8 ай бұрын

Awesome, thank you!

@navarrodba 8 ай бұрын

20 years as an Oracle DBA. Finally I’ll move do DE

@haridatta5495 7 ай бұрын

Watched till the end!! Great content. Will start learning from today

@simpledrawing7284 8 ай бұрын

Hi Darshil, can you make a video on role of data engineering in Generative AI and LLM models.

@DarshilParmar 8 ай бұрын

I’ll put this in list

@ayandazwane_ 7 ай бұрын

Thank you, this video was very informative, I watched all of it and I will certainly become a data engineer.

@thedataholics 8 ай бұрын

Thanks for the knowledge, you seem really truthful. Keep helping us sir

@joelsarpong1847 8 ай бұрын

I will become a Data Engineer in the year 2024 due to your road maps and your course provided. Looking forward to the rest of the courses coming up.

@guillermocelisprada 8 ай бұрын

Thanks for the video man, I found it very informative and well-structured. Keep up the hard work! 👍🏻

@user-uy7fd5zq8h 6 ай бұрын

Hi. Thanks for the roadmap, everything was clear. I didn't undrestand just one thing. You said we better to stick to just one web service like AWS. I wanted to know its function in the learning path, like what will we do in the web service

@DarshilParmar 6 ай бұрын

Get started, you'll understand it, create account and do my KZfaq data Analytics projects

@soumobratamanna6681 8 ай бұрын

I alreasy started learning data engineering after watching your past videos , I will be become data engineer . Thanks for the clear roadmap and instructions .

@DarshilParmar 8 ай бұрын

All the best

@soumobratamanna6681 8 ай бұрын

@@DarshilParmar Thanks a lot for sharing such insightful content

@soumobratamanna6681 8 ай бұрын

@@DarshilParmar Previously I was working as a data engineer in a early growing startup company in Bangalore. I resigned last month only due to some project issues. Now actively looking for a job . Would like to get some suggestions from you regarding job hunt in data science domain

@jbh7079 7 ай бұрын

Excellent content. Keep the good work up!

@karthikvibuthi9635 4 ай бұрын

I will start my data engineering journey today. With many projects I will gain skills from these road map as well as essential technologies outside the course and as they emerge

@deepthireddy3480 8 ай бұрын

I am going to become data engineer.. I have watched your video till the end.. And i wanna say you something that you are going to reach 200k subscribers very soon... Thank you so much for spreading such a great knowledge..

@DarshilParmar 8 ай бұрын

Thank you, jai Shree Krishna

@deepthireddy3480 8 ай бұрын

@@DarshilParmar jai shree krishna

@VirajPrabhu-nn9ny Ай бұрын

As a Data Engineer I can say this is a very good roadmap! Very well structured

@prof.mangabhai 8 ай бұрын

Highly anticipated for spark and aws courses !!

@user-om7lq4yk8s 8 ай бұрын

thanks for the video Darshil.Pls keep posting such awesome content.

@DarshilParmar 8 ай бұрын

Thank you, I will

@Cheer4SK 8 ай бұрын

I’m going to become a data engineer! 🥳 You guys have got this, too! 🌟

@reynardtcloete6959 7 ай бұрын

Great video brother :) thanks man

@vasundras4746 2 ай бұрын

This is the best video on KZfaq right now 💯 we need more of this energy !!

@kalyanishende618 5 ай бұрын

Thank you Darshil... I am on the journey to become DE now watching this video and the roadmap you mentioned in the video is pretty cool. Thanks for sharing such useful content

@the_witcherr275 Ай бұрын

I watched this this video till the end. I really want to become data engineer. Thanks for this amazing roadmap.

@narenbot Ай бұрын

Awesome god tier roadmap.. I will become a data engineer by this year .. thanks bro ❤

@DataScienceAndEngineering 8 ай бұрын

Thanks for the video. Really Great Video.

@jigarrthaker2441 3 ай бұрын

Hi Darshil, Thank you so much for providing such clarity Data engineering path, Currently I am working at organization which handle Splunk stacks and also i handle and work on splunk stacks which are on Snowflakes and also i work on Docker and kubernetes so somewhere this roadmap gives me clarity and open opportunity path to get carrier Data Engineering. Thank you once again and keep updating us with such of informative knowledge and also from your experience.

@yogeshkumar-du4jx 7 ай бұрын

Hi Darshil, its really a great content 👍 . I have watched your video till the end and you have shared the information in very good manner. I want to become a data engineer

@RaoAli-on8ns 7 ай бұрын

Thank you Darshil. Please make a video about Linux. How we can start learning it for Data Engineering and which Linux OS is to start with, and how to practice either on a virtual machine or install directly on our PC. Also , which fundamental, intermediatory and advance Linux included in Data Engineering. Waiting for your next video. Please do include which OS of Linux should we use . Thanks.

@viveksingh112 7 ай бұрын

Thanks Darshil, I watched this video till end and will keep following you to build my journey to become a Data Engineer

@hazarijayesh9807 8 ай бұрын

Intrested , watched video till the end. ready to start journey for data engineer.

@ShivamKumar-pk1rf 7 ай бұрын

Got the roadmap Thank you so much

@manibaddireddy5477 8 ай бұрын

Everyone is teaching roadmaps but no one is teaching domain knowledge like retail, hospital,aerospace etc. please do a video.

@DarshilParmar 8 ай бұрын

I have many projects, those are enough to get you started with all of these

@manibaddireddy5477 8 ай бұрын

@@DarshilParmardoes a video regarding your day to day life.

@abhinavtiwari4238 8 ай бұрын

Domain knowledge matters the most with the advent of AI which can be utilized for technical part

@ankitapandey398 8 ай бұрын

Thank you so much for this video, This has cleared all my doubts. You are doing great job for beginners like me and many others. I have watched this video till the end

@DarshilParmar 8 ай бұрын

Glad it was helpful!

@farantaimoorbutt5028 8 ай бұрын

Hey I am from a non tech background.I am a self taught full stack engineer.I couldn't find a job .Now thinking of switching to data engineering after watching you video .Whats your take on this? ->I am going to become a data engineer

@_k_a_m_a_l. 8 ай бұрын

I am working as an ETL developer and I will become a Data engineer in 2024 we are using Vertica Data Warehouse My doubt is which is good Vertica or Snowflake I saw some blogs that Vertica is more powerful in query performance than Snowflake

@mr.amitdas1541 8 ай бұрын

i ve completed this vedio.. This is very insightful.. Thnk you for the guid.

@yousef5886 7 ай бұрын

Thank you for this video Darshil! I am already a Data Engineer but my foundational knowledge is not the best and there is so much I still do not know, so thank you for explaining so thoroughly 😊

@wckd6726 7 ай бұрын

Watched it till the end and I swear to god I'm going to become a data engineer this year

@SatyaMalik-id5vm 5 ай бұрын

Thank you for being this clear and complete roadmap i just started my data engineer journey ❣️❣️

@sanjanagoud1155 7 ай бұрын

Thank you for the video Darshil..... I am going to start my career in the field Data Engineer. for sure in future i will become an Data Engineer.

@Gokulprasanth-gs8zi 8 ай бұрын

I am already in the journey, thanks for sharing valuable content

@Korudo 6 ай бұрын

I’m going to become a data engineer. I do data integration in my day job, and I want to grow and become truly excellent at this.

@moinshaikh8551 6 ай бұрын

I’m going to become a Data Engineer! Thank you for your useful information!

@xavier144 8 ай бұрын

Will AI take up Data Eng job? please make a video on it. also how to search for data eng jobs can you make a video on this also?

@DarshilParmar 8 ай бұрын

I already have two videos on this, just check my channel

@radhikab4864 8 ай бұрын

i have watched this video till end. I love the way you are teaching concepts and projects.

@hamzaabbas8563 7 ай бұрын

My goal is to become a data engineer by the end of the year 2025 mastering all the concepts and technical tools related to this field

@fahadbawazir562 6 ай бұрын

Data Engineering + DEVOPS (Aws or GCP or Azure) that is Docker + Kubernetes = ULTIMATE Data Engineer

@sriharisrinivasan1307 8 ай бұрын

Quick question: If we purchase your all-in-one combo course, Will you add the Apache & Pyspark course, or will it be launched separately as an additional purchase? Would like to have all round course material at one place for All the Data engineering topics.

@DarshilParmar 8 ай бұрын

It will be separate, current price is according to 3 courses only

@sriharisrinivasan1307 8 ай бұрын

@@DarshilParmar Okay thank you

@fahadbawazir562 6 ай бұрын

please make Apache Spark + Data bricks along with Snowflake complete video with handful of exercises & its application in the real life scenario, much awaited your reply

@mrafam626 8 ай бұрын

Glad I found your channel! How adept do I need to get in SQL and Python (as a complete beginner), or should I get the basic knowledge then focus on aspects of both programs that are used in data engineering?

@njoy2075 8 ай бұрын

I would say get the basics bare minimum. Then find one YT python and SQL project and start building it. Build 2 , 3 projects then go deep in python sql

@dhyeyhiteshkumarsuthar7150 7 ай бұрын

@@njoy2075 Can u provide complete roadmap pls? Because in this video i didn't understand that much. I m learning python and SQL. Should I make projects after learning this or after learning whole data engineering thing. Pls just can u provide roadmap like what topics to master in sequential manner?

@njoy2075 7 ай бұрын

Once you are okay with basics move to projects then again back to sql python advance then again project

@icon5819 8 ай бұрын

Hi Darshil, can you please drop rest of the courses quickly. I want everthing related to the course at a single place and your course is worthy enough for that

@SanjayKumar-rw2gj 8 ай бұрын

I'm going to become a high skilled Data Engineer in 2024. Will follow the roadmap thoroughly. You mentioned that we can get all these skills in 6-8 months, how much time daily I need to spend and there are bunch of tools to learn like, sql, python, Kafka, spark, cloud and many more is there any specific order to be followed or can few things be done in parallel?

@angellavado763 8 ай бұрын

I'll become a master in Data Engineering in 2024! 💪🏼🤩 Greetings from Peru 🇵🇪.

@sukritisachan5773 8 ай бұрын

Hey Darshil ! I had a question, my free subscription for Azure Dats Factory has ended, so is there any other way I could do hands on for that(without pay as you go model)?

@DarshilParmar 8 ай бұрын

Use different email and card, this is the only way! if you have student id then you can get some free credits

@manojreddy2871 2 ай бұрын

thank you sir, for giving us a helping hand to become a data engineer.

@yogendrasharma7534 8 ай бұрын

Hi Darshil, you are doing good thing at best, and i am sure that i'll be a good data engineer in 2024 itself, although I have 7 years experience in SCM in manufacturing sector.Hoping to connect both in 2024😊

@MrAd540 6 ай бұрын

My Goal is to Become DE in 2024

@Moses.Daryoue 7 ай бұрын

Thanks a lot, you have given me a lot of information and clarity on how to start data science engineering

@ManiKanturi 7 ай бұрын

Thank you :) I'm going to become the Data Engineer. Please make the course for the remaining topics or tools.

@Reactionz.3402 2 ай бұрын

I'm going to become by Data Engineer before the end of 2024. The journey just began.

@harishparamathmananda1385 7 ай бұрын

Watched till the end! Thanks for the tips and tricks, I'll become data Engineer in less than 6 months. 🎉

@karenemody 5 ай бұрын

Thanks Darshil. Very informative. Watched till the end.

@lamboarun 2 ай бұрын

Informative & Well Structured 😊

@neeleshvishwakarma6583 8 ай бұрын

I am going to become data engineer before may 2024

@akhilsrivatsav 8 ай бұрын

I will be data engineer by March. And watched your video completely. ❤

@Rider-jn6zh 8 ай бұрын

Hello Darshil, What do you think of Microsoft fabric ? Will it replace traditional data engineer tools. Kindly help us to understand this MS Fabric. Its advantage and scope in future.

@mellowtones1985 8 ай бұрын

I'm learning Fabric now and so far, so good.

@Rider-jn6zh 8 ай бұрын

@@mellowtones1985 for what solution u r learning? For DE, DA, or DS

@mellowtones1985 8 ай бұрын

@@Rider-jn6zh At the moment DE and DA. I’ve been using all the Microsoft pieces separately ADF, DataLake, Azure Spark, PowerBi etc. So far it’s very interesting to see how all can be used under one platform.

@Rider-jn6zh 8 ай бұрын

@@mellowtones1985 can you please send me the resource link that u are going through for ms fabric

@LOFI_WORLD_SONG 7 ай бұрын

@@mellowtones1985is it easy to become data engineer, no or very less coding, with less responsibility and continuous learning. Which devops or data engineering is best ?

@okorochaobinna7701 7 ай бұрын

You're a blessing to an aspiring data engineer

@surajshivakumar5124 8 ай бұрын

Hey darshil! Can you give us some tips/advice on contributing to open source as a data engineer?

@arindamsinha5857 Ай бұрын

Does your course spark with databricks use pyspark? Is your databricks and spark course cover end to end basic to advanced or need to refer anything else other than your course. Please help with these answers and also say if there is any pre requisite for the course apart from python, sql dwh?

@arjitsingh3519 8 ай бұрын

I would like to start my career as data engineer. I already familiar with Python, SQL , DSA , and linux. Learning all those additional technologies/tools its not a big deal for me. But the problem, I have a 4yr career gap. Few people told don't go for data science related domain because it is difficult for a fresher with a gap to get job. Is it that true can i go for full stack rather than DE? I'm stucked Kindly enlighten me bro..

@DarshilParmar 8 ай бұрын

Hi, You can always explain your career gap and also showcase you took efforts to learn new things, if you can prove you have skills by projects then no one can stop you

@DANNYEL20122 8 ай бұрын

So I'm learning python specifically for data engineering. What aspects of python do you think i need to focus on

@njoy2075 8 ай бұрын

Think of building a project where you have to parse a log file then join it with scraped web data then once the final dataframe is available, push it to sql. Now you can Google the components needed for the project. That's what you need to learn.

@amitjaiswal781 8 ай бұрын

Nice Video, Please also make a video on a GCP Data Engineering Roadmap.

@sibusisokanityi 7 ай бұрын

Great road map to data engineering I'm currently pushing data science but I also love data engineering....so now I'm pushing both 😅

@rudramohanmishra4578 4 ай бұрын

I am on the path of becoming Data Engineer. Hey Darshil , I have watched till the end & committed to this path. can you share the link to CS50 lectures please ...

@trygreenfirst1275 8 ай бұрын

This was helpful. Thanks

@aditisingh4018 7 ай бұрын

Stayed till the end ❤ A data analyst here, will be a data engineer in 2024 ❤ Thanks for the video

@atharvajoshi1003 4 ай бұрын

Watched till the end. Really inspiring video for all data engineer aspirants.

@Soneone-kz4fp 8 ай бұрын

I am gonna become a data engineer Plus, watched your video till the end, I also followed your uber analysis project, it was a real good one

@nagalakshmia1291 8 ай бұрын

i have already started my data engineering career that's from now

@hemanthsagar3537 5 ай бұрын

Thank you for your inputs! Completed Databricks certified Data Engineer associate after following your certification path. I am following every video that you upload. Thanks alot ❤