Will Data Engineering Exist In 5 Years - Is Data Engineering A Good Career Choice?

  Рет қаралды 51,515

Seattle Data Guy

Seattle Data Guy

Жыл бұрын

Should you become a data engineer?
Will ChatGPT replace you?
Will vendors replace you?
What will happen in the next 10 years?
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you're looking for an Airflow replacement, check out Mage.
bit.ly/445Kacm
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/​​
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Пікірлер: 69
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k
@justinballas7299
@justinballas7299 Жыл бұрын
I think a lot of the concerns are coming from students and people that haven’t entered the working world yet. When you see how much of a mess the vast majority of companies data infrastructure is you’ll realize data engineering won’t be going away any time soon.
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
THIIIISSSS! Yes...I should do a whole live just on this topic. I remember being a student, yeah it felt like everything was solved and everyone was doing ML...I promise most companies still don't do ML now. They just write articles about how they are doing it to stay relevant.
@idkwhy77
@idkwhy77 Жыл бұрын
@@SeattleDataGuy demand in EU is high and AI doesn't have scaling capabilities or problem solving the human data engineer has
@ibexy
@ibexy 9 ай бұрын
Absolutely 😄
@parlor3115
@parlor3115 6 ай бұрын
That's for existing projects. But for fresh ones where AI or AI-assisted engineers would be responsible of developing the entire infrastructure, development will be conducted having a strong forsight and following good engineering approaches.
@elkinhuertas2137
@elkinhuertas2137 5 ай бұрын
@@parlor3115This will take years. Implementing such an infrastructure is crazy expensive. Until it can be offered as a service up and down and across industries, we’re good 👌
@_muzzles
@_muzzles 10 ай бұрын
I think your point at 7:52 is super relevant. There is so much tech debt and the requirements keep changing. Maybe the world moves more towards point and click, but the core of data engineering will still exist and there wont be less of it.
@WisomofHal
@WisomofHal Жыл бұрын
I’ve been a backend developer. On recent interviews, hiring managers really asked a lot of questions about developing data platforms. There were technologies I wasn’t familiar with, which sucked because I was expecting a typical backend developer interview. The industry changes so fast.
@WisomofHal
@WisomofHal Жыл бұрын
@@rouenyu Yes, I noticed this as well. It seems, there are plenty roles for dedicated frontend developers and in those roles frontend, design skills, knowledge of APIs and browser side scripting with js are what’s expected and, honestly, that’s no surprise. Backend developers, as of late, are expected to know the full stack and be able to contribute on every layer (backend, architecture, cloud, storage, frontend, CI/CD, getting coffee and so on). Just a few years ago, I was able to look for a plain old backend developer role and I was set with the regular old java skills for backend. The industry evolves so fast and, it seems, you must with it while always be learning. My advice would be to set short term and long term goals: what I did was study frontend (react/vue) for 6 months until I can build full stack project. In that time I also studied cloud and architecture. The next 6 months I studied databases (Relational and Non Relational). I’m currently on miscellaneous technologies like Kafka, Flink, and streaming technologies). Luckily I got plenty of experience on my first job with CI/CD with Jenkins and such. Also, keep in mind, if you’re < 5 years in like myself, it just takes time to get familiar with ALL these technologies. As of late, there just seems to be more requirements on software engineers and an explosion of technologies. I don’t see the pace slowing down anytime soon. If not, then just ignore what I said haha. You’re probably extremely proficient in the technologies you’ve been using and could use that as leverage when you learn fullstack. You’ll crush it man!
@splashoui3760
@splashoui3760 Жыл бұрын
I don’t like you should set click bait video titles which you do not need. I see your channel as high quality channel. And it’s being disappointing this kind of titles.
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
I think this is fair. I am thinking of how I might change the thumbnail. Whats kind of annoying is that usually my CTR is like 6% and this video is 13%
@roncrudup2110
@roncrudup2110 Жыл бұрын
@@SeattleDataGuy I personally don’t think the title is that bad. You have to make a title that pops for the largest audience.
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
@@roncrudup2110 Did you see the old thumbnail? I just updated it...before it was like "It's Over For Data Engineers" with Fire. I think the Plain Bagel did an analysis and found those types of thumbnails got an average of like 3-4% more CTR which is massive.
@splashoui3760
@splashoui3760 Жыл бұрын
@@SeattleDataGuy I think important thing is to create your channel community with robust, real subscribers. Not the ones coming with click bait titles. That’s what I think. Still I appreciate your videos and thank you for your effort 👌🏼
@ulysseschang129
@ulysseschang129 Жыл бұрын
I clicked through when I saw the new thumbnail title. I think if I had seen the original "clickbait" title, I probably wouldn't have bothered checking it out.
@shardulkatareaz9984
@shardulkatareaz9984 Жыл бұрын
Hi @SettaleDataGuy, I have been following your channel for quite some time now. Your channel is great! and really breaks down what Data Engineering looks like on a day to day basis. Currently, I am working on a Data-engineering project. I need to pull data and compile it in excel for Reporting The data comes from several databases - Athena, Oracle and SQL Database, and some from Excel. This data becomes available at different points in time as each is for a different country. Currently this process is done manually and takes about 1-2 days. I am thinking of building a datapipeline using Apache Airflow. Do you think this is a good solution? Any of guidance that you can provide will be really appreciated or any material/video that you have created in the past to reference. Thank you! And please keep doing what are doing
@arbaazm3761
@arbaazm3761 Жыл бұрын
Yes you should
@kunalmishra8326
@kunalmishra8326 Жыл бұрын
Very nicely explained, I am working as data engineer for around 7 years? Do i explore more into MLOPS career now, or further explore data engineering ?
@gautam_505
@gautam_505 Жыл бұрын
Hii brother I'm interested in data engineering will I get job at this role at first place and what do they expect in skill sets?? Currently I'm doing my intern where I'm learning to build ML models
@Nick-du9ss
@Nick-du9ss Жыл бұрын
What do you think about data analyst and business analyst job role regarding next 5 to 10 years
@TomT-ds9vn
@TomT-ds9vn 4 ай бұрын
Keep getting hyped about the lower pay scales coming due to less technical skills necessary to do the job and job loss.
@Mari_Selalu_Berbuat_Kebaikan
@Mari_Selalu_Berbuat_Kebaikan 4 ай бұрын
Let's always do alot of good ❤️
@SeattleDataGuy
@SeattleDataGuy 4 ай бұрын
yes!
@mhn17
@mhn17 Жыл бұрын
I'm a freshly graduated, however I'm stuck finding my first DE job. I've tried to apply for many possible SQL/Python-related jobs, but it seems like no one is going to reply to my applications. The requirements for a freshly graduated are seemingly a lot. I've applied for many intern positions also (that means I'm willing to work at least 2 jobs, 1 for DE and 1 for making a living while gaining DE experiences). Furthermore, the market for which I'm currently seeking my first job is in Vietnam, which made I think that my English would be considered to be a fairly strong advantage. But, apparently, it's not. Do you have any recommendation for personal projects to put on my CV? I'm looking to gain more knowledge on building data pipelines, ETL, etc. I'm advanced in SQL, intermediate in Python and other languages such as Java, C++, web languages. I'm also familiar with data modelling, NoSQL, data visualizing (Power BI) and server-side data processing (NodeJS and PHP). How could I utilize the mentioned skills and tools? I do really need help now!
@WisomofHal
@WisomofHal Жыл бұрын
I don’t know the market in Vietnamese, but I couldn’t imagine the competition being any less fierce than the competition in the US. I asked my mentor this question, his advice was to: 1. BUILD projects 2. Contribute to open source 3. Become more visible (blogging about popular technologies that many people use and will need help tutorial help with, blog about niche technologies you know that don’t have much resources, or blog about your own product and challenges you’ve faced, share those challenges because others might run into them). I wish you all the best, my friend!
@sherwing9054
@sherwing9054 10 ай бұрын
There's too much supply of python DEs, what companies lack are framework/enterprise skilled DEs. Like example, they want to find people that excel in Talend, DataStage, Teradata, Informatica, Snowflake, to name a few. To land a job, use actual tools, not just python. For example, I only knew mysql back then, but landed an SSIS/SSAS/SSRS job. And with that, I landed an informatica and snowflake job. Now I get frequent calls for tools like DataStage and Talend which I don't use, but since I already have exp in other frameworks, they got no choice.
@datpham9526
@datpham9526 4 ай бұрын
Hu đọc được cmt của a em cũng thấy hơi hoang mang vì tình trạng tìm việc của DE fresher ở việt nam , a cho em chút lời khuyên được không ạ
@sickair45
@sickair45 4 ай бұрын
now in 2024 would you recommend going to school for a masters in data science/analyst or bachelor's in computer science?
@SeattleDataGuy
@SeattleDataGuy 3 ай бұрын
Hmm, I have mixed feelings about masters in data science and computer science because i assume its better to just do..but i don't have a masters so maybe someone else would find it useful
@rafayowais480
@rafayowais480 Жыл бұрын
Data Engineer vs Backend Developer in future?
@idkwhy77
@idkwhy77 Жыл бұрын
data engineer.
@WisomofHal
@WisomofHal Жыл бұрын
Backend developer is slowly becoming data engineer. Or is it other way ‘round?
@coder7078
@coder7078 9 ай бұрын
@@idkwhy77why ? Any specific reason ?
@MeLLiFLuOuSViDeOs
@MeLLiFLuOuSViDeOs Жыл бұрын
The question is not that will humans will be required or not... The question is will they pay you 300K USD per year for something that the AI does completely and you are just being specific and looking over it? Humans will have the Job but those making 300k will be reduced to 100K
@WisomofHal
@WisomofHal Жыл бұрын
Hmm. Good point, but have you thought of other alternatives. If AI can do the work, it’ll be more productive and produce more value (more money). Why reduce the salary of the human if more value is being produced? Like you said, human still needs to monitor: if no human is willing to monitor then what’s the use of AI that can have many mistakes? Therefore human can, still, demand very good pay. It will still take deep knowledge to know what to look for, mistakes, corrections, additions and so on. We reduce human labor. Also, human labor has always been added to any work ticket, the cost of that labor. If we are making labor cheap, this can also make other aspects of life cheap while producing more output. As computation power and storage becomes less expensive. I believe the overall goal is to give people more time to do other things, things we actually enjoy, we spend so much time at work. Wouldn’t you want to be with family, reading a book, skiing, or playing music? Basically, it’s why we wanted slaves in the first place, only exchanged for AI. Now we’re talking ethics… it’s a complicated chat, my friend. The idea of AI is still young; imagine a world with AI powered robots who can build without rest, who can clean without breaks, who can drive you from A to B, who can give you all the answers, cool your meal, read your kid a book. Would the cost of new construction be more or less expensive and who’s incurring the cost if we humans are not working? If robots can construct non-stop, where will all the material come from. Say construction becomes so cheap, countries decide to build homes for everyone. Is that feasible? Now, we’re talking policy. If we go another route, your route, and even highly paid jobs are now paying nickels and dimes while less paid jobs have already been automated to AI/Robots and we’re still expected to pay these enormous bills then how do we pay the bills? Or will government step in and develop more policies. It’s a very sticky situation. Now, what is the value of money? If money is no longer the god, then what becomes god for sheep? AI, a new god, or do people go back to spend more time with their one true God? Just something to think about. In my opinion, I don’t see salaries going down in our lifetime.
@MeLLiFLuOuSViDeOs
@MeLLiFLuOuSViDeOs Жыл бұрын
@@WisomofHal That cannot work with capitalist system... And is polar opposite of current capitalist system
@WisomofHal
@WisomofHal Жыл бұрын
@@MeLLiFLuOuSViDeOs That’s my point exactly 🙂 Don’t you think AI will relate a world like this? A world where we do less, if anything at all? Where monetary value is significantly less important or not important at all. The assumption is AI will automate and takeover most, if not all, jobs. Wouldn’t the next logical step be that government will morph into something completely different and have to establish more and more policies? I don’t know. There seems to be multiple ways this whole AI thing can go. I guess, we’ll just have to wait and see. Why would I have a human build my deck when government begins to produce machines that can do it perfectly in 5% of the time it takes a human to. What are the ramifications of government becoming more powerful? Why is it that, in capitalist societies, the idea of socialism and socialist policies are appealing to more people, how is that leaders that promote these policies in the most capitalist societies are being elected as leaders and what’s the effect on society? Why have countries in the West promoted, and some implementing, less working days (4 day weeks, 10 hour work days) and will 10 hrs a day slowly decrease back to 8? Officially reducing productivity? Or are we more productive because of AI and automation?
@dekippiesip
@dekippiesip 7 ай бұрын
​@WisomofHal your market salary isn't really determined by your added value, but by the factors of supply & demand on the labor market. It is profitable to hire you if and only if your market salary is lower than your added value, as with all products. Added value is, therefore, more like a salary ceiling. It can be the case that new technology enables you to be more productive and simultaneously reduces the skill floor of anyone wanting to do your job. In that case your market value actually goes down even though your added value goes up. We associate added value and productivity with higher salaries, because often if a machine tips over 3 out of 4 workers, the remaining 1/4 has to do less manual work and needs to work at a higher abstraction level. THAT translates into more salary, not the added value in itself.
@collinsnnadozie5092
@collinsnnadozie5092 Жыл бұрын
I'm still wondering why a programmer will create an AI to take away our jobs
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
Because the business who creates that tech will become very rich 😅
@collinsnnadozie5092
@collinsnnadozie5092 Жыл бұрын
@@SeattleDataGuy indeed, but at the expense of taking away people's career 😭
@Zuranthus
@Zuranthus Жыл бұрын
@@collinsnnadozie5092 google "luddite"
@collinsnnadozie5092
@collinsnnadozie5092 Жыл бұрын
@@Zuranthus Are you the developer of AI Chat GPT ?
@Zuranthus
@Zuranthus Жыл бұрын
@@collinsnnadozie5092 no, i'm just your average day rational individual that understands advancements will be made and you will find some other job to do. or do you propose we all go back to tilling the land and raising barns? cause you know, that would make sure there's plenty of work to go around for everyone
@Dheeraj7998
@Dheeraj7998 Жыл бұрын
Hi Ben, I'm a entry level into IT industry. Im more interested towards the field of data. Im confused to go with machine learning or data engineering.. please suggest . Thanks.
@gavinkalaher7314
@gavinkalaher7314 Жыл бұрын
entry level = data analyst. Microsoft Certified: Power BI Data Analyst Associate is the best course.
@Dheeraj7998
@Dheeraj7998 Жыл бұрын
@@gavinkalaher7314 thanks
@janbecker4744
@janbecker4744 Жыл бұрын
👍🏻
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
😀
@SaveThatMoney411
@SaveThatMoney411 Жыл бұрын
Please make a video on why you left Facebook. 😢
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
Have you seen this video? kzfaq.info/get/bejne/jNqehNleqLTDfnk.html
@WisomofHal
@WisomofHal Жыл бұрын
Love your name. 💚
@SaveThatMoney411
@SaveThatMoney411 Жыл бұрын
@@SeattleDataGuy Thank you. You don’t discuss why you left in that video though…
@SaveThatMoney411
@SaveThatMoney411 Жыл бұрын
@@WisomofHal Thanks. Someone called me aggressively NPC.
@WisomofHal
@WisomofHal Жыл бұрын
@@SaveThatMoney411 Was there any reasoning?
@sharad3877
@sharad3877 Жыл бұрын
your linkedIN?
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
I am! You can look up Seattle Data Guy!
@TomT-ds9vn
@TomT-ds9vn 4 ай бұрын
It will shrink data engineering teams.
@Drakvod
@Drakvod Жыл бұрын
If many of the things that a data engineer was in charge of are now becoming a commodity, there might still be work, but paid significantly less.
@SeattleDataGuy
@SeattleDataGuy Жыл бұрын
I don't know if that's true. In many cases, as we automate work, it increased productivity which increased salaries. If AI acts as a tool, then I imagine it will be the same. Of course the ICs will see a decent jump in their pay but the biggest winner is always the corporations.
@mahmoudelleithy6908
@mahmoudelleithy6908 8 ай бұрын
Omg. All this time and didn't answer the question !! 👎👎
The Harsh Reality of Being a Data Engineer
14:21
Jash Radia
Рет қаралды 212 М.
Cat story: from hate to love! 😻 #cat #cute #kitten
00:40
Stocat
Рет қаралды 14 МЛН
NO NO NO YES! (50 MLN SUBSCRIBERS CHALLENGE!) #shorts
00:26
PANDA BOI
Рет қаралды 102 МЛН
Why I Left Data Science - And Picked Data Engineering Instead
7:14
Seattle Data Guy
Рет қаралды 15 М.
The Harsh Reality of Becoming Data Engineer
36:02
Yudi J
Рет қаралды 37 М.
The Harsh Reality of Being a Data Analyst
7:39
Sundas Khalid
Рет қаралды 527 М.
Why EVERYONE Is Struggling to Become a Data Analyst
8:03
Learn with Lukas
Рет қаралды 249 М.
Why NOT to become a Data Engineer
13:45
Karolina Sowinska
Рет қаралды 58 М.
Что если #тест #обзор
1:00
Orion
Рет қаралды 2,2 МЛН
Смотри до конца 😻💔
0:44
mafo fashion
Рет қаралды 3,3 МЛН
Азат - ол менің бизснесім  І АСАУ І 6 серия
28:42
ПАРАЗИТОВ МНОГО, НО ОН ОДИН!❤❤❤
1:00
Chapitosiki
Рет қаралды 962 М.
Самолёт Падает! Но Осталось 2 Парашюта... @NutshellAnimations
0:35
Глеб Рандалайнен
Рет қаралды 3,2 МЛН