Everyone's Data Infrastructure Is A Mess - The Truth About Working As A Data Engineer

No video

Everyone's Data Infrastructure Is A Mess - The Truth About Working As A Data Engineer

Рет қаралды 7,574

Күн бұрын

Is everyone’s data a mess?
Recently, I came across a post in the data engineering subreddit that asked the question.
The answer is yes, but no.
As someone who has seen data infrastructure at FAANGs, Enterprises, start-ups, and every other company in between, all companies need to make some concessions that can build up and become messy over a long period of time.
So let’s discuss some of the causes of data infrastructure becoming messy and how some companies are trying to deal with it.
Also, I forgot to cover a very important topic!
That is all of the mess often starts at the data source.
You can read the fuller version of this topic here
seattledataguy.substack.com/p...
If you need consulting help, set up some time with me here -
calendly.com/seattledataguy/3...
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Пікірлер: 27

@SeattleDataGuy 8 ай бұрын

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

@GoWmaster27 Жыл бұрын

Based on the thumbnail, I really expected this to be a 3 second video where you just record yourself saying “yes.”

@SeattleDataGuy 11 ай бұрын

Hahaha - yes

@urip_zukoharjo 8 ай бұрын

3 secs is too long to spit yes, 0.01 sec is all you need to say "Уe"

@Yavin4 Жыл бұрын

There are other layers to this problem. E.g. regulatory compliance. Who can/cannot access the data? What and where can you access the data? Think GDPR. Other factors include external vs internal data. Is there a cost to accessing/collecting the data? Most companies are not even close to having good Data Governance fundamentals, and many of them may never meet a high standard given the constant turnover. The Data Engineer role will evolve into a greater one of overall Data Governance.

@edwardmitchell6581 Жыл бұрын

I find this unlikely. Data engineers tend to be highly technical order takers. It’s in there interest to use technologies that lead to high salaries. It’s not in their interest to have data quality 4 quarters from now. On top of that, data governance is a topic that never goes below C Suite at most companies. I’ve seen job requirements follow the trend you predict, but I think that’s just about technical knowledge of metadata rather than business skills.

@Yavin4 Жыл бұрын

@@edwardmitchell6581 You are describing the current state. I am talking about future state.

@richardduncan3403 7 ай бұрын

I have noticed that automation needs quite a bit of manual maintenance

@SeattleDataGuy 6 ай бұрын

hahaha...if you've ever had to backfill a table....

@sigmapi1989 Жыл бұрын

Ha soooo True! Everywhere I've worked its been a mess!

@SeattleDataGuy 11 ай бұрын

its just always a fight to try to bring it to some level of sanity

@SuperLOLABC Жыл бұрын

So with companies data governance being such a mess, would you say that the field of data engineering & governance still has a future for atleast a decade? Or will it all be automated since automation seems a huge part of Data Engineering already?

@edwardmitchell6581 Жыл бұрын

How can you automate data strategy or data management?

@SuperLOLABC Жыл бұрын

@@edwardmitchell6581 Today in a data engineering team of 10, about 1-2 people take care of data strategy and data management. The remaining 8 build and maintain the solution. Their main job is to automate the engineering solution. If data engineering can be automated sufficiently then the total amount of DEs required will go down.

@SeattleDataGuy 11 ай бұрын

There is plenty of work to do. I can't speak in decades, but 5 years, yeah probably

@ceejay1353 Жыл бұрын

For those who want to get into consulting, assuming you're starting from 0 exerpince, how many years of experince would you say is good before you can reasonable make a living off of consulting?

@KshitijPatil1 Жыл бұрын

Consulting is entertained mainly with the logic that someone with MORE experience than them is going to help solve an unsolvable problem. So if you have 0 experience, what in your opinion is it that you would be even offering to them?

@ceejay1353 Жыл бұрын

@@KshitijPatil1 I think k you missunderstood my question, I'm asking howany years is good to start consulting in general

@KshitijPatil1 Жыл бұрын

@@ceejay1353 My bad. So you're asking how many years does it take to make a living off of consulting gigs, should you leave your current job, right?

@ceejay1353 Жыл бұрын

@@KshitijPatil1 Yeah!

@KshitijPatil1 Жыл бұрын

@@ceejay1353 Got it. So my assumtions about getting into these types of career paths is that you already have your first 2-3 clients when you start. This means the people you've worked with, trust and respect your contribution are happpy to commit their company's dollars on a weekly/monthly basis. This helps you to a) anchor your price and b) provide references for your potential clients to get social proof from. The reason point a) is important is so that you know how much is the max you can earn per month, and deduce the number of clients you need to juggle. Point b) helps you to go on an aggressive client acquition excercise, because till you get your schedule packed, there's no financial upside to this excercise.