What Is A Data Pipeline - Data Engineering 101 (FT. Alexey from

Рет қаралды 25,191

Күн бұрын

Data pipeline
A term that gets thrown around a lot in the data space.
Does it involve streaming, batch, Ipaas or all of the above?
Guests in this video include
Alexey Grigorev - / agrigorev
Andrew Madonna - / andrewmadonna
Colin Shapiro - / collin-shapiro-54673766
Also, if you're looking for a new data orchestration solution, check out mage.ai - www.mage.ai/
If you enjoyed this video, check out some of my other top videos.
Top Courses To Become A Data Engineer In 2022
• Top Courses To Become ...
What Is The Modern Data Stack - Intro To Data Infrastructure Part 1
• What Is The Modern Dat...
If you would like to learn more about data engineering, then check out Googles GCP certificate
bit.ly/3NQVn7V
If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here.
seattledataguy.substack.com/
Or check out my blog
www.theseattledataguy.com/
And if you want to support the channel, then you can become a paid member of my newsletter
seattledataguy.substack.com/s...
Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio
_____________________________________________________________
Subscribe: / @seattledataguy
_____________________________________________________________
About me:
I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data.
*I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.

Пікірлер: 28

@SeattleDataGuy Жыл бұрын

If you guys want to learn more about data engineering, then sign up for my newsletter here seattledataguy.substack.com/ or join the discord here discord.gg/2yRJq7Eg3k

@sng9x Жыл бұрын

Great content, simple enough to understand and reflect on as to learn how the same knowledge can be interpreted by different people.

@SeattleDataGuy Жыл бұрын

Glad you enjoyed it!

@NaveenKumar-cn2pc Жыл бұрын

Wow As an Informatica ETL developer myself planning to switch to Data Engineering I was just trying to understand this concept of Data pipeline, this is as much more simplified explaination although would request you a video on what kind of pipeline would you suggest based on different business scenario that you mentioned as we have some ETL or I say ELT tool build native to cloud such as Matillion ,IICS or manually coding the entire thing.

@SeattleDataGuy Жыл бұрын

Glad to hear this was easy to understand! That is my goal. I will put that video on the list

@karan7843 Жыл бұрын

Good to see Alexy🙂

@SeattleDataGuy Жыл бұрын

yes I loved that he said yes to joining in!

@Cybersader 7 ай бұрын

I do security log management using Cribl and everything youve said is everythinf ive been thinking but in a different language 😂 Great to hear from someone else. We dont have data engineers where I work. Just me

@SeattleDataGuy 6 ай бұрын

Sometimes that's how it goes. One person doing all of it. Good luck!

@christiansanchez9880 Жыл бұрын

Good job on mentioning that there are too many tools. I’m a Business Systems Analyst right now who is trying to learn data engineering and there seems to be too much to learn.

@SeattleDataGuy Жыл бұрын

For real...its never ending

@alphar85 Жыл бұрын

I am here lookong to switch from analytics and digital lead to data engineer as i see myself doing it alongside software engineering (although they might sound different). So as i lean more towards SQL rather than Python. I got trained on SQL and SSIS and i never knew that was part of data engineering. In SSIS, you really learn a lot about the ETL process. This video really taught me to revisit the SSIS as there is a learn in the system and plus it is MS product.

@SeattleDataGuy Жыл бұрын

Awesome! I would still say learning some python is a good idea!

@alphar85 Жыл бұрын

@@SeattleDataGuy thank you very much mate

@SeattleDataGuy Жыл бұрын

@@alphar85 You're welcome!

@arahso Жыл бұрын

A lot of this content is geared for beginners and for folks trying to break into the role. What about advanced topics or in-depth technical debriefs?

@SeattleDataGuy Жыл бұрын

What are some specific topics you are looking into?

@bryanneremita2860 Жыл бұрын

Being someone who wants to migrate from a data analytics role into a data engineering role, what tools would you guys reccommend I start out with ? From the ETL/ELT operation to data warehousing and maybe post that some work on cloud tech. Any help would be appreciated.

@SeattleDataGuy Жыл бұрын

If your SQL, Python and data warehousing is solid. Then you can start working on developing some basic pipelines and building core tables in something like snowflake, bigquery or databricks.

@bryanneremita2860 Жыл бұрын

@@SeattleDataGuy thanks for the quick response. My sql and python is alright but i need to get Into warehousing and etl. Any tips on that?

@SeattleDataGuy Жыл бұрын

@@bryanneremita2860 Have you read kimball and inmon's books? Great for understanding warehousing as well as high level etl concepts. Oh and the fundamentals of data engineering too www.oreilly.com/library/view/fundamentals-of-data/9781098108298/

@bryanneremita2860 Жыл бұрын

@@SeattleDataGuy thanks a lot. This will be helpful.

@Anant_Maheshwari Жыл бұрын

apart from being excellent in sql what else one should know if a datawarehouse guy wants to become a data guy, not the seattle one but an engineer one.