Practical Projects to Learn Data Engineering On AWS

Рет қаралды 44,323

Күн бұрын

This video comes with some practical project ideas to learn data engineering on aws so you can handle a variety of real-world challenges. This includes architecture diagrams of:
1. Event Driven data processing with AWS Lambda,SNS,SQS
2. Batch Processing Data with AWS Glue
3. Build a Dashboard with AWS Quicksights on a Data Lake powered by AWS Athena.
4. DMS To Migrate Data from a Database to S3
5. Process streaming data from Amazon Kinesis Stream
Diagrams and article link:
aws.plainenglish.io/most-comm...
Buy me a coffee : www.buymeacoffee.com/dataengu
Timelines
00:00 Introduction
00:31 Event-driven Data Processing
02:59 Batch Processing with AWS Glue
04:40 Serverless Dashboard
05:23 Database Migration
06:29 Process Streaming Data
#aws #dataengineering

Пікірлер: 60

@sylarguo7741 Ай бұрын

I'm a newcomer to AWS a data engineer, Your channels really help me! Appreciate it!

@Theoldmad Жыл бұрын

Excellent video ! I wish I found this earlier.

@viciesone Жыл бұрын

this is the best channel on KZfaq! thank you! explanations are to the point and easy to grasp.

@DataEngUncomplicated Жыл бұрын

Hi Rodrigo, wow thank you for your kind words. I'm glad you found this video helpful! thanks for watching.

@shivthedev 10 ай бұрын

please do a comprehensive course on this! I really love your content 🙂

@mehmetkaya4330 Жыл бұрын

Thank you for great video!

@khandoor7228 Жыл бұрын

Great video, clear concise and valuable!!!

@DataEngUncomplicated Жыл бұрын

Thank much appreciated!

@trk1139 Жыл бұрын

Awesome !.You cleared all of my doubts in this video...Thanks a lot.

@DataEngUncomplicated Жыл бұрын

You're welcome! I'm glad it was helpful.

@momijigz 4 ай бұрын

What an amazing video. Solved an issue I was facing in my own implentation of a data ingestion pipeline. You got a sub today.

@DataEngUncomplicated 4 ай бұрын

Thanks for subscribing!

@mikethomson1872 Жыл бұрын

Yiour videos are so consise and easy to follow. Glad I found you

@DataEngUncomplicated Жыл бұрын

Thanks Mike, much appreciated!

@jasper5016 9 ай бұрын

Your videos are fanstastic. Please bring them more.

@DataEngUncomplicated 7 ай бұрын

Thanks Jasper!

@joudawad1042 2 ай бұрын

one of the best channels on youtube for data engineering! great content, keep the great work

@DataEngUncomplicated 2 ай бұрын

Wow, thanks for the kind words! It made my day 😄

@sjvr1628 Жыл бұрын

Hi Tutor, Since I don't know your name, Can you please do a video on end-to-end implementation with sample project to use the AWS Data services like S3, Glue, Kinesis, Redshift, Data pipeline, Crawlers etc. for both batch and streaming data ingestion.

@gudguy1a 4 ай бұрын

@sjvr1628 My friend, his name is: Adriano Nicolucci capisce

@dzhuwao 2 жыл бұрын

Great video. Well presented.

@DataEngUncomplicated 2 жыл бұрын

Thank you kindly!

@onuabah3001 Ай бұрын

You're an excellent teacher, keep it up...subscribing now

@DataEngUncomplicated Ай бұрын

Thanks for the kind words!

@the_c0mm0n_man Жыл бұрын

thankyou for providing this video

@DataEngUncomplicated Жыл бұрын

You're welcome!

@jeremynx Жыл бұрын

thanks, everything is clear, i like your voice

@DataEngUncomplicated Жыл бұрын

Thanks Saltanat!

@DeepakSingh-of2xm 2 ай бұрын

Can you please make a video showing implementation of event driven architecture using event bridge, sns, sqs and lambda? Thank you!

@biswajitpattanaik3157 10 ай бұрын

The theory is very well explained. Could you please make a video showcasing the implementation practically, with a small example.

@DataEngUncomplicated 10 ай бұрын

Thanks, This is a good idea. I will think about what I can do!

@SreeDevi-ee1tj Жыл бұрын

Looking for more examples with practical session

@shreygupta1253 Жыл бұрын

Pretty neat and easily explained !! ♥️

@DataEngUncomplicated Жыл бұрын

Thanks Shrey!

@bohdan134 Жыл бұрын

Hi thanks for these project ideas. Would like to ask, can you advice any public data sources/providers for data engineer practice?

@DataEngUncomplicated Жыл бұрын

You're welcome! I use www.mockaroo.com/ to generate my sample data. It gives full control of the column names and data types for randomly generated data.

@harvestingdata Жыл бұрын

Well presented. For the first pattern, what's the advantage of using EventBridge and implications of skipping it?

@DataEngUncomplicated Жыл бұрын

Thanks! Well one benefit of Event bridge is event bridge rules is you can have many different AWS events used as the source to start your job. But with glue it is more limited. If you just want to run it on a schedule than glue orchestration should do the job fine!

@Zingaro27 Жыл бұрын

@@DataEngUncomplicated ace edx

@groundingtiming 6 ай бұрын

will you please make an end2end production grade code that will also combine terraform?

@DataEngUncomplicated 5 ай бұрын

Hi, I actually just created a video on terraform with glue jobs. I think you will find this helpful: kzfaq.info/get/bejne/eb-KZr2bvcq1gGw.html

@redhawasssim5111 10 ай бұрын

well explained, is there an implementation video ?

@DataEngUncomplicated 10 ай бұрын

No I don't have one yet! but this might make a good follow up video. Thanks!

@andrewting3081 Жыл бұрын

What percentage of real industry work would you say fall into these 4 categories? Also, thanks again for these sweet sweet videos man. Super concise and informative.

@DataEngUncomplicated Жыл бұрын

Hi Andrew, that's a good question! I don't have the data to answer this with any certainty but I can speak to my own personal experience. From what I see, the majority are batch processing workflows. However, there is a growing trend towards more event based architectures and streaming architectures as companies mature in the cloud and need real-time analytics grows.

@paulaganbi5236 8 ай бұрын

Hey Adriano, thank you for your content. I am a data engineer and I will take the aws data analytics specialty exam next month. Please what materials do you best advise me to use to study for the exam? Thank you.

@DataEngUncomplicated 8 ай бұрын

Hi Paul, your welcome. I bought Stephane maarek's Udemy course for this exam and I felt it did a great job to prepare me for the exam. I recommend it.

@the_c0mm0n_man Жыл бұрын

hi can you do a video on working with the incremental data stored in s3

@DataEngUncomplicated Жыл бұрын

Hey, can you provide more details what you are looking for?

@justevil9743 2 жыл бұрын

Loving your content. Do you provide any courses on udemy for projects like this ?

@DataEngUncomplicated 2 жыл бұрын

Thanks so much! Unfortunately I don't have any courses on Udemy, I would love to make one in the future though! I'm currently just focusing my extra time outside of my full-time job on KZfaq videos for now.

@andrewting3081 Жыл бұрын

@@DataEngUncomplicated Youre doing friggin great so far -- Please keep it up! At the same time, I am still rooting for you to create a full course sometime in the future.

@hritiksharma7154 Жыл бұрын

Hii , could u please make an end to end aws data engineer project . It would a great help for everyone .

@DataEngUncomplicated Жыл бұрын

Hi Hrithik, what pattern do you think would be most useful? It might be a challenge to cram it into one video given there are many services to learn and know about. The video could be several hours.

@hritiksharma7154 Жыл бұрын

@@DataEngUncomplicated You can make a series of video of 20 to 30 minutes. Small small steps in each video.

@DataEngUncomplicated Жыл бұрын

This is a good idea! This would be more manageable as a video series. Thanks!

@krishnadebata5403 Жыл бұрын

Can u please help with these project videos. I would love to connect if u plan for a project day luve on youtube

@DataEngUncomplicated Жыл бұрын

Thanks Krishna, I'm slowly working on technical tutorials for these architecture patterns. My next batch of videos is using AWS Glue for reading, transforming and writing data.

@ashwinisolanki2801 4 ай бұрын

I loved your videos and subscribed them instantly. But one think I didn''t like is the subscribe notification that comes in between and distract the whole focus and then I need to play that particular part again and again. You ask for subscribing in the middle of the video, nobody would pause and will go to subscribe but instead ask for subscribing in the end so that people will remember to subscribe in the end. Thank you.

@DataEngUncomplicated 4 ай бұрын

Thanks your feedback and subscribing! In my newer videos I have added it to the end or at least it's not as distracting. Unfortunately KZfaq doesn't let you edit existing videos without uploading the entire thing again.