AWS Data Engineering Tutorial for Beginners [FULL COURSE in 90 mins]

Рет қаралды 89,438

Күн бұрын

Course Material Links
📁 github.com/johnny-chivers/aws...
🖥️ aws-dataengineering-day.works...
❔www.thequestionbank.io
ℹ️ johnnychivers.co.uk
☕ www.buymeacoffee.com/johnnych...
00:00 - Intro
00:48 - Downloading Required Resources
01:06 - What The Course Will Cover
01:12 - Prerequisites
02:34 - What Is Data Engineering
04:17 - AWS Kinesis Theory
08:32 - AWS Kinesis Data Streams Theory
13:29 - AWS Kinesis Firehose Theory
15:35 - AWS Kinesis Data Analytics Theory
16:53 - Realtime Streaming Kinesis Lab
39:33 - AWS Database Migration Service Theory
44:13 - AWS DMS Lab
01:05:03 - AWS Glue Theory
01:12:58 - AWS Glue Lab
01:30:50 - Outro
In this free AWS Data Engineering course we take a deep dive into the services provided by AWS to help us with out with our everyday data engineering needs. The course is created for both AWS beginners and seasoned pro’s alike. I have loosely based this course on the AWS Data Engineering Immersion Day. In fact the Kinesis lab is straight out of the Immersion days booklet. The rest of the labs take the ideas from the booklet, but translates them so we can build our own use cases at home without being in an instructor led environment. During the course we cover AWS Kinesis, AWS DMS and AWS Glue from both a theory and practical perspective. All slides and code are available on GitHub - link above. I will also make all this information free via my website - again link above.
😎 About me
I have spent the last decade being immersed in the world of big data working as a consultant for some the globe's biggest companies.My journey into the world of data was not the most conventional. I started my career working as performance analyst in professional sport at the top level's of both rugby and football. I then transitioned into a career in data and computing. This journey culminated in the study of a Masters degree in Software development. Alongside many a professional certification in AWS and MS SQL Server.

Пікірлер: 76

@JohnnyChivers 2 жыл бұрын

FYI-> You can not leave the SMS field in the Kinesis template blank. You must enter some numbers or the stack creation will fail.

@hugoczerniawski166 Жыл бұрын

It's not often to find someone on YT who is explaining it so meticulously and simple at the same time! Huge shoutout to Mr.Chivers 10/10!!!

@gustavofernandez5914 2 жыл бұрын

fantastic! Thank you so much for this tutorial Johnny.

@JohnnyChivers 2 жыл бұрын

Thanks for watching.

@kaedien 2 жыл бұрын

fantastic work Johnny, just binged the whole thing! 😁

@JohnnyChivers 2 жыл бұрын

Thanks kaedien!

@The_Bold_Statement 2 жыл бұрын

Long awaited video. Thanks for the upload

@JohnnyChivers 2 жыл бұрын

Hopefully worth the wait.

@davidbaillieul7601 Жыл бұрын

Really enjoy your videos Johnny. You can make what could be dry content interesting.

@JohnnyChivers Жыл бұрын

Thanks David!

@jirasaki6672 2 жыл бұрын

Thank you Johnny... Its helps me a lot for project building and prepare for specialty exam.

@JohnnyChivers 2 жыл бұрын

Thanks for watching!

@ElboxD 2 жыл бұрын

Exactly what I was looking for!

@JohnnyChivers 2 жыл бұрын

Thanks for watching German. I appreciate it.

@lifewithchels 2 жыл бұрын

Lifesaver. Those slides are comprehensive. Liked and Subbed!!!! Let's GOOOO

@JohnnyChivers 2 жыл бұрын

Glad it was helpful! Thanks for watching.

@Shubh22 Жыл бұрын

Thanks man, the Glue lab demo helped me a lot in understanding it's usage.

@JohnnyChivers Жыл бұрын

Thanks for watching

@brandonjapol9853 2 жыл бұрын

I gotta say, between yourself and Stephane Maarek, the accents for learning AWS have been absolutely spectacular. Great content, thank you so much!

@JohnnyChivers 2 жыл бұрын

Thanks very much for watching l!

@Stonk5331 Жыл бұрын

You forget to mention what services we should delete at the end to avoid costs^^ Beside that, it was a wonderful course and imo set a new standard for other similar youtube videos

@mehmetkaya4330 2 жыл бұрын

Thank you so much for the great tutorial!!

@JohnnyChivers 2 жыл бұрын

Thanks for watching!

@redrum4486 2 жыл бұрын

Awesome video bro! Took me like 7 hours while stopping and reading other things and setting stuff up... top video!!!

@JohnnyChivers 2 жыл бұрын

Thanks for watching. Glad it was useful.

@VyNguyen-xy3il Жыл бұрын

Thank you Johnny, totally helpful to me!!!

@JohnnyChivers Жыл бұрын

Thanks for watching!

@umairqamar2672 Жыл бұрын

You are amazing, dude

@alanflynn3612 2 жыл бұрын

Johnny, you the man

@JohnnyChivers 2 жыл бұрын

Thanks as always Alan.

@tunggw7763 2 жыл бұрын

Thanks so much. Sharing good knowledge for free, not a lot are willing to do that. :)

@JohnnyChivers 2 жыл бұрын

Thanks for watching!

@shibadhamija4305 2 жыл бұрын

This is THE BEST !!!!!

@JohnnyChivers 2 жыл бұрын

Thnaks for watching!

@DiaaKasem0 2 жыл бұрын

Super, And Thank you for sharing..

@JohnnyChivers 2 жыл бұрын

Thanks for watching

@valterdiasrosa4117 2 жыл бұрын

Tks Johnny!

@JohnnyChivers 2 жыл бұрын

Thanks for watching Valter.

@ryandavies4894 2 жыл бұрын

Loving the course content very unique. One thing that would be interesting would be how to do CI/CD in AWS.

@JohnnyChivers 2 жыл бұрын

Definitely something I can cover.

@kpicsoffice4246 2 жыл бұрын

@@JohnnyChivers Seconding this request. It will really help to set up a CICD pipeline & promote code changes following good software practices for data engineering.

@yearsofai Жыл бұрын

@@JohnnyChivers Any chance you created this sort of content and I missed it? Thank you for the wonderfully helpful content!

@yogeshkumarshankariya642 Жыл бұрын

Amazing Johnny😊. Kindly request you to make whole video series on the AWS ML Speciality exam.

@vivekjoshi3769 Жыл бұрын

Are you working in India on cloud technologies?

@scar2080 2 жыл бұрын

Fantastic ,😀❤️❤️❤️❤️❤️.. sending it from india ..great work

@JohnnyChivers 2 жыл бұрын

Thanks Ashlin.

@asmaamahmoud7465 Жыл бұрын

thank you

@andynelson2340 2 жыл бұрын

Awesome 😎

@JohnnyChivers 2 жыл бұрын

Thanks Andy.

@reniermeyer8274 2 жыл бұрын

Liked and subscribed

@JohnnyChivers 2 жыл бұрын

Thanks for watching

@ashraf950901 11 ай бұрын

Hi Mr.Chivers, I cant create application using SQL runtime. The option only shows Apache Flink. Help.

@Bee-ib1pb Жыл бұрын

Hello, please can you create a tutorial on how to migrate data from redshift to dynamodb from python code (using cloudformation)

@jacobide2647 Жыл бұрын

Great Video. Minor update that's cause me some headaches. Since DMS 3.4.7 has come out, you have to onfigure VPC endpoints. The docs aren't super clear for a beginner like me. I am spinning down the insnace and going to redeploy with 3.4.6

@jacobide2647 Жыл бұрын

Didn't work with 3.4.5 either. I'll try again tomorrow.

@vivekjoshi3769 Жыл бұрын

@@jacobide2647 Did it work for you?

@John83118 5 ай бұрын

I'm enchanted by this content. I recently read a similar book, and I was truly enchanted. "Mastering AWS: A Software Engineers Guide" by Nathan Vale

@midhunrajaramanatha5311 2 жыл бұрын

Hi Johnny can you make video about AWS lambda function and its use case

@JohnnyChivers 2 жыл бұрын

Hi. Thanks for watching. Would you be interested in seeing how lambda can be used in the data space? Or more broadly? Lambda has massive scope within AWS.

@tomerenshtein 10 ай бұрын

@johnyChivers, this is a great video, unfortunately kinesis analytics was deprecated and now there is only Apache Flink so the setup process described is no longer relevant..

@ogunniransiji2686 2 жыл бұрын

I am new here, it will be helpful if we 'beginners' can be informed about the billing conditions. so we can prepare beforehand before practicing. I think that will go a long way

@JohnnyChivers 2 жыл бұрын

Hi Ogunniran, I think I do mention in the video how much it cost me to carry out the implementation on AWS. Off the top of my head it was less than 10 USD.

@mehmetkaya4330 2 жыл бұрын

@@JohnnyChivers yes you do!

@t-566 6 ай бұрын

At 09:00 - 09:45 you mention a solution with API gateway and lambda with authorisation to put data onto the Kinesis stream. You say there is a video about this on your channel, but I can't find it. Any chance anyone could share a link?

@ricardorodriguez4180 2 жыл бұрын

Wow, all this for free? Subbed.

@JohnnyChivers 2 жыл бұрын

Thanks for watching and the sub!

@IsaacHsuHe1028 Жыл бұрын

Hi Johnny, thanks for your courses. Really appreciate. That would be great if you can also provide the course about Sagemaker and Airflow? Thanks

@JohnnyChivers Жыл бұрын

Airflow has been on my video todo list for a while now!

@channuangadi7504 Жыл бұрын

AWS or someone should bring videos on kinesis Video stream Because there is no one who explains it fully Hope you bring it in future

@Hi-uw5uz 2 жыл бұрын

can you make a aws lambda seri? tks u so much

@JohnnyChivers 2 жыл бұрын

Hey Hi. I certainly can. What would like covered? There is a beginner's guide video on the channel already, but it's just a quick introduction to lambda.

@Hi-uw5uz 2 жыл бұрын

@@JohnnyChivers yeah i see. But i still hope you will make a full course about it

@laszlokozma7710 Жыл бұрын

Thanks for the great content Johhny! I think the assignment of the destination Lambda function is missing from the video. It should be around here: kzfaq.info/get/bejne/mdGBasdnyMeWemQ.html Or did I miss something? :)

@tech-n-data Жыл бұрын

As of June 2023 you must select SQL Application (legacy) to create the SQL application. AWS is moving away from it.

@weblakey Жыл бұрын

hmmmm - I click the link in the lab instructions to get to the "SQL applications (legacy)" page, but the Create SQL Legacy button is disabled. Looks like you cannot create this any longer - deprecated.