Building a Data Lake on AWS with AWS Glue, Glue Studio, Amazon Athena, and S3

Рет қаралды 21,111

2 жыл бұрын

Build a simple Data Lake on AWS using a combination of services, including AWS Glue Data Catalog, AWS Glue Crawlers, AWS Glue Jobs, AWS Glue Studio, Amazon Athena, and Amazon S3.
All open-source files on GitHub: github.com/garystafford/ticki....
This video represents my own viewpoints and not of my employer, Amazon Web Services (AWS). All product names, logos, and brands are the property of their respective owners.
📣 Please subscribe to my KZfaq channel for future videos.

Пікірлер: 17

@donaldmahaya2689 Жыл бұрын

Excellent hands on video covering building a datalake in AWS.

@hareepjoshi Жыл бұрын

Hey Gary, found you through your medium articles and now I'm watching your youtube videos. Excellent content!

@mohammedgt8102 Жыл бұрын

Gary, that was just perfect! Fast and straight to the point. 👏👏. Thank you!

@DarioRomeroDeveloper Жыл бұрын

I always enjoyed the 'down to earth' practical business cases from @Gary Stafford. This one is really good. Thanks for sharing. I've learned a lot with this tutorial.

@rixonmathew 2 жыл бұрын

Great video. Many complex concepts have been explained using simple language and examples.

@gatorpika Жыл бұрын

Really great presentation, thanks for that.

@_truthful_q_ 2 жыл бұрын

This was excellent 👏 Top marks my man!

@swapnilbops1486 10 ай бұрын

Very Useful 🌟

@RashaadFontenot 2 жыл бұрын

Great video

@freakinmonkey85 2 жыл бұрын

Thanks a lot! This video deserves more views. It’s the first concise to the point video I’ve found where actual data and actual results are shown end to end. I have a question I hope you could answer. How would you handle data that changes. E.g. in a couple of days a customer cancels a ticket with id 1234, concert id 321. Now the calculations needs to take this into account, no?

@GaryStafford 2 жыл бұрын

kzfaq.info/get/bejne/aJuDp8Sk0qm6g6s.html

@ericpho4060 2 жыл бұрын

Hi, thanks for this video. Very interesting ! Would be curious to know how would you handle incremental updates of this aggregated tables through Athena SQL queries ? with that architecture, would you run full calculation for entire set of data all over at each execution ?

@GaryStafford 2 жыл бұрын

Here is the documentation on the current level of integration possible with Amazon Athena (docs.aws.amazon.com/athena/latest/ug/querying-hudi.html): "Currently, Athena supports snapshot queries and read optimized queries, but not incremental queries. On MoR tables, all data exposed to read optimized queries are compacted. This provides good performance but does not include the latest delta commits. Snapshot queries contain the freshest data but incur some computational overhead, which makes these queries less performant."