AWS: How to use AWS Glue ETL to convert CSV to Parquet

AWS: How to use AWS Glue ETL to convert CSV to Parquet - Tutorial

Рет қаралды 13,884

4 жыл бұрын

** FREE AWS Professional Consultation (United Kingdom) available here: firemind.io/free-consultation/ **
Video: AWS Glue is a managed ETL platform, and can be used for storing your data Schemas, as well as ETL tasks in Python, or Java. A common ETL use case is to convert CSV files to the much more efficient Parquet files. Glue makes this easy, and can automatically handle this transition from your objects stored in S3.
Learning Objectives:
- Updating IAM policies to allow access to new prefixes in S3
- Creating a AWS Glue ETL job
- Configuring a AWS Glue ETL job to convert to Parquet Format
- Querying Parquet files using Amazon Athena
***
Full AWS Playlist:
• AWS: The Basics
Find out more about Firemind:
www.firemind.io
#AWS

Пікірлер: 11

@puneetsharma3032 3 жыл бұрын

Very Nice. Earlier, I was keep getting error but after watching your video its resolved. Thanks :)

@firemind 3 жыл бұрын

We're glad to hear it helped you Puneet.

@akhileshmahajan9626 2 жыл бұрын

Hey Firemind - thanks for this video!

@firemind 2 жыл бұрын

You're very welcome Akhilesh!

@AbhishekDubey-td4ks Жыл бұрын

Thank but I want to convert multiple files CSV to parquet from the same folder target s3 output s3 pls help me out

@mattph76b 3 жыл бұрын

good video, short and to the point - quick question though, I noticed your timestamp fields were set to string datatypes - have you had any success converting them to timestamp? thank you

@firemind 3 жыл бұрын

Hi, thanks for the feedback! We had no issues with setting the field to DATETIME - which should be as simple as modifying it in the transformation script - this then outputted with the DATETIME format, however for this dataset we did not try with the TIMESTAMP field.