Deploying a Glue Job to AWS with Terraform: A Step-by-Step Tutorial

Рет қаралды 3,694

Күн бұрын

This video is a guide on how to deploy an AWS Glue Pyspark Job using Terraform. It covers the IAM role required to run your glue job, configuring a glue job in Terraform, and how to add variables that can be different depending on your aws environment.
Github Repo: github.com/AdrianoNicolucci/d...
From this Video's Sponsor:
Get one month of VPS FREE at hivelocityinc.net/48vyem1. Use code
DATAENG1 at checkout.
#Hivelocity #VPS #VPShosting #hivelocityhosting #ad
#collaboration #terraform #aws #awsglue
timeline
00:00 Overview
01:16 AWS Provider
04:19 IAM Role
06:24 Glue Job configuration
13:01 Terraform commands to deploy to AWS

Пікірлер: 9

@calvinfernandes1054 5 ай бұрын

Great video! It would be great if you cover Job bookmarks with Glue

@DataEngUncomplicated 5 ай бұрын

Thanks, What are you looking for the video to cover, just the concept at a high level?

@maheshkharawatkar4356 4 ай бұрын

Thank you for this wonderful video, The content is so useful, I have a question; Is it possible to create an aws glue job with json file instead of python file using the terraform?

@DataEngUncomplicated 4 ай бұрын

So glue jobs are written in python. If it's a spark job, you have an additional option of scala. I'm not sure what you mean by using a JSON file?

@alessandroariu6090 5 ай бұрын

Thank you! Great video, is it possible to also stop a job run and start again a job? I mean for a CI/CD process or it is necessary to use the cli

@DataEngUncomplicated 5 ай бұрын

You're welcome! What do you define as job run? A glue job run?

@alessandroariu6090 5 ай бұрын

Thanks for the reply, e.g. if I have a streaming job running and I want to deploy a new version of the script

@DataEngUncomplicated 5 ай бұрын

I think you will need to add a step in the ci/cd process to stop the current job if you can't wait for it to finish on its own. Calling the cli in the ci/cd pipeline is one way or having a Python script using boto3 is another

@alessandroariu6090 5 ай бұрын

I see, so it is not possible only with terraform. Thank you again