Thanks for the kind words! Yes actually this was going to be one of my next videos. How to deploy a glue job to terraform with terraform.

I don't know an elegant way to do this but you can go into the docker container and install the python libraries you need directly that way.

Hi, Can you explain how your approach is different than how I created the unit test in the video? If you design your functions to do one particular thing, it makes it much easier to write unit tests for it.

The approach is to be able to run the unit test without a glue environment, no docker image, pure local développement (my laptop). Mocking GlueContext and DybamicFrame. The tests would use the mocks of these instances so there's no interaction with AWS glue at all.

Yea I don't know how you can achieve this. your environment you are running the glue jobs need to have the python libraries installed so you can execute the code. The way I set it up is I am doing 100% local development but glue is in a docker container. If you can't use docker, you need to install and set up spark directly on your local machine. I tried to do this following the documentation but it was messy and I couldn't get it to work in the end

I made the script just for this KZfaq video, sure things could be organized into classes if it makes sense?

Do you have a Github for this Pytest example?

Hey, checkout my videos on local development for AWS glue. I covered topics like using interactive sessions, pycharm and vs code with a docker container with AWS glue. In order to upload them, I recommend managing them with IaC with terraform or cdk.