You're welcome!

Thanks a lot!

You're welcome!

Thanks Julian!

Thanks, if your doing the exact same transformation on all 300 tables, I would put the table names in a list and then loop through each table to add this load date column and then write it to another target. If your tables are very large I would be careful trying to bring all 300 tables into memory at once so perhaps looping through might be a better option of this is the case. The other option is Create a Parameterized Glue Job...Write a Glue ETL script that performs the required transformation on a single table, making use of parameters for dynamic values (e.g., input table name, output table name, etc.).

I had to install python in the container and now i see the run button.

Hi, it sounds like you don't have admin permissions on your own machine. I would probably talk to your IT team to see how they could help. I didn't experience any permission issues.

@@DataEngUncomplicated It looks like the glue_user does not have permission to access the mapped volume folder location for AWS configs. I had to specifically give read permissions from my local machine

For this step, just navigate to where you have your vs code workspace file and open it in vs code. It might look differently on a Mac I did this on a windows machine.

Strange, at what stage did you get this error?

Hi, to someone who found this error. I also did but solved it by using the command: "python3 " instead of "python "

Hi, I'm not sure what you mean it almost purchased a subscription. But you need pro to use docker in pycharm.

Hi Ryan, since your deployed AWS glue eventually runs in the cloud you would need your data in S3. You can manually upload your data to S3 through the AWS S3 console or use boto3 or the AWS S3 cli to do this.

This one is with visual studio code vs my last video was pycharm. I got a lot of comments that more people use vs code and wanted a tutorial specifically for that ide

Hi Rahul, The workspace location is specific to the individual developer. When we run code on our docker container, we are telling the docker container to run some code we have on our local machine. So to answer your question about "how would developers use a common code base". This should always be in a code repository like github, codecommit, or bitbucket. I wouldn't advise an s3 bucket for storing everyone's code.

Yes

That's a good question, I'm not sure how to visualize the data in vs code from spark I use pycharm mostly.

@@DataEngUncomplicated sir on pycharm also dynamicframe is shown as json line-by-line in your video

@@DataEngUncomplicated Sir On pycharm also dynamic-frame is not shown as table instead it is json line-by-line as seen in your video.

I don't think pycharm supports visualizing spark dataframes, you can bring the data into a pandas dataframe and then you can view the data in debugging mode as one method. @@dipeshg8581

@@DataEngUncomplicated thank you sir 🙏🏼