Thanks sarfaraz!

Good I'm glad it was more efficient than your other method. This only works for inserting new records. I plan on making another video on how to update records with pyspark and AWS glue at some point soon

@@DataEngUncomplicated oops, I meant adding new records but looking forward to that video as well.

@@DataEngUncomplicated I have a file (test_report.csv) that gets uploaded to S3 (let's say new version every 24hr) -> Glue crawlers pick it up and eventually I have a job that populates a table in Redshift. However, after creating a Schedule for my job I noticed that the table in Redshift is not getting updated with the most current csv file it picks up from S3; is there no overwrite? Is there a way to update the table in Redshift dynamically, otherwise how is this service practical?

​@@denisgomonov2320 Hi Denis, yea absolutely this is possible. I am doing something similiar to this in aws glue where I am upserting records by a particular composite key. I am planning to make a video on how to do upserts in AWS Glue to redshift.

Can we insert the data with same columns but has different ordinal positions?

Thanks Luk!

Hey I believe the pyspark uses copy under the hood to write to redshift. One of the parameters it asks for is a temp directory in s3. It uses this to write the temp parquet files before copying it to redshift.

You can use pre and post queries to perform basically an upset to solve for this. I'm going to make a video soon on this.

I'm using pycharm professional

Yes this is possible, one of my next videos is on this with redshift.

Are you thinking real time loading or just a bulk load of all the files? But yea it's definitely possible

Instead I ll say use s3 presigned urls. Create api and lambda. User passes filename to api and lambda provides presigned url. User makes post request to url. Till 5 GB data can be passed via presigned url

Hi, this is outside the scope of this video. I don't know if this is even possible to be honest. With dataframes the files are abstracted away from us. Please post on repost your question to see if someone can help you out. Also Google search or reading the docs can be your friend!

Hey, unfortunately I do not. If I had more time I would love to make an aws data engineering course.

@@DataEngUncomplicated I have subscribed. If you have plans to do paid courses on Udemy. I hope you will make an announcement. Yeah?

Hi Shashikant, it's important to understand where the bottle neck is. If you are running a pyspark job one way is to add more worker nodes if you are processing extremely large data sets

@@DataEngUncomplicated Hi, Thanks for your reply. Actually I have created redshift JDBC connection and trying to write the dynamic frame to redshift table. datasink0 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = applymapping_for_ids, catalog_connection = "redshift", connection_options = {"dbtable": "track_order_ids_test", "database": "dev"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink0") This dynamic frame has one column(ID) with 8M data and approx. size is 800MB and it's taking around 32mints. What's your suggestion on the same. Thanks in advance.

Are you looking for the script? If so, I started uploading the scripts a while after I made this video sorry.

These errors can be tricky and require you to look into the logs further to see what is causing it. It sounds specific to your data.

Hi Sumit, hmm is there any more error messages provided? That seems to be a very vague.