I'm happy you found it to Farsim! Thanks for subscribing!

No problem! Yup this approach would work. Why do you need to use pyspark though? Are you analyzing millions of records? If it's only 1000s or 100,000s lambda functions or just using a glue shell job might be sufficient

Thanks, can you elaborate on what videos would be helpful on multiple conditions?

@@DataEngUncomplicated thank u for replying. i want new videos 1)using triggers for crawler, and connect to sns service for msg similar like that. 2)join oracle database to glue for querying. I really appreciate your efforts.💟

Hey, sorry for the late reply. Whether to read from GlueCatalog or directly from S3 depends on the specific requirements and constraints of your project. Here are some factors to consider: Performance: Reading data directly from S3 can be faster than reading through GlueCatalog, as GlueCatalog adds an additional layer of metadata management. However, the performance difference may not be significant, especially if you use partitioning and indexing in GlueCatalog to optimize queries. Schema evolution: If your data schema is likely to change frequently or unpredictably, using GlueCatalog can provide a more flexible and automated way to manage schema evolution. GlueCatalog can automatically detect schema changes and update table definitions, which can save you from having to manually update your code. Cost: Using GlueCatalog can add some additional cost to your AWS bill, as you are paying for the metadata management and indexing that GlueCatalog provides. However, the cost may be small compared to the benefits of using GlueCatalog for your specific use case.

Thanks! Sure I will add the change data capture to my video suggestion list. I have a couple of videos on writing data to s3 using AWS lambda service and AWS glue you can check out. Check out this blog post on aws related to CDC with aws glue aws.amazon.com/blogs/big-data/implement-a-cdc-based-upsert-in-a-data-lake-using-apache-iceberg-and-aws-glue/ This might be helpful if you can leverage the iceberg file format

There isn't an existing role that will give you everything. You need to add permissions to your s3 bucket if you are using it for reading and writing data as well.

No reason to avoid Catalog, I made this video because you might need to read files with aws glue that have not already been configured in the aws glue catalog. Is your schema defined in your first column of your csv files? That is one reason I could think of why it's not showing up in the catalog.

Hey thanks you. I've seen this when no data is being returned...can you confirm there are no records being returned?

Hi Shashank, these are great video suggestions I will add them to my list, I have broken my videos down into smaller segments but it having a video end to end might be beneficial esp with the permission challenges

@@DataEngUncomplicated what number is my request on your list 😅😅😅😅

In AWS Glue, you can use PySpark to read a CSV file and ignore non-UTF8 characters. Here’s an example if you convert your dynamicFrame into a pyspark dataframe # Replace non-UTF8 characters for column in df.columns: df = df.withColumn(column, col(column).cast("string").alias(column))

Hi Malvika, it sounds like you did not define glueContext correctly, I would check to make sure you included the template python code that comes when you first create a new glue job

@@DataEngUncomplicated Thank you 😊 I will check and get back. Thank you for the response. Merry Christmas and a Happy New Year to you !

Thanks! Merry Christmas and happy new year!

Hi, try posting on AWS repost, you might get a quicker response for this particular problem.

Hey Harsh, do you want to replace the same data on AWS S3? There is a parameter on right that will overwrite the partition which could be an option

If you're asking if databricks and glue are the same then no they definitely are not.

Hey, Unfortunately I don't have a formal course but I am building out a youtube playlist related to aws glue for reading transforming and writing data: kzfaq.info/sun/PL7bE4nSzLSWci0WpYafgTOBcqpdtO3cdY

Yes you can! Aws glue actually supports a streaming mode which supports kinesis as a data source!

@@DataEngUncomplicated did you try?

@@DataEngUncomplicated I can put data from KDS to S3 or grab data from S3 and put it on RDS but can't do directly from KDS.

It says it supports it: docs.aws.amazon.com/glue/latest/dg/add-job-streaming.html

@@DataEngUncomplicated I saw that. Unfortunately it doesn't work.

Hi there, although in this tutorial I am not using and glue transformation methods, I am using the create_dynamic_frame_from_options method to load the data which is from the GlueContext class. This is why we need to use gluecontext. Dynamic dataframes can be for structured data s well not only just semi-structured or unstructured.

That works too but it's not using the aws glue library to do it.

@@DataEngUncomplicated Sorry, I'm new to Spark and Glue. Would you mind to elaborate about glue library are you referring to? I know about Glue Data Catalogue, but it is not affected when I use df = spark.read.csv(path).

Give a read on the aws glue api and the transformations that come with it: docs.aws.amazon.com/glue/latest/dg/aws-glue-api.html