Thanks for the comment Patrick, much appreciated!

Thanks! Absolutely! More videos to come!

Thank you my Chilean friend!

Thanks Nehal!

Glad it was helpful!

Your welcome, glad you got it working

Hey, that's a good idea, I can put them all into 1 playlist. It will be a lot of videos though, I kind of broke them down into different aws services

Thanks! Much appreciated!

As of today, there are about 6k views! That's a lot more people than you could reach through normal means. I think they're doing a great job!

Hi Joe, You're welcome! If you are trying to pass your event payload to your lambda function through step functions, when you are running your step function execution in the console manually, you can paste your test payload there. You should set up your step function so the payload gets passed directly to your lambda function with the parameters your lambda needs. I hope this is what you are looking for.

Hi Pradeep, if you attempt to configure a rule with eventbridge with the glue sample, it will tell you what the general payload will look like being passed to sns: for example: { "version": "0", "id": "66fbc5e1-aac3-5e85-63d0-856ec669a050", "detail-type": "Glue Job Run Status", "source": "aws.glue", "account": "123456789012", "time": "2018-04-24T20:57:34Z", "region": "us-east-1", "resources": [], "detail": { "jobName": "MyJob", "severity": "INFO", "notificationCondition": { "NotifyDelayAfter": 1 }, "state": "STARTING", "jobRunId": "jr_6aa58e7a3aa44e2e4c7db2c50e2f7396cb57901729e4b702dcb2cfbbeb3f7a86", "message": "Job is in STARTING state", "startedOn": "2018-04-24T20:55:47.941Z" } }

Hi Steven, can you edit your sentance, I don't understand what you trying to do.

Yes, you can “wrap” your bash script within a supported language like Node.js or Python. For example, in Node.js, you can use the child_process module to execute a bash script. Remember to package your bash script and any other necessary files into a ZIP file and upload it to AWS Lambda. Also, ensure that your bash script has the appropriate permissions to be executable.

Hi Oscar, It just had the sns:Publish action. The full statement looks like this: { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:us-east-1:account#:glue_jobs" } ] }

The choice between using AWS Glue in Step Functions vs. Glue Orchestration (Glue Workflows) depends on the complexity of your data pipeline and the services you’re using. AWS Glue Workflows are beneficial when you’re chaining together multiple Glue jobs and/or crawler. They are particularly useful for batch processing, where you can schedule workflows directly. However, Glue Workflows lack several features common in flow control tools, such as conditional branching, loops, dynamic maps, and custom steps. On the other hand, AWS Step Functions are more suitable when the complexity exceeds simple triggers and the services used extend beyond Glue. Step Functions provide more advanced orchestration capabilities, including support for error handling, parallel execution, and conditional logic. They also integrate with over 220 AWS services, making them a more flexible choice for complex, multi-service workflows. In addition, Step Functions can handle quick start and shutdown, which can manage a reasonable throughput. They also allow for the execution of parallel jobs, which is not possible in Glue Workflows.

Yes, you can pass the S3 path and database as input parameters to an AWS Step Functions State Machine that includes an AWS Glue JobRun S3 Step. When you define your Step Function state machine, you can include an input parameter section that specifies the input data that will be passed to the state machine when it is executed. You can define the input parameters as key-value pairs in JSON format.

@@DataEngUncomplicated thanks,

Working as a data engineer and in the data analytics field for 10 years. Also doing Udemy courses, AWS certifications and side projects to continue to learn as the field is changing so fast with new services coming out all the time.

Hi Mallikarjun, there is a parallel state which will allow you to run whatever jobs in parallel

In the sns step there should be a box where you can customize the message

@@DataEngUncomplicated then how do i use the parameters of the job? for example if i want to send "The job state is (~SUCCEDED~ or ~FAILED~). At this time ~endtime~ ", thanks