AWS Tutorials - Using AWS Glue Workflow

Рет қаралды 12,905

3 жыл бұрын

The Workshop URL - aws-dojo.com/workshoplists/wo...
AWS Glue Workflow help create complex ETL activities involving multiple crawlers, jobs, and triggers. Each workflow manages the execution and monitoring of the components it orchestrates. The workflow records execution progress and status of its components, providing an overview of the larger task and the details of each step. The AWS Glue console also provides a visual representation of the workflow as a graph.
In this workshop, you create a workflow to which orchestrates Glue Crawler and Glue Job.

Пікірлер: 61

@7sandy 2 жыл бұрын

Best and to the point, your channel should be the official AWS learning channel.

@AWSTutorialsOnline 2 жыл бұрын

Many thanks for the appreciation

@liguangyu6610 2 жыл бұрын

I just want to say thank you for all the tutorials you have done

@AWSTutorialsOnline 2 жыл бұрын

Glad you like them!

@ladakshay 3 жыл бұрын

Really like your videos, they are simple which helps us easily understand the concept.

@AWSTutorialsOnline 3 жыл бұрын

Thanks for appreciation

@hareeshsa8381 3 жыл бұрын

Thanks a ton for all your effort in making this video

@AWSTutorialsOnline 3 жыл бұрын

Thanks for the appreciation

@tientrile5733 Жыл бұрын

I really like your video. Thank you so much for a wonderful video.

@AWSTutorialsOnline Жыл бұрын

Glad you liked it

@vascomonteiro2297 3 жыл бұрын

Thank you very much!! amaizing videos

@AWSTutorialsOnline 3 жыл бұрын

Thanks for the appreciation

@radhasowjanya6872 2 жыл бұрын

Hello Sir, Thanks for the wonderful session. I have a quick question: I was able to create 2 different data loads in the same glue job and it was successfully loading 2 targets. But i would like to know how we can configure the target load plan(similar to Informatica ) in a AWS Glue studio job.?.

@AWSTutorialsOnline 2 жыл бұрын

Glue job supports parameters. You can parameterize target location when running the glue job.

@atsource3143 2 жыл бұрын

Thank you so much for such a wonderful tutorial, really appreciate. Can you please tell us how we can set a global variable in glue job. Thank you

@AWSTutorialsOnline 2 жыл бұрын

Apologies for the late response due to my summer break. There is no concept of global variables. But jobs can maintain states between them in the workflow - here is a video about it - kzfaq.info/get/bejne/fZyUaZCSx8-1nqM.html Hope it helps,

@prakashs2150 3 жыл бұрын

Nice video.. very easy steps thanks!

@AWSTutorialsOnline 3 жыл бұрын

Glad it helped

@pradeepyogesh4481 3 ай бұрын

You Are Awesome 🙂

@rokos1825 6 ай бұрын

Good tutorial, but the audio fades in and out. AWS Glue has been updated enough to make some of this information irrelevant. I would update with the latest UI and correct the audio issues. Thank you.

@mallik1232 3 жыл бұрын

Very good explanation in detail...

@AWSTutorialsOnline 3 жыл бұрын

Thanks

@rexe1166 3 жыл бұрын

Hi, Is it possible to move an s3 file(csv) after it has been imported to RDS mysql table by a glue job to an processed S3 folder? Great content as always.

@AWSTutorialsOnline 3 жыл бұрын

Sure, it is possible. I created a workshop for this scenario which might help you. aws-dojo.com/workshoplists/workshoplist33 Hope it helps,

@rexe1166 3 жыл бұрын

@@AWSTutorialsOnline Thank you and much appreciated.

@shubhamaaws 3 жыл бұрын

You are doing great work. Please keep making videos on glue. Your content is best. Can you make video on reading from rds with secure ssl connection using glue.

@AWSTutorialsOnline 3 жыл бұрын

sure - I will put it to the backlog.

@akshaypunewar3887 3 жыл бұрын

Thanks for sharing knowledge... I am not sure why we should use workflow instead of stepfunction... we do have better control in stepfunction... can you please advise ?

@AWSTutorialsOnline 3 жыл бұрын

You raised a very good question. Simple answer is - use Glue Workflow only when you are orchestrating jobs and crawlers only. If you have need to orchestrate other AWS services, StepFunction is more suited. I personally believe - over period of time, StepFunction would become main orchestrator service for Glue as well.

@akshaypunewar3887 3 жыл бұрын

@@AWSTutorialsOnline Thanksl you..

@abiodunadeoye9327 2 жыл бұрын

Please How do you make use of the properties, is there another tutorial on that? thanks

@AWSTutorialsOnline 2 жыл бұрын

yes there is - kzfaq.info/get/bejne/fZyUaZCSx8-1nqM.html

@atsource3143 2 жыл бұрын

Sir, is there any way were we can set a trigger for S3 and Glue Job? What I mean is , whenever a new file upload in S3 one trigger should get active and it run the Glue job and same thing for Crawler also. So whenever new file upload in S3 it active trigger for crawler and job. Thank you

@AWSTutorialsOnline 2 жыл бұрын

You can do it. Configure event for S3 bucket which gets trigger on put and post event. On the raise of the event, call a Lambda function. In the lambda function, use Python Boto3 API to start glue job and crawler.

@rajatdixit4912 2 жыл бұрын

Hi sir, My que is, when any push happens in s3 that time my workflow is runs automatically how i can do plz help.

@AWSTutorialsOnline 2 жыл бұрын

Configure event for S3 bucket. Event will call a Lambda function and the Lambda function will call Glue workflow using SDK like python boto3

@alphaprimer6485 Жыл бұрын

It was a good tutorial but I would recommend a better mic as it is hard to hear you at some times.

@prasadkavuri8871 2 жыл бұрын

How we can add DPU'S in Glue job using glue workflow.

@AWSTutorialsOnline 2 жыл бұрын

Not sure why you want to add DPU to Glue Job from Workflow. When you configure Glue Job, you can configure default DPUs for it.

@venkateshanganesan2606 3 жыл бұрын

Nice and clear explanation. I have query here, how can we run one after another workflow (not job/crawlers) i.e. one workflow for dim and another for fact. once dimension is loaded it should another workflow for fact.

@AWSTutorialsOnline 3 жыл бұрын

Nested workflow is not available. The best approach will be - at the end dimension workflow, you run a job (using Python Shell) which simply starts the workflow for fact. You can also use other mechanism such as orchestration using Lambda based business logic or Step Function but it will be little complicated because between dimension and fact workflow you need to make API call to check successful end of the dimension workflow before you start the fact workflow. So probably - the first approach I talked about is the best way.

@venkateshanganesan2606 3 жыл бұрын

@@AWSTutorialsOnline Thanks for your time. I really appreciate it. you answered my query and i got an idea what to do, let me try create one specific job to call fact workflow at the end of dimension workflow using python scripts.

@venkateshanganesan2606 3 жыл бұрын

Hi @@AWSTutorialsOnline, I tried some blogs and google, I don't find code to call AWS workflow using python shell, is that possible to share any our blog and git where I can find some info regarding to execute the workflow using python. Thanks in advance.

@AWSTutorialsOnline 3 жыл бұрын

@@venkateshanganesan2606 Hi, basically - you need to use boto3 Python SDK in python shell based job. You can google plenty of examples for that. if not let me know. In this job, you use Glue API to start the workflow. API for this method is here - docs.aws.amazon.com/glue/latest/dg/aws-glue-api-workflow.html#aws-glue-api-workflow-StartWorkflowRun Hope it helps. Otherwise - let me know,

@venkateshanganesan2606 3 жыл бұрын

@@AWSTutorialsOnline Thanks a lot, it works as you suggested. I used the below piece of code in end of my dimension job to invoke the fact workflow. I really appreciate that your sharing your knowledge wisely. import boto3 glueClient = boto3.client(service_name='glue', region_name='eu-west-1', aws_access_key_id='access_key', aws_secret_access_key='secret_access_key' ) response = glueClient.start_workflow_run(Name = 'wfl_load_fact') Thanks again for sharing your knowledge.

@prathapn01 23 күн бұрын

you better use a headset or earphone while speaking.. otherwise the session is very good.

@bhuneshwarsingh630 2 жыл бұрын

Thank for sharing knowledge but can you create video on read data from s3 and writing to database while we need to handle bad records while reading and only insert good records in rds table and badrecords in s3 location

@AWSTutorialsOnline 2 жыл бұрын

how you differentiate between good and bad records?

@bhuneshwarsingh630 2 жыл бұрын

@@AWSTutorialsOnline if record don't not match schema I mean data type is like datatype is int like 1,2,3 are coming but sometimes it comes as four ,five i will share you example link

@bhuneshwarsingh630 2 жыл бұрын

Basically i m looking for whenever any corrupt record found so I want write in S3 path and normal record I want to write in database ,i don't want my job to stop corrupt record found then it must continue my job running in AWS glue

@AWSTutorialsOnline 2 жыл бұрын

I need to see some example of corrupt data in order to understand how to check for the same. But once you know whether dataset is corrupt or not; you can use dynamic frame write method to write to S3 bucket or database.

@AWSTutorialsOnline 2 жыл бұрын

@@bhuneshwarsingh630 I am publishing a video in 1/2 days about doing data quality check. Please have a look. I think it might help you.