Crawl different datastores in a single Glue Crawler job

Рет қаралды 2,377

2 жыл бұрын

Prerequisite:
-------------------------
Below is the learning path for AWS Glue in-depth --
An automated data pipeline using Lambda, S3 and Glue - Big Data with Cloud Computing
• An automated data pipe...
How to Use AWS Glue with Snowflake | PySpark-Snowflake Connectivity
• How to Use AWS Glue wi...
Glue Python Shell with Snowflake Wheel
• Connecting Snowflake w...
Transform Data Using AWS Glue and Amazon Athena
• Transform Data Using A...
Schema Evolution in AWS Glue using Glue Crawler | AWS Athena
• Schema Evolution in AW...
Simplify Amazon DynamoDB data extraction and analysis by using AWS Glue and Amazon Athena
• Simplify Amazon Dynamo...
AWS Glue as Hive catalog
• Using the AWS Glue Dat...
AWS Glue as spark sql catalog
• AWS Glue Data Catalog ...
A very frequent technical requirement in big data domain--
You have to write spark dataframe but with kms encryption, if you are using Glue , then this is one approach you can try to improve the security of your pipeline by enabling server side encryption
• Security Configuration...
Incremental Glue Crawling with S3 events
• Incremental Glue crawl...
Tracking Processed Data using Glue Job bookmark
• Tracking Processed Dat...
AWS Glue Job Workflow
• AWS Glue Workflow in-d...
Empty Success File while writing Glue dataframe in S3
• In-depth intuition on ...
Manage Glue Job with Step Function
• Manage AWS Glue Jobs w...
Automated Serverless Datalake using Glue, Lambda , Cloudwatch
• Build and automate Ser...
Check this playlist for more AWS Projects in Big Data domain:
• Demystifying Data Engi...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZfaq CHANNEL

Пікірлер: 12

@shobhapoojari1694 10 ай бұрын

Thanks a lot

@KnowledgeAmplifier1 10 ай бұрын

Most welcome

@marquinho1p 6 ай бұрын

great!

@KnowledgeAmplifier1 6 ай бұрын

Thank you @marquinho1p! Happy Learning

@user-kb8fe5uw7j Жыл бұрын

what if i have multiple csv files in 1 folder and i want to create Data catalog for each file to query in Athena can help me ?? any documentation or any reff ??

@bhagyarajgopalam2254 5 ай бұрын

It is compulsory to put each file in separate folder Can we put all files in same folder and if we give datasource as that folder will it work? I am facing issue

@ravikreddy7470 Жыл бұрын

Thank you so much for all the videos related to ETL pipelines. Do you have playlist or some material for Glue. I was able to learn so much with snowflake - python- aws pipelines and different implementations. But I would like lo learn about Glue & Redshift too.

@KnowledgeAmplifier1 Жыл бұрын

Ravi K R, as of now , there is no video on AWS Redshift in my channel , for AWS Glue , you can refer this link -- qr.ae/prNOn7 Happy Learning

@ravikreddy7470 Жыл бұрын

Thanks a lot for all your videos. From traditional warehousing background with python skills. I was able to learn to so much in cloud with your videos. Please let me know if you offer any 1 vs 1 paid courses, I am willing to learn and improve my skillset. Thanks again !! :)

@KnowledgeAmplifier1 Жыл бұрын

Glad to hear that the videos are helpful to you Ravi K R! Thank you for your support ! As of now , I am not providing any 1 : 1 paid course or any paid course , in future , if I plan to do so , will update in this channel ... But for now , all videos are available for free here ... Stay tuned for more interesting upcoming projects

@ravikreddy7470 Жыл бұрын

@@KnowledgeAmplifier1 Thank you. But please do let me know if you provide any training. Thanks again

@KnowledgeAmplifier1 Жыл бұрын

@@ravikreddy7470 sure , Stay Tuned