AWS Glue - Access On Premise Database and Transfer Data

Рет қаралды 10,060

Жыл бұрын

I will show you how to connect to external database (on-premise) via AWS Glue. Will be create a Python script using Spark and connect to database via JDBC and save dataset to S3 bucket.
Database in this example will be a MySQL database location in Digital Ocean.
Support my Channel
www.paypal.com/donate/?hosted...
Book a 1 on 1 meeting with me at:
calendly.com/iamlu/30min

Пікірлер: 31

@user-lt1ez6ef4v 4 ай бұрын

I am excited to use this kind of way to upload s3 in glue.

@peterpan9361 Жыл бұрын

Great content !! Subscribed :)

@codingwithlu Жыл бұрын

thanks for the sub! move content coming!

@francismagnusson378 3 ай бұрын

THANK U FOR THIS 😭🙏 im currently working as a data engineer and man glue feels kinda limited (probably because im new to it). ive been trying to connect to our DB using only the python shell glue job, but the thing is, i have to setup a driver for it. theres a workaround using this library called pymssql but sometimes it doesnt work. i hope u can add tutorial on how to add drivers and use it on python shell jobs

@marcoap Ай бұрын

Perfect video! I don't know if you will see my question. Did you create a "Connection" in AWS Glue, or is the on-premises database IP exposed on the internet? In this case, should I use something like a VPN from AWS to create a secure way to connect to it? Thanks a lot!

@krishnareddy4948 Жыл бұрын

Great content. Please do a demo on Oracle source system in Glue

@codingwithlu Жыл бұрын

Will do!

@xroydacute Жыл бұрын

@@codingwithlu Hi Lu, any chance you have done a video for Oracle source system?

@sunilmakwana2898 Жыл бұрын

Great Information! Please make a video on how to load multiple table to the S3 bucket from on prem server?

@codingwithlu Жыл бұрын

Sure will make a video of that

@andrezzo80 2 ай бұрын

what about using coalesce to make sure the data stays in 1 partition?

@poojabharathi4826 Жыл бұрын

what all service access must be included in iam role to load on premises to s3

@AY1986R Жыл бұрын

thank you for this great video, but I do not see where you have created the connection, an example with JDBC oracle will be great

@arunnair1392 Жыл бұрын

In this case, I guess , no incoming ip restrictions are in place on the server side . Mysql server port must be open for all traffic . So no connection is required. But for enterprise implementations , we must create a secured connection

@ashutoshsharma4525 10 ай бұрын

Thanks for nice video. Could you please provide the link for your code. I am getting some parenthesis missing errors while running it.

@kshitijbansal3672 Жыл бұрын

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

@rafaelfpk2079 Жыл бұрын

Does your glue applications takes long time to run for the first time? I'm having a huge problem with this here, with 2 DPUs, just connecting to AWS RDS and WRITING into S3, like your application, it took 30minutes and gave timeout.

@codingwithlu Жыл бұрын

It shouldn't take too long. I have a few glue jobs, and some of them are querying a few million row of data, and it takes just a few minutes. How huge is the data that it's querying?

@rafaelfpk2079 Жыл бұрын

@@codingwithlu Very small, 50 rows...

@appurvagnihotri2218 9 ай бұрын

how to get mysql database info to add in code?

@user-pi5dq7eb8w 6 ай бұрын

How to fetch table data from aws rds postgresql and store it to s3. can anyone help?

@nizor_ Жыл бұрын

Awesome content. How can i connect to a firebase realtime db?

@codingwithlu Жыл бұрын

Glad you liked the content. I would need to do some testing to see how to connect to firebase db. I would let you know what I find.

@tintin1678 Жыл бұрын

How to connect to Oracle Db in AWS Glue ? I'm getting error using cx_Oracle lib

@codingwithlu Жыл бұрын

I will make a video over this

@achyuthvaleti3639 Жыл бұрын

How can I get the Mysql Connection details

@codingwithlu Жыл бұрын

If it is a MySql instance in AWS RDS you can get it from the AWS console. If it is a on-premise instance, you can connect via IP address.

@kshitijbansal3672 Жыл бұрын

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

@akarshba3052 6 ай бұрын

Hey, were you able to do it?

@kshitijbansal3672 Жыл бұрын

How to make a pipeline with Aws Glue to move data from On premises SQL Server to S3? Pls reply in detail sir.

@codingwithlu Жыл бұрын

I will make video that is specific for On Premise SQL Server to S3