Automate File Handling With Python & AWS S3 | Five Minute Python Scripts

Рет қаралды 73,984

Күн бұрын

Hey everyone! Quick Example on how we can automate the handling of different files to automatically be uploaded to amazon S3.
Sorry for the green screen issues - hope they're not too bothersome!
Few notes on Creating IAM users:
- Create a user
- Enable programmatic access
- Add user to a group with the desired permissions
( In this video we used S3, but you can use this client with many more aws services. You would assign those permissions to a group the same way as we do the s3 permissions here)
- Copy access and secret access keys
Legit crazy that I'm writing this with 11,000+ subscribers -- How awesome! I write these notes in the descriptions of most of my videos, and I still remember writing it with 250 subscribers and being so thankful. I'm honored I get to create these videos and hopefully they're of use to some of you. Thank you for all the support and kind words throughout the last year!
Support the Channel on Patreon --
/ derricksherrill
Join The Socials --
Reddit - / codewithderrick
FB - / codewithderrick
Insta - / codewithderrick
Twitter - / codewithderrick
LinkedIn - / derricksherrill
GitHub - github.com/Derrick-Sherrill
*****************************************************************
Full code from the video:
from secrets import access_key, secret_access_key
import boto3
import os
client = boto3.client('s3',
aws_access_key_id = access_key,
aws_secret_access_key = secret_access_key)
for file in os.listdir():
if '.py' in file:
upload_file_bucket = 'youtube-dummy-bucket'
upload_file_key = 'python/' + str(file)
client.upload_file(file, upload_file_bucket, upload_file_key)
github.com/Derrick-Sherrill/D...
Packages (& Versions) used in this video:
boto3 1.11.13
os
*****************************************************************
Code from this tutorial and all my others can be found on my GitHub:
github.com/Derrick-Sherrill/D...
Check out my website:
www.derricksherrill.com/
If you liked the video - please hit the like button. It means more than you know. Thanks for watching and thank you for all your support!!
--- Channel FAQ --
What text editor do you use?
Atom - atom.io/
What Equipment do you use to film videos?
www.amazon.com/shop/derricksh...
What editing software do you use?
Adobe CC - www.adobe.com/creativecloud.html
Premiere Pro for video editing
Photoshop for images
After Effects for animations
Do I have any courses available?
Yes & always working on more!
www.udemy.com/user/derrick-sh...
Where do I get my music?
I get all my music from the copyright free KZfaq audio library
kzfaq.info...
Let me know if there's anything else you want answered!
-------------------------
Always looking for suggestions on what video to make next -- leave me a comment with your project! Happy Coding!

Пікірлер: 111

@breadenthusiast 4 жыл бұрын

Your awesome Derrick! Some more videos on AWS and APIs would be wonderful! Keep up the great work!

@petermcentee9922 3 жыл бұрын

This is exactly what I needed for work, thanks so much!

@lancehoward1609 4 жыл бұрын

Great job. Finally that can explain step by step what you are doing!

@abhinavkannojia1053 4 жыл бұрын

Big Fan of these pythonic tricks..Nice work there. Please keep it coming !!

@unitech23 7 ай бұрын

Amazing Derrick. Great Job Dude!!

@RenandOliveira Жыл бұрын

Great Video Derrick! Go Vols!

@vivekr4690 4 жыл бұрын

Keep uploading such useful videos! Thanks! I learnt some new concepts here!

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@oldguywholifts 3 жыл бұрын

Has to be said.. direct and precisely what the title says. EXCELLENT!

@HopeUnveiled 2 жыл бұрын

Wish i was this beautiful hahah Great vid dude! Helped me a lot.

@Anwar-Mian 2 жыл бұрын

Excellent, Derrick!!!

@colorfuljosh 2 жыл бұрын

Great video, thanks from Australia dude

@just5two163 2 жыл бұрын

Thank you Derrick. Nice tutorial. Thanks for your time! It helped me a lot.

@Sagar_Tachtode_777 3 жыл бұрын

Awsome Job. Thank you Derrick...

@Reptar4000 3 жыл бұрын

You’re the man. Need more content!

@fbzz64 3 жыл бұрын

You explain very well, thanks for sharing your knowledge .. Greetings from Argentina!

@amnksngh29 2 жыл бұрын

Thanks Derrick!

@pepex90 3 жыл бұрын

Thank Derrick !!

@vijaypingale7 3 жыл бұрын

Thanks man ! Great video!

@stahmedpendragonguinevere2711 4 жыл бұрын

Keep up the good stuff matey!

@bismarckandu6069 2 жыл бұрын

This is natural teaching.

@renyildiz5155 3 жыл бұрын

I am fairly new to data but you are awesome. Thank you!!

@kunaljha6204 2 жыл бұрын

Stupendous tutorial....Need more such tutorials

@DonellGrace 4 жыл бұрын

Amazing. Well done

@chadripley2776 3 жыл бұрын

great video. One issue I had was you have to set the region in the arguements ofr creating the client object if it isnt configured already.

@adrianmisak07 2 жыл бұрын

thank you, this was great

@3h936 2 жыл бұрын

Thank you, it's amazing to video !!!

@AbhishekBhuva 4 жыл бұрын

I am a Vlogger but as a professional, i am a software engineer too. I think your video is more interactive since its 6-10 min.

@anilct6749 3 жыл бұрын

Thanks man. Much appreciated!

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@yaseenimuhammadraja9461 3 жыл бұрын

Great video 👍

@vamshidharr3434 Жыл бұрын

Awesome !!!

@nowyouknow2249 4 жыл бұрын

Well done bro. Please more vids on AWS.

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@keithsmalling9723 2 жыл бұрын

Brilliant tutorial well explained

@AdityaBolt 4 жыл бұрын

loved the video..

@GG1440 3 жыл бұрын

This is very helpful, thank you.

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@DanielArturoSanchezDeoleo 2 жыл бұрын

thank you greetings from the Dominican Republic

@MuhammadAli-zg2nw 3 жыл бұрын

you deserve the like and sub

@rajkishan3690 3 жыл бұрын

Well Explained.

@amatulathok597 Жыл бұрын

Thank's a lot dude

@AshisRaj Жыл бұрын

Excellent

@paulschrum9222 3 жыл бұрын

3:36 -- file is a python keyword. Don't use Python keywords for variable names. Use something like a_file or my_file.

@japeshgawale9640 3 жыл бұрын

You are awesome Derrick! Could you Please upload a video of automating data transfer from AWS S3 to AWS Redshift (Using AWS Glue) or any other appropriate method .

@lotusmojo 3 жыл бұрын

Awesome

@ncTechlearning 4 жыл бұрын

Could you please make a video on file transfer from sftp to amazon s3 using python

@gabopython Жыл бұрын

good video

@lucasneves3465 2 жыл бұрын

amazing, it was very useful I have a question: how can I send the file to a specif path in my bucket?

@erich.3247 3 жыл бұрын

Thanks for the vid man! Noob questions here. Can the S3 subdirectory be created on the fly from within the python script? Say I get a .py file but I have no python subdirectory in my bucket. If my script calls for storing the file in my bucket's non-existent "python" subdirectory will it be created or do I have to manually do it beforehand? Thanks!

@andrzejandrzejos6674 4 жыл бұрын

Hi! Do you think Python allows us to scrap data from the screen, not only from a website? I see there is a lot info about web scraping but I want to scrap words that I type in an app. I dont have acess to the API.

@BehruzbekOtayev 4 жыл бұрын

Thank you

@romachd01 3 жыл бұрын

@Derrick Sherrill everytime I try to run the code it picks up files from windows32 directory. I have the .py file inside a folder with few test .txt files but it keeps picking up txt files from the win32 dir. Whats happening, please help

@dreambig9794 3 жыл бұрын

I have my bucket like "parent_Bucket/Chield_Bucket", Now I want to upload my file to Chield_Bucket. Could you please suggest what I need to do for this.

@velnathravanan1765 2 жыл бұрын

In the Python file, write a program to get all the files from a public S3 bucket named in random list. In there there might be multiple files, but your program should find the file with the prefix __cb__, and then output the full name of the file. You should use the boto3 module to solve this challenge.

@granand 3 жыл бұрын

Thank you so much. Derrick Please can you take me along with you in a journey to use Python for data manipulation using strings, lists, tables, SQL and excel with regex.

@willd0g 3 жыл бұрын

That’s great

@surajiyer7406 3 жыл бұрын

If we need to read multiple files from AWS S3 bucket, what must be the code in Python?

@mainakbiswas6541 2 жыл бұрын

Nice video. I have one question. I don't want to download the content from S3, and I want to know the content details like no. Of .txt, no. Of. py within that bucket. I am using aws s3 ls bucket_name/ - - recursive - - human-readable - - summarize It shows the total size and objects no. But can't find the file counts with respect to their extensions.

@AnAyahaDay 3 жыл бұрын

How can we delete all files from a S3 folder? ( not the folder, but all files in a folder )

@sathishsuganya2013 2 жыл бұрын

If i wanted to add some file in to the specific folder (Existing on S3 Bucket)how i do that?

@aajadsingh8050 4 жыл бұрын

Nice

@ShivamGupta-dj3eh 4 жыл бұрын

It's not working for me. Showing error: boto3.exceptions.S3UploadFailedError: Failed to upload upload_data_S3.py to new-bucket/python/upload_data_S3.py: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

@BekaBelle Жыл бұрын

Thanks Derrick! What if i wanted to upload csv files one each day and to also check if the file is already in s3 then dont upload?

@islauddin 2 жыл бұрын

Looking for a way to transfer files from our Sharepoint location to a AWS S3 bucket using Python. Any good suggestions on how to go about doing it?

@philippephilippe1265 3 жыл бұрын

Hi Derrick and all I tried this method and I am getting the error below ImportError: cannot import name 'access_key' from 'secrets' (C:\Program Files\Python37\lib\secrets.py) did anyone had same error? and how do you go about it?

@rupeshbhuju2897 2 жыл бұрын

Hi there! If you could suggest some python approach to write a file into multiple target buckets from source bucket, that would be helpful. Thanks

@withkonan Жыл бұрын

how can i upload a 1 gb video to s3 bucket quickly . or if i could upload it asynchronosly(i.e., my page should get redirected to next page and the video can get uploaded at background)

@omkarbabar4420 3 жыл бұрын

Hi.. Can you please tell how to do same thing on heroku platform?

@sarahkim3856 3 жыл бұрын

Hi, I was trying this and I'm getting an error of "FileNotFoundError: [WinError 2] The system cannot find the file specified:" I tried searching it everywhere but I couldn't figure it out. Any advice?

@priyankapatel9461 2 жыл бұрын

How can I Write a Python code to create AMI and send email to all for AMI creation. also create spot instance on based of backup AMI?

@parag4982 3 жыл бұрын

what to do when the s3 has multiple buckets and we need to choose a particular bucket??

@mwredfern 2 жыл бұрын

just reviewed another video from someone that literally walked through this same scenario. however, their video was 57 mins! no. i did not watch the 57 min video. i watched about 3 mins and saw your video sitting in the sidebar screaming '7 minute video over here!' haha. to the point and just the facts. no showboating knowledge and skills. if im ever drowning, i hope you are on the shore to save me and not the person who makes a 57 min video on 'how to jump in and save a drowning person'. haha

@un_known7299 2 жыл бұрын

How to securely insert the Access key and password to the python or c# Because using your method any one can decompile my program and have my access key and secret key ?

@raja4usekhar 3 жыл бұрын

Can you do /suggest best automation tools to work with aws

@soniathomase7757 Жыл бұрын

Do you have any idea how i can access files in a s3 bucket with just my IAM role? I dont have the access keys. Is this possible?

@hesamulhaque2197 2 жыл бұрын

Great video like always! Can you also make a video on cron job?

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@rahulmeena79 7 ай бұрын

I uploaded files to S3 yeahhhhhhh

@xin6129 3 жыл бұрын

This is great! Thanks for sharing! Could you also do a FTP one? I am very new to python/

@DarkStrado 2 жыл бұрын

You should never use FTP anymore. SCP is more secure.

@lucasrosa7781 2 жыл бұрын

Hi Derrick!! I want to know with its possible to do this with a AWS Lambda for automation pourposes?? (sorry about the english, im from Brazil)

@senthilcm 3 жыл бұрын

Hi How can read the files browser for S3

@jamilmjq Жыл бұрын

need idea how to copy files to s3 bucket in UC4

@carlbjorklund2703 3 жыл бұрын

Hi, thanks for great video. I did though run into an issue if you have time. I followed the tutorial and I can get the file (image.jpg) from my images directory (I see this image by running print("file", file) in the os.listdir for-loop) but when I run: "client.upload_file(file, upload_file_bucket, upload_file_key)" I get error "FileNotFoundError; [WinError 2] Can't find the file 'image.jpg' Do you have an idea what the issue can be? Thanks

@tahmeedhossain2910 3 жыл бұрын

Did you find any solution for this ? I am having the same issue.

@tahep3906 2 жыл бұрын

same with me, any solution?

@gregf9160 4 жыл бұрын

More on AWS and Python would be great!

@harisantho Жыл бұрын

Hi.. thanks for this video How to copy one file from ec2 instance particular path to s3 bucket

@celsoalbertoreyesugalde4792 3 жыл бұрын

Excellent video!!, could you do something similar but uploading automatically to Google Cloud Platform?

@rishvish 2 жыл бұрын

Can we download files from AWS in any similar way?

@MichaelDavis-hk8vl 3 жыл бұрын

Can I do this from Juypter Notebook?

@alejandroaparicioguerra1818 3 жыл бұрын

Good sample could you please make an example of how to send email with lambda too

@rattaponinsawangwong5482 4 жыл бұрын

I wonder if you could do the same with google drive? Google drive is more popular in my region.

@wilianuhlmann5284 3 жыл бұрын

how to copy file in other directory?

@mayukhchattaraj5552 3 жыл бұрын

Please upload to a code for S3 bucket to S3 bucket transfer of files.

@narendrasuryawanshi5815 4 жыл бұрын

Can system run a code even its shutdown. Lets say i don't work on Saturday & want to run a code. Will it possible?

@Byomakesh1 4 жыл бұрын

Narendra Suryawanshi For that you need to deploy your script on a server and schedule it

@rickramirez94 3 жыл бұрын

Do a aws series !

@ajtam05 2 жыл бұрын

Are you able to download, as well?

@MrPawan412 2 жыл бұрын

Awesome Derrick , I have one requirement where i need to upload the file from S3 to Sharepoint Site using python in AWS EC2. Any help please ???

@deepquest 2 жыл бұрын

All about S3 Bucket; Subscribe for more such videos. kzfaq.info/get/bejne/otCkpbWIm5_eYWQ.html

@mwredfern 2 жыл бұрын

if you have a video for SFTP, please share link. :)

@preyasprathap 4 жыл бұрын

Nice video.. i am thinking of directly deleting the files after it gets uploaded.

@CodeWithDerrick 4 жыл бұрын

Thanks! You would just add this inside your for loop after the upload statement: os.remove(file)

@mukeshk2553 Жыл бұрын

Bucket to bucket automation how to do

@harisantho 2 жыл бұрын

Hi 👋 thanks for your valuable KT. I'm requesting you to help me with a automated python script to check RDS snapshot name and snapshot created time for multiple Rds in multiple regions. Like N.V, ohio, Sydney, Singapore, Ireland, Paris, Belgium, etc..