Upload files to S3 using API Gateway

Upload files to S3 using API Gateway - Step by Step Tutorial

Рет қаралды 58,505

Күн бұрын

In this video, I show how to upload files using API Gateway and S3 integration. In the first part of the video, I explain the required IAM role and policy to run the demo. After that, I create the API Gateway with the S3 bucket as the backend service. I'll show how to map the path params from the API Gateway to the S3 bucket and object key.
00:00 Introduction
01:20 Solution Architecture
03:13 IAM Role and Policy Creation
05:30 Create API Gateway
09:33 Deploy the API
If you want to learn more about AWS Services, make sure to
subscribe to the channel:
KZfaq 🎥 - / @enricoportolan
Medium: / enrico-portolan
🌎 Find me here:
Twitter - / enricop89

Пікірлер: 115

@jonty0001 2 жыл бұрын

This is a great video on API gateway integration with S3. Enrico, thanks so much for sharing this knowledge with the community.

@KyleMerl Жыл бұрын

For anyone getting an "internal server error", I had to enable cloudwatch logs in order to debug, and discovered I had a typo in my "path override". You can enable cloudwatch logs by going to "stages", select your stage, click the Logs/Tracing tab, and then enable cloudwatch logs. Keep in mind I think cloudwatch logs do incur costs, so turn them off when you're finished.

@sandeeppanchal8615 2 жыл бұрын

Clean, clear and straight explanation. It solved my problem in 10 minutes. Good start with your video.

@EnricoPortolan 2 жыл бұрын

Thanks 🙏

@purplishjen Жыл бұрын

Clear and straight to the point. Good job!

@Leonquenix 2 жыл бұрын

You are a lifesaver! This saved me so much time. Thank you so much!

@yuming123 2 жыл бұрын

Great video and just cover enough details for me to pick up in 10 mins, cheers !!

@paul-macfarlane Жыл бұрын

Very helpful, thanks! I initially setup API Gateway with a Lambda Integration because I didn't know this was possible, but I definitely agree this is the preferred approach since with Lambda you have to deal with cold starts and managing code.

@jonathanvargasv2009 2 жыл бұрын

Practical, helpful, and precise. Thanks for all your help.

@markcheng7254 2 жыл бұрын

Worked perfectly. Amazing stuff!!

@nicktran8595 2 жыл бұрын

Thank you so much. I just got stucked with AWS doc. You saved me :)

@arpankhetani7818 2 жыл бұрын

Thank you for your explanation.. Very well explained and you kept it same..

@glassin1093 2 жыл бұрын

Great thanks for this video - exactly what I needed

@karikalanramakrishnan9286 2 жыл бұрын

Excellent guidelines, able to successfully upload png and csv file too. Thanks.

@theautomationchannel6920 2 жыл бұрын

Thanks ..Really great effort..happy to find you channel

@rcapilli 11 ай бұрын

Thank you for this. You’ve saved me a lot of time

@inversebrah 2 жыл бұрын

great tutorial, thank you Enrico!

@RBusch_ Жыл бұрын

Thank you for this tutorial (from Brazil)

@principe.borodin 11 ай бұрын

WOW, it works fantasticly!!!

@manojreddy152 Жыл бұрын

Awesome Job!! Well executed!!

@greendsnow Жыл бұрын

AWS is absolutely ridiculous. I couldn't do it without you. I can't believe how people are put through learning this stuff. I'm actually very angry with the lack of intuition here.

@greendsnow Жыл бұрын

now every file uploads but none of them opens...

@federicovdemartino4915 2 жыл бұрын

Man, you cannot imagine how you saved my live here!

@EnricoPortolan 2 жыл бұрын

Great to hear!

@ahack210 2 жыл бұрын

Very useful video, thanks!

@undertaker9138 2 жыл бұрын

Very helpful, thanks.

@WhiteJass 10 ай бұрын

legend! very helpful thanks a lot!

@ezequielsanchezventurini2184 2 жыл бұрын

Thanks, this is really useful

@Bemidbar 2 жыл бұрын

Super! Thank you!

@nil.hunter Жыл бұрын

I don't understand why do I need to use PUT instead of POST to upload a file in a folder in S3. Could someone to break it down, please?

@Ritunjan 2 жыл бұрын

Great thanks it is awesome!!

@tsielomudau1849 2 жыл бұрын

it worked for me thanks

@gujg2010 Жыл бұрын

save my day, Thanks

@nevilholmes5900 Жыл бұрын

thank you so much

@hebahijjawe3489 2 жыл бұрын

Thank you, i have a question ... can i send file ACL list in Simple Storage Service (S3) PUT ? because i need the file public read access

@gregbanister 2 жыл бұрын

Would it be possible to create a single SAM template that will generate all of the resources and permissions. If so, an example would be great.

@munozhoracio 2 жыл бұрын

help, MethodNotAllowed The specified method is not allowed against this resource. PUT SERVICE ..... ....+...+...=

@zhongtom2625 Жыл бұрын

Thanks for the video Enrico. Is there a way to get the file url in the api's response? That will be the complete use case of the api. Thanks.

@lifeessense4283 2 жыл бұрын

I had similar requirement but additionally I have to save the path and few additional parameters in db, will the first approach works? Can you explain that as well how to configure.

@yonatantre Жыл бұрын

Hey thank you for the solution. Can you provide serveless yml solution as well for same issue?

@anilpoudel5974 2 жыл бұрын

Sir suppose I have AL3 file in my desktop and I need to convert that format and read using third party API how I will upload file in that API and retrieve that file in particular format without the use of any object just normally passing desktop file.

@joaoluismoraes7215 Жыл бұрын

Is it possible to get a custom response? Once I upload the image, can I get the file URL or something like that?

@harishkp8631 Жыл бұрын

hey, quick question, instead of postman, how can we upload the picture through a frontend code that is in a s3 bucket and this s3 bucket is connected to the same api gateway

@ramyahello Жыл бұрын

Very useful video !!! Thanks a lot. Should we add authentication is it really necessary in this case ? Since u r already using iam role

@nick_jacob Жыл бұрын

Do you have a way to add meta data with the image?

@Sony-il2ti 2 жыл бұрын

please can you explain and clarify a bit what an API resource is and how it is different from a method? thank you

@user-sb8nb5id6c 10 ай бұрын

Hi, Thank you for this demo. Just one question, I want to use the same pattern to send XML structure in memory not stored in a file system to be be then stored as object into S3. Is this working directly using API Gateway or do I need a lambda to manage S3.putObject(buffer)?

@allofmystuffs 2 жыл бұрын

We didn't limit the file size to 4MB. How to do that.?

@geetanshawriteme Жыл бұрын

Can we upload the file with POST method instead of PUT?

@sport-jz8dd 2 жыл бұрын

Hi, Thank you for the Clean, clear, and straight explanation. can you please let me know how to create folders and place images in them using this API.

@EnricoPortolan 2 жыл бұрын

if you specify a folder path in the API, such as images/website/myphoto.jpg, it should create the relative path. I haven't tested but it should work

@n14r Жыл бұрын

Hi Enrico, This is very helpful can you do the same using cloudformation?

@ArjunKumar-df6pr Жыл бұрын

How to give the filename same as what we are uploading from postman

@user-py1xm2zz6g 2 жыл бұрын

After uploading the image, it shows me a white small box in the browser. I also added "*/*" in the binary media types section.

@javascript_developer Жыл бұрын

You might need to re-deploy the api again in order to work.

@stevebuonincontri6853 Жыл бұрын

Thank u This is just what I needed. It turns out my videos are too large for lambda. So it is now going directly to S3. So is it true that lambda has a 6mb max for video processing? My files were even smaller having issues ~5.1mb?? Then I realize there is no way to control how large a user's video upload will be and needed to go directly to S3. Maybe I should have gone directly to transcribe.??????

@sumit84ky 2 жыл бұрын

First of all, Thank you so much for such a wonderful video explaining the problem use case with solution. I have one request, how do we secure the API gateway with basic AUTH or AWS Signature (preferred) ?

@EnricoPortolan 2 жыл бұрын

I think this guide can help you: docs.aws.amazon.com/apigateway/api-reference/signing-requests/

@kushmishra Жыл бұрын

hey , im getting a method not allow error when im calling PUT call to the api through the postman please HELP !

@muralikrish9515 Жыл бұрын

By using api url, can i upload log files from an application

@prashantsingh9645 2 жыл бұрын

It's is showing internal server Error ! tell me what is the problem, or what am i doing wrong for some reason

@harishstealth9112 2 жыл бұрын

How we can Mutiple files Upload ? Any Idea .

@anandbabu01 9 ай бұрын

Configuring bucket is working fine. I want to put the file into specific nested folder in the bucket. Anyone tried this?

@kishores6830 4 ай бұрын

Invalid mapping expression specified: Validation Result: warnings : [], errors : [Invalid mapping expression parameter specified: method.request.path.bucket]. I got this error for adding url path parameters. Can you help me out

@uttamtarsariya8492 2 жыл бұрын

Can Have first approach?

@jasontwinters 2 жыл бұрын

Great tutorial! In my case the bucket name will always be the same. How can I eliminate the bucket name from the URL?

@EnricoPortolan 2 жыл бұрын

You can set a custom dns record with route 53

@lollo4ever 2 жыл бұрын

Great video! It does work on Postman, but what about CORS? What about calling that API from a front-end? Thank you

@EnricoPortolan 2 жыл бұрын

Hi Alessandro, sure. If you are deploying your app with Serverless Framework the easiest way to enable CORS is the following:www.serverless.com/blog/cors-api-gateway-survival-guide/. Otherwise for a more general guide you can follow: docs.aws.amazon.com/apigateway/latest/developerguide/how-to-cors.html

@EnricoPortolan 2 жыл бұрын

Let me know if it helps, otherwise feel free to ask :)

@TheRealKosalaHandlersMail 20 күн бұрын

Can this be done using Terraform only?

@divineer627 2 жыл бұрын

This was very helpful. Is it possible to have a video on uploading files larger than 100MB in S3?

@EnricoPortolan 2 жыл бұрын

Yes it is using multipart upload docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

@lucasvargasquintana8396 Жыл бұрын

Great video! Thanks! I have one question about it: On Postman, I'm getting this error: IllegalLocationConstraintException Message: The unspecified location constraint is incompatible for the region specific endpoint this request was sent to Can you give me a hand on it?

@lucasvargasquintana8396 Жыл бұрын

Nevermind! Solved, API Region must be same as Bucket Region

@raviullagaddi2089 2 жыл бұрын

Thank you so much. and one question How to return s3 file location with this api?

@EnricoPortolan 2 жыл бұрын

I think you can add it on the Response mapping - Body

@raviullagaddi2089 2 жыл бұрын

@@EnricoPortolan : Getting this error : { "message": "Unsupported Media Type" } What is the content type need to pass in header? i have added this --> image/jpeg in binary media type

@sidduk5389 2 жыл бұрын

Great video, what are the solutions fot multiple images and over 10 MB size?

@EnricoPortolan 2 жыл бұрын

The best approach is to use multipart upload docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

@calm2584 2 жыл бұрын

How to fetch these files

@jiongnanliu5541 2 жыл бұрын

some follow up questions, I followed your approach and successfully uploaded object to S3 bucket, but is there way to add metadata value at upload time? I added the x-amz-meta-* key in the headers according to the docs, but failed. Do you know why? Thanks!

@EnricoPortolan 2 жыл бұрын

have you tried following this guide: aws.amazon.com/premiumsupport/knowledge-center/custom-headers-api-gateway-lambda/ ?

@jiongnanliu5541 2 жыл бұрын

@@EnricoPortolan Thanks! Is there anyway that I don't need to use lambda, directly putting x-amz-meta-* in request headers sent to API gateway, which directly put object and metadata in S3.

@sebascardonac 2 жыл бұрын

@@jiongnanliu5541 I try to do someting similar but I don’t how configurate the apigw to add header to send tags to the bucket.

@nithintm1748 2 жыл бұрын

How to do it with lambda

@deepshikhagarg5661 Жыл бұрын

This was really helpful. Also how can i connect it with my html code. I want to upload image in s3 using UI..

@javascript_developer Жыл бұрын

You can use this url as a restful api url and use html form file upload and call the api on submit

@mjr8771 Жыл бұрын

mine is saying missing authentication token

@user-mt2ud3ir8y 2 жыл бұрын

while successfully uploading the image to S3, I cannot open these images after downloading them from S3.......

@user-py1xm2zz6g 2 жыл бұрын

Have you found the way how to fix this?

@moeyk97 2 жыл бұрын

What if I want to store an RFID Tag Number and then connect the S3 bucket to Elastic Beanstalk? (where my website is located) pls if you have any idea how this can be done let me know I would really really appreciate it, and it would help me greatly with my project.

@EnricoPortolan 2 жыл бұрын

What do you mean by connecting S3 to beanstalk? Can you elaborate please?

@moeyk97 2 жыл бұрын

@@EnricoPortolan I developed a website using html/php/css/javascript and I want my website to interact with the items that I upload to S3, I uploaded the code of the website to Elastic Beanstalk and successfully connected it to a RDS too

@moeyk97 2 жыл бұрын

and when my website can fetch the items from S3 bucket, I want it to behave accordingly to what I have in the RDS, e.g. if what it has read is available on the RDS then it would return the item. I hope that makes sense.

@riazahmad5975 Жыл бұрын

Hello Sir ! please make a vidoe how to upload csv data to dynamodb using nodejs in serverless framework

@lifeankit 2 жыл бұрын

Can you please upload a video with the first architecture? I dont know how to read the uploaded file in lamda. Thank you. Great content. :)

@EnricoPortolan 2 жыл бұрын

I will

@lifeankit 2 жыл бұрын

@@EnricoPortolan thanks 😊

@tnmyk_ 2 жыл бұрын

I got an error on Postman saying "Missing Authentication Token". Can you please help

@EnricoPortolan 2 жыл бұрын

That error usually means you are using the wrong endpoint/url. Make sure you have spelled the url correctly or configured the parameters on the api gateway

@sumit84ky 2 жыл бұрын

I was mistakenly using the "POST" method and hitting the same error. I guess, you are using POST method too in postman, we need to use PUT. Hope this helps.