How to Run a Python Docker Image on AWS Lambda

Рет қаралды 28,780

Күн бұрын

Learn how to run a Python Docker image on AWS Lambda using Lambda container images.
Using Docker containers for your Python app on AWS Lambda means you can create a special environment for your app with its own tools and settings, making sure it works everywhere, and also making it easy to include all the things (libraries and packages)your app needs to run correctly.
🛠️ Project Code: github.com/pixegami/aws-lambd...
🔗 Links
Lambda Base Images: docs.aws.amazon.com/lambda/la...
AWS CLI: aws.amazon.com/cli/
AWS CDK: docs.aws.amazon.com/cdk/v2/gu...
Docker: www.docker.com/
📚 Chapters
00:00 Why run Docker on AWS Lambda?
01:04 Create CDK Project
01:45 Create a Python handler app
04:07 Create Dockerfile for Lambda
06:23 Test the Docker image locally
07:11 Create AWS Docker Lambda with CDK
09:44 Deploying to AWS
11:59 Wrapping up
#pixegami #aws

Пікірлер: 103

@tangled55 4 ай бұрын

Dude, I cannot tell you how AMAZING you are and how AMAZING this video is. I could literally hug you through the screen. Thank you so much for your video man. It was a PERFECT explanation.

@pixegami 4 ай бұрын

Thank you! I'm glad it was helpful.

@sweetymjct 6 ай бұрын

You are brilliant!! Following your steps, I deployed several Python apps to AWS Lambda. Your instructions were clear, code was perfect (worked fine without any changes) and explanations were precise. Very rare in free KZfaq tutorials. Thank you.

@pixegami 5 ай бұрын

That's awesome feedback. Thanks for sharing that! Glad it was useful for you.

@igortrindade-dev 5 ай бұрын

Thank you for sharing this! It saves me a lot of time trying to create a bunch of layers to execute some heavy deps functions on AWS Lambda 🤯🙏

@pixegami 5 ай бұрын

You're welcome! I'm glad to hear it helped :)

@visheshpandey2001 2 ай бұрын

How do you even know what we want? Short simple to the point explanation. Thank you so much. Please keep creating such amazing videos!

@pixegami 2 ай бұрын

Thank you, I appreciate your comment, and I'm glad you enjoyed it!

@eb8931 3 ай бұрын

Fantastic video. It covered so much (CDK, Docker, Lambda) in just 13 minutes. And it actually all works really easily. My first interaction with lambda was a few years ago and decided I would never, ever touch it. But after this video, with CDK and the other progress since then, I am reconsidering moving my workflows over to lambda instead of FastAPI. Pretty much all because of watching this video.

@pixegami 3 ай бұрын

Thank you! I really appreciate this kind of feedback, and honestly it's hearing things like this that motivate me to make this content. I hope that project all goes well for you!

@alphasauroxviix4235 Ай бұрын

Thank you for putting so much effort into this video, the quality is just magnificent. Your channel is super underrated, I hope you get the recognition that you deserve soon.

@pixegami Ай бұрын

Thank you! Really glad to hear you enjoyed this :)

@therydn784 23 күн бұрын

This is was so helpful! Thank you so much!

@user-ng2zv3ro5s 8 ай бұрын

I'm very thankful for this video. I really struggle with GUI tools like websites. This command line interface solution really helped me to ship and urgent work I was struggling to configure on aws web interface. Thank you for your work.

@pixegami 8 ай бұрын

Glad it helped!

@champikadilshan5920 Ай бұрын

Perfect! This video is brilliant, If anyone need to Run a Python Docker Image on AWS Lambda. Check this, You will definitely able to solve your issue

@pixegami Ай бұрын

Glad it was useful!

@Acumen928 11 ай бұрын

omg exactly what i needed thanks so much!!

@pixegami 11 ай бұрын

You're welcome!

@beratgenc7695 3 ай бұрын

This is perfect! Great work, thanks!!!

@pixegami 3 ай бұрын

Glad it was useful, thank you!

@frkael8255 8 ай бұрын

New skill added to my resume, thank you so much :D

@pixegami 8 ай бұрын

Congrats! Well done :)

@__sjduebsj Ай бұрын

Great video! Thanks :)

@anaves 9 ай бұрын

wow! Excelent video. Thanks so much!

@pixegami 8 ай бұрын

You're welcome!

@BrannigansLaw21 4 ай бұрын

You've earned a sub my man.

@user-ns4zu3re8q 6 ай бұрын

Thank you, gained a sub.

@pixegami 6 ай бұрын

Thank you :)

@user-vi5tp4xd8i 7 ай бұрын

I absolutely needed this bro. Thank you so much. +1 follow

@pixegami 7 ай бұрын

Thank you! Glad it helped :)

@user-fm4fx2jf4x 5 ай бұрын

Great. thank you

@pixegami 5 ай бұрын

No worries!

@bengoudifaoussama7020 4 ай бұрын

hello thank you for your video , i have a small question , in my lamda function i have some api keys ,how to do the .env config in aws with docker

@immanuelpeter1579 Ай бұрын

What if we want to use env variables? Where would we place them and how can we retrieve them in the py file?

@davidcasanova2951 11 ай бұрын

Thanks a lot ! Nice tutorial. Do you plan to make a video tutorial for a larger project, such as building a full-stack website or something similar ?

@dchenk 11 ай бұрын

Just use docker-compose for this.

@pixegami 10 ай бұрын

Thanks! I do plan some in the coming months, but right now I'm just trying to build a base of smaller videos (like this one) to cover all the basic tools first :)

@ueiwqoak 8 ай бұрын

@@pixegami bump on @davidcasanova2951 's request. Fantastic video

@spare4337 2 ай бұрын

THANKS

@pixegami 2 ай бұрын

You're welcome!

@jeffwoolfe 11 ай бұрын

Great video! When creating the CDK project why do you choose typescript as your language and not python? When I choose python as the language the lib folder is created in the venv folder.

@pixegami 10 ай бұрын

I tried Python before and I'm not sure I like the experience as much as Typescript. CDK is natively developed in Typescript, so I feel it makes the quality of code hinting better, error messages easier to read (without all the JSII stuff they used to translate it to Python) and most importantly, it has the largest example base online and in tools like GPT.

@terryyyzhang 7 ай бұрын

Thanks. It could be more helpful if you walk through the permission setting in IAM, as I end up spending an hour configuring that during bootstrapping as a newbie.

@pixegami 7 ай бұрын

Ah I see, thanks for sharing that. IAM is one of those things that I often forget can sometimes be very challenging unless you've used it before. Noted for next time!

@hamishrobertson946 7 ай бұрын

fyi for me these issues popped up around the 943 mark (Deploy to AWS). Assuming you are running as a managed user you'll need to add the appropriate policies to your IAM group as 'Permission denied' starts popping up each time you run cdk deploy/bootstrap. I had to add stuff like AmazonS3FullAccess and AmazonSSMFullAccess

@jeromedormoy6955 9 ай бұрын

king

@jaxontn 4 ай бұрын

Hii can you make a new video for Docker Python API deployment?

@jayanths2788 2 ай бұрын

Error saying invalid entry point i am using arm64?? Im using m2 any fixes?

@Randomize-md3bt 3 ай бұрын

I came here after trying to install my libraries to my python lambda function and it give up so I am trying to upload it as a docker image

@pixegami 3 ай бұрын

Yup, uploading a library with Python *without* Docker is surprisingly difficult, especially if you want to use libraries that have an OS-dependent library under the hood (which includes AI/ML libraries, numpy, and more). That's why the Docker way is really good to know :)

@eugenechua7239 13 күн бұрын

I don't quite get the part on LAMBA_TASK_ROOT? is it an environment variable that u need to set up when u first create the lambda function? the video doesn't seem to cover it..

@muustafa 8 ай бұрын

thanks for tutorial. What is your terminal that suggest commands?

@pixegami 8 ай бұрын

I use fish terminal. See this video for my Mac terminal setup: kzfaq.info/get/bejne/q82noKuQ3eDPc3U.html

@prajganesh 5 ай бұрын

Great explanation. What are ways you can pass parameters to the container?

@pixegami 5 ай бұрын

Thank you! Well, it depends on what kind of parameters you'd like to pass. Environment variables are probably an obvious and easy way to get started. You could also potentially modify/override the "command" or the "entry point" it is executed with.

@claudiolaas456 8 ай бұрын

Would this setup also work to deploy a streamlit app? Or is it only for request-response type apps?

@pixegami 8 ай бұрын

Ah, I don't think it works well for Streamlit. Streamlit relies on a session and an active connection with the user, but Lambda doesn't really do that. But stay tuned, I'm also working on a tutorial for deploying Streamlit with Docker to AWS as well.

@volodymyrtruba7016 4 ай бұрын

Fantastic video! However, how do we handle logic that takes more than 3 seconds to execute, considering Discord messages have a 3-second limit?

@Acumen928 11 ай бұрын

im trying to create a docker image for a python lambda from an amplify app, how would this work for that setup?

@pixegami 10 ай бұрын

I haven't used Amplify, but from what I understand it creates and manages the Lambda runtime for you. I'm not sure if you can extend it to support images. Let me know if you find out otherwise. That's why for more fine-grained control Iike to use AWS CDK for the infrastructure.

@erikcoding1238 9 ай бұрын

what is the terminal autocomplete feature you are using it's nice how it autocompletes for you

@jeronimoahumada7817 8 ай бұрын

Looks like zsh-autosuggestions

@pixegami 8 ай бұрын

Yup! Or you can use `fish` terminal and get it out of the box. I have a video on the terminal setup here: kzfaq.info/get/bejne/q82noKuQ3eDPc3U.html

@techaisolution Ай бұрын

Hi, this setup spike my billing very high, The setup was to build lambda function to read the latest file from the s3 dir and make transformation then finally to s3 target dir, So this all setup with the python script has to run once the s3 notification to lambda function that an file just came to s3. But it went into a loop and made the s3 and lambda billing spike Let me knew what is the issue in my setup that i didn't noticed at first while running this python script in lambda

@pixegami Ай бұрын

Uh oh. It sounds like you have a recursive Lambda function. I guess after you update the S3 file, it probably triggers another event (which triggers the function) in a loop. Normally, if you want to trigger Lambda from an S3 event, I wouldn't recommend to update/write back into the same S3 - maybe just use a different bucket for the output. For the billing spike, you can create a ticket to AWS support to explain the mistake, and see if they'll be able to give you a refund on the cost (it usually works if it's the first time you are making an honest mistake).

@saml.8295 10 ай бұрын

I follow your steps and faild to build on the RUN pip install ... I commented it out and can build the image. What it take to get pip command to work? I use Windows 10 and curl command (in PowerShell) does not work with the URL. What's the ../2015-03-03/... for?

@pixegami 9 ай бұрын

Hmm, what error do you see when you try to run pip install?

@mohamedabdo-xg6tw 9 ай бұрын

@@pixegami I built my image, it runs with no errors but when I do Curl nothing happens. so, as saml asked, what ../2015/... stands for? or where it comes from?

@kepitingtiara 5 ай бұрын

@@mohamedabdo-xg6tw , I did have same questions, here the GPT help me The 2015-03-31 represents the version of the AWS Lambda API. When you send an HTTP POST request to this endpoint, it triggers the Lambda function handler specified in your Docker container.

@praveenpoonja95 5 ай бұрын

Amazing. I'm facing an issue in Dockerfile in "RUN pip install -r requirements.txt". when I run cdk deploy, I get Failed to build asset error. In local, it is running correctly. Any idea how to resolve this? meanwhile I commented 'RUN pip install -r requirements.txt' and returned just dummy message without numpy. This time Cdk deploy worked and the url was giving me the json response

@pixegami 5 ай бұрын

Hmm, I think we'll need to see what the actual error is - the build could fail for a variety of different reasons.

@crabmeat88 2 ай бұрын

Is there anything extra I have to do to get environment variables working with this setup?

@pixegami 2 ай бұрын

You should be able to apply environment variables via the CDK construct for Lambda. See the `environment` parameter: docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html

@juanfranciscoestradagonzal7826 4 ай бұрын

Does the cost of lambdas change with the use of ECR? Or is there any additional cost generated by using ECR?

@pixegami 4 ай бұрын

ECR cost would be billed separately to Lambda. I think ECR also has a free tier? aws.amazon.com/ecr/pricing/

@andreaizquierdobouldstridg2926 5 ай бұрын

Trying your tutorial on windows, prompted the error: 'curl: (52) Empty reply from server'. Any help will be apreciated!

@pixegami 5 ай бұрын

Is this for when you are testing it locally? What about the Lambda docker output itself? In the terminal where you are running the image locally, do you see any messages?

@RatherBeCancelledThanHandled 6 ай бұрын

Solution : on M chip Mac : change credstore from desktop to osxkeychain , use sudo nano ~/.docker/config.json Thanks for sharing but I’m getting the error ERROR: failed to solve: public.ecr.aws/lambda/python:3.11: error getting credentials - err: exit status 1, out: ``. Can you please advise a solution ?

@pixegami 5 ай бұрын

Hmm, sounds like a permissions issue. I recommend checking how you are setting up your AWS credentials for this. AWS goes through a series of steps to try and determine your credentials: docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html#credentialProviderChain

@arthurpiresdafonseca5081 7 ай бұрын

07:02 it's interesting that I've done the same with Postman and it didn't work. CURL works, though

@pixegami 7 ай бұрын

Hmm, I wonder why. Does it throw an error?

@aacesta 7 ай бұрын

On the pre-requisites. How to install cdk on aws 2023 linux image. sudo yum install nodejs npm sudo npm install -g aws-cdk sudo npm install -g npm@10.2.5 sudo npm install -g aws-cdk

@KillianTattan 2 ай бұрын

Great video, thank you! Wondering how this would work if using poetry instead of requirements.txt? I've seen a lot of different answers on SO and yotube.

@rexthundathil3828 3 ай бұрын

Is it possible to deploy docker image using terraform? If so, can you do a video on this please?

@pixegami 3 ай бұрын

I'm gonna guess it's possible, but I haven't used Terraform before. I'll add it to my list to check out!

@user-kf4zg8pn3z 8 ай бұрын

does not work for me, my curl command gets stuck without any response

@pixegami 8 ай бұрын

Have you run the container successfully? Do you see any output from the container terminal?

@itsanishjain 6 ай бұрын

I wish real world senerios are this simple lol, I am currently using node and canvas and that's needed a lot of Os level dependies like gcc-c++ cairo-devel etc, how to cache them because everytime I make changes I need to build docker image again and that is taking too much time

@pixegami 5 ай бұрын

Well, tutorials and examples are always going to be a simpler cleaner version of real world problems which does have its own challenges :) But your problem does have a solution. Docker builds images in layers, so anything you want to update/change, just put them in a later layer (e.g. lower in the Dockerfile) so it doesn't have to rebuild everything up until that point. That way it'll just re-build your changes into the image without having to go through the dependencies installation again. If you want to take it a step further, check out using GitHub actions to automate your workflow and deployment (it also has caching): kzfaq.info/get/bejne/b9t9lrFnm9PKp3k.html

@aacesta 7 ай бұрын

On the pre-requisites. How to instal Docker on aws 2023 linux image. sudo yum update -y sudo yum install -y docker sudo systemctl start docker sudo systemctl enable docker sudo groupadd docker sudo usermod -aG docker $USER docker version

@pixegami 7 ай бұрын

Thanks for sharing this!

@aacesta 7 ай бұрын

After the above you may also have to restart your machine before commands such as docker ps or similar can succeed in connecting to the daemon.

@DeepakKumar-nl5jr 8 ай бұрын

Nice video. Thanks @pixegami

@pixegami 8 ай бұрын

Thank you!

@TC-gz1ko 4 ай бұрын

Dumb question, what terminal are you using?

@SiddharthSharma360 7 ай бұрын

It's not python it's typescript

@pixegami 7 ай бұрын

The Docker image is a Python app. If you mean the CDK - yes that's Typescript. You can use Python for it too, but I generally go with Typescript since it's the 'native' language for CDK.