Big Data Processing Using Distributed Maps and AWS Step Functions (S3 + Lambda)

Рет қаралды 11,940

Күн бұрын

AWS Step Functions are a powerful orchestration or workflow service. Distributed Maps are a feature that help you implement a series of tasks in parallel. They offer much higher scale over normal Step Function Maps and have useful features like S3 CSV / JSON readers for your event source. In this video, I show you how to use the distributed map feature by processing records in a CSV file located in S3 using a Lambda Function.
Sign up for my Newsletter to receive regular AWS updates AND get a FREE PDF for 5 AWS Project Ideas: beabetterdev.com/aws-project-...
📚 My Courses 📚
AWS Learning Accelerator - Learn AWS Through a Hands On Project - courses.beabetterdev.com/cour...
AWS Lambda - A Practical Guide - www.udemy.com/course/aws-lamb...
🎉SUPPORT BE A BETTER DEV🎉
Become a Patron: / beabetterdev
📚 MY RECOMMENDED READING LIST FOR SOFTWARE DEVELOPERS📚
Clean Code - amzn.to/37T7xdP
Clean Architecture - amzn.to/3sCEGCe
Head First Design Patterns - amzn.to/37WXAMy
Domain Driven Design - amzn.to/3aWSW2W
Code Complete - amzn.to/3ksQDrB
The Pragmatic Programmer - amzn.to/3uH4kaQ
Algorithms - amzn.to/3syvyP5
Working Effectively with Legacy Code - amzn.to/3kvMza7
Refactoring - amzn.to/3r6FQ8U
🎙 MY RECORDING EQUIPMENT 🎙
Shure SM58 Microphone - amzn.to/3r5Hrf9
Behringer UM2 Audio Interface - amzn.to/2MuEllM
XLR Cable - amzn.to/3uGyZFx
Acoustic Sound Absorbing Foam Panels - amzn.to/3ktIrY6
Desk Microphone Mount - amzn.to/3qXMVIO
Logitech C920s Webcam - amzn.to/303zGu9
Fujilm XS10 Camera - amzn.to/3uGa30E
Fujifilm XF 35mm F2 Lens - amzn.to/3rentPe
Neewer 2 Piece Studio Lights - amzn.to/3uyoa8p
💻 MY DESKTOP EQUIPMENT 💻
Dell 34 inch Ultrawide Monitor - amzn.to/2NJwph6
Autonomous ErgoChair 2 - bit.ly/2YzomEm
Autonomous SmartDesk 2 Standing Desk - bit.ly/2YzomEm
MX Master 3 Productivity Mouse - amzn.to/3aYwKVZ
Das Keyboard Prime 13 MX Brown Mechanical- amzn.to/3uH6VBF
Veikk A15 Drawing Tablet - amzn.to/3uBRWsN
🌎 Find me here:
Twitter - / beabetterdevv
Instagram - / beabetterdevv
Patreon - Donations help fund additional content - / beabetterdev
#stepfuctions
#aws
#serverless

Пікірлер: 15

@Langstonrocks Жыл бұрын

Thanks pal, I've been loving your videos for years and this one helped me to quickly solve a current task at my job!

@andyweeks2216 Жыл бұрын

Can't wait for you Step Function Course, Daniel. Thanks a bunch for this video.

@haiderh1339 Жыл бұрын

I was just researching on this topic for a project and saw you uploaded newest video about it 4 hours ago :D

@mariumbegum7325 Жыл бұрын

Great content!

@tamaskiss3237 Күн бұрын

Is there a way to overcome the overhead what map run adds to the overall state machine execution? The execution time seems to be around 8s in your video but the individual lambda executions seem to be ready around 2-300 ms. Is there any recommendation for an alternative solution if latency is critical (around 5s)?

@deepak.rocks. Жыл бұрын

Great 👍

@WiredMartian 2 ай бұрын

Is there a way to preserve order of execution here? Suppose I need to aggregate results from the CSV and I need to maintain the original order of items from the input CSV.

@artbart9080 17 күн бұрын

Hi.Tried to reproduce and stuck with error:States.ExceedToleratedFailureThreshold.Cause:The specified tolerated failure threshold was exceeded. CSV file was an issue. Initially I saved excel file with test data as CSV UTF-8 and after error I saved as CSV. Execution succeeded.

@vinodreddy1722 7 ай бұрын

How to store the data in csv after modification?

@tello9504 Жыл бұрын

How could we show this in github? I know the first step would be to create a design doc about the architecture but I would like to know if you have any examples. I want to put together a portfolio to showcase my work but I would like to explain it effectively on my github.

@MrAbdel776 Жыл бұрын

A great channel. Thank you! I have tried this code on a large csv file with half a million records. Unfortunately, it takes forever. I am not sure what is wrong. I hope anyone can provide some help.

@BeABetterDev Жыл бұрын

Hi there! Are you sure you enabled the "Distributed Map" mode and not using inline?

@MrAbdel776 Жыл бұрын

@@BeABetterDev Thank you for your response. Yes, I used the "Distributed" mode, as you indicated in the video. The code runs reasonably with a small number of records. However, with a large file, it takes more than an hour.

@MrAbdel776 Жыл бұрын

I tried the batching and it makes a big difference. I use batching of 250 elements. I am able to read 50,000 records in 17 seconds. I will try the 500,000 soon. Thank you!