Lambda Error Handling features for Kinesis Data Streams and DynamoDB

Lambda Error Handling features for Kinesis Data Streams and DynamoDB | Kinesis Error Handling Demo

Рет қаралды 2,290

Күн бұрын

Kinesis Error Handling if using Lambda as a consumer. When using Lambda as a consumer, it becomes very tricky to handle poison messages (or problematic messages) which can cause your Lambda to fail and keep on re-trying the same batch of messages again and again.
In this video, I have covered Lambda Error Handling features and Retries in the case of Kinesis Data Streams.
Do watch the theory part if you are not clear about the tricky issues which you might encounter if you are using Lambda with Kinesis and not using Lambda Error Handling features. The link is below:
Lambda Error Handling with Kinesis (Theory Part): • Lambda Error Handling ...
Kinesis Producer Example (with Java): • Kinesis Data Streams J...
GitHub Repo: github.com/ajaywadhara/Lambda...
Chapters:
00:00 Introduction
01:17 Create Lambda
01:40 Create Kinesis Data Stream
02:07 Add Kinesis Trigger to Lambda
03:29 Send Stream of Messages to Kinesis
05:02 Write Lambda Code
09:45 Test Lambda with Error Handling features
12:26 Add Error Handling options in Lambda
14:19 Test Lambda Again with Split Batch and Retries
16:02 Check SQS (DLQ)
17:40 Complete Flow with Error Handling
#KinesisErrorHandling #LambdaTutorials #AwsTutorials

Пікірлер: 12

@mrnuke1 13 күн бұрын

I watched all 4 of such videos and found them very helpful! Thanks a lot!

@asupat12 Жыл бұрын

Thank you for the video. Such videos are very practical and to the point.

@nitinkulkarni7942 2 жыл бұрын

Excellent Tutorial.. Very well done. Thank you

@nitinkulkarni7942 2 жыл бұрын

I cant believe there are only 434 views. This deserves a lot of views.

@AjayWadhara 2 жыл бұрын

😀 KZfaq’s algorithm is harsh. But i do get motivated by such comments. Thanks 🙏🏻

@eshwarnag Жыл бұрын

Excellent Demo! Answered all my questions.

@AjayWadhara Жыл бұрын

Glad you liked it

@nitinkulkarni7942 Жыл бұрын

Incase of bisect function on error, lets say we get 100 messages. We process first 49 messages and publish them to SNS topic. Then the 50th message is an issue. In this case the batch will be divided into 2. I am still going to get all the messages that I already published to SNS right? I will end up publishing the messages multiple times until that bad record is isolated. How do we solve this issue?

@AjayWadhara Жыл бұрын

Hey Nitin, you have to think about making your system Idempotent. You might have to add another service such as DynamoDb that takes care of this and use SNS with DynamoDb streams. Also think about Event Filtering if that is possible in your use case

@meashish Жыл бұрын

I wanted to know, how to redrive the messages in DLQ back to Lambda once I fixed the Lambda code.

@meashish Жыл бұрын

I had to write a script for redriving the message in DLQ. 1. The script first take the shardId and sequenceNumber from the DLQ message. 2. Fetches the record body from Kinesis using shardId and sequenceNumber. 3. Construct the Lambda input. 4. Invoke the Lambda. 5. If lambda invocation is successful then delete the message from DLQ. If it is more than 24 hours and the data retention for Kinesis is 24 hours then you won't be able to get the message from Kinesis later. Default retention period for kinesis is 24 hours.

@AjayWadhara Жыл бұрын

Perfect solution. 👍