What makes Kafka special?

What makes Kafka special? | System Design

Рет қаралды 15,643

Күн бұрын

Visit Our Website: interviewpen.com/?...
Join Our Discord (24/7 help): / discord
Like & Subscribe: / @interviewpen
You've probably heard about Kafka, but you really know why everyone is so excited about it? This video dives into the low-level details of what makes Kafka fundamentally different from other message queue systems, and when you should use it over other solutions.
If you liked this video and want to learn more about fundamental systems and how they can be used to solve problems, check out our full course on interviewpen.com/?... !
Table of Contents:
0:00 - Introduction
0:13 - Message Queues
1:32 - Message Routing
2:03 - Kafka Behavior
2:37 - Consumer Offsets
3:25 - Sequential Disk Access
4:51 - Use Cases
6:02 - interviewpen.com
Socials:
Twitter: / interviewpen
LinkedIn: / interviewpen
Website: interviewpen.com/?...

Пікірлер: 39

@prashantkumardhotre5695 Ай бұрын

What I like most about your videos is that they are short, crisp, to the point, great visuals and no BS

@interviewpen Ай бұрын

Thank you, glad you like them!

@harisridhar1668 4 ай бұрын

Thanks for really going over the differences between Kafka versus TMQs - Traditional Message Queues - in this video. It helps to understand Kafka for its benefits with retention, replay, and fanout delivery pattern to multiple consumer nodes/services. Also I like your presentation style and the use of multiple colors to highlight the many components and network passing going.

@interviewpen 4 ай бұрын

Thank you!

@shaikhmubin02 4 ай бұрын

Bro you are doing a great job, keep it up💪🏻

@interviewpen 4 ай бұрын

Thanks!

@mertbozkir 4 ай бұрын

I appreciate you, awesome channel!

@interviewpen 4 ай бұрын

Thanks!

@48_subhambanerjee22 4 ай бұрын

Bro... This is like OS... Semaphoring with producer consumer... There also you send process PCB into queue and when queue is full producer sleeps and when empty reciever sleeps

@sanketagarwal8570 4 ай бұрын

The content is just too good!

@interviewpen 4 ай бұрын

Thank you!

@Labandusette 4 ай бұрын

thank you very much !

@interviewpen 4 ай бұрын

Of course!

@pieter5466 4 ай бұрын

Another great video. A great follow up might be Samza. [edit: Samza as a example of stream processing between queues]

@interviewpen 4 ай бұрын

We'll add it to the backlog! Thanks for watching.

@AshrafMMA 4 ай бұрын

Wow.. now I get why it so much more scalable . How do you determine size of queue, and will you cover usages of multiple queues in kafka?

@interviewpen 4 ай бұрын

We have a full video about using queues in large scale systems on interviewpen.com :)

@rustix3 4 ай бұрын

4:08 Wow the page cache of the operating system need another video on its own. How does it caches, what strategy it uses? What happens if the consumer becomes available and Kafka need to re-read all previous messages? Do OS cache stores anough of messages, should or even can it be twicked?

@interviewpen 4 ай бұрын

Yes--tons of OS level enhancements can be made to optimize for specific usage patterns. This is an advanced topic, but we're working on expanding our coverage of these sorts of things in our course at interviewpen.com . Thanks for watching!

@quachhengtony7651 5 күн бұрын

Can you name some of these traditional message queues?

@martyallen3801 4 ай бұрын

RabbitMQ also provides exchanges and fanout, right? What's the big difference between them in that case?

@interviewpen 4 ай бұрын

Yep--Rabbit would be classified as a traditional MQ involving exchanges and messages delivered to one consumer.

@elstp2 4 ай бұрын

please an review of account system

@interviewpen 4 ай бұрын

Is this what you're looking for? kzfaq.info/get/bejne/q9CPZ9mp0p_ZaWQ.htmlsi=eaD-ZVsmh2Ha2q3N

@69k_gold 4 ай бұрын

So because we have the event dequeueing in traditional MQ we are losing the throughput capabilities?

@interviewpen 4 ай бұрын

It's really more about messages being removed from the queue once consumed--this results in different consumer semantics but makes it harder to optimize for throughput at an OS level. Hope that helps!

@rustix3 4 ай бұрын

3:01 What just happened? Did the offset below(which is the offset for Consumer below) just went directly through 1 and 2? Or was it going through 1 at the same time as the Offset above was going through 1? I mean the offsets and consumers work in parallel, right?

@ygormartinsr 4 ай бұрын

The second consumer hadn't pulled data from the queue yet, so its offset was still at 0. Once it did it received the two events that were sitting there

@interviewpen 4 ай бұрын

Messages can be received by consumers in batch, so when the second consumer polled, it received both messages.

@santoshpinisetty7796 4 ай бұрын

But wouldn't saving all msg waste memory how does it handle the it

@abhishek_ar97 4 ай бұрын

Default time is 7 days, after which the message gets deleted. You can find them in the kafka-config file

@interviewpen 4 ай бұрын

Kafka will automatically delete old messages when it runs out of space :)

@imnormal-sl8mj 4 ай бұрын

Hi, I still don't get why Kafka can get much higher throughput than to traditional MQ, is it because of its scalability? If so, why isn't traditional MQ more scalable?

@interviewpen 4 ай бұрын

Kafka efficiently uses optimizations built into the operating system, such as sequential disk access and the page cache. For reference, this is covered at 3:25 in the video--thanks for watching!

@yuzhao5268 Ай бұрын

helpful but shallow. there's gotta be more details. what if the machines running the queues crash? What if publisher publishes faster than consumer consumes? what protocol is used to communicate? if multiple producers are producing, what orders are enforced?

@interviewpen Ай бұрын

If you want an in-depth video on how queues are used in fault-tolerant systems, check out interviewpen.com :)