What is Kafka and How does it work?

Рет қаралды 140,622

Күн бұрын

In this video I will explain what is Apache Kafka, how does it work and the main components of Kafka such as The broker, connection, Producer, Consumer, Topics, Partitions and Consumer group.
Chapters
0:00 Intro
0:10 Broker
1:10 Connection
2:00 Topics
2:55 Kafka Producer
4:20 Kafka Consumer
5:40 Kafka Partitions
9:30 Pub/Sub vs Queue
10:20 Consumer Group
🏭 Software Architecture Videos
• Software Architecture
💾 Database Engineering Videos
• Database Engineering
🛰 Network Engineering Videos
• Network Engineering
🏰 Load Balancing and Proxies Videos
• Proxies
🐘 Postgres Videos
• PostgresSQL
🚢Docker
• Docker
🧮 Programming Pattern Videos
• Programming Patterns
🛡 Web Security Videos
• Web Security
🦠 HTTP Videos
• HTTP
🐍 Python Videos
• Python by Example
Stay Awesome,
Hussein

Пікірлер: 90

@hnasr 2 жыл бұрын

Learn the fundamentals of database engineering grab my udemy course database.husseinnasser.com (link redirects to udemy with coupon applies)

@Bruh-jw2ze 3 жыл бұрын

Bro I literally went through entire documentation But this solved the consumer group concept in 5 mins 🙌

@k.l7111 3 жыл бұрын

The explanation of consumer group to achieve queue and pubsub is gold. Thank you.

@mahdibentahar9908 3 жыл бұрын

Lolll i Like the Jordan Peterson bit

@ayoolafakoya9841 3 жыл бұрын

Wish someone talks about Machine learning like you do with backend engineering. You are awesome.

@JinayShah 2 жыл бұрын

Checkout Srikanth Verma

@FUTETrends 4 жыл бұрын

Wonderdul info Video about Kafka n its components Thanks Hussein

@Saurabhandsonu1994 4 жыл бұрын

This is amazing content! The one spoof in middle was pretty cool!

@user-im9qy4kw6h 10 ай бұрын

Hussein is hands down the best at explaining complex topics. I love the light-hearted approach - hussein always FTW!

@dengzhonghan5125 3 жыл бұрын

This 15 mins video is better than a 2-hour lecture from my professor.

@Bruh-jw2ze 3 жыл бұрын

*7 min video (at 2x)

@TravisTennies 2 ай бұрын

Yeah. The idiot jobs that REQUIRE a degree... they're idiots and you don't want to work for fools. Smart people learn from any/every thing possible, and better than what schools offer.

@section9999 3 жыл бұрын

Bro, you and Gaurav Sen are my two new favorite youtubers. Free high quality content. Awesome stuff! :D

@hnasr 3 жыл бұрын

DataSurgeon 369 Thank you! I love Gaurav content he is so passionate with his work

@section9999 3 жыл бұрын

@@hnasr Yeah definitely, and he's pretty funny. That makes it a lot easier to say engaged.

@python2448 2 жыл бұрын

Very quick and deep explanation. You made it very interesting.

@amospan14 Жыл бұрын

8:05 Absolutely love that Jordan Peterson reference about how life is suffering. That was awesome! And likely Jordan would add, the antidote to that is voluntarily confronting our own suffering will begin to cure it. Thank you for this Kafka educational video! I very much appreciate your teaching style! =)

@moeyali123 2 ай бұрын

data pipelines are suffering

@romantsyupryk3009 4 жыл бұрын

Thanks so much for this video tutorial.

@basheeral-momani2032 2 жыл бұрын

الله يعطيك العافية مبدع، شكرا جزيلا

@aymanmahmoudabdelshakour4092 3 жыл бұрын

Thanks, Hussein, Great Job!

@Zmey5656 28 күн бұрын

Now I know enough about Kafka after you video, thank you.

@nd8774 Жыл бұрын

your explanation is chef kiss.

@1987eslamsabry 3 жыл бұрын

man , u are awesome. i just wanted to know what Kafka is as it was mentioned in some topics related to my job and found ur video, loved ur explanation and simplicity. well explained bro.... keep it up man.

@hnasr 3 жыл бұрын

Thank you SemSem

@quangminh4847 Жыл бұрын

thank you so much, this video is very helpful. so easy for understanding

@vnd2489 8 ай бұрын

Wowww, your explanation is funny but easy to understand the concept. Many thanks.

@ozanerturk9715 3 жыл бұрын

OMG you just use JB Peterson as a meme in a tutorial. Best combination for me !

@brijeshkp2000 2 жыл бұрын

You are guru, you are awesome, take a bow 🙇‍♀️👏

@brightstar1927 Жыл бұрын

Awesome explanation !

@moeyali123 2 ай бұрын

Great video, entertaining, simple and educational at the same time. Subscribed and liked

@teja.648 3 жыл бұрын

superb!!

@harshitsaxena__ 3 жыл бұрын

Amazing videos , would be great if u can add these slides as well in desciption

@sbylk99 3 жыл бұрын

I have a question "Can the number of partitions changed during execution?" I did some research and share the answer here: 1. Number of partitions can be increased, but can not reduced. 2. When increasing number of partitions, Kafka servers are kept running. No code shut down time. 3. Rebalancing traffic is handled by Kafka, producer will send messages by the new partition and consumer will consume messages from new partitions too. Client does not need to handle the rebalancing.

@KrishnaJaju24 2 жыл бұрын

Best starter video on Apache Kafka. One question: How is the order of message arrival maintained in different partitions?

@ParthShah1504 4 ай бұрын

"Kafka guarantees that messages from a single partition are delivered to a consumer in order. However, there is no guarantee on the ordering of messages coming from different partitions." - from the kafka's published paper

@bhawanajohari 2 жыл бұрын

Very well explained.

@flamencodeveloper4963 3 жыл бұрын

You are a good teacher sir subscribed

@PAVANKUMAR-vj8oi 7 ай бұрын

Best in world tutorial

@thatguyadarsh 2 жыл бұрын

Good to know you too learn fron mr Peterson

@GeorgiiRychko_G 3 жыл бұрын

Amazing! It would be really cool if you could make a video on Apache Kafka streams in NodeJS with demo examples, I’ve noticed that there are no proper resources on this subject.

@hnasr 3 жыл бұрын

Georgii Rychko thanks Georgii!! Will do

@GeorgiiRychko_G 3 жыл бұрын

Thank you a lot!

@anand.prasad502 2 жыл бұрын

true gem

@artemv3160 2 жыл бұрын

Thanks!

@jamasamado6333 Жыл бұрын

Good work Huesein Ramadan Kareem

@miroslavisikiyski4876 2 жыл бұрын

Hussein, what is your opinion about beanstalkd? Compared with Kafka (used as a queue) which one would you prefer to and why? Thanks

@shiwang789 3 жыл бұрын

Nice video. please add a video on activeMQ also

@linkow Жыл бұрын

I literally subscribed because of the Jordan Peterson reference 😂

@jonsnow9246 4 жыл бұрын

Hi Hussein! Great video! 👍 Can you please make a video on elastic search?

@hnasr 4 жыл бұрын

This should be next because sooo many requested it

@testacc9381 2 жыл бұрын

@@hnasr Bro, still waiting for it

@ChristianAltamiranoAyala 3 жыл бұрын

One thing to keep in mind is that kafka hands "offset" which is a grest feature in case of kafka restarts

@amlife180 4 жыл бұрын

Excellent video, Was expecting nodejs Kafka demo tho 😕

@hnasr 4 жыл бұрын

Hey Dean! Watch the full kafka video for the code 🙏

@user-yo8iq3sm5j 3 жыл бұрын

Please describe a Replication factor and how can we add a new consumer group if there is no "free" replica but we need to do it?:)

@claushellsing 4 жыл бұрын

Something that I can't understand about Kafka are zookeepers, what are they? What are they used for? Where are they used? Its something that puzzle me

@hnasr 4 жыл бұрын

R. Rod. Aha you asked about the hardest piece in the puzzle that I cut from this video, checkout the full kafka video (end screen) I cover that in details

@claushellsing 4 жыл бұрын

@@hnasr thanks for you answer, again your content is amazing teacher Hussien

@claushellsing 3 жыл бұрын

@André Rosa what are paxos?

@claushellsing 3 жыл бұрын

@André Rosa thank you sir for your answer

@Emmanuel-px9lk 4 жыл бұрын

Liked the Jordan Peterson clip !

@hnasr 4 жыл бұрын

🦞

@danielequattrocchi2090 3 жыл бұрын

sorry for the question.. but it can be true that kafka seems to be the "facebook" of a distribuited system?

@ducthinh2412 2 жыл бұрын

Just want to clarify: at 12:27, you said: "One partition'd better be consumed by 1 consumer" This only applies to consumers within the same consumer group, correct? Let's say we have: - 2 consumer groups G1 and G2. - G1 has consumer C1 and G2 has consumer C2 Can a partition P be consumed by both C1 and C2 since those consumers are from different consumer groups?

@hnasr 2 жыл бұрын

Thanks that is correct, I do mention this in the full kafka video maybe this part was cut during this highlight. Appreciate it

@shivakumarn7963 2 жыл бұрын

1000th LIKE. :D

@georgiatsatsani4963 3 жыл бұрын

@t The Bucket Rider: From Kafka to a Film Adaptation

@johnsailor3590 2 жыл бұрын

I feel partitions and sharding are completely different things

@lilj1702 2 жыл бұрын

lol Jordan Peterson has made it in the Kafka vid lol.

@nadertarek4822 2 жыл бұрын

Wohooo I got published xDDDD

@shiwang789 3 жыл бұрын

at 8:05 why do subscriber need to know which partition it is publishing the message to? The broker can itself figure out based on the rule of partitioning. And due to some case if it is not possible then do I need to change all my publish requests if add a partitions to an existing topic? That is a very bad design i think.

@hnasr 3 жыл бұрын

I think they made this flexible by allowing the publishing decide which partition, otherwise how would they configure the broker to decide which partition? But i agree with you it has its cost

@shiwang789 3 жыл бұрын

@@hnasr For eg. in the case that you mentioned, (N-Z) goes to partition 2, so Nader will automatically go to partition 2. Am l missing something here?

@AniketSomwanshi-ll7mz Жыл бұрын

@@shiwang789 I agree the broker is absolutely capable of appending to the correct partition by looking at itself

@pvsk10 3 жыл бұрын

Let's give credit where it truly belongs, "Life is Suffering" - The Buddha, 2500 BC

@asknavid7752 3 жыл бұрын

🇧🇩🇧🇩🥰🥰

@tambolaking5383 4 жыл бұрын

Where is Kafka used. In which cases is it used?

@hnasr 4 жыл бұрын

When a simple request/response system isn’t cutting it. If the cost of calculating response is high OR if the same response is required by multiple requester services. having a pub/sub system helps in those cases (kafka is just one example) I give some use cases in KZfaq here kzfaq.info/get/bejne/hZeAmtSG1sCvhHU.html Thanks for being long time subscriber and asking great questions over the years Vrunda.

@tambolaking5383 4 жыл бұрын

@@hnasr I have built a game. It uses subscription provided by Firestore. game.cvfy.in - Will Kafka be useful in replacing Firestore? The game is if Bingo. So, random numbers between 1 and 90 are shown to players and they claim dividends. So, it's all players are on subscription to one document where updates happen.