Apache Kafka Crash Course
Рет қаралды 420,774
Күн бұрынApache Kafka is a distributed stream processing software developed by LinkedIn and written in Scala and Java.
Chapters
0:00 Intro
2:32 Kafka Broker
5:39 Kafka Producer
7:11 Kafka Consumer
8:34 Kafka Partitions
12:50 Queue vs Pub-Sub
16:47 Consumer Group
23:06 ZooKeeper
29:45 Coding Example
1:02:48 Kafka Pros-Cons
1:17:00 Summary
Download slides here
payhip.com/b/Y0Cq
(Members get all slides for all my video courses for free)
In this video I want to focus on the basics of Kafka, talk about how it works give finally spin up a Kafka cluster and write a producer and a consumer.
Kafka Components
_ producer
_consumer
_topic
_partition
How Kafka works
Example ( Ride sharing )
Spin Kafka cluster
Write a producer
Write a consumer
Kafka Applications
Pros and Cons
Source Code
github.com/hnasr/javascript_p...
Docker commands spin up kafka
//if you ever mess up or you want to start fresh just stop and remove the containers and rerun the command
docker stop zookeeper kafka
docker rm zookeeper kafka
docker run --name zookeeper -p 2181:2181 -d zookeeper
docker run -p 9092:9092 --name kafka -e KAFKA_ZOOKEEPER_CONNECT=husseinmac:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://husseinmac:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -d confluentinc/cp-kafka
Get my fundamentals of database engineering course link redirects to udemy with coupon applied (database.husseinnasser.com)
Resources
KIP 500 (Remove Zookeeper)
cwiki.apache.org/confluence/d...
Become a Member on KZfaq
/ @hnasr
🧑🏫 Courses I Teach
husseinnasser.com/courses
🏭 Backend Engineering Videos in Order
backend.husseinnasser.com
💾 Database Engineering Videos
• Database Engineering
🎙️Listen to the Backend Engineering Podcast
husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
partner.canva.com/c/2766475/6...
🎙️ Mic Gear
Shure SM7B Cardioid Dynamic Microphone
amzn.to/3o1NiBi
Cloudlifter
amzn.to/2RAeyLo
XLR cables
amzn.to/3tvMJRu
Focusrite Audio Interface
amzn.to/3f2vjGY
📷 Camera Gear
Canon M50 Mark II
amzn.to/3o2ed0c
Micro HDMI to HDMI
amzn.to/3uwCxK3
Video capture card
amzn.to/3f34pyD
AC Wall for constant power
amzn.to/3eueoxP
Stay Awesome,
Hussein
@hnasr 3 жыл бұрын
Check out my fundamentals of Database Engineering udemy course with over 20 hours of deep database systems lectures. Head to database.husseinnasser.com for a discount coupon. Link redirects to udemy with coupon applied.
@markyoshikawa4278 Жыл бұрын
I love it and the way it’s presented! Will announce on LinkedIn when I get my cert. Thanjs for putting this together!
@_sr 2 жыл бұрын
27:28 answer: Every broker on a Kafka cluster is also named a Bootstrap Server. All brokers have the metadata required for the clients (producer or consumer) to discover brokers. When a client connects to one of the brokers (which are already configured as bootstrap servers in the Kafka configuration) it makes a "metadata request". The response includes information about topics, partitions, leader brokers, etc. Once the client gets this info, then - in the case of a producer- it makes the write request directly to the leader broker for that specific partition.
@hakimbencella4242 2 жыл бұрын
Thanks for the explanation
@matejlaskomulej 2 жыл бұрын
This should be pinned. Thank you!
@HarshJain-lk8cw 2 жыл бұрын
if all the brokers maintain this metadata then what's the role of zookeeper.
@boobalangnanasekaran3381 2 жыл бұрын
@@HarshJain-lk8cw The use of zookeeper is that only one leader is there for a partition at any given time by using some kind of lock registry.
@syedsabeehahmedfatmi1099 Жыл бұрын
This should be pinned. Thanks!
@mubashirimtiaz3836 Жыл бұрын
I just wanted to express my heartfelt appreciation for the amazing Kafka crash course, Your dedication to simplifying complex topics shines through, and I can't thank you enough for making Kafka so much easier to understand.
@mikejeffery8371 3 жыл бұрын
This was an awesome tutorial. You're incredibly good at explaining things and keeping people engaged when consuming so much information.
@stmasanti 4 жыл бұрын
One of the best tutorials i’ve seen. Great class and very funny. Subscribed and waiting for more content!
@jithin_zac 3 жыл бұрын
I always got demotivated by the complexity I experienced when I tried to learn this on my own but this one is a gem. I know it might have taken a lot of learning and experience for you to explains this in a simple manner Nevertheless thank you for sharing your knowledge!
@hnasr 3 жыл бұрын
Thank you Jithin! And Remember to ask in the community to get access to the slides if you would like to since you are a member❤️
@ershad003 Жыл бұрын
Sweeeeet... did not feel like a tutorial, but like a friendly chat between deeply connected friends, every word makes clear sense. Kept me engaged every second till the end, yes I made to the end. Well done Sire... 👏👏👏
@asht7290 3 жыл бұрын
Dude, I made it to the End. You kept us engaged throughout. I have soo much to learn about Comp Sci!
@bokilsbk23 3 ай бұрын
Great tutorial as always. Literally a crash course fit into about an hour. Has everything to get started with kafka, especially because of the hands on.
@manishbolbanda9872 2 жыл бұрын
i had to deal with Kafka in one of my project, so searched for some tutorials, As soon as I saw that you have made a video on this, i had a sense of relief that i dont have to worry about the resource atleast. Thanks for all your efforts man.
@eduardotp541 Жыл бұрын
Not only the content, but the way you say it, and your enthusiasm, is something else. You sir deserve all the best, because you are, with no doubt, in the better's shelf..
@nextifydev 3 жыл бұрын
Just started watching and already loving it. I wish you a happy new year and hoping to see more great content like this.
@ilyakushlianski6519 2 жыл бұрын
Amazing, Hussein! Thanks for the overview of Kafka. I've always been a bit afraid of this beast, but now I understand at least the basics
@myviewsmyreviews2001 3 жыл бұрын
Amazing tutorial! Really find the topic interesting and easy to understand broker, producer, consumer etc.. Thanks for coming up with a great video. Hope we will get to see more on this.
@MOHDSALMAN-sj2zu 3 жыл бұрын
I made it to the end. :P Thank you so much for the amazing content Hussein.
@rubensingh6809 2 жыл бұрын
bro your tone when explaining these videos are so good. Didn't even feel like someone was teacahing me. It was as if a friend was talking to me lol. Good job man. Your excitement is intoxicating. :)
@satalkar 4 жыл бұрын
I rarely comment on videos but this was genuinely one of the best computer science tutorials I've ever seen!
@hnasr 4 жыл бұрын
Thank you Anish for taking the time to write this appreciate you!
@jasonngan5747 2 жыл бұрын
The explanation and illustration is clear and concise! Thanks for the comprehensive tutorial!
@nihinlola 2 жыл бұрын
Excellent practical explanation of Kafka, I wish every technical tutorial was as easy as this. Thumbs up!
@aatifbangash 4 жыл бұрын
Another perfect tutorial. Everything is explained very nicely. Thanks for making Kafka easy to understand in this tutorial.
@hnasr 4 жыл бұрын
Atif Ali thanks Atif for your support! Happy I could make such complex topic easy to absorb.. still I didnt cover all what kafka has to offer! Cheers
@nimasoufiloo2626 3 жыл бұрын
Couldn't learn kafka better in such a short time. Thanks Hussein.
@mutasworld Жыл бұрын
I made it to the End! Thank you for that video, I can now say I understand Kafka a lot better than 1 hour ago :)
@alexsiuwh 4 ай бұрын
this is the only tutorial i found that go in the depth and details of a pro should know , and i can compare Kafka with other messaging system like JMS and MQ. Thanks so much for sharing👍👏🙏
@manishtanwar4803 2 жыл бұрын
made it to end. Amazing video. you made Kafka look so simple, and the best part-> you showed a code demo. clap. clap.
@beatagryczka7108 3 жыл бұрын
Amazing video! You have the talent to explain the complicated stuff really clearly😊
@atexnik 3 жыл бұрын
This is the BEST educational channel!!! Bravo!
@linwang7476 2 жыл бұрын
Very nice tutorial. Got a clear understanding of Kafka, especially the part of queue, pub sub, very helpful! Thank you so much!
@sy5294 3 жыл бұрын
Your skillset is brilliant, amazing and fantastic. Thank you so much!
@user-lq7yg5ej1u Жыл бұрын
This is well detailed even for a beginner, I completely understand whats going on. Brilliant content.
@abhasupadhayay6420 3 жыл бұрын
Really amazing!!! So many concepts explained in an easy and interesting way. Please keep up the great work.
@ComisarioLobo 3 жыл бұрын
Man, you are the best, thanks a lot for creating such valuable content for free
@luqmansen 4 жыл бұрын
I just watched your RabbitMQ crash course and at the end of the video you mention how Kafka better at some point, then I bumped into this great video. Very clear explanation, love your video!
@hnasr 4 жыл бұрын
Luqman S thanks Luqman! Each software its use cases, I think RabbitMQ Is trying to be more than it really is (a message queue) where as kafka was designed to be a pub/sub and a message queue. If your app needs a queue and a pub/sub (or maybe streaming) than Kafka . If you just need a queue RabbitMQ is the way to go and kafka will be an overkill I think.. I would even consider zeromq as a lightweight mq.. Good luck!
@ibknl1986 3 жыл бұрын
I made it to the end. Excellent. I don’t remember which video previously I saw was so long MD not boring at all. In fact you made it so interesting. May Allah (God) guide and bless you.
@mikaeledebro1144 3 жыл бұрын
Great explanation and overview over the possibilities of Kafka! Looks like a cool technology!
@peternamagonya5310 Жыл бұрын
I made it to the end. Followed every step and it worked nicely. Thank you for the effort... really helpful
@yuanqingzhang3648 2 жыл бұрын
This course is great because of the example. It's much better than some videos in which only some introductions are represented.
@a.yashwanth Жыл бұрын
You explain by answering all the questions people get when learning a topic. Most tutorials just explain concepts.
@vbrtrmn 3 жыл бұрын
Great video, thanks for making it. Love your accent! You're very easy to understand, follow, and your excitement is fantastic!
@hnasr 3 жыл бұрын
vbrtrmn glad to know thank you! Lots of people have trouble with my accent glad you like it!
@bhaveshgujarathi926 Жыл бұрын
made it to the end 🤟. It was a very good explanation with great clarity. Really Appreciate your work! Thanks a lot 🙏
@alishoman2826 3 жыл бұрын
the simplest and best tutorial ever!! Good Job
@flippolis71 2 жыл бұрын
This is a great video to understand what Kafka is and how it works, very well explained!
@priyaranjankumarverma2341 3 жыл бұрын
I really appreciate you from my deep of heart for explaining such complex concepts in super simple way, Excellent :)
@abrahammoyo2939 3 жыл бұрын
Hussein, You are a legend!!! Any one can be a master if they follow to your conversations. I like the way yo have a full conversation with yourself - such a good dialogue asking and answering yourself
@hnasr 3 жыл бұрын
Thank you! I am happy you noticed the conversation style, i have been working on it 😊
@93kazem 4 жыл бұрын
Thank you for this, don't know how I missed it
@nidalshater4027 Жыл бұрын
Watched this video in one session, also applied the practice on my machine, what a great content you are creating Hussien, a must rewatch as always 😉
@wanitooo Жыл бұрын
Amazing video. A gifted teacher like this is quite rare ♥Thanks a lot!! btw made it to the end 😆
@shubhamchandra336 3 жыл бұрын
Really enjoyed this informative session and yeah, made it to the end !!!
@umair.ramzan 3 жыл бұрын
made it to the end :))) Thank you Hussein, you are the best!!
@hirenjoshi2198 2 жыл бұрын
Amazing explanation Hussein. Every single bit of your lecture is fully understandable. You made difficult thing easy with practical example. You rocked! Good Luck.
@emanuel0723 4 жыл бұрын
Enjoyed following this... as always, thanks Hussein!
@t529615 3 жыл бұрын
By far the best introduction video to Kafka I have seen. Great examples and nice build up to demos that puts theory into practice.
@hnasr 3 жыл бұрын
Hans-Jacob Melby thanks Hans!
@t529615 3 жыл бұрын
@@hnasr One question though.. Shouldn`t the consumer allways get all messages in the topic (within its partition)? I can see that the property "fromBeginning" i set to true, but when i restart the consumer, it still waits for new mesages. It does not replay the old ones?
@simonhuang1357 3 жыл бұрын
I made it to the end. Thanks for the tutorial, it was very informative and I appreciate your efforts.
@amirhosseinmoayedi6205 10 ай бұрын
truly amazing-so simple and incredibly useful. Great job!
@shantnukumar1642 Жыл бұрын
Made it to the end! Awesome video Hussein :)
@DavidBwire 2 жыл бұрын
Thank you for putting in the effort to make a short and clear to understand video.
@stefanmiller5818 2 жыл бұрын
You really know how to do tutorials, great video!
@hvuupadhyay5418 Жыл бұрын
Made it to the end! Yay! Nice video Hussein!
@robbenbahati 3 ай бұрын
made it to the end, and every detail was fully explained thanks.
@sindhumohana6164 3 жыл бұрын
Amazing video Hussein ! The way you explain helps to register the information in our memory :-)
@satheeshprabhakaran5330 2 жыл бұрын
Made it to the end!! Amazing content!
@shivangitomar5557 2 жыл бұрын
This is THE BEST video on this topic. Thanks a lot!
@amizan8653 3 жыл бұрын
This is explained simply and perfectly. Thank you for posting this.
@Goodwin454 2 жыл бұрын
I was looking for your channel , you really good at explaining things you helped me once , I'm here again , thank you andn jazkAlllah khayran .
@luniacllama8373 Ай бұрын
made it to the end , noice, hopefully i consistently get better at backend and devops :3
@jassinakhlaqi5474 2 жыл бұрын
Thank you for this amazing channel. I enjoy watching your videos and learning more.
@susmitvengurlekar 2 жыл бұрын
Made it till the end ! You can explain something in a super simple manner if and only if you understand it And you do, Thanks !
@mzaman8660 4 жыл бұрын
This is brilliant and concise. Been looking for a channel like this 👍🏾
@hnasr 4 жыл бұрын
M Zaman enjoy the content 🙏
@PrashantKhanolkarUSA Жыл бұрын
Amazing video, you have made a complex topic extremely simple with your style of teaching.
@atharwaadawdakar4144 3 жыл бұрын
Thank you so much for such a crystal clear and crisp explanation !!
@pongching8697 4 жыл бұрын
Clear and easy to understand. Although it's a long video, but not boring at all.
@gokulkpeethu6642 Жыл бұрын
The best stuff on kafka . Made it so simple
@mosespeter9711 2 жыл бұрын
I made it to the End!!! Thanks so much Hussein!
@xonmaina 2 жыл бұрын
Thanks Nasser I ve made it to the end. wonderful stuff
@jongxina3595 Жыл бұрын
best video on kafka. Thanks a ton!
@pavanr92iitm 3 жыл бұрын
Made it to end. Thanks you so much for the quality content.:)
@saurabhdeshpande8450 2 жыл бұрын
Best explanation of Kafka on the internet!
@adilsheikh9916 2 жыл бұрын
Appreciate your efforts...like the way you clearly say what you know/understand & what you dont...else I see most people in Industry are faking the intelligence & understanding about any technology & they pretend it so perfectly like they themselves have developed that tech...while those who actually know/understand the tech properly are very few & rare... So always good to watch your videos to understand something in simple words rather mugging those funky & techie words.
@diptikaushik8250 3 жыл бұрын
I made it to the end. :P Fabulous content! You rock. :D :D
@princeabdul4290 3 жыл бұрын
Brother you're videos are phenomenal they are just top notch.
@hey............. Жыл бұрын
Thank you man! Really appreciate the hardwork.
@sonjoychakraborty571 Жыл бұрын
Awesome, such an wonderful session and easy way to represent can only be possible by Hussain Nasser only. It truly cleared lot's of my confusions. Hats Off. Keep up going ..
@NickLeenknecht 3 жыл бұрын
Answer to - 27:25 - The producer sends a Metadata request with a list of topics to one of the brokers in the broker-list you supplied when configuring the producer. The broker responds with a list of partitions in those topics and the leader for each partition. The producer caches this information and knows where to redirect its produce messages. In case of failure while producing, failed broker's data (topics and its partitions) dynamically linked to existing replica which is present on another broker via topic's replication and new leader's information is communicated to the client (producer).
@nicolaseckell9691 Жыл бұрын
Thank you very much Hussein, this video helped me a lot. Have best see you next time.
@adewaleShotobi Жыл бұрын
Made it to the end 😁. Thank you!
@hangduong6683 Жыл бұрын
oh my god I love this video so much =]] this is such an amazing tutorial and how funny the instructor is : D
@sanjitselvan5348 2 жыл бұрын
Wow! Wonderfully explained video, including the commands. Thank you
@vispharaon Жыл бұрын
I made it to the end :) great tutorial, thanks!
@uwaifoidehenre7976 3 жыл бұрын
I made it to the end . Great content. Thanks .
@realohh 23 күн бұрын
Amazing 🎉. Love your work and your passion to teach.
@sergakrem 2 жыл бұрын
Cool and neat into 👍 Thank you!
@unboxingsparkles5786 3 жыл бұрын
I made it to the end. Love the long videos and the content 👌👌👌
@hnasr 3 жыл бұрын
sidra tariq thanks Sidra I like long format too 😊🙏
@jiun683 3 жыл бұрын
Thanks for such an awesome video on Kafka!
@obiwanus 3 жыл бұрын
Huge props for trying to understand things before you teach them!
2 жыл бұрын
Excellent tuto Hussein! Thanks a lot.
@stevensonicteam 7 ай бұрын
Thank you. Best video so far for learning kafka. 👍
@AhmedElfakharany 4 жыл бұрын
I made it to the end! Thank you for such great content.
@hnasr 4 жыл бұрын
Awesome! Thank you!
@pavankumardittakavi5325 2 жыл бұрын
So much quality content. Thanks a lot Hussein.
@crypto-saiyan9283 2 жыл бұрын
Today i understood, its not the subject its the way of teaching...thanks for sharing this wonderful knowledge
@shaunpx1 2 жыл бұрын
I love your content man, very awesome 👌 , great lessons!
@kramnedski1988 5 ай бұрын
I appreciate the content. no boring part :)
Devoxx Poland
Рет қаралды 193 М.
BalcevMMA_BOXING
Рет қаралды 14 МЛН
SmashCrush! Arabic
Рет қаралды 39 МЛН
Hussein Nasser
Рет қаралды 34 М.