Apache Kafka Architecture

No video

Apache Kafka Architecture

Рет қаралды 22,950

Күн бұрын

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com
▬▬▬▬▬ Experience & Location 💼 ▬▬▬▬▬
► I’m a Senior Software Engineer at Juniper Networks (12+ years of experience)
► Located in San Francisco Bay Area, CA (US citizen)
▬▬▬▬▬▬ Connect with me 👋 ▬▬▬▬▬▬
► LinkedIn: / anton-putra
► Twitter/X: / antonvputra
► GitHub: github.com/ant...
► Email: me@antonputra.com
▬▬▬▬▬▬ Related videos 👨‍🏫 ▬▬▬▬▬▬
👉 [Playlist] Kubernetes Tutorials: • Kubernetes Tutorials
👉 [Playlist] Terraform Tutorials: • Terraform Tutorials fo...
👉 [Playlist] Network Tutorials: • Network Tutorials
👉 [Playlist] Apache Kafka Tutorials: • Apache Kafka Tutorials
👉 [Playlist] Performance Benchmarks: • Performance Benchmarks
👉 [Playlist] Database Tutorials: • Database Tutorials
▬▬▬▬▬▬▬ Timestamps ⏰ ▬▬▬▬▬▬▬
0:00 Intro
0:40 Apache Kafka Event (Message)
2:56 Apache Kafka Topic
4:22 Apache Kafka Partitions
6:52 Apache Kafka Producers
7:48 Apache Kafka Consumers
8:38 Apache Kafka Cluster Architecture
9:38 Apache Kafka Broker
10:23 Apache Kafka Controller
▬▬▬▬▬▬▬ Source Code 📚 ▬▬▬▬▬▬▬
► GitHub: github.com/ant...
#kafka #kubernetes #devops

Пікірлер: 44

@AntonPutra 6 ай бұрын

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - me@antonputra.com

@pradeep_kumar_ind Ай бұрын

Awesome! I love these videos. They are so simple to understand. Thanks.

@AntonPutra Ай бұрын

thank you!

@GabrielPozo 6 ай бұрын

Thanks for this high-quality explanation!!!

@AntonPutra 6 ай бұрын

thanks trying to improve

@ravipanchal2259 3 ай бұрын

Your content is awesome. Hats off to your visualization and explanation .

@AntonPutra 3 ай бұрын

❤

@agun21st 6 ай бұрын

Wow, Anton sir, this breakdown of Apache Kafka's architecture is incredibly insightful! Understanding how topics, partitions, and brokers work together is key to leveraging Kafka effectively in real-world applications. Thanks for this great overview!

@AntonPutra 6 ай бұрын

thanks!

@boyanbalev4428 6 ай бұрын

One unorthodox questions do you think creating those videos help you getting deeper understanding of these technologies. I write articles and I found that it actually benefits as much me as helping others. Great work.

@AntonPutra 6 ай бұрын

thanks, of course it does

@user-vl4sg2vj8b 2 ай бұрын

best primer on Kafka. kudos

@AntonPutra 2 ай бұрын

thanks!

@aymenfayadhi3694 26 күн бұрын

Thank you. This was actually helpful :)

@AntonPutra 26 күн бұрын

welcome!

@pieter5466 4 ай бұрын

Great Kafka video, very clear and great diagrams.

@AntonPutra 4 ай бұрын

Thanks!

@zacclifton5479 6 ай бұрын

Well done! I would recommend doing Pulsar next

@AntonPutra 6 ай бұрын

thanks! i'll take a look

@csmithDevCove 6 ай бұрын

this guy ^ always bringing useful information. Well done @AntonPutra.

@AntonPutra 6 ай бұрын

thank you!

@drivetrainerYT 3 ай бұрын

Can't help subscribing ;) thanks!

@AntonPutra 3 ай бұрын

🙏

@shokhrukhbekyursunjonov6203 3 ай бұрын

Good day Anton, thank you so much for user-friendly content! Could you please make a tutorial video on properly deploying secure (SASL/SCRAM) Confluent based full Fafka stack (2024 edition)? For several weeks I am attempting to deploy full kafka stack (zoo+kafka+schema-registry+kafka-connect+rest-proxy+ksqldb+conduktor-console) using SASL/SCRAM_SHA_256 method, but having errors in additional components such as registry, proxy connect and ksqldb during SASL SCRAM authorization... (docker compose solution). I am sure it would be really helpful to the kafka devops community here... Sincerely, Shokhrukh Yursunjonov

@AntonPutra 3 ай бұрын

Sure, I can do it. A couple of questions: Is it Kubernetes-based? Also, why not use Kafka without ZooKeeper (KRaft)? Do you have any legacy applications that require ZooKeeper?

@shokhrukhbekyursunjonov6203 3 ай бұрын

@@AntonPutra it is docker stack based (cause I am given one server to deploy kafka and I am writing compose file to deploy all in one server). Ah, I almost forgot about KRaft, good idea, I might try to use this mode, I heard that it is more intelligent and faster! According to my info (what devs said to me) apps are not legacy (mostly .net containerized apps), thus I can try using single/double broker KRaft mode, thank you. But I am afraid to have the same issues configurning SASL/SCRAM auth in it

@AntonPutra 3 ай бұрын

@@shokhrukhbekyursunjonov6203 yes it's faster since kafka does not need to keep offsets in zookeeper and it's scales better without zk. I'll get to it maybe after EKS playlist will see.

@dmitriydiachenko5648 5 ай бұрын

perfect explanation, ty

@InvincibleMan99 11 күн бұрын

Hello brother Great explanation Recently in interview, i was asked what is the data structure that is used in kafka..

@AntonPutra 10 күн бұрын

key/value :) for the value you can use RPC, avro, json etc.. Many companies use protocol buffers to reduce the size of the payload - grpc.io/docs/what-is-grpc/core-concepts/#service-definition

@InvincibleMan99 6 күн бұрын

@@AntonPutra Got it thank you. Yeah so message data structure is key value. But as kafka is a queue, it maintains a log file in append mode on each broker, as you have covered. So it also has query data structure.

@suvajitchakrabarty 6 ай бұрын

In 4:55 should the partition in kafka broker 3 be partition 6, 7, 8 instead of 3, 4, 5, since you mentioned a partition always belongs to one broker.

@AntonPutra 6 ай бұрын

yeah, copy paste mistake :) it should be 6, 7, 8

@suvajitchakrabarty 6 ай бұрын

Thanks for the clarification@@AntonPutra

@blender_wiki 6 ай бұрын

How you menage message locking in Kafka in case of a consumer task is time consuming?

@rahulsingh-iq4gd 6 ай бұрын

Hello sir can you make video on kubernetes helm chart please

@AntonPutra 6 ай бұрын

@rajun9208 2 күн бұрын

Which tools you are using for making this animation?

@AntonPutra 2 күн бұрын

adobe

@rajun9208 2 күн бұрын

@@AntonPutra thank you for the quick response. Could you please tell me full name of the software

@AntonPutra 2 күн бұрын

@@rajun9208 adobe suite, multiple apps illustrator, after effect, premiere pro etc

@RD-fb6ei 6 ай бұрын

So the upper bound on the number of consumers for a topic is the number of partitions? That’s quite limiting compared to something like Apache Pulsar or NATS Jetstream where you can horizontally scale the consumers as much as you want without worrying about the partitioning scheme of the server.

@AntonPutra 6 ай бұрын

No, you can have as many consumers as you want, not limited by partitions. Depends on your app, in some cases, to scale you need to increase both partitions and consumers.

@suvajitchakrabarty 6 ай бұрын

@RD-fb6ei However in those applications when you scale the consumers, I believe you lose the guarantee of events arriving in order.