Kafka Streams 101: Getting Started (2023)

Рет қаралды 101,609

Күн бұрын

► TRY THIS YOURSELF: cnfl.io/kafka-streams-101-mod...
In this course, Sophie Blee-Goldman (Apache Kafka® Committer and Software Engineer) gets you started with Kafka Streams.
Kafka Streams is declarative, so you state what you want to do, rather than how to do it.
Imagine that you have a topic, from which you'd like to filter all records marked with the color "red." You could accomplish this with plain Kafka, but the equivalent Kafka Streams code would only take a third of the lines.
For a complete IMMERSIVE HANDS-ON EXPERIENCE: cnfl.io/kafka-streams-101-mod...
--
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion - designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#kafka #kafkastreams #streamprocessing #apachekafka #confluent

Пікірлер: 34

@johnkelty4001 2 жыл бұрын

Thanks, this is very helpful info in a short amount of time!

@elbowdashizzle Ай бұрын

The subscribe button in KZfaq lit up when you said “subscribe to the topic” 😂

@mehrdadk.6816 Жыл бұрын

That was a great introduction

@tomknud 2 жыл бұрын

Good talk, thanks

@jasper5016 7 ай бұрын

Fantastic tutorial. Thanks so much Sophie!

@ankitlakum1 2 жыл бұрын

Good info 😊

@krishnasharma-nr3yg Жыл бұрын

Thanks, very helpful

@avimehenwal Жыл бұрын

Does kafkajs in nodejs provide support for kafka streaming API ?

@kartikprabakara3126 2 жыл бұрын

Excellent

@sonnyps413 Жыл бұрын

4:50 Is when she starts explaining about Kafka Stream

@komiljonaliev3415 Жыл бұрын

Thank you for giving helpful information

@SzymekCRX 4 ай бұрын

Thank You!

@KarthikeyanSkarthikeyanmadurai Жыл бұрын

okay my long pending doubt is cleared now 🙏🏽

@ighsight Жыл бұрын

Excellent. Wondering can I use Kafka Streams with the KafkaJS library for Node JS?

@adityanjsg99 9 ай бұрын

you may.. i am using with python

@stream2learn 2 жыл бұрын

Can we use Kafka Streams and Spark Streaming in the same project ?

@ConfluentDevXTeam Жыл бұрын

Hi, I'm Gilles Philippart from Confluent. I don't think there's anything that would prevent you from doing so, Kafka Streams is just another library that you pull into your application dependencies after all. I'm not sure there would be a good reason to do that though, as it might prove harder to tune the performance, scale, secure, troubleshoot, etc. Have you considered creating another microservice that would consume or produce to a topic to interact with the Spark Streaming app?

@sghsghdk Жыл бұрын

So kafka streams is just a convinience client api? will the network traffic be equivalent?

@gillesphilippart5986 Жыл бұрын

Kafka Streams is a streaming processing library to join streams (topics) and simplify the building of stateful applications (windowing, aggregations). If might consume more network traffic due to the fact that most of the local state is also stored in Kafka for backup, but that's usually a good thing as it make your application more reliable!

@sapientum8 2 жыл бұрын

@shyammohan2007 Жыл бұрын

If each topic is a file, how is about a partition? Does it means the aforesaid file will be divided into a subset of file again where each sub-file is a partition?

@ConfluentDevXTeam Жыл бұрын

Hi Shyam, I'm Gilles Philippart from Confluent. Sophie says things like "This log is a file and it's a special kind of file" then later on "the topic itself really is just a name that is given to the log. So on the broker, they store each log and directory names by that topic". I agree that this part is a bit confusing as we normally associate a " Log" to a "Partition" not a topic. Sophie glossed over the details of how the data is stored physically on disk because that's not really the purpose of the course, but let me get this right: 1. A "topic" is a logical stream of records, and it is the primary unit of data organization in Kafka. 2. A "topic partition" is a logical unit that represents a subset of the data within a Kafka topic. Each partition is an ordered and immutable sequence of records. 3. Now, Kafka stores the data for each topic partition in a set of "log segments". Each log segment is a file on disk and represents a sequential chunk of data. The size of each log segment is configurable and typically ranges from a few kilobytes to several gigabytes. As new records are appended to a partition, they are written to the active log segment. For example, the directory structure of the first partition of the "account-deposits" topic would look like this: /var/lib/kafka/data/account-deposits-1 00000000000047926734.log 00000000000047926734.index ... 00000000000052497535.log 00000000000052497535.index If you want to learn about Kafka internals, check out developer.confluent.io/learn-kafka/architecture/broker/ Hope this helps!

@javadahmadzadeh7129 Жыл бұрын

Thanks. it was really helpful but for me as a non-native English speaker, the speed of talking was so fast, and I had to use subtitles for better understanding