Azure Event Hub Tutorial | Big data message streaming service

Рет қаралды 165,788

Күн бұрын

Azure Event Hubs is highly scalable big data event processing service capable of processing millions of events per second.
This episode will deep dive into Event Hub basics, partitions, namespaces, shared access signatures, even hubs, consumer groups and more!
Source code for demos: github.com/MarczakIO/azure4ev...
In this episode live demo of
- Creating and managing namespaces and even hubs
- Sending data with .NET SDK
- Sending data with Logic Apps
- Receiving data with .NET SDK
Next steps for you after watching the video
1. Azure Event Hubs overview
- docs.microsoft.com/en-us/azur...
2. Creating event hubs in portal
- docs.microsoft.com/en-us/azur...
3. Even Hub terminology
- docs.microsoft.com/en-us/azur...
4. Throughput units and auto-inflate
- docs.microsoft.com/en-us/azur...
5. Authentication and authorization with Event Hubs
- docs.microsoft.com/en-us/azur...
6. Event Hubs FAQ
- docs.microsoft.com/en-us/azur...
Want to connect?
- Blog marczak.io/
- Twitter / marczakio
- Facebook / marczakio
- LinkedIn / adam-marczak
- Site azure4everyone.com

Пікірлер: 240

@harithik1 3 жыл бұрын

A 3hr Pluralsight video = 30 mins of your video. Well done.

@AdamMarczakYT 3 жыл бұрын

The greatest complement! Thanks! :)

@pesetskyps 3 жыл бұрын

@@AdamMarczakYT True story. I thought the same watching your video about App insights

@Tom-sl7ni 3 ай бұрын

I love the way you explain. No distracting or unnecessary information, the tone of voice makes it easy to listen, and the slides help to understand the complex concepts of Azure Event Hubs. 👍 Thanks

@franciscovilches7424 Жыл бұрын

this is such great content. structured very well and explained in very simple terms. It's perfect for anyone starting out with EventHub. Thank you for this! subscribed

@shovonku 3 жыл бұрын

Fantastic presentation Adam. To the point without any unnecessary detail.

@AdamMarczakYT 3 жыл бұрын

Glad it was helpful!

@anirbanmandal4788 3 жыл бұрын

All of your videos are extremely helpful. Very informative. Clear presentation. Absolutely marvellous. Thanks a lot.

@AdamMarczakYT 3 жыл бұрын

Glad you like them! Thank you :)

@apsrain 3 ай бұрын

Welldone ! Your video covers the key concepts and cuts all the noise ! Much appreciated

@martinshiveral6604 2 жыл бұрын

Great content, very well presented Adam! Keep up the great work.

@Marcelk86 2 жыл бұрын

what a talented guy - high quality content - excellent explanations!

@maitreyeekar7509 3 жыл бұрын

Awesome Adam!! At first I was not able to understand from where I should start but watching your video really helped me! Also, I was able to do hands-on for this! Thanks Adam for this video!! Hope to see more videos :)

@AdamMarczakYT 3 жыл бұрын

Glad I could help!

@HarshalGeet 3 жыл бұрын

So I have Udemy and Pluralsight both subscription - But man Hats Off. You make it some simple and crystal clear - Just what is needed and with needed details. Thanks a lot

@AdamMarczakYT 3 жыл бұрын

Thank you, glad to help out :)

@georgetuccio6053 Жыл бұрын

Just finished your Event Hub Tutorial and very much enjoyed it. Excellent. I wish you would've also dived into Autoscaling and Geo-replication with region pairing. You only mentioned them at the end but didn't address them. I was looking for a good tutorial on Event Hub and found this. Excellent job. I see you have some other, related videos as well which I will study as well. Thank you Adam.

@Jamesatighe Жыл бұрын

Really nice overview of Event Hubs! Well done and informative! Subscribed! :)

@VBA4u 2 жыл бұрын

Thanks Adam very well presented and explain in simple language.

@adamzachary6947 2 жыл бұрын

Great demo, Thanks Adam

@mohinderkaur6671 3 жыл бұрын

Great tutorial - Thanks

@AdamMarczakYT 3 жыл бұрын

You're welcome!

@afroinfotech6705 3 жыл бұрын

Great video Adam, Thank you. However, there are some unclarity 1. at 5'15, Throughput units represent the number of 10 k events or 1mb messages per sec. We can get a max of 20k events per sec or 20mb msgs per sec 2. at 18', you can have up to 20 consumer groups per event hubs (NOT partitions). The partition is just for horizontal scaling of the data across multiple nodes in the event hubs 3. at 18.15' it is advised to have one reader per partition in a Single consumer group (if not the consumer application has to handle dups). However, multiple consumer groups (with different readers) can read the same partition of an event hub

@AdamMarczakYT 3 жыл бұрын

Great comments. I checked those times and I see how I could be more precise at those times. For point 3 I did say it's not recommended to have more readers than partitions as per docs. You can have less, but you are 100% correct that ideally you should have 1 to 1 ratio of consumers to partitions. For point 2 I said 1 to 20 consumer groups, but I think I said mistakenly we've seen that when we created namespace which could be confusing. Thanks for the pointers :) Will make sure to keep an eye for the details.

@samraja 2 жыл бұрын

Thanks for the great Video. It would have taken several days if I want to learn from the documents from Microsoft site. But this one helped to learn in hours.

@vekzdran 3 жыл бұрын

Very good work Adam. Thank you honestly.

@AdamMarczakYT 3 жыл бұрын

My pleasure!

@mallikarjunamatam9782 3 жыл бұрын

Awesome, great presentation. Thanks Adam.

@AdamMarczakYT 3 жыл бұрын

Glad you liked it!

@NIYANTAjmp 3 жыл бұрын

Very concise and effective video. Great work

@AdamMarczakYT 3 жыл бұрын

Much appreciated!

@ashishdeora8522 2 жыл бұрын

This is the best video I got on this topic. I am going bananas for last 2 days. This is simply awesome

@AdamMarczakYT 4 жыл бұрын

⚠ Attention ⚠ I was pointed out that there is a small mistake in dotnet package adding commands. Both Azure.Messaging.EventHubs.Producer and Azure.Messaging.EventHubs.Consumer are not separate packages as such they don't need to be added specifically. They are part of Azure.Messaging.EventHubs package. You will see during video error in the console when adding them. Github repo is updated. As per very good comment from user hellhax, please note that Auto-inflate feature is "auto-scaleUP" not auto-scaling (up and down). I oversimplified the explanation here. If you want automatic scaling up and down, you would have to add some custom logic!

@AdamMarczakYT 4 жыл бұрын

Will update this in future if MS adds full auto scaling features to event hubs. :)

@thiagus4056 3 жыл бұрын

Good presentation! Articulated very well.. Thank you!

@AdamMarczakYT 3 жыл бұрын

Glad it was helpful!

@rafae111111 4 жыл бұрын

Great presentation! Much appreciated :)

@AdamMarczakYT 4 жыл бұрын

My pleasure!

@mohamedgalal3198 2 жыл бұрын

Very informative thanks Adam .. just little comment to be considered as mentioned in the documentation "Auto-inflate does not automatically scale down the number of TUs when ingress or egress rates drop below the limits."

@takon100 8 ай бұрын

This was very clear! Thank you

@undefined_io_ 5 ай бұрын

awesome. First I watch your video then I go to Microsoft Learn documentation. Best way to retain knowledge.

@MegaJagveer 2 жыл бұрын

Amazing video and great instructor. Keep it up!!

@AdamMarczakYT 2 жыл бұрын

Thanks, will do!

@danielesalvigni 2 жыл бұрын

WHat a beautiful perfect introduction :), thank you!

@digvijaykumar8923 2 жыл бұрын

thankyou so much adam i cannot tell you how much this video help me , i am stuck in something and this video slove that. thanks man.

@n.s.karthik5628 2 жыл бұрын

wonderful demo

@flaviucristian3658 11 ай бұрын

Great video! keep up the good work!

@anshumanpandey7500 3 жыл бұрын

Awesome presentation... short and crisp 👍👍

@AdamMarczakYT 3 жыл бұрын

Thanks a lot 😊

@RainLaw 3 жыл бұрын

Fantastic presentation, good job!

@AdamMarczakYT 3 жыл бұрын

Thank you very much!

@run-forrest-run 3 жыл бұрын

Thanks again Adam, really good content.

@AdamMarczakYT 3 жыл бұрын

Cheers! Appreciated!

@arupnaskar3818 4 жыл бұрын

Lots of thanks mr. Adam.. very very helpful tutorials ..👌

@AdamMarczakYT 4 жыл бұрын

My pleasure!

@bergimusic7726 3 жыл бұрын

This is very good and clear, congrats!

@AdamMarczakYT 3 жыл бұрын

Thanks a lot!

@mekasamshaikh 4 жыл бұрын

Good one...nice clean detailing of topic..Thanks a lot!

@AdamMarczakYT 4 жыл бұрын

Glad you liked it 😁

@petercheung63 4 жыл бұрын

This video is so great, thanks Adam

@AdamMarczakYT 4 жыл бұрын

Glad you liked it!

@jamshadm 3 жыл бұрын

Nicely Explained...And very interesting and informative..Thanks a lot..You saved my time..

@AdamMarczakYT 3 жыл бұрын

Glad to hear that

@arvindpattar155 3 жыл бұрын

precise and crisp.. well narrated.

@arvindpattar155 3 жыл бұрын

few issues resolved like dotnet add package Azure.Messaging.EventHubs.Processor --version 5.2.0, requires a version number to install the packages and to build

@AdamMarczakYT 3 жыл бұрын

Cool! Thanks for letting me and others know :)

@andriizilnyk4927 Жыл бұрын

Thanks for your work❤

@buguannishi 2 жыл бұрын

Great video Adam

@saronyobose Жыл бұрын

You are so clean in your description .

@vishnusharma1402 2 жыл бұрын

Excellent explanation in short vedio.

@padmanathanramasamy8354 2 жыл бұрын

Very interesting video with clear explanation. I always recommend your video to my friend and colleague if they want to start learning Azure. Can you show a example in Event Hub like 1) Two consumer were reading the data in different time internal. 2) How do we can configure a message to publish the message to specific set of consumer.

@antirus5481 2 жыл бұрын

Nicely explained, keep it up 👍

@maunico123 8 ай бұрын

Thank you brother.

@bartosz9476 10 ай бұрын

Dzięki za to video!

@MrCybergoku 3 жыл бұрын

Thanks , very clear !

@AdamMarczakYT 3 жыл бұрын

Glad it helped!

@rn9085 3 жыл бұрын

Thanks, that helped me a lot!

@AdamMarczakYT 3 жыл бұрын

You're welcome!

@philburtonfrench 4 жыл бұрын

Great work Adam, really useful video. I'm still a little unclear as to the purpose of Consumer Groups

@AdamMarczakYT 4 жыл бұрын

Thanks Phil! Think of consumer groups like separate applications that need to read your entire stream of data from end to end. Consumers in a consumer group are processes within the same application. This design allows for maximum parallelization.

@OpTechMarketing 3 жыл бұрын

@adam ... Thanks for a well put together tutorial.

@AdamMarczakYT 3 жыл бұрын

My pleasure!

@gnomek101 18 күн бұрын

bro, it's great!

@leandroqueirozdossantos9434 6 ай бұрын

nice content. Congrats

@mischameyer1012 3 жыл бұрын

Great video!! 👍

@AdamMarczakYT 3 жыл бұрын

Thank you! 👍

@samtsai8369 3 жыл бұрын

super clear!

@AdamMarczakYT 3 жыл бұрын

Thank you! :)

@leandroezequielmolinari9551 3 жыл бұрын

Hi Adam, how are you? just to say thank you! your azure video tutorials are really good!

@mcelona71 3 жыл бұрын

Good Job, Adam!!!

@AdamMarczakYT 3 жыл бұрын

my pleasure!

@AaronHoffman 4 жыл бұрын

Great overview, good balance of information and demos

@AdamMarczakYT 4 жыл бұрын

Much appreciated!

@alwaseem5309 3 жыл бұрын

Way too Good. Keep it up.

@AdamMarczakYT 3 жыл бұрын

Thanks a lot 😊

@Gq_Guy7846 Жыл бұрын

Good Job

@samatanarvankar1211 3 жыл бұрын

Compact and nicely explained. Please make a video for building micro services.

@AdamMarczakYT 3 жыл бұрын

Thanks! Appreciated :) noted the idea.

@randommoments1263 3 жыл бұрын

Azure saviour thanks

@AdamMarczakYT 3 жыл бұрын

Hehe thanks! :)

@BijouBakson 2 жыл бұрын

Thank you

@malohatibrohimovna2939 3 жыл бұрын

thanks, Adam, for a concise and powerful tutorial! I would like to use Java to read the events from an event hub and to send them to an IoT Hub. Do you have something similar or could you recommend me some other source?

@domagojhorvat2646 2 жыл бұрын

Adam, thanks for an awesome explanations! Quick question...what are the options if we run into situation when our consumers can't keep the pace with producers and need to scale out to multiple instances (a lot of i/o on consumer side, e.g.)? what are the options to make sure there are no duplicate hadlings? tnx!

@carl-heinmostert6481 3 жыл бұрын

Awesome

@AdamMarczakYT 3 жыл бұрын

Thanks!

@marlonmorales8672 2 жыл бұрын

Very helpful tutorial sir Adam. Just wondering if there's a way to test event fetching without implementing and using the SDK. To test it via API call using postman for example?

@sorontar1 3 жыл бұрын

IMO it is important to mention that there is a limit of the number of event hubs in the namespace. This essentially means that you can’t use event hubs in the same way as you would use kafka where you could create lots of topics.

@AdamMarczakYT 3 жыл бұрын

Thanks Pawel. In general your remark is very import but not only to event hubs but any service in Azure. Always understand limitations of each service that you are planning to use. This page is great for that docs.microsoft.com/en-us/azure/azure-resource-manager/management/azure-subscription-service-limits?WT.mc_id=AZ-MVP-5003556

@misko566 3 жыл бұрын

thanks for video

@AdamMarczakYT 3 жыл бұрын

Most welcome :)

@Unforgiven2218 3 жыл бұрын

Hello Adam, nice video and very clear explanations. Can we send structured data in the event like JSON ? is there any limit of the size in byte for the event message ?

@AdamMarczakYT 3 жыл бұрын

Hey, thanks! Yes you can send JSON. For limits and quotas for event hubs check out this doc docs.microsoft.com/en-us/azure/event-hubs/event-hubs-quotas?WT.mc_id=AZ-MVP-5003556

@vedantshirodkar 2 жыл бұрын

thanks Adam for clean explanation. Can you explain what is the role of epoch in event hubs ? I went through few articles but didn't quite got it. :/

@olexiyusov1279 3 жыл бұрын

Adam thank you for video! Is packing events in batches a preferred way to send events or we can send them one by one?

@AdamMarczakYT 3 жыл бұрын

If you send a lot of events very fast then I'd batch them together to reduce networking load.

@vaasu 4 жыл бұрын

Hi Adam, Nice presentation!! I have a question for you. How we can pull data from Server sent Events (SSE) to Azure ? ( Azure Eventhub or Blob storage)

@AdamMarczakYT 4 жыл бұрын

Hey Please elaborate on your scenario as the description is too generic. If you are talking about azure web applications then you should use Azure Application Insights service.

@hemantchandekar3465 2 жыл бұрын

Thanks for great learning of EH, can you also add confluent kafka and difference between EH vs CK.

@JOHNSONLOBOlobojony 4 жыл бұрын

I am just enjoying your presentations and Demo. they are so clear and to the point. could you help us by covering topics related to AZ 204 certifications as well?

@AdamMarczakYT 4 жыл бұрын

I'll check what will be my next goal after I will finish az-900 course :).

@GG-uz8us 4 жыл бұрын

Very informative. Thank you. I am a little bit confused how to do checkpoint in Databrick (Notebook) when preforming stream processing from Event Hub?

@AdamMarczakYT 4 жыл бұрын

That is a great question. I think it's simply using .option("checkpointLocation", "") when using readStream as per spark documentation. spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing

@MiszterTorpedo 2 жыл бұрын

Great video. Is it possible to set role base access for the consumer groups? I have basic tier so only have the default one and I can't set it for that.

@nayeemuddinmoinuddin2186 4 жыл бұрын

Thanks Adam for wonderful video on Event Hub. Just a request can you also make a video on Azure IoT Hub.

@AdamMarczakYT 4 жыл бұрын

Great suggestion! It's on the list :) thanks!

@liborj1997 Жыл бұрын

Hi Adam, thank you for the great video. I wanted to ask you if you have a video about how to configure a VM to send OS logs to the Event Hub. I am struggling to see how to set it up so I can consume the VM's OS logs and send it to Splunk. thank you for your suggestions.

@strategistoptiontrading 3 жыл бұрын

great work adam, but I am still not clear were was .net code was written? It was written on local system visual studio?

@kumoart7767 3 жыл бұрын

Great tutorial Adam! Do you have any tutorial for send recv in Python also?

@AdamMarczakYT 3 жыл бұрын

Not yet! Thanks for watching! :)

@sonofspm 4 жыл бұрын

Your efforts in preparing the content are commendable (y). Which Softwares do you use or recommend recording and editing the videos? I also wish to prepare some videos.

@AdamMarczakYT 4 жыл бұрын

I use Camtasia to record screen and Davinci resolve to combine recording from camera with screen and do extra effects. But in most cases camtasia is good enough.

@sonofspm 4 жыл бұрын

@@AdamMarczakYT Thanks for letting me know. I'm inspired by the way you present and thought of creating the tutorial videos. Though I'm new in creating videos and I would need expert to advise me to make the perfect videos. I got Filmora and created a small video on Git Introduction. kzfaq.info/get/bejne/a8BpnKdkz77Fj40.html

@sonofspm 4 жыл бұрын

I have subscribed to your channel because you have good content and nice presentation skills.

@AdamMarczakYT 4 жыл бұрын

Thanks. Some day in near future I plan to release small tips and tricks video of what I learned over the last year doing my videos. This could be helpful to you. Thanks for stopping by.

@run-forrest-run 3 жыл бұрын

@Adam Marczak, I am confused about which criteria should be used for the amount of partitions needed in case of maximal throughput. The official documentation states: "The number of partitions in an event hub directly relates to the number of concurrent readers you expect to have". later the documentation also states: "Choose number of partitions based on the downstream parallelism you want to achieve as well as your future throughput needs." I would think that the partition should be based on the incoming data. Not on the amount of readers (consumers) who are eventually reading the data. Maybe I am missing something here?

@AdamMarczakYT 3 жыл бұрын

In general all those sentences mean that ideally you should have 1 separate process reading from 1 partition or you might get duplicates while processing. But partitions allow you to scale horizontally across nodes.

@deepanshuaggarwal7042 3 жыл бұрын

Hi Adam, are you creating any other azure exam playlist just like AZ900 ?

@TheVask21 4 жыл бұрын

thanks for the video Adam, however I was asked to use function app for both producer and consumer so my question is - since function app needs a trigger, is there a way use the event hub as trigger ? in your case, i can use function to push but for consumer how will make consumer to listen which is not a trigger point. or where would you upload the consumer code in Azure ? also in my case i am supposed to use kafka which is configured in a different place.

@AdamMarczakYT 4 жыл бұрын

Documentation in this case should always be first thing to check. It very nicely explains what you are looking for docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp Thanks for watching :) feel free to check my intro into function apps if you are just starting!

@TheVask21 4 жыл бұрын

@@AdamMarczakYT thank you much Adam :)

@govthamreddy 3 жыл бұрын

Great video.. I have one doubt if I want to handle 20k messages/ second. Do I need to add 20 partitions with 20 Throughput units? Is it one to one ratio?

@AdamMarczakYT 3 жыл бұрын

It's a little bit more complex than this. Please check this doc as it nicely explains key things to note docs.microsoft.com/en-us/azure/event-hubs/event-hubs-scalability?WT.mc_id=AZ-MVP-5003556

@govthamreddy 3 жыл бұрын

Thanks for great video. I have one small query i have two partitions in my event hub, Can i trigger two function apps using eventhub trigger with specific partitions?

@AdamMarczakYT 3 жыл бұрын

Unfortunately seems like default event hub bindings do not have this as configurable property docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp&WT.mc_id=AZ-MVP-5003556#configuration

@anassaifi3319 3 жыл бұрын

Hi Adam, Great work, I just wanted to know that is it possible to send real time frames of a video to event hub? or does event hub deal with image data?

@AdamMarczakYT 3 жыл бұрын

it's possible but not recommended, maximum event message size only only 256 KB, better upload this to a blob storage.

@anassaifi3319 3 жыл бұрын

@@AdamMarczakYT Okay, Thanks Adam

@anassaifi3319 3 жыл бұрын

So for this will iot edge work fine?

@anassaifi3319 3 жыл бұрын

And also don't we need some sort of intermediate between device and blob storage for capturing real time frames. Will iot edge work there?

@manisharajan3773 2 жыл бұрын

Hey Adam ! I was really expecting that u will create multiple event hubs under single namespace. Could you please explain that?

@dhanunjayreddyvalluru8161 3 жыл бұрын

Great presentation. I am new to this, still understood. What is the difference between Kafka topic vs event hub. Both has got partitions and consumer group concepts.

@AdamMarczakYT 3 жыл бұрын

Check this stackoverflow response on this stackoverflow.com/questions/63953085/azure-event-hub-vs-kafka-as-a-service-broker

@neil3507 3 жыл бұрын

Wow great explanation..any tutorial on sending msgs from TCP to azure event hub ? So basically I will need a sender application from TCP to azure and a receiver application in azure to capture results ? Or the sender application can directly store into adls ? Am confused

@AdamMarczakYT 3 жыл бұрын

TCP is a messaging protocol. So it's like a car. You can't move people from the car to the shop. You can move people from the home to the shop using the car. That said, in the video I've shown how to use SDK to send messages to event hub (it actually uses TCP underneath the scenes). Maybe that is your solution. But sender can store directly in the blob if it wants to. It really depends what are you planning to do with that data. Thanks for watching :)

@neil3507 3 жыл бұрын

@@AdamMarczakYT can we read from azure event hub and store it directly into our mapr location? Without storing in adls or blob ?

@xavierjohn9 3 жыл бұрын

Azure has fixed the UI, now you have to pick at least two partitions.

@AdamMarczakYT 3 жыл бұрын

Did they? I think it depends on event hub tier whether it's a standard or a basic one. I'll check later :)

@ravikumarkumashi7065 Жыл бұрын

Great tutorial❤ is it possible you make vedio on connecting to azure event hub using azure service principal without using shared access key

@ry-financials 3 жыл бұрын

Thanks Adam for very good demo. Would you mind sharing the dotnet code that you showed ?

@AdamMarczakYT 3 жыл бұрын

All my videos include link to the source code in the video description :)

@shouryaslife3606 2 жыл бұрын

Hi Adam, nice video. I implemented tha same by watching this video. But further I want to read avro files and extract JSON data from body section in avro file. Further I want to send that extracted data in to SQL server DB table. Could you suggest How I can do it? Thanks in advance.

@joseandreslopezmunoz9972 3 жыл бұрын

Thanks! And the next pass to read the avro?

@AdamMarczakYT 3 жыл бұрын

Use Azure Data Factory to read Avro files.

@Umerkhange Жыл бұрын

At 29:51 - When you have set the retention period 7 HRS and CAPTURE:ON and TIME WINDOWS: 1 min. Will it still save the event in Azure blob storage and why as our retention period was 7 hours

@saurya72 3 жыл бұрын

One request ...don't know If I am asking more as you already have done a great job....May you please add some videos containing the business used cases and how these services interacting with each other in real time . Thanks in Advance.

@AdamMarczakYT 3 жыл бұрын

If I can I will create implementation videos on the future, thanks!