 Hey, Thomas here for another series of Azure Unblocked. Today's episode, we're going to speak with Manoosh Prasant about Event Hubs on Azure Stack Hub. So stay tuned. Hey, Manoosh, how are you doing? I'm doing good, Thomas. How are you? Doing very well. Happy to have you on today's Azure Unblocked. Perfect. You are a PM working on Event Hubs on Azure Stack Hub. So can you quickly introduce yourself a little bit on what you're working for? Yeah, absolutely. My name is Manoosh Prasant. I'm the PM for Event Hubs on Azure Stack Hub. I've been part of Azure Messaging team for four months now. It's been great and as a product owner, it's on me to drive the roadmap and take the features to the release like GA and things like that. So those are the things that I manage for Event Hubs on Azure Stack. Okay. That is amazing. By the way, that sounds like a really interesting job. Thank you. Absolutely. For the people who are watching this and are probably not so familiar with Event Hubs, can you give us a very short introduction of what Event Hubs is? Yeah, absolutely. So Event Hubs, think about the streaming jobs. If you want to stream the data and if you want to ingest those streaming data, Event Hubs is a perfect service, whether it be it on Cloud or on-prem like on Azure Stack Hub. So think about all the use cases where you're trying to ingest the streaming data, Event Hubs are perfectly capable of handling pretty high workloads. Okay. That is awesome. Yeah. I remember I worked with Event Hubs in Azure itself, but now, again, as we are here for this Azure Unblocked, we're talking about also bringing Event Hubs to Azure Stack Hub, which basically allows customers now to run Event Hubs that service we have in their own data center or at their own Edge locations. So can you explain me a little bit of the use cases why they would, for example, do that? Absolutely. So the first thing is, let's think about the oil gas industry, where think about those smart grids. Think about those smart grids trying to stream the data, and you can use those data to understand, if any of those grids needs any maintenance, for example, I can get some insights into those data and predict or prevent some kind of a batting that's happening on those grids, for example. So maintenance use cases, you can pretty much stream those data through Event Hubs on Azure Stack Hub. Similarly, healthcare, think about those hospitals where there's a lot of devices trying to stream the data, and also it could be small IoT sensors, for example, also trying to stream the data related to healthcare. You could as well use Event Hubs on Azure Stack Hub to ingest those streaming data, and you could also find some good insights from the data to understand if any of those devices needs any maintenance, or sometimes you could also use the data to get some insights on general profile, like maybe understand how the patients are doing, in general, things like that related to healthcare, that's going to be very beneficial if you use Event Hubs to ingest those data. Retail, for example, I'm offering something to the consumers, how do I get feedback from consumers, whether they're happy with my offering, like I'm doing some promotional events and pricing schemes, for example, how do I know that whether those are effective? How do I know that my consumers are always engaged? So Event Hubs could be helpful to, again, ingest those data, and you could always want insights to get those feedback, that's going to be a great use case on retail. In general, application operations, like think about those logs. Logs have a lot of times of valuable information. I could use the logs to do some monitoring, some troubleshooting, or pass the logs and maybe find some patterns to see if there's some issues, for example, things like that. You could always use Event Hubs to ingest those logs and hence do some analytics on those logs as well. Now, financial, think about those data related to those stocks, for example, you could use Event Hubs to ingest those streaming data and also like credit card transactions, for example, like anomaly detections and things like that. If you want to find some anomalies, for example, in some credit card transactions, you could always ingest those data through Event Hubs and run some analytics or anomaly detections to find any kind of a fraudulent activity. That's one, just an example, use case of financial. Security, for example, in your home, you have all the security devices trying to stream the data. You can always ingest them through Event Hub and run some threat intelligence, run some analytics on that to identify any kind of, like maybe do you have any security holes, for example? Is there something you could do better? Things like that. In general, using the data to get some feedbacks, you could always use Event Hubs to ingest those data. Now, there's many, many more use cases, right? I mean, these are just like simple, quite popular use cases I'm mentioning, but there's many, many more things you could do with Event Hubs. No, there are some pretty cool use cases, absolutely. I mean, the healthcare makes for me a lot of sense. Also, finance, obviously, all of them make a lot of sense, but there is, whenever you have basically various team, a lot of data, and you don't necessarily want to rely on the connectivity to the cloud, for example, you don't have like bad or no connectivity to the internet at all, you probably want to have a service which can handle that in your own location, not necessarily need to, again, rely on this. In some things, like as you just showed, in some use cases, it can be very critical, right? So you want to take that out of the equation. Absolutely, and a lot of these use cases, they might not be needing them to be ingested into the cloud, for example. Before it enters the cloud, they might want to do some cleansing of the data, for example, or some enrichment, for example. That's where even Terps on Azure Stack really comes into play. That's awesome, that's awesome. So that is super interesting, and I'd love to hear more about that. Now, what you showed, what your experience basically said, okay, you're working on events on Azure Stack, can you give me a short overview about how that actually looks like in a technical sense? Absolutely. So when you look at Azure Event Hub, there's two things that's important, right? One is the producer, the one that produces all these events, and the other part is the consumer. It consumes all these events from Event Hub. Now, producers, they could use any of the three protocols that are available, which is like HTTPS or A&QP or Kafka. They could use any of these protocols to produce the events. And on the other hand, the receivers too can use any of these three protocols to consume the events as well. And now, when you have these consumers, it's not like one or two, it's many, many consumers, potentially there would be many, many consumers. And in order to have many consumers, the concurrency is pretty critical, because one consumer trying to consume events and waiting for the other is not really practical, right? So the concurrency is critical, and in order to enable the concurrency of consuming the events, partitions are designed. So within Event Hub, all the events just get into this different partitions, which basically then consumers read from the different partitions, and that's how they concurrently do that. And one consumer consuming events should not impact the other consumer, also trying to consume the same events. So which means there might be some kind of an isolation required, right? Like each consumers might need like their own private snapshot of events in the Event Hub. And how is that enabled? That's through this consumer groups. So consumer groups provides the consumers a private view of events in the Event Hubs. And in that way, each consumers can independently view their data irrespective of what the other consumer is doing. So from a high level, I would say these are all the big players that are involved with the Event Hubs on Azure Stackup. Okay, now that's pretty interesting. I mean, obviously the system needs, you just brought up a really large amount of points. Like it obviously needs to be scalable and it needs to work for multiple consumers. So again, there's a lot of engineering work which obviously goes into this, right? Yeah, absolutely, absolutely. And that's why the partition thing is pretty critical for a scalability aspect and also the concurrency aspect and consumer groups. So yeah, so pretty much. Yeah, no, that is fantastic. Again, I always love to know really understand how things are working. The next thing I really want to talk about is we already covered this a little bit with the use cases but I want to understand a little bit deeper now on like what can Event Hubs do for me and why would a customer need it? And so I'm sure there are like a couple of things you can explain here. Absolutely, I mean, great question, right? I mean, before even diving deep into Event Hubs I probably need to understand what are the key things that Event Hubs is bringing onto the table and why do I need to use that, right? Like why do I need to use it on stack up? So let's try to answer those questions. The first and foremost thing what Event Hubs brings is processing large volumes of events per second. Like, you can ingest huge volumes of events per second and also consume huge volumes of events per second. So processing like really, really large volumes of events per second is the key feature of Event Hubs, you know? And now with Event Hubs you can also do processing if you need real time or also batch if your use case demands. Now real time, so generally what you could do is you could have as your stream analytics job, for example consuming the events real time from Event Hub and maybe do some kind of a processing like maybe do some kind of analytics on that real time and maybe through a Power BI you could easily visualize it real time as well. So just a simple use case of how you can do a real time solution. In terms of batch, Event Hub supports a feature called Capture where it will, if enabled it's going to store the events into like a storage like block for example. And now applications can start reading in a maybe in a cadence from the block or in a batch to process the events and do some kind of an analytics on that maybe. So that's why Event Hubs will allow both the cases like maybe a real time or even the batch processing. Now the third point we touched upon this previously where the concurrency if multiple consumers need to concurrently consume the data then you really need the partitions and Event Hub is already supporting that partition logic. Kafka, now the Kafka support is amazing. The Event Hub is going to expose an endpoint for Apache Kafka producer and consumer APIs. So which means it's actually going to support the pass on prem model. So let me explain what the pass on prem model is. Without this, for example, you might have to manage your own clusters and things like that. But with this support, Event Hub is already going to do that for you. Meaning on your stack up with Kafka support you can get the, you can manage the Event Hubs is going to manage all those clusters for you. And hence, you can actually truly get the pass on prem expedience with this Kafka support. Okay, this is pretty cool. This is really cool. Again, I don't need to like care of it. It really is a service which is offered to me to actually take care of it. Absolutely, absolutely. And the other cool thing about Kafka is you can produce with one protocol and you can consume with other. For example, you can produce with Kafka and then consume with the MQP or you could do vice versa where MQP and Kafka or you can do both Kafka, things like this. It's going to support this multi protocol which is pretty cool, I would say. So I can also use it kind of like to translate from like one protocol to the other protocol if I need to, right? If I have like the consumers, they work with a different protocol I could definitely leverage that. Yes, absolutely, absolutely. Now, so let's answer the why question. Why do we need to use Event Hubs on Azure Stack Hub? Think about those use cases where, I need to ingest the data to maybe Azure Cloud but having said that, I need to do some kind of a processing where I need to do that on-prem. I need to do that locally on-prem. Maybe do some kind of an enrichment or maybe sometimes due to privacy concerns I probably want to remove all like maybe private PII like identifiable information, for example. So you could do any of those cleansing or enrichment on-prem through Event Hubs. So basically you can ingest those events through Event Hubs on Azure Stack Hub and then have like consumer do this cleansing or enrichment and then have those data ready to be ingested into the cloud again through Event Hubs. So the great news is Event Hubs is supported both on Azure Cloud and Stack Hub. So which means it's going to support this perfect hybrid cloud model where you can use Event Hubs locally, do some processing, and then again use Event Hubs on the cloud. Okay, that is also something pretty cool and I hope we can talk about that in just a minute because that is really like this is just real power, right? So I could even like let's say I have a use case where I just need Azure, I could run like Event Hubs obviously in Azure, then I have use cases where I only want to run it on-prem but then the great thing now you just told me like is like I can even combine them. Like I can say, well, let's do some processing on-prem. As you said, you probably have data sovereignty challenges there, you have like network connectivity challenges. So you want to do some like of the critical parts on your Azure Stack Hub system but then you can take the data which can actually go to the cloud and then use Event Hubs in Azure. And so I'm really waiting for that one, that you show me that one. Great, great. So stay tuned. I think we're going to double click on this a little bit with the thought for details. So the next use case is, suppose I, for example, I want to ingest the data into my on-prem and I do not want the data to leave my on-prem system. For example, I do not want that to be visible outside from my on-prem network, for example. Then in those disconnected solutions, for example, I do not want my data to be exposed to the internet, for example, or even Azure or some of the public clouds. Then guess what, Event Hubs is also supported for those disconnected solutions. In fact, we just did Event Hubs for disconnected solutions just last month. So with this, this scenario is also enabled, which is great. Okay, this is like when I, for example, I have scenarios where obviously I work with government or with companies which, again, have scenarios where they don't have internet connectivity or, for example, typical cruise ship scenario, which we all show a lot of times. But obviously a lot of customers have that request that they also need to run it without having any connectivity. And so that is absolutely great to have that. Now, one thing I want to know, and it's like, so obviously one is running in Azure, one is our Event Hubs in Azure, and then I have now Event Hubs in Azure Stack, can I use the same tooling or do I need to use different tools or how is that working? Great question. So we support this parity where, for example, if you write any applications on Azure cloud, for Event Hubs rather, the producer consumer applications, now you could use the same on Stack as well for Event Hubs. So any applications you have already developed on Event Hubs on Azure cloud, you can reuse the same because the SDKs, even the portal experience, we are going to take a look at that a little bit later, the PowerShells, the CLIs, the SDKs, the samples, all is pretty much the same. There's a parity on both the platforms, which is great. Oh, that is awesome. Like this is like really what it is about. So if I already have tools, if I'm familiar with the tools in Azure, I'm automatically familiar with the same thing on Stack. Absolutely. Yeah. Okay. So I want to come back about that hybrid scenario. You really mentioned that I can actually take advantage of both. So I can actually run, again, some parts of my application on Event Hubs on Azure Stack, and do some processing there, and then take it further for other parts like on Azure. Can you show me and talk a little bit about that? Yeah. So let's dive deep into this. So I'm going to show a sample architecture of how that hybrid solution could look like. Again, this is just a simple architecture diagram. By no means this is complete because there's plenty of things you could do as well. So let's navigate from left to right. So of course, you have all this multitude of data sources, different types of data, for example, streaming data, it could be logs, it could be weather data, or different business applications. So this box here, what you're seeing is the Stack Hub, and this box is the Azure Cloud. So let's dive deep into each of these boxes. So you have this data, and like I said, I'm not comfortable ingesting this data directly into the public cloud. Rather, I want to ingest it locally and do some processing, and then have the process data, then go into the public cloud, for example. In such cases, this is the workflow that we can follow, where all of this data gets ingested through Event Hub on Azure Stack, and now we'll have a consumer basically consuming these events and doing some kind of a processing. It could be enrichment. I mean, I want to add some kind of data to enrich the ingested data, or I want to remove some, for privacy concerns, I want to remove some personal identifiable information, for example, I could do any of those, like cleansing, enrichment here, and I could optionally store it in the blob as well, if my use case demands that. But basically, once I'm ready with the process data, that's when I'm just gonna ingest this into Event Hubs on Azure Cloud. So think about this, this is actually doing two roles. It is a consumer for this Event Hub, but actually a producer to this Event Hub on Cloud. That is the cool thing that this consumer does. Now, once it's ingested into Azure Cloud through Event Hub on Azure, then depending on my use case, I could do, I mean, any of these workflows. For example, if mine is a real-time scenario, I want to get some real insights, real-time, from these events that are that we're ingested, then you can run these events through an Azure Stream Analytics, where I could write some queries, and based on my use case, I could do some kind of an analytics, and I could also have a Power BI dashboard to visualize these insights that Azure Stream Analytics is generating, for example. So this is like a real-time workflow, I would say. Now, on the other hand, for example, if I need to store these events, maybe in a SQL format, because there's already a capture feature that's enabled on Event Hub, and if it's enabled, then it's by default going to store all these events into like Blob, for example, some storage. So that's already available by default, but my scenario is that I need it to be in SQL format, for example. Then what we can do is, as soon as the capture, the events are captured from Event Hub, it's going to generate these capture event, and we can leverage that event through Event Grid, and have Azure Function, for example, get that event delivered, and once that event gets delivered, then Azure Function can go read those events from the Blob and maybe do that conversion into SQL, and maybe store it into a SQL data warehouse. So this could be a SQL kind of a scenario, I would say. And many more, this is just a couple of scenarios I'm highlighting, but really, once the data is available, based on my use case, there's many, many different workflows that's possible. Yeah, it looks like there's now, especially with the last thing you showed me, basically, then from Event Hubs, I can actually trigger functions, and from there, I can really play around through really everything I actually need to do, and Event Hubs is just a great tool for this. And I also like, again, the part where you said, coming back on the first part of the thing where you said, well, we could also animate, do some of the data to make sure that it's not, there's no personal information in it, right? I mean, sometimes I really have these requests, or as you said, I can also enrich the data and add something to it, which I need to for my analyzers in Azure, I want to run it there or for long-term storage. So that is pretty cool. Now, I know that, obviously, because Azure Stack Hub is a very, like kind of like a small version of, like if you're a part of Azure, compared to Azure. So I'm sure that there must be some limitation, even though we have great consistency with both of them, like you said, I can use the same tooling, I can use the same code and all of that. But is there, are there some like limitations or some features which are not available in one or the other? Yeah, yeah, so let's look into this comparison of our offering on EventTubs on Azure Stack versus the, versus the Azure. The good news is Kafka support is available on both. So that's a great news. The Capture feature, which I was talking earlier, that's available on Azure Event Hub, but currently it's not available on Azure Stack, but the good news is it's already part of our roadmap. So we are already trying to plug this gap to say. And then the good news is the cluster creation is available on both. What this means is on Azure Event Hub, we generally support three tiers, standard, basic and the dedicated. Now the basic and standard are multi-tenancy models, but the dedicated is like a single-tenancy model where the customers can create something called capacity units, which is basically clusters. The customers can create those clusters and basically it's single-tenancy model. So they can use it pretty much, the entire resources for themselves to use. So what we have done is we have supported this dedicated model on the Event Hub on Azure Stack. So which means the cluster creation experience from Azure Event Hub is exactly the same as Event Hubs on Azure Stack as well. So the customers can again, create the CUs and then have the entire cluster basically for themselves. Now the admin and operator, admin and operator diagnostics, those are pretty Azure Stack Hub related concepts. So that's not really applicable on Azure Event Hubs, but the Azure monitor, which is really useful for diagnostic settings and in general diagnostics and troubleshooting, that support is available on both the platforms, be it on Azure Event Hub or the Event Hub on Azure Stack Hub. Pricing, like I said, the dedicated model that we support on Azure Event Hub, that's the same model that we support on Event Hub. And as a result on Stack Hub, the pricing is more based on core comp. Like how many clusters you're using, you're basically gonna get charged based on that usage. Okay, that is awesome. So you basically get the dedicated part, no basic or standard in that sense. You really get like the high level part, but you get obviously, because it can obviously be used in disconnected scenarios, you will be using another pricing model. And obviously it makes much sense. Obviously why will be with the operator experience. As you said, in Azure is not needed because we basically Microsoft operates that part. And then obviously it's cool that we already have all these features, especially the integration of Azure monitor and then also that you already highlight here the roadmap. So this is pretty awesome. Now, again, thank you for showing this. Now, what I wanna see, obviously, we wanna see that and also the viewers of this video, I wanna see it in action. So is it possible that you show us a quick demo about what you can do with Event Hubs on Azure Stack Hub? Absolutely, I'll be glad to provide a demo. And in fact, I'm gonna provide the demo based on the hybrid cloud model that we talked about. So before we dive deep into the demo, let's quickly look at from a high level the architecture, so that the viewers understands, what are the things that's part of the demo and what's the architecture? What are the pieces that are involved with the demo? So suppose let's say, I'm a business owner and I have some stores maybe in like Redmond area, for example, and then my intent is to ingest all those business transactions, which has pretty card details and things like that. My intent is, I would like to know, among all my business transactions, how many of them have fraudulent credit card usage, for example, how many of them may have some anomalies that I would like to know that in all, because it might be impacting my business, if I have more of this fraudulent credit card transactions. So that's my intent as a business owner that I would, number one, I want something to ingest and have the data ingested someplace, but also I need some kind of anomaly detection to understand how that's impacting my business. So with this, what I have is, as a business owner, I do not want all the customer details, which has credit card transactions directly in the Azure Club. So what I would like to do is, all this data that's coming from my different stores, I would like to ingest them on-prem, on my Azure Stackup instance, for example, through Event Hubs. So I'm gonna ingest all this credit card transactions and other transactions through Event Hub, and then I have this consumer basically trying to cleanse the data. I would like to remove any kind of a PII information from that and maybe do some enrichment as well. So I'm gonna do that and then basically ingest all this details or events into the Azure Event Hub. And once I get the events on the cloud, I would like to use the Azure Stream Analytics and write like a query to basically run real-time, a query to understand like, how many credit card transactions of fraudulent. In my scenario, what I've done is, if I see any two transactions coming from the same credit card, which is in different locations and five seconds apart, then I'm flagging that basically. I mean, to be practical, if I have one credit card transaction which is coming from Redmond and the other credit card transaction which is coming from Chicago, for example, and they're merely five seconds apart, same credit card, then something is wrong, right? I mean, that cannot be practical, right? So it's a simple query that, I'm simple logic that I'm using to flag it, but it could be much complicated than that. But this is just to show the, this is just to demo the, to highlight the importance of this hybrid model, let's say. So now, so what I've done is, I have this Python scripts to basically simulate these, these transactions. And then I have the event tab stood up, on my instance of Azure Stackup. And then, I have this Python script here, which is consuming this events and then, ingesting them into event tab, and then I have the stream analytics job running. So let's dive straight into those. Now, so this is the portal. So like I said earlier, the portal experience is the same for, when you create an event tab on Stackup, or event on Azure, the portal experience is going to be the same. So this is the portal for my instance of Azure Stackup. And this is the event tab on my Azure cloud. So you can see like portal.azure.com. So this is the event tab instance on the cloud, versus this is on my Stackup. And you can see that the portal experience, the creating the namespace, and creating like an event tab, it's pretty much the same, you know? Okay, so then we have this event tab that's already created. Now, just to save some time, I've already created this before, just to get going to the demo. And I have this stream analytics job set up, which is running right now. And this is the query. As you can see, it's a simple query. What I've done is it's looking for, if there are any two transactions with the same, if the credit card ID is the same, and the time difference is just like sometime between zero to five seconds, and the most important thing, the location is not the same. Then it's basically taking account of how many of those transactions I'm seeing. So that's what this query is trying to do. So let's run these scripts and show you the output really quick. So again, this script is ingesting events into the event tab on stack. This is the receiver, this is the Python script, which is doing two things. It's number one, consuming the events from Azure Stack Hub, and it's ingesting them into the cloud, into event tab on the cloud. And the final, this one is a simple one, it's just consuming events and just printing it out, just to show that the transactions has made it all the way from Stack Hub to the Azure. So let's run these scripts really quick. So this one is simulating those credit card transactions and it's already sent to the Azure Stack. Now this is receiving it from Azure Stack Hub and you can see it. It's already starting to receive. So you can see in the prints, it might be difficult to see, but it's basically saying I received this event and I sent this event to the event tab. So it received the event from Stack Hub and sending the event to the event tab on Azure. Now last, we'll start receiving it from the Azure event tab from the cloud. And you can see all these transactions which we injected here to the Stack Hub, it has made it all the way to the cloud because from these prints, we can see that it's receiving these events from the Azure event tab. And now let's go back and run the query. So now basically what I did is simulate like 12 such transactions where the credit card ID were same and the location were different and they were just five seconds apart. And now the stream analytics job is detecting those 12 transactions and it's basically, it's displaying that there were 12 such transactions which were fraudulent, which may be fraudulent. Let's put it that way, which may be fraudulent. And now what I can do is I can take it forward from here, maybe just have a Power BI display this information. It could also display many more which this query can generate and have Power BI dashboard up and running which means like I could keep seeing the dashboard periodically to understand how these fraudulent transactions are impacting my business, for example. Yeah, that is pretty cool. I mean, this is exactly the example like it shows the power of having something running in your own data center or at your edge locations and then combining it with running it in Azure to do some advanced analyzes. And as you said, with that data now and also the show, like when you showed the different architectures before, it would be now also something we could either as you said visualize it in Power BI or you could build a function basically to like run a certain task or send out an email or a message in Microsoft Teams or any of that, right? To make sure that we actually get notified about this. So that is really, that is really, really cool. No, I really like that solution. That's right, that's right. Yeah. So perfect, this was awesome. Absolutely great to see event hubs on Azure Stack Hub. So I'm sure now people have obviously questions where can they learn more about it? How do they get started? Where can I find more about this? So can you tell me where would people go? Absolutely, I mean, there's plenty of documentation on event hubs or even on Stack Hub or like event hubs on Azure Stack Hub. So plenty of documentation which talks about what event hubs is and what are the features for example and portal experience, like there's plenty of samples to show how I can get started using event hubs to start like producing events and consuming events and things like that. There's also many videos actually. There's plenty of videos which actually goes in depth. Remember that one slide we talked where I introduced, I gave a high level overview of that event hub. There's actually videos which goes in depth, explaining those consumer groups and partitions and Kafka and things like that. So there's plenty of resources available. That is awesome. So we will definitely link all these resources in the description below this video. So with that, I really wanna say thank you for your time today. That was really, really interesting. For all the viewers, thank you for watching and hopefully see you in the next one.