 Welcome, everyone, to Ops 115. I'm here with the Meir Mandolovic principle PM working on Azure Monitor Logs, formerly known as Log Analytics. So stay tuned. Hey, Meir, how are you doing? I'm doing great. Thank you very much, Pierre. So we're talking about Azure Monitor Logs, or formerly known, as I mentioned, as Log Analytics. This is a service that basically is underpinning a lot of other services in Azure, which makes it very, very important to get the design right, correct? Correct. We are the foundation of services like Azure Sentinel, like Azure Security Center, automation, and few others. So we have this concept of let's keep all the logs in one place and architect it correctly, and you get lots of advantages out of that, and I will touch some of them in the session today. Oh, that's perfect. So exactly, what are we going to be covering today? So I would talk about, sorry for that. So today we would look mainly on two topics around like high-scale deployments and enterprise deployments. One of the reasons for that is that throughout the last year and a half, what we see is huge uptake of the service. We see lots of customers. When I'm saying customers, I'm saying like customer from all sizes, starting in Mamas and Papa's shops, and going all the way up to 1450 companies. So every vertical in the world, every geography you can think about. So you can think about like we have banking, even like the most sensitive type of banking investment, and we have manufacturing, we have retail, every vertical you can think about. We have lots of big and the leading companies that are using us. It's one thing to deploy one system in one place. As I explained, like logins, like the most trivial problem you can find. You just have a file in you, or make logs. It's not a problem in small scale. The problems and the challenges that you have start to raise when you go to very, very high scale. This is our focus, how we deliver a service that can work in terabytes or tens of terabytes per day, that can get you analytics and results in this scale, and can get you everything working and visibility into your services. And not doing it just for like one machine, two machine, three machines, but to do it in org-wide experience. So this is our challenges, and this is what I would share today. Okay, so when we're looking at designing for a higher scale, are we really looking at, like you mentioned, mom and pop shop all the way up to the big like top 50 companies. Does that mean that the way you architect for a mom and pop shop and the way you architect for a very large corporations are like completely opposed to, not opposed, but at the different end of that spectrum? Or is it a common design approach that you take? So we are SaaS service. We don't look at ourselves as like we give you silvers and do whatever you want with them. We are trying to provide the service. As part of this paradigm, we take care about all the aspects of managing. So of course, if you have like a very small company that just like sends like few megabytes of logs, and they are not even paying because they are on the free tier, they start with their workspace. And they might grow and grow, and we love to see customers growing because they have business success there. And they can grow. And behind the scenes, we are making lots of stuff to move them, to support them in whatever scale they can do. So we have workspaces that started with megabytes per month and moving all the way to tens of terabytes per month without doing anything. So the system is completely supporting that. Still, large organizations and complex organizations have additional needs. And this is what we would touch here. Like they have additional considerations that we have to talk about. But the platform itself is completely to respond. You can go from zero to huge enterprise without doing anything. Just pump and pump and add more data. There is no need to deploy anything. There is no need to configure anything. You just won't. But when we talk to this like very large and complex organizations, they have their unique requirement. And this is what we would touch today. OK, perfect. So teach me. So let's start talking a little bit about what is Azure Monitor. Because logs doesn't stand by themselves. They are part of a bigger thing. And the bigger thing is Azure Monitor that we deliver. And if you look at Azure Monitor, we start on the left side with set of collection mechanisms. Some of these collection mechanisms are infrastructure that automatically send logs. So if you go to almost every Azure service and you go and there is a diagnostic setting there and you just click on it, you checkbox the entire thing and you get your logs inside log analytics. This is one way of doing stuff. We also have application level monitoring, like as you know from application insights, that you go there and you add instrumentation to your application. It might be code based. It might be code less, but it is application level. It has the notion of the application. It has the variables that you have in your environment and everything there. So that's another method. And of course, we have agents. And we have the existing agent, which is very widely deployed like the MMA, very, very common. We are in the process of moving to the new agent called Azure Monitor agent, which consolidates this. Agent is always like a mess. And we are working hard to consolidate it together. So we started with MMA was a consolidation of few other agents. And now we are consolidating the few ones that are not there. And we would have one agent that is modular and allow you to do everything, to collect stuff like that. Okay, so that agent deployed on virtual machines in Azure or on any machines on prem will then send whatever you configure it to send back up to Azure Monitor logs. Correct, correct. We have now like, if you look at VMs, we have like three models. We have VMs in Azure. We have VMs that are completely off Azure and have nothing to do with Azure. Where you just put the agent. And then we have in between, we have Azure Arc. And Azure Arc actually opens us lots of options. Azure Arc is much easier to manage, much easier to deploy. So in the past, if you have deployed an agent and forgot about it, Azure Arc allow you to set policies and it's allowed you to enumerate the agent and lots of stuff that the customer asking us for years. So we highly recommend you to go through the Azure Arc option and you can deploy Azure Arc everywhere. We have customers deploying them in like crazy environments that are completely different. And I have heard that some customers are using environments not Azure. I don't know why, but they might do that and they can put Arc also there. So it opens a lot of variety. So because what we hear from our customers that they want a central place to put all their logs and metrics and they don't care where the data is coming from because they don't want, when you have two kind of data sets within the organization where you have two kinds of languages, there is a problem with the translation. There is mismatch in the data. So having like, they are asking us all the time to keep a single way of collecting stuff and reporting them and querying them. Okay. And on top of that, we also support all kind of other things, what we call custom sources. So if you need a dimension, a measurement that we don't have, you can ingest it. If you have like your own code that is generating whatever it is and you want to send us logs through HTTP API, that's also available. So we have very wide, a lot of different collection mechanisms. Okay. Now, all these mechanisms boils down into two repositories of data. We have logs and we have metrics. And if you want to have good visibility, good observability of your platform, you need both. So in the past I heard like, we need only logs or we need only metrics. The truth is that if you are in a, if you're doing real application in real complex environment, you need both of them. Okay. And each one has its own strength. So logs gives you the verbosity. Logs give you much more retention and analytics capabilities. And metrics give you the very fast moving, the very instantaneous data points. So we have in Azure Monitor both of them and they are working very well together. And then top of them, just having logs and just having metrics is not enough. We need to provide tools. Okay. And tools to use them. So we have, if you're seeing in the top line, we have what we call insights, which is out of the box components that we give you. And the insights are stuff like VM insights and container insights, which is super popular. We see so many customers enjoying that. And of course we have application insights, which is our most veteran insight in the market. It has huge array of ways to analyze what your application is doing. And we have tens more of them. And new ones are coming all the time. So that's like the out of the box, very, very rich in time. Okay. That's not what we call, that's not what we use to refer to in the days of OMS has solutions, is it? It is similar. We kind of evolve the solution concept into what we call insight. Insight is something more transparent. It's not deployable. It's there. Now, we know that the insights we provide out of the box are great, but customer have more needs. They need to do more things with that. And this is why we actually give the customer the actual tools that we use to build the insights. And this tool called walkbook. So all the insights that we build are based on walkbook. And you as a customer, you can get the option to build your own walkbook, put your own logic, your own graphic, your own terminology within that. You can actually fork some of the insights. So the insights are not like black boxes. You can like take them and you can build your own stuff based on them. And of course we have other like super powerful, super useful people love power BI, people love dashboards of all types that you want and you can create. We have lots of customers using us with Grafana, for example, Grafana has a very good support for Azure Monitor. So all of this is like a way to consume and visualize your data. But the next level is about analytics. So it's about how you take this data and you digest it and you get the information out of it. If you ask me that this is why, what I love the most, we are of course using Azure Monitor for ourselves, to monitor, to monitor ourselves. And I like every my day to day is like analyzing what customers are doing and what's working, what's not working. And I'm writing tons of queries, super complex queries to get that. And we have the Log Analytics UI for that, which let you query logs. And we keep on investing there. We keep on adding new capabilities, new, better query auditing, better query management. And we are keep on like adding like new visualizations and way to walk inside. Because we talk to customer and we see like these people that are sitting there and is trying to get the insights out of the existing logs. Okay. And then the next level is actually doing something automatically. So like looking at your logs and just it's good or analyzing them, that's very proactive. But free live environment is super, super dynamic. And you want to code and you want to operationalize everything that you can. So we have alerts that you can take and you can be notified when something bad is happening. We have stuff that you can configure basically every logic that you can think about in the alerts. And you can take these alerts and you can hook them to all other things that we have in Azure. And this is where we ask being integral part of Azure kicks in because like you can take an alert and you pump it into Logic App. Logic App is doing stuff that you call something that is unique to your organization. And we have this mechanism where you can take things in and out. You can use the data or the insights to do autoscale. You can use it to do all kinds of things that are true for your organization because there are not two organizations that are alike. Okay. So that's in a nutshell, Azure Monitor. And we would focus today mainly on logs and describe how logs, what the architecture behind it. So we'd focus today basically on this area and how you design and what's the features and capabilities that we provide for very high scale. Okay. We'll talk a little bit about our buck and we will talk about workspace topology. Oh, perfect. When we talk about workspace topology and our back, just a little qualification. Like by default, Azure Monitor logs, they don't, like you can't, it's not gonna store your logs for an extended period of time, correct? By default, it stores your data for 31 days. For 31 days. So if somebody wants to keep like historical data so that they could do modeling over longer periods of time, are we gonna talk about how we address that? Yeah, that's very easy. It's a property of the workspace. It's one of the things that differentiates the workspace from the other. Okay. We allow retention, we call it retention. We allow you to set retention for the entire workspace or for specific table within this workspace. Okay. All right. The first question that we always get is how many workspaces do I need? This is like the top, the top question I'm getting asked all the time. So I have this slide that try to summarize the issue. First of all, our recommendation is to have as few workspaces as your business requires. And this is like, really like tricky stuff. People will say, well, what does it mean my business requires? It means that we give you a container for logs and you give it the interpretation. You decide what to do with that. Now you don't need to overuse it. So stuff like RBAC, I would explain how you do that without a container. So lots of the other types of containers have this like I need container for every team. I need container that scales high. I need container for RBAC. But with actual monitor logs, this is less of an issue. It's not something you need to think about. Tells me how your organization is working. Like, let's, we are talking to very large organization that have very few workspaces. I mean, like one or three or four of them. And we see other organizations that have quite more of them. And that depends on how your organization is working. How much centralized is your cloud? Do you have internal, internal regulation? Like if you have two teams that have nothing to do between themselves, they have zero overlap now and in the future. And there is not a single person that needs access to both of them. Then there is no reason for them to share the same container. Okay, because that's the question I've gotten a lot in the past is from companies or individuals working for a larger organization that wanted to do like a hub and spoke type of organization where they say, well, we want our this division to have their own log analytics and we want this department to have their own log analytics. But we want to be able to replicate their data into the corp log analytics so that we could have a top-down view of the entire organization. And you're telling me that that's not an approach that we should be going after. Yeah, that's of course we support. You can have multiple workspace. You can have hub workspace and spoke more workspace. And we do have customers that are doing that. But when you do this, you get into multiple other problems. So for example, you ingest data twice. And if you have that twice, you have to pay for this twice because we have to process it twice. And you start to have discrepancies between the copies. And we build a lot of mechanisms to help you avoid this. So it's valid, we have customers doing that, but it's definitely not where we recommend customer to do. And then another thing that comes up with this idea is then it's very hard to manage. So workspaces becoming another thing you have to manage, another thing you have to allocate resources just to keep them healthy. And we don't, there is no need for that. Because we are used to the fact that if you are a different team, if you have different albac needs, you need different workspaces and that's not the case. And of course you can have this if you have like a completely different regulated organization. So if you have like, we are talking to organization that have kind of Chinese walls within the organization where like this part of the organization cannot have any overlap with this part, that's fine to have like different workspaces. Another very common need is you have like prod and dev environment or staging, pod staging and dev environment is because like for example, you don't need retention on dev environment. Dev environment can have very short retention while in the pod you want to have full two years of retention or one year of retention, there is no need to pay twice. So if you have these kinds of like walls within the organization, it makes sense to separate the stuff. But if there is no need, there is no regulatory need, there is no one forcing you, we highly recommend you not to go there to avoid the management of your head and the need to double ingest data, which is expensive and cumbersome and gets you to a whole slew of other problems because you now have two different copies of the same piece of data. Okay. Regarding scale, as I mentioned, you can have one workspace, you open it, you ingest almost nothing. We don't charge you for the workspace, you can ingest actually one byte or even not that. And then you can simply grow up, up and up and up and get to the tens of terabytes per day with the same workspace, there is no need to do anything else. We take care about the scale behind the scenes. And another like frequent questions that we get is about regions. You know, Azure have lots of regions and this is one of the biggest advantages that we have. And one of the, it's actually features that we provide today, is the ability to have logs in whatever region you want. And then like come to question, do I need like a workspace per region? And usually say, why, what's the need? You don't really need it. Like you might have assets here and you might have assets there and they can both send data to the same workspaces in different regions or in the same region, whatever work for you. But there is no need, you don't have to have workspace per region. Okay, so if there's no workspace, you don't need for workspace for region, how do you deal all because it's a past service and so you don't have to worry about redundancy across region? Yeah, I will talk about high availability through in this session. Okay. But yeah, redundancy is a little bit of a different topic here. When I'm talking about regions that highly, again, you should not think about regions, you should think about regulation. So there are regions that are unique. And for example, you have a regulation that force you to keep certain data within certain geography boundary. That's a good reason to have a separation. But if you have like three regions in Azure in the same geography, there is no reason to have like three different workspaces. Okay. Now, the one problem that usually people are resping with is Albuq. Yes, I was gonna bring that up. If we have this one big workspace, how do I make sure that everyone gets only the logs that they should see? And this is a very big investment we made a few years ago and it is available now. I think we are almost the only vendor that provide this in this strength and the ability to set Albuq policies per log or per record. And let me show you how this is, how it's working. Okay. So the basic model is that you have containers like workspaces and each workspace has its own data or data coming from all kinds of log, of resources. And what we are doing, we are actually, that's the familiar workspace model where you have this like padlock on the workspace, you have Albuq on the workspace and we support it, we would keep on supporting it because there are people that need it, like you have within the organization, you have security people and usually security people have no compromise. They want to see everything in everything. Yep. And you have people doing central admin of the entire thing, you want to see the entire thing. And you have people dealing with cost management and they want to have visibility to what people are keeping there. So for this like people, you want to give workspace access which is basically give them access to the entire boundary of the container. And that's great. But that's not all your user. I would even argue that it's not most of your users. What it is super important for us what it is super important for us to open the system not just for these unique people but we want to open it for the entire organization. So if I'm a developer and I'm sending log, I want to have them and I want to have only my logs. And the way we are doing that is that we are now tagging each and every log record in the system with the resource that created it. So the logs are now associated is strongly associated to a resource. It could be the VM that was sending that. It could be like the AKS cluster. It could be even if you have like activity logs coming from a subscription, it is also a resource. And each and every resource in the system because we are in Azure or we are talking about resources that are known to Azure either because they are in Azure or because they are Azure Arc we have our back for them. And now we can apply this our back. So the model that we have is that you can actually access the system within the context of a resource. And when you access the system within the context of the resource we don't check your workspace access. We check your resource access, whether you can see whether you can look at the resource itself. So our recommendation in large organizations is not to give workspace access to anyone other than the central teams. And for the different application teams they automatically get the access because they have access, read access to their resources. By the way, as an admin you can block it. So you can say, no, no, no, I want only workspace access but the default mode now let these people access. So in an environment that was looking at maybe doing hub and spoke in terms of having each team have complete access to their own data and the central team having access to everything this really in effect negates the need of that hub and spoke because you have one log analytics but each team can only see the logs and metrics for the resources that they have access to. Correct, correct. And it's not just like when I was saying logs that they have access to it they might spend several different work spaces. As we say, they might like you as an organization you might take the decision to have like two work spaces in two different regions. But if I'm running an AKS cluster I don't care about like this, all of this. So these users, the one that are coming from the resource context they don't even know about the work spaces. So the work spaces are the tool for the central admin because work spaces are there because of regulation all stuff that the DevOps team doesn't care about they don't need to care about they don't even need to be aware to. So people ask me how do I switch between them and the answer is really simple. Like if you go to AKS Azure Kubernetes Services you just click on logs and you get the logs of this AKS cluster. And this is how we like we have so many users that are using the system that way. They don't know anything about work spaces they don't know the work spaces even exist they just click there and get the logs. Because if I'm a DevOps team this is one less problem that I need to take care about. This is one less headache that I need to have like where so the logs how I manage it. Usually you have the central team that is doing that and that's fine. And in this model we also have hierarchy. So if you have like a DevOps team that has a subscription they usually manage the RBAC on this subscription. And now they automatically have access to the logs of this subscription. You don't need to have RBAC for logs and RBAC for resources. It's the same RBAC permissions that you manage. You set it once and it propagates across logs, resources and everything. So you're if you set it up right the first time then you don't have to worry about having access to something that you're not supposed to and on the log side of things. Yeah, I was talking to very large organization that has like they had like a log system that had had its own RBAC. And they found out that many people had access to that without having access to anything else. So like when you have two systems where you have two different policies, it's always a problem. Either it's just like more tedious to get access to and you have, or you just open up things that you shouldn't and people have access to stuff that they shouldn't have access to. And in the end of the day when you don't have good systems people simply don't use it. And you see people trying to do all the stuff around it. So we made it like super easy. You go to the OAKS cluster, you click on logs or you click on insights and you get the stuff that you need. As an admin, you don't need to care about anything else. As a DevOps team, you don't need to care about anything else. You just get the stuff that you need. And if you wanna have access to the logs so as to query the logs for let's say all of the VMs in the subscription, as long as you've got access to those VMs within the subscription, but do you get that from the VM and the log menu or do you have to go in the log analytics workspace and then go to the logs and query it that way? Yeah, so you just go to your resource group and within the resource group you also have logs and you click on the logs and then you get access to all automatically all logs of all resources within this resource group that are available. Another option you can go to Azure Monitor or Hub and select logs and then you can select which subscription you want to look at. So both option work, we see customers using both of them. But it only show you what you have access to is basically the bottom line. As long as you don't give this like users, you don't give them the access to the workspace. So if you give them access to the workspace they can basically have access to everything. Gotcha. All these details and I know how complex this topic is because we give all these options and all these capabilities. We have like you can see in this slide we have aka MS slash logs access where with all the details, nitty gritty details and we are always there for you to help you and make you successful in the process. So we know our back is usually something you have to define once, but we are aligned, we are totally aligned with the Azure Alback. So you have to do it, start that you do for Azure, for your resources, also applies here. Well, the same it's whether or not you're in an hybrid environment or a cloud native, it doesn't matter. Basically nobody should have domain admin level access to everything. Exactly. Like you do need those like break glass type of accounts but they should be used only by the appropriate people but also at the appropriate time and everything else should be assigned based on what's needed. Yes. Okay. All right. Now, one of the mechanism, if you want to understand, this is an example of how logs look like. So you see in this example, I took a screenshot of how it looks like within a VM. So you go to a VM, you click on the logs and then you see here at the top that you have the name of the VM as the scope. And by the way, you can select now a new VM and do whatever you want, but you can change the scope but you start with the scope of this specific resource with the RBAC of this specific resource. Okay. And that resource, if it's on-prem or an Arc server doesn't matter. As long as it's listed in your, which is why you mentioned earlier, if I'm interpreted your comments properly, which is why we kind of recommend if you're going to tie a hybrid server, a server on-prem or in another cloud to your log analytics to do it through Azure Arc so that that VM is basically registered and visible through the portal and everywhere. Yeah, the VM is not just registered, it's also have its own RBAC. So when you enroll a VM into Arc, you tell us which resource group it's part of. Okay. So you don't have this like often machines that are not collected to anything. So when you tie it into a resource group, you tie it into subscription that has RBAC. So you don't have like this like resources that have their own different paradigm. You have one paradigm that goes throughout everything and it supports hierarchy. So if you set it on the subscription level, you get it down, percolating down the resource group and to the actual resource. And the beauty here, it's not just for logs, it's basically everything. All the goodies that Azure is doing, stuff like policy and ARM and everything is walking through across the board in whatever cloud you are in whatever data center you are, it's just there, it's just walking. And we hear like an amazing stories from customers that they in the past, they didn't even know what they have or they found all these like quirks and stuff like that. They found that people have access to stuff that they shouldn't, but now we have like a very well-organized paradigm to keep this in control. And I realize that onboarding all of those VMs or workloads through something like Azure Arc makes a lot of sense versus taking an agent and manually deploying it or automatically deploying it to a bunch of server in a data center because then they're not specifically tied to any group of resources. They're just sending data in terms of logs and metrics to the workspace, but they're not actually being managed in terms of resource access for what it's uploading. Exactly. And we talked about like agent deploys and stuff like that, which is a big headache in the past. But with, because the agent is just an Azure extension, it's also very easy to deploy it. So if you have Azure Arc, if you want to remove something, if you want to take it back and forth, it's really easy. You can use the same policy, the same automation, the same mechanism that you have to manage in the resource group level. You don't need to manage it machine by machine. You don't need to write like this custom sprint and whatever it is. It is because we're using the same mechanism, the same deployment mechanism that you have across the board. Okay, and this Azure Arc basically decides or helps you kind of manage which agent, because currently we have multiple agents. I know you mentioned that we're going towards a more streamlined model of agents, but right now Azure Arc will basically decide or manage how those agents get deployed to each of those servers. Correct, correct. And that's very, very different than what you had before. Basically everything you deploy through Azure Arc is getting through Azure extensions. Whether or not the machine is in Azure or not. Yes, yes, it's very easy to manage. It's very easy to enumerate and to remove, to add and stuff like that. Yeah, and visually as well is a Azure Arc machine in the portal shows up as a different than a Azure machine, which if you're looking just by enumerating the resources in a browser, you can tell which machines are which. Yeah, we actually talk to customers that don't want to see the difference. They have like, they want to see one set of VMs, that's fine. If some customers do manage it differently, so they want to see it separated, but we support both modes. You can decide what view you want to make. Okay, that's perfect. Now, that's all like the general architecture, but when we talk to very large enterprises, we hear questions and we hear topics that are very, and features that are very different. So we have like a set of stuff that we have developed that is used again by the very high end companies, but it's not limited to the iron companies. And I would like to share it with you because we made lots of progress over the last year in this area. And I would start with the first topic, which is dedicated clusters. Okay. So when we talked about the single workspaces that grow all the way up from nothing to very large one, we do a lot of stuff behind the scenes. But for several reasons, we wanted customers to have some control over this process. And this is the dedicated cluster. So the dedicated cluster is when we actually come to the customers and tell them, hey, we give you some control. We commit to you on some things. You need to commit to us, so you need to have reserved capacity, but now we give you a few things. First of all, we give you lots of features that are very high end, usually features that requires differences and settings within the hardware level. And these features are mainly customer managed keys, lockbox, and we would talk, we would see also double encryption and additional encryption mechanism, and we will talk about a few more that are coming. Okay. That's one thing. The second thing is because you commit to us and we commit to you that you have your own cluster, we can apply less restrictions. We can let you play a little bit more with the platform and we make sure that you don't disturb other than other are disturbing you. Okay. We make sure that thing. And the third thing which kind of happened, we do have customers using many workspaces that you might have like your production and dev, you might have different units, or we actually have customers that decided not to consolidate the clusters, but then not to consolidate the workspace, but then they do have the need to put the workspaces in one place and to have capacity reservation in a single place. So let's say you have like Dev and Pod, and Pod is where you have most of your data. We don't want you to pay extra money because the dev is too small to get to get the capacity reservation discount. We actually want you to have as much discount that you get on the other one. So by allowing you control the clusters, we allow you to move which workspaces you want to the same cluster and to reserve the capacity for them together. And that has a big cost advantage for several customers. And another by side effect is the fact that like cross workspace queries work now very, very quick because they are located on actually the very same hardware. Okay, but if you're going to a dedicated cluster and you want to move two workspaces that are separated because of mandatory requirements for that geography. For example, if you're in healthcare in Canada, your data must reside within Canada. So... It's a good question. Clusters are physical entities. So they are limited to a single region and we don't allow you to move data between the regions. So you can link to them only workspaces within the same region. Okay, so that was my question is if you're going, if you want to have a dedicated cluster because you're large enough to warrant that type of capacity reservation, it doesn't necessarily mean that because you've now got a cluster, let's say in the US or in Israel or wherever else in the world that you have a footprint that you can move workspaces that are. So if you want that cluster in both the region is where you have data. You need two clusters. You need two clusters which means you have to double your capacity reservations. You have to pay for it twice. Yeah, if, again, dedicated clusters are optional features. You don't have to use them. True. We take care of your workspace whatever you want to do with that. So we have like customers that decided to have dedicated cluster only on their main region and keep the others dispersed. That's fine. We saw customers that are actually consolidates their stuff to the same thing. And then they actually move like the side that they would not send data to the other workspaces that would send to the central ones. And we see customers that actually have more than one cluster. So if you are an organization that is usually you have it with Europe and US and you have like two large operation and we have a customer and like every very, very large organization have it. It justify to have like one cluster here and one cluster there. Okay. But the point is I think is that all of those features are optional but they're available should you need them. Yes, correct, correct. And it's important to say if you decide not to use dedicated cluster, that's fine. We support, we have very, very, very large customers that haven't decided to move there. We have today workspaces that are almost 100 terabyte per day and they are not on dedicated cluster. Like I think like our top three customers our top three workspaces are not on dedicated cluster because the customer didn't need it. And we have other customers which are smaller that decided to go there. So it's a decision. It's another option that we give for customers and it is usually more connected to the question whether you are regulated and you need to have especially stuff like customer managed keys that we will talk about in a second. Okay. So, and again, we don't charge you for dedicated cluster. There is no additional cost here. The only thing we require you to purchase more than one terabyte per day capacity reservation the need is to have this like two way commit. We commit to you and you need to commit to us because there is a physical reservation of hardware when you allocate this kind of cluster. Something happens on the backend. Okay. We, another area that we talked a lot about like very large customers is to give like the top notch about security and control. And we talked, I'm always surprised like organization that I would never think that they would go to the cloud because they have like the top most security needs. Whether you can think about like the most sensitive financial organizations or government organizations that have like the most secure data like you can think about, they are moving to the cloud. And it's our obligation to provide them the tools all the tools that are needed to make sure that this is working and they get like full control. So when you talk to very large enterprises they want to have full control. So as I said, like we are doing a lot of things behind the scenes, but you have customers coming to us and tell us that we do want to have this like control. We do want to have this point where we decide on stuff. And that's fine. That's great. We understand. We see it like it's, we are honored to have all these workloads working on us. And the way we do that is by providing few features that are specifically tailored for this situation. The first one is customer managed keys. With customer managed keys, we are using your keys to encrypt the data without us as a service or as a Microsoft personal having any access to this thing. It is based on Azure Key Vault. So you have to put your keys there, but it's your keys. If you take them away, the data is now unaccessible. There are many different regulations. There are many different situations where you need, it's kind of like an extreme scenario, but we are talking about the very high-end, and the need is there. And organizations need to have the ability to revoke all their data. And we give this with customer managed keys. Okay. Is this something that we are looking, and you don't have to answer if that's not something that we're allowed to say, but is this a feature that we are looking at opening up to everybody that has a workspace, like a regular workspace? So we, because customer managed keys are integrated in the low level of the hardware, we need dedicated clusters. So it's available to anyone on dedicated cluster. And again, there is no additional cost to that. It's free of charge. Okay. No, I understand. I just wanted to know whether or not, this is something we could expect for a normal or a standard non-dedicated workspace, but you've just answered the question. So thank you. We are looking on other options. We are looking on other options, but nothing in the short-term around that. So dedicated cluster gives us the assurance that the boundary is well-secure. The next thing is lockbox is that you have, and again, we, because we run the service, we need in some situation access to the actual data or to the, to maintain stuff. And with lockbox, we actually give you the ability to control whether you allow us to do that or not. So it's not just us asking, we need it. We are required to go there for you. So that's lockbox. It applies throughout Azure. And we support that as well on dedicated cluster. Okay. And the last thing, and again, we can drill down on this forever. So I would just like put it here and we can define it, talk about it more is like the double encryption. We have the best encryption experts there and they came up with this paradigm where we actually encrypt the data twice in two different keys. And that's like give you like whatever your regulator is asking you, trust me, it's not, this one would actually keep you in compliance. So there are some organization and again, like the very high end that have this like super complex encryption asks, and we support like the top notch compliance requirements. So let's go to something more like common is what we call, it's about high availability. You asked me, Pierre, you asked me about that earlier. Yeah. So high availability is something everyone needs. And we like, if you know how Azure regions are working, we are providing high availability out of the box for everyone. But for right now, we are going to provide what is called availability zones. The idea here is that if you go to Azure region, they are physically segregated within themselves to at least three zones. We do have several regions now that have more than three but we have at least three in everywhere we deployed. And these three have like separate connectivity to the internet. They have separate electricity field and they have separate cooling mechanism. Everything is separated. And then we deploy our systems and we deploy your data in a way that is redundant, meaning that if one of these three regions goes down, the service keeps working. And the beauty of that is that you don't need to take care about anything. So behind the scenes, we are using something like ZRS, zone redundant storage. And we're using VMSS and all kinds of other Azure low level mechanism that gives us this kind of of redundancy. And we don't like just trust it. What we are doing, we have like once a quarter, we have a procedure. We have a special region somewhere in the world where we have our system deployed and we actually deploy a mock organization there. And once a quarter, we go there and there is someone physically going and unplug a wire somewhere to make sure that everything keep on working. And we are doing this for several quarters now and we always find new stuff. This is why it's coming soon. It's not already there. We are almost there. We are not 100% there. And once we have it, we would announce it as something available. It's coming really soon. Now the beauty here is that you as a customer doesn't have to know about anything. You don't have to take care about anything. It's just walking like everything we are doing as a service. We are doing, yeah. If the entire region was to go down for example, like including all of the zones within that region. We don't have for log analytics or for Azure monitor logs. Currently we don't have a capacity to replicate that to another region. Yeah, we are working on mechanism to support also this like doomsday scenario. Okay. It would have cost. It would have cost because you need to reallocate. We want to make sure that if something bad happens, it's not enough to keep stuff on another region. You want also to keep the VMs there that would be able to give you access. Okay. So this thing is coming. It's more further down the road. If you have an organization that doomsday is like super important for them, let us know. But the mainstream, the one like most of the organizations availability within the region is actually enough. And whatever I talked to a lot of customers, including customers from telcos and stuff like that, they don't have these kinds of like availability within a data center. It's pretty unique and it's super strong the ability to do that. And again, we are testing it over and over and over again and we would provide it out of the box. Okay. The first step that we would provide, it's for newly created dedicated clusters. People creating dedicated cluster can all, even now can see the flags that are, whether it's turned on for them or not. And we are going to widen, make it available in more places, in more regions and for more scenarios. So thank you very much Pierre for this. Okay. I just had one more question on the availability. Yes. Because I've read not too long ago about Azure monitor logs data export to standard to storage. Yes. So are we, is this kind of like a backup or is it just like an offline or what does that mean in terms of availability, in terms of scale or is it just like exactly why, why is that a feature that people should look at? Yeah. So that's a new capability that we have added few months ago. And we see customer using it for all kinds of stuff. So again, if you want to use it to export your things to put them, to make them available somewhere else or to archive them, another very common ask is, I need to archive stuff because this is my regulator is asking me to do. Okay. I want to make it available somewhere in the board. So that's one use. We see other customers that needs like mass export for doing like kind of machine learning stuff on the logs. And we have like all kinds of like reasons and scenarios that people are using it. So you can export, you can export now all logs, almost all logs coming into the platform. It would be, it would include all logs pretty soon. And these logs would be stored in whatever storage account that you want. It could be in the same region. It could be in different region as well. And that allow you also, if you have the requirement to have a copy of your data somewhere else it's much cheaper option than what we talked before about like having like always on one plus one option. So if you just need to retain a copy somewhere that that's a very good decision to have export. What about cost? If we're, if you're looking at a workspace and that's in your retention, you've set it to a year versus the normal 31 days the cost difference for that and exporting the data to storage just so that you can keep that full year as per requirements. Is there a significant cost changes here? Yeah, there is, there is a different costs like export costs you to export, retention when you pay per month. And the question is why you need to retain it. So whether you query on that. So the stuff that you retain in Azure Monitor Logs is instantly queryable. Yeah. Stuff you export, you can actually, we have power diamond, we have documented how you can run queries on top of that. But it's not really trivial. The queries would be much, much, much slower. Okay. And so it's mostly for compliance that we would recommend to do that. If you're just looking, if your compliance is to retain all logs for a year or two years, then you would go the export route. Yeah. So if your compliance requires you to store them somewhere and you don't really need them like on day-to-day basis or maybe like, I had a customer that told me that he just got like a cold supina for a record. So for this like exporting and putting this on storage is great. If you actually want to run investigations, so we had now a pretty loud high-profile breach and we had lots of customers going back six months ago and even a year to find out evidence of these hackers getting into their environment. So they had like very, very demanding need here. So they ran a query. We also have like customers that keeps data for a year or even two because they want to have this comparison. I mentioned the fact that we are big users of our platform and one of the things that I'm doing like constantly is like looking at a pattern. So December, how does December look compared to the other December? So for this export would not be good. You need to retain the data. So if you want to do historical modeling, you want to retain the data if you're just need to keep it for potential access later whether it's because you decide to or because of requirements, then export is the way to go. Yeah, and we also see customers that have other analytics platforms where they need to share the data with. I mentioned machine learning, that's a very common thing. Like they have a different platform and you don't, like you can do lots of stuff that you do on machine learning on our platform. It's very well documented. But if you have your machine learning guys working on, you have something already built, already ready and you just want to have this data available, that's another reason to export. Okay, well, that makes sense because I was a little confused in terms of what the use case scenarios for the retention versus the exports. So I'm glad you cleared that up for me. Yeah, thank you. Okay, so we are done with the formal session. Thank you very much, Mir, for spending this time with us. For you, the viewers, if you have any questions at the URL that you see below is a link to our Discord text channel for this particular session. So you have questions, put them there. We'll make sure to get them answered. And in also this link below is where you're gonna find all of the resources that Mir has mentioned throughout the session and some more documentation that we will put together for you for you to actually go and try it out and find out more about Azure Monitor Logs. Thank you very much, Pierre. All right, thank you, Mir. Thank you everyone and stay tuned for more content on IT ops talks by all things hybrid. Ciao.