 Good afternoon again. My name is Vinod and I'm a product manager with Microsoft. I Work in a team called visual studio team services, which is basically our DevOps offering And today I'm here to talk about You know our journey towards DevOps. This is not just about VSTS But this is also about the 82,000 Microsoft employees actually moving to DevOps, right? So The journey includes three aspects. It talks about the culture the process and then tools And just to set the expectation right my presentation is not so much focused on tool right now because I Saw there are plenty of demos happening here The intent of this presentation is to talk about, you know, what we did right or wrong What are some of the learnings that you can take in your journey? What are some of the mistakes that you can avoid? right I Have worked here in Bangalore also previously and I love the opportunity to come, you know to this city. Just love this city Except for the fact that my in-laws also stay here. So And That's definitely not the reason I left the city, right And I hope this is not getting recorded right This that is certainly also not the topic of discussion. I know a lot of you I can see a lot of you smiling, right? The only thing I can say about that is, you know, there are some challenges in life. We cannot solve We'll look at DevOps. Okay. All right. So with that, let me give you a quick introduction of The team that I belong to right so Microsoft has a big Organization which is called developer division right which has about thirty two hundred employees and This is essentially all your visual studio You know vs for code vs Mac Visual studio mobile center and then obviously, you know visual studio team services as well So it's a large organization and we are a part of that organization So we're about 800 a team of 800 people Spread across the globe. Okay. So We have a team in Redmond. We have a team in North Carolina and then we have a team You know in Hyderabad as well, right? We have about 40 feature crews So the size, you know when I look at 800 it reminds me that my first company didn't have you know The total strength was in 800. It's a big team Spread across the globe 40 feature crews working on you know various features This is the only slide I have about the product Okay, and I'm not going to sell VSTS here because my intent is to talk about how we embraced DevOps, right? So visual studio team services has two offerings One is a TFS which is the on-prem solution and the other one is VSTS Which is you know the cloud solution, right? It provides you source control It provides you build release continuous integration continuous deployment package management Dashboard analytics, so it's kind of an end-to-end solution. You know if you're looking at onboarding to a DevOps offering So that's the only slide I have on the tool then I have this slide which actually talks about Our internal journey like as I said there are about 82,000 Microsoft employees Actually using VSTS as the DevOps tool This includes, you know, the largest, you know, the largest code base that we have on earth, which is windows basically includes windows office You know any team your name, I mean there are 50,000 engineers actually working on you know on a dedicated basis So when you what you look at when you look at this dedicated user that means people using it continuously for 11 days in a month Right when you look at occasional these are people who come in for two days at least right and then there are One-day users as well who may be experimenting with it, right? So our journey so far has been good You know the way we have moved to almost 50,000 dedicated employees and it's a challenge It's easier to sometimes move people outside the organization than inside and in Microsoft You have plenty of tools which have been you know kind of homegrown, right? So you really have to have a reason To be able to move these teams around So let's now get into a journey and this journey is about how we started how this team started so Once upon a time You know we used to have this as the cycle which is two-year release cycle so something That I that I'm working right now is actually going to get released in 2020 Right, it's going to be two years down the line. I know exactly what that feature is going to be, right? So to be able to do that we had a huge planning cycle like months of planning, right? And then we had few milestones, right where we would have some part of the product to be enabled for our customers The planning itself was a huge cycle. I'm a product manager. So this is my area I would be writing specs, right? So I'd be spending like months writing specs and those specs would be nothing but you know 500 page document that the team is expected to go through and develop a product, right? But as I said the assumption was whatever I'm doing right now is going to get released two years down the line So I have to know What the customer is expecting two years from now. It sounds funny right now, you know when we when we think about it in today's time, right? And we also knew it was right, right? We knew because we have spent so much time. We know this has to be right What happens to the development? We had an equally long Development cycle where people would write code and this code would be written months before it's actually going to be released, right? It's like The developer is writing it now and six months down the line We will actually have a release because there is a huge test and stabilize phase Think about the plight of this developer Where a tester comes at this stage and asks the developer. Hey, here is a bug in the code Can you tell me what is the problem with that code? The developer has almost forgotten the code that you know he has written It's almost like me asking you like, you know, what's your marriage anniversary? And then you're thinking, you know, I don't remember right so It's it was that kind of a challenge right and think about the plight of a developer who is new So the developer who actually coded has left the organization and there's a new developer poor guy He doesn't even know what the code is but he has to look at you know, what that bug is understand that code So it was a huge huge huge debt for us Then came the beautiful day right when we are going to release the the product So we had we used to have release candidates and then we would have released to market, right, which is the RTM and What happens? We have done the planning. We have done our first deliverable and now our beta is ready So we go out right we go out and we talk to the customers We're just waiting like I'm standing here and you're the customers and you know You're just looking at what this product is going to offer and I'm excited to know what's your feedback and then you come back and tell me That doesn't seem right, you know, what you've built doesn't seem right And we're like no, but we spoke to you. We took the requirements This can't be wrong, right? So what do we tell them we have to have an answer, right? So we tell them Fine, you know, it's a great feedback We'll implement in the next two years Because that's how a really cycle is right. So we tell our customers wait We are booked solidly come back after two years Right now think about if you were that customer We've had a lot of challenges We've not been able to give features that people have asked for we've not been able to prioritize So we realize that you know, there is a problem with this, right? Why did it work at the first place because at that point in time? That is how products were developed like windows for an example was a three years really cycle, right? And it was fine. It's almost like if I tell you 15th century if I were to tell you in we're all in 15th century and I tell you a Hey Cars are going to be the most common way to commute. You're probably going to laugh at me at that point in time, right? At the same time today if I tell you that horse is going to be the most common way to commute You're still going to laugh at me, right? So this fitted at that point in time and so it worked, right? But today it doesn't right This is anyways a statement. So I'll skip through this is basically to say that hey You know with with the amount of agility that we are expecting in the business You know, you can't work at the pace that you've been working with, right? So what's the solution? I'm assuming Everybody knows about this light, which is a standard sprint, you know the scrums You know diagram and I'm sure through the you know three days or four days you've been here You would have seen this right so this is the solution to the problem and that's also my last slide No This is just the beginning we thought we will adopt this we heard like all of you we heard, you know This is what we should do, right? So We said, okay, let's go to sprint model. So what do you do? You simply break the schedule Right and this has like even even though it will not be perfect But the first step is to simply break the schedule. We did so so we we decided we are going to break everything into Sprints, right and these prints are going to be three week cycle. Why three? Any guess You know you are smarter than us because we didn't start with three weeks Actually, we started with two weeks first and then four weeks. Okay, and in both the scenarios We realize that it's not working out, right? So you're smarter. You figured it out in the first place, right? A good one So we started off with two weeks and then we realize it's not working We were not able to deliver enough increment, right? So we moved to four weeks but the four week cycle was a problem because if there's something that slipped beyond four week That meant another four weeks, right? So we said, okay, we're gonna go get to a three week cycle which seems to be reasonable But we were scared of quality, right? We think about this like we had been doing this two-year releases and one fine morning You know, we wake up and we say, hey, we're gonna do three weeks sprint Like it's just gonna be a very smooth transition, right? Two years till yesterday. Let's switch to three weeks now, right? So that was the transition that we were going through. So definitely not smooth What we figured out is that we do we are not we are not confident we can deliver quality So we added a stabilization sprint like everybody does and these are by the way small waterfalls Within the sprint, which is where we also started, right? Nothing new everybody does that Now what happens? We have two teams and this is actually by the way a practical Like actual on-ground example that I'm saying obviously I can't name the people but there are two teams One of the team decides that they're going to build Features that just going to focus on features. So they'll add feature on feature on feature They're excited developers want to build feature, right? So they went overboard and they started building the features which is team B Team A decided that while they're going to build feature But they're also going to make sure that there is not in your more technical debt that is accruing because by the end of sprint five While there is a stabilization phase. It is still a sprint, right? You do not want to accrue the technical debt, right? So what happens? Team B is red Okay, because there is so much technical debt Team A is green, right? So what do we do to team A? We reward them No We actually punished them we told them go and fix their debt as well What do we do? We have a you know limited timeline So we punish them, right and you can imagine the level of motivation that team A would have, right? After knowing that they have to fix team B's debt By following all the process as it was expected, right? Definitely, I don't want to be part of team A for sure So today we are here Today we are at three weeks sprint We don't have that, you know Long two-year cycle and I'm going to talk about some aspect as to how we reached here, right? I'm sure this audience would have you know people from different background like you know program managers developers you know testers and I hope that there is something for everyone out here to take okay as I said at the start the large part of DevOps is about culture and people then the tools and that's why I'm not covering tools You know to the detail This is a pretty known statement So what did we do to change the culture? We did not give people Dan Pink's book By the way, I don't know how many of you have how many of you've read this book, but it basically talks about The importance of autonomy the importance of mastery and having a purpose behind it, right? So we didn't give this book But our thought process was kind of aligned to this to say that we need to give autonomy to the team because Creativity comes with autonomy. We need to give people a sense of mastery For us it was easy because I'm building a product which is VSTS, right? It's a product for DevOps and if I'm not going to master that product I cannot stand here and talk about it, right? So for us the purpose came naturally because we were building that product And obviously, you know mastery I'll talk about it, you know, how our teams do that But let's look at what does that mean? What does autonomy mean because autonomy can also be risky, right? So this is one example that all of you may know about which is you know in Kata and You know Wikipedia the way in Kata came in was that, you know huge business plan Well-funded hiring the right people, you know that they needed and then Wikipedia completely You know different model People who are interested to contribute in their free time not much of funding and we all know what happened, right? And by the way that was actually funded by Microsoft In Kata, so we all know what happened, right? So clearly it's not the money that's working, right? You have to have something beyond right so So this is what our guiding principles are We have an organization which is of the size of 800 people What that organization is going to look like? What are the rows that are going to be there in that organization? Which teams are going to be there like testing or CI CD or work items? What cadence we are going to release product, right? Is it three weeks? Is it four weeks? Is it five? Is all Alignment today today Any of those 800 people if you wake up any or any one of them middle of the night and ask them Which is the sprint that is going on right now? They'll be able to they'll be able to say s 132 dot 2 Anyone and that's it. That's a challenge, you know that I can take openly That's the taxonomy part of it How do we make sure that you know the entire organization is aligned on certain things, right? But Without risking the autonomy so where does the autonomy comes the autonomy comes in? How do you plan and how do you practice that which means as a product manager? I have full freedom To decide how that feature is going to be built What are we going to do in that feature and nobody can take that away from me, right? But am I going to run a four-week sprint? Am I going to run a two-week sprint? No That's aligned at an organization. Am I going to run a sprint starting next week to you know The next three weeks and the other team starting this week to the next three weeks. No You don't have a choice there, right? So you have to have alignment and you have to have autonomy This is from a tixonomy standpoint. So we do 18 months of Planning strategy planning and let me just get to the next slide actually So we define epics which are like at you know at the range of 18 months Something that we want to target for 18 months next 18 months, right for an example If I were to decide, you know, if we were to decide that we want to do something around functional testing, right? That's a large strategy, right? So we would probably take a call that this is what we want to invest in right in the next 18 months From there we decide what feature are going to be, you know part of that strategy, which is roughly around six months From there we come to feature chat feature chat is nothing But we do kind of a three sprint planning. We have a sprint for three three weeks We do a three sprint planning, which is what we call a feature chat, right? And then we have the sprint which is three weeks itself now What is interesting is we've also figured out What does it mean in terms of how accurate we will be on each of those? So if I were giving a strategy for 18 months, I know I'm somewhere at the ballpark of 60% It it would be only 60% that I'll be able to deliver if I am doing a feature which is six months Then I know it's probably around 80% Right, but when I come to feature chat when I come to a sprint, I have much more clarity I know that I can deliver 90% of that right and I'm not just saying it Let me just I have a few slides open a few pages open that I wanted to show you as well Yeah This is Feature chat for all the 40 crew crew members that we have right 40 crews that we have and this is only for the sprint That is going on right now, right? So everybody has to come with a feature come up with a feature chat and then we have a set of meetings where Every, you know GPM which is basically my boss, right? Who's going to talk about his feature chat about his three feature crews and then everyone else is going to talk about that And we do it over like three meetings which are each for about two to three hours It is important because teams are distributed and there are dependencies across team, right? So that's the you know the feature chat part Let me just quickly Okay So this is the feature chat part that I said This is our sprint Three weeks print. So we have three weeks print which starts at week one completes on week three week fourth is a deployment and our deployment Happens in rings. So we have a progressive deployment And I should probably show you you know what that really means so These are the rings that we have. Okay, so these are the environments you can assume, right? And this is ring one. This is ring, you know, this is ring zero one two three four We have seven rings, right? The idea here is that there are a lot of people who are sitting here in ring one This is kind of a canary kind of a deployment where you know, I have my MVPs Which are you know a community of people that we work closely with, right? This is us? Us itself like you know one of this ring holds our code the VSTS code itself like can't be like Being part of this team, you know, I feel so excited thinking that What I am developing is also actually getting deployed by VSTS Which means VSTS is being built using VSTS, right? So these are the rings where you know, you would basically pilot test And and you know get some of the early early feedbacks out, right? And then you do a progressive deployment. So when I said the fourth week is deployment in the five days we would go from here till ring seven right and we intentionally delay You know in that week because you get service health related information If there are issues you would get issues as well for an example We had an issue with with our service, right? And since I'm talking about culture Let me let me show you something that Am I only intend to show you this is because as I said a large part of it is actually culture, right? So we had an issue. This is this is from my CVP And he he is very active on the blog, right? And so we had an outage Just a few days back, right? And he blocked about it He blocked about and this is all available in public to view, right? So he blocked about How there was you know an incident that happened and how we did a post-mortem of that and how that was available publicly for everybody to view, right? And the same post-mortem, which he's talking about is actually here So we talk about what was the customer impact and it's all in public, right? We talk about what caused the initial failure Why didn't the system auto come automatically recover, right? And then what happened at the you know the application tier what happened at the back end All of this data is available for all of you to consume for our customers to consume So that's the kind of transparency that we're talking about. It's a very important part of DevOps, right? It's not just about the three weeks, right? This is where the culture part of it comes in, right? So let me just go back again Yeah, so the three-week sprint fourth week we deploy and then while that is happening We've already started on the next print right We spent sent sprint emails Every every feature crew has to send it by the end of the week third everybody has to send it and All of these print emails are actually read by rcbt, which is great Basically, but the idea here is to say that there is one this is alignment This is alignment to say that Everybody in the team has an opportunity to know what's happening across a you know different feature group, right? So you have the opportunity to know that But where is the autonomy you decide how you want to send that sprint? What you want to highlight the sprint emails are usually about customer value and stuff But you have freedom how you want to do it, right? But you have to do it, right? The other thing that I want to highlight is anything that we do, right? This is A feature that I worked on. Okay, and this is actually a work item in the sts And this is my release notes of when that feature got completed, right? I have to document that right, but once it is there It is available for anyone else to see. So let me just go and see where Yeah What I've put in my release notes is actually what the customer is going to see So you saw this Which is the feature that I had added, you know the one that I had worked on basically it's on vsts Which is also available publicly, right? So after every sprint you can come here and see what we delivered Anybody in the world, right? Um So that's that's another cadence that you know, we follow as a team so um In terms of you know, how that is getting picked up and how that is getting posted is something that is internal to us But Here's what I would say. It's a very lightweight process that we have built and it's not even part of the product Okay Yeah, that's what I said like there's a lightweight process that we have built which is basically going to collect that And then publish to our, you know blog site. It's not part of vsts, but it's part of our process sure sure All right, um, let me just make sure So this is what I spoke about, right? We also made organizational changes because you Like it's not just about tooling, right? So what are the changes we made? We used to be a program management Uh development and a testing org, okay, and this might be a little sensitive But this is our journey, right? It doesn't have to resonate with you necessarily, right? So we used to be a program management Development test and each of them will be like a vertical reporting into you know the director Uh in each of these orgs, right? But what we realized was The challenge we had with that model was the accountability wasn't there I as a developer would write code throw off the wall and say, you know, it's testers responsibility Right test it is testers responsibility to test and provide the feedback And those three months I'm having fun time, right unless there is a bug And it's a tester who is basically testing it and once the tester is done He has thrown it back to the operations to say hey you go deploy. I have done my job, right? We wanted to break that we wanted to bring accountability So we got we we got rid of a test as a as a role or I would say, you know As a specialized skill instead we merged dev and test, right? And and that kind of became the you know, we used to have sdats sd is previously We don't have anymore, right as part of that transition We had to let go of people as well Because you know a lot of work was About manual testing and we had to let go of people as well And we also had opportunity for people to kind of free skill But we were clear we wanted to have you know automated testing, right A large part of it was You know manual and we wanted to drive accountability more important, right? It was about accountability what you write you own it, right? And when we did that basically so this is just you know, how a feature crew looks like like my boss Has probably few pms and then I am one of them and then you know My you know boss's engineering peer has You know few teams under him leads and then you know, there is a feature crew So I probably work with you know a feature crew on a particular product, right? This is how the team looks like this is This you know the standard we think 10 to 12 is what works for us Thanks to those guys, you know, they've tried to control their laugh While clicking that picture, but you know, that's a team that sits in Redmond Um So what happened when we made that change when we when we kind of you know merge the devint test We had a lot of tests which were manual functional tests and stuff, right? And we wanted to get that automated, right? So this is how a test look test scenario look like this is how the test landscape look like this is This is where we were previously This is where we are today now that l1 l2 l3 l4. I'll quickly tell you l1 is basically a unit test No dependency with anything no network. No storage. Nothing. It's just an independent unit test, right? We coined term for l1 as well because we had a huge dependency with databases. So we said We will have l1 tests which are basically just using a database, but we are going to run them very frequently Right. So that is what is l1 Then we have l2 which is more of an api kind of you know integration testing kind of scenario And then finally l3 which is more of you know functional functional testing ui and things like that Our belief is that we want to have as less l3s and l2s Compared to the l1s and l0 we want to be high on the you know l0s and l1, right? today we run about 70 000 tests every pr build so As a developer when I make a change when I send a pull request, they're going to be 70 000 tests running with that single pull request And it's includes both l0 and l1 tests, right? And there are you can imagine there are 150 200 runs happening In a day actually more than that Actually more than that, right? So that was a huge transition and you know Part of it is also because once we did that whole transition We wanted to basically have all of this automated, right? less debt in the system Yeah l1 is where back end comes in so we are a very heavy sequel, uh, you know dependent Service so that's where l1 comes in. So this is how the quality was before right where you would have these Stabilize and the debt has gone up. This is uh This is how it looks today, which is After every sprint You know, you got to make sure your debt is controlled. There is by the way a report I'll show you at the end of it If you have more than x amount of debt The the team cannot take a feature period. You cannot take a feature Right, so you got them put and that's again, um That's alignment autonomy What is that? Thank you Yeah, there's a lot of technical data It's a lot of productivity as well because what was happening is the code merging is also going to happen here If you look at the previous model, most of it is happening here Right, which means 40 different crews merging code here Is a You know, it's a pain, right now that is happening here The third week everybody merge Right and just to add to that. We have a very lightweight branching model So first of all, we don't use tf. We use get okay Internally and a lot of team use get and we accept it right get works for us. We use get right But What we do is we have a very lightweight branching model, which means there's only one master branch Everybody is going to take from master I'm not going to start my own branch And then work on it and then come back one fine day to say, hey, let's go and merge it, right Everybody has to work on master and everybody has to merge that small increment Right and once the master is ready to be taken for a release And we'll take a release right, so Yeah, when we do the release, it's a release branch action from where we are taking it, right? Yeah Yeah Right Sorry No, no because the deployment is You know, there are dris. Um, there are points of contacts for deployments as well So it's not that the entire team is basically involved in the deployment We have a small ops Operations team as well that works very closely Which is partly looking at the service health and everything else But you know, they do help around that as well. So it's During that one week. It's not that my feature crew is actually doing the deployment It's a dev response since you asked that Yeah, sure Um This is where we are getting queries from customers. This is today. Obviously. I've been here. They're 37 emails team is actually looking at it Uh And responding this is a developer He's actually responding. So the team is responding to the queries and then there is a di as well Okay, so I'm going to skip through this one because I already spoke about the you know, how people Uh, basically create branch and we have a light, you know branching model and then we basically merge Uh, the idea is also if you look at it, you know, the code is fresh in developers mind So if an issue comes I know what happened, you know, what I did two weeks back versus saying a six six months back I don't remember, right? I mean, we have hard time finding, you know, stuff that we place at home and stuff. So, you know Okay, so this is This is something that we do which is what we call feature Flag, right and this has helped us immensely because the next question that should come is What if I'm not able to complete a feature in three weeks, which is a common scenario? I I speak to ton of startups and One of the common problems that they face is that they say the timelines are driven by customer So if the customer says something has to be given in two months, we have a two month cycle The customer says three weeks. We have a three week cycle, right? The problem there is the size of the feature So we came up with something we call feature Sorry feature flag, right? What does that mean? Now assume this is my code, right? Which is today and it goes through a Like when I when I go to a particular page and access a feature it goes through this path Okay Now I've decided that I want to enhance this feature So what I do I actually start working on it separately in the same code base under a feature flag Okay, so feature flag is nothing. I mean if I were to say simply it'll be an if condition where my code would basically branch out Right, obviously there are optimization behind in terms of how we handle it at application layer Or the database layer, but it's simply saying that I'm going to branch it up, right? Now. This is not visible to anybody This is still not complete, right? Then what I my feature flag is still off for most of the customers they can't see it They can't see this workflow. I am continuing to build on this and I might just test it within the team as well right I am now ready because I've completed this feature But I am still not ready to release it to the customer for everybody as a default, right? So then what I do is at account level I can enable that workflow So in vst as account is like, you know, I as a user if I want a particular, you know a subscription with five users I would go and Uh register myself and create an account and then I can set up a team of you know x number of people right so Now since the feature is complete. I can redirect few users here Who can start testing it and providing us the feedback? It's still the same code base that is getting deployed, right? Now I am confident that this works for everyone Right and at that stage I might You know switch on the feature flag by default and now everybody is flowing through this Right, but my previous feature is still there It's still there. It's a dead also, right? And we don't let it Grow large, right? So what we do is we just get rid of the previous feature, right? And then this becomes the default workflow right and Trust me. This has helped us immensely because you can't complete a feature in three weeks. No, it doesn't matter. You would still have to check into master It would still have to be there in the in in the master It would still have to be deployed you come back in the next print start from there Right, it helps us in testing because you know, I can give you alternate flows and say you you know You are a certain set of customers who can go and basically test Uh, you know this new feature and give us your feedback, right? By the way, if you're using so this is just an example to say that I have enabled it for these two accounts For one I have enabled the feature the other one it is off So one would basically go through the flow and you know be able to see the new feature the other would It'll still be in the master No, so the the the point there is that unless that feature flag is enabled, right You don't go through that workflow and that feature. Yeah, that feature flag you could limit and say hey It's just going to be MVP. It's just going to be my team It's just going to be a few set of folks and if it breaks breaks because the whole expectation is that you know There is a new feature. You just want to have a look at it, right? Look as I said, we are moving towards L zeros I cannot move my code beyond pr Like I if I send a pull request it will get rejected if it doesn't have test Compliance so pr you can you can set the compliance And it will run all the tests if there are failures you need to go and fix it before you can you know move to ci so if So reviewers would not even look at it, right? The second thing is that When you look at our prs, we will have failures. It'll be around 70 75 percent So tests will fail there, right? When it comes to ci you will see suddenly that it is around 95 96 percent, right? Because you've reduced all your You know bugs within the pr itself, right? Which is your pull request, right? And by the time people reach to ci It's a much more mature You know service But we still have failures, right? We still have failures so In the interest of time. So this is also the last section Everything that we do we measure also right and measure on three parameters. We measure on operationally how it is doing we measure on business You know kpi is we measure on how that feature is basically being used, right? And every pm has a responsibility to basically look at that data, right? So instead of me showing you you know these slides which is basically our stats around the service and staff This is the root cause analysis again publicly available This is the dash, you know, this is this what software engineering uses so you can see number of large-scale incidents for every team number of You know the bugs that each developer has you can see red where we already know that you know people need to basically go and fix it Otherwise they can't move, you know proceed further. So these are some of the things that we track at operational level as well, right? um Few things that i want to show you before i get to the summary part which is This is our dashboard And it refreshed sorry, but those boxes that you saw there they are environments where all the code for master branch is going And it's basically testing for You know 10 to 20 these are all builds these are all the builds right and then we have You know, obviously some error. So this is the default for for an example an engineering manager to just have a look at How the quality is across all the environments, right? Since i spoke about Telemetry I mean this is just one of the features that i basically We enabled right and i can see actually how many people are actually coming and basically hitting that that kind of stuff So you need to know how the usage You know how your service is actually being used is it really valuable So there's a lot of focus on telemetry. I mean um This just to make sure that Everybody is basically telemetry is like a default Component of your specification. You got to know what you're building is something that you know will be used So every everybody has to do this so that is there from you know operational perspective business and other other You know aspects as well. I'm just thinking is there anything else? That I wanted to show you before cool so Yeah, perfect on time, right? Sorry, there's the deck. So we probably don't need the deck In summary We didn't do it in six months right We didn't do it right at the first place Right and we are not done yet, right? It's going to take time Right, it's a journey at every stage you have to figure out if there is waste Like we figured out, you know, we've been spending time on manual testing. Let's uh, you know invest in automation, right? You you got to like You know, keep it going because it's going to be a journey, right? And all I can say is all the best from microsoft for your journey And if there's anything that we can help with we would love to thank you so much