 Hey everyone, Donovan Brown here. We're going to learn how the Azure DevOps team actually uses Azure DevOps to build Azure DevOps. But better than that, we're also going to show you how you can pick and choose the parts of the services that you need to use in your DevOps pipeline. Hey everyone, Donovan Brown here with another episode of Visual Studio Toolbox. I'm here with my friend Gopi, and we're actually going to show you how the Azure DevOps team uses Azure DevOps. Yes. Azure DevOps uses Azure DevOps for every item. Exactly. It can be planning, it can be pipelines, all of those. Right. So just to give you introduction on Azure DevOps, Azure DevOps suite contains boards. Boards is primarily for planning and work item tracking. Pipelines, pipelines is the one that will help you to do continuous integration and continuous delivery. Azure Repos is for all code storing the code in GitHub or TFPC. Azure Test Plans is for testing, manual testing. Azure artifacts is all about how do I store all my artifacts and you get feeds into the artifacts product. Right. It's always funny, it's like an inception thing, but we actually use the product that we build to build the product that we use. So we are close to 600 engineers who are continuously working on it. Given that the product actually uses the same product, we make our product much better. Great dog-fooding. I think in France, they say drinking your own champagne, but it's very similar thing. Right. Cool. Let's show me how the Azure DevOps team actually uses it. So let's do a quick day in the life. Right. All right. I'm now just switching to the product. This is Azure DevOps product that we have, the entry point. You can see this is the Repos where we actually just store the code. This is the pipelines where we have build and release. This is the Test Plans, this is the artifacts, the same part that I just talked about. Right. Now, how do we use our own product? I'm just starting with primarily the CI CD, I'm just assuming people know how we do the boards or how we do the code, but I'm starting with the bills. So from bills perspective, if I just switch to the build hub, we have a definition called VSO.pr. This definition gets triggered for every pull request any developer commits to your master. Right. For what I heard, it's like 600 a day on average. 600 a day on average. Actually, you can just to see it, these are the numbers that we just keep incrementing on a continuous basis, all the bills will keep going. Got you. I'll just pick up one of those and then just show you, what is the scale and how do we run it? Like, if I just click on the test, literally for every build, every PR build, we run 85,000 tests. Right. Any time you keep seeing it, this number keeps growing up, growing up, growing up. What's amazing is not only is that number increased, but we did it in six minutes and 75 seconds. It's just unbelievable the amount of code covers that we get, and that we do this 600 times a day to make sure that the quality is baked into the product itself. Into the product. Yeah. And this also is what I show people when I want to teach them that this scales, right? Trust me, we can handle whatever workload you're going to throw at it. Right. Because we do this 600 times a day. 600 times a day. Unbelievable. Exactly, right? And then the way that we actually just manage is I just showed you one called VSO.pr. But once the PR is passed, then we have CI builds that get executed too. Like onto the master, now we go to the master and then generate another build. We exactly go through the same validation. And in fact, sometimes we start adding more tests in the CI. Whereas in PR, we do only unit tests. When it goes to the CI, we will do some more additional tests. Right. And we take a good build from there and deploy to all our developer environments, test environments, and then do a lot more testing. Yeah, because even in the prod environments, we had to do a ring safe deployment throughout there to get our code out into production as well. I'm just switching it to our dashboards to show you how we do manage our builds. Like this is the dashboard that is available. You can see this is the master CI. Like every, after the PR is done, every master CI build that comes out, these are various test environments that we have. Okay? Various test environments and on. Every test environment, there is a different kind of testing that's happening. And literally every build gets covered. Like you can just to see that on seventh, like we have covered from the master roughly 54, 56, and it's just going on. Sure. It's currently just going on. All the builds get generated and then going on. And if I have to today fork from this branch and then create a release, like we do continuous deployment. How do we do continuous deployment? If I just to see this is one particular column, everything is green. Right. So if I were to just, I pick up this code commit and then fork it into my release branch if I want to create a new build and then take it forward. Gotcha. But not just that. This is a good dashboard for any developer, you know, manager or a director who is sitting on it. And I just wants to see what's going on. I can just to see that there are multiple reds. But this particular one, you know, started sometime at this point. I click on that. It actually literally takes me to what has started failing. Gotcha. Right. On a particular stuff. Not just there. Now, if I go to the pipeline, it's a self-host environment. If I click on it, how many clicks you think it will take for me to identify what commits actually cause this? Well, I know the answer is not very many because it's all tightly integrated. And that's the key. Even though the services are individual and you can acquire them individually, they're better when you use them all together. Right. Because you get this amazing traceability because boards wants to talk to repos, repos wants to talk to build, build wants to talk to release and they all tightly integrate. So it's really neat to be able to pinpoint a failure down literally to the line of code that I wrote because it's all integrated together. I'll just show you from here, literally in a few clicks, I'll get to know what commits are actually causing me. Everything was passing 100 percent, 100 percent, 100 percent on this. We just suddenly we saw zero percent. Yep. One click. Okay. It takes me to this. The second click that I will do is I just click on the pipelines. Second click. Third click. I click on Commits. And then this is the code. Now it actually just tells me one of the code changes of these two actually made all the tests fail. Exactly. And if the test fail, then user can look at it, he can pull his code back and then get it back to the healthy state. Absolutely. And it's not just the comments. We also show if there are any work items that are associated with it. Sure. Here is the feature that is actually causing. Right. The bug I was working on, the task I was trying to fix, it all gets shown at the top. Full traceability. Absolutely. Intune traceability. Now let me just show you what I have shown you is the developer workflow with respect to how the PR is done, how the CI is done, and then the CI gets into and tested. How does our production environments look like? Sure. Right. So what I'm now just clicking on, this is a production definition. Right. And I shouldn't have edit permissions. Yeah, I don't have edit permissions. Typically we show edit button, but. Yeah, I don't have any there. Like, you know, we just have view permission. So, right. Right. I'll just click on one of the versions. Like the way we have actually divided our deployments are in stages. We have something called, you know, ring zero, ring one, ring two. You can model this pretty easily. You can just see, like, you know, every, you know, scale unit that we target are available here. Right. Right. And then, you know, typically we just take care of starting from ring zero if ring zero passes, ring one, ring two, ring three, ring four. So today currently there is a deployment that's actually just going on for the ring four. Right, it's already gone through ring three. It clearly is successful. And one thing I like to point out when I show this screen is that ring zero is the ring where the actual Azure DevOps teams worked. Correct. So we're the first ones to get any of those changes to make sure that they're good for us. And if they're not good for us, we don't push them on any of our customers, right? We send out another fix. And then only if it passes ring zero, and I believe it's 48 hours, it sits there with us, which means it goes to Hyderabod with you. It comes back to Raleigh. It comes back to Redmond. It does that twice. Correct. And if it survives 48 hours with us, we then promote it to ring one. And then ring one I believe is our internal people, like our MVPs and RDs and things like that. And that into a ring two, if I'm not mistaken, does it actually reach a public paying customer? Right, exactly. That's how it works, got it. But now, again, it's not just that. Now, I assume if I want to now see, hey, the Singapore cluster deployment is pending. If I want to know what payload actually the Singapore cluster customers are going to get, I can literally click on that exact stage. And then I know that, hey, here are the 30 comments are actually getting deployed to them. Right, we will exactly get the same traceability that I just talked about. These are the bugs, these are the user stories, these are the features or the tasks that they are going to get benefited. Yeah, I can look at it at every stage of the lifecycle. Right, and like I said, it's beautiful when you use it all together. But what you and I both know is that we cannot go into a customer and say, rip out everything that you have and replace it all with Azure DevOps. That's what we think they should do sometimes. But there's no way they're going to do that. They have investments in Jenkins, they have investments in Ansible. But I tell them, no matter how mature you are on your DevOps pipeline, there's something that hurts in that pipeline. And what I want you to do is find which of these services from Azure DevOps solves just that one problem. Let's go fix that with Azure DevOps because what's the cool thing is, and I think you're about to show us, is that Azure DevOps can integrate with everything else that you already have. Right, right, you know, you can use our product suite to get the benefit, the site integration. But the Azure pipelines can work with any of your existing tool. Gotcha. If you are using GRAF or your issue tracking, we do integrate with it. Today I have shown you Azure repose integration where the code commits, but if you are using GitHub, we support you. If you are using GitLab, we support you. Bitbucket, the whole nine yards. Bitbucket, we support you. And then if you are using for CI system as Jenkins, we have actually support for it. And we have an internal approval system for customers to approve and then take it forward. But you have a, you know, ITSM, you know, service management system that is available like ServiceNow. Yeah, very popular. Right. And we have a set of tasks that are available for, you know, our own deployment, but you want to use Ansible, we have support for it. All right. So the next demo, I'm just going to switch the demo to show you how we actually, you know, take a simple application and use various different systems. Sure, like Jira and GitHub and Jenkins and Artifactory and ServiceNow. And I'm deploying to your Linux and I'm using a Java application and I'm deploying to AWS. So essentially other than Azure Pipelines in this, rest all or the third party products that the customers want. Perfect. And I will also show you, you know, do I get the same benefits if I use Azure Pipelines? Do I get the same capabilities? Same traceability. Same traceability is what I'm just going to show you. Okay. So I have a definition that is set up and I'm just going to show you at a high level what is this definition doing. As you can see, I'm actually consuming a build from Jenkins. Okay. Right? So as soon as the Jenkins build is available, it will trigger. Then I'm deploying to AWS, QA environment and I have an approval that is configured. Then I'll deploy to production. Correct. Let me just do a code change and then submit the code change. Then we'll come back and then see it. Okay, got it. So, you know, this is GitHub. This is, you know, I'm in GitHub. I'm doing a code change. Like, you know, we have a set of institutions that are available. Assume, you know, today, you know, we have an institute in Redmond. I am now adding Cambridge. Okay. Okay. Now, while adding it, as part of the pull request, I am now going to link to Jira. Right, I'm using GitHub here. The next thing that I'm doing is I'm just linking it to Jira. Jira, you know, issues. And the format here is KOD is the project name in the Jira and then the, you know, 11 is the issue type. Okay. You know, whatever are the issue types you can say and adding another in institute. Okay. Okay. And then I commit my changes. Well, as soon as I commit my changes, you'll actually just see Jenkins, Assured, Trigger, another build here. Yeah, on the left-hand side, we should see. Yeah, you should see 36. There it is. There you go. Yeah. Got it. Now, you know, you will know that the 36 will get done as soon as the 36 will get done. It is pushing the artifacts into the artifact tree. Now I'm just jumping back to our release. And if I just to see the view releases, you can see that there is a new release that got created. Right, so this is the result of your Jenkins job completing. And then that triggered the fact that there is a release. And now all the different parts that have been produced can now come together and be deployed using pipelines. Yes. So now, you know, before I deploy to the QA environment, it's, you know, it's waiting for approval. You know, I can just approve it. And there are many people, I think, you know, just one person approving is fine. And we do have policies that are set up, too, saying that, hey, you know, all users has to approve. Any one user has to approve. Gotcha. Or, you know, you can approve in a specific sequence as well, right? Because if I reject it, there's no point in bothering my boss to get an approval because I've already rejected it, right? Correct. Got it. While the execution is going on, we literally get live logs that are available. You know, now this is the QA, you know, because I approved it, you know, it started executing. If I click on the logs, it just tells me what's going on in it, right? So in my application, I'm actually just deploying to the backend first. But once the backend is done, I'll just deploy the front end. Got it. And so right now it looks like you're deploying to AKS. So that's an AKS cluster. That's Kubernetes. Right. And we have all the tasks that you needed, obviously, to deploy there. Correct. And now you're deploying your web app. And what's interesting is you did those sequentially, but you actually could have the power sometimes to do those in parallel as well. Yes, exactly. I can actually just execute them in parallel as well. Got it. While this, you know, this is almost getting done, now, yep, the QA environment in AWS is complete. The next one, as you have seen, the first one, you know, we did an approval using our own system. Sure. But what I have configured the production environment is instead of using our own approval system, how about using service now? Integrating with a third party system. Integrating with one of the third party systems, you know, for it to work. So, you know, I have this environment. I think, you know, once the environment, you know, gets picked up, it's waiting for one of the agents to be picked up. I think in the meanwhile, let me just, you know, show you how exactly our, you know, the workflow is configured here, right? I think, you know, I see that there is a service now change management that is actually configured. Now what it does is it'll actually just talk to the service now until the service now change management approval comes into our system, we will not take the deployment. Right. So right now, like you said, we got a bill from Jenkins, but we did not go instantly into QA. It basically said, I have to get approval from a human being to say it's okay to deploy into QA, which you did and it deployed. Now it's going to say, the next approval has to come from service now. So service now, you go over there, you perform whatever process you need to perform in service now, and when that's true, we're going to be able to continue into production. Exactly. Yeah. So you can see service now is triggered, it actually just failed for the first time because we haven't approved it yet. Right. And you know, in the next four minutes, if I approve, actually this workflow will continue. Okay. So I'll just jump to service now. This is my service now portal. If I click, I should just see a new request waiting for me. This is the new request. This is the one prod AWS. Okay. Right. I will now follow the service now. And then you can configure any service now workflow based on your enterprise requirements. Okay. Right. So in this particular case, you know, first it has to be requested for approval. And once it is requested for approval, it has to be approved by some person inside the service now. And then the person should mark it as, go ahead, implement this. Right. Okay. Our task will get it, you know, executed further. So I'm just being David Lu now. And then let me just mark it as approve. I have marked as approve, update. So the next one is, I would say it is scheduled. I should just change it to schedule to implement. Okay. And... And now we've completed everything we need to do in the service now side. Yes. Got it. The service now side is complete. Right. So let's just switch back. Whenever the evaluation happens, sorry, this tab, whenever the evaluation happens in this tab, that's when, you know, our workflow will continue. Okay. So why don't you show me while we're waiting this three minutes, like what are the tasks look like and how do we add them to a pipeline? Right. Okay. Now I'm just going back to the pipeline. Like let me just show you how does the pipeline will look like and if I want to add more, how do I add it? Okay. Like the tasks are, you know, similar. I just clicked on one of the tasks, right? So here, the first part is doing Helm. And you can see that we are downloading the RDFact from RDFactory. And then we are deploying to AWS using the Tomcat. Gotcha. Right? But if I just click on this, you can literally see all the tasks that are supported. Right. And a lot of these come right out of the box and those are actually developed in a GitHub repository. Correct. Because I've had two of my pull requests actually end up in the box, which is really awesome to realize that a whole new macro, like my code is actually now available in every instance of Azure DevOps, which is really cool. Yeah. So most of these tasks are available in open source. Right. In GitHub, you can just search for Azure Pipeline tasks. You'll be able to see it. You know, it's not just, you know, I think one of the key thing I want to just to say is you can literally, you know, build any application and you can target any platform. Any language, any platform. Yeah. Because anything that, what I say also is anything that you can do from a command line, you can do. You can do. From Mac, Windows or Linux, right? Anything you can do from a REST API, you can do from Mac, Windows or Linux. So the sky is the limit, but what I like about the task is it takes the complexity of REST API calls and command line interfaces and it kind of takes all that away from you and makes these tasks even easier. And if you don't see what you need here, what I love is the fact that we have a marketplace too. Right. Right. So can you bring up the marketplace and show us what it looks like? I think about just how the marketplace that's, you know. Yeah. So here you can search for security, you can search for testing, you can search for anything. And what you do is you get this amazing, rich list of extensions that you can use inside of Azure DevOps. And some of these are built by us, some of these are built by the vendors. For example, the AWS one is built by Amazon, right? And then there is a Sentry extension that is available. There is a Slack extension that is available. The service now is the one that I was just showing you. Yeah. Like, you know, pretty much, you know, there are many extensions that are available. If you look at the Azure Pipelines, this is where the maximum, we have 650 extensions that are available. That's awesome, right? 650, you know. You can do pretty much, you know, any task that is needed using one of these extensions. Absolutely. Let's just, you know, go back and then see if our pipeline is completed. Yeah, have we been talking long enough? That's the question, right? Have we been long talking for three minutes? And if so, we're going to see some post up here. So you can just to see that the first gate actually, you know, failed because it is waiting for us to complete the service now. Now, once it is done, it'll just to go ahead and take the exact, you know, same payload and then take it forward. Awesome. Now, while this is going on, I'll just show you, like, if you have seen when people were using the entire Azure DevOps product as is, we had a great integration story. Yes, right? Will we have the same integration story? I'm just going to click on the AWS environment. If I just click on the AWS environment, it actually just to tell me, you know, what version that I'm just using, right? And now if I click on the commits, it should show me the commit that I have done. I just added another institute and then this is coming from the GitHub. GitHub Enterprise, it just gives me the exact same link. Not just this, if I go to the work items, this time these work items are coming from Jira. Yeah, and it's also, it showed me the Jenkins icon, letting me know that that was actually a Jenkins artifact that I was actually deploying as well. So you're still getting that same traceability that the Azure DevOps teams gets by using all of Azure DevOps, even if you don't use all of Azure DevOps, which is amazing, yeah. So one of the stuff is, you know, one question that customers ask me is, hey, I am today on Jira, I am today on Jenkins, I am today on GitHub. If I want to transform to, you know, DevOps, when I'm adopting Azure pipelines, do I need to just change my tools? The answer is no, you're exactly. You can just, you know, keep your tools, bring in Azure pipelines. It'll just help you integrate all of these and then, you know, get the transformation. Right, it's the orchestrator almost, right? It's the, people always argue, like, why do when I define DevOps, do I not use the word tools and I use the word product? Because it takes a product like this, to stitch all the tools together to actually give you a DevOps pipeline. This is a perfect example of where we don't require you to rip and replace anything, right? There is a point that you need to fix, and I tell everybody, when you're on this DevOps transformation, fix what hurts most first. It wasn't Jenkins, because Jenkins was working, it wasn't Ansible, because Ansible was working, it was the fact that you couldn't stitch them together, that's what hurt most, and then putting in pipelines, literally glued all that stuff together, and what's so amazing, it gives you such great traceability, even though you were picking and choosing your tool set. Exactly, and you get the traceability and then you can build a similar dashboard that I have just shown you, that, you know, we use for Azure DevOps, with, you know, whatever test framework and whatever test infrastructure that you are, you know, working on, it'll show you the dashboard, it'll show you the results with complete richness. Gopi, thank you so much for coming out and showing us this. We're learning everything that we need to know about the Azure DevOps team using Azure DevOps, and again, if you wanna use it yourself, you do not have to use all of it, pick and choose the parts that works best for you. Thank you so much, everyone, thank you.