 Feature flags, again, so I'm kind of going through our progression of evolving into a service as part of becoming a service. One of the things that we needed to do was be able to control disclosure of features. And we have a couple of conferences, as you all know, that we tend to target, build and connect with new functionality. Feature flags serve multiple purposes, and this is very much a cloud thing. We never had this notion in the on-prem product. So, as I mentioned, I need to be able to couple deployment and exposure. If I want to go to announce something at a conference, how do I allow the engineering team to go build it, test it, deploy it, all without necessarily showing it to the whole world and, let's say, ruining the event in some sense? Feature flags provide this kind of control, and it provides a lot of granular control. All the way down to the individual user, and we'll talk about the granularity there. I can make changes without a redeployment. There's no need to deploy anything. When I make a change to a flag, the whole system knows about it. There are a set of notifications that go out. All the services react to it. It's controlled by PowerShell or WebUI. PowerShell, of course, for the automation, we may do a deployment. After the deployment is done, we may go turn on a feature flag. That can all be fully automated. But it might be that somebody needs to turn on something for either a personal account to test something out, or my best example is SSH. When we rolled out SSH, we had a set of feature flags that controlled it, of course. Basically, one feature flag that controlled it. It's a feature that not everybody wants. Not everybody wants to go use SSH. We've got a standard way to turn feature flags on, but sometimes you need to do something different for the feature because of the audience. In this case, I literally wrote a blog post and said, hey, if you want SSH, turn it on. Let me know. Every time somebody sent me an email, I'd go turn the feature flag on or actually have the lead who owns it turn it on for me. But they would go through the WebUI and I'll show you that. The whole purpose here is we want to be able to do early experimentation. Some of you have seen some of our features very early on when they're still pretty rough because you're in the early rings of four feature flags. And getting this feedback has been incredibly valuable to us. We can expose something very early on knowing that it's not fully baked, get feedback, iterate without waiting until it's all done and goes global. And sometimes it's very helpful to be able to turn something off. Let's say that you develop a new feature, it's buying a feature flag, you turn it on. Something goes wrong. Whatever that thing is, you've got a quick way out. You can always turn it off and that turns out to be very, very valuable. And sometimes, by the way, we use feature flags for architectural or infrastructural changes as a way to use this kind of granular control to be able to either test it out incrementally or to be able to turn it off if something goes horribly wrong at any point in time. So to turn on feature flags, we have a standard process that we use. We call this stages because we have two separate concepts, rings and stages. Rings are for deployment, stages are for feature flags. Stages are all about disclosure or exposure. So for example, our stages are our team's accounts, our personal accounts. So our team's account meaning MSH. There's a lot of stuff we deploy. There's stuff running on service right now that you don't see because the only place it's turned on for is MSH. And that way our team can use it, go experiment with it. Our personal accounts, many of our personal accounts, we go turn stuff on early so we can try it out. Select customers, that includes many of you. You're in early rings on early stages of feature flags. Opt in. So these days when you look at the UI, you see that preview panel that you can get to from your avatar. You click on your avatar, click on preview features. They're all feature flags. But in the UI it doesn't look that way, right? It just looks like a switch. Oh yeah, I want to turn on that new release editor. Great, hit that. You actually just flip the feature flag yourself. And then of course we can turn things on for every customer when we're ready to roll it out globally. We have a great deal of control too. We can control it down to the individual user, to global, and everything in between. And this flexibility has proven immensely valuable over time. The other thing that's key by the way, see how these feature flags, you've got a way to incrementally expose things. That's really only going to have the most value if you've got the right monitoring in place. So you deploy some new feature, it's turned off. You're turning it on. Without the telemetry and the monitoring to tell you, is it working right or not? And by the way, the avoidance of failure is not success, right? It's like, oh I turned it on, nothing bad happened. Well okay, not that you can tell. But we all know if you listen to your system, if you listen to your service, it can tell you things before things really go wrong. So you want to think about, when I turn this feature on, what are the important metrics I should look at to know if it's truly healthy. Yeah okay, nothing bad happened. And of course, we can revert it. As mentioned earlier, you can turn it off. Let's take a look at a quick example. This is the, we added a couple of buttons here, cherry pick and revert in the pull request UI. We did this about a year ago. And we actually did these under feature flags. And it makes for a nice sort of example of how this works. So let's take the revert button, sitting there in the UI. Even in, so how do we define that? We actually have an XML file that defines all the feature flags. And yeah, it's kind of long. But that's okay, there's one per service. And it's kind of self managing. And the fact that if a team ends up with their section getting really long and they look at it and go, I got a bunch of old flags, they'll delete them, especially when they delete it from the code. So it cleans itself up over time just because there's natural motivation to kind of control the size of that file. But again, going back to, we've got this server framework makes for a lot of reuse economies of scale across the whole team. In order to use a feature flag, I've just got to go add it to an XML file. The rest of it is handled for me. All this granularity, the web UI, PowerShell, I get all this for free. So I define my feature flag. I call it source control dot revert. Here's the actual type script from that page. The actual as of a year ago. And you can see that it calls the feature availability check. And there's one in .net. There's one in type script. So you can use it either place. And it checks and says, hey, if this thing is enabled, then great. Go put this button up there, on click, call revert. If it's not, then skip it. And this one, of course, is particularly easy, right? Because the button's either there or it's not. But it's all controlled by this. What I had to do as an engineer to add this to my code. And so how do I control it? I mentioned earlier, you can control it through PowerShell. And it's as simple as you might expect. Get feature flag, set feature flag, specify user account, whatever your scope is, et cetera. All defined on a wiki if you want to run it. Run PowerShell manually. It may also be part of your deployment. As part of your deployment, once a scale unit is upgraded, you may go flip feature flags. This is a set of PowerShell calls that are part of that deployment, fully automated. Here's an example of using the web UI. Now, like most internal web UI, it's particularly beautiful. And you can see here I've got a feature flag for source control.revert. I've got two personal accounts listed here. One called Halux and the other one is in West Europe. And I've set the state to on for Halux, which happens to be in scale unit one. And West Europe, I had it turned off. And this is great, as I mentioned earlier, with the SSH example. I said, hey, email people. We even had EWALDO let people know in the user voice. Hey, we've got this feature now. Send us email if you want it turned on. This is how we went and did it. OK, so I've got two questions. You first. So is this tightly coupled with VSTS and TFS itself, or could this feature flag system be easily replicated in any other system? The way it's implemented, it makes use of other features in the framework. So it would actually be hard to separate it as it is. If you want to do something today, our standard recommendation is to have a look at LaunchDarkly because they've built a whole service around this. And quite honestly, they've got a bunch of features we don't have. We can do analysis on this stuff. But it's still not as rich as I would like it to be. We don't have AB testing built around it, et cetera. There's more capability there. And they've built a business out of it. So I would recommend going with theirs. At some point, we could, but it's not a ton of motivation. And it would actually be fairly expensive for us to try to extract it. And Buck, I think when the partner summit starts tomorrow, I think LaunchDarkly will be here. OK, great. Yes, so you can meet with LaunchDarkly if you'd like to talk to them. There was one in the back, and then I'll get you. So this is custom built, right? Yes. OK. And you guys have, like, a lifecycle for this feature of flags, like when we make something available publicly, do you go back and delete those feature flags? Or you just, you know, Mark, that's just true when they just continue to stay? Yeah, so it's a good question. Many feature flags go away, and the teams themselves take care of that. They decide when to go delete the feature flags. And again, as I mentioned earlier, it can get unwieldy after a while, so there's some natural motivation to go clean it up. And of course, it's easy cleanup. You're just deleting things. There are a set of feature flags that are sort of persistent for things that are more infrastructural, where somebody says, hey, if I ever deploy a regression in this, I want a way to turn it off, so they leave it in there. But there's a downside to that. If you've got two ways for your code to run, if you don't test it both ways, it may not work when you go flip it in the opposite direction. So if you're going to keep it, you've got to make sure it stays working. So there's also a tax to keeping it beyond just the code maintenance. Just one more thing. So is this allowing you to have that flat branching strategy that you showed yesterday, or this is not related to that? It's a great question. And yes, this is highly related to that. I'm glad you asked that question. So by having things behind feature flags, I can have half-baked features in Master and not affect anybody else. As long as it's turned off and I've feature flagged it properly, then I can have it in there, not affect other people. There are still other things you have to do. You have to think about things like security. Like if I feature flag something in there, but I leave some of the code quote exposed because you can't really get to it, but let's say it's a rest endpoint. If I add a new rest endpoint, I either have to make sure it's fully shut off or I've thought about security, right? Because we don't want to ship security holes. But yes, that is a key part to allowing us to work in Master. The other thing I tell people in that particular question is, there are certain destructive or disruptive changes you might make, and you may choose to create a feature branch for a small amount of time. One of my best sort of extreme examples is that transformation of going from a wide schema and work item tracking to a long one. By wide schema, I mean, it used to be, literally, there was another column in the SQL table every time you defined a new field in work item tracking. That's what I mean by wide schema. That took months to pull off. When they first started doing it, as you imagine, they're making some major changes to work item tracking, very destabilizing changes. So they did the destabilizing changes in a branch separate. But what my rule is, people come up in a Master as soon as they are not causing any harm. And so all this is related, absolutely. Just a follow-up to that. So what kind of feedback are you getting from your developers that are actually using this flat branching strategy? Because now they're going to have to deal with each time they're making a change, they have to create this feature flag and all of that, versus creating just a feature branch. So what kind of, do you get any feedback from developers that, okay, we should probably start thinking about that, or this is just, so would you recommend, because the reason I'm asking this is, if I'm in front of a customer, would I actually recommend like a flat branching strategy, because we are doing it and we've tried it and it works really great? Yep. So your question is actually, there's a lot that goes into answering your question, so I'll give you a little bit of it. A lot of it is actually answered in, Manil Shah is going to do a presentation on quality. All of that talk really talks about why we can work in master. We started working in master before we had our testing as good as it is today, and it was rough. But working in master, having feature flags, 100% love it. There is some tax to it, right? You've got to test it both ways. But the advantage you get of being able to have your code go out to production sooner and be able to iterate and experiment, there's so much value that you're willing to pay the price, and I don't have to ask people to add feature flags. They just do it, because there's that much value to them. So yes, I absolutely recommend to customers, use feature flags. The whole test transformation talk that Manil's going to do is just really key to having, not only having everybody working master, but do it and do it well. Because like I said, we started before we really fixed our testing. It was a bit miserable at times, but I would absolutely recommend it. Question? So it's pretty much every feature, every new feature in TFS or VSTS developed under a feature flag, or how do you guys decide when do you use them? So I leave the decision to the feature teams. So the feature teams are empowered to figure out do I need a feature flag or not? Now, not every change requires one, right? So if I choose to change some element on a web page or whatever, I mean, it's a judgment call, right? But in the case of something like the revert feature I used as an example here, it was great to be able to have that behind a feature flag because that meant they could put it out in production. We actually tried it on MS Inge. We sent an email to the team and said, hey, we just rolled this out. Would you go try reverting some stuff? Let us know if you have any problems. So it adds enough value that people just naturally gravitate to it and I let them make the decision. They own their feature area. Aaron talked a bit about autonomy. It's part of it. And quite honestly, I wouldn't be able to know enough to give them a better answer than they themselves could come up with. So we let them take care of it. Yes. So if you just showed the button, that's something that you can really imagine like show the button or not show the button. But if you are in the back end code, for example, so you have to change a few methods, change the behavior of a few methods, you add new ones and how do you prevent that people call that new method instead of the old methods. So how do you work with back end stuff instead of the front end? Yeah, and it's interesting. And it all becomes, it all depends on exactly what you're trying to do because sometimes it's, you've got the same API, you didn't change the API, but the underlying implementation is now different. And so as a result, you're going to have in the worst case two separate implementations, your original and your new one. And the feature flag determines which one it goes into. And those are also changes where you as a user are never aware of them. So some engineers made that change. They put it behind a feature flag because they want to be able to roll it out gradually. And they're looking at telemetry, deciding if they got it right or wrong, et cetera. You as a user actually never know. But so you could roll out a performance change this way. And we have a variety of sort of internal things. And sometimes you mention adding a new method behind a feature flag. Yes. So in theory, it could call something that wouldn't be enabled at run time. It could happen. Quite honestly, it doesn't seem to happen very often. And most of it is because of testing. So if by default that feature flag's off, when you start writing a piece of code and trying to call that new thing, it's going to fail because it's not there, quote there, right? So I leave it to the teams to figure that out. But quite honestly in practice, it really hasn't been a big deal. Yes. When you do feature flag cleanup, do you also remove references to the feature flag name or different versions of methods? Do you clean up the code afterwards? Yes. So since you add the feature flag to the XML file, it generates constants. So it generates constants for C-sharp. It generates constants for TypeScript. And so when you remove that, that stuff goes away when it gets regenerated, which is nice. So you've got the actual code that's using it. If I take the constant away, delete the line of XML, I regenerate the constants. The constants disappear. Then at compile time I get an error if anything references it. So it becomes a very clean way to take it back out.