 Okay, so really excited to introduce Caitlin, she's going to give us a great talk, let's give her a round of applause. Can everybody hear me? Yeah. Yeah, okay, awesome. So I'm Caitlin, hello. What we're going to talk about is intentional deployment, so best practices for feature flag management. And I know you guys are all thinking like amazing title Caitlin, like great work, right? But what does that actually mean? Let's talk about it. So intention, let's go down to like the smallest block we can talk about. Doing what you meant to do, super important for software development. So we're usually writing code with intention. I don't know if any of you are just, you know, banging your keyboard against your standing desk and assuming perfectly syntactical pipeline just shoots right out. No, nobody's doing that, right? We are writing code, we're testing code, we're reviewing code, we're putting a lot of effort into making sure that our code does exactly what we want it to do. And then somebody had this amazing idea, which was, okay, when it comes time to, you know, throw that perfectly tested reviewed bundle of code that we wrote out into the world, maybe we should be being intentional about that too, right? So let's be intentional about our code when we deploy it out into the world. And not everybody has, you know, CI, CD canarying systems, not everybody has very fancy or automated ways to deploy their code into the world and have a lot of control over it. So what is a super easy way to have a lot of control over your code once it's out in production? Feature flags, right? So if you don't know what a feature flag is, it's basically a light switch. You can wrap your code in a feature flag and say when the feature flag is on, we're going to do something. Otherwise, we're going to do something else, right? Pretty simple. So let's look at what that process actually looks like, right? So we wrote this amazing piece of code. It solves every problem ever, it's called do something, and it's time for us to deploy that. So we wrote the code. We're going to put it behind a feature flag. We're going to say if the feature flag is on, we're going to use our amazing function. Otherwise, we're going to use what was already existing there. We deploy that, this code just as it looks right now to production, except maybe it's, you know, more syntactical. But then we put it behind a feature flag and this one is going to be a release toggle, which means that we're going to slowly roll this feature out to a number of users by turning the flag on in production. So everything goes great. We turn the flag on. We start rolling it out. We roll it out to 100% in production. And the important thing to note here is that we didn't actually have to do a code change to turn the feature flag on. So we did this externally to do any code change. We were able to just roll out the percentage of the feature to production. So this lucky user has the feature on. They were able to see your feature. Maybe something went wrong. Maybe your feature didn't work as intended. Maybe you just want to roll it back. You turn it off. And then all of a sudden they don't have access to your feature anymore, right? And none of this required a code deploy. Everything is working great. So you had this amazing idea. It worked out perfectly. You're thinking, I'm doing so great. I'm going to get a promotion. Everything is working out great for me. The only issue is that everybody else in your company also had the same amazing idea, right? So when feature flag becomes feature flags, now we're starting to get into a little bit of like a hot mess situation, right? And then not only the volume of these feature flags is an issue, but also they're all different types. So they all have, you know, different purposes, different lengths that they should stay in the code base, different functions, and that's contributing to this mess. So your feature flag from before was a release toggle. It turned things on or off and, you know, slowly rolled something out into production. Maybe this feature flag over here is a kill switch, right? It's just intended to turn a feature off if something is going wrong in production. Meanwhile, this feature flag over here is an experiment. It's trying to test two different things with two different groups of people. But this one over here is an opt-in. So if people have this feature flag on, they're getting a feature. And then this flag over here is an opt-out. So if people have this one on, they're actually not getting the feature. So all of these are coming together kind of to form this mess of feature flags. You don't know what's going on. You don't know what any of these do. And they all look kind of the same. So this is a problem, right? How do we fix that? Or rather, a better question to ask is, what's the best way to do feature flags? What's the right way to do it? So this is a problem that I've been trying to look into for the past couple months. And the reason that I'm trying to look into it is because I'm a software engineer and I work at a company called Optimizly. And Optimizly has a feature flagging product, right? So I work on a feature flagging product. I use feature flags in development. I hack on feature flags. And I also talk to a lot of people who use feature flags every day or who manage feature flags every day. So just for this talk, I spoke to a lot of people, right? I spoke to, you know, some back-end engineers, some front-end engineers, some SDK engineers, some people in management. So some PMs, a director of development, some people who are, you know, post-sale cycle talking to customers, so sales engineer, solutions architect, and some people on DevOps and Quality. And I learned some things about some best practices that I'm going to try to share with you today. All trying to figure out what's the best way to do feature flags, right? Answering that core question that we had before. So we talked about a couple things. We talked about feature flags and we talked about some people. What goes wrong when we compare those, when we combine those two different things, right? Because that's kind of the more fun part of it. So let's talk about, before we talk about how to do feature flags, right, let's talk about how we can do them. Really, really wrong. So let's start on, like, the lowest risk end of this spectrum. So inherently, when you're putting a feature flag into your system, you're adding tech debt and confusion into your system. We're increasing the complexity of the code and we're making it harder for engineers to understand what's going on. So if we compare these two code samples, right, the one on the left has a feature flag in it and the one on the right does not. The one on the left is inherently much harder to understand and even impossible to understand, right? Because the one, I mean, it's more lines of code, obviously it's a little bit harder to read, but you don't actually know if that feature flag is on or off. So literally not one person in this room can tell me what the code on the left is going to evaluate to. You don't know if it's going to do something or if it's going to do something else. Meanwhile, the code on the right, we're all pretty clear, it's going to do something. So we're increasing the complexity of the code. And not only that, but let's take that confusion that we felt with those two code snippets and let's apply it by the amount of feature flags that we have in our code base and the number of lines of code that they're gating. So I took like a little look into our own code base to see how many feature flags optimally has in our code base. I found over 40 instances of feature flags and they're all pretty big, so they're all gating like thousands of lines of code. We're introducing a lot of tech debt, a lot of opportunity for things to go wrong. The next thing to talk about is taking out a feature flag kind of sucks. I don't know if anybody here has tried to do it. This is from a real feature that I worked on that was gated by a feature flag. This is just the PR to take out the feature flag, right? Over a thousand lines of code, 100 comments on GitHub, and it stayed in the code base for over 60 days, right? So none of this is super great, none of this is best practice, but it just shows like taking out a feature flag, it can be a lot of work, it can be a lot of tech debt in our system. So now we've talked about things that were kind of more low risk, right? We're not directly impacting customers, we're just kind of pissing off a bunch of engineers. Now let's talk about something where, you know, we're kind of impacting customers now, right? We're seeing an impact on our customers. So we can have performance impacts on our system just by using feature flags. What happens when, you know, your feature is broken, your feature tanks, your feature is improperly implemented. These are things that you have to think about and they're going to have performance impacts on your system. So we once had a feature flag that was improperly implemented. It added 48 seconds to one of our API endpoints. I don't know if anybody is like super patient in this room, but I personally don't have the patience for 48 seconds for any answer to any question, much less than API response. Okay, so now you guys are like, okay, Caitlin, how bad can this really get? It doesn't sound super risky, how bad can it get? The answer is really bad, right? So we can get into a total disaster scenario pretty quickly. Has anybody heard of Knight Capital Group? I see a few nodding heads. Okay, so if you haven't heard, they were a financial services group. The key word here is were. We'll go into why. So they were the largest trader in U.S. equities in 2012. They were trading $21 billion every day. It's a lot of money. At 9.30 a.m. on August 1st, 2012, they held $365 million in assets. Remember that time, 9.30 a.m. By 10.15 a.m., they were straight up bankrupt. So they had a $460 million loss in 45 minutes. Let's talk about why. If you might have guessed, it has something to do with an improperly managed feature flag. They had this code deployed to their servers in production that said if a feature flag is on, we're going to do this old thing. This code was eight years old. It wasn't being used because the feature flag was off, right? Now they add some brand new functionality. Uses the same feature flag. But now we're turning the feature flag on so that we have access to the new code. But the old code is still there. So when we turn the feature flag on, we're actually bringing back from the dead this eight-year-old code. What does that eight-year-old code do? So they're a trading company. The code was responsible for processing orders. So this old code that came back from the dead is responsible for processing orders. Doesn't have validation to stop processing orders. So that's a very important part of the puzzle here. And it went rampant for 45 minutes, non-stop processing orders, mounting to $460 million in losses, right? So it's a pretty big deal. If you want a good visual of that, this is their stock over that day. Definitely illustrates the risks of, you know, an improperly managed feature flagging program, right? Total disaster. So now everyone's scared, right? We're all like, oh no. We're going bankrupt 45 minutes from now by the end of this talk. No, actually we're not going to because everyone here is going to follow these practices and be really intentional about how we're deploying code using feature flags, right? So how can we mitigate these risks by being intentional about how we're using feature flags and how we're controlling them? Because if you think about it, we used a feature flag because you wanted to be intentional when you wrote the code. So why are we not being intentional about controlling these flags after they're out in production? So let's talk about some best practices, right? These are the things that we're going to follow so that we don't have any disaster scenarios. We're implementing feature flags correctly. The two things I want to talk about are visibility and accountability. These are the two things that we're going to strive for in our feature flagging systems. So let's talk about visibility first. The core concept here is I basically want you to know where your flags are and what they're doing every hour of every day of every night. I want you to be a helicopter parent on your feature flags. You're going to stalk your flags, right? So for every single flag you should know the state, the history, the metrics, and the expiration at any given time. So the state is pretty simple, right? We set a feature flag as like a light switch. I want to know if it's on or off. The history could be a little bit more complicated, a little bit more related to what you're actually doing with flags. So, you know, some questions you might want to answer. When did it last change? Why did it change? Who changed it? Why did they change it? And then for every single change that was made, was it turned on or off? Was the rollout percentage effective? Who was it being targeted towards? Did that change? You're going to want to know all of these things for every decision made in your feature flagging system, you know, throughout the history of the flag. The next thing you want to do is have some metrics. So the first thing you want to know is, is this flag even being used? Is it being evaluated? Are users hitting it? So you want to know how many times has it been evaluated? When was the last time it was evaluated? The next thing you want to know is, you know, what metrics are tracking this flag's success or failure? Is it properly implemented? Is it just tanking? What metrics do we have around the flag? And then the last thing you want to know is the expiration date, right? So we said before that you're adding technical debt to your system when you add feature flags in. So it's really important to know exactly when is this technical debt coming out of the system and who is responsible for taking it out. So we talked about things that, you know, you're going to want to know for every single flag. Now we want to have a bird's eye view of your entire system. So for your entire feature flagging system, how many flags are in the code base? Where are they in the code? The percentage of your flags are rolled out to X percent, right? So on any given day, how many of your flags are rolled out to 100 percent? How many of your flags are rolled out to zero percent? They're not being used. And then also which flags are expired? Which flags are expiring in X days? So which flags are going to expire in 30 days and 60 days and 90 days? These are really important questions to know the answer to for planning. The next thing you're going to want to think about is collaboration, right? So other people might be using your flag, not only in like the disastrous situation which we saw with Knight Capital Group, but also in, you know, other ways. A feature flag is really cool because it can turn what used to be a code deploy and engineering decision getting the code into production into a business decision. So now, you know, you can deploy the code and then separately you can turn the feature flag on in production. But the important thing to note here is that you might not be the one flipping the switch on your feature. So who else needs to know about these things, right? Or what else needs to be gated behind this flag? It might not only be code. You might be gating docs. You might be gating marketing material. There's other teams that might be infected by this flag. The other thing to know is who's controlling the flag, right? So is it a developer? Is it a release team? Is it a PM? And then the last thing is who needs to be notified of this change, right? So we said you're affecting other teams. You might be affecting other departments with your flag. Who needs to know about it? And the important thing is that we're going to know the answer to all of these questions before we put the feature flag into our code base. That's really the key part of being intentional. None of this has to be particularly high-tech. You know, we have a very low-tech stack here where to find your feature flags, you're just searching for them in the code. And then also to track your flags, you have a shared spreadsheet. Really easy. Note down every time somebody has a change, have a whiteboard in the middle of the room if you're a smaller team. The important part is that you're just tracking these and you're asking these questions. So here, because I work on a feature flagging product, I can have like a little bit more of an automated higher-tech solution. But the important thing to note is that we're answering the same key questions. Where is the flag? You know, up here, you can find it in GitHub. Is it being used? This one was last accessed five hours ago and 10 times overall, because it's kind of a small sample. And when does it expire? Answering the same questions, we're being thoughtful about these things. So now we follow the best practices. We have a very visible system. It doesn't really mean anything unless we do anything about it. So there's two parts of intention, which is I'm going to plan what I'm going to do and then I'm actually going to do it. I want you guys to be accountable for your feature flags. So for every single flag, we want to make sure that it's tracked in a shared place. So everybody who has a stake in this flag or the outcome of this flag has access to it. And we want to make sure that we have a learning setup on our monitoring and our metrics so that we're not just screaming into a void, nobody cares. Otherwise, there's no point in having monitoring. The other key piece of accountability is basically a lot of procedural change, right? So we want to decide as a team or as a company who owns, managing your feature flags as a whole. Somebody needs to be in charge of this. Somebody needs to be driving this. So is it a release team? Is it a PM? Is it an engineering manager? Making sure you have this visibility and accountability for every single flag. The next thing you want to think about is maybe you want to have a whip limit. So this is a work in progress limit. It means that you're limiting the number of features or feature flags in the code base. So personally, my team has a whip limit of two features in flight at any given time. Can't go over that. So because we tend to use a little bit larger feature flags, we tend to only have two feature flags in the code base at any given time. The important thing to note here is that we're considering feature flags to be work in flight, right? So it counts into the discussion of how much work in flight do you want to have at any given time. And the important thing here that you might want to do this for is that the people feeling the consequences of tech debt or of leaving these feature flags in the code base are not always the people prioritizing the work to take them out, right? So if you have a whip limit or something limiting the number of feature flags in your code base, they're going to be incentivized to take old features out before they put new features in. Another thing that you could think about as a team, as a company, is what are the list of must-dos before somebody implements a feature, right? Super easy to just have a feature flag checklist before anybody puts a feature flag in. We have to be thinking about a couple of things. So some of the questions that you might want to ask, what tracking do they need to have around the feature? What metrics do they have set up? What's the procedure they need to follow if you're using a shared spreadsheet? What do they need to enter? What do they need to be tracking? What kind of changes should not be behind a feature flag? Definitely a lot of changes that shouldn't be behind a feature flag, right? If you have destructive database changes or things that can't be rolled back or turned off, those should not be behind a feature flag and everyone should be checking that they're not making those kinds of changes before they put it behind a feature flag. And then the last thing you want to talk about is the lifespan. So for any given type of feature flag, how long can it stay in your code base, right? So for a short-lived flag, maybe for your company that means a day, maybe it means a week, maybe it means a month, do you want to have long-lived flags or do you not? The important thing here is that you want to know how long does your system take to get meaningful results, right? So maybe if you're getting a lot of traffic to the feature, you can make these decisions based on an hour, based on a day. Maybe if you're not getting a lot of traffic, it takes you longer. The important thing is to think about, you know, setting up limits for these things, and then at least if somebody has a really good reason for why they can't, you know, follow this limit, they've thought about it. They give you a limit, a reason for why they're going against the rules. Instead of just the default being, I'm going to do whatever I want. Okay, so we've talked about kind of ways to be accountable. Now let's talk about expired features. So we need to think about features as expired features as part of the work of the feature. So the feature is not completed until the feature flag has been taken out. And that's going to help us retire flags which are expired or unused, which are the flags that we want to go. How do I make sure that expired features are removed? So there's a couple of really easy ways that you can make sure that you're getting expired features out of your code base. You can have a feature stand up every week where, you know, this is where visibility comes into play because you can say pretty easily if you have a very visible system, all of these flags are either at 100% or at 0%. They need to come out. They've expired in next days. They are expiring in next days. Here's the ones that need to come out and plan for that. The other things you can do is you can ticket the flag removal when the flag is created. So if you're using, you know, JIRA or something like that, you can make the ticket to take the flag out as soon as you, you know, JIRA out the feature. You can also, even the most extreme version of this is write the PR to take the flag out as soon as you put it in. You're going to have the most context right then and if it's a pretty simple flag, you can write the PR to take it out right then. And then an important thing to think about here is that these are procedural changes. So like any procedural change at your company, it's really important that you do a couple things. You need to ensure a view of the processes that you put in place with all of the people who are stakeholders here. You need to gain buy-in from all the people that are stakeholders in this process and you need to communicate these standards widely once you decide upon them as a team or as a company. A couple other just kind of rules of thumb. They might not work if, you know, you have a very specific use case or depending on what you're using flags for, but it's good to keep flags as small as possible and as small as possible, I mean the code directly gated behind the flag. Important to keep that as small as possible if you can because it'll make removal a lot easier. Keep flags, try to keep flags as short-lived as possible so as soon as they can come out, they should because they're added tech debt, they're going to be harder to take out down the line and as top level as possible. So what I mean by that is bubbling up feature flags so the highest point they can live in in the code. If you just think about, you know, it's an if-then statement. It's kind of a branching structure. If you have one branch up here as opposed to a million little baby branches down here, it's going to be testing a lot easier if you only have, you know, two cases to test rather than a bunch of little ones. Okay, so in summary, we are all going to manage our feature flags otherwise some bad stuff can go down and we're going to do that by stalking them 24-7 and we're going to have monitoring, alerting on our flags, keep ourselves accountable and retiring our flags once they're, you know, time to go. So with that in mind, hopefully we are all going to go from this hot mess of a feature flag situation to our beautifully organized feature flags on their flag poles ready in their feature flagging system. Thank you.