 Great, thank you so much for having us today. My name is Andy Marshall with Harness, and we are excited to talk to you about testing in production with feature flags and continuous deployment and all the benefits that come along with this process. So before we jump into the topic, we want to level set and make sure that we're on the same page when it comes to what exactly feature flags are. Feature flags is a technique that was popularized by companies like Facebook and LinkedIn who wanted to release new features to a small group of users for instant feedback and to kind of use that as a way to influence their decisions for rolling out features across their wider base. So today feature flags have become an essential part of any developer's toolkit and lots of companies are adopting them into their process for delivering features. They've been in production use for about 10 years but it's really becoming more mainstream in the last five or six years. So as a result, we're still coming across people today that are new to feature flags and how to leverage them. So as we begin our time, we're just going to take a few minutes to kind of take a look at why feature flags, what are the benefits of them, why you should be considering them, some of the use cases and then how that radically changes the delivery of features. So in order to get going, let's just kind of talk a little bit briefly about kind of traditional ways we used to deploy features before feature flags. Traditionally, developers coded a feature, committed that code, they tested it, they signed off on it, QA did their pass and ran through the new features and put it through its paces. And then there was a wait for all the features to be written and tested and kind of held in a hopper. And when all things were new and ready, we crossed our fingers, we flipped the switch and we'd execute the deployment and all the features were pushed to production. So this probably sounds familiar to you. This is the way we've been doing it for a long time. And sometimes that would take quite a bit of time waiting for everyone to get their features written, have their code ready, reviewed and ready to rock. So sometimes that would take months, weeks at least at a minimum and it was quite the process. And if you think that for a minute that this is kind of old and maybe antiquated and it's no longer reality, I can tell you that we constantly talk to even Fortune 500 clients who are reporting that they still only released maybe four times a year and they're working as fast as they can. So this is a conversation that we have daily with companies that are trying to speed up their release process because they're kind of doing it in the traditional way. There are some consequences that come along with this though. After you deploy your outcome, sometimes features worked, sometimes they didn't, sometimes people got what they needed, sometimes people did not get what they needed and some features may not have worked at all. So traditionally releasing features is kind of the consequence of an all or nothing type release. And there's some consequences that we've noted to this. First, feature releases, the date is often determined by developers instead of a business. And the reason being that if developers wrote a feature and put it into production, they just released it as soon as they were done. And this often causes friction between engineering and operational teams who might want a specific release date to coincide with a business objective. There's another consequence to doing it this way and that is slower development of new features often happens because if we're only releasing every couple months, then it's a little bit slower to get features out. Another consequence is we can't test multiple solutions or multiple implementations because when we release all the features together and deliver that it's kind of an all or nothing approach. So there isn't any time or ability to test those individual solutions. There's also no way to or kind of it's cumbersome to release individual features to production. So I've got to wait and wait for that one big deployment. So if I have a feature ready, I can't just release one at a time. And finally, big huge consequence here is if something goes wrong, I have to roll back the entire deploy, which means I'm rolling back all the good features due to the poor implementation of maybe just one feature. Meaning I have to roll back all the good with the bad. So that's kind of traditionally how we've looked at deploying software. And along came feature flags and there's kind of a better way to do it. The mighty feature flag. It's a little tiny snippet of code that you can wrap around features within your code base and then conditionally turns certain sections of your code on or off. And you can think of feature flags really as an extension of your CD process. It's a way to put chains into production behind a flag and then turn them on in a controlled way later. You can use them to clearly define features and components that can be changed and then toggle off and on individually. And so what this effectively does is it separates the deployment from the release. And they are deployed in an off state. So you can push it all the production and then at a later time, you can turn them on. So if you are deploying code, the result is you can release when you want and you can release to whomever you want. And if a particular flag doesn't have a desired result, you can change the status of that flag and the target. And in the end, we're seeing that we've got happier users who are getting the correct features at the correct time. That is a quick 100,000 foot view of the power of feature flagging. So with that, let's just discuss real quick some of the use cases that we see and kind of how they work. Use cases for feature flagging. The first one in the upper left-hand corner here really is about increasing velocity. So as developers create a new feature, they can park it behind a flag and then they can ship it immediately into production and they get right back to working on new code. And this creates a lot of speed because you are constantly shipping. You're constantly pushing to production. And if a particular feature is broken or incomplete, it's not a big deal because it's deployed behind a flag in an off state. So the results are, it's pretty safe and it isn't impacting any of your end users. It's simpler, it's faster and allows your engineers to speed up significantly and push all new features to production. And when the time is right, you can turn it on after it's been fully tested and it can go live. The second major bucket of use cases we think about when it comes to feature flags has to do more with controlling the release. So because feature flags allow you to target just a subset of users, maybe to try out a new feature before it's ready for prime time and rolled out to the entire user base, it gives you the opportunity then to get some real world feedback. And then you can make any changes and experiment before your entire user base receives that particular feature. So conducting AB testing, experimentation to see how features are being received before it's ready for your entire user base, that is totally a use case that we are seeing currently. In the lower left hand corner there, this box is all about operational control. And so these are use cases that are really for the development teams and allow them to quickly have access to things like a kill switch or safety valves. So if something goes wrong with a new feature, you can quickly turn that feature off and within half a second or less, every connected device, every browser, every mobile, user receives the same command and the offending feature is turned off. And so this goes a long way to mitigating any sort of blast radius when something fails. And it's always easier to fix something or correct bad code when there aren't dozens of support tickets being created and coming in. You can just simply kill the bad offending code and immediately correct the issue. The last kind of area that we think about is that of customer accesses. Access and this is kind of a series of use cases that are fairly cutting edge. It involves inviting other internal teams in your organization like sales or product or support or marketing to be able to have access to turn features off and on for particular clients. And so previously, turning a feature off and on was kind of an engineering function and something just reserved for devs, but Harness is now in feature flags is giving teams a little bit of a, that are closer to the client, the ability to manipulate flags, turn them off, turn them off and on without tying up engineering resources. And that goes a long way for progressive delivery. So pretty powerful. We have a question here from Christopher Strickland. He asks, do you find any organizations are using feature flags with microservice architectures or are those architectures often small enough with that feature flags aren't worth implementing? And if so, could you describe the microservices with feature flags use case? And I think this slide actually maybe is a great one to illustrate that. What we're talking about with microservice architectures where the size of our code changes are small enough that they are independent of everything else. Our services are very independent. So that kind of de-risks that and actually might potentially solve that top left use case there of increasing velocity where a developer as soon as they're changed to their services down, they can deploy it right away without waiting for the others. However, microservice architectures do not solve for the other three use cases that are shown here potentially really around, hey, maybe we wanna have different behavior of that one microservice for different customers or for testing out different iterations of a feature. Maybe I wanna have a flag. So I'd be the top right one. Bottom left one is maybe I wanna have a flag that is on but I have ability to turn it off whenever something bad happens, right? And that is outside of my control and outside of the case of a new deployment. And then of course lastly on the bottom right there, that's really around changing the behavior of the application based on a customer request or a type of a customer or letting a sales team enable a feature for a customer or something like that. Something that those three I would say are not addressed by microservice architecture alone. Excellent. Thanks Chris for appreciate that question. Keep them coming and we will answer them on the fly. Just to kind of wrap up a little bit of the benefits here of feature flags and kind of what you can expect when tied to your CD pipelines. First of all, it gives you a flexible way for the business to determine the date release of features. And this is one that again, has caused a little bit of friction in the past because marketing product and often sales team have their own dates and kind of have their own campaigns and thresholds for wanting to turn on particular as Connor was talking about client facing features. And so feature flags allow other teams to come in and get these releases done on their own timeline. Another great benefit of feature flags that we see is the ability to release new features much, much faster to iterate rapidly without any negative consequences. So there's no more waiting for that monthly release cycle to push or that weekly release cycle to push a bundle of completed features all at one time with feature flags you can release and push them as often as needed. So those can be pushed for production anytime whether it's fully functional or you're just wanting to do some testing on it or just beginning to work. Another great benefit is testing in multiple solutions testing solutions across different implementations simultaneously to kind of determine the best approach for this particular feature. So test solutions, you can test these solutions to a problem in production with customers. You can have different cohorts. You can target individual groups of people and create the feedback for how customers are receiving this and interacting with your new features. And this creates really valuable intel before a feature is going to GA and ready for prime time. Another great benefit that we touched on is you can easily release those features to production daily. You never have to roll back a good feature along with a bad one. This is the throwing the baby out with a bathwater type thing. When something goes wrong in production, just turn it off. That's fine and you can work on that. No deployment rollback is necessary for that particular feature. And all the good features that are still working can be out there in production. And this leads to some incredible positive outcomes. This is one of those tools that we actually see a real tangible outcome in the lives of engineers and developers and the people that are using the tools for feature flagging. Believe it or not, there was a study just recently done that reported 75% of feature flag users are reporting a reduction in stress when deploying new code. I absolutely love that, this makes me so happy because you think about the life of a developer and an engineer, I really like to call engineers really inventors. You're inventing something, you're writing new code, you're producing a new feature that hasn't existed before in a way that maybe it's never been used on a platform where it just hasn't been there before. So you are expected to invent in half the amount of time with half the amount of budget and get it done right the first time. That's pretty stressful, that's a stressful situation. But when you are using feature flags, great benefit in being able to hide them behind a flag, release them when you're ready, test them in production, see how they're gonna work in the real world and then quickly be able to mitigate any damage should there be an offending a feature that doesn't quite work. So that is your quick fly by on why feature flags, unbelievable benefits. Let's just talk really quickly about testing in production. Back in the early 2000s, it was a great joke and I remember this working in development shops where some devs would wear a T-shirt and I saw it occasionally and it says I don't always test my code, but when I do I test in production and everybody laughed, we all thought that was funny, so edgy, so radical of you to wear that shirt and why would they ever test in production? It just seems unbelievably risky and why would we do that? That joke has actually become a reality. And so let's talk about that. Why can we test in production with feature flags and what are the benefits of that? So to begin with, when you test in production, let's talk about what it is really quickly. What it is, is enables your QA and devs to gain real world insight and you are able to then learn at scale. And so feature flags allow you to test in a true production environment with real world scenarios. And let's be honest, sometimes it's really hard to recreate a testing environment with mockup data at a smaller scale. That's a really tough order. So testing in production environments answers some questions that you can't simply answer just by writing great unit tests, right? So it gives you insights and answers to questions like, what does your real load do with the latest changes? Like how is this actually working in production? And what do users do differently than it expected? Is there any way that they're interacting with this new feature that we didn't see? Are you seeing the performance changed that you hoped for? What does the performance look like? Are users reacting the way you intended? How does your change react under various stress situations? These are questions that you have difficulty answering in pre-production environments. But when you can wrap a feature behind a flag, push it to production, target yourself as the engineer of the QA, you can actually start to see in a real world environment how this works and get valuable, valuable insight that you normally wouldn't have. And so that's kind of what it is. And we want to be really clear here that what we're not saying is that it's simply being cavalier and just pushing to production everything that you want to release and deploy and roll back consistently and rinsing and repeating. And this is not a best practice to just be cavalier about what you release. This is not simply using live data to make sure that things work. There is, that's still what QA is for in testing and pre-production environments, but it's more about seeing how targeting yourself, it works in a real environment, but it's not a substitute for testing. It's not skipping the pre-production testing. It's not a substitute. It's a helpful addition to your testing process. And when it's done correctly, there are incredible benefits. It highlights unexpected problems. You can instantly switch things off and on without disrupting any of the application. You don't have to redeploy. You can target specific end users or cohorts and see how it works in a real world environment. So lots of benefits to testing in production. Andy, we have a question here from Karen. Karen is asking, how are the flags implemented? Is it a config file? Is it like a hidden settings page or maybe a remote server that's delivering the flag settings? I can take a first pass at it and you can add more details to this if you'd like, Andy. So there's a lot of different ways to achieve the outcomes of a feature flag. Like just generally speaking, a feature flag is a boolean value in your code, true or false, on or off, show or hide that you then change the behavior of your code based on. So at its simplest, it's a boolean variable that you have an if statement where you might show a page if it's true, you might hide a page if it's false, you might implement the new feature that you're trying to release wrapped in an if else statement. And the question is, well, where is that boolean value coming from? And so at its simplest, it could be an environment variable. A lot of our customers that are looking at harness are maybe starting with having toggles that are in a database, right? They may have a database table that has a bunch of key value flags that are either turned on or off. And then where we're seeing the market really moving to is actually having standalone feature flag solutions, harness being one of them, where it's actually a SaaS based solution where you can configure flags, roll out features and flag values across different environments, kind of applies all the kind of governance and roll-based access control that you need to be able to do that and also decouples it from the actual environment you're deploying to. So hopefully that addresses your question there, Karen. If you have follow-up, feel free to ask another question. But like I said, at the simplest, it's a boolean value, but the question is where does it come from? And that's really where some of the power can come from is having a centralized solution for that. Anything you wanna add to that, Andrew? No, I think that was really, really well said. I'll look for any other questions on that, but would you like to talk about how we're using feature flags today to roll out new features and even how it differs from canary deployments? Sure, certainly. So first off, a lot of times, you can go ahead and go to the next slide. So really, we're talking about the difference between flags and canary deployments. So I just maybe wanna first define what a canary deployment is and then we'll talk about how it is different than feature flags. So canary deployments are an advanced deployment strategy for really reducing the risk of deployments. Kind of think of it similarly to, you might be familiar with blue-green deployments. And the idea with both blue-green and canaries that we are deploying our new artifact into an environment and giving it a fraction of traffic. Why it's called a canary deployment it's kind of named after the canary in a coal mine that is giving us an early warning that maybe the coal mine that we're working in is running out of oxygen. And so really the benefit of a canary is to give you that early warning that's that thing you just deployed is having a problem before it affects everybody. Now, an important thing to note with a canary deployment is we are deploying a new code artifact to a fraction of traffic. And what I mean by that is we're not necessarily targeting individual users or groups of users, it is just a percentage of our traffic. And so what that looks like in your environment is exactly what that graphic is looking like. We have our old version V1.1 deployed. We are deploying just a few instances of version 1.2 side by side with version 1.1 they're each receiving a percentage of the traffic. And really the goal of a canary is to identify if that new artifact that we're deploying that new code version is working or not. And what I mean by working is is it connecting to the database? Is there errors happening? Is there a performance issues happening? Really kind of focusing on the functionality of the artifact being able to start up and work, okay? And you wanna know that before that new version that new artifact you've deployed has 100% of your traffic subjected to it, right? If we're gonna have errors in this new artifact if it's not gonna be able to connect if it's not gonna be able to start up we don't want that to be affecting 100% of our users that once we just can just try it out with 5% of traffic or 10% of traffic, something like that. In the event of a failure with a canary deployment the next step would be to do a rollback and that rollback's easy it's just delete the canary release and because we still have our old version still running in production. So you're right back to your previous version as soon as you remove that new version, okay? So that is kinda, yeah, go ahead, Andy. Sorry, quick question that just came in. Is it considered feature flagging if you want to switch a feature on the fly for maybe a group of users? I'll take a quick pass at this if you wanna add some thoughts to it. I would say probably yes. There's really kind of two types of feature flags that we think of as far as time and there are kind of short temporary flags and there are more long-term flags that you might have that are permanent in your system. So Connors talked about some of these short-term flags for rolling out just for a new feature to roll out to your user base. You might do it progressively over time and once your feature is considered GA for your user base you may then want to go in and delete that flag. Flags are not the number one cause of technical debt for sure but they've got to be in the top five, right? So just leaving flags long-term in your code bases generally frowned upon as it leads to that technical debt but most of them are gonna be short-term. I would say probably in your system maybe north of 85, 90% will be short-term. However, there are some long-term flags that you may want in your system for long periods of time and this could be flags that target a group of users that might be in a different level of products. So maybe you have a gold standard level or a silver level of clients and you go in and are able to turn flags off and on and features off and on depending on their engagement with you as a client. So in that example, even for Harness we have flags that are permanent in there by nature sales can go in and turn trials on and off for particular groups of people. Maybe you've got the flags that are set aside just for particular geographies of West Coast versus East Coast. There are some great use cases for long-term flags in your system and it definitely is considered a flag if you're gonna switch it off and on on the fly. Anything you would add to that, Connor? No, and actually on the next slide I'll be actually going into some of the benefits of future flags and really how they are different than canary deployments. So yeah, I will add on to that as I go through this slide here. And so just to kind of summarize what we're talking about with a canary deployment and also a feature flag. Yeah, go ahead and progress. Canary deployment being we've deployed our new artifact. We know that artifact is working, it's accepting traffic, has low error rate, performance is acceptable if there's a problem we can roll it back, right? That's really very much from an engineering and like artifact release perspective is what canary deployments are all about where flags is really about releasing that new feature, the new functionality and code after or more importantly, separately from a deployment, right? And being able to release that feature independent of a code deploy allows us to kind of separate the is it working to are we ready to release, okay? Second part, second bullet there on the right is now we can turn on those features just for a specific subset of users in a controlled way. So after we've released the ability for our code to serve a feature, now we can say, well, maybe we want our QA team to see it first in production, enabling testing and production. Maybe wanna say we have a cohort of our customers that are beta testers, let's turn it on just for them and enable them to test this out before we roll that out to 100% of our users. So going back to the question that Karen asked of how our flags implemented config file, we talked about it at the end of the day it's a boolean value, but that boolean value doesn't have to be true or false and nothing, there's an in between. It could be true for just a subset of our users and false for the remaining. So that's really where advanced feature play techniques and solutions really come into play. So we talked about testing and production, beta testers, operation toggles too. Releasing a feature and then maybe in the future we realize, hey, you know what, this feature that's been out there for six months, it's currently pounding our database. We got to turn it off. Well, you don't wanna have to hot fix that code and redeploy code to be able to turn off the feature, just click toggle, switch off the feature. We have now a kill switch to allow us to change the behavior of our application in real time, separate from the deployment of code. And that's really where the value comes in. So quickly to just kind of summarize here when we're talking about answering the question when should I use feature flags versus when should I use canary deployments? Both techniques are really to enable progressive delivery. Both techniques reduce the risk of making changes in your application. You need to release code in a production before you enable feature and canary satisfies that. And then a canary really isn't appropriate for testing of features or running beta programs. That's what feature flags are for. So I would pause it and recommend it. In fact, you should use both, right? Use canaries for your deployment strategy, feature flags for your feature release strategy. And it is additive and it builds on to each other to really unlock the power of progressive delivery. And ultimately our goal here is to de-risk the release of code while decoupling the release of code from features. I'm seeing a bunch of questions come in, Andy. Anything you think we could get right now? Or he is asking, how do you deal with obsolete feature flags? And what he means is features that should be removed because they wouldn't be used anymore. So what do you do with the accumulation of obsolete code or obsolete flags? I think I summarized that correctly. Yep, so obsolete flag would be a flag where maybe we first create a feature and we roll it out to 10% of our users. Maybe we test out different iterations of that feature. We roll it out to that and find a winner, find the one that works the best. We roll out 50% of our users and finally we roll out to 100% of our users and it's on. That feature is enabled for everybody. So at that point, the if statement that's left in our code is always true, right? There's no reason to have that if check. There's no reason to be able to need to ever turn it off. Now, so this gets away from, well, what if we want to have a kill switch? You may want to leave it in if you eventually want to be able to turn it off. But assuming we've gotten to the point where this feature is done, it is released. It's out in the wild and we're never gonna turn it off. Now we have some dead code. We have, in this case, an obsolete feature flag and the best practice there is actually clean that up. Once you have flags that are 100% released, there should be part of your software development lifecycle that is to remove that dead code, to remove that technical debt that is that feature flag. So it is important to accept that as we adopt creating of flags as part of our software development lifecycle, removing the flags also needs to be considered at the same time. Anything to add, Andy? Yeah, that's good. That's good, I wouldn't add anything to that, perfect. Thanks for the question. I think we're gonna end on number three here, which is just touching on how feature flags give you incredible speed. It really speed up your software delivery lifecycle. So a few years ago, actually quite a few years ago when I climbed Mount Rainier in Washington, it's one of the largest mountains in Lower 48. I was taught that the quickest and easiest way to climb a vertical mountain is with something that Mountaineers called a rest step. And it involves taking just one step and then stopping and breathing. And then you take another step and you breathe and you take another step and it does this continual process. And so the result is you're constantly moving towards the summit and you actually end up moving up the mountain at a much faster consistent pace because you're not doing these quick sprints and these stops and quick sprints and these stops. And reminded me of the old turtle versus the rabbit example that we learned about when we were kids. And the same is true for feature flagging. So a couple of benefits here that the feature flags give you that help you speed up. And we've tested on it in prior slides here, but this whole idea of decoupling the deployment from the release, this really is kind of the secret sauce of feature flagging. It means that I can ship consistently dozens of times a day even more than that and then release when ready or even better yet turn that release over to a complete other team internally, another business unit so that as a developer I can continue to code and produce more features rather than having to go through the whole entire delivery system of that particular feature. So huge, huge benefit in the end result is that I get a higher velocity on the amount of features that I'm deploying because I'm able to decouple that deployment from the release. And going back to the mountain analogy is that smaller steps lead to this higher velocity. So if I make a misstep in a particular deployment of one feature, I can correct the course pretty quickly. I don't have to pull back the whole entire build and redeploy. So it's this whole idea of I can aim well and just shoot once. And again, just touched on it. I don't have to roll back all the good features along with the offending feature, the bad feature. And so the idea here is I had a client who was in North America and the engineering team created a new feature. They released it to their production server and they went to bed. The team in Australia woke up to a complete mess. They woke up to their Twitter feed blowing up and support tickets coming in and all kinds of issues because the particular feature didn't work. Thank God they had some feature flags that were able to go in, turn it off immediately and isolate that blast radius and leave it to the North American team who originally made the feature to be able to fix it. And it's always easier to fix things that you've produced and to correct that issue. But the Australian team, they were able to mitigate it within moments. And so it speeds you up. You can only just roll back the bad offending feature and you can leave the good features in production. So it eliminates this all or nothing approach, meaning either everyone gets all the new features or they get none at all. And you can roll back just a small percentage roll it out to a small percentage of the users at a time. You get this granular control, real time feedback from clients about new features and it potentially closes that loop there and creates more velocity. And also, yeah, faster feedback, I hit that. And on all these loops of being able to push for production become much shorter, increase your velocity. This is one thing that we're seeing at Harness. We used to release to production Connor, I think probably once a month, we got it down to maybe even once a week. And I just recently heard we're pushing to production over a hundred times a day now. And the way we're doing that is through having a very robust, mature CI CD pipeline. However, that's not enough. We add feature flags to the tip of that pipeline in granularly control, kind of a microwave individual features to who is being targeted, which groups, which cohorts and we're able to have that kind of micro micro control. And it's critical. It's critical that we are able to learn from testing and production. It's critical that we can target granularly and then in turn that creates a lot of velocity. And so routinely we're seeing big companies adopt this, adopt feature flags, be able to leverage it and create that velocity in their code production. And so we're excited to share this with you as well. It's the number one thing that is helping clients speed up drastically. So with that, we're going to go into a little bit of time of Q and A here. We've got a little bit of time left. Connor, I'll turn it over for you. There's a couple of questions in the hopper there. Before we go to questions, I think you want to see if we, we're going to try something new. I want to actually want to pull the attendees here. And really the ultimate question that this is about is how do we get started and how do we adopt feature flags? And what I want to know is for the folks that are attending participants of this webinar, there should be like a raise your hand feature in the Zoom view, hopefully it does. But what I'd like to hear is how many of you have some way of changing the behavior of your application in real time? So that could be a config value in a database. That could be an environment variable. That could be changing a file or something like that. Some way of modifying the behavior of the app to turn things on or off in real time. And I'm just going to give it a minute for folks to start raising their hands. And really what the reason I'm asking is because this whole notion of a feature flag isn't like completely revolutionary. It's been around forever. It's just like part of a regular software on breakfast. We have Boolean values that change the behavior of our app. That is normal. But by calling them a feature flag and putting parts of our application behind a feature flag is when we start to unlock all the benefits that Andy and I have been talking about here today. It's now not just a engineering effort that is within our code. It is now exposing this toggle to be able to be used separate from the deployment of code, right? So let me take a look at how our poll is doing out of the roughly 50 attendees we have. I'm seeing eight hands up. So what's the math there? So that's about 15% of attendees are saying they already have some kind of capability around this, right? And I guess the reason I was asking is because clearly this provides value. That's the reason why you're doing this. And so I'm glad to see that some folks are already starting with that. And really it's all about how can we operationalize that this idea of a feature flag and have it exposed outside of just a Boolean value and a database or something like that. Cool. Andy, any comments on that or thoughts? I actually do have a question. And this question has popped up several times. It's one I get frequently. Since there's only about 15% of people that are actually using feature flags, it sounds like or implementing some kind of solution. How do people get started? So what's your advice, Connor, to a new team that is looking at putting feature flags in place? How should they approach that? Is it across the whole entire org? Do they target a specific avenue to start with? What's your suggestion there? Well, yeah, so like what I mentioned earlier, it is kind of part of the software development lifecycle in that it needs to be a part of, it needs to be planned, right? Maybe you are in your sprint this week, you're working on a story, a JIRA story that is a new capability or a new feature that your developer, your customers are depending on. A story is a feature, let's just say that's the case. And all the code that you write around that to be able to change the behavior of your app or introduce a new UI experience or something like that, wrap that in an if-else statement and have that variable define if it's on or off. So in that case, your application should be able to support both on and off by the old behavior and the new behavior. Or maybe if you're introducing a brand new module in the application that you build, hide that module behind a flag, hide the menu item to get to that module, whatever is a different, a way that you can have that be more dynamic, okay? So I guess to get started, it's just whatever you're working on today, wrap it in a flag and keep it dark until it's ready to go up. I think another great rule of thumb that I'm hearing from a lot of the folks that I'm working with is they're using flags even if they may not necessarily think they use it in the future. That doesn't mean they're leaving it in there permanently, but they're putting a flag in on new code that they're producing, new features they're producing just in case they are needing that flag in the future. It's great insurance should, you know, a fending feature not work properly and then they will deprecate that flag when it's ready for GA. So that's one way to think about it. You could wrap anything in a flag, which leads me to an additional question, which is, is there anything you should not wrap in a flag, Connor? I would say things that might be considered more application configuration, like not necessarily changing the behavior of the app, but not necessarily changing the behavior of the app from how your end user experiences it, but just how the app should work, like connecting to a database or, you know, what server should it connect to to get it back in data, things like that. Things that would typically be very strictly configuration of the application. I would argue not really appropriate to be a flag, although it definitely could be used for that use case. And then I don't know, I guess I'll just leave it at that. There's really nothing else that I can think of that is not appropriate to be a flag. I think this is one area of, or a area of software development that's really fun because we're hearing about all kinds of new use cases that are coming out all the time. And so my knee jerk reaction outside of what you said, Connor, is wrap everything in a flag. As long as it's not, you know, adding to your technical debt, that you're deprecating those flags at another time, you're making sure that you're removing those, but we're seeing lots of uses for feature flags that are pretty inventive. Another question here, in case of a rollback and you just simply turn that feature off, then what kind of is the question here? Are we gonna leave that broken code in production till the next deploy? Yeah, of course not. The ability to have a kill switch on a feature or the ability to use a feature flag to turn off a feature that is part of a failed release is about reducing the duration of time it takes to rollback, right? Instead of having to quickly code up a hot fix and redeploy it or, you know, if rollbacks of the actual code take a while, the flag is about immediately turning it off. However, that is only the first step in the actual remediation of a failed release that should then kick off the actual development time that needs to be done to actually fix the code. But the benefit there, the value is now we, it's our leisure, like it's not here on fire for, you know, nine o'clock on a Friday night that we're trying to hot fix. We turn it off when it's not working and we'll get back to it on Monday because production is not being currently affected. Yeah. Another question that came in as interesting and powerful as feature flagging sounds, what are the best practices for avoiding creating a spaghetti code? So the term spaghetti code is probably referencing the fact that if as you're suggesting, Andy, if everything's a feature, by, you know, in, you know, only a matter of time we're gonna have a class or a method that has like 10 flags in it. And we have a lot of nesting of if L statements, like it can get complicated, certainly. And that's just where, like, if you are adopting feature flags in such a way that it could result in that, now you just need to have a little bit more intentional of a plan and maybe decide at what granularity flags are, right? A lot of my customers, a flag is very high level. Is this module turned on or off, right? For other, my other customers, it's every single PR is a flag, right? And we have it, so in which case there could be potentially lots of them. I would recommend to however, to avoid that is to have a flag for a story. And in which case, now you're not having millions and millions, you're having only hundreds and hundreds. And what I mean by that is a story describes the experience that the user can have. And then that way it reads well in your code because the code that you write should relate to the story. But again, James, your question about how do we avoid spaghetti code? It's a problem that's gonna play you no matter if you're using feature flags or not, right? It's just good development practices that are going to help you out there in general. Connor, another thought I have on that is if you're using feature flags to make sure you have a great tool that will give you some dependencies too. So if you create flags, you can have dependencies and which flags are relying on which flags because you can actually start to get in a situation where you're nesting flags. I was talking to a national brand, Fortune 500 company not too long ago. And one of the engineers said we have 600 flags, but they're all tied together and we didn't track the dependencies. And so we're not really sure what is connected to what. And if I flip a flag, I don't wanna take down portions of my application. That's a great question. And so I would suggest if you're struggling with spaghetti hell or having too many nesting flags, get yourself a great feature flagging tool and that will help create a lot of clarity and give you a lot of confidence when you are turning flags off and on. I just wanna step back to one of the questions we asked, that was asked earlier, like how do we get what's appropriate to be the first flag that I do? And I mentioned like it should be for a specific new feature or a specific story, but another common use case. And you guys tell me if you agree that this might happen in your organization where we're building this new feature, but we're not really quite sure how it should be work, right? We don't know necessarily should, as simply as should the button be blue or should it be green or should the suggestive cell be in line or should be a pop-up in the user's experience, right? And we can go through all this effort of developing this new feature, release it out and realize, hey, you know what? I wanna make some changes to it. Maybe we wanna have it be blue instead of green. Maybe we wanna have it be a pop-up instead of and really what this potential other use case and the first time to get started with feature flags is to test out a multivariate feature. I wanna build this feature and I wanna have two different iterations of it. And I wanna release both and I wanna then test A and B, right? Have half of my users get this experience, have half of my users get this experience or maybe test one experience for a week and then at the end of that week, switch it to the other experience. That's another important use case and now just getting back to what I was saying earlier, we're getting away from engineering practices and we're getting into product management practices where we're enabling more iterative and quick iterations of our product development and making that independent of code deployments and releases like that. So an important thing to consider. We have less than 10 minutes left and so I wanna make sure that if anyone does have questions, they should get them into the Q&A quickly. And Andy. Yeah, quick question here. Are there any tools which can show you a dashboard of all of the feature flags in their state? Absolutely. If you head over to harness.io, you can download our feature flagging tool there for free and you can play with it. It is really powerful. There are other tools on the marketplace as well but a great feature flagging tool is one that will give you fantastic insight. We leverage Looker in our dashboard to be able to slice and dice the data, hundreds of ways for you to be able to have the visibility you want as to which flags are deployed in which environment and what their current state is. So that is paramount. If you're using feature flags at you, you get a platform that will help you administer that. Yep, great. And Christopher Strickland has a question. How is the logic like for a multivariate feature flag given that the default implementation is binary? So I did state earlier that at the simplest, a feature flag is a Boolean variable, true and false. However, multivariate feature flags would be typically a string or maybe a JSON object or a number, like an integer or something like that. So feature flags don't just need to be Boolean type variables. They could be strings or complex objects like a JSON object or something like that as well. So that is how you would achieve multivariate feature flag. Maybe the actual value of the feature flag can toggle between off, the string off, inline and pop up or off, blue, green, yellow, right? And actually the actual evaluation of that string is how you change the behavior or maybe like literally set to be a CSS error, yeah, HTML class or something like that to be able to change those variations. Great. Well, I don't see any other questions just checking here. Connor, did you have any other questions that came in on your side? That is all I have. All right. Well, with that, Marisa, we're gonna head it back to you. Thank you so much for inviting us today to talk about feature flags. It is a topic we are pretty passionate about. Lots of information over at harness.io for you. Oh, one last question that snuck in under the wire here, Connor. Yeah, that was good, James. Good job. Can feature flags be self-service? In other words, at your own risk. Not sure if I'm understanding that. Let's go with the first part of that. Can feature flags be self-service? I believe what James is getting at is do we need to have a third party solution to being able to feature flags? And I think that, or could it be self-service? Can you build it yourself? Clearly, as DevOps engineers and engineers ourselves, we could probably build anything, right? You can build your own, and the question is, would you want to maintain that? It is really the question. Like I said, a lot of our customers that adopt harness previously had database toggles or environment variable toggles or something similar, and they're looking for a more enterprise-grade solution that gives features like access control, automation, things like that. But certainly, feature flags could actually be self-service or DIY. And Christopher says, thanks for the presentation. I thank you all as well. Thanks for joining everybody. Yeah, we appreciate it. Thank you so much. Awesome. Thank you so much, Connor and Andy. I, for one, learned a lot from the presentation today. So I really appreciate you guys taking your time to be here with us. And thank you so much, everybody, for attending. Just a quick reminder that this presentation will be up on the Linux Foundation's YouTube page later today. So we hope you'll join us for future webinars. Thanks so much.