 Hello, hello, there we go. Thanks, everyone. I hope we have enough space in this room. There's, you know, it's not a very big room. Thanks, everybody, for coming. I'm Todd Bart. I'm a Dynatrace. I'm a software engineer at Dynatrace and maintainer of the Open Feature project. I'm hoping today that I can kind of show everybody how, as the slide says, we can reduce risk, conquer compliance, and perfect previews with feature flags and Open Feature particularly. I guess first to start, I want you to picture this, OK? Maybe you've been in this situation before. Everything is green in our test environment. Our E2E test suite is working great. The PMs are happy. We've delivered a new feature on time. Everything is basically perfect. Everything is going well. And we're ready to go. We've actually completed our tasks ahead of time. Then suddenly, compliance issue, right? So maybe all it comes down to is a particular jurisdiction just to introduce the new compliance law. And legal, they run down the hallway at you and they say, we can't release this new feature. We think we're implicated in these new compliance laws just in this one jurisdiction. But we've got to hit pause. We can't release. The problem is, there's already been contract signed. Customers have been asking for this feature for years. So where does that put us? Well, feature flags for the rescue. Like I said before, maintainer on the Open Feature project, it's a CNCF incubating project since just a few months ago now. And really what we're doing is developing a standard for feature flagging. So yeah, we provide a vendor agnostic community-driven consensus-built API for feature flagging that works with your favorite feature flag management tool or even your in-house solution. There's a lot of companies that develop their own in-house solutions for feature flagging. I think before we can go further, though, we've got a just level set. We've got to define feature flags for everybody, because I find there's lots of different levels of organizational experience when it comes to feature flags. So this is a quote from Pete, who's on our GC, our governance board. And he says, he has this great blog on Martin Fowler's website, feature, where he defines feature toggles, feature flags. So he says feature toggles often refer to as feature flags are a powerful technique allowing teams to modify system behavior without changing code. We also have the definition from our site here, feature flags are a software development technique that allows teams to enable, disable or change the behavior of certain features or code paths in a product or service without modifying the source code. This is my personal take on feature flag maturity. It's kind of inspired by the Richardson maturity model for building REST APIs, if you're familiar with that. And I'm gonna outline these really quickly. I'm not gonna go into too much detail, but I think level zero, the first level, because we do zero indexing, obviously, is basically what most people are familiar with just as a base level. So it's like an environment variable, maybe a settings.properties file or whatever, if you're using Java, maybe it's a config map in Kubernetes, but it's static. It just maybe defines something that turns the feature on and off, and you have to redeploy your application to really take advantage of this. I'd say the next level of maturity after this is what we call dynamic configuration. And that's similar. The big difference here is that we don't need to redeploy. We can just maybe hit a REST endpoint on our application or use some kind of mounted volume or something like that to actually change the behavior of our app without redeploying or restarting it. The subsequent level, and probably where we're gonna spend most of our time today, I like to call dynamic evaluation. So this is different in that not only can you change the behavior at runtime, but it takes contextual data into account in evaluating the feature flag. So this could be HTTP request headers, user data, that kind of thing. And then the last level, as I conceive it, is what I call operationalized feature flags. So this is where the whole organization is feeding data back into the feature flagging system. You have robust telemetry around feature flagging. You can see how feature flagging is impacting performance, how new variants are impacting user experience and customer satisfaction. Like I said, I'm gonna spend most of our time today talking about dynamic evaluation, the kind of the third level there. And I'm gonna dig into that a little bit deeper. So yeah, characteristics of this, the return flag is determined dynamically. So you use targeting rules, sometimes what we call them, or flag evaluation logic to do this. And the return value is based on relevant contextual information. I just have a couple of examples up here, app version, user geolocation, org or user entitlement, the current date and time. I'm sure you can come up with like tons of different examples. The nice thing about this is it means that flag evaluation logic can be centralized and it's independent of the application. So we don't need to actually code our flag evaluation logic, like I called it, into our application at all. It doesn't need to be mixed with our domain logic. It can be extracted and hosted and defined and manipulated elsewhere. And application authors don't have to worry about that. And it's also centralized, so it's not in multiple applications. And I'm gonna probably come back to this a few times during the presentation. I want you to think in terms of deferring and delegating the flag evaluation logic to others, to later and to others, not the application author itself, themselves. Practically this means no more of this sort of code. Maybe you've written something like this before, if customer and legacy customers, or if customer in this particular geo, you don't have to write that in your applications anymore, especially multiple times. It means reduce blast radius. So if you release a new feature, you can just test it on a small set of users initially. It means that you have a robust platform for experimentation and it doesn't require a developer intervention. You've coded these flags in once and then later, marketing team, PM, maybe some kind of compliance officer, they can actually change the rules. Again, we're deferring that type of thing. Like I said, compliance agility is something else you can disable, perhaps based on geo, in accordance with legal requirements. This is our SDK. This is just one of our SDKs actually. It's our JavaScript SDK. And so this is what it looks like to actually use application context to evaluate a feature flag. Here we have a very basic feature flag. It's called My Flag, it's Boolean. And you see some code here it looks like you might see it as part of a express request handler. If you're not familiar with express, this could be like a Java servlet or something like that. And all we're doing is taking the IP from the request and the email from the session and we're embedding those in a context and then we're using that as a factor in the flag evaluation. You might be thinking this sounds a little tedious. I need to have this information everywhere my developers want to use it. We have some I think elegant solutions to that. This is what we call a transaction propagator. So what this does is allows you to kind of set this stuff once. This is an express middleware. So again, if we were going to make a comparison to Java, this might be like a request interceptor or a filter. And all you're doing here is setting the same information but it's going to be available for the entire duration of the rest of the continuation basically the rest of the call stack. If this was Java, you might be taking the same information and putting it in the thread local storage. The key to understand here is that the developer doesn't have to explicitly pass that context everywhere. You have access to this context now in the data layer in the business logic and it doesn't have to be explicitly passed because you may not want to alter your methods signatures all the way down to your database just to get an IP address or something like that into the feature flag. Okay, so we have this contextual data but what do we do with it? I mentioned before targeting. So targeting is the application of rules, user overrides, fractional evaluations, that kind of thing in flag evaluation. This is a screenshot from LaunchDarkly. It's a vendor for future flag solutions that actually works with open feature and you can see they're doing something similar to what I kind of described. You're setting a user. You're saying user is in one of these geos and we're going to return a certain variant. This is a similar UI from Flagsmith another project that supports open feature and you're doing something similar, right? So Geo matches America and New York and the last one is this is DevCycle CLI and you see again something kind of comparable, right? So we're defining a new, we have a new targeting rule here, number two and if the email contains DevCycle we're going to serve a particular variant. Like I might have mentioned before we actually have our own cloud native reference implementation as part of open feature as part of the project and we call that flag D. So it's written in Go. It's easily containerized. It can source flags from multiple sources, multiple syncs is what we call them including files, HTTP endpoints or probably most interestingly for KubeCon, Kubernetes custom resources. And it targets flags based on basically a custom DSL we've implemented on top of JSON logic. So you can basically embed that right into a custom resource. Here's an example of what that looks like. We have a flag called enable mainframe access. It's enabled, it has a couple variants. It's a very simple flag so it's just on or off and you have a targeting rule here. So similar to the examples we saw before if the user's domain ends at ingen.com then they're gonna see this variant. I'd like to take a quick second here to show you what that actually looks like how that works. This is our playground for flag D. So these are the sorts of rules that you would embed in a custom resource definition. But I'm gonna show you how this looks. I think we basically have the same example here. So you can see here's our rule. Here's our context so it has a user email. And if I just hit evaluate, yes it's true because it's targeting this user. I'm gonna go ahead and just, you know throw some other junk into this email domain and you see it evaluates to false. So you can embed all kinds of logic in these. We'll see some more of that in a bit to target specific users or services, whatever it might be. I don't have my normal level of agility without my mouse here. Apologies, thank you. Okay so evaluation depth, is that it? I kind of spent some time talking about how we do request context and context in general and how we use that in targeting rules. Is that all it comes down to? Well, in short, no, as often as the case when you take into account architectural, infrastructural, UX, compliance concerns, all of those things get a little bit hairier. So I'd like to take a little bit more time just to dive into three hairballs that you might encounter. And these, you might encounter them whether you're selecting a vendor solution or implementing your own kind of homegrown solution or using some of our cloud native stuff. So hairball one, context contents. We wanna give stakeholders maximum flexibility in building their flag rules. Like I said, remember, we wanna allow that to be deferred. So you think, well, if I just put as much information as possible in there, doesn't that give me maximum power to make decisions later? Yeah, but I think we should consider a few things. I can just throw all of this stuff in the context, right? Every HTTP header, I can throw a session ID in there. The user's battery level, I'm sure you can think of a ton of stuff. But we need to be worried about a few things. So one is PII. So some of this could be sensitive. It probably all identifies users in some way. There's also serialization costs and network costs. Some solutions are gonna store a percentage of this data or some of this data. So we should consider the storage costs. It also introduces telemetry challenges, especially with high cardinality values. So the TLDR is you don't just wanna put everything in there. You wanna be a little bit choosy. Hairball two, client side or feature side evaluation. Now caveat up front, when I'm talking about client versus server side, I'm talking about applications that actually run on end user machines. So maybe it's a mobile app, maybe it's a webpage. When it comes down to those, we kind of have a choice in terms of where we evaluate our feature flags. So some implementations evaluate everything on the server, while others evaluate them on the client. And that has a huge impact on all kinds of performance characteristics, security, availability of the system. So it's critical to kind of make the right decision here. And I have a little table here. There's kind of pros and cons to both choices. So targeting rules, obviously, they decide what feature flag is shown. If you're doing it on the client, you have to send them to the client. And that's a problem because they may contain domains. They may contain user names even. So you have to consider that. If they stay on the server, you don't have to worry about that at all. On the other hand, PII is basically the other side of the coin. If you're evaluating on the server, you have to send personal information to the server to do that. If you do it on the client, it doesn't ever have to leave the client. Latency is also really low on the client because everything is evaluated locally. There's no network hop. While on the server, obviously, you have to make a network request. This can be mitigated with caching and kind of bulk evaluation. And I'll mention that our most recent SDK, our web SDK, took a lot of time. We took a lot of time in the spec and in the implementation to really make sure that we had robust semantics to facilitate caching and caching validation around this. This is like one of the biggest challenges, I would say, in a client-side evaluation. Or a server-side evaluation, pardon me. Implementation difficulty. If you want to support multiple clients and multiple languages, you have to implement that same engine usually multiple times. At the very least, you have to create a WASM module or wherever you're going to do it, to only do it once and then ship that in multiple languages. Whereas obviously, if you do it on the server, you really only have to implement it once. The last hairball I'd like to mention is sticky pseudo-random evaluation. So one of the key use cases for feature flagging is to do experimentation. And generally, a lot of times, that involves assigning us a pseudo-random value. And I want to imagine a hypothetical really quickly here. So we have, say, two UI versions we want to test. Would we really want the user to have a different experience, page load to page load? I'd say generally not, right? That's a lot of thrash. So usually we want our assignments of flags to be sticky for a particular user. And really what that involves is to do some kind of pseudo-random assignment based on a static identifier of some kind, and then use that as the basis for a deterministic bucketing algorithm. So to determine which variant the user sees. Here's some pro tips for pseudo-random assignment. Your bucketing value is probably not going to be the same for every flag. You may want to bucket some flags based on an org ID and others based on the user. The bucketing algorithm should be based on a fast hash. It doesn't need to be like cryptographic grade. It just needs to be fast and random enough. And it's nice if it's widely implemented too. It's also optimal if the algorithm exhibits low thrash when we add a new bucket. So you can see in the last slide I had three buckets, now I've added a fourth. It's ideal if the users aren't completely reassigned. It's nice if we can add a new bucket and have a few users from one, two, and three all move into four just to reduce the thrash because you never know when you want to add another variant to your random assignment. Demo time. So what is the demo? You can go ahead and load this QR code if you want to participate. It's just a small app that's going to run on your phone. There will be a prize, there's going to be a giveaway at the end if you participate. So you can just load that QR code. You probably won't see anything except for a gray screen right now. But what's happening here is we have a small app it just has two feature flags and we're using flag D to basically read those flags and get them from a Kubernetes custom resource. And I'm going to show you some basic targeting with this application. So everybody should, does everybody see a gray screen? Cool. Yeah, it'll do that. It's a very basic app, very, very simple. So basically what we have here is we're just rendering, we have this request handler and it just renders a very simple view with a few variables here. This is our JavaScript SDK again. We just have two flags, hex color and emoji and we're just rendering those based on the feature flag settings. So the other thing that's relevant is our Kubernetes resources. I'll start from the bottom because it's probably the least special here. We have a service. It's exposing this stuff on Google Cloud. We have a deployment. A couple of things to note here. We have some open feature annotations. These are what kind of allow a flag D to work. They configure flag D with our particular feature flags. Our feature flags themselves are defined up here in this custom resource. So we have two flags, one hex color, like I said, one emoji and you see there's actually a bunch of commented out targeting rules. I'll get to those in a second. But let's just do something really simple. I'll just switch this to grin. So now we're going to return the grin variant and I'll apply this to my cluster. And now that we see a grin. So I didn't redeploy anything obviously. Kubernetes is not that quick. Basically what's happened is we updated the value and the flag D provider that's plugged into our open feature SDK saw the change and the next time your page was refreshed as you noted, we got the grin. I'm going to set this back to non, which was the default. So now let's do something a little bit more exciting. I don't know how many of you here have Macs, but if you have a larger monitor, you'll get an opportunity to kind of brag here. So I'm going to just comment out this and through the magic of YAML anchors, I can actually go and just change this targeting to a existing definition. So basically look at this targeting rule here. What we have is basically we're saying we're going to multiply the window height by the window width and then see if that's greater than 4 million, AKA 4 megapixels, and then it's going to return indigo. Otherwise, it's going to return blue. If it's 24 megapixels, sorry, yeah, it's 24 megapixels, or 2.4 megapixels, you're going to get green. What this amounts to is basically we're going to color shift your background color based on the resolution of your screen. And I'll save that and apply it. And so now if you're, yeah, most phones are probably going to be like yellow. Yeah, does anybody have like a, anybody have purple or blue? Anybody have a really nice display? It's orange. Oh, you have orange? Okay, most people have orange because that's like not a super high pixel count. I see a net book back there that's orange, yeah. So let's try something else. This is going to be a little bit of a callback to our example in the demo. Or sorry, our example in my slides here. There's probably a decent representation of people here from different countries. So let's go ahead and change this targeting values. So this is going to change our emoji and I'll go ahead and take a quick look at this. This is basically going to look at the language that's being sent with the browser. So we're grabbing this language variable and we're seeing if it has the substring FR in it and then we're returning FR, which is linked to the French flag in our configuration. I'll go ahead and save that and apply it. And you should be seeing, again, a gray background with some kind of flag on there. Does anybody have, anybody have German? Germans, hold your phone if you're in German. Okay, a few Germans. What about French? Anybody here French? Yeah, a whole bunch of French people and English. Okay, there you go. So this is obviously targeting language, but you could see you could use this exact functionality to target based on geo location. Most phones here would be in France, so I didn't exactly target geo for this demo because I don't think that would have been very diverse. Seeing as how we're probably all in the same room right now. And there's one other one I'd like to demo here before we do our little giveaway. So this is gonna randomly assign you a card deck icon basically. So hold up your phone if you have spades. Spades, okay, hold up your phone if you have clubs. Okay, hearts. And diamonds, I'm missing diamonds, yeah, okay. So it's about an even split. And if we go to the targeting for this guy in our CRD, you'll see it's basically 25, 25, 25, 25. So it's an even split between the four card hands. Now lastly, I said there'd be a prize. So what I'm going to do is take a quick look at some metrics here, bear with me, and I'm gonna see how many people we have participating. According to this, 144. I don't really believe that, but maybe it's accurate. So I'm gonna do one more change here. I'll have this giveaway targeting. And I think I want to give away about 10 t-shirts. So let's do 5.95. So if your phone turns green, you win a t-shirt. You can come to the open feature desk. Who has green? Okay, come see me after, okay? If you have green and you get an open feature t-shirt. But you can all come by the, sorry, you can all come by the booth. You don't need the green phone for that. Okay, so I have a couple more slides here. Yeah, this is basically our contact slide. So you have all our information here, social media stuff. I'd like to do a quick commercial for two upcoming things we have with open feature. We have the ContribFest. So the ContribFest, basically you can come and you can hack with us. Open feature has a bunch of SDKs in different languages. We have a bunch of cloud native components. We have a bunch of providers, for folks like DevCycle and stuff like that. If you're interested in helping out, please come. We have so many open issues, so many languages. We'd love for you to help out. And even if you just get started today, you grab an issue, you can finish later. So it's a great way to get started and open source. We also have a really cool session that's kind of gonna go a little bit beyond mine into that next level of feature flagging, what I called operationalized feature flags. And that is really by combining open feature and hotel. So there's a presentation later in the week on Friday by my colleagues, Mike and Dan. And they're gonna really talk about how you can kind of feed metrics back into your feature flag evaluations to really get a lot of really cool, robust information. Yeah, I guess that's it. We have a little bit of time for questions. Hello. Oh my God, that's loud. Love it, really cool. I like the open specification, everything. We like doing A-B testing and feature flagging and stuff on client side and on front end, that's great. But we're somewhat resistant to doing it on the back-end server side, mainly because we work in quite a regulated area where it's like essentially four eyes principle of like each change needs to have like two people look at it. And changing production behavior through a feature toggle is scary when you need to do four eyes. So of the platforms that are available, obviously you work for one, are there systems for doing like permissions on toggles to ensure that like two people have toggled it before it actually gets enacted? Like is there permission systems that essentially can or at least some sort of system to ensure that those sorts of compliance checks are done before it's actually feature flagged on? I mean, that's an excellent question. It sounds like something I believe that some of the vendors would probably have implemented. I can't speak to any particular vendor having that specific feature, but certainly like enterprise grade features like single sign on and I would guess something like that, like some kind of robust permission model. Those are the sorts of things I would expect from vendors. Yeah, I can't name one, but I would expect that there's some that have it. Good question though. I have another question. Yeah. So you talk about sticky sessions basically, how it would be implemented on the front end? Well, it's not a session in particular. Like you can use the session ID as what we call the targeting key. That's kind of the identifier that by default is what's used to bucket people, but really it could be anything. So you could generate, actually what I do in this app is generate a random number between one and a million and stuff it in local storage and that's what is sent subsequently. So you can really, you could generate a UUID. You could pipe the user session from the server and put it into the client, for example, and use that. You really just need some kind of static identifier across whatever duration you want to make sure you're bucketing sticky. That's what's necessary. Yeah, so basically we're storing ID in browser cache. Browser local storage. Maybe local storage, yeah, something like that. That could be a solution, yeah. And if the user clears it, then they're gonna lose that. But if you really wanted to make sure you could store it in the database as well, it's really a question of how sticky you need that to be, I guess. Hello, I have a question. I wonder is there any prescriber solution how to observe different feature flags? Like just to give you an example, like I want to turn on some feature flag for 10% of people, it will impact some metrics, and I want to see results like how much it changes. And if it grows to 10 feature flags, like it's obviously become much harder. So is there any ready to use solution on that problem or any best practices? I would say, I mean, one that we've advocated for and that we have kind of first class support for is open telemetry. So you could use open telemetry metrics, for example, to kind of get your head around some of that data. We actually have hooks where you can like arbitrarily add basically any kind of code during flag evaluation for each flag. And then in this case, it would be adding a metrics hook for hotel, and that would increment the counter that's then flushed to your metrics. Probably small follow up. Yeah, that's, we'll be just showing how many machines are using certain feature. What I want to understand is how to bind this data together with some metrics like, for example, I turn on some feature, my CPU goes up, and I want to see that for this machine, this CPU metric had certain feature turned it on. So is it about just for every single metric that I'm interested in, we need to attach it as a dimension like certain features or any better solution for that? Yeah, well, in that case, I would probably use open telemetry traces, which we also support. So you can basically say on this trace, we observed this feature flag value and then you could, depending on what kind of backend you have for your telemetry querying, you would be able to associate, okay, we have an increased failure rate or a reduction in performance associated with this particular feature flag variant. Yeah, that's actually the sort of thing that Mike and Dan are probably gonna talk about a little bit in their talk. Okay, thank you very much. No problem. Any other questions? Go for a cheeky second question. No problem. In terms of resiliency for like if things go, if feature flags are like evaluated and then like apps go down or the connection to the deferred place is broken. And so that's off. I'm assuming there's like safeguards and defaults. Yeah, the biggest safeguard is the fact that we force you to pass in a default value at the evaluation time. So that's like our ultimate fallback and most vendors do that as well. Like I said, we developed the specification in concert with a lot of vendors and we saw that was a very common pattern. So at the end of the day, you have the default there in code. And just realized I remembered another thing. You showed the like a Kubernetes resource model version of the feature flagging, but I'm guessing if that would be available through UIs and then through the UI, does that update the CRM? Well, so the Kubernetes resources I showed, that's part of our reference implementation for open feature. There's no UI to drive it. This is more of a solution targeted like towards people who are really interested in doing GitOps and that kind of thing. So there's no UI in our case. I mean, if you use Argo, you kind of have a UI because it's a custom resource there, but it's one of those solutions that we kind of recommend like you combine other solutions in the open source ecosystem to kind of get where you want to be. Cool, thank you. Cool, thank you everyone.