 So, thank you very much for joining. My name is Alex Jones, and I'm going to be talking to you today a little bit about Open Feature. Just before we go into that, I want to set the scene of who I am and why I'm talking to you about this today. I'm on the governing board of Open Feature, but I didn't start there. I started as a contributor, as often people do. It was just something that I don't have anything to do in my work with feature flagging, but I find it interesting. So, aside from that, I work as an engineering director at Canonical, and I also contribute to the CNCF in a small way as well. I would really encourage people to try and think about how you can contribute back to the CNCF, especially after the great talks we had from folks this morning and throughout the week. So, feature flagging, it's an interesting thing. So, let's get into it and talk about why we should care about it, what's Open Feature, and even more interestingly, what sort of experiments can we perform. So, this is kind of a looser gender of what we're going to do today. We've got what is Open Feature, how can I use it, and how can I perform experiments. And then maybe you'll have some ideas around how you might integrate it into your own workflows and your own organizations. So, let's just quickly recap on what is a feature flag, right? I think many of us have a view in our minds of, yeah, it's kind of a toggle thing that goes on and off, and I really like the words that a colleague of mine, Pete Hodgson, who also works with the Open Feature project, put together. If you haven't seen Martin Fowler's blog, it's excellent, but this effectively encapsulates how I would think of a feature flag, right? It's a way of being able to modify a system dynamically without causing any impact to the code. Of course, in reality, feature flagging is pretty much changing the color of a website or a drop down, a modal, or maybe a new piece of functionality comes out. So, I think feature flagging has been sort of typecast as something you see only in the web browser. When the reality is anywhere, you can put a feature flag. So, where are the gaps and why is something like Open Feature needed? Well, the gaps are that first and foremost, it's assumed to not be cloud native, right? It's assumed to be something purely in the web browser. There isn't really a singular open source API, right? There aren't really well known SDKs and free and open things that you can build feature flags with in a ubiquitous way. There are some great efforts from many communities, but there's not a standard across those communities. And because there's no standard, it makes it hard to be extensible. You know, if I decide to use one provider one day and one another, I have to rewrite my feature flags, I have to maybe take my middleware layer out and do something else with it. So, it becomes really tough to really put feature flags in for long duration. And that's also why people get scared about putting them in infrastructure. If you're running a controller in Kubernetes, I don't want to put a feature flag provider in SDK that we might not use in six months' time. So, I want to talk to you about Open Feature. Open Feature is a really interesting project because I think it tries to solve a lot of those things that I just mentioned, and I think it does it in a really good way. First and foremost, it was created to support the community and to grow that community. So, what we're trying to do here is take those existing feature flag providers and make them feel included. The ambition here was not to go ahead and say, here's a new standard, you all got to use this. This is the cool thing and you're all wrong. It was the exact opposite. It was to bring those folks in who were experts at writing feature flags. How do you do things like targeted feature flags? How do you do things like evaluation that's proportional? And all those really interesting things that have been developed over the years with lots and lots of research into feature flagging, but how do we also bring cloud native to them? We have some great providers of feature flags out there, but they don't necessarily have access to the infrastructure community. The folks who are building and running Kubernetes, the folks who are building cloud native projects. So, there are a lot of ambitions to bring those two groups together. It started last year. It was around about, I think, Valencia. We had our first meetup and it was just a group of eight to 10 of us. Some from companies who have motivated to build feature flags. Some from companies who are in observability, but everybody there was very passionate that this was sort of a problem space that had a lot of potential resolutions and things we could do to add value there. In terms of who's actually using it today, we have commercial adopters, big names who have got open feature in their production environments and that's in the space of a year, which is pretty fantastic, I think. So, what I'm trying to encapsulate in this tiny, tiny little piece of JSON here, I apologize, is that the feature flag kind of world revolves around this idea of a spec. Open feature has an API and has an SDK and has providers, but fundamentally, all of that pivots on this idea of having a standard. When you have a standard, it means that you can build SDKs towards it. You can build providers towards it. And the standard is really important for us to start to feel like we can build out features ubiquitously across these providers. When I said that it had SDKs, we went ahead and wrote a bunch of SDKs straight out of the box. So, you've got, you know, you're really popular languages there, like Go, C Sharp, et cetera, et cetera. And even more recently, I just contributed the Rust SDK, but I dare somebody to try using that. We've also got a provider library. So, what's interesting is that you see some pretty well-known logos there, right, if you're in the feature flagging game, and they will come to the table and say, hey, okay, we'll build an SDK. We'll try and, you know, come along with you. And on that journey, what they've found is that they've learned some stuff from each other. They've discovered a new community. And I think the really exciting part is they're also becoming more involved with the cloud-native community, as I'll go on to talk to you about a little bit later. So, how does this all play into Kubernetes and what's the overlap there between writing SDKs and providers and Kubernetes? Well, let's sort of take a thought about where a feature flag is used. And I said that they're primarily in browsers, but that's an arbitrary reason for a whole slew of points, you know. I said it could be everything from people being fearful of putting feature flags at a lower level, not having the API necessarily, and also the accessibility, because if you think about how feature flags have traditionally worked, a company would have a SaaS, and you would hit the SaaS API endpoint that would tell you where the feature was turned on or off, right? So, all those things don't really seem compatible. So, we kind of need to have a way of taking feature flags into the OS, right? We need to have a way of taking feature flags into Kubernetes, and we need to have a way of taking feature flags into the cloud, right? So, arbitrarily, wherever I want to have some sort of control flow where I'm doing some sort of testing, feature flags need to exist. So, this is where come sort of last March, maybe a little bit later, actually, I think I started thinking about this idea of how do we take us something like a system deservice, and we wrap it into some sort of feature provider, and this was the initial incarnation of that, where I thought, well, okay, you can have something that runs, it reads a JSON file, it has some sort of IPC, you know, and I'm glad to say it's evolved since this kind of napkin diagram to be a little bit more of a sturdy service called FlagD. I've got QR codes throughout, so if you do want to snap this, feel free to check out the project. They're all on the GitHub for open feature as well. FlagD was this idea of, well, if we ran feature flags as a native process rather in my web browser, I'm suddenly providing accessibility for feature flags for a whole bunch of places I didn't have previously, right? I could run it in Ubuntu and read feature flags locally off the disk, and I thought about this and I experimented with it locally, and I thought, well, what actually is this useful for? And soon enough, I was turning kernel modules on and off through feature flags, which probably wasn't a great idea. So, FlagD fundamentally is pretty simple. I always run it as a system deservice, I make and then install it, but you could just run it as a Golang process, and you can see here, it's got a nice little HP API. It also supports GRPC. It originally also had AF Unix socket support, but nobody apart from me used that, so we got rid of that. But fundamentally, the idea is this Unix philosophy of keeping a single job for it, and it just reads afar and it effectively lets you query feature flags. As the spec was evolving, some of the behaviors also evolved with FlagD. So now today, it's a bit more sophisticated. It supports streaming of feature flags. It supports eventing and triggers and all sorts of other good stuff that you can't really see well represented here, but I'm glad to say that it's actually starting to innovate on that original idea, and the collaborators now have done a really great job of taking it even further. But that wasn't really enough. We had FlagD, and so I was really tickled to say, well, how can I put this in Kubernetes as I try to put everything in Kubernetes? And what it was interesting to me was this idea of, well, what happens when we have FlagD running as a sidecar? What then does that mean? So the workload could be, I say, a Golang or a Python application with FlagD next to it. That means everything's always on localhost, so I could do gRPCs at localhost, and almost like proxy-ing, I suddenly now have access to feature flags from my container. When we took the idea a bit further, it became evident that you could run a controller or an operator that would effectively link up your workload to some feature flag configuration. And the beauty of this is that configuration, as it's a CRD here, that could also be in the cloud, right? If you've got web access, that could be your flagger, your flagsmith, your launch darklays, etc., and you could hit it from your container if you had network access. If you didn't, you could use a feature flag configuration. The beauty of this as well is because you're using the client-go-watch API, you've got real-time flagging very, very quickly. So the applications start to grow. Everything from a kube-builder Golang application where you want to put a feature flag in, on a particular reconciled loop, all the way up to, say, a web server where you want to have something like on-boy proxy and you just sort of extended a bit by saying, hey, I want to have certain web requests do x, y, z. So it became very, very exciting. To give you an idea of what these actually look like, because I think that's always important in these kind of sessions, on the left is a super simple implementation of what a feature flag custom resource definition looks like. So you can see that the default variant is bar and then the variants it lists to bar and then that's the response of bar. So it's pretty accessible. It's not too difficult to get going. And on the right, how do you actually implement it? Well, if you see the annotations over there enabled true and then it actually links to the resource that it's using in the cluster. That wasn't entirely intentional because for any sort of adoption, you need to make that dx really, really easy to get on board with. And as I said with flag D, open feature operators also evolved. It now supports proxy and use cases. It supports use cases for organizations who don't necessarily have network connectivity outside of the cluster. And so we're trying to make it effectively a provider as well. So you don't necessarily have to go off to the web. So just to recap on these two technologies before we get into this idea of experimentation, we have flag D, which is the injected sidecar. It could also just be run as a process on Ubuntu or wherever you like to put it. And we have open feature operator, which is the idea of how do we inject it into your workload. And together they have a feature flag file, which is a custom resource. It's a provider in the web, or it's something on the local disk, your choice. So with all this said, where does this give us opportunity? Like what can we do with this? Well, when I look at feature flagging, I don't think it's just for product managers to test whether something works. Yes or no, because typically in the Kubernetes world, you don't really have a product manager testing the reconcile loop of a controller function, or you don't have a product manager who wants to help you with that on-vib proxy routing decision. So I thought, well, what could be some experiments that are useful to think about? Before we get into that, you don't really have a talk on this. You have some sort of arbitrary quote, but I just wanted to have Milhouse in there. But it is an experiment. And so what I'm trying to say here is it's not testing. And I think that's an important distinction is that testing is part of that, but experiment is proving a hypothesis. And so I think what this is trying to enable is to say that I want to be very intentional as we go through this and say that there are companies who are trying to prove business value of Kubernetes, business value of placement of pods. They're trying to understand things like financial cost, whereas testing, we're looking in both fuzzy testing and the intentional testing we do in units and integration testing to make sure things functionally work. And so as we go through this, I thought, well, what are some of the things that we really want to figure that out? Because often we do actually test our infrastructure. We do actually test our code. We don't usually test them together at the same time. We don't usually test our pod placement strategy or our HPAs or our horizontal pod auto scaling strategy with new code all running at the same time across a bunch of different microservices because it's too hard to orchestrate in a deterministic way. It becomes chaos. And that's another thing. We chaos test, but it's very difficult to replicate the chaos testing because of the very nature of it. And then microservice tests don't usually think about the active infrastructure changes, right? If you're trying to test a car, you're not usually doing it in the middle of a war zone because there are too many extraneous variables going on that you can't really repeat in any meaningful way. And so really, feature flagging, if you think about it, lets you turn on and off capabilities now across the stack in different ways simultaneously. So it's almost like being a conductor of an orchestra and you can start to bring that symphony to life over and over. So we take a very simple example of this. This is a web, like an SDK or a web server, like engine X or something, that you can extend with flag D. In fact, I think I've used some sort of like flask-esque example. Like, you know, take candle up here, right? Okay, so we've got the take candle API. And you know, we're getting a feature flag value and we're returning the JSON of yes or no, right? So super simple. But what you can do with open feature is you can actually do a targeting key based on the email address, right? So you can use targeting key on any kind of arbitrary parameter or value that comes in and you can then respond, right? And what's super interesting is this is one of the first experiments I did aside from the kernel module one that was actually running in a container and instantly you can see the application of that extends humongously because you can start doing conditional routing. Now, just a note on that. Service mesh has been able to do stuff like A-B routing and all that sort of jazz for a long time, but we're actually able to do this now in the application code. And I think that what's really interesting about that as well is what it then lends itself to further on. You could start saying, well, okay, my microservices have the ability now to have specific types of logic rooted through them based on the feature flag configuration. And when you start to think a little bit further along, you can start to imagine that you can pass them through to other services. And so therefore the feature flag configuration object, it's changing its purpose, right? This is almost becoming a state for the configuration of the microservices on the underlying cluster beneath it. And so I think as we start to follow that pattern along, we can do some interesting things. So the next thing that I did was to say, okay, well, I've got my feature flag configuration. Why don't I rewrite it so that my feature flags are actually different services and I can do proportional weighting so I can start to root things here and there? Again, it's not a new concept, but we're bringing it to a new layer in a way that we can start to aggregate out. And also remember, because you have a centralized feature flag configuration, you can start to use that across services as that orchestra conductor. So as an example, I thought, well, I'll build an operator for this demo. You know, I wanted to show something that was interesting. I wanted to sort of bring a real world example. So, you know, typically in clusters, when you do your provisioning, we kind of let Kubernetes scheduler decide where things should go. Maybe you set a topology key. Maybe you have some sort of anti affinity or affinity. It's not that typical to do dynamic affinity changes of workloads. So I made a quick operator to do that so that this layout operator can actually move a stateful set and deployment pods all over the place. And it can do some simple allocation. But what's interesting is when you couple that very simple concept with the web server concept, you've suddenly got two levers you can pull to perform some interesting experiments. Because as traffic comes through, you can start to think about how that works. So just as a very quick example of what that does, we see at the top we've got the placement application demo and dynamically it shifts workloads around here and though it eventually ends up on the same worker. Again, as I say, these things aren't necessarily new concepts. What we're doing is trying to bring them all together in a reproducible way to make a toolkit so that you can start doing kind of fun experiments. Again, what does that actually look like? So on the left, we just zoned in a little bit to the feature flag spec. That bit there is the layout flag. So that's effectively saying I'm either going to use the stacked or the balanced layout. This is what I called my placement operator modes of operation. I have stacked or balanced. Again, just trying to sort of have a more dynamic way of scheduling. And on the right, this is the targeting key. So I've described that based on certain email addresses coming through the initial evaluation on the web server, we should be returning different services. And so you can see that the evaluation is weighted. That weighted evaluation has given me a 50-50. And that's built into the spec. Again, coming back to the spec, you could take that spec out of Kubernetes, put it in your web browser, and it would still work. Of course, you'd have to figure out what service A and B meant in the terms of your web browser. But again, the spec is ubiquitous, which makes it really powerful. So as we started to put these pieces together, you start to see that you get this real flexibility to almost dynamically map the topology of your cluster and change it and mutate it in a way that is both... You're able to put it into something like Git. And so the feature configuration, you can save that. So you know the different experiments you can run. But also, as I said, you can link multiple services to the same configuration flags, which makes it really powerful. I like to think as well that when we're testing things like this layout topology, there's always a business reason of why we're doing this and what we're trying to prove out, coming back to that idea of hypothesis. So in this experiment, I was trying to show, well, you know what? I think that it's probably better business values to run all pods on the same node and to run that node near max utilization than to run three nodes at low utilization with the pods distributed among them. And of course, you can do that in a really easy way by splitting off traffic. Also, when you combine this with additional tooling and observability, you can say that, hey, this is the feature flag configuration and these are the observability results. So you're creating effect of the catalog of data for that experiment at that time. But how do we take it further? So this was only the start of that experiment. We wanted to take it to a place where we could actually do some more repeatable experiments. And so the place that we're getting it now is when we actually set the feature flag configuration up and we set the placement operator and the web server up and we start pushing requests through with something like Vegeta, which is a popular Golang, like a test slinger kind of service, we can start seeing that we get data back from observability, right? So when your observability systems are there in the cluster, like kubestate metrics, and you know the feature flag configuration, you can push that into a kubeflow pipeline and I wanted to give a quick shout out to kubeflow because there was a great keynote earlier on it. Those ML pipelines then can start training the models and we're starting to see, you know, with all this hype around LLM models, we're gonna have to start being able to see these hyperspecific domain models for things like experimentation within Kubernetes. Then what we do is we query the inference API and we can make PRs against the repository based on what it thinks will be the next experiment or the next place to take that. So what's really the value of all this? And I wanted to recap because I've discussed a whole bunch of stuff there. The impact of blending features together across a variety of services has been made simpler. It's been made something you can repeat by using feature flags as a mechanism for switching services to certain states. That's effectively the high-level takeaway there. It also means that we can change the inputs, we can mix the inputs across services. I said that it's not exactly testing, it's more to do with experimentation. We can postulate that if we have certain services that are going, say, all to a certain node, that the outcome of our observability, which is really effectively our experiment output, will be a certain way. And that these experiments can go across boundaries, right? You could be running the feature flagging mechanisms within, say, an Ubuntu host OS, also in a pod and a service. Think about, like, CNI testing, right? If you've got a container network interface and you might want to switch out something on the host OS and then you can then prove out certain behaviors of that of doing so. If you're interested in this stuff and if I tickle your curiosity at all, there is a really nice little killer code thing. You don't even need to set anything up on your local computer. The QR code is there, and that will create you a beautiful little environment. You can play around with it. You can create a deployment. You can create a feature flag. You can, you know, learn all about it. And as I said, there's also the open feature material, which I'll link just at the end. I would just say that the open feature operator is just one example of how you can get feature flags into Kubernetes. My example of experiments is just one way that you might think about it. I'm sure you have much better ideas on how you can do experimentation. What gets me excited is that open feature, which probably was only of a thought of to be a ubiquitous API and replacement for the disparate different feature flag providers, has so much more application than we're seeing or that we envisioned initially. And I'm excited to see where it goes. So if you found this interesting, please do visit openfeature.dev. It's a really great community. I represent a very small portion that gets excited about Kubernetes and kernel module feature flagging. There are some great folks who are more interested in web development and everything in between. And again, thank you so much. I appreciate your time.