 Hello, thank you for coming and listening to my talk today. This talk is called GitOps is a means, where is the end? And my name is George. I snuck this slide in real quick to maybe give you a hint as to where I'm going a bit more with this. This talk is not so much about my frustrations with GitOps but maybe the Git part in GitOps. And just maybe where the GitOps experience is lacking a little, but we'll get to that later on. So just to introduce us a bit more, this is the Flipped team. I'm a member of this team. Mark, our founder created an open source feature flag solution back in 2019 called Flipped. And in 2022 founded a company around it and that company consists of just us three, myself, Mark and Yofi. Yeah, so that's Flipped. But today I wanna take you on a little bit of a journey. I wanna rewind the cockpit, go back to a past role of mine where we had a complex system that embraced GitOps practices to deliver software. I wanna talk you a bit through that so that I can sort of give some context as to some of my experiences there, the influence some of the things we've been doing since then. And then taking on to talk about what we did and what we've been doing at Flipped, the open source feature back solution, what we've done with those GitOps experiences to bring feature flags in line in that world. And then I wanna briefly touch on some more problems that we've still not managed to quite solve yet. And I think that broadly apply to GitOps as well as well. And then end with a little experiment that we've built, open source experiment that tries to address part of the problem, part of the equation. Great, so as I said, wanna start with past experience of mine prior to working full-time on Flipped. I worked at a company with Yofi called Influx and we built a time series database that was deployed onto many Kubernetes clusters. And this platform probably looks quite familiar to those that embrace GitOps. We had a number of multiple Kubernetes clusters across three major cloud providers. And in order to configure those, we used Argo CD to watch a single configuration repository and apply the configuration there to these various instances of Kubernetes across these cloud providers. And we also had a separate application repository where the engineers were building and writing software and writing code. And we were building container images pushing those and then the image digests that came out of that process were automatically written to our configuration repository. And that's how application code changes flowed all the way through to be deployed onto our Kubernetes cluster. So this is probably not that surprising. We had developers who contributed primarily the application repository. We had developers who contributed primarily to the configuration repository and we had some developers in between who did a bit of both and sort of straddle both worlds. We also had automation and code that was contributing to these repositories too. So we had humans and machines writing commits to these repositories. So hopefully that's not very surprising architecture. It's a very high level. There are a lot more details than just this. But one thing to say about, there were certain members, teams, particularly ones focused on application development that weren't particularly, they didn't particularly enjoy having to contribute to the configuration repository. This configuration repository, as you might imagine, got quite big, complex and deep and sprawling. We had all sorts of variations in configuration because of the different environments that we deployed to and the number of clusters, the different phases of these clusters, staging, production and so on. And that made this repository like, it had a quite steep learning curve. And if you wanted to get something simple done, it was kind of an off-putting experience. And this kind of had a couple side effects. If you're an application developer and you just want to get something done, you've got a lot to think about, a lot to build and you want to change the log level in an environment or something like that, you don't want to necessarily be a painful experience. And so because it was so painful, you had to clone a repository nowhere to find the file that you needed to change, make multiple files, make those edits, commit, add, push, open a PR, get something to approve it, get it merged. That was all kind of a high barrier for some relatively small changes. So sometimes you get some of these engineers throwing that work over the wall, just throwing it over to our team with an issue and hoping it gets solved for them or worse, sometimes they try and work around this problem by doing things like baking the behavior they wanted into their application based on the environment it was deployed to. So they would write code to be like, oh, if I'm in this environment, then behave this way, which made the software that they're writing hard to test outside of those environments because you couldn't change these behaviors by configuration anymore. And that's something that we didn't want them to do, but when we challenged that, it would be like, well, it's too hard to change an environment variable. So this is what I do. So that was one pain that I observed and it felt like git was kind of the only way to solve it that we were offering them and we want something better. Now, additionally to this, we also had another thing, we had feature flags. Now, the feature flag system we used was this thing outside of our GitHub pipeline and they were super useful because they allowed developers to decouple their deployments from their releases, right? Now developers can safely change some behavior, protect it with a feature flag, get the code deployed up to all these environments and then they could progressively enable that feature environment per environment basis thanks to this feature flag service, which was great and perfect. But my long criticism of this is that feature flags are just configuration like anything else, right? They change the behavior of your application at runtime, they reconfigure it to behave differently. Now, when something goes wrong, which things do constantly, you're going to want to understand both the current state of your system and how that state of your system has changed, right? And that's one of the major benefits of GitOps is we have this unusable history of how your system's configured. But if you use a traditional feature flag system that sits outside of this flow, you've now kind of subverted that benefit and you have to correlate both what changed in Git with what changed in your feature flag system to try and debug and understand which thing caused the effects that you're observing. And so that was something I always wished we could bring feature flags back in through Git and so that Git continued to provide the full picture of how the system looked. And so this was sort of some of the things I took when I left Influx back in 2022 with me to flip. Feature flags, how can we bring them in line with GitOps? And GitOps in general, like not everyone wants to contribute to make configuration changes through Git. Is that the only interface we have adding, committing, pushing for requests? Can we make something better? And also my team who looked after the configuration with Audrey, we spent a lot of time trying to improve these experiences. And along the way, we spent a lot of time fiddling with the tedious parts of Git and the Git protocols to do that. It'd be nice if there was some tools to help kind of encapsulate that behavior and take it away so that more platform-focused teams can think about just the behaviors they want to manage. So I want to take that experience and fast forward to Flip. Last year, I joined Mark. The founder reached out to me, we had worked together back in 2016 to 2019. I watched him start this open source project and then he reached out and said, hey, I want to do this full-time, do you want to come and join me? He explained to me that he wants Flips to be a developer-focused feature-flag tool to help primarily focus on the problem of decoupling your releases from deployments. So that really resonated with me. And I was like, well, I've had all these problems with my feature-flag experience too and I'd love to bring those. And so we talked about that. And one of the first things we've done this last year is try to tackle Github's for Flags. So these numbers here are the versions of Flip that we shipped over the last year. Back in June, we shipped 1.23 and we launched our first experimental support for a Github's experience in Flip. So you can now, at this point, with some experimental feature flags enabled, you can store your feature-flag configuration as files in either a local directory or in a Git repository. And Flip will just follow the state of your remote repository. We made this generally available back in, I think, August in 1.25. And at the same time, an awesome contributor added additional support to put these configuration files in S3 and so S3 can help to scale this experience with Flip. And then only this month, November, when I recorded this, we added OCI support. So just like with S3, you can use any OCI-compatible registry as the distribution mechanism for big-to-fag state, which again helps with the experience of managing this state and scaling a large sort of set of flipped instances sourcing their feature-flag state. So we've taken a bite out of this problem. Now you've got the delivery mechanism for feature flags from declarative files in a number of sort of Git or container adjacent sort of backends. But then comes the next problem. This is a problem we kind of expected. And that is this product doesn't want to learn Git. And what I mean by that is that feature flags are often not just for developers, right? Not just consumed and used and toggled by developers or personas. Feature flags are often given to, you know, maybe product or project management or other functions of the organization to change behavior maybe on a per-customer basis. They have a broader set than just the deploying and releasing. And these audiences don't necessarily have in the first place knowledge of how to clone a repository and make a change in a file in order to do that. And that's a high bar to expect from this audience. And I think additionally, teams who want to commit these feature flag files, particularly alongside code even in the repositories probably don't want to give the full sort of scope of the repository access to these audiences because there's too much in there for that audience to need to have to worry or think about. So this is not a great experience. Folks don't wanna, these folks don't wanna have to clone, add, commit, push and so on. But this sort of started to seem familiar to me as a problem as in when people brought it up. I feel like I've heard this before. Going back to when we had a configuration repository and our own application developers didn't really wanna have to clone branch, find, edit, add, commit, push, open a PR just to change an environment variable. We're now not only do we have this problem exists, but it's true for people and product who wanna enable a feature flag. I think I can think of times where I didn't wanna have to go through this whole process just to change a log level. And I think again, it broadly applies to a whole host of common configuration like changes that we make, like even maybe adding a whole new service. These things that are quite templated, repeatable processes, why should we always have to go through this sort of set of steps just to get the same result? And I think this is really a platform engineering problem. To me, this makes me think of platform engineering when I hear these problems because that's the goal of platform engineers is to build these self-service internal developer platforms that provide a nice and intuitive experience for application developers to get what they need. And yeah, this all sort of feels like the same problem. And instead of just getting on with the problem ahead of me, which is getting feature flags to work with our UI and supporting a UI over just flipped, instead got a bit distracted at the maybe the next level of abstraction around this problem and started thinking about the general space of making a click-ops experience for GitOps repositories and this more broader platform engineering problem, I think, with GitOps at the core. So that takes me on to this last bit where I wanna talk a bit about a solution that we've experimented with, which looks at maybe slightly broader picture than just our feature flag system and that GitOps in general. So I should illustrate that I think the GitOps space and the tooling that has been built is absolutely excellent, particularly for taking a configuration repository with a bunch of desired state and acting on some real world systems to make them match the desired state. I think tools like Argo and Flux, even things like Terraform, which again, I think are tools for taking a desired state and making the changes that are necessary. Those are all excellent and they've allowed us to commit our code through things like GitHub and GitLab and have that experience. I think they continue to get better and they're also already quite excellent. But I think there is a bit of a gap when it comes to maybe helping to improve the experience of automating the creation of these code changes because I want my cake and I wanna eat it. I want, what I wanna see is a kind of click-ops experience but over the GitOps pipelines. So I wanna bring back that sort of Heroku-like delight, which I think every sort of internal platform team is probably trying to achieve that kind of, when I make a configuration change, like adding an environment variable or changing a log level. What I then wanna see is that actually materialize as a code change in the relevant space in my repository. I want that to be put in a pull request. I want the relevant people assigned to that pull request. Maybe it goes through CI. Maybe it automatically gets put in an merge queue. I want all these kinds of, I want the capabilities to build these kinds of experiences. And as maybe like an additional bonus, it would be nice if this solution was less focused on the, a specific way of organizing your GitOps repo. If it was extensible and could manage a whole variety of these great tools that already exist, like Argo, like Flux, like Terraform, there are so many ways that teams construct and organize their configuration repositories today. So many languages that consume, I mean, okay, there's Terraform for configuring cloud infrastructure, but then there's also things like Pulumi, there's CDK, we've got YAML for Kubernetes, but we've got JSON, we've got JSON app, we've got Q, we've got customize. There's just so many tools and they're all great and everyone wants to get what they want from each of these tools. So it would be nice if there potentially exists a solution that doesn't dictate that part of the equation that just helps you to build the experience you want over these existing project structures and tooling. So spoiler alert, I don't have, I haven't got all the answers to this, but we did, we have tried a little experiment to look at what something might, what one piece of this puzzle might look like to help solve that problem. And this project that we devised is called Cup. You can find it here at github.com.slip-io-cup. And we kind of describe it as an API server for Git that helps you to present an API over your Git repository that can change the contents of your Git repository based on some sort of desired state. One of the sort of best analogy I can come up with is basically to sort of draw some similarities to Kubernetes. So this is a very simple illustration of what Kubernetes does, right? On the left hand side, you've got Kubernetes' own API server and its own state. Kubernetes stores a desired state of the world and then it's constantly reconciling the desired state of the world with the actual state of a cluster. And it's making changes to that cluster to reflect and match the desired state. I think similarly, tools like Argo CD and Flux CD do the same thing, but this time in front of Kubernetes, they take the desired state of the configuration in a Git repository and they make sure that the actual state of the Kubernetes API matches that. So taking that, you could say that Cup goes in front of the Git repository and it presents an API which you can pass your desired state of your Git repository and then it's Cup's job to reconcile that with the actual contents in your Git repository. And when desired state changes, it can actually turn those changes into PRs for requests to make the physical changes in the Git repository. So you can imagine, you can stick all these things together and you've just got reconciliation loops all the way down. You actually have a little labs repo which demonstrates an end-to-end Cup with Argo in a Kubernetes cluster. So this can be created, but yeah, I just wanted to sort of draw this little diagram of that and sort of make these connections between how Cup works and how similar it is to Kubernetes because we really did steal a lot of the concepts of Kubernetes when writing this first iteration of Cup. Cup actually has the concept of custom resource definitions if you're familiar with those in Kubernetes and it has the concept of controllers. Cup lets you define these things. These are the extension points so that you can write your own custom behavior for Cup so that you can integrate the project structure layout of your configuration repository to meet your needs so that you can leverage the tools that you need whether that's JSON or Q or just YAML and HOME or whatever it is that you're using and build your own controllers to manipulate your repo content structure to meet the needs of whatever you're solving. So it's all very conceptual but this is kind of like a little flow of how sort of Cup handles a request to set the state of a resource in it. So here we have this payload that's very Kubernetes looking being put into the Cup API server. This payload represents a service, maybe some conceptual service in your organization and it has a very sort of constrained set of three parameters in its spec. It has an image, maybe some replicas and some environment variables. And it's Cup's job to find the right controller for that resource type, invoke it with the contents of Git at some tract branch, maybe it's the main branch. Cup will do that, it will clone the repository, it will find the right controller, it will call that controller with the new desired state of the world. And it's the controller's job to look at Git at that particular reference, that particular head commit, look at the contents, figure out where this particular instance of the service lives currently or if it doesn't exist, find a place for it. And then to materialize and to write out the changes to that repository that are necessary to make that instance either exist or be updated. And then Cup intercepts those changes made to the file system, made to the repository and it bundles them up into a new branch. It pushes that branch out, adds commits to changes, pushes the branch, opens up a pull request on some configured source control like GitHub. And that's kind of like very high level flow that sort of goes through. You start with this new desired state of some high level concept like a service and then the controller turns that into a change and Cup opens up a pull request for you. And so we built a tool that can do that and we use, we have a sort of very general purpose controller that comes out of the box that sort of uses Go templates to help you sort of with a low code but rigid sort of quick get started experience with Cup. But we also support a more general Wasm interface for writing controllers that allows you to really broadly extend the Cup's capabilities and the kinds of repository content structures and tooling that you might choose to use with Cup. And we've actually written an implementation of a controller with Wasm for a flip zone configuration format, which is its own custom format. And I have a quick demo of it here. So this is very quick. This is a flipped instance. And here we're using the Cup CLI to interface with a Cup D instance, which is pointed at this flipped instance, the flip state. And here we're gonna edit the flag that we saw. Here Cup edit is like cuba-cutter edit. If you're familiar with that, it quickly pulls down the reference. We're changing the flag to enabled and Cup has gone and turned that change into a pull request for us in the relevant file in the right place. So once we approve that, we merge that as I said before. Flip these days can support sourcing the state straight from Git. So here we're seeing a flag has been enabled because we've merged it straight into the source repository. So this is a real, I was a real whirlwind video of the experience, but here we've sort of got an end-to-end, in this case, CLI experience, Cup ships with a CLI for interfacing with it. And just sort of shows you how now in order to turn a feature flag on, I don't need to know which particular file I have to look in to edit and add and commit and push those changes, Cup can simplify that process through the flipped controller that we have configured here. So that's it. That's my super whirlwind tour of Cup. If you're interested, you can come and read the docs at cup.flipped.io, or you can come and find the repo at github.com.flip.io.cup. It's very early experiment. I'm sure it's got all sorts of rough edges, but hopefully that kind of sparks some curiosity in you to think about how you might be able to put an API in front of your Git repo and how that might then power further upstream tools like UIs and CLIs and so on. Big shout out to other projects, I think are trying to bring a ClickOps experience to GitOps. Gimlet.io is an excellent example of that. I know there's critics, I think, is sort of helping towards that. It's really exciting to see people focusing and feeling this pain and thinking about this problem. I'd like to see more tools come about for helping template out some of these experiences. Yeah, so we can get away from just pull requests and having to know repository structures and things like that as a barrier to entry for more application developers. Thank you for taking the time to listen to this. As I said, my name is George. You can find me on Twitter here. You can find me on GitHub here. Or you can come and check out our site, flip.io. We also have a Discord at discord.flip.io. Come and chat to us. You wanna learn more about flipped feature flags or you wanna talk about CUP. We'd love to hear from you. Cool, thanks for listening.