 Well, hello and welcome to another Dev Nation Live. We are excited to have all you guys here today. We have another great topic with another great presenter, Edson Yanaga, who's gonna walk us through feature toggles and hypothesis-driven development. If you've not seen this topic before, it's gonna be awesome. Edson is one of our best speakers here at Red Hat and he's gonna deep dive in this topic. So remember, use the Q and A tab or the chat tab to ask us questions. I'll try to respond in real time where I can, but we'll hold questions towards the end and I'll verbally ask them of Edson when he's ready to go. Okay, when he's finished his formal presentation. We have about 30 minutes to spend together today. So feel free to chat us up in the chat tab and Edson, it's time for you to get going. Hello, always a pleasure to be here. So as we're said today, we're gonna discuss a bit about feature toggles and I have a challenge. I have a lot of demos to show and I also have to try to speak slower than usual. So let me try to share my screen with you right now. So today's subject is feature toggles and hypothesis-driven development. I'm Edson Yanaga, my Twitter handle is at Yanaga just in case you want to follow me. I discuss a lot about DevOps, microservices, Java, software craftsmanship and other subjects, interesting subjects I hope. I'm also a Java champion and a Microsoft MVP, which I believe is a very nice combination. And since we're talking about feature toggles for hypothesis-driven development, of course we have to discuss a bit about DevOps. And all of this discussion around DevOps and of course continuous delivery. You need a fully automated software deployment pipeline. You have all of your deployment problems already solved. So you can get to the third step, which I believe is to test your business hypothesis. But so far, everybody doing DevOps and continuous delivery, the main goal of trying to establish a continuous delivery pipeline was to try to reduce the bugs that we were deploying to production between each one of our releases. So if you're discussing this DevOps thing so far, the main goal of that was to reduce risk. We were deploying really software releases with bugs into production, where we're trying to reduce our batch size. So we could reduce the amount of bugs that we had into production. From a business perspective, we wanted to reduce the risk. But now that we have solved this deployment problems, we can dig into the hypothesis-driven development. And just to give you a comparison on how we traditionally do deployment pipelines, traditional deployment pipelines, you always assume that the next version is better than the previous one. So if I'm developing release 1.0, then the next version I'm performing some commits into my master branch. And I'm assuming that the next release 2.0 will be better than previous one. I'll have fewer bugs. All of my features will be better than previous one. I'm also assuming that the features that I'm developing right now are the best fit for the business. And if I release 2.0 into production, when I'm developing 3.0, I'm always assuming that 3.0 will be better than 2.0. And the analogy that I like to make with this traditional pipeline approach is that we are developing software as we were cooking or as we had a food buffet as a business. Because somebody, probably the owner of the business, just the chef, decides the dishes that are going to be cooked that day, you're just assuming that people will consume the dishes in the same amount. So maybe you don't know if people will eat more meat or will eat more vegetables. So you just have to cook like our pasta or you have to cook. And enough amount that will satisfy all of the users that are going to come into your beef fit today. And as you can imagine, someday you have a lot of the return people going to a restaurant. So you have a lot of meat that is not sold, not consumed, or the opposite could happen too. You have some people that own meat or steak and they don't consume the vegetables. So you're just wasting a lot of time, a lot of effort, a lot of resources developing food or cooking food that's not going to be useful into production. So the traditional pipeline approach is that we assume that these features are going to be useful into production, but we don't know for sure. So we have to spend a lot of time and effort developing these features, even though we don't know yet which are the results that we're going to have into production. And when we're thinking about hypothesis, dream and development, we can, the analogy that I like to make is that instead of trying to cook the whole beef fit with all the dishes that we want to present to our consumers into production, we're instead just picking small plates of each one of our features and we just cook a small plate. We offer to our customer, well, the customer decides, well, I like this food, but maybe you could put a little bit less of spices or put a bit more salt or I didn't like that at all because if the user enjoyed the food or the feature, well, you can spend more time producing more with that same feature, but if the user just rejected the feature, well, I didn't like the feature, I'm not going to consume this feature at all, you just can stop cooking that feature of that dish at that moment and you can invest in other kinds of foods or features that would be more useful to you, okay? So when we're talking about hypothesis, dream and development, I don't know what will be the outcome of the feature that I'm developing right now. That's why I need to test to give the user just a simple taste of what might happen. So the user, in fact, the data that the user is providing me can tell me which feature I should be spending more resources on. So if you're talking about hypothesis-driven development, if you ever heard about A-B testing, well, A-B testing is certainly one of the ways that we can achieve hypothesis-driven development and once I have this kind of A-B testing, we have version 1.0 and I have two different business hypotheses saying, well, maybe if I implement my recommendation engine this way, I'll have more sales into production or maybe if I tweak my recommendation engine this other way, I might have better results. So you don't know for sure and the best way for you to test your hypothesis is to deploy them into production. So maybe you have two hypotheses, recommendation A and recommendation B and the data in production tells you that, well, recommendation engine B performs better into production. So that's the branch that, that's the feature implementation that we should be choosing to keep moving forward into our software. Traditional way of trying to achieve hypothesis-driven development with A-B testing is trying to implement with feature branching. So feature branching and A-B testing are done like that. You just, since you don't know and you have the different trees of committing your source code, you just implement, there are feature A in feature branch A and implement feature B or feature branch B and later you deploy both versions of production and you monitor the behavior to test which hypothesis performing better into production and the traditional way of trying to, well, two different deployments into production and you need to route the requests of your users to your two different deployments is to use a smart router approach. And with that, you just, you add an artifact on your deployments which means your requests are going for this smart router and this smart router has the logic to route the request between feature branch A and feature branch B. It looks simple, but it does also assume that you're going to have a fully automated software deployment pipeline for both feature A and feature B and people will be developing this feature branching separately. Of course, we also have our dangers. We have some problems that become even worse. The bigger the team that you have. So imagine, for example, that you have all of the best practices for continuous delivery, your continuous integration too is integrating your trunk or your master branch for every single commit that you're performing. And of course, if you're developing your feature branch A and your feature branch B, for every single commit or maybe even hourly or daily, you're pulling the branches, the changes from the trunk or the master branch to your own future branch. So you won't have like later integration problems. But what happens if, well, I'm developing both the trunk, feature branch A and feature branch B, and later I decide that, well, I test as both hypothesis into production and I'm going to keep one of the versions, but on the meantime, some other team is also performing some A, B testing and they're developing their other features separately and they decided that their feature branch B should be merged into trunk. The problem with that is that, well, you were performing continuous integration of your feature branch A against the trunk and the team developing feature branch B also was performing continuous integration with the trunk. But when the team developing feature branch B decide to merge their changes into the trunk, on the next pool, on the next continuous integration cycle that you'll have in your feature branch A, you just received like a thousand commits, well, a thousand so much, but maybe a hundred commits from feature branch B that you were not continuous integrating. So you just had a huge conflict because this feature branch were sold different apart that now you have to solve this huge problem. And with all of the teams that I've ever talked to, the problem with that is that when they have such a huge merge conflict problem, most of them just decided, well, it's practically impossible for us to solve this merge conflict. We're just going to copy paste the files that are important in their future branch. We're gonna discard everything. We're gonna get the trunk again and just copy over the files that we think they're important to our future branch and just tweak the rest that should be implemented. You might be guessing it's not a right or it's not a nice approach for you to be developing software, copying, pasting things and then copying over the changes of other teams, right? But that's what usually happens if you're developing A-B testing with different feature branches. You're just one of the purposes of DevOps and continuous delivery pipelines is to reduce the batch size between each one of your releases. But every time you create a separate development branch, you are creating a huge batch size on that branch and all of that commits that are being put together in that separate branch, they're creating a huge batch size that is going to be integrated all at once at trunk. So you have like a batch size of 100 commits. All of these 100 commits are going to be merged in the single request when you decide to merge that to your trunk development. So many people get burned with this kind of feature branching approach and huge merging. So that's why we decided to develop another A-B testing approach, another hypothesis-driven development approach which is to use feature toggles. And with feature toggles, we just do trunk-based development or master-based development. Everybody commits on trunk but the features are not released all together on every single deployment. You just delegate to a feature toggle the moment that you're going to release this feature into production. So in the DevOps terminology, we're saying that with feature toggles, we're just uncoupling releases from deployment. And the most popular example that we have from uncoupling release from deployment was the Facebook messenger approach because Facebook didn't know if their backend would support the amount of requests that Facebook messenger would send to their infrastructure. So they just, well, they deployed the Facebook manager feature into production but the users couldn't see that. But in the background, your Facebook page was sending random requests to your friends, to the backend. So they could test if the backend would support the amount of requests. And on the release day, somebody inside Facebook just decided, well, let's flip this feature toggle so we can release the Facebook messenger to the users. So now they can use that. But in fact, in Prats, people have been like shadow using that into production for many months before the release. So we deployed the feature much before it was released to production. And how can we do that with code? So today I'm going to show you two different demos. One, I'm going to show you the Togo Z framework for feature toggles and the other one I'm going to show you the FF4j feature toggle framework. So let me just show you here the web pages. So ff4j.org is the web page for feature, FF4j and Togo Z.org, the web page for Togo Z. Since most of us are probably Java enterprise developers, I chose two different Java implementations. You have different implementations from different languages and platforms too. The only one that I used into production was FF4j because at that time, two years ago, I compared both frameworks and in my opinion, FF4j was more mature. But it just in the past couple of months, many people have told me that they were using Togo Z and Togo Z evolved a lot in the past few years. So I decided to give it a try and in my like one week testing of both frameworks, I can tell that, well, they're both comparable in terms of features. So now maybe it's a matter of choice, which one should you be using? So in the past, I only knew about people using FF4j in production, but now many people reported to me at least like four different teams in the past couple of months reported to me. Oh, we're using Togo Z successful into production and they gave me some reasons for choosing that. And I said, awesome, because it's very nice for us to have competition in this field so we can each one of the frameworks can improve in their own pace. And with some competition, it helps us to push forward the different implementations. So I just created two different projects. The first one that I'm going to show you is the FF4j demo. And can you see my source code right now? Mer, can you confirm that? Just let me double check with you in the chat. Yes, awesome. Thank you very much. You're amazing. So, so you can see my source code right now. Here I have a Spring Boot application. If I want to deploy an FF4j application, I just have to add some beans. So basically I'm using the simplest possible approach for configuring FF4j. I just create an FF4j.xml And on this FF4j.xml, I declare here two feature toggles. I'm declaring feature hello, which is enabled by default. And I have a description which I'll be able to see. And my administration panel, and I have a new recommendation engine feature toggle, which is not enabled by default. And I have this beautiful description. So how does it work? If I have here my hello controller, suppose I just inject my FF4j instance as a Spring Bean, and I just need to check here, if FF4j.check, and the name of the feature hello, if that's enabled, I return hello. But if it's not, I'm going to return hola, which is Portuguese. I'm Brazilian, so I'm going to return that in Portuguese. So I just added this if in my source code, and I want, now I want to run my application. So if I decide to run this application, I'll be able to see that in my browser. So application is starting up with my feature toggles enabled, and it's already running. Okay, and now let's see localhost 88 slash hello. So hello, my feature toggle is enabled by default. So if I want to tweak, how is it working? Let me open the console. FF4j console. And now I have here my feature panel. It's toggled a zone by default. If I turn it off, then I can just go to my web page back, and now it's showing things in Portuguese. So it's very easy to implement the feature toggle feature with FF4j into production. But this I'm adding this kind of if statements in my code. Right now, maybe we can have a better approach for that. So in some cases, we want to have like a different algorithm implement implementation into my production source code. So I want to test. So I created an interface recommendation engine, which has one method. And I have a default recommendation agent here. So it just returns like a default recommendation and which is burger, hot dog and milkshake, which is very appropriate since yesterday, here in the US, it was four for July. And I can tell that people consume a lot of this when we have holidays here in the US. And I implemented recommendation controller, which basically injects for me a recommendation engine instance, and I just return as the list of strings from this recommendation. So if I go to this implementation into production right now, go to recommendation. I have this beautiful JSON recommendation, which are burger, hot dog and milkshake. But how can we do that changing dynamically into production or source code? So basically the recommendation controller, I'm going to say it out. Well, this method, I want to add a ff4day feature. And the name of the feature is new recommendation engine. All right. And now I'm going to provide on my application two different implementations of the recommendation engine interface. So I'm going to add like public, improve it recommendation engine, and it's going to return a new improve it recommendation engine instance. And it's going to be a bin. For those spring lovers here, you know that, well, since I'm providing two different implementations of the recommendation engine interface at runtime, spring would complain because spring wouldn't know which one to inject since I have two different options. So I have to mark one of them as primary to be injected. Well, if I have a conflict, you're going to choose this one as a primary. And I have to restart the application because I changed the source code. So let's see how it behaves. So let's get back to the recommendation, hitting reload and still getting the burger, hot dog and milkshake. So let's go back to the panel. And here I'm going to turn the new recommendation engine on and let's see which is the recommendation, steak, steak and steak. And for those that know me, well, that's my particular favorite recommendation because for me, that's my favorite food. And so that's how you flip different implementations using FF4j into production. The backend repository that I'm using to store my feature toggles using FF4j today is just an e-memory repository. So it means that every time I restart my application, I'm going to lose all my data. But of course, FF4j provides you a lot of different repository implementations. So you can store your implementations using Redis, you can store that using MongoDB, you can store that using NEJ cache implementation, which means you could use in FinisPan, you could also use JDBC. So you have many different options for using your FF4j. And for J, I think that the benefits that you have a very nice spring integration, which at the moment for the amount of knowledge that I have from both frameworks right now, I would say that the spring integration, the spring proxy integration for me to be able to do this flip switching with FF4j is a bit more mature than to go Z, but both are very nice frameworks, okay? So next demo that I want to show you is, just let me stop this application. Now I want to show you how can you perform the same kind of thing using to go Z. So now let me switch my projects. You can see even, let me show you my to go Z implementation. And I'm going to run to go Z. So the basic endpoint is basically the same. And this one I'm using a wild flies form as the implementation. So basically what I can do, the nice thing about to go Z is that it has a very nice CDI implementation. So if CDI is your favorite implementation of dependency injection, maybe you want to give a try. I just create the configuration also is very simple. I just need to create here, I just added the to go Z dependencies on my own dot XML and that created a class that implements to go Z config. So I have here, it gives me the class that is implementing my features, the state repository. I'm using a file-based implementation because I think it was cooler to show you right now. And I have a user provider which provides me the users for authenticating the console. And if you go, what does this class means? This in FF4j, I'm basically implementing since I chose to use an in-memory repository, I needed to create a FF4j dot XML and I had to declare my features there. But if I'm using to go Z, I can create an enum and I can declare my enum implementations here. So I can say that, well, I'm going to have two different feature toggles. Hello, and it's going to be labeled hello to be shown on the my interface. And I also have WordCup teams who were playing the WordCup right now. So we have, I have a different feature toggle called WordCup teams. And you have to implement these methods which is the default implementation. So if you think about how we implement the feature toggles, maybe to go Z, my particular taste is that I prefer to code, to implement things using code as in Java. So for me, it's a win that I can use enums for my feature toggles. So basically that's all that I need to do. And in my hello worded point, I just have to check here using code. I just say, if my features dot hello is active, my enum is returning is active on this method return hello or else return hola. So let's see, let's go to my endpoint here. And hello, if returning hello because it's enabled by default, but now I'm going to open my to go Z console. To go Z, oops, local host to go Z. So if I open my administration console, I can turn it off. So if I turn it off, I go back here to my local host and it's returning hola because now it's returning Portuguese. So it's very similar the way that we can use both of them, my administration panel. But since I'm using my file repository to go Z automatically changes into production here's dot properties. It already changes dynamically that the values of my properties file here. So if I decide to change it, but to enabled, it's going to show hello again. And if I go back here, hello equals to true. So you have a sync here. It's an e-memory repository. It's a file-based repository. If you had JDBC, it would work the same. But I will want to show you other things like how can I do to, let's create another endpoint. World cup endpoint. So I decided here I want to return some kind of JSON. If I want to implement that, I can do just the same if like my features dot world cup teams is active. Maybe I want to return world cup favorites dot better favorites. Yeah, or else I can return. I don't have a best practice, but I assume that maybe you want to show your future toggles using if our statements. I think it should be clear for you to see how the flow of the code is behaving. So let's restart my application because I just added another future toggle. And just in case you were wondering, does to go Z provide me the same proxy feature that you can switch the things dynamically without adding the if statement? Yes, but as far as I can tell, it only works with spring. And since I'm using wild class form with Java EE, I couldn't find a similar implementation for a CDI. But I guess, well, that's the tip. Somebody could be implementing that. Let's see how it behaves into production. So hello, it's still working. And if I go here to my endpoint world cup, it says that, well, the favorites so far are Croatian, England and Sweden. But if I just go back to my administration console and I enable my new features saying, well, let's have better favorites from now, I would say that, well, maybe the future right now should be Brazil, Brazil and Brazil, which by the way is playing tomorrow afternoon. And that if you don't have a favorite team so far, I strongly suggest you to cheer for us because I will. And that's what I wanted to show you for now. So I'll stop my screen sharing. And if we have any questions available, I'll be more than happy to answer them. Back on the scene here. All right. So John had a great question for you, Edson. It was around the concept of, what about performance of a given feature? So think in terms of an A-B test, if I am doing an A-B test between red versus blue or green versus black, whatever the two things I'm testing, Brazil versus let's say Germany, Brazil still in the tournament. How do we measure the performance of one of those particular features? You mean business performance or back and forth? Yeah, the business performance. Business performance, okay. Yeah, as I mentioned that very quickly in the slide, in the beginning of the presentation, you need to monitor a behavior. And traditional way for you to monitor application, well, you're just monitoring CPU, memory, network on these kinds of things, which is our back and statistics. You have to start sending your statistics to a monitor platform. And these statistics needs to be kind of business statistics. So maybe if you have an online sales application, you're going to measure how much you're selling at this time of day, during one week, you're going to measure that and compare to your baseline on the week before. If you normalize all of the other variables, you can compare the algorithm between one implementation and the other one. Okay, and I would mention that for A, B testing in particular, there are a lot of great online resources to describe what statistics you need to capture to validate that you are doing it correctly and making sure that A versus B is being tested correctly. Yeah, and just- Another question that came up, oh, sorry, go. Please go. I was about to say another question that came up was related to what about not Java? You saw all your demonstrations were in Java, which of course is a primary user base for people like you and myself. But what about Python or Node.js or .NET? We did find a website, featureflags.io, that seems to gather all the different toggle Z-type things, but do you have any other insight into non-Java platforms for feature flags? Well, yeah, no, I don't. Unfortunately, my experience is purely Java and that's the, well, as I said, I used FF4G in the past and I've been playing with Togo Z in the past week so I could have a baseline, but I don't have knowledge. I know that people are using feature toggles in other languages and platforms too. I just can't recommend anyone because it's out of my knowledge. Okay, one last question and we're out of time, but Vicky asked a question that I think is good to kind of recap and kind of summarize everything here. What is the exact significance of this framework and if you already have instrumentation frameworks in place? So if you're already capturing metrics, what is the real value this is on top of those metrics? Okay, well, metrics are a requirement for you to be able to perform hypothesis really on development. You're already measuring the performance of your business results into production. And now, as I said, you can switch your, maybe you want to develop a new version of your recommendation engine, but you're not sure if the new version will be better than the current one. Or maybe you have two different teams that will be implementing two new different approach, but you don't know for sure which one will behave better in production. So you can implement both features and then flip them dynamically into production just to monitor the results. And one of the things that I wanted to compliment in the first question is that you need to monitor for a long enough time so you can compare with a baseline. And you have to be sure that all of the other variables are the same. I have to tell a famous taxi app in Brazil, for example, they were performing A, B testing. So they released a new version of the application and they realized that the number of rides that they were having at that day was much lower than they had in the previous week. So they said, what's happening right now? And they just realized that, whoa, Monday at this time last week was raining. So we had much more rides. So it's unfair to compare with the metrics that we're having right now. So you have to monitor a lot of things to make sure that all the variables are the same to have a meaningful result with comparing your future titles. Okay, well, I'm afraid we are out of time for today. The recording will be available to folks that are interested in it. I did add both of Edson's GitHub repose for the demos that he showed today to the chat. So you guys should have those already. But always feel free to email me if you have a question, have a need and also have a future topic. You'd like to see us discuss. I'm now working on the next round of August and September and October definition live session. So if you have an idea, feel free to reach out to me at birdredhead.com. And I'll get that information back to you. Also, the recording will be available as soon as we have it. Thank you so much. Thank you, Edson. Thank you. Thank you all.