 Welcome everyone to the webinar Future Toggle Promotion via CACD Pipeline by Nilesh Mevada and Naresh Jain. Without further delay, over to you guys, over to you Naresh. All right, good morning, good afternoon, good evening wherever you folks are joining in from. Thanks for tuning in on a Saturday or a Friday night. It's a real pleasure to be presenting here with you. I have my colleague and the man behind the whole Future Toggle stuff, Nilesh. Hello everyone. Welcome to the session. Thanks for joining. Cool, so the way we're going to split this is I'm going to do the initial overview then we'll jump into a demo then I'll do some more overview then Nilesh will jump into a demo so we will swap between us. We'll swap between the overview and the demo sections. Before we get started, you know, if I just want to see how many people in the audience here are familiar with, or are using regularly Future Toggle. If you can just put your hand up, just use the raise hand. It'll give me a sense of how many people are familiar with Future Toggle and then we can base our session accordingly. So use the raise hand option. If you are familiar with Future Toggles. Okay. Wow, okay. So I, is it a fair assumption that majority of you are kind of new to this topic and would like us to cover a little bit more basics. If you agree with that you can raise your hand. Or you can put it in a chat as well. Yeah. Okay, so I'm seeing few folks in the chat saying I've heard about it but you know, not used it, it would be good. It's a new topic. All right. So, in that case, change of plan Nilesh. We're going to try and do a little bit more basics but of course we will try and cover the core of the topic because that is also equally important. All right, so I'm going to quickly share my screen here. And let's jump to this and. All right, so we're going to jump straight in Future Toggle's promotion using CI CD pipeline, and the big claim that we want to make here is basically the death of release bundles and you know independent deployment is is the independent releases coming forward. So let me quickly touch upon why Future Toggle's right why Future Toggle's are important and today, if you are trying to embrace continuous deployment and you're trying to move towards. I think some of our keynote speakers talked about how you know some of the companies are releasing core to production every second, you know, and you know how is that possible. So I think one of the secret sauce behind that speed and agility to me is the core concept of Future Toggle's. So Future Toggle's allow us to do independent deployments what do we mean by independent deployments let's say in a typical service oriented architecture where you have a set of microservices and a set of micro front ends different teams or the same team is working on these and as in when your services are ready or micro front ends are ready, you want to start deploying them to production so that the teams can kind of, you know, practice trunk based development and other good practices, and also you know they're not waiting and you don't have this complex maze of coordinating between different people and getting all the jigsaw puzzle in place before you can start deploying things right you want the ability to decouple your deployments and as in when each service or microservice or micro front end is ready, you would like to deploy it right and so Future Toggle's is a essential ingredient to be able to do that, of course, it is not the only way to do it. So, you know, I just want to make sure I put that disclaimer that Future Toggle's is not a silver bullet it's not going to solve world hunger and climate change but if you're trying to do independent deployment and especially if you have changes that are not backward compatible then Future Toggle's become kind of essential from an independent deployment point of view. It also helps you separate your, you know, deployments and releases so you could keep deploying frequently but when all the pieces are there then you can actually release or make that feature available. You could also use some of these Future Toggle's techniques for advanced concepts like AB testing for canary releases for gradual rollout and so forth. And the most important thing from a measurement point of view is some of you might be familiar with the Dora metric of change lead time. So if you're trying to reduce your change lead time which basically means from the time developers are done to the time it is a feature is actually available in production. You know, if you're really trying to automate your entire pipelines and you're trying to optimize for speed to delivery, then reducing CLT the change lead time becomes an essential aspect and you know, this is kind of how we've been focusing with several of our projects in terms of trying to, you know, reduce the CLT using Future Toggle's and independent deployments. So that's the why, right? Like I've just put the why this is important and why we need to pay attention to this. Let's start with an example. So we are all on the same page and we are talking, you know, from the same angle, right? So let's think of a typical e-commerce company, right? It's built the product, you can buy the product online, etc. But so far in its the evolution of this e-commerce company, it did not support customers to cancel the orders. Okay, now the business feels that hey, you know, the ability to cancel orders is important. However, we want to only allow the orders to be canceled before they are dispatched. And if someone's prepaid for the order, then on cancellation, we need to issue a refund back to the user. And upon cancellation, we also need to adjust the inventory accordingly, right? We need to increase the inventory in this case on cancellation. So all of these things need to be handled as part of this particular feature, which is the order cancellation feature. Now, you know, in a typical way, you might say, hey, I need to get all of these things in place before I can release this feature or before even I can deploy this feature out, right? But how with continuous, if you're trying to practice continuous deployment and then how you could use feature toggles as a way to, you know, release this. So first let's look at the current state. Okay, the current state is you have an app through which people are ordering, you know, that basically then calls an order service in the background. This order service talks to an inventory service and then this inventory service talks to a bunch of other services. And that's kind of how, you know, it will respond back with basically what was the, what are the available orders and what are the available inventories and then that goes back into the available orders, you know, with the statuses in terms of out of delivery, out for delivery, it's confirmed, etc. This is the current state and we would like to change this current state to something where you should be able to cancel the orders, right? So let's look at what the new state should look like. So here, the app is going to check with the order service just like before. The order service now is going to check with a feature toggling system. In our case we're going to be using feature hub for today. So feature hub is a feature toggling system. I'll talk a little bit about this. I'm going to talk to a feature toggling system and it will check whether cancellations are enabled or disabled. Okay, let's assume cancellation is enabled. Then it, and the system that is actually the service that is checking with your toggling system, your feature toggling system, we call that as the decider component. We're going to do this concept of decider component because that is extremely important in your ability to be able to do continuous deployment as part of the pipeline. But yeah, we have the order service which we are calling as the decider component. It checks with feature hub, checks whether this particular feature is enabled. Feature hub says yes, this feature is enabled. Now you might call a slightly different version of the inventory service. Okay, and then it will do whatever it used to do before. You're going to respond back and you will notice now that you are responding back with a cancelable true or false option as well. And this change has happened because you're, you know, the flag for enabling cancellation has turned on. If this flag was turned off, then it would not return the cancelable parameter back. Okay. Now what this allows you to do is this now allows you to decouple each of these pieces, each of these services now could independently be deployed including your front end. Right. How could you release your app? You could, you know, basically if the app was done first before the order service was ready, the app could assume that, you know, if cancelable is part of the response, then it will show the cancel button against it. If it is not available, then it will simply not show the cancelable field at all. Right. And so this way, and if the field even does not exist, then it assumes it's as false, like the false is the default state. So now if the app was ready, you could actually just release the app whenever the order services ready, you could release the order service. Of course, as part of this, you also need to now build out a new service which did not exist before. For example, you wanted to build a refund service. If the refund service is available because it's a new thing that it was the last thing to be enabled, then you would add that later. And till that point you would have cancelable as false, so everything works as before, only when let's say the refund service is available, you will then go and turn on this cancelable flag. And that is what will enable, you know, the changes also in the inventory service and so forth. So this is kind of a quick just a high level concept in terms of if you move towards independent deployment and you want to use feature toggles how you can start decoupling the releases of or rather the deployments of each of these components out of order right they can go in any order, and you could control what is visible or what is the impact to the user through a feature toggling system. So, and the, and the service that are the component that is actually controlling this decision factor we generally that's that's called the decider component would be the one that essentially we will see later on how that orchestrates the toggle through the CI CD process. Now, with that, I think enough of talking, let's request Nilesh to quickly share his screen and, you know, show a demo of what I just explained, so that everyone can be on the same page. So far, this makes sense just a quick yes no thumbs up forever, just so that I know I've got the basic concept through perfect. Okay, cool. All right, Nilesh over to you for a quick demo. Thanks. Okay, thanks, Naresh. So I'll quickly share my screen I'll go take it over just give me a minute. And sharing. So, what we saw just now is that we so now it's already talked about the scenario right so we talked about that whether the, the clients, you know, whenever they're deploying they're waiting on the toggle and without deployment we can basically enable or disable the feature which is like a very powerful thing to do. Now, you know, this is the system, you know, we were talking about feature hub that's the open source platform that we're using for the toggle. What I've done is, so these are the list of toggles out of that. You know, there is one toggle which is called enable cancellation that you know Naresh was talking about just a while back. This has been configured in the front end side. So the front end is basically waiting for this toggle. So, you know, as you mentioned, if the order status is basically dispatched. Then, you know, the cancellation is essentially not available, but if it is not dispatched and if the toggle is enabled, then this new feature should allow the cancellation to be available to the end users. So, so front end is basically depending on that feature to be visible to end users at the same time. Let me quickly show the back end as well. So here is a back end as well. On the back end side, once again, you know, we are just, you know, when the cancellation is, you know, right now this has been disabled as we can see in the screen. Once again, yeah, so this has been turned off right now. So, since it is turned off, basically the cancellation is not available to the end users right now that feature is not available. Now what we do is we just, you know, try to switch it on right in production and immediately we will see that the thing the feature will be available to the end users. So now it has been turned on as you can see in the app, you immediately see the reflection of that. And, you know, as we expected, you know, there are few items which has been not dispatched and since the toggle that feature is available, we are enabling that cancellation to the end users. Similarly, if we, if I make a call, once again, we see the exact same effect, the feature which was not available to the users to initially consume at all. That is now, you know, responding as whether, you know, the cancellation is allowed or not based on whether it was dispatched or not. And, you know, accordingly, it will reflect the status. Right. Any, any questions? Cool. I think, Narish, can we, you know, start with the promotion then, I think. Cool. All right. Thanks, Nilesh. I think there's one quick question here in the audience. Let me see if, you know, is that cancel enable disable is auto handled by feature hub. So basically what happens is that, you know, and I'll explain the architecture of feature hub in a second. So some of this will become a little bit more clear. Let me quickly share my screen. Okay, hopefully you can see my screen. So that was just a quick demo that Nilesh showed of feature toggle. So here, I think Nilesh already mentioned that we're using this open source project that Richard and Irina to, you know, awesome folks from New Zealand. So a quick call out to Richard and Irina for building feature hub and open sourcing it. This is, this is a platform that we are using and we'll deep dive into it. We'll also explain why we ended up on this platform and what happened but feature hub.io is the URL you can go it's free open source tool that allows you to do feature toggles and a lot more things including AB experimentation and so forth. You know, it's important to just quickly explain the architecture so everyone understands what was going on when Nilesh was demoing things. So here you will see where you are seeing, let me get my highlighter or laser pointer. So you'll see here, you know what Nilesh was doing is he was interacting with the feature hub web application from where he was turning on and off the feature the moment he turned on the feature what essentially happens is obviously goes and sits in the database. It also then, you know, changes the value through NATS a component that is there and then it kind of kicks into the cache, which is called dacha. And then from there it's available through an edge API. Now, the edge API is kind of your, you know, API gateway in some sense, which is essentially talking to the cache and has all the values that of your feature toggles. The feature hub also provides you SDKs, both for like several languages will talk about that and what they do is essentially they establish a, you know, SSC server side event, so they register for a server side event with the edge. And anytime someone changes this value, it obviously propagates through the cache comes to the edge and the edge notifies the SDK saying hey, you know, there's a new value and that's how you saw that without refreshing the page. You know, the cancel button where automatically visible. And all of this is happening because the SDK that they provide is talking to the edge and is listening to server side events. And so whenever you toggle things it's readily available and the UI can get reflected. Same thing on the back end as well. So both for back end and front end. Plus also you they provide rest APIs. So you can query the edge API through rest and get that. So I think it's a pretty neatly architected system, quite scalable. We'll come to that in a minute, but they provide a huge variety of SDKs already out of the box. Pretty much all popular languages are available as part of the SDKs. They have a lot of interesting features and stuff like that again, I'm not going to go too much into it but just highlighting some of the things because a few folks said they're new performance when it comes to this because this is going to sit on your critical path performance becomes important. So we actually did quite a bit of independent performance testing around this, you know, trying to scale it up to about 300 concurrent connections. And we saw that it was basically very smoothly just scaling out on a fairly commodity hardware. And, you know, with 200 concurrent users we could go up to 129 RPS with 300 concurrent users we've almost up to 1600 RPS. And another important thing to notice is only the edge is one where the CPU and memory is actually utilized but the rest of the things is barely using anything right so it's architecture also deployment architecture allows you to auto scale the edge servers and which is where you would hit most of the traffic. So it's a pretty neat, you know, what I would say cloud native way of building feature toggles. We did comparison with a whole bunch of different tools that are available including a homegrown solution that we had and then eventually I think feature hub seem to kind of tick all the boxes that we had. Again, the comparison itself could probably be a whole new talk in itself, because there's a lot of interesting aspects that one needs to look at in terms of, you know, which platform to go with various options from Infra to toggle value categories, different kinds of targeting, etc, etc. But anyway, this is not the session for the feature comparison just wanted to highlight that we didn't do our homework and then pick up this particular thing. So now, with that out of the way, let's jump into the core of the topic. I think we've taken about 20 minutes so far to kind of set the context and now we will jump into the core of the topic. But before I proceed, let me just take a quick pause here and, you know, just quickly see so far as everyone with me. Does this make sense? Yeah, I'm assuming everyone's with me so far cool. Okay, so then let me go back to sharing. And now we will jump into the feature toggling. So what is the problem statement, right now, Nilesh showed the UI through which you can turn on or turn off the features. And it is instantly available. And of course, in any typical, you know, organization you would have multiple environments, you would have an integration environment or let's say SIT, you would have a staging environment or a replica environment which is similar to production, and then you would have production at least you would have these three maybe a dev environment as well. So what you want is basically you want, you know, you probably would do some progression testing for a given feature you would also do then a regression testing before you release. And this could be automated or manual. Again, that's a whole different topic in itself. But you know, we all I'm trying to say is this is agnostic to how you're managing your testing. So it supports both kinds of, you know, continuous testing kind of a model that everything automated, or it could be more heavy on the manual testing or exploratory testing. But assuming you have different environments, and in each environment you want to do certain kinds of testing. So in your integration environment you want to test out that the feature integrated with all the other systems is working as expected, you want to then put it in the staging environment, get some of your business folks or some of your product folks to do a UAT, and then you may want to release it into production. Even in production you may want to do a gradual rollout or canary releases are very targeted releases and so forth. It's a key challenge when you're using a feature toggling system. One is obviously the whole design and thinking in terms of decomposing your components and finding the decider component. But the other important thing is how do you automate this toggling process so that, you know, when someone's turned on certain flags in an environment and let's say sit and they did regression testing and they said yes this all looks good this is working fine. So what you want to make sure is the same set of toggles are then promoted to the higher environment, let's say replica or staging to make sure that you know what was tested and certified in a lower environment is actually what is replicated in the higher environment and then all the way to production. If, if you don't ensure that then you may accidentally be turning on or turning off something, which means something that was not tested is now available to users, and that could break things in production, or something that was working before is now suddenly disappeared because someone accidentally toggled it off. Someone can relate to this problem what I'm saying about why it is important to ensure that what was tested and certified in a certain environment actually gets is kept intact and then promoted to higher environments. It's the same philosophy that we use with our code right when we, when we basically test our artifacts it could be a Docker image it could be a jar it could be whatever. We want to make sure that the same artifacts are actually promoted to higher environments and goes all the way to production. You don't want suddenly someone to give you a new artifact saying okay deploy this into production. Because then you have no sanity around whether this will work or not. Correct. So, the same thought process in the same philosophy also applies to the state of these toggles, because you have to keep them intact and you want to make sure that they move forward. So that's the key point about this presentation today is how do you ensure that what toggle states feature toggle states where, you know, accepted in or tested in a certain environment they are kept intact and promoted to higher environment that's what we call. And that's where we want to make sure that it is part of the CI CD process just like your configs like your DB scripts like your code, and it basically goes along with it through this promotion process of the CI CD. So let me quickly just first run you through the whole process, and then Nilesh will give you a demo on the same. So the first thing one needs to do is we call ft.yaml feature toggle.yaml file. So in the decider component that you have, you would add this new file the developers would add this new file, they would declare the various toggles that they have the name of the toggle the URL to where this feature that this toggle belongs to. And what is the default value is it a boolean is it a JSON is it some more complex data structures so you can have different kinds of things so you declare these three things for each of the feature toggle that you're introducing. Once you have that, then just like, you know, you would check in your code, you would check in your ft.yaml file as well into the decider components repository git repository. And then you would create a pull request when you create a pull request as part of the pipeline. What we first do is we validate whether the entries in the ft.yaml is valid or not. And we also check with feature hub whether you know someone's duplicating an already existing feature ID toggle ID, right, because you want to make sure those are unique else you will accidentally be turning off something else. And once everything is good once you merge the this thing in the build will produce the four artifacts right a code artifact which could be a jar could be a Docker image could be something else. It produces a config artifact which is basically all the configuration for different environments it produces a DB change set if you're basically doing automated DB changes, you know, then you would have a DB change. And then also the feature toggle yaml file is also an artifact and these all get created as an artifact. Now once you have the artifact that your build pipeline, your CI pipeline has created then your CD or the release pipeline will pick these artifacts. And essentially it will deploy these artifacts into your SID environment and if if that is successful, then there is a stage in the release pipeline which basically syncs the toggles to feature hub so it will basically take the ft.yaml file, find what all the toggles are in it, and then you know basically set up through an API call will create these in feature hub. And then of course your release process will continue as normal but at this stage you have, you know, basically taken all the new toggles or deprecated toggles and synced it up with feature hub so they are available. Now once the syncing is done, then, you know, either a set of automated tests could run, or a set of manual tests could run. So you would go ahead and do the progression testing at this stage. During progression testing of course you can keep toggling on and off and make sure that on state the feature is working fine off state the feature is working fine, because you want to make sure both on and off testing is done. And you play around with this but before you start your actual regression testing where you are basically saying hey this is good to go. You just need to make sure that whatever state you want to promote the toggles to the next environment you set that before the regression testing starts. And once you've established that in the feature hub then you would kick off your regression testing and at this stage when you basically say I'm about to kick off my regression testing. So what we do is we basically put a marker in feature hub saying hey we are about to kick off so basically take a snapshot of the of the toggles now and this snapshot is also saved in in our case we using Azure DevOps it could be any other platform. So basically store this snapshot against the feature in in Azure DevOps for later references. Then you start your regression testing you do your regression testing as normal. If there are no bugs found in the regression testing. Essentially you would say hey I'm good I'm about to finish my regression testing I'm going to give a sign off on my regression testing. And at this stage what we do is we basically during the sign off we verify with feature hub, whether what we started in terms of the toggle state where they remained intact whether you know during regression testing people did not go and play around with the toggles because then you lose sanity whether something should be on or off in the production right. So we basically verify that no toggle state changes were made. We again take a snapshot of that and we store it so before and after snapshot is stored for later references. Again to ensure that during regression testing when you're supposed to not be playing around with the toggles everything is fine because you set everything in a particular state, which is ready to move forward to the next environment at that stage you want to basically do a regression test. This could again be a completely automated process in some teams we do that and some teams it is not fully automated so you do it manually. Both ways it works fine. And this is when you would start deployment to the next environment once the sign off is given then the deployment to the next environment gets kicked off and pretty much the same process will continue. I'm going to jump through some of this you do the uat and finally once the uat successful you would basically deploy to production and at this stage you would take the, the, you know, the snapshot that was captured and just promote that to production. So this ensures that you are keeping everything intact and not mucking around with the toggle states. And this is a completely automated process without any manual intervention no one's going to the UI and changing things. You know, once you've given the sign off so once you've given the sign off everything is fully automated from there before the sign off you may want to play around with different states on off states and validate so that's perfectly fine. Once you give a sign off then the snapshot is promoted through the pipeline just like any other artifact is promoted and goes all the way to production. So that is a quick run through of something that we've spent many months thinking through and trying to come up and now we have this fairly successfully running in several production pipelines. So let's quickly pause here and see so far if this is making sense. If everyone is good so far then at this stage, we can jump in and have Nilesh kind of quickly give you a live demo of how this is in action. So Nilesh jumps in let's quickly take a pause. Everyone with me understand why it's important to make sure that you need to have the toggle state and promote them through the whole pipeline. We have one person raise the hand I'm not sure they have any question and we can see one question in the section probably can take it. Another question from Sarah here which says do we not have to test with both toggle states in the SIT and UAT. So generally what you do is in the SIT you would basically do the on off testing, you would make sure both in the on state and off state things work fine once you've actually verified that then you decide what is the state that you want to promote to the next environment all the way to production. And at this stage you basically set it to a given state right beyond this you generally don't want to keep playing around because if let's say a feature is not ready you're only deploying some components of that feature then you want that particular feature toggle to be off. However, you would have tested both the on off state, you might have done some progression testing with on but since the whole feature is not ready, all the components are not available you toggle it off then you do a regression make sure nothing breaking, and that is the state you will promote to the higher environment. Once you go to the higher environment you ideally don't want to keep playing around with these toggles you want to keep them consistent also from a UAT point of view they are just saying okay, maybe as part of this release train or as part of this release we've got five features, five of them are fully baked in but there may be five other features that are not fully set up. If they're not fully set up then they're off but from a UAT point of view you don't care about those in process ones and so this is why you don't turn on the flag for those in a UAT environment or in a staging environment you just keep them off and then you proceed forward. Hopefully I've answered that for you Sarath. Cool. If there are no more questions then over to Nilesh for the magic and hopefully the demo gods are with us. Yeah, yeah sure. Okay I'll quickly share this screen. There are lots to show and we just have about few minutes so I'll try to be a little fast. Well Nilesh has already mentioned a lot of things anyway so let's start from the beginning. So what we are going to do is we have about, so this is what you see on the screen right now is basically the current state of the toggles into the system. What we are going to do is we are going to add few toggles into the system so as Nilesh mentioned a while back. You know we have this ft.yaml which is baked in and we are basically declaring that okay there are some few there are few toggles that are going into the system now is part of this release. So he also mentioned about the atomic release means that you know whenever this code goes and the configuration goes or whatever the DB scripts goes along with it. You know together as a bunch this thing basically is promoted in the pipeline and goes to the production. That's how it makes it you know an atomic release in that sense. So let's say we are trying to deploy this now. So these are the toggles that we have declared now. What I'll do is I'll quickly you know queue a build to this and you know the one step that he mentioned was that during the build process. We basically check whether you know the only entries are which is defined into ft.yaml is proper or not etc. So let's you know see it in action. So right now so build has picked in and it is basically trying to figure out whether the ft.yaml you know the three entries that we created is basically you know proper. Both from the syntax point of view and whether that basically exists within the existing system or not. If it is a duplicate then it should basically fail and all those you know normal checks are basically taken care of in this. So as we see it has passed I'll quickly run through you know the output once. So there are a bunch of checks that it does you know valid values are there into the features whether the name was correct or not whether the Azure URL that is used. So if you have noticed you know we have used the URLs over here which is like you know just a link to the feature that is being released and to which this particular toggle maps to. So why that is important I just you know get in a minute but yeah so it basically and you know checks for all these basic entries it also checks whether there are no duplicates and stuff like that there are a bunch of checks that is carried out. Once that is successful you know we have this artifact ready which we need to you know release along with the code into the you know the respective systems like the workflow is going to be as we see over here. There are like three environments so we are basically starting from a site. Then we move to replica then we move to prod replicates basically a staging sort of an environment. So we'll start from that. So let me create a release out of this particular build that was just successful. Right, so let me get to that. There are manual steps which have introduced just so that you know it does not quickly run through every step stage at one go. So we'll have to you know so there is some preparatory stage that it is you know trying to get into post that till the time that it runs you know I'll quickly you know you know talk about what is going to happen next. So once this basically is successful. There is this next stage that nourish already gave a brief about all these steps anyway but you know the next step basically is about syncing those entries into SIT what that is going to do is, once this particular stage runs, we should be able to see this new entries, you know, popped in over here for you know the SIT and QA guides to you know stay ready with him to test upon etc. So let me see whether this has taken yeah that has been successful I'll quickly resume this operation and let's see the things in action now. I'll switch on the logs as well so that we can see what's going on. So now the entries are being synced into the environment. So let's see, and that has been synced now so let's see I just quickly refresh this screen. So as you can see there are just three entries that has been created you know demo 22 demo 22 so there are like, you know we have this entries now available to us. There are three entries, three ft.yaman entries all three of them have been created now, as we can see right, all are by default you know of state, and once, you know as I'll just you know get back to this particular stage, you know this is the stage where you know basically QA folks will start doing the progression test so they can basically you know go to the screen they can do on and off into this one and they can just check whether the feature is working fine. That component is working fine or not. Once that is through and when they are about to start the regression test right. They will run this stage again what this is going to do is it is going to please. You know this particular environment for any further modification so that nobody you know, you know please around with this particular environment and the essential regression the core configs, you know the db scripts whatever we have into a site environment that is checked with the exact state. You know this particular site environment is into. So, let me run this stage right so it's in progress. So along with you know locking. So if you've noticed this three toggles right those are essentially unlocked you see this small icon over here, these are like the toggles that were already in earlier. So these three toggles are basically that has been freshly created or unlocked so that you know QA can essentially do their on and off testing. But the moment we start the regression right all these three toggles will be locked for any other modifications. Let's see whether that has been successful. Yeah the regression has been successfully done so let me refresh this screen what will happen is this you know entries will be locked in a site for any further modifications. So as you notice all this you know entries have been locked now for any further modifications because the regression is going to start so no more fiddling around with the toggles. At the same time, you know, let me see there was if you see this particular entry has been created just now. What it does is you know the entire snapshot of this feature toggle right of the site the environment. It basically takes the entire snapshot that what was the state in which this society environment was before starting the regression. And it creates a hash around it and it basically stores it for against the feature ID so that you know the respective you know operations or release managers can have a look as in what was it initially before the regression started. I'll just quickly show the, the attachment as well what is there in the snapshot the snapshot generally consists of, you know, Jason, if I go to downloads and if I open a snapshot file. Let's open it with Firefox for now. So it can say it consists of the entire society environment what the particular state was. It was into so it was logged the state was false etc etc so that basically it gives a snapshot of that particular environment at a given point in time that was before starting the regression. So now we are at the stage where the regression is already started. What happens is after say if the regression is automated it will basically finish within say a couple of parts. If it is manual then it might take a couple of days or something. But the movement the regression is ended right you know we can basically run a step which will basically sign off this particular stage. The sign off is basically just ensuring that nothing was changed during this regression phase in terms of changing any states into future toggle although it is lost. It also does another level of verification wherein it checks the version so any change that happens to the toggle the version gets changed so it basically checks you know that aspect as well that actually there was nothing changed. And once that is confirmed it gives you know it verifies that yes everything was fine the regression is successful. The confirmation of that is also once again you know provided over here as you see I'll just you know refresh this once again. So if you see three minutes ago regression was started and there was one snapshot that was taken. Once the regression the sign off has been given the sign off has been given. Once again the SIT snapshot is taken and you know we compare all the toggle states whether anything was changed or not. The version numbers and as you see the hash is similar that means you know nothing was changed we guarantee that. So this is this guarantee is basically done in the automated fashion so that nobody has to you know individually come in check whether this was changed that was changed or not. So it gives that particular assurity from that aspect and we are out of time but you could just wrap up here at least. Sure okay yes and at the same time once the sign off is given we essentially promote those particular states so over here the promotion was so initially into SIT the toggles were into the off state as you notice. What we have to do is we have to promote that particular state into you know the replica and the prod environment as well. So essentially this the remaining two stages are basically about that it essentially you know runs the stages and whatever diff that was there within SIT and prod right. If something was promoted as on it will basically you know promote that as on to replica and the production environment in the automated fashion. So that is what it does that is essentially what this promotion process is all about. This is how we ensure you know the automation as well as you know the assurity that the environments are not fiddled around accidentally as well. All right thank you guys thank you everyone for joining the session and thanks Nilesh and Arrej for insightful session. Thank you thanks everyone.