 All right, good afternoon Now that the sugar levels have dropped to an all-time low Fantastic, so listen I you know as Andrew and I prepared and we you know start to think about what we wanted to talk about I was actually reminded of a story. I heard and it was about Some moose hunters and I can tell you this is particularly relevant Andrew is from Canada So he brings the moves I'm from Texas So I bring the guns seems seems important and the story goes like this There were apparently a couple of hunters who wanted to go up in a very remote region and hunt moose So they hired a pilot with the plane one of those planes with the pontoons Flew him in there landed in the lake let him out and said knock yourselves out see in three days takes off Three days later the pilot comes back lands the plane here come the the two hunters They've got all their gear and two giant moose Meese moose moose have to ask Andrew. He's the expert And so the pilot says they start to get their gear in the plane and they start to put the moose on the plane And the pilot says hey, oh, you can't do that You know one moose tops. I said no no no We came and did this trip like a month ago and just like this same gear To moose no problem. The only problem we got is we got a different pilot Back and forth back for thought are again finally the pilot says Okay, look you did it before fine. I don't think it's I think it's too heavy, but we'll give it a try Backs the plane all the way down to the end of the lake turns it around they get the brush out of the way He says let's go gives it the gas Full speed down the way gets up enough speed plane is shuttering Starts to pull up plane gets a little bit out of the water and then boom crashes Stuff flies everywhere guys fly out of the plane Moment of silence crickets in the distance and the guys kind of popped to the surface of the lake. Oh, Bill. Bill. Where are you? I'm over here Man, where are we? He goes? Well? Looks like about 500 feet further in the last time Now I think the moral of the story here is what we're gonna talk to you about continuous delivery It's a form of continuous improvement and software right how do we adopt these practices for ourselves? And so what Andrew and I are gonna try to do is let's face it There are companies who were born on the cloud built your applications from that from the start But the truth of the matter is there are a whole lot of us in this room including us who have been building our Applications before the cloud and we have a way of doing things long-release cycles right Architectures that maybe predated some of this stuff and yet we want to take advantage of continuous delivery, right? Then we for all the reasons that we all know So what we want to share with you is how did we go through that transformation what we're all the rocky points We've been through what did we learn from that and what do we do about it? So as much as possible want to be very practical about that and so when we thought about this We said you know to be honest this is you know just like an addiction I mean we've done things a certain way for a long time. We've been trained to do that We've been rewarded for doing that and the fact of the matter is despite the fact that our hearts tell us We want to go in a different direction the rest of our body still feels that pull towards the way We used to do things and it's hard to change that behavior right, so maybe you can introduce us I was just gonna say you've been talking all this time, and you haven't even introduced. Sorry. Sorry What's your name? So my name is Kendall and I have a problem with long-release cycles that I do on a repeated basis Hi, Kendall. Come on everyone Hi, Kendall. Thank you. Hey, I'm Andrew and I am an addict Hi, Andrew So maybe you're a little too good at that which makes me worry So what we want to do is best way to talk about this kind of a problem is the 12-step program So we're gonna take you through the 12 steps of breaking our own addiction to long cycles talk about how we got there So let's go. How many of you are familiar with the program? No, you don't need to show your hands Give it to yourself. Step one admit you have a problem So here's some good examples of this right the emotional reaction We say hey, you know the reason we like big releases is because big rocks make big splashes, right? I mean we want to make a big dent in the market big press releases big big big We love that and how am I gonna do that if I just do it one bite at a time, right? The truth of the matter is you can still accumulate a set of things and make as big a splash as you want That's just marketing. It's something you can choose to do Second thing hey nine months long right the classic arm wrestling with product management if I don't get it in this release It's gonna be nine months plus another nine months till I see it We fight to the death right and there's cage matches We put people on an island sometimes the people that swim ashore get their release features in It's no way to work and at the end of the day What we found is if you deliver incrementally turns out you have a lot more Flexibility and a lot more voting power than you ever did because if you don't get something now You can change your mind a month from now and you still got a chance of getting that done right away and Last you know the worst the worst problem to have Don't you know try not to talk to too many customers because if they give us feedback that they don't like something What are we gonna do with that? We're already full Right worst possible thing we need to go out to the next feature because we promised it Even though we did a crappy job of the last one no good Andrew talked to him about what we did for step two so for Continuous delivery we said you know let's let's try doing what Everybody's doing out of born on the web. We ought to apply the same techniques and you know Kind of extend that continuous integration where you've got the build the publish the deploy automated testing and What we found is just getting through that first part of the The pipeline was really challenging a lot more difficult than we expected and that's what we're gonna take you through a lot More of the details of what happened what went well what didn't go so well and then lastly even then that the next step of Operating that software in a production environment also turned out to be a lot more challenging than we expected and In fact one of the things that we found will tell you a little bit more about it Is that it's not just one box, but we also started to see the need to have multiple production Environments hanging off the end of this chain So just to set this up so you understand a little bit about our roles So I'm the managerial part that means I get to be all rosy and talk to you about how great everything is and everything Work Andrew for those of you who know him is the smart one of the two right so he gets to actually tell you the reality So let's start with ugly truth part. I'm the extinguished engineer. I mean distinguished engineer so So I you know I Like most of you engineer types, you know you just you call it like you see it And the truth is you know one of the one of the first problems that we found was That our stakeholders our stakeholders just that's not the way they roll and they're they still want to play the same game that they've been Playing and and then but you say well wait a second You know what we want to do is we want to take some time out of delivering features to go and Establish this continuous delivery pipeline and and you know build up our automated test cases and they're like yeah Yeah, that's all really nice But can you do these 52 other things because at the end of the day that's what they're being measured by And so that was really a real problem And of course then you get the other one is like okay, and that's fine But then we get this urgent customer request so this battle between the things that we need to do to be successful in this and You know all of the pressures from the external world around us The next ugly truth that we came across was even within our team, you know and we're we're a global team We have folks in China, Italy Poland Germany you name it we have people scattered across even Canada a Yes, that's right. They're a couple of essay But one of the things that we found was that a lot of the developers we'd find out that they weren't actually Writing the tests and we told them, you know You can't check in code until you've got the automated test and the guy would be like well Yeah, but you know the other guy hasn't written the test yet What but you're supposed to write the test no no no that's not my job It's a tester's job and she had the kind of this cast system that was ingrained into you know People's minds and behaviors and so you know trying to turn that way of thinking was was not trivial and actually I think one of the things that Became really good. This is more on the good side. So I probably shouldn't tell you about that That's a managerial job What do we do about that right so first of all deadlines are dangerous right in the sense that there's always this lure that hey You know doing the next feature for the customer somehow more important than making the system that works and that she can update regularly That's that's actually not true. The other guys aren't coming. So they're us right test automation is part of our fundamental job And as Andrew said we had to do a few things kind of in a in a forceful way to make the change happen quicker Just talking about it and saying that's who we want to be Was enough so we literally said okay? Nobody checks in another feature until a certain set of tests are written period Can't be done right and after that in order to check in a new feature It comes with tests or it doesn't go in the build period We even invoked a few hostage scenarios We had some video clips of family members and kids But I wouldn't encourage you to do that in a lot of countries turns out that's illegal Well in some countries in the other countries seal the doors right But you know teach their own whatever works for you just a quick show of hands I'm curious, you know some of what we were talking about before about these different mindsets Are they familiar to anyone in the room? Yeah, so it's not just us not my job. Alright, so we got a bunch of addicts in the room Let's we'll be pumping it never mind So The next hunk of not such happy truth was we did stitch together this Beautiful looking on PowerPoint Continuous delivery pipeline That took six hours and to end Unbelievable, I mean literally for developers to get code in and get it through and especially when you'd have you know one Developer is dependent on another developer getting a change through it could literally be days from the time You you're ready to do something before you actually see it on the end of the pipeline Which is just you know, it just grinds people to a halt and the second thing was that we were just constantly hitting Test blockers where the system would just it wouldn't work You know 50% of our builds more than 50% of our builds in the early days just didn't work So that was you know a pretty nasty situation We even at one point we kind of instituted a manual fire brigade kind of approach where we'd manually have you know One guy in China would kind of crunch through it You know start fixing it and making sure that something's working and then you know He'd kind of pass out out of sure tiredness and then somebody would wake up in Germany And he'd kind of pick up where that guy left off and go until he was you know This follow the Sun thing is not as good as it sounds So Step four take inventory of yourself Right, what kind of automation do you really have so for us? It turns out we had little bits and pieces for sure But as Andrew said we had big blind spots big holes and it cost us I can't tell you out of those 50% or more builds that failed the amount of time We spent a large portion of the team installing deploying that build and beginning their work to realize I can't go any further and I got to go back Right. We you know across the team this distributed across the world you can imagine that adds up in a hurry second thing Continuous delivery in the sense can't mean continuously not available, right? Please come back tomorrow. Sorry system being updated So, you know, we we learned from a lot of the design patterns here in open stack Right this idea of multi-instance and how do you kind of roll through these things and keep your user-facing services alive? That was an important principle for us and the third thing which which I can't understate and I have to admit This is something that we didn't probably recognize as we should The leaders who get this who really want to be this kind of team will not necessarily be the ones you're used to So what I'd encourage you to do is look very very hard for these people and help bring them to the front Right they you might have had a traditional lead who's the superstar in your domain for 75 years Right comes in of the Walker, you know mumbles puts his teeth in and then codes something that you think is awesome But that might not be the dude who's gonna say you know what continuous delivery is really gonna help us get there Find the people who are gonna get you there put them in the light Let other people see them reward that behavior and the rest of the people will follow very very important so this is a Bit of an eye chart of what we did what our environment is so we you know we use our rational We use a lot of our own tools from IBM And in fact there's a RTC rational team concert. That's where our source code is We've got these build engines and that would kick off over to Jenkins Which in turn would go and combine all of the components from the various different component teams And then we put together some really neat stuff one of the things we called Deployinator which would then go and stand up, you know multiple environments Where you know we just have as many servers that we could put together to go and run a lot of these automated tests And we developed a selenium grid. How many people use selenium for for web-based testing great tool You know we found initially that it was pretty fragile and so we actually kind of wrap it in around Sort of what we called a kind of a grid where we just fire them up whenever we need them and you know turf them and at least that Made us very productive But at the end of the day it took a lot of horsepower and a lot of work to get this kind of stuff together So that led us to step number five admit your wrongs and do something about it, right? So one of the first things we learned so we're development people right development people think of course my stuff's gonna work Right. I'm good. I'm paid a lot of money, and I'm worth it Right the reason you have me here is because you know I get it done Truth of the matter is no matter how good your automated testing is at any given point in time and particularly at the beginning when you're Kind of crawling your way to get to that critical mass You're gonna let some things through that you're not proud of right and so the first thing we learned right so open Stack is wonderful at spinning up a system quickly right with an image on it So that's great That means we could deploy a new build that we have that we're proud of or we think we're proud of pretty fast Turns out it's equally good at going the other direction and it's really important that you figure that out first So if you walk a wire between two buildings the first thing you're gonna do is put a really strong net down there Because you're just gonna feel a lot more confident on that wire knowing that if you fall odds are it's gonna catch you Although after a pretty healthy eating experience here in Hong Kong I'm not sure there's a net that can handle me at this point, but for many of you in the audience lean and fit I'm sure it will work So so that was something to be honest. We kind of blundered our way in so we made mistakes in public in front of our users The good news was it taught us a valuable lesson. We needed to react quickly It also taught us a principle of let's make sure we can keep these people up And even when we make a mistake we need to think first about defense before offense And then the second thing and again, this is something we've seen actually done very well in many places in open Stack itself, which is you know the idea of the feature switch or the config setting right? I'm not yet confident in this. It's not fully cooked looks great on the outside The icing is good if you put your thumb in the middle quite gooey in the center right not ready for eating But we want you to mess with it So it's there, but it turns out if it's still not quite done flip the switch go back to what we know works for sure And we'll gradually go live with it when we're ready So the idea of these kind of strangler patterns sort of choking out the old features when you're confident in the new ones and Allowing this kind of continuously available system was very important which brings me to some more ugly truth so we said You know so now the test automation. It's part of our lives go do it Unfortunately That's not enough direction for a team in fact We we even had kind of one of the guys you kind of let it. He's like yeah, you know just just go and do it right tests You know when you have a team of you know hundreds of people scattered across the globe in different languages and stuff in Italian How do you say right? Just do it. What what does that mean? So, you know, we had to get a little bit more prescriptive and start telling a lot more specific How do you do it? How do you write good test cases? In fact? That was one of the other things really you know of all the tests that were being written how many of them were actually good and that would would actually work from iteration to iteration So it's very easy to make poor tests and and fragile tests So we hadn't in no other way was but to invest in learning it and then communicating that back So that others could benefit from it because you know at the end of the day when we just kind of let it loose You get anything from awesome to oh my god a drunk man. You just like when wrote a test anyways So that led to step number six remove your defects Right find out what's wrong. So the first most brilliant inspirational moment for us. Hey, it turns out test automation is code What a shock and I you know it sounds silly But I'll tell you we didn't treat it with the same respect and we paid for that just as Andrew said You know we had build breaks, you know red builds constantly that we would go and look and say no actually that was fine Just the tests were bad, right So lots and lots of lessons here very practical ones first of all right treated like anything else you do So if you do code reviews do code reviews if you you know have something around You know promotional system go through some gates do that for your test automation just like your features Step number one second thing we introduced very late by the way the concept of what we call a non-voting test case And I think that concept exists here in the open-stack efforts as well Meaning that hey, this is a test that is not yet robust enough to stand up and give me solid results under every case But it's necessary. I do want to run it, but it doesn't if it fails. It does not stop the train Right so having this idea of saying these are in progress And these are sturdy and a benchmark for me to say green or not really really critical right and having that that two-class system Second thing tests are atomic. We had a lot of folks as Andrew said, you know We started with this wonderful idea of well, you know, we got professional folks all we got to do is say dude Right test automation y'all know how to do that right go do test automation I Is a little even more inspiring than that. It's hard to imagine right, but I mean I can really whip into a frenzy So, you know it turns out you let the flowers bloom and you get a lot of weeds So we had lots of cases where people wrote a test case that said well, you know I mean the one I wrote three things ago surely that's run and left behind some stuff that this test case is expecting Well, not really turns out when we mix and match those things That stuff didn't work. So they need to be self-contained third test can have allergies meaning that you know sensitivities So we found things selenium is a good example that right where we're looking for something on some part of the screen Well in the meantime, we do a little user feedback the user say I universally hate you And I will kill you and everyone who looks like you plus your family members just to throw it in It's that bad and we say you're right feel terrible go back and fix it change the screen They love it our test break because where's pixel 47 down in 32, right? So think about that stuff think about the stuff you want to change and make sure your tests have some way of beaconing in on those Changes IDs immutable objects and enforce that right step number four What will you vary we found a lot of wasted effort because one person running a test case would come up with three images that they thought Tested some combination of interesting things the person sitting right next to him literally would write a test case and create three Different images sort of kind of like that, but not really that did this now You can imagine the fun we had with build breaks where we said dude I don't know what you're talking about it works great with mine So yeah, but we had that other image that the other dude put there and when that ran it what what images he use and All right, so at the end of the day these things you know your variation points your environment Your variables your configurations your content right have a strategy for how to do that There is a repository a URL where these are the images that will test all tests are written to loop through that That way we can change our mind if we have some new interesting combination We saw it a customer we throw it in there We run the whole battery and we see if we live or if we die right but that takes effort takes thought takes design This is code right in last load and stress test your system So we have some fabulous engineers here in the room who helped us a lot with this So functional testing is great, but then we realized we need to put the system under stress We need to vary our workloads. We need to simulate lots of users hitting this at once Simulate concurrency. This is where the really fun problems exist So when we got better at doing the basics we started to layer on the more advanced things and again It pointed out a lot of crocs in our foundation very helpful Step seven then remove your shortcomings with humility What this meant is Andrew talked at the beginning about this six hours, right? The from the moment I put my line of code somewhere then we churn through this incredibly long pipe So that was you know the Alaskan pipeline when we really needed one from about here to the front door, right? So how do we shorten that first of all make it a lot easier for people to Sample their work sandbox build how do I take my one little change and Incrementally test it against something. That's the known content so that I got a pretty good chance Then unless my buddy next door checks in some conflicting change. It's gonna pass tonight. So I don't get bothered right awesome Test as a service. So our initial version of this was done in a you know more manual way Load these things in your workspace try to run them. That's pretty expensive And again, we found cases where people had very different versions of things that got very different results So why not apply a cloud to this problem, right? If these are the tests I want to run then I can provision the tests run the tests that I want put them away Voila consistency on demand. This is what clouds do right and then last publicize your results I mean this is what we do right builds are like breathing so We wanted a way to have peer pressure be the mechanism So, you know, there's a website every member of the team understands you go out and it says here's the last good build Here's the one that was done before that here's all the tests and that one is wrong That one failed right and everyone goes sweet Edmund nice Right, and then we have a pack of people with torches who follow into the garage and beat them unconscious But that's because we're filming that's only because Edmund lives in Texas, right? That's perfectly acceptable. We don't beat people in other places, but in Texas. That's how it gets done and Edmund You break it again. I swear to God. I will pull this car over and you don't even want to know all right So that takes us to step Pete Apologize to those you have harmed right not not Edmund not Edmund. I'm not apologizing to you Edmund You broke the bill. I'm gonna One more Okay So for this one we said, you know what? We have an interesting problem as well, which is and and I'm sure this doesn't happen anywhere in the room I think this is unique to us, but on the drawing board. It looks beautiful. This is a work of art pure Picasso with the sideways nose And everything just the way I love it and when I translate into code I run my test and I hang it on the wall And I say man, I love that turns out Sometimes when the paint goes on the canvas it doesn't quite turn out the way I had it in my head and in front of our customers It yes, it does work. Yes, it can be made to work, but in practical terms It's awkward. There are some things you didn't get right tangible things About what it takes for me to keep it running not to get it running, right? So we decided you know what the first instance of this continuous delivery pipeline is gonna be for us So we forced our team and members of our team to use the product. We were building every day why we were building it Right why because we wanted to learn from ourselves And we want to make sure we didn't put something in front of our clients that we ourselves were not capable of using That's pretty embarrassing and secondly It was a way for us to get this system working built tested in a little bit less public way Right, you probably don't want to go live with your continuous delivery pipeline in front of your banking application for your customers I mean you might because you're you know Good and really confident But this might be an alternative for you And then the last part Andrew talked a little bit about that pipeline and said that last box called production Right. Well, that's the last step right go live bring the bring the new change up. Well, it turns out as We got into this more and more we realized there's lots of different ways we go live We go live for our internal cloud. We go live for a beta site It's hosted that our customers can try things out and write little Facebook like comments on the wall You suck or that's awesome, right? Hopefully more of the latter than first And you know, we want to do these things for multiple development teams in multiple geographies Ultimately, even some of our clients started to see this stuff and say, you know I can envision a day where I actually subscribe to the same pipe and take updates to your product from you in that Same way still ahead of us not quite there, but again, you can imagine it So we really adopted this idea of a publish and subscribe meaning that our product had the idea like a yum repository Right look for the changes pull them on a schedule or policy that that particular instance requires If it's the internal cloud I want to update it daily if it's a beta site Maybe once a week because updating every day is not good for the customers. They can't keep up with their stuff, right? Might be different for a production customer says I got a change window every 30 days. Who knows but this idea of sort of done once Placed into an approved repository and then pull to multiple instances actually turned out to be a very powerful concept Well, and ultimately it's giving the customers the choice, right? How how do they want to get me? You know, who knows there may be some customers that are still happy with those old school ways of you know Every six nine months or whatever boom major upgrade But you know for those other more progressive Companies that want to get the latest and greatest there. They're happy to do that too, right? I think choice is a pretty important piece Which leads to step nine just make direct amends except when doing so would harm them, right? And so this is feedback right this says that again No matter how well you do at the end of the day You should assume that sometimes things aren't going to go the way you want and you need to make sure that in order to save Your face right to make sure you're strong in front of your users that they have a quick way of talking to you Let me know right because as long as I think you're listening as long as you could do something about it You can use that safety net if you need it or you can roll forward quickly Things are going to be fine And they're going to be a part of your community people who are helping you make themselves be more successful If you don't listen and if they can't talk to you you are the enemy You are the people who are in the way of them getting their job done and this whole thing is going to fall, right? So let them talk to you talk back. Let them know you're listening And at the end of the day make it easy because let's face it, you know surveys and all that other nonsense None of us have any time we got three minutes a day That's free and that's for me walking from here to the bathroom, right? So if I can sort of Bluetooth you something that's about all I got So keep it short and keep it sweet Well as you can imagine there's a little bit more of the ugly truth and and frankly You know it's great when we see all these born-on-the-web type apps out there And they're awesome, and they're taking advantage of cool things like, you know Netflix Simeon army isn't that awesome But how many of you out there like me are addicted to legacy apps that were built some time before The this, you know kind of web world anybody out there? Yeah, okay. They're a few of you And this is the thing is they just weren't designed for This kind of rolling updates and this kind of incremental, you know Maybe take down parts of the system while you're upgrading other parts, but overall the system stays alive. Unfortunately We're just not all there yet And so you know in some ways we started trying to take advantage of a lot of the techniques from OpenStack Or that you get from an OpenStack world But you know in some cases we use kind of more traditional HA techniques and and you know even using things like DRDB and you know Kind of the traditional style for traditional clustering, which is fine At least that kind of gives Our customers something reliable while we're incrementally evolving our system to take advantage of the more modern approaches to develop To develop our apps, but that takes time and that takes change Which leads us to step number 10? Continue to inventory yourself and deal with your shortcomings. So as Andrew said, I mean the trick here is Don't wait, right? So we all have these applications. They're a little bit of a Frankenstein whether we like it or not There are parts of the system that you say wow, this is nice We've been able to renovate that that part of the house. It looks pretty good We're happy to have the guests come in and it kind of does what we want. We never let them go in the back room Right because the back rooms got dead bodies cats that you know, we haven't seen in two years, you know food Stuff I don't even want to talk about right so that's that's just the nature of our applications You can you know get caught in this idea that well until I redesign my application in an open stack like man or multiple Instances separate out my data, you know, it's just not ready for this sort of live updating Don't wait. Don't wait fact of the matter is as your application as you renovate one room of your house at a time Your system will get better and better and these short downtime windows will disappear and that's what you want So initially it may be that you say well I got to be a little careful when I make changes to one part of my system because it's not very well designed for that And I have to take the system down for 10 minutes in order to bring it up. Okay, not ideal But get started do it and then fix that part of the house so that the guests can kind of wander around on their own All right, and the last part is lights out operation. I'll tell you this is another one We underbid so Andrew's right in the dev environment. We found that oh, you know It's great to be a developer because we have no rules at all Right, I mean you can whip stuff up you could tear it down You can put whatever you want in there you can try something different if you don't like it you scratch it reboot Whatever, right? I mean, it's a very flexible environment the minute we crossed over this very thick line where he said, okay Love this change. It's awesome. We're gonna put it in production for our users, you know We walked there's like a little trip wire sort of a laser beam an alarm goes off a bunch of people wearing black Flight through the windows put you know tranquilizer darts in you and then you examine you for security threats, right? So the rules take over and I guarantee every one of us has them, right? We have these procedures We have to follow we have auditing we have HIPAA regulations. We have Lord only knows we all got them these scrolls of You know ancient text from the Roman Empire that you know has been translated into it So the rules matter and there's no way around them you may evolve, but you'll evolve slowly So what we also figured out is It's not only important to automate the deployment of the application and the change to that application We also got to automate the wiring of that thing to these procedures and these tools and management practices That our company requires because if we don't all that fabulous flexibility all that speed We build for ourselves comes to a screeching halt and days or weeks literally can go by Right, so we used a lot of workflows and other things that we had in our solution in order to put those things The put the wiring diagram in place and it allows to snap those things up but maintain the auditability and compliance that we needed Very important which is frankly one of the reasons why we built this product exactly right So last step 11 Meditate to continue forward right when you found inner peace Continuous deliveries working for you the features are rolling steadily out the door. There's music Incense perhaps maybe something stronger just saying don't advocate it. No it happens You know the other thing that we learned and this was my own change right as a manager, right? What am I trained to do? I'm trained to walk around with a clipboard and track because we love to track. Where are you how long is that gonna take? You know, do you really have to go to the bathroom now because I really need this checked in What are you doing and we tap people with pencils and stuff like that? It's actually a lot of fun So it's kind of hard to put that down because I enjoy it a lot But it turns out in this world what you're really trying to do is if you're gonna change your behavior You got to change what it is you inspect right and what we really want to do is I don't want to know pace I don't know how fast you are doing something when is this going to be done the classic question that pretty much every developer hates Instead what I want to know is who's using the stuff? What are they telling you? What did we do about it? How well do they like it? What's the next thing we're gonna do? Right if we can shift ourselves into that mentality then everyone's thinking forward everyone's thinking positively, right? What can we do next did we get it right or people digging it and what's next right? Power powerful psychology. I promise you this will make a huge difference makes my job lie these here Right, so it doesn't mean that speed doesn't matter. We don't live in markets by ourselves. There are competitors all that stuff is fine There's people who are angry with you almost always But it's different it's speed for the sake of being delightful to your customers and that's what matters That's a different way of thinking Which brings us to the last step so step 12 you get through the entire program your addiction is broken Right the last thing you do once awaken carry your message to others in despair Which I think is why all of you are here in this room and hopefully we've we've had some success in doing that I could give you some examples so in six months since we implemented this Literally and you know because we're IBM and Lord knows we love measuring stuff We got more metrics than you know, we're measured by the kilogram of metric apparently it's part of my appraisal system So but literally we got you know reduction in labor the meaning the kinds of things We usually had to do for build for testing for things like that seven to one and and believe me There is a lot more we can do a lot more we can do we are far from the end of this journey, right? More than 3,300 builds in that length of time and the amount of time that it took us to generally resolve some problem Went down by more than half right That's you know, and that's not small change. It's not it adds up day after day after day Now we'll tell you like any addict We still have our days where we kind of Fall off the wagon we fall off the wagon the stress comes such-and-such customer called Such-and-such competitors doing something we got to do something today. Can't you just do this one special thing, right? So you're gonna feel that pull don't be afraid of it. Just remember find your partner as I do Talk each other down right put the glass down Let's go back implement the feature make sure the test automation is right Deliver it as soon as it's ready, and we're confident in what we've got Okay So that said thanks everyone. Thanks at your time. Hopefully this will help. Good luck on your journey