 story about our journey of continuous delivery. Before I do that I'm gonna tell you some things that I'm gonna tell you in that so you can follow along a little bit about me and where I come from and then these couple of stories. So first of all to start off I'll tell you what a continuous delivery means to me and then I'll tell my two stories and then I'm gonna give you something to take home and remember. So before all of that yeah that's me if I wasn't here that's what I would be doing somewhere. I'm Susie Prince I am the head of product for ThoughtWorks product group. I have been in software for about 10 years primarily doing product management and I'm currently the product manager for a continuous integration and a continuous deployment tool called SNAPCI. I'll be talking a little bit more about SNAP later on and some of the other products that we make but if anybody's interested in that you can talk to me about it later and if you want to complain at me or say anything you can find me on Twitter. I work for ThoughtWorks and ThoughtWorks is quite a big company now. We have 4,000 people in 40 offices in 14 countries and we're fairly diverse because of all of that and hot off the press I wanted to share this with you because it's particularly important to me is that this week ThoughtWorks was announced the winner of the top companies 2016 for representing women in tech basically. It actually is really exciting and ThoughtWorks have given me a really nice home for the last 10 or 11 years so I feel really proud about that. Some of the things that we do at ThoughtWorks we provide professional services but outside of that we're actually really big proponents of software excellence in the industry and we like to give back as well as you know make a living. So some of the things that we've done a lot of my colleagues they write books, they do speaking events and we also build some products some of which are open source and you may be familiar with. We made the first continuous integration server cruise control which I think I heard somebody mention yesterday and the first continuous delivery server with pipelines. It was not Jenkins 2, it was GoCD. I can also talk to you about that later as well. And something that might be of interest to most of you here is that my colleague Jez Humble and Dave Farley they were actually ThoughtWorks when they wrote the continuous delivery book and if you haven't read it I'm going to just barely touch the surface of all the good things in this book so I would highly recommend that to you. Okay so that's enough about me and all the ThoughtWorks-y stuff. Let's start with telling the story. So I wanted to start off by telling you what continuous delivery means to me so that we can all understand each other when I go through the rest of the series of stories so that we sort of have the same baseline. I'm going to use the word CD probably but interchangeably I may say continuous delivery when I say that I'm saying the same thing. So continuous delivery is the ability to get changes of all types, feature changes, configuration changes, bug fixes, patches, experiments, all changes into production in a safe, quick and sustainable way. That's quite a mouthful so the two most important things are all changes into the hands of your users safely, quickly and sustainably. So what does that really mean because that's just like this phrase. To me it sort of means like feedback cycles. It's basically the ability to put something small in the hands of the person that's going to use it as quickly as possible so you can find out if it's the right thing that you should have done. So as a product manager that means was it the right feature to build like is it going to be what my business needs to do to enhance our product. But for other people it might be those when my code the right code or is the configuration correct like it's basically the ability to get this feedback as quickly as possible. And it sort of looks something like this. It starts off I don't have a laser but at the top there that first loop that's the continuous integration loop basically starts off with checking in your code. I recommend to mainline trunk master whatever you call it into your main source repository. You build it test it and you get a result and that result was your first bit of feedback. And then to build on top of continuous integration is the part that I call continuous delivery, which is you're continually putting that same piece of code that's more change into different levels of testing. They could be integration tests, they could be staging tests, they could be user tests, they could be into different environments. They're basically different types of ways of getting different levels of feedback. So you go through all these stages and the closer you get to production, the more confidence you get the risk is reduced that we're going to put out there doesn't work. And you you're more confident when you go to production. So it's basically levels of feedback on top of that first continuous integration cycle. And we call this these stages in a deployment pipeline. And I'm going to distinguish continuous delivery continuous from continuous deployment. I'm doing that because I heard sometimes in the open spaces people asking questions. I think for the rest of the talk it doesn't really matter if you want to believe what I'm doing with continuous deployment. I think the learnings I want to share are relevant to that as well. But I just want to be clear that to me the real difference between continuous delivery and continuous deployment is basically this little finger here in continuous delivery, there's human intervention. So somebody gets to decide when we go to production. And in continuous deployment that would be an automated process. So my feedback loops would go through automatically. Every time I got a green build, or something was successful in one of those stages, it would move to the next stage in the pipeline. And it would automatically be deployed. And in delivery, someone would just make that decision. Okay, so that's that's what I mean when I'm talking about continuous delivery. Now I want to tell you two stories about how our product group at ThoughtWorks got themselves from a state of maybe not continuous delivery to a somewhat better state of continuous delivery. There are two separate stories, one followed by the other, and then wrap up at the end. So the first story is about our product, Mingle. I think most people will not be familiar with Mingle. So for those of you who are not, Mingle is essentially an agile project management tool. You've probably all seen one of these before. This is our particular version of it. It's a card tool. And it's sort of our main view. And it helps you manage your work, put it through workflow. And it's highly customizable. It doesn't really matter that you understand what Mingle is. The main thing that you need to understand for this particular story is that Mingle used to be on premise software. So we basically used to build an installer and we would provide that to our customers and they would install that on their own hardware and run it themselves. And about four or five, maybe six years ago now, we wanted to move that product to be basically a service, a SAS software as a service product. And so that's the story I want to tell you between how we went from an on premise deployment to actually providing, we actually provide on premise and software as a service right now. So the story starts back in 2012. We're an on premise product and we are releasing four times a year, normally on a quarterly basis. And that was driven by our business. We felt pretty good about it actually. We delivered regularly. We're on time. We ran to schedule. We felt good. We delivered it to our business. We went out there. It was all hunky dory. Our process we also felt pretty good about. We had really nice continuous integration. We were doing trunk based development and everybody committed into that main line multiple times a day. We had a high degree of automated testing. And every single commit went through this automated testing suite. I think on a daily basis we automatically deployed to a staging environment so anybody could actually get one of those installers that we built and test it on their local machine if they wanted to. And then on a somewhat regular basis, we actually had a different staging environment. I called it user testing just so you can understand it here that basically went out to ThoughtWorks. So this is to our bigger business at the time, probably about 2,000 people where they could sort of dog food this and use it themselves. This is also I believe an automated process at the time. But we didn't release that regularly to our customers. So we were doing this part of our process fairly regularly. But the final part where we went actually out to production, like I said, was just once every quarter. And at that point of time, we actually felt pretty good about it, like I said. But we started to think about moving to the cloud. And I was the product manager for this product at the time. There I am. That's my doughnut. I started to think about, okay, what is this going to mean for us to have this installed product and then also be out in the cloud? And I probably got ahead of myself. I was talking about feature toggling. I was talking about, you know, A.B. testing, turning things off for some set of users, turning it off, trying it out, doing all these experiments. And we all got like all really excited. And then we sort of took a step back and thought about our process and said, like, are we really ready? Like, can we just go ahead and do that? And we sort of really needed to reassess what we were doing. And one of the things that we found when we looked at our cycle time is that we really noticed that about 25% of our release cycle, so that three-month cycle that we were running to, was actually in that tiny last bit of the process at the very end. The piece that we didn't do very often and actually took up a considerable amount of time, I think sometimes up to three or four weeks in that three-month cycle. We kind of been frustrated by this before, but there was no real driver for us to change it. We thought automating those tests was actually going to be costly and perhaps not worth it. It was only when we were really driven by that business need and the want to be able to move to SAS that we actually reassess that. So we reassessed it and we actually decided that we could automate those tests instead. But automating them and putting them at the end, we actually felt like it wasn't the right thing to do. So what we ultimately ended up doing is we moved from these manual tests at the very end of our process and we brought them forward and we actually put in these installer tests into our process a lot further on. This meant that they happened more regularly and we got feedback sooner about whether there would be a problem. Because one of the things we learned in that three weeks is that actually there were problems. So the three weeks wasn't just testing, the three weeks had been like fixing. So we decided to move it forward, get the feedback sooner and get payback. So then our process looked something like this. So we were obviously now doing a lot more of our process more often. We brought that pain forward. We got feedback sooner and we were doing a lot more of the process before we went to delivery. We still were delivering every three months because for the installed product that's what made sense to our business. And it still does. Actually we still only do that for an installed product every three months. But obviously we've covered a lot more. We've got a lot more feedback sooner. And this little piece at the end now, we actually have cut that down to about five percent. So we do do one or two days testing now at the very end. That's still manual, but we feel a lot better about that. So that was like the big thing, right? We saved all this time. But one of the other things I want to share is that actually we learned something by doing this that we didn't expect to learn. And that was that we had hidden sort of hidden barriers inside our team. So we are a small development team. I think at max we would have been 20 people. We were all co-located. We have opened desks. There's no pods. You know, stand up. We do all all the right things. And so there was no way that we would have believed that there was a silo in our team. But actually what we realized was happening in our previous model is that we get to an end of a release. Product manager, BAs, developers would have done all of their work. And they would sort of move on to thinking about the next release. And we would leave our QA's behind doing some testing. And when they sort of came to us about problems they would find, we would have already sort of moved on to this other world. And we sort of realized that we had this big barrier where they were really responsible for this piece of the process on their own. And the rest of the team sort of neglected it. And I think that's maybe one of the reasons it stayed that way for so long. Because we all didn't have the insight into it being such a problem. And when we removed it, it was actually really nice to like not see them struggling through that on their own. And actually have them in the initial conversations about our next release so that they could highlight issues sooner, provide feedback faster. So I think the point there is that you don't know what you're going to learn until you sort of do these things. Okay, so just to wrap up this particular story, we moved our process to this process. And then when we finally did go to the cloud, which I think we did about six months or maybe a year after we moved to this process, it was actually so much easier for us to take that step. Because delivering to the cloud, we just, we actually didn't need to do those three or four days of installer testing. And we're actually able to get to that process I talked about. Had we wanted to do continuous deployment and automatically go to production, we would have been able to. We don't. We choose to have somebody make a decision when something goes to production. But we had the opportunity if we wanted it to. So that's what our process looks like now. And it's been four years since we moved to this process. And the one thing people often ask me when I tell them the stories, but you spent a lot of time doing the automated tests. Was it worth it? Like, was it worth doing that? I would say absolutely within the first year, the time it took us to do those tests was actually made up by the savings. And now four years later, we're using that same process. It's outlived the people who automated the tests. And it's definitely made up for the time we spent doing it. So the learnings really are there. Yes, I encourage you all to automate. Well, I know all of you will be like, yeah, but like we have this reason why we can't do this thing. So did we. It's not true. You really need to try it. You will save time in the end. I'm sure there's an example that someone will give me afterwards that isn't true. But really, you'll be surprised how much benefit you gain from just the cognitive overload of like not having to keep doing that work or remembering how to do that work. Obviously, you've got to look after your automation, but it becomes a whole, a lot less of a burden, at least it did for us. And like I said, you will learn some things that you didn't know you're going to learn. We learned about the silo that we had. So that's our first story. Our second story that I want to share with you is fairly different. This is about our product, Snap. And I need to give you some context about Snap for some of the story to make sense also because I want you to know about it. Snap is a continuous integration and a continuous delivery product. It's also a cloud based product. So you don't install it on your hardware. We look after all the servicing for you. And it essentially takes a check-in from GitHub and helps you put it through a deployment pipeline, so the thing I talked about before. So in this, you can see I have a bunch of different stages. And my check-in is automatically kicking off those stages and when they pass it moves to the next stage. So the reason I'm telling you this is that this is a continuous integration and a continuous delivery product built by this company that I told you wrote this book about continuous integration and continuous delivery. So one, you might think that we would kind of think we know what we're doing. And that's exactly what happened in the case of the team that were working on Snap. They were all really well informed. They'd been to loads of conferences. Maybe some of them had even spoke at conferences. They were very, very confident that they knew how to build this product and do continuous delivery or continuous deployment when they were in this process. So much so that I think they got way too ahead of themselves before they even had a product. They were just, they were in CD Nirvana let's say. And if there was a check box for like everything they could have done they would have done it because they basically like they did all the things, all the things that you were meant to do. They had microservices. They had RabbitNQ messaging. They had Nagios which is monitoring. They had an automated infrastructure. They had deployment pipelines and they had containers. We didn't have Docker. We had, we have Alexi. But you know, all these things that they'd heard and read that you should do, they had it in there. And they would have got 10 marks, you know, if it was a check box, if there was a cookie cutter template for CD they would have done it. Except for they weren't doing it. You'll remember this description that I gave before of continuous delivery and the two things I think are important, all changes into the hands of your users safely, quickly and sustainably. They weren't doing it very quickly. They had not managed to get to automatic continuous deployment. They deployed about every two weeks. They weren't really doing it safely because when things went to production sometimes they didn't really work. So the users never really get hold of it. And if the users did get hold of it and tell us that it didn't work, sometimes we didn't even know why it didn't work because we just, you know, we didn't have that insight. We had all these monitors but not the right alerts. So we just didn't really know. And so this whole thing where they shoved everything in just wasn't really working for them. And they actually did, you know, took a step back and had to reassess everything. And if my colleague was here he would tell you the whole story about everything that he, that they changed at that time. But I just want to focus on one of them. And that is microservices. And I'm definitely not qualified to tell you whether microservices are good or not. I told you I'm the product manager. But I can tell you this story about why the choices we made were not the right choices for us at that time. And I hope it's useful to you. And the reason I chose to talk about microservices is that a lot of people talk about microservices being essential for continuous delivery. Oh, oh, well, you saw them. Etsy was there, I think, as expected. Yeah, people talk about microservices and they talk about continuous delivery or continuous deployment and how it's super important for you to do that. And I'm not making a judgment call about that or not. I'm mainly asking you to think about is it the right thing for you by telling you the story about whether it was the right thing for us. So we did have this microservices architecture. We had about nine different components. And they were all in separate repositories. And in theory they could all be worked on independently, except for in practice they really couldn't. If we made changes in one part, we had to change another part, particularly around the messaging I mentioned to you before. So although we had this cognitive overload, and one thing I forgot to mention is there were only four or five people working on this product at this point of time and it was only six months old. So it was four people, six months old. They had nine separate components and nine separate repositories. They had all this overload of remembering which is separated here and what's for all of this. And then when they went to deploy, they basically had to deploy them all anyway. So the feeling that they should have had it was overwhelming to them and that's why they did it, but it just really wasn't helping them achieve the thing they wanted to. So we sort of, we went back, we went old school, we put it all back together again. And we sort of let the design of our product and our business needs and what we were trying to achieve lead us to where we needed to be. And so instead of sort of rushing to do something because we'd heard that you should do it, we actually sort of let, yeah, let ourselves drive it out. So we are actually back here now and we do have, I think we have seven components now, maybe six. And let me guess that's my point, it's not to say that you shouldn't do it, it's just that you shouldn't do it preemptively and you shouldn't do it because somebody else told you at a conference like this or in a blog post or something like that. So yeah, my points here with this story are that I really think like some of us have the title DevOps engineer or we're on a DevOps team or a continuous delivery expert and that's great. Like those skills are super important and what we're trying to do is obviously make a change in our organization. But I think we heard this like yesterday morning, our key job is really like to understand what our business is trying to do. You can't just shove all the things in and achieve the ultimate thing that you're trying to achieve. So think about like, what am I actually trying to do in this organization? It's not do CD. It's deliver this thing or make this choice. And that it's a journey, like you're not going to get it right, like you're definitely not going to get it right. But that's okay because you're going to learn some stuff you didn't know and you're going to make some choices and you're going to make different choices to me and that's going to be okay. That's important. So I think I have just a few minutes left. I'm going to wrap up with some takeaways for you. And I don't know why but the doughnuts I thought maybe that would be really nice. If you thought of these as like those little doughnuts. So I got to eat the doughnut at the start and then I had the doughnuts. So you can think about these as like little snacks to take away with you on your journey into continuous delivery. So the first one is I encourage you all to practice continuous integration. And most of you are thinking, yeah, we've got we've got that. That's like 1999 people talking about continuous integration. We're doing it. I really bet you aren't. I bet a lot of people aren't. You need to be committing at least once a day to your main line to be doing continuous integration. And if you're not, it means that there is some code somewhere that is not being tested regularly. And I have quite a lot of things to say about this but I really encourage everybody to think about starting with the basics and that means that your continuous integration needs to be successful and it needs to be working well. Next little snack is that frequency reduces difficulty. And this is really about the idea that the things that are most painful you need to be doing sooner. So that is from our first story where we had the installer testing. It was like a pain in the ass and we didn't want to do it. So we left it to the end. I encourage you all to bring those bits of pain forward. And the more painful it is, the more you'll think about automating which is also one of the things that I think you should do. So think about doing something difficult often and it will actually become easier for you. Next, next thing is, I just talked about this a lot so I won't labor it here but like being, being the best at CD or doing a checklist of CD or following a cookie cutter for CD is not the most important thing. You really need to move up a level and think about what am I trying to do here, what's important. You might not need to do half the things I talked about. But that's okay because it will be your choice then based on the information that's available to you about your business. So I encourage you to ask questions about why we're doing things. Why is that important? What are we trying to achieve before you think about oh, I'd like to use this infrastructure and I'd like to use this tool and I'd like to like put this process in place. I encourage you all to ask bigger questions about what it is you're trying to do. And then you need to involve the whole team. So I know this is really hard sometimes and there are silos but I think we can start by breaking them down by maybe saying thank you to people. Like you really need to think about how can I make this not about me and my role and my team but how can I make this a bigger organizational change and involve people and it can start small and get bigger from there. And I think you'll find like we did, we had those silos we didn't know about. Really, really try to look for those. And then the final one I thought maybe people don't like doughnuts. So this is for the people who are healthy. I want something else. Automate, automate all the things. Do it all automatically. They're much, it's much better than us doing it. Somebody once told me that I have the benefit of not actually having to do it myself. Somebody once told me that computers do anything and I'm such a believer of that. So yeah, don't it's not true that you can't save yourself some time. You will save yourself some time. You will bring the hard thing forward. You will get the benefits. It will give you more time to ask the big questions that I have encouraged you to ask and it's super important. OK, so that's pretty much it. My final thing here is just some things for you to take away in terms of the idea that it's a journey and that it's not going to be easy but it's going to be OK. And that's it.