 Mae'r ddweud eichhetion yw'r ddweud eich harae y fuddefnod yn siarad agfynodig a'i gynnig oherwydd yn dweithio'r cyfo'i gyngliedd ar gyfer y dyfynodig ar rydym ar y dyfynodig a'r hynny'n cael ei gydag i'r cyd- Northfield P hypoc Fynodig i'r cyfo'r cyfnod i'r gyfer y dyfynodig a'i gyfynodig a'i gynnig i'r cysylltu'r dogion gyda inneb. Fynodig i'r gyffodus yr edrych IAM IAM. Ond ym mwy o'r newydd yn i'w meddwl roi'i'i phobl iawn ar y sgol yma, wrth i ddweud o gyfnodion ym gwaith o'r ddwylaethau, yr idea yr ad hyd yn i'i phobl iawn yn ei pan iawn i dwyliadau yn wefynnod o'r dnnun cyflw perfectly. Felly â'r ddweud o'r ddweud o'r ddweudol, a bod hynny'n gilydd i'r ddweud o'r ddweud o'r grannol dw i'r gwrth i'r ystyried yn ddifrif, ac yn ymgyrch o'r ddegyn nhw ydych chi'n gwybod bod yn sgwrdd maen nhw'n ei ddweud yn gweithio. Felly, yn ydych chi'n gofyn am y dyfodol yn gweithio'r ddechrau'n gweithio'n gweithio'n gweithio'n gweithio. Felly, ydych chi'n gallu gwahodd yn ddifrif i'r ddweud o'i gweithio ar y ddegyn nhw, ac yn ymddangos, ac yn ymwneud ydw i'n gweithio'n amser. Mae'n gofyn i ddadw'n gweithio ar yr ysgol sy'n mynd iddo i'w wneud ei wneud wedi'i yn gyfeirio'rῆs er oeddwn sydd wedi'u byw. Felly byddwn yn cymryd o bwell o'r cyfeirio'n cyhoedd llwydo ac oed beth e'w chir pa yna wnaeth wedi'u hynny fel yw'r cyfloguaeth a'r cyfeirio'n cyfrwg sy'n gwych ac yn gydweithio hyn ar hynny rŷ. A esoie yna'n ddod cais ei wneud o'r meirio'n cysun. I have got a presentation, as you can see, I'm going to talk to that, and mostly this is going to be me talking. I'm not going to be doing lots of different workshop exercises as part of this. I'm going to be talking about some of the ideas that you might want to think about. But please do feel free to make this interactive. Please do feel free to interrupt me if I'm not being clear or if you'd like to ask a question about what I'm talking about or for any other reason. I have a very bad habit of talking too much. And if you let me, I'm just going to talk at you for an hour and a half. So let's try and avoid that. Let's try and make it interactive and then we'll all have more fun as we go in. So I showed this picture in my talk earlier today if you were there. And this is the kind of the stereotype that I tend to use of a deployment pipeline. This is actually the minimal deployment pipeline that I think of. I think that in order to qualify, I can't imagine writing any project so simple that it wouldn't need this. This is my minimum deployment pipeline. And the idea of this is that fundamentally this defines the core of my testing strategy. I'm going to want to give development teams fast feedback on the work that they're doing. We want to be able to evaluate every change as quickly as we possibly can and get an answer back to the development team saying, yeah, that's good, carry on. But that's not enough. That's not enough to be able to release the change into production. We also need to know that the software does what the users need it to do. That it's deployable. And so we need acceptance testing as well, which is a different kind of testing. I'm going to talk a little bit more about those things as we go through. But we're going to need those in addition. And then it's a deployment pipeline. So the job is to produce something that's ultimately releasable into production and therefore we need to be able to deploy the results of our work into production. So this is what I'd build in the first iteration or the first sprint for the first story on a new project. This is the easiest time to build this at this point and making your software testable and deployable. The job at the point at which we're creating the deployment pipeline at this point in the life of a project is we want to start as we mean to go on. We want to be able to give developers feedback on those early stories. But we also want to kind of set the rules and the scope of work so that it's easy for people to do the right things. We want development teams to start growing their software based on automated testing, evaluating its releaseability in terms of automated testing. So let's start there. Let's start while it's easy. The last time I did one of these from scratch, I did it as a demo, I did it in GitHub Actions for a very, very trivial walking skeleton of a web project which had a web page, a couple of fields and a database. That was the first story and I built a deployment pipeline and it took me about a day and a half to do the whole thing. So this is really easy to start at this point. Later on, if your system is big and complicated, your deployment pipeline is probably going to get big and complicated. But right now, it's really easy. So this is a nice place to start. And even if you're retrofitting this to an existing piece of code, I'd start pretty much the same. Get these fundamental pieces in place and this is a good starting point for any project, really. It sets you up on a good footing. The more general picture, the picture that I more regularly use is this one and this is based on the deployment pipeline that we built at the Elmax project. And it's meant to show the more sophisticated picture. Actually, the Elmax pipeline was more sophisticated than this because lots of these stages were running in parallel. They were running copies in parallel so that we could optimise to run more tests. And so some of these things, so there were multiple deployed environments in the acceptance test stage. There were multiple deployed environments in the performance test stage. There were multiple deployed environments in the commit stage as well to get feedback quickly enough in this more complicated pipeline. In recent years, I've started to think about this in a slightly different context to that which Jess and I wrote about in the continuous delivery book. That doesn't mean that the continuous delivery book is wrong or out of date, but there are some new takes on how this might work. The first thing is what I've already said. The first phase of a deployment pipeline is what I call the commit cycle. And this is strongly focused on supporting the development teams. The objective at this point is to give early fast feedback and a high level of confidence in the developers that if all of these tests pass, then their change is probably good. So I usually advise organisations that I consult with and teams that I work with to aim for very fast feedback from this cycle. Feedback in under five minutes. Actually, the evidence suggests that 90 seconds is ideal for build feedback. But you have to go quite a long way. If it's a big system, getting feedback in 90 seconds is quite a lot of work. But that's the ideal. But those are the kinds of numbers we're looking for. What we're looking to do is to try and really lower the barrier of entry so that we can get really fast feedback and really limit the amount of pauses in the development process while somebody's waiting for this validation. What we want is that we want developers to commit a change, wait for the results of the commit cycle, and if all these tests pass, they're going to move on and they're going to start working on something new. And so this is one of the most important parts of the deployment pipeline to optimise the fast feedback. This is one of the places where it's worth investing time, money, engineering effort to get really fast feedback at this point. The next phase is something I talked about in my talk this morning is really about determining the releaseability of the change. And in this phase, we're going to do whatever it takes given our project to determine that releaseability. As a minimum, as I've already said, we probably want to run acceptance tests. And these are behavioural tests, usually written, often commonly described as BDD tests. But the objective of these is to define what a user wants of the system. So what we're trying to do is that we're trying to capture a desirable outcome. We're trying to identify some feature of the system that a user would like to see added. And it can and probably should be ideally a small feature, a small increment in the behaviour of the system. But we want to capture that idea, that goal, if you like, that target without saying anything at all about how the system is going to achieve it. So we don't really want these tests to say fill in this form and press this button or call this message or add this column to a database. Those aren't really behavioural assertions. Those are design choices that comes later. At this stage, what we're trying to do is that we're trying to assert an outcome, an outcome from the perspective of an external user of the system. And we're going to evaluate that in a production-like test environment given realistic scenarios, realistic patterns of use. If these tests pass, they're going to say, yeah, that does what you intended. That does what the users wanted to do. I've recently come, again, this is a new way of thinking about this for me anyway, I've recently come to think about this part of the process as a translation process. So what we're going to do is that we're going to start out with kind of a vague wish from the user's point of view, I wish the system did this. That's where we're going to start. We're going to take the next small step into a slightly more formal way of expressing that in the form of a user story. And then we're going to identify a series of examples that if those examples existed would demonstrate that the needs in that user story existed in the system. And then for each of those examples, the next step is we're going to create an executable specification that defines that that example exists in the system. And that's going to guide our development work, is the way that I prefer to organize things. This is straying a little bit off topic, but I just wanted to paint the picture of what it is that I'm talking about when I'm thinking about a deployment pipeline. This is very strongly organized around BDD-style thinking. And there are some things that BDD misses in terms of the testing that we might want to do, but this is a good starting point in terms of thinking about defining things. And it's a good way of communicating and maintaining that understanding of intent all the way through the development team and the development process. And then whatever else it takes, is it secure, is it resilient, is it scalable, is it regulatory compliant, whatever it takes, we're going to try our best to automate all of those things. Our goal is to eliminate any dependence as far as we're able to on manual regression testing, because that's too slow, too inefficient, to be frank, too low quality compared to running tens of thousands or millions of tests of different kinds. So we want to eliminate dependence on manual regression testing, but that doesn't close the door on other forms of manual testing. So we might want to include manual exploratory testing as part of the development process, as part of the development team, happening in parallel with the developers working on the changes, looking for things like is the software nice to use, are there any dumb things in terms of UI layout, that kind of stuff. But those aren't really gates. We're going to automate all of the gates in terms of releaseability. And the job of our deployment pipeline is to be definitive for release. If the pipeline passes all of its tests, this change is releasable. That's our goal, that's what we're striving for. The last phase is the production phase, and this might be very simple if you're just building a simple web app, you might be just dropping in a container and starting up a container somewhere in the cloud, or it might be incredibly complicated. Netflix, for example, track the terminator on the planet. They look where it's nighttime, and then on an automated basis, they choose data centres where it's the middle of the night to release new changes. They constitute something called a canary index, which is a measure of the success of the change that's tailored to that particular change. And if that canary index, if you don't meet the goals of the canary index in the middle of the night, then they pull the change before it starts to become primetime in that time zone, and they stop it being rolled out in other time zones. That's pretty sophisticated stuff. There's quite a lot going on here. The other part of the production cycle, at least in the way that I think about it, is the observability of the system. Monitoring the system in production, I've just described what Netflix do. That doesn't work unless they're monitoring what's really going on. I quite like site reliability engineering as a model for this. I don't quite like the way that people often talk about site reliability engineering quite as much. Site reliability engineering, which is Google's approach to DevOps, is extremely good. But mostly when people talk about it, is technical measures of success. Those aren't the only kind of experiments that we want to do in production. One of the ideas that I really do like about SRE is the idea of service level indicators and service level objectives. Measuring the success of the developed delivery of your software is really, really important. It's hard to find generic measurements that make sense. The doorometrics of stability and throughput are sensible generic measures, and you probably should build those into your deployment pipeline. Stability is measured by change failure rate. How often does you release a change in production, introduce a failure into production, and meantime to recover? How long when you discover that change does it take you to get back to a stable, fixed result? The other doorometric is measured by lead time. How long does it take you to cycle around here? How long does it take you to go from commit to something releasable? How often do you exercise that by deploying into production? These are measures of quality and the rate at which we can produce software at that quality. These are really good things. They're pretty generic. They're hard to game. I would recommend that those are a good choice. Beyond that, it's pretty hard to generalise. For some kinds of change, the site reliability indicator, the service level indicator that we're interested in, might be something like op time, or latency, or throughput, those kinds of things. For others, it might be how much money do we make, or how many customers do we get signing up, or the kind of more business-focused measures. What I'm saying is that this is contextual for every different feature, not just every different project. In an ideal engineering context, for every feature, we say, we'll know that this feature is successful when we measure the success of this thing based on this scale. How many users we get, how much extra money we get, whatever it might be, how many sign-ups we get. Then we're going to set an objective. If we can hit this number, success, if it's not, it's a failure and we've got to do more work. That's a service-level objective. This is a really good model for monitoring things in production. This might be part of the production cycle. This might be quite complicated. The pipeline in the... Yes, certainly. I went back too far, but carry on. I think it should be production-like, but not production. For exactly the reason that you've just pointed out, what many teams do... I think many teams in this context, from my perspective, optimise for the wrong thing. They optimise to minimise the amount of work that they do on testing. No. Test everything that can go wrong, and that minimises your work in terms of development. That's a better way around. I think that just playing production data through the system is not good testing, because usually what you're doing is that you're taking a cut of production data and you're just winding it through the system. When you think about that, it's just kind of a sampling exercise, and it's a sampling exercise of reasonably normal cases. I want to start testing things when things are under stress or when things are going wrong. How does this component work when it can't make a connection to these other components? I want to test those sorts of things, as well, the failure cases. Those are the points where most production failures happen. The commonest place in code, this is based on academic research, the commonest place in code where systems fall over in production is in exception handling. The commonest line of code at that point says, we should do some exception handling here. Testing that kind of stuff is important. Evaluating those kinds of exceptional cases. For that, I think that you need to get the system precisely into the state that you want to be in to be able to then exercise those behaviours. Testing those corner cases I think is important, and just replaying production data doesn't do that well enough. It should be production-like enough so that the scenarios that you're playing are realistic, but that's not the same thing. My preference, certainly for acceptance testing, is to always use synthetic data. Stuff that's inspired by, if you like, the production data, but he's not the production data. It's generated by the test at the time that the test is running. There are some tricks to do. This is probably beyond the scope of this workshop, to be honest, but there are some tricks to doing that to use the data to isolate tests so that you can run them in parallel and all of those sorts of things. I might try and just point those out if I can fit those in when we're talking about optimising things. I pressed the button too many times but there you go. So those are the cycles. As we start working in this way, the deployment pipeline becomes a more and more important strategic resource. We're going to use the deployment pipeline to be definitive for release. All changes flow through the deployment pipeline. My preference is, if you're going to upgrade the operating system, it goes through the deployment pipeline. If you're going to reconfigure your database service, it's going to go through the deployment pipeline. If you're going to, I don't know, change the schema in your database, it goes through the deployment pipeline. Every change to production goes through the deployment pipeline and in my ideal situation is that there is no human access, no human right access to production systems. So this is the only way that you get to change production is ideally the way that you exert best control in my view. And that means it's important because essentially development stops when this is broken or too slow. So thinking about it in those sorts of terms, we're going to need to start worrying about keeping the deployment pipeline working and keeping the information flowing through it. Here are some of the common places that might be problematic when we're building our deployment pipeline. So a slow commit stage will have an impact on development because this is what drives the development process. If our commit stage is slow, developers are going to start backing off and committing less frequently. That means that the size and complexity of the change goes up. That means it's a bigger deal when something goes wrong. So keeping the changes tiny and small and incremental is the better way to go. My preferred way of working is with test-driven development to write a test, run it, see it fail, write some code to make the test pass, run it, see it pass, refactor the code and the test and at that point I'm probably going to commit the change and trigger the pipeline. So I'm probably committing just me as part of a team. I'm probably committing once every 10 or 15 minutes just me. So the whole team is going to be producing change frequently and so we don't want the commit stage to be slow because that would if it takes more than 5 minutes to get the result back from that commit stage, I'm going to wait for longer. I won't commit to the pipeline yet because I'll have to wait too long. Oops, sorry. Next, slow acceptance test. These are big complex tests. The aim of these sorts of tests is to deploy the system into an environment that's like reality in production and to evaluate it in life-like circumstances. These sorts of functional tests people have been trying to do these kinds of things for literally decades and often with poor results because they're hard tests to live with. So there are some things to get right to make that work but also they're complicated so they're going to be slower, they're going to be more costly in terms of time. And this is one of the reasons why I tend to break them out in the pipeline as a separate stage because it gives me opportunities to the fast stage, the commit cycle differently to the acceptance cycle. And then there's failing tests. How do you deal with failing tests and intermittent tests even worse? How do you deal with intermittency? And last of all, there's changes to the pipeline itself. If I introduce a change to the pipeline then it's not properly version controlled and managed and so on and I can't step back to safety and it breaks, I've just compromised the whole release ability of the system. And for a big team that's a really big deal. I've just stopped everybody working. Not only have I stopped them releasing I've really stopped them changing things as well because it's very scary once you get addicted to this stuff to start changing stuff without the safety net of all of those tests running all of the time. So let's look at those in turn. So the commit stage is here and these are the kinds of things that are very common in building a commit stage. We're going to probably need to compile the system if it's a compiled language run some unit tests of some kind maybe some analysis tests verifying our coding standards or data migration tests or whatever else and at the end of a successful commit build in the commit stage we're going to build the installers the installed images of our system docker images Kubernetes images DLLs executables JAR files whatever our deployed unit of software is we're going to build it and from then on that's what we're going to deploy and test in all of the rest of the stages. So the idea of feedback cycle for this is under five minutes as I've said a few times now and so what can start to compromise that? Well first compilation if we've got a compiled language and so one of the things that we can do is that we can spend time and effort and money on making sure that our build scripts are efficient. There's something weird in software developers we seem to take our brains out of gear sometimes for some kinds of work and this is one of the pieces of work that we seem to take our brains out of gear. We might agonize over by building nicely structured well designed code we build spaghetti meatball mash of build scripts that are inefficient. I consulted with a client a couple of years ago and their release process was to run the build seven times because there were so many circular dependencies in it that the first six times didn't produce a deployable result. So it was only after seven times of running the build that all of the stuff was compiled correctly which is just insane if we're trying to be efficient. So thinking about rewriting things is worth doing. The other thing here is that as I said this is one of the most valuable pieces of real estate so buy faster hardware if it takes this. I worked for a financial trading organisation to be fair they were pretty wealthy and that we had a problem with a very large C++ build the build and the running the test took nine and a half hours and so they did an overnight build. Every morning they'd come in and they'd look at the results of the build and they'd cherry pick the components where all the tests had passed and they'd allow those to be released into production and discard the components that had failed any tests. That was okay as long as the bits that were green didn't depend on changes in the bits that were red and sometimes they did so they were releasing broken code into production sometimes. We did all sorts of things all of the things that I'm talking about here we did all of these things to try and optimise this build we bought faster hardware went to my boss and said we need bigger hardware for the build and he said okay how much is it and I said it's going to be $150,000 and he said okay which surprised me but we built really big hardware we ran things I think it's one of the things I say later we ran the build off RAM disk because that was faster than SSD or any other way that we could do it and C++ builds our IO band and it was worth it because in the for the three years that they'd been running the overnight build I was told that there had been two occasions maybe three occasions when the whole build had gone green or every test had passed in the first two weeks after we got the faster build including the faster hardware we got two fully green builds and from then on after that every day we had at least one release candidate where all the tests had passed and we just picked the newest release candidate and we'd released that the following day just optimizing this stage with no other changes is a fantastic tool for improving the quality of the system so invest in it buy faster hardware do whatever it takes parallelize the build modern hardware is phenomenally good incremental build techniques are phenomenally good start to parallelize things and we can go really fast you can think about the solution that nearly everybody jumps to is to think about decomposing the system into more independent modules but this is actually more complex than it looks because the danger here is that you find yourself independency hell I had another client that I worked with over a series of years who were building some complex system they were building scientific instruments and they were building everything from the firmware on the devices to apps in the cloud and everything in between and they they said there's no way that we can do our building the time scales that you're talking about so we're going to break it down into a series of services put them into separate repos work like that and they did that and they got the build feedback from those things fast but for the next 18 months they were fighting a losing battle of trying to identify a change set that all worked together and that was taking weeks and months to do so they were still not in continuous delivery world in terms of being able to produce something releasable every day so if you want to try if you want to try and get there be very wary of this as a solution the way that this works is when you can get independently deployable pieces but if they're all together I think the easiest strategy is to optimise the build to just get the feedback and if you use the sorts of techniques that I'm talking about here you can do massive builds in 5 minutes with most languages look to modern build systems the incremental build systems and distributed build systems that are available now are phenomenal Bazel and Book are respectively the build systems that are used in Google and Facebook Meta now and what certainly Book does is it has a distributed cache so if a developer has compiled a particular piece of code on their workstation that's now in the cache and it's available so it doesn't have to be built again and that's reliable so these can be extremely efficient for extremely large builds Google does trunk based development in a single repo with 9.5 billion lines of code in 25,000 developers so you can scale quite effectively if you apply the engineering effort to this try running try running tricks like measuring things and running the build from RAM disk or whatever it takes to just try and optimise the build understand what's going on you might need to bring in some expertise to do some of these things but this is all a worthwhile investment if you've got a large build and this is part of a problematic for you because this is a good place to speed up certainly older languages like C and C++ are amenable to these sorts of techniques because they were designed a long time ago and the compilation is slower than most more modern languages the next thing that's likely to be slow is the tests themselves and so the first piece of advice is look to the tests try and figure out whether they're fit for purpose often tests are written as a kind of afterthought and one of the downsides of writing tests after you've written the code rather than writing the test before you write the code as in test driven development is that the tests tend to be more complex and more coupled one of the big advantages of test driven development is that it forces us to write testable code it's kind of obvious really because why would we be stupid enough to make our own lines harder by making the test the code that we haven't yet written hard to test of course we wouldn't what we're going to do is we want to make our lives easy by at the point at which we write a test we'd like the code to be easily testable and that means it drives the it reduces the complexity of the test and the code and testable code also has many of the properties of high quality code it's more modular, it's more cohesive better separation of concerns better abstraction, looser coupling all of those things that are the hallmarks of high quality in code so we can use the testing as a tool to drive better design but we can benefit from the better design by making our tests more efficient as well as a result because our code is more easily testable commit tests on the whole the vast majority of commit tests are best captured as pure unit tests pure unit tests don't talk to disk don't talk to start of the processes don't start the application certainly don't talk to a database don't talk across a network they're in process with the code that they're testing they're right up against the edge of the code and they evaluate the code in controlled circumstances and that can be blisteringly fast for these kinds of pure unit tests on modern hardware you can easily run hundreds of thousands of such tests in five minutes easily and probably many many more without doing much more than just sort of throwing them in and running them so pure unit tests are the ideal kinds of tests for this stage there's a lot that they're not going to do but what they are going to do is that they're going to validate that the code is doing what the developers think the code is doing and that's a good step forward because nearly 60% of production failures are caused by the common mistakes that we as developers all make in software off by one errors and condition of the wrong way around and scoping problems with variables all that kind of stuff we just eliminate those kinds of failures when we do test driven development and do these sorts of tests there are times though in the commit stage where as well as wanting fast feedback we want a high level of confidence that if all of these tests pass and we let the pipeline continue then what we want is that most of the time the pipeline is going to say yeah it's all good so my general advice is that we're looking for two things from the commit stage we want feedback in under five minutes and we want to achieve about an 80% level of confidence that if these tests all pass everything else will be good it's easy for us to measure that in the pipeline because we can just count the number of failures and determine the ratio of the failures so we can monitor that easily but that's what we're trying to do so in order to get that 80% these two ideas are really intention we want a high level of confidence that all is good and we want the answers really fast so we want lots of tests so occasionally there are tests that will tell you something that you're not going to find out any other way that might be really slow and painful so that's okay but accept them grudgingly be very cautious be very aware of just how valuable that five minutes of time is to you this is, as I keep saying this is amongst the most valuable real estate in the pipeline so be very very wary of letting anything in there that's too slow there are some teams that I speak to that just put all of the tests essentially in the commit stage that's fine if you're building a really simple application if you can get the results for your app for everything that determines the releaseability of your system in under five minutes then cool, lucky you personally I'd still organise it with a separate acceptance test stage and run that alongside so that when my tests grow and they become slower than being able to run them in five minutes I can start doing other things and other optimisations which we'll talk about but that five minutes I think is important to try and aim for and you might not always hit it the war story that I told you about the big C++ build the nine and a half hours when we'd done all of our work we'd got the commit stage down to 12 minutes we couldn't do better than that whatever we did and everything else was giving a result in 40 minutes so be careful about how you how you accept slow tests in the commit stage or anything slow really it's worth always thinking about efficiency and breeding the culture in the team of considering the efficiency of the tests to some degree if we're testing code we don't need I don't there's at one level I don't really believe in over testing I want to test everything that can go wrong in my system but that doesn't mean I need to test every variable I might test the edge cases where the values in the variables are extreme or normal cases but I'm not going to test every potential variable there's an idea like chicken counting the joke is that chickens can count one, two, many so use that in the design of your tests if your tests include loops then don't loop for a long time in a test that's just wasted effort why do you want to do that just trying the thing out twice is enough to check that you've got a list or whatever it is if you really want to be extravagant three times but don't loop for hundreds of thousands of times it's just waste use mocks and stubs as a way of faking out expensive interactions and just giving very quick results this is quite a good way particularly when you're starting to test towards the edges of the system to fake out interactions with IO devices primarily that are going to slow you down in the context of a test so you abstract the IO and then you test to your abstraction and get really fast results that way periodically it's worth sorting the tests by duration so you can monitor that in your pipeline figure out which tests are slow and then have a test performance day have the team focus on just taking them off the stack in order and trying to find a way of making that test operate more quickly more efficiently the slow tests and you'd be surprised how effective that can be if everybody's just focused on doing that periodically just going back thinking again about how to test this thing more efficiently and that's a worthwhile effort and run tests in parallel again this is difficult if your test isolation is poor so don't allow your test isolation to be poor but otherwise it's really easy if your design is reasonably good if you've not got global variables if you're not sharing state on the disk or something like that in the scope of tests you can run these things in parallel very easily and very effectively and modern build systems will help you to do that as a last resort if you can't find a way of speeding up the commit stage enough then look at those slower tests and just consider maybe moving them out are they really telling you something that's worth the time that you're spending on them maybe just move them out on the later stage of the pipeline where you can afford the cost of those tests a bit more easily and defer running them this is one of the things that we did when we were doing the big C++ build we just we just cheated basically we did an analysis of all the tests we defined any test that ran under two seconds as a unit test and any test that ran over two seconds as an acceptance test in that way it's not ideal I wouldn't recommend it as a starting point but tactically, practically pragmatically in a real world scenario it will move the game on for you the next part of the commit stage testing is analysis tests I like to do various kinds of static analysis as a minimum I'll usually fail a build if the code change doesn't meet my coding standards I usually turn the compiler warning up to the top level and fail the build if I get any warnings in the areas because I want the logs to be clean those sorts of things those are hard to retro... certainly the second one but the first one as well they're hard to retrofit after the fact because you'll find lots of mistakes in the code and you'll be a big cost of correcting them but they're a good way of starting off a new pipeline and that can be really valuable there are other kinds of analysis that can help you you may choose to assert architectural patterns you could say in this tier of code we're not going to allow any libraries that access a database for example in the UIT or something like that but different things and you could have an include and an exclude list of different resources in different parts of the code base or whatever else you cared about in the scope of your tests and validate those and fail the build on those kinds of things during the commit stage this can be really useful but if it's too slow it's not worth it so prefer kind of relatively lightweight lint style checks largely rather than deep code analysis checks these sorts of static tests they need to pay for themselves they need to be finding bugs because if they're not if they're slowing down this very valuable time in the commit stage they're not worth it if they're not finding bugs so they need to be useful any of these sorts of analysis tests because they are more costly usually because they're looking over the whole usually whole of the code base or large parts of the code base as a last resort you can move the analysis testing into the slower stages the acceptance phase of the pipeline rather than the commit phase the other last resort is just decide to bin them if they're not giving you useful feedback if they're not detecting defects then they're not paying their way I'm not somebody who lightly advocates to developers to delete tests because I think that's a really really bad idea and sometimes some developers are too prone to delete tests when they fail that's a bad mistake but in this circumstance if you're taking a considered view and the test is more costly than the value that it's providing that's a really make the hard choice the installers for our software that we're going to create at the end of the commit stage we want to be able to test our software in life like circumstances so we want to rehearse the deployment as though we were going to deploy into production and using the same tools the same mechanisms ideally close to the same configuration in our test environments for this and we want to test that release into production this picture I think is a real picture of a man testing a bulletproof vest presumably the man that's being shot at is the inventor of the bulletproof vest rather than one of his employees I hope so the output of a successful commit stage is a deployable units of software and we want to do that the good news that might take a while to assemble so that deployable unit if it's big and complex if you're doing this for a big legacy system the assembly of the deployable thing is sometimes a costly thing you can look to optimise that you can look to speed that up you can look to modularise it and parallelise it and all sorts of different techniques but for some kinds of technology it can still be difficult to achieve at this point if creating the installers is slow it's a relatively low impact to move this out of the commit stage because this is not the kind of thing that breaks very often you're not going to break the building of the final deployable unit if you've already compiled all the code run all of your tests and it all makes sense this is not usually a fragile part of the build process so you can more easily defer this remember you're still now adding that to the cost of the acceptance stage which we're also going to want to optimise but it's not quite such a valuable real estate as the commit phase there's one of my ex-colleagues Dan Boda who's a very very smart guy and very amusing was bored on a project some years ago and decided that he was going to see how far he could take these kinds of ideas and he optimised the commit build on his project so that he took 10 seconds this was a big Java project and it took 10 seconds to do it all he tells the story in this blog post here I will make the slides available to the organisers so that they should make them available to you but it's surprising how far you can take this if you take this stuff seriously and work at it higher Dan's pretty good at this kind of stuff so this is a rather extreme but you can take this quite a long way so in summary for the commit stage the performance of the commit stage is vital to encouraging good behaviours in our software development process we want people to work in these smaller steps to get fast feedback understand where they are all of the time and the commit phase is what's going to do this fundamentally the commit phase is just continuous integration it's continuous integration plus building our deployable artefact but it's continuous integration so we want to optimise to have great continuous integration as the first part of our deployment pipeline as a grumpy old man it's one of the reasons that I feel slightly grumbly about saying CI-CD rather than just CD because we're saying CI twice because CD includes CI in my mind but that doesn't matter it's me being grumpy there's a link between the ability of teams to keep the build green and how fast this goes go back to my example of the C++ the large C++ build an overnight build, think about that for a moment if I'm a developer and I commit a change that introduces a defect I'm not going to find out about that until the following day because the builds run overnight now if I know what I did wrong if I did something really stupid and it's absolutely obvious to me what I did wrong and I commit immediately it's not fixed until the day after that so your real cycle here is twice the duration of your pipeline and if you're waiting for a day and half of that double that time is the time when stuff is broken in reality what you're going to do is that you're going to find it really really hard to stay on top of failures my gut feel estimation of looking at projects like this is that if you have an overnight build which is pretty good by most standards if you have an overnight build you're probably going to have 5 or 10% of your tests that are always failing there's a random 5 or 10% that are always broken so you're not getting the clean signal about the releaseability of your system because now you've got to guess is it okay to release when that test is failing or not and you don't want to be there within the working day bringing this so that you can have multiple tries within the working day is one of the reasons why I suggest optimising for a really fast commit stage now I've got several attempts at trying to fix problems if I introduce a problem and that means by the end of the day I'm almost certain to have corrected any problems that I've introduced treat commit stage performance as an important thing optimise for it assign development to achieve it spend money on resources this is a worthwhile investment in the efficacy and efficiency of your development process to my mind the next stage is the acceptance test stage so this is the first part of the acceptance cycle this is the stage where we're going to run these tests that define the utility of our software that we're expecting usually the kind of the pseudo code for the commit stage sorry the acceptance stage looks vague like this we're going to configure the environment in some way we're going to deploy the release candidate we're going to get that up and running we're going to just check that it's ready for use and then we're going to do whatever it is that we want to do with it ideal feedback cycle for this is somewhere in the region of about 45 minutes or less ideally we'd like the whole cycle to complete in under an hour this is another one of those things that gives if we can determine the release ability of our changes in under an hour and we introduce a defect we've now got two or three goes at fixing any problems that we introduce during a working day usually on average and so that's going to allow us to largely keep the build green slow acceptance test the way that I prefer to work as I described earlier is to use acceptance tests to guide the development process we're going to create these executable specifications before we start work on a new feature as part of the development of this new feature and so we're going to grow lots of these tests because we're going to build these tests new story that we write we're going to end up with lots of these I'm not a big fan of the testing pyramid that say that you should have few UI tests and a few more integration tests and whatever other tests and lots of unit tests certainly lots of unit tests but probably also lots of acceptance tests in my world view because they're important they give us different parts of information and they help us to drive the development process one of the ways of thinking about this is thinking one of the things that we can can be slow is the configuration of the host environment itself now this is alleviated to some degree with modern system or modern cloud based tech like docker, Kubernetes those sorts of things to some degree but it's worth thinking about so it's worth thinking about the rates of change of different layers so if you're deploying you know some regular software running on a version of an operating system you're probably not updating the version of the operating system quite as fast as you're updating the application that you're building so that's changing faster so you'd like that to be very efficient even if it's a bit slower to upgrade the operating system just as an example there are different ways that we can use to optimise that but thinking in terms of the layering and tuning the configuration of environments using the layers in those environments is a useful strategy particularly if you're not working with cloud native technologies for any reason so we can use things like layering the configuration you had a question I want to evaluate the system through the natural interfaces to the system that the end user would use it's got a UI I prefer to test through the UI I'm okay with using other APIs or interfaces to the system if they genuinely publicate interfaces to the system to populate the system to get it into the state ready for the test but if the interactions that I'm testing are through normally through a UI I want to test them through the UI I want to test the system as though it's been used by real people and the guideline is to write acceptance tests that test via public entry points into the system what I mean by that is that I don't want to build backdoors into the system just to support the testing that I want to do I don't want to go in and put some test data directly into a database that backs up my software I want the database to be populated through the code that populates it in real life because otherwise I'm not testing reality so I think that's worth it and there might be times where you kind of compromise on some of these things a little bit but I would strongly recommend that the safest stance is to always test through the public interfaces I think there's lots of things that are talked about testing through the UI being slow and for some things that might be true but it's not as slow as it seems the Elmax case I showed a picture this morning in my talk this morning that showed the numbers of tests in different stages and we had somewhere in the order of about 20 odd thousand, 22 thousand, something like that acceptance tests for the Elmax system a large section of those thousands of tests tens of thousands probably we're going through a web based UI because that's how people were interacting with it and we could get a result in under 45 minutes from that phase so I don't think it's always costly you do need to think about the performance of these things of course but it's worth thinking I'd rather not break encapsulation if I can avoid it of the system I don't think that integration tests for me integration tests aren't part of the strategic decision for testing so my test strategy is based around acceptance testing and test review and development for many systems that's all you need if you think about what the acceptance tests are we're going to deploy all of our system that we're responsible for we're going to test that through its public interfaces in life like scenarios from the perspective of an end user in a production like test environment this is like a super integration test this is testing the system integrated as it will be in production it's also like a staging test so we don't need a separate staging environment either because we're evaluating our system deployed as it will be and configured as it will be in production on the whole so this kind of fulfills multiple purposes which means that it's quite an efficient way of working we're using these things as specifications as a way of communicating into the development team the intent of the requirement these executable specifications and we're using these executable specifications to guide the development process in terms of we're going to do TDD underneath until the specifications are met so all of these things add up and we get an awful lot of value out of this one type of tests for me the role of other types of tests are more tactical so if there's some particular aspect of my system that I can learn more quickly with some integration tests I might do that but it would just be for those things I don't integration tests for me are not the kinds of tests that you have for every feature that you develop you have them occasionally when you have a particular need that they teach you something that you don't, more quickly that you don't learn elsewhere the acceptance tests are going to cover the ground I think so we can think of layering our configuration we can think of things like pre-baked templates this is one of the techniques that Netflix used for a long time where you kind of you pre-populate it's part of the environment and then you just layer in the application changes to pre-prepared environments that you took off the shelf modern technologies like docker images solve some of this problem generically because they have a layered version control system built inside of a docker image that may not, as I've said before that may not always be enough if you're doing this for a legacy system and you can't use docker for some reason or if you've got particular needs the generic solution in technologies like docker might not be completely optimal so there might be other opportunities for you to speed things up if that's a difficult thing but now we're getting into technical edge cases really so using containers like docker or Kubernetes and those sorts of technologies are an effective way of managing this part of the problem and managing the layering and then you can keep these images and use images off the shelf and then populate them with changes and then snapshot the image and it's a good way of optimising the things successful deployment ends with a running system in production and so for efficiency we need to configure the time to deploy the system the time to migrate any data that the system uses and the time to start the system up and ready for use optimising all of these things is sometimes worthwhile but certainly considering them because what this does is that this kind of puts defines the theoretical limit for how fast you can get results from your tests let's imagine for a moment that compute and money are unlimited then the limiting case for how fast you can get results from acceptance tests is that we could imagine deploying a dedicated version of the application for every single acceptance test and running all of the acceptance tests in parallel against their own dedicated version of the app if we were to do that then the maximum duration of the test would be how long it took you to deploy your application into the test environment get it up and running and ready for use and then the duration of the slowest test so after that you just decide how close you can get depending on how much money you want to spend on on test hosts so it's worth thinking about these kinds of things I think if you're in this kind of game of trying to optimize the things there's also then the time to validate the time for the test to run so keep deployable artifacts lean don't carry too much weight I did some consultancy for a client in Silicon Valley and they were complaining that it would took too long to deploy the application into production and when we looked they were deploying 114 gigabytes of data and that's quite a lot of bits to move around when we looked though 112 gigabytes were some test data that they were lugging around everywhere and so were useless in nearly all of the environments so just discarding that has just sped the build up so just think about these kinds of things look for ways in keeping keeping these things as lean as possible don't carry too much stuff around as the deployable image for the system and try and keep the artifacts lean try not to produce a vast deployable artifacts if you can avoid it you can start thinking about packaging strategy as a tool for allowing you to deploy things if you package things appropriately you can then start thinking about parallelizing the deployment think about network infrastructure and those sorts of things whether that's a constraint it might be something interesting that you want to test but certainly in your test environment optimizing your network infrastructure and your environment to allow you to shift the bytes around that represent the system quickly and efficiently is worthwhile thinking and again kind of caching builds and stuff like that getting them closer to the point where you're going to run them when you want to run them might be an option in some kinds of environments particularly if you're doing the layered configuration minimize data sets where you want this goes back to the question about using production data production data is usually too big and unwieldy so use targeted data sets for your testing and that tends to make life a little bit easier and these sorts of tests are a bit more tractable and think about better modularization of the code again along with the packaging and maybe deploying the system in parallel try to avoid dependency ordering between the parts of the system so you can start the system up and when all the pieces get ready it kind of knows it's ready for use so let's start quickly I've already talked about that it's worth making this part of the code efficient this is commonly not a place where we think of making the code efficient but actually there's lots of nice properties of having your system that's able to start up quickly not least if something bad happens in production and you need to fix it and get it back up and running again it's going to shorten that time so it's quite a healthy sign if your system can start up quickly to improve the efficiency of this part of the system very often I've talked about that already I'm whizzing a bit because I've just realized I was talking slower than I thought I was there's a common cycle here with acceptance testing in particular because acceptance testing we're using to drive the development process we're going to be writing acceptance tests for every new feature of the system it's just going to grow and grow and grow as we develop more software that's going to happen we start off small we start off with a few of these tests we usually run them on the same server that we run the commit test originally because there's loads of spare capacity at some point we'll have added enough acceptance tests where the build is starting to slow down a bit it's getting annoying so we'll move our acceptance tests off and we'll buy a new server and we'll run our acceptance tests on a dedicated host environment and then those will do okay for a while and then we'll add more and more tests and then we'll get to the point where that starts to slow down as well what usually happens at this point is that we buy ourselves another server we double up and nearly every time a team that I've ever seen they start off by just manually dividing the tests up every test that starts with the letter A to M runs on the first server everyone that starts with the rest of the alphabet runs on the second server and then they watch that and they find out that everyone always finishes before the other and so they start tinkering and moving the tests around to balance it out and to try and optimize that for a bit and then we get some more servers we double up again, we get four servers and then that starts to get painful all that balancing and messing around with tests and keeping it aligned at some point we're going to want to think about probably automating the allocation of tests to environments to try and optimize this process a bit more effectively we can do things like test avoidance, we can try and track I'm always a bit wary of this to be honest some people are better at this than I am I've never been able to do this to my own satisfaction well enough I'd rather just optimize to run off the test really fast but test avoidance can work in particularly at a massive scale if you can say this change here never impact, this change in this part of the code here never impact these tests, they're not related and we don't have to run those tests but I think it's sometimes hard to be able to do that so I'm a bit wary of it so be careful that you don't spend too much time the last time I tried to do this which to be fair was a long time ago we analyzed it and we literally our build was spending more time analyzing which tests to run than it was spending running the tests so be careful what you're doing if we do all of these things then we're able to scale up easily if we care for the way in which we work these things what we can start to think of now is that we can start to think of tricks like sharing out the cost of deployment of our system between many tests we're going to run many tests in parallel or sequentially in a shared environment and effectively we're immortalizing the cost of deploying the application and getting the environment ready so now when our acceptance test environment becomes free it goes and looks for the newest release candidate that deploys that and then it spawns out a whole bunch of test hosts that can then evaluate in parallel against that shared environment this is quite a good strategy for optimizing things and we can collect the results at the end of the room and that's another way in which we can start to really scale things up and really start growing our ability to get fast results this scales quite well this is an animation of the elmax system allocating tests, the different test environments on this part this is the normal case so this is one environment or several environments that are shared between multiple test hosts and so multiple test cases running in parallel this environment is running special kinds of tests these are mostly time travel tests where we manipulated time in the scope of tests and you couldn't time travel at two different rates in the same system because that would really screw things up and so each of those tests had a dedicated version of the application to time travel with and so that's a recording of a live run test run that we used when we were building the software that managed our test grid to optimize these things so you can take this quite far and you can handle very large, very complex builds with these sorts of techniques if you want to go in this direction let's move on and talk about failing tests the idea of continuous delivery really is to keep our software in a releaseable state and if any single test fails our software is no longer releaseable so at the point of a test failure we're now not releaseable so the job at that point is to get back to releaseability as quickly as we possibly can so the job of the development team in that situation the first part of the discipline really is that the person that committed the change and the failure is responsible for correcting the failure if I write a change and I commit something it breaks one of your tests that's my problem, not your problem so I'm going to go and try and fix it as quickly as possible there's some disciplines that are at this I want me to be able to spot it before anybody else and so on but fundamentally my job now is as soon as I've broken something is to try and get back to a stable place when starting a clock from the point at which you detect a failure give yourself 10 minutes to commit something and you either commit a fix or you revert your change don't spend ages trying to describe what the fix is revert it and then figure it offline if there's a problem because you want to get back to getting the build into a releaseable state we've talked about the time scales that we're looking for we're trying to maintain that and be able to use that tool within the team so prioritise fixing failing tests always when there's a failure if somebody else commits a change and breaks it and they've gone home or gone away revert their change on their behalf keep the build in a releaseable state one of the most profound problems for these bigger more complex kinds of tests are intermittent tests and these are always these are always difficult to deal with intermittency is painful and a difficult thing to stay on top of but I think it's important to try and do so too many teams live with intermittent tests it's at the stage where some build tools for example will do things like give you the option to flick a switch and say if a test fails run it again what on earth does that mean what's the point of that if I run the test again it's lying to me if I run the test again and this time it passes it's lying to me one of the times is it lying to me when it's passing or is it lying to me when it's failing I don't know I can't tell yeah so it's a race condition but you don't know you can't tell that so you've got to deal with that you've got to deal with that you've got to deal with that you've got to deal with that kind of problem too many teams live with intermittent tests and spend ages analysing is it okay to release with this failure or not or re-running the test both of these are crazy so don't live with intermittent tests solve the problem and treat intermittent tests as failing tests it might be worth we built some tools into our deployment pipeline that would track the history of the last 10 commits and would be able to spot patterns if there's a particular test that went pass, fail, pass, fail, pass, fail or something similar it had a high score on an intermittency index and so it was a suspicious candidate that we'd then go and look at and try and figure out why it was doing that so treat intermittent tests and don't allow the release of the software until you've kind of got to the bottom of it as a last resort if your tests are intermittent I think it's probably better to delete them to leave them there just nagging at you because they're not telling the truth the technical problem that you were referring to if you've got nearly always intermittencies down to one of two things it's a poorly written test that hasn't, in the case of the browser it hasn't thought about the case it might not be finished by the time it does the evaluation so pull for the result until you get the result and then time out or not something else or look for a concluding event or something to determine when the evaluation is or there's a race condition in your application and that's a real real world bug so it's usually one of those two things so either one is important to fix in my view Elmax this was one of the things that we got wrong we lived with intermittent tests and we built up all sorts of approaches to try and stay on top of them for a long time and when we got down to it we just started let's dig in we'll just always fix the intermittent test sometimes that took a lot of work and a lot of effort because these are sometimes complex tests but nearly always if it was a hard problem to find it was a real bug in the application that was subtle and tucked away but it could have caused problems and we were glad that we found it when we did the common causes of intermittent tests are race conditions as I mentioned poor test isolation, poorly designed test cases and sometimes it's really something serious in the application that you ought to care about the last on the list is pipeline changes this is as I said before a deployment pipeline is a strategic resource it's our route to production for all changes and so it's worth thinking about high availability strategies clustering the systems for the pipeline applying continuous delivery principles to your deployment pipeline infrastructure as code principles certainly I have in the past for larger teams built a deployment pipeline for my deployment pipeline changes it sounds a bit crazy but it allowed me to evaluate my changes be confident that I wasn't rolling out rubbish to the teams and to step back from the changes in the pipeline if they were bad think about service level agreements for the pipeline so that people know that it's going to always be there and invest in the hardware software and practices to keep it available if it's central the pipeline is a complex system its own right and so we ought to be applying good engineering principles to it as well as to the rest of our software think of version controlling it and all of its tools if you think about it if we want to have a reproducible state in our system the deployment the version of the deployment pipeline is the dependency of the system so if I've got a particular version of the application and I'd like to recreate that older version of the application I need to know what version of the build tools the environment the deployment pipeline was in play at the time when I originally built it that's a bit of an extreme case I confess I've not done that very often but I have done it a couple of times in mission critical kinds of software systems so it's worth thinking about consider writing test cases for some pipeline behaviours you don't need to go to town and do this all of the time but I've done some work with test driven development for infrastructure as code and it works surprisingly well it's not always easy to think about but once you get into the the way of working it's quite a nice secure feeling when you're able to work with those sorts of changes that way we can start thinking about blue-green deployment strategies for the pipeline if you've not come across that concept that's an idea that we wrote about in the continuous delivery book the idea is that you have two environments two production environments in this case two pipeline environments one called blue, one called green and the blue one's live I'm going to provision and update the green one test it and when it's ready I'm going to switch it over and make that one the live one and that just keeps the pipeline always available use infrastructure as code approach for all pipelines I would advise doing this from day one when you first build your pipeline use infrastructure as code techniques to deploy it and that way you're going to be as you build it and it gets more and more complex you can maintain that and that's the easiest way to do these things and using infrastructure as code it means that all of this stuff is version control so you can step back to safety if you make a mistake too often I've not done that that's it that's my talk for today if you'll forgive me advertising my own ways I'll be honest with you I have some online training courses that go into some detail on a lot of this stuff testing strategies in particular but deployment pipelines continuous delivery in general they are very highly reviewed it's difficult for me to sell them into India because of the differences in our economy and I'd like to do that I've tried advertising different price rates and so on and if anybody ever asks we'll give you a discount always so if anybody's interested anytime ask but for this now you can use any of my courses at a 50% discount for the duration of the conference so do take a look courses.cd.training and for the people that were particularly asking about acceptance testing there's a really good course if you'll forgive me saying so myself about acceptance testing how to build the infrastructure how to operate the process as I sketched it today and how to do the test isolation stuff so you can run these things in parallel in your pipeline thank you very much I hope you continue to enjoy the conference and thank you for attending this today any questions we have some time for questions anybody doesn't have to be about this stuff that we were talking about yes so in one of my experiences it has happened that due to some past experience there is a fear factor within the developers so although as you mentioned in acceptance test they should focus only on that particular use case focus on that only but because of past experience they tend to do extra tests so is it something which is it a psychological thing and we have to deal it that way or is there a process by which we can handle such scenarios I think in my experience it's commonly down to a few things and the first is that development teams are reticent to do test-driven development and so they lean on these bigger more complex functional tests, the BDD style tests, the acceptance tests instead of test-driven development I think that's a mistake I think that you want the fine-grained validation that test-driven development gives you that's faster, quicker, cheaper to run easier to maintain and they have a better impact on the design of your software the acceptance tests are expensive and you probably don't want to be testing every little possible edge case through them which is why I prefer strategically I want both I want the value of the fine-grained detail in the test-driven development things and the acceptance tests the other reason why people are nervous is that they haven't got good enough test coverage clearly because otherwise they wouldn't be nervous so I think part of it is to try and build the culture of testing this is unpopular these days but I think that pair programming, mob programming, ensemble programming are a good way of trying to level up the culture in a team if I'm having a bad day and you and I are pairing and I'm not testing you're going to say no, we should be doing the test and you're going to make me do a better job and when you're having a bad day I'll make you do a better job so you tend to level up you tend to get a better result with that kind of thing and that's particularly true when you're trying to change the culture of a team I think it's a really valuable tool for doing that kind of thing I do want to make test cases easy to write sorry, I'm not trying to advertise my courses I suppose I am, but I'm not but that's one of the big themes of acceptance test driven development and BDD is to try and make the test cases easy to write because these big functional tests are difficult to live with and you want to be able to make them easy to write so they go really fast and you make them reliable so you want great test infrastructure and you want the test cases themselves to be simple and easy to write and you just tend to lure you into over using them so you have to be a bit wary of that but I think the real answer is to focus more on TDD rather than these things for that really thank you, pleasure anybody else? great, okay, thank you enjoy the rest of the conference sorry, one more question can you give an example about installer? installer, yeah sorry, it's just jargon that Jess and I took to using for the build image really so what we want is that we want to as well as testing the behaviour of the system we want to test its deployment as though it's in production so we want to use the same techniques the same technology, the same mechanism to deploy the software into test environments that we will use into production and it should be automated so that's the installer what I'm talking about at the end of the commit stage is building the installable image which I tend to refer to as a release candidate which is the bits and bytes that we will deploy into production and then any of the associated mechanisms that are going to get that image into place and up and running and ready for use okay there's some stuff talked about that in the continuous delivery book which I know that you've got a copy of because you asked me to sign it earlier we do talk about some of that stuff in the continuous delivery book thanks everyone, thank you to you