 Thank you very much for choosing to be here and not in the other room. I know that it's kind of going to ask Mary Popendikis. It's very difficult and You know they've scheduled two talks about continuous delivery at the same time. So I'm going to tell you why continuous delivery is bad. If you want to hear why continuous delivery is good, that's kind of in the other room. I'm going to talk about how I've made a couple of really stupid mistakes and how we lost a lot of money because we started doing continuous delivery. And I want to talk about the kind of side effects of continuous delivery that people doing software delivery often do not think about. So that's kind of what this talk is going to be about. If you want to learn why it's so brilliant and things like that, that's kind of another room. And I know that for an angel themed conference kind of trying to talk about continuous delivery is a bit dangerous. To set things clear, I'm not against continuous delivery. I love being able to deliver quickly and things like that. But I just want to point out a couple of things that generally surprise people when they realize that they are not. And if you pay attention to those things, you might turn out to be a lot more valuable to your company and product. If you ignore those things, you risk completely destroying the business. That's kind of where we're going to go with this. Now, I assume most people in the room are kind of building or delivering software. Is there anybody from the business side of things in the room? No. Okay. So I have this relatively strange position now where three years ago some friends and I started building a product and kind of developed the business at the same time. And because I'm primarily a developer, we focus too much on the development side of things and kind of that misled us slightly. And that's how I started kind of paying attention to some other things as well, because when it's your money, then it creates quite a lot. And after I started spotting this, then kind of going to conferences and speaking to lots of other people, I started seeing that lots and lots of other companies have the same problem. It's just because it's somebody else's money, people don't pay attention that much. And the biggest thing when we talk about continuous delivery is that one thing that I really like to kind of think about, and it's kind of a slightly philosophical question, is who decides when software gets actually delivered? And where does it get delivered? Because that's not necessarily easy to answer in today's world where there's multiple background services, there's mobile phones, there's websites, there's multiple environments, multiple customers, internet of things, lots of other weird things happening. And there's been a massive change happening in our industry over the last 10, 15 years. Of course, processes and things like that, but the commercial models for how we distribute and how we sell software have changed quite a lot. And the answer to this question kind of changed quite a lot. And one of the really interesting things that we are now starting to see, and I'm starting to see as a consultant with lots of other people, is that we've generally solved the problem of building software. Most companies I work with today as a consultant can shift any software in a reasonable amount of time. That's no longer the problem. 20 years ago there was a huge problem kind of long delivery phase in Italy, but we won. The whole angel thing won. We now know how to shift software to the bottleneck somewhere else. And one of the really interesting research papers I've read over the last year or so is this paper from Ron Foghavi from Microsoft, who went back to the PowerPoints that were used to justify these grand initiatives to build stuff. And his conclusion was that kind of about one-third of ideas actually improved the metrics that were supposed to improve. About one-third of ideas they implement do not have any statistically significant change on the metrics that were supposed to improve. And about one-third of ideas actually damage the metrics that were supposed to improve. That means that if you're anything like them, two-thirds of things coming into delivery might find are bad ideas. They're not bad because, you know, we have a bug or we implemented it badly, they're just bad ideas. And lots of people talk about scaling. There's kind of several consultants that are going to sell you how to scale your agile and things like that. But what if with the same number of people you could build and maintain three times in a software? If you're just able to spot the bad ideas earlier. And that's where I think kind of, if we do continuous delivery right, that can massively, massively, massively open up this opportunity. And kind of generally as said over the last 15, 20 years, lots of stuff has changed in software. But one of the things that's kind of changed beneath us, or at least I have not noticed that, is who decides when software gets stopped and where. 20 years ago, that decision was mostly with the customers. There was mostly with the people paying, we'd kind of get a box of software to install it. Regardless of when the software company ships something, it's the customer's decision when it actually gets installed. And it's kind of, I remember buying Wattcom C++ compiler 8.5 or 95 floppy disks. And then spending a whole night installing that, and then disk number 87 was kind of corrupted. So I had to go back to the shop and it was just insane. Now, you know, the internet came and everything became brilliant and we started shifting stuff. But all of a sudden, one of the really interesting things that started happening with the internet is the customer started slowly losing the decision point when it gets delivered. Because you have a customer who pays for something and then there's kind of people using it. And as kind of the internet started taking hold, first thing I've kind of noticed was anti-virus software started getting these updates in the background. You know, you buy the software and then, you know, the screen pops up to the user and the user says, well, yes, I'd like this update to not, and it installs. So it started getting these kind of updates, decided by user, now we're getting this completely insane thing where, you know, it's asking, well, this other support for Hungarian, do I want it, do I not want it? Can I refuse it? Probably not because kind of, I'm being offered this choice that makes no sense to me at all because it's irrelevant for me. But I'm kind of forced to take it because if I don't take this, I can't take something else in the future that I need. So it's kind of a decision with the user, but it's not necessarily with the users anymore. And kind of, we're getting to the third point where pretty much it's not that the customers are not the users that are deciding when something gets delivered, it's the developers. And the developers are kind of taken over and it's insane. I've done this presentation in Sydney two years ago where in the middle of a presentation, the update for Acrobat Reader popped up and it says it's a very important update. Well, important to whom? It's not important to me because I'm doing a presentation now. It's important to a developer who was embarrassed with the bug. And by the way, you know, how is it possible for a reader to have so many stupid bugs and security problems? It's supposed to be read only. I mean, that's insane. So more and more we're getting to this point where developers are in charge. And that's an interesting thing because it opens up some really interesting opportunities, but it also opens up kind of a whole Pandora's box of potential problems. For example, I worked with a media company in the UK where we got brought in to help them figure out how to test the software better. And generally, when I go into companies like that, it's almost autopilot. It's a bit of test automation, a bit of better communication between test developers in the business, a bit of kind of focusing on the risk and that sort of thing. Take the money around. With these guys, they pretty much already had in their process everything that we wanted to propose. This is a strange situation. So, you know, if you look at the normal metrics, their software is actually pretty good. The process actually pretty good. So, we went and talked to the kind of business guys who called us to figure out what's going on. I said, no, no, this is horrible. It's absolutely horrible. You know, sit with us in the seat. And we spent a day in the content management center where kind of they're putting interviews. In the middle of kind of the busiest period when everybody was typing interviews, there was a message and everybody screamed and we need to log out of the app and kind of restart and log in again because there was an update. And one guy decided not to do it because he was in the middle of writing this kind of big article and then finished writing, clicked on save and got an out-point exception on the screen because the service in the bank and his law were combating over the front page. And this is kind of the whole quality problem they were facing. Not the quality, not the software is kind of bad in itself, but people were shipping it at the wrong time. Then we went to the developer and said, why are we shipping it in the middle of the day? He said, well, this stakeholder in that building over there told us we need to fix his bug very quickly. And here we are kind of competing priorities, competing stakeholders, competing opinions and everything. And you know, one guy might think this is really great because my bug was fixed quickly. Everybody else thinks this is horrible because it's getting updated all the time. And go and talk to anybody in a call center and ask them what they think about the software being updated all the time. They'll kick you out because there's no documentation changing this, they can't support their users. So there's a lot of things to think about more than just shipping software whenever you want. And that's why it's kind of dangerous for developers to be in charge of this now. Continuous delivery and kind of shipping software mandated by developers opens up some really interesting opportunities. For example, there was this big thing a couple of months ago where General Motors was hit by a recall order for about half a million cars. Because the brakes didn't work that well. So they had to get people to kind of drive cars back in, fix them, pay for labor and pay for parts and kind of it's a massive problem. Around the same time Tesla actually ended up with this problem where one of their cars using this autopilot feature ran into a truck and the driver got killed. And they had to kind of figure out, you know, why that happened and what happened and things like that. And as a precaution, what they decided to do is kind of just set an update over the wire to everybody to kind of prevent something like that from happening again. And plus they started detecting parts of the roads where other vehicles have passed and things like that. So the market is safe or unsafe. And they were able to react to a very, very similar situation to what General Motors had with the software update. It was kind of just went out. No need to pay for expensive parts, no need to pay for labor. And when you have two companies like that that, you know, one company needs to inconvenience their customers by taking a day off to drive a car into a service, waiting and paying for labor and parts. The other one doesn't get pushed. They're not even playing the same game anymore. So it's kind of, there's a massive business advantage to that. There's a marketing advantage and things like that. Where at the same time, there are companies that have suffered quite horribly by not understanding what's going on with delivery. Because continuous delivery also starts interrupting sales models. For example, as we started getting this internet and getting updates and everything, people got used to getting updates for free. When we were buying boxes of software, it was usual to pay, you know, $100 or something like that for a new version. Spend the whole night installing it. And people were keen on buying software. Nobody buys software anymore. We kind of ended up with these upstores where everything costs $1.99. And users expect that for $2 that they give, you're going to support them for the end of their life. Just commercially unsustainable. And that you're going to give them all the new versions forever for free. And kind of then people have to basically finance their old users by selling stuff continuously to new users in order to finance development. When you have to support your old user base from the money that's from the new user base, that's called a pyramid scheme. That's the definition of a pyramid scheme. And those things grow up to a point and then they explode. That's why it's so difficult to sustain a business developing mobile apps now. And this whole kind of updates on the internet all the time that the free has disrupted the game industry significantly. In 2009, I worked with a gaming company. I was going to release some multiplayer games when Zinger shopped through the roof as a business overnight success. Because they figured that people don't want to pay for games. But they would pay $1.99 for virtual diamonds to put into a virtual farm or something gross. And this whole micro-paying thing was amazing. Everybody was really, really impressed with Zinger. And then a couple of years later they started firing people after I dissented because they've killed interest in the game. You buy virtual crystals only so many times. And then after people push you and push you and push you to buy virtual crystals all the time, that's no longer that interesting. But because they're not selling a game anymore, they're selling virtual crystals, they have to somehow pay for that. And there's this whole problem now where it's almost impossible to sell an operating system. Because Apple started giving away operating systems for free. And Microsoft had to start giving away windows. And they can't make money on windows anymore because nobody's going to pay for it. So what they do is they start stealing people's personal information, selling it as advertising. It's not what you want from an operating system. But it's changing commercial models. So this is where the two effects that I've started seeing from continuous delivery. The software development people often do not see user entitlement. Users feel entitled to all your updates all the time. They feel entitled to stuff that 20 years ago they would have to pay for it. And that starts changing what we can sell and how we can sell it. For example, continuous delivery and continuous updates tend to work really well for selling stuff like capacity. GitHub and Dropbox are making a killing because they're selling capacity. They're not only selling software. They're selling megabytes of storage. They're selling other things. So it's almost impossible to sell features, but selling something that grows over time, we're able to find it. So the commercial models are changing here quite significantly. And the other thing that is happening is we start getting users that are never really satisfied with what they're getting. They feel really kind of dissatisfied with the stuff that they're getting for free. For example, I've mentioned this collaboration tool I'm building. And one thing that can, this is the user growth graph over the last kind of two years. So this is when we released version 1.0. And we got a PC World reviewed us and we got on Hacker News for two days on the homepage and things like that. This is where we got reviewed by, made use of, and it's kind of another thing we bought on the website. This is where we got reviewed by Light Hacker, which is number five kind of website on the internet. And after that you can see kind of a very, very slow growth up until this point where we've done a ton of features. This is kind of almost a year and something where we'll be releasing lots of features all the time. And we've never really been able to get this growth and repeat this stuff, although we've tried to become. And one thing we realize is that by doing continuous delivery, we have taken the drama out of our releases. There was no technical risk. There was no big risk of anything. We were doing continuous stream of small features. But by taking the drama out, we've also taken out the excitement. In a continuous delivery cycle, there's never anything important enough for newspapers to write about. It's a stream of small features. And kind of user growth here is from a popular website writing about us because we're new. And by doing continuous delivery the way everybody says you should do continuous delivery, we've pretty much killed the biggest marketing driver. Because there was nothing to write about us. And that's a really unexpected effect and it took us a long time to understand what's going on. Because, you know, there's nothing new to announce. And you look at all these big companies where there's a conference, people announce crap and everybody writes about that it's a massive success. And these people are still doing continuous delivery but they're doing continuous delivery in a slightly different way. And I've mentioned Tesla. Tesla is famous for being able to patch up software and release new features to their cars. And that's an interesting thing because when you think about them being able to do something to their cars, that pretty much kind of kills any argument that, oh, you know, we are this kind of company because of continuous delivery. So if they can do the cars, you can probably do it as well. But even kind of they started having a problem with user entitlement. Because every kind of, they became popular for releasing new features to their cars on the time. And two years ago when they started working on this autopilot thing, the first version that came out was actually just automatic lanes. They're really good doing kind of MVPs and things like that. And automated lane switching was if kind of you're driving and the car decides that you're going to crash, it will try to kind of switch your lane automatically for you. There was no kind of, nothing else in terms of automatic driving, but for that they started releasing a new car in their cars. It was like eight cameras, two radars and a sonar. And, you know, why on earth would you need a sonar? I guess, you know, if you start running away from dolphins, but why not, you know. So what they've done is they've put in a bunch of cardboard there and into new cars and then they started building up software that uses that car. And everybody was really happy about that, but then a huge percentage of their customers started complaining. There was this online petition where a guy called Richard Wolpert from LA started saying that, you know, this new feature is really important, it might save your life. And previously all the new features that Tesla really, he was able to use on his car basically from the point where they released it. And he did not have to do anything. He didn't have to drive the car into the kind of repair shop. He didn't try to do anything. He just woke up and there was a new feature. His car doesn't have a sonar. His car doesn't have eight cameras and two radars and things like that. So the new feature obviously, you know, is not going to work. You and I know that. We understand the difference between software and hardware. But this guy bought a car and his car continuously got new stuff. And in the news he read that, you know, Tesla's now support plane switching and his car doesn't. So he felt like to go and disappear. And he started complaining and kind of raised a lot of noise and people started signing up the petitions. And early Tesla doctors worldwide started sending letters to Tesla how, you know, it's irresponsible for them not to give them this new thing. Now, you know, it's one thing to do a git push and the software magically goes out there. It's another thing to kind of give people away, eight cameras, radars and sonars for free. And kind of build it in so that they know the costs are completely different. But people felt entitled to that because of a continuous stream of updates. I've never heard anybody saying that's about a different type of car because you understand the bottom car and it has these features. That's pretty much it. So as we kind of doing more and more of these kind of frequent updates and continuous delivery, the value of each of these incremental updates has gone down, which means that people expected it for free. And then when you do something that's really expensive and really important, people expected to get it for free anyway. And they complain if you don't give it to them. So that's kind of an interesting effect that, you know, developers often don't pay attention to. So at the point where software delivery or the way to software delivery starts interrupting our sales models, starts interrupting our pricing models, starts interrupting the marketing, starts giving users who we want to satisfy reasons to complain. I think, you know, we are pretty much doing it wrong. And we need to figure out different ways of doing continuous delivery to prevent all that from happening. Because in an ideal world we should open up all the opportunities, open up a continuous delivery but kind of not kill any of the other stuff. And what I've realized is that for a long time our approach continuous delivery is purely a technical thing. And there's incredible side effects on marketing and business. And, you know, at this point if you start interrupting other stuff that's more important for a company than software, software is going to suffer. The software process will suffer. People will say, okay, you can't do any more business during the day. And, you know, this whole problem of basically the robots taking over. When developers decide how to run marketing, how to run sales, they have an impact on the robots taking over. Now, Issa Kasimov had, you know, published this book 60 years ago on how, you know, robots are taking over the world. And he decided he created three rules for kind of how to prevent the technology from taking over humanity. Does anybody, do we have kicks in this room who know what the three rules are? Give me one, that's okay. Well, it's something about the machine that's not its creator. Okay, the first one, the first rule is the machine must not harm a human being. Yes, that's pretty good. Okay. Does anybody remember the second one? The second one you should obey apart from number one. So you shouldn't be able to make the machine kind of, you know, which is really interesting because if you look at self-driving cars like that, everybody's really kind of excited about the possibility of self-driving cars. I've read a couple of months ago about a self-driving system called Unicom, developed in Russia. There is six autonomous tanks. They can go around and kind of decide on the targets themselves. And because you don't need humans in them, you can load them up with ammunition a lot more and they can have a lot more weapons. And this whole, you know, do not harm a human is, you know, don't worry about it. So does anybody remember the third rule? Yeah, not necessarily. Okay, so the third rule is the machine should protect their own existence, which is, you know, the Russians I'm sure have done that a lot, apart from one and two. So the three rules are kind of declared as, you know, how do we prevent technology from taking over humanity? And more is a joke than anything else. I kind of, I'm going to rephrase this now and look at kind of how I think we need to approach continuous delivery. So I'm going to give you kind of my three rules of continuous delivery. I will fix this up. And in terms of continuous delivery, if you talk about may not harm a human, hopefully you're not going to work on an automated tank. So, you know, in this case, that's a feature and that's not really a big risk. I mean, people who work in medical devices stuff like that, you know, don't do continuous delivery. But if you're working on something that's not that risky, I think this first rule translates pretty well in kind of, it might make massive differences. So, you know, we can't end up in a situation where half the screen is updated, half is not updated. You know, half of the website is updated, half is not. And this now poses a big challenge. Especially when you look at stuff like updating, like user design. Kind of, UX designers value consistency quite a lot. Marketing people value consistency, they value the consistency of brand, they value consistency of design. Anybody who's doing support values kind of consistency. And the big problem with continuous delivery is, hey, you have a website that has 500 pages. You're not going to be able to update all those 500 pages this week. At the same time, you don't want to wait three months updating all 500 pages to go live. So there's this kind of conflict between doing continuous delivery and having consistency. And one of the kind of really interesting problems that happened to me was PayPal. PayPal updated their business dashboard a year and a half ago. And you know, they're actually good at doing continuous delivery in a good way. And one day I kind of was at a conference and I got a call from one of my partners. He said, have you taken money out of the kind of PayPal accounts? Well, we have kind of 5,000 pounds missing. And I can't figure out what's going on. It's insane. We started looking at the statement. We started looking at transactions. There's no obvious thing where the 5,000 pounds are gone. And I noticed a small link in the top right corner says, do you like a new business dashboard? And I've done enough software testing in my life to know what's going on. I clipped it. No, no, no. Minus one million. I do not like this at all. I lost my money. And then it was like, would you like to switch to the old business dashboard? I said, yes, please. And then my money was there. And the problem was that we sell books to PayPal in multiple currencies. And having a multi-currency account, the old business dashboard would convert everything into our primary currency and give us an estimate how much money there is across all the accounts. As they started to continue changes and kind of small changes, they obviously selected to do only kind of one currency account. And they were giving us the current value of our primary account without converting any other one. And that's why all the dollars and all the euros and stuff like this are not showing up in the toolbar. But that doesn't mean the money was gone. I'm just not displaying it correctly. And there's a big conflict here where I completely understand, you know, they said they had to slice the delivery somehow not to wait months and months and months until everything gets updated. At the same time, I panicked there. And, you know, being from the software, I could kind of fix the problem myself, but I can bet that their kind of support concept was covered that day with people complaining that their money was lost. So we have to be able to avoid this problem. Now, the second one is where we have to obey orders, you know, we must obey orders given by humans and things like that. And I think one of the most important things to do there is the developers should not be able to force people to interrupt a session while something is going on. So this whole situation where somebody is writing an article and they decide not to update and then this whole thing kind of breaks beneath them is a big problem. Or me when a dialogue pops up and says, you know, would you like to update it? I said no, please stop. And then it takes off my kind of, you know, internet pipe anyway to update. It's horrible. So I think, you know, we mustn't interrupt users while possession. The software update is never going to be as important as something that somebody is doing at that time. And the third thing where kind of the robots must protect their own existence, I think that translates into not interrupting marketing because at the point where continuous delivery starts interrupting marketing, somebody from marketing is going to say, no, you're not allowed to do this anymore. And that's pretty much it. So kind of game over. So I think kind of from that perspective we must design our kind of continuous delivery pipelines never to interrupt marketing, which is a really strange kind of thing to think about. Now kind of, I mentioned this collaboration tool working at the moment. And we started trying to figure out how do we not cause these problems for ourselves? How do we keep doing continuous delivery without causing these problems? Because I don't want to wait six months to release that introduces technical risks. At the same time, I want to have something that newspapers want to write about. And I want to have kind of a way for my users to keep using the stuff even if I'm updating it and things like that. And we realized thinking about this big mental problem that we had that's mostly terminological. And it was like an eye-opening for me. I realized that kind of we were confusing two terms that people in books use interchangeably that are completely, completely, completely different. And for the next five minutes you're going to look at me like a crazy and then I'll explain what I mean. And the thing that kind of, I really realized is that deployment is completely different from what it is. In the software industry, deployments and releases are almost the same stuff. We talk about continuous delivery, continuous deployment, continuous release, potentially shipable stuff, shipable stuff. It all means the same thing. And we realized that kind of by thinking about it that way we've completely broken everything. Because we created side effects to our process that do not need to be there. Now, I'm not saying that you need to shift to a testing environment and then you're done with things like that. But I've realized that it's really, really useful to think about these two concepts as two different things. Deployment is a technical act of moving code to production hardware. Again, forget about testing environments, staging environments, but moving code physically to the production hardware is deployment. That's like putting stuff into a package, isolating it nicely so it survives the transport and shipping it. And that's a technical event. Now, release is a marketing event. Release is something that's incredibly well timed to create the biggest impact. And kind of confusing development and release got us to the point that kind of is almost like, you know, getting this summer zone to deliver a present and then children interrupting, intercepting the delivery guy, kind of tearing everything up. And yes, at the end the child ends up with the present, but the magic of Christmas is gone. And that's exactly what we did. In order to reduce technical risks, we've killed the magical software releases. And this is where one of the key things we've done to kind of, you know, get the spike going is decoupled developments and releases. So we're able to shift stuff to the production hardware where it's not necessarily made available to users. And that has some really interesting effects because that means that we can decide when it gets available and to whom. And we can, for example, if there's a particular part we need to fix for a subset of users, we can give them the new stuff without interrupting anybody else. If there's a really important thing we want to kind of slice and develop, we can decide how do we switch people and who do we switch and things like that. So imagine going back to the paper business dashboard if they gave the new business dashboard only to people that had a single courtesy card. First of all, they would get a lot better feedback because asking people if they like it or not, you know, people who lost money are not going to like it, not because it's bad or ugly or whatever, but you know, you lost my money. And if they limited that to only the people that have a single courtesy card, then they would get much better feedback, people would be happier, and they would actually be able to then start expanding it slowly to other users, which would be amazing and you know, the whole problem. And this whole kind of, you know, how do we market it? How do we sell it? Starts revolving completely differently if we can decouple deployments at least. And the really interesting technical part of that, it's a technical problem we need to solve in order to make this work is how do we start being able to run multiple versions of our software on the same environment, on the same hardware at the same time. So this is where multi-version becomes absolutely critical. And I'm not talking about feature toggles. Feature toggles are a cheap way to create spaghetti out of your code that can survive for the next 20 minutes while you're doing an experiment. I'm talking about proper designed multi-version, so you can have multiple versions of the same stuff running at the same time. You can decide who gets to see what, you can decide who gets kind of upgraded to what version. And it took us a couple of months to really kind of figure out how to do this for our software. The software was mostly kind of in the data. And once we started kind of looking at multi-version in the data and upgrading and downgrading data and things like that, that's a huge technical problem. And this is something that's going to cost a lot, but the value of that is amazing. So, and this is where I think, you know, instead of combining deployment and release where we effectively cause a technical event to have a marketing impact. If we decouple that, we can use a technical solution to create incredible business opportunities. Like opening up new features to only set the sets of users or kind of rolling out things fast when we throw it fast, slowing it down and kind of opening it up only to a certain environment. Kind of leading companies that do online software kind of figure out how to do this while and why. I have a couple of friends in Australia and they're always complaining how Facebook is constantly broken for them. Because Australia is a big enough market so that it can get statistically relevant results, but small enough not to cause kind of financial problems and to screw it up. So Facebook seems to be continuously experimenting with features in Australia because, you know, they don't even need to translate it, they can just start releasing there. And then when kind of it matures, then they release it to the rest of the world. Which is kind of really interesting because, you know, they can do these experiments. I've mentioned two thirds of stuff you're going to do are going to be bad ideas. Now, if we can figure out multi-version properly, the biggest value of that is we'll be able to kind of not pollute our software with bad ideas because we can actually kind of go and look at is this a good idea or not? I can pretty much guarantee although I wasn't involved in, you know, the people think that the feedback they got from the new business dashboard was horrible. But it doesn't mean kind of, you know, what they started being was a bad idea. They just kind of didn't do their deployments and releases properly and if they did it differently, they would have gotten kind of proper feedback to discover what's good, what's bad. And that's where I can have a lot of people now go around talking about lean startup UX research, lean UX and there's a ton of kind of talks in this conference about trying to figure out what to do properly. But think about kind of how you're releasing stuff and whether that's polluting the data that you're collecting and whether, you know, you can act faster to those things where how do we take the things that have bad ideas. So generally kind of the message that I want to get you to think about is although multi-version is a technically difficult problem to solve, if we're doing continuous delivery but we're not doing multi-version, they're just irresponsible. That's irresponsible because we as technical people are causing unintended effects on sales or marketing or user entitlement and all of other things. And by approaching the architecture differently by solving a technical problem, we can start decoupling those two things and we can start allowing the marketing people to decide when something gets released. We can start kind of opening up these really interesting business opportunities for kind of releasing new things only to substitute users doing better experiments and doing kind of lots of other things now. I assume that most people in this room when they think about multi-version in their current system that they're going to get ahead, they can start vomiting. So don't do it for this one. But the next system is design. Design from scratch so it supports multi-version. And I think we're going to start seeing the same effect as we did with kind of unit testing and testing and development. Fifteen years ago, talking to people about unit testing and development, everybody would say, oh, that never worked for my system. Because people were not designing it to be like that. Once you design your system to be unit testable, it works perfectly fine. So the next system you design, think about multi-version from day one. Because that's the future that kind of is going to open up all these new opportunities. And kind of the three things to remember, kind of, you know, there's a takeaway from this is the most important thing we need to do is decouple the development technologies. There's two completely different things. And, you know, for 20 years I thought about them as kind of the same thing. One is a technical event, one is a market event. There's absolutely no reason for that to be the same. What we've done as a community is by doing faster, did more frequent releases, we've kind of forced it to be like that. It's still perfectly possible to do continuous delivery to production where we reduce the technical risk where we expose it to 1% of the users where, kind of, we see that it's actually working and then, you know, figure out later when we're going to release it to the world. And continuous delivery is a kind of fantastic force for good, but, you know, please don't use it to force future users with traffic. Whole kind of, you know, we now have a Hungarian translation. Who cares? I mean, and kind of this whole fatigue of, you know, there's a new version, there's a new version, there's a new version, just because you can release something doesn't mean you should. And the third thing that I think is really, really important, once we start thinking about that, we can start targeting our releases so that subgroups of users get 100% of the solution they need. Rather than everybody getting 2% of scope and then 3% of scope and then at the end kind of people get something useful. Because if we can do multi-version, then we're liberated from implementing everything for everybody at the same time. Yes, you have kind of 5 million users and in order to satisfy all 5 million users you need to spend 5 months redesigning the whole website. But maybe there's a thousand users that only care about the single currency account. So if you do multi-version, you can kind of slice yourself to a like that and expose it to users as they kind of can use it, which is going to give you much better feedback and you'll get much, much better at discarding two thirds of the stuff that were just bad ideas. They're not having to waste time implementing anything. So thank you very much. I hope I've kind of challenged you thinking at least a bit. And that's pretty much it. Thank you. Okay, we have 5 minutes Q&A session. Okay, we have 5 minutes Q&A. Any questions? We have a question over there. You mentioned about multi-version software. I mean nowadays it's very common to have a platform as a service to have one release for everybody. So we are in a lot of this collaboration too that we are doing this pretty much a platform where we're running lots of different things. Platform as a service means there's one environment you're kind of offering to everybody but kind of the request coming in when it hits your system you can decide where to route it. And that's what we do for example based on user kind of attributes we decide once a request comes in where does it go. And that's part of kind of your deployment architecture and your design where you can say well this request comes in I'm going to look up the user. Yes, the user has only a single currency account. They can see the new version. Or the user doesn't have a multi-currency account, they see the old version. And that's part of the kind of deployment by doing multi-version properly. For example, what we ended up doing is absolutely all the requests coming from the website have a version tag in the requests. So then when it hits the router, the router can decide where to go plus we can decide which version of the service to execute for what. So the infrastructure complicates quite a lot but platform as a service is perfectly doable with multi-version. One way would be to do that but you know we also for example our versions are typically time stamps smaller when we deploy the software. So we can say now we are letting 5% of our users use time stamps or now it's not necessarily just kind of feature based. It's designed so that everything kind of has a version from to back. So you might do it for features, you might do it in other ways but it's not necessarily features. It's kind of designing it from start so that you can have multiple versions. Good, thank you very much. Thank you.