 Hello everyone, I guess we'll go ahead and get started. So my name is Keef, I'm a consultant at ThoughtWorks and my role normally is to normally our teams go in and help organizations with software delivery projects and my role has been to work with the operations and IT side of our clients to try to help bridge that gap into production, the path to production and continuous delivery and that sort of thing. And over time, because my background prior to ThoughtWorks involved a lot of infrastructure automation and systems administration, as clients began to get more and more involved in adopting cloud and automation technologies, I got more involved in helping them and advising them on ways to do that and so I wrote the book Infrastructure as Code to kind of try to explain some of the common patterns and things that I was often helping clients with and Infrastructure as Code is a phrase that's been around for a while, I didn't coin it so I was kind of mainly trying to capture ideas and things and share those out so that was the idea of the book. What I'm going to talk about today is this idea of, well, what we're trying to do with cloud technology and how we, what we need to do in order to make it work to help organizations become more agile. So typically what organizations are trying to do these days is to move more quickly, right? This kind of digital transformation is about ways of being more flexible to try to get, have an idea and be able to quickly get it in front of users and then to be able to iterate on it and improve it and respond quickly. So speed is kind of that driving need. And the IT side of the organization has often been the impediment to that so oftentimes it's, you know, it's just taken quite a long time to say get a development environment much less to get kind of production environments and everything set up so that's been one of the constraints. So what was typically happening a few years ago and then to some extent is still happening in some organizations is cloud is seen as a way to bypass this. So for an organization, let's say a bank or a retailer or, or, you know, some large organization that's been around for a while and has an existing IT organization, they've often said that what it used to be is that they would set up a new digital unit and say we're going to bypass our existing IT organization because they're too slow and we're just going to go directly to the cloud and that will help us to move more quickly. And then over the past few years as these kind of digital groups within within organizations as a separate unit, what's happened is as they become successful they have to take things more seriously, right? Or also there's the main part of the organization which which has to it has to adapt as well and can't kind of stay in the past. So what's what's come to the the realization is that we can't simply just jump onto the cloud and ignore all of the things that the IT organization is saying about, you know, you know, compliance and governance and security and stability and all of those kind of things. As digital becomes more core to the business and as it becomes as what's going on online and on the on the cloud becomes more important, it becomes important to take it, you know, handle it responsibly. So where I kind of frame this is with this iron age cloud age thing and the idea is that in the iron age so when I was starting out as a systems administrator and for some number of years in the old days where I was dealing with physical infrastructure all the time, making a change was expensive, right? It was if when somebody said I need a new server that meant ordering hardware, waiting for it to be shipped, then assembling it, you know, installing an OS all those kind of things, taking it down to the data center, putting on the rack. So it's like it's weeks right to create a server and if I made a mistake if I'd specified something incorrectly that was an expensive mistake to fix because I might have to order more things and I might have to go back down to the data center and power it down. So the impact and the amount of time involved in making a mistake was very high and so our processes that we had then were based on let's invest time and money and effort in preventing mistakes. This is that more upfront, right? The kind of waterfall approach was a fairly rational approach when your cost of change is high and when the cost of making a mistake is high, you want to spend more time to get it right and try to avoid those mistakes. But what's happening now with cloud technology is that the cost of change is very low. If somebody needs a server, it's a matter of minutes to create a new server and if it's specified incorrectly, if it's got not enough RAM or if it's wrong type of hard drive, it's again it's kind of minutes to correct that mistake and have it going again. So this ought to help us become more agile and more quick overall. However, what a lot of organizations are doing at this point in kind of the adoption of this technology is they're adopting the technology but they're still keeping the old processes. So I'll go to, you know, I was at a client where they were very proud that they had implemented a private cloud in their data centers using OpenStack and they were saying our engineer can spin up a new server and it takes, you know, 20 minutes and we can have a new virtual machine. I mean, isn't that great? But then when I went to talk to the application teams, it still took them like two weeks to get a virtual machine and the reason was because of all the process that still had to be done. So they can get the virtual machine. First of all, they had to submit designs and plans. They had to be reviewed. You had to go to a change advisory board. Then that engineer would spend those 20 minutes creating the VM, but then other groups would have to do more things to it. So you need a different group that has to install certain types of software, create user accounts. And so you still have this kind of, you know, a lot of the old school things that we're doing to it, even though the technology should help us go fast. So, you know, the classic response is to say, well, it's just, you know, get rid of all that process and just let everybody, you know, have access to the cloud and create whatever they want and do whatever they want. But as we've said, that's not always, you know, that's not necessarily viable, right? Because you still have to worry about those concerns. You still have to worry about security. You still have to worry about stability. So it's not really the right thing. I think the issue we have here is we've got this mindset that you have to choose between the spectrum, right? We have to choose either we're going to do things carefully and do them correctly, which means we have to go slow. We have to have heavy weight processes to do that. Or we can say we want to go fast, but that means kind of abandoning quality. We're not going to kind of test stuff. We're just going to go willy-nilly and speed at all costs. And so we have in our minds, it's kind of a natural thing, I think, to believe that this is the choice that you have to make. But the reality is different. So how many people here are familiar with the state of the DevOps report? A fair number. This is a really useful thing because it comes out every year. And what they do is they survey companies. So it's based on, you know, statistical evidence from organizations or real organizations around what they measure are things like the practices that organizations have. It says DevOps, but it's kind of agile and lean in all these kind of things. It applies to all of these things. And what they found is that things like the frequency of change, how often an organization makes change, rather than indicating that they're more likely to have failures and to kind of have spend more time, say, fixing errors. Actually, it's the opposite. Organizations which change most frequently, make the changes most frequently, have the highest success rates. They spend less time fixing errors, less time remediating security issues, for example, and they tend to have more success in the IT space and even in the commercial space, quite interestingly. And that seems counterintuitive, but I think what it is, the way I like to think of it is rather than being this continuum between going careful and fast, well, like a quadrant. My quadrants are cool. And so what you can have is you can have choice, the traditional choice that you make, you just say, okay, we're going to prioritize going quickly over doing things correctly, right? Typically, you're not going to start up or we're having to go fast, right? What tends to happen is that you end up getting pushed over into this quadrant of actually things become quite messy. And even though you were prioritizing going quickly, after a while, it actually takes a long time. It actually, things are slow. And I think the kind of reasons why you see that are, it's that technical debt, right? So you do things quickly, but if you make a mess and you're not careful about it, then it's very, actually over time, it becomes difficult to make changes to it. And what I often see in startups that I've been involved with is that after like a year or so or a year or more, the founders have this thing of, oh, we've lost our edge, right? It used to be in the old days that if we wanted to do something, to implement something, it was so quick, but now everything takes so long, and it's because we've lost our edge and we don't have that sense of urgency when actually it tends to be down to this technical debt, which is actually slowing you down. So if you go the other way and say, okay, we're going to prioritize correctness, overspeed of change, we're going to do things carefully, we're going to engineer things very, very well. And so the larger, more traditional organizations tend to have this approach. What happens is, again, you end up in the kind of the same place. And I see this a lot, and I think it's because of things like the time pressure comes in and makes you take shortcuts, and the overengineering tends to happen. And so the processes that are involved, those heavyweight processes tend to make things kind of worse because the responsibility for the system is divided and different people are doing different things and nobody has kind of responsibility to keep things kind of simple and clean. And so in large organizations, what I often find is that we'll go in and we'll be talking about, say, Agile and DevOps, continuous delivery, and they'll say, no, we can't do that stuff here because what we're doing is too important. We're not a startup, we're a bank, we're this, we're that, we have to do things properly. We can't do all that kind of silly stuff. We have our processes have to be very strict and rigid, and they have very strict and rigid processes. But when you kind of peel back and find out how they actually do things, it's actually quite a mess. And so they might have a six-week process involving design documents and cab meetings to review that, you know, the plan changes and everything. But at the end of all that six-week process, what happens is somebody goes to a UI and clicks on buttons and clicks apply and they get it wrong and it breaks everything. And so what you see in these organizations is change freezes, you've run into change freezes, a change freeze to me or that it's a signal, it's a smell that your infrastructure, your systems are fragile. And that you don't have a sufficient understanding of how to manage your systems. It typically happens when they grow over time, right? It's what happens is you'll have those periods, the kind of busy periods, maybe the holidays or whatever, where we say we're going to do a change freeze because nobody touch anything because we don't know what you might break if you touch something. We don't, you know, we don't have the visibility or if somebody does break something, then it's like we have to have a change freeze because we don't know what went wrong, why it went wrong, and we have to just kind of stop everything. And you know, to me, this is a smell that you're not in a good place if you're having to do that. So the kind of alternative, what we're really aiming for here is with agile and lean DevOps, to me, I think the, in that place where the state of the DevOps report finds the most effective organizations is in this upper quadrant. And the idea is that you're trying to be careful about what you do, do it correctly, do it responsibly, but also have that speed. So how do you do that? So it's things like, first of all, it's building the simplest thing. So it's not kind of over-engineering and deciding, you know, we need to build something that's going to handle all these future needs, like what do we need right now, build just that. It's the simplest thing we can do to achieve that goal, and then make sure we build that really well, that we test it very properly. So we want to have that fast, frequent feedback. So test-driven development, continuous integration, this is kind of saying that we're building each piece and making sure it's correct and strong. We're also giving power to the teams, the people who are doing the work to make decisions because they have the responsibility, we're giving them the responsibility and the ownership of what they're working on. And ultimately it's about having our process optimized for change, as opposed to optimizing for speed or optimizing for kind of some theoretical correctness. It's like, let's make sure our process guarantees that we're always going able to go quickly. And when it starts slowing down, that's when that's a sign of, say, technical debt, right? If our change process, if our cycle time for change is slowing down, that means we have to address what's the cause of that and fix that and keep that going. And if we do that, we'll get like the other benefits kind of fall out of that. You can't go fast if you're not doing things carefully and correctly. And if we were successful at this, then we're able to make changes to our systems rapidly and reliably, so we can use measurements like cycle time and track how those go and failure rates of changes. How often do we have to roll back when we make a change? We should also be able to start new projects quite quickly. We should be comfortable and confident that if we have a new project that we've got a way to get that going very quickly. So one of the, I guess, principles that I think is important is that large batches of changes, so we all know from like waterfall, large batches, it's bad. We want to do can band lean. We want to have flow, but in terms of automation, those large batches are difficult and this is why a lot of automation projects fail. That's just infrastructure. So if you look at like automated testing, when you have a testing team that is responsible for the automated tests rather than the development team being responsible for their own tests. Typically with this kind of UI tool. So you've got this UI tool with a big suite of tests that poke the UI. When that's managed by a separate team, when there's this lag, the bigger the lag is between releases and between times that you kind of run these tests and make sure that they're all correct, the harder and harder it is. So if you've got, if it's like every iteration, say, that the QA team gets the latest release and then has to run the test suite and then they break, not because necessarily anything's wrong with the code, but because stuff has changed in the code and the tests aren't up to date to match that code. And so it takes a lot of work. And so a lot of automation test automation projects fail just because of that. It's just you can't keep up because the, and it's because the gap of the time is too long. And similarly with infrastructure and with automated deployments and all of these things, you have to be doing these things continuously. Giving another talk tomorrow where I go into some more details around some of the practices for this. But this is kind of one of the the core principles is that when you do those kind of small frequent changes, small changes is just easier to manage, right? It's easier to make that change, to test it, to debug it if something goes wrong because it's only a small bit of code that you've actually changed, you have to have to dig into. It's easier to write the automation for the deployment, for the tests, whatever infrastructure is involved, and just to understand it and do all the things to it. And so when it comes to the, when I mentioned before around the processes, the cloud age processes or the cloud age technology and the iron age processes, what's the alternative? To me, pipelines, continuous delivery pipelines is really the key enabler for being able to go fast as well as correctly. So it's taking each one of those changes and putting it through and doing validation. It's not that you necessarily need to do continuous deployment and put each change straight away into production. It's just that you have to make sure each change you make very frequently is ready to go live. And so what this means is thinking about those risks that would normally happen that you normally check for at the end of that longer process. So what happens when you're about to go live into production? And what has happened at the end to get there? So security reviews and operations folks, you know, kind of checking things over, what are the risks that we're worried about at that point? How can we bake that into the pipeline? So we're testing for it as we're developing and making sure that our code base is always from a technical viewpoint ready to go live. And so that becomes an enabler for governance and compliance. So continuous delivery pipelines are a really, really strong tool when you're in a regulated market, a regular regulated type of business, to be able to say we're going to manage all of our changes and manage our validation of our systems and our change, our change management process is basically embodied in this automated pipeline. And so where people get involved, people are often wondering in a new world, we're doing continuous delivery, we're doing infrastructure as code, we're using cloud and trying to go quickly, where it is, you know, security folks fit into it, where are the kind of operational, you know, all those kind of people, where do they fit into things? And the answer is that they fit in at the beginning, so rather than getting a hand over from development at some point down the line and inspecting releases, what they should be doing is working on the pipeline to say, okay, what are the things that have to happen? How should the software be deployed? How should the infrastructure be configured? All of those things should be defined in code that can be looked at and audited. So it's not documentation that says our change process is somebody does it this way, except when they're in a rush. Our change process is an actual script that you can look at and you can see the log output from every environment that that is exactly what was done and what the results were when we ran tests against it. So in terms of how to kind of make this move, how to kind of do the transformation, the kind of process side of it. So one of the things we're finding a good success with is this idea of the lean value tree. And what this is, is where you kind of articulate as an organization, what you're trying to achieve your vision, because of the change program or just generally for your group. And one of the different goals that we think will help us achieve that vision. And these need to be quite concrete. So we actually use this internally at ThoughtWorks and our technical operations department. And we found it works quite well. And so what they do, so let's say in terms of infrastructure, we might say one of our goals is that we want to reduce cycle time. We talked about cycle time. So we want to say that let's reduce the time it takes to get things into production. And so what you can then do, what you then do is you look at, OK, what are some of our hypotheses? What do we think will help with that? And how do we, you know, for the goal level, you kind of, you have a measurement ready there, right? Because it's cycle time. That's a measurement we can engage our progress. And then you come up with a hypothesis which says, well, OK, we've done an analysis and we've found that a lot of time is spent fixing things because our environments are not consistent, right? You deploy things into a development environment. You run tests. But then when it comes time to deploy it into production environment, it actually doesn't deploy very easily. And you have to kind of do custom things and spend a lot of time getting it deployed and working in that environment. And then things don't always work the same as they did in our test environments. So our hypothesis is if we can make our environments consistently managed, then that will reduce our cycle time. So that's a very kind of clear statement of something that you can try to measure. And then you can have some initiatives to, OK, we're going to implement terraform, we're going to implement puppet, what have you got to, you know, the specific things that we're going to put in place, which we think will get us some progress along those goals. You can map the things down. And this really helps. So I mentioned that we use it internally. It helps because it helps to focus what you're working on because it's very easy to make the big list of things, all the work that you need to get done or that you think would help. But you can kind of, you know, decide, you know, you need to really kind of limit and be very kind of brutally, you know, brutal about your prioritization and say, we're only going to work on a certain number of things. So what we've actually started doing is putting percentages of budget because this also comes down to investments. So how much are we going to invest in terms of people's time and any other kind of resources that we need to put into these different things? And so we'll have like a percentage across. There's, you know, 40 percent on this goal, 30 percent on that, 30 percent on that. And then very frequently review how we're actually making progress on that. Are we actually working on these things? Have we taken on too many things we need to rebalance our priorities in any ways? Then another thing that we find is quite helpful for these kind of infrastructure projects where we're helping people to adopt cloud technologies and build platforms and these kind of things is it's actually taking the techniques that we take from software development. So I think traditionally when we have these projects that happen more on the IT side of the organization on the infrastructure, it's kind of viewed as well as a technical thing. So we're just going to let the technical folks, you know, assign a few technical people to work on that and they go away and do it and we'll see what they come back with kind of thing. We find about thinking about it the same way that you would think about a software that you're working on for your end users is quite helpful. So who is actually going to use this platform that we're building? Because what typically happens when you don't do this is that you spend a huge amount of time, you say we're going to spend six months building this platform and then 12 months, 18 months later, you're just kind of finishing up and you release something and then it's kind of not what people actually want to use and it flops or you just get pulled before it gets finished. But thinking about, you know, who are the users of this platform? Well, it's the developers, it's testers. What do each of them want to get out of it? So a tester wants to get, you know, so a developer comes to a tester and says, I finished a story. I need you to do some testing on it. So the tester needs to get that build into an environment where they can go ahead and test it. And so what's involved in them doing that? You know, is it they have to go and raise tickets or go and find somebody to deploy something to an environment, make sure that see if there's actually an environment ready? Or can we, going back to that lean value tree, can we actually prioritize some of the effort to say we're going to make an automated process so that the QA can click a button and then their build gets deployed and the way they go without having to interact, you know, to interact with anybody else. So that becomes very efficient. So it's thinking through what those working processes are for the various people who are involved in your platform. Turns out to be very useful. And then another thing we need to think about is across functional requirements, non-functional requirements. Call it what you will. It's those things like performance and security and so on. And these are often challenging because we want to just, you know, do all the things, right? We want to be as secure as possible as perform it as possible, scalable and so on. But really we have to think about those tradeoffs. So one of the exercises this comes from the Building Evolution of Your Architectures book where they talk about fitness functions. The idea is to work out the relative priorities of these different qualities of your system, characteristics of your system, what they are, how you measure those things and then be able to kind of map that out. And then you can put that into how you actually build your stuff. So getting into that, this is like taking the evolutionary approach to building out your system, including, so here I've kind of simplified in our application is just like one column of stories. And it's kind of like this is the sequence, the first, second, third. So it's kind of like this is how we typically will manage a backlog of work for a piece of work. And so these might be epic level things. And you have different kind of concerns or capabilities. And so in terms of the platform and infrastructure, what we try to do when we're going into a client who says we want to build a platform or want to adopt a cloud, whatever it is we say, let's find an application and find a team that's building, you know, that needs to build something on this and let's work in conjunction with them. And so this lets us take, if you think about, so identity management is a good one. I was on a client that's a global finance organization that was looking about to do move into the cloud with the system. And in terms of identity management, they had active directory in their data centers. And what they were kind of their initial stats, initial stats was we need to have active directory in the cloud. You know, that's that same active directory federated into the cloud and all the kind of systems and tools. So things like the monitoring tools and, you know, the CI server and all these kind of things, the source control server that were being hosted there, all need to be integrated with our active directory in the same user accounts, which is great, right? But it's a it's a big piece of work, especially for an organization which is just getting started with the cloud and also because of their security concerns, they're reluctant to like use the cloud hosted things like for the source control and for the active directory. They don't want to use the cloud hosted ones. They want to stand up virtual machines. They can have more control over the data and everything. It's kind of fair enough. But it's a lot of work. And so what we can do is say, actually, let's kind of sequence this out so that we can build this up. We can we can build out. We ultimately want to get to that federated identity where we're synchronizing with the cloud. But if we can come up with some interim stages to that. And so what we said is, well, first thing we're going to do is so we're going to have an application that's being developed. And it's just going to be internal for a while. So they've got their backlog with, you know, it obviously doesn't look like this. It's got real stories and everything. But at some point in their backlog, we've got enough stuff that we want to do and internal beta let people inside the company use it. And so we can do kind of a minimal set of stuff. We don't have to have this at that point because actually at this point, we've only got our Jenkins server and our BitBucket server. We don't have a whole lot of other servers. So we can actually say, well, just use Jenkins user accounts and BitBucket user accounts. And there'll be a separate user account on each. And it's not, you know, single sign on or anything. We will put in some basic hygiene around security. So we're not going to have these things are going to not going to have no password on them. And they're not going to have like a shared admin, you know, log in with admin, admin and do anything. We're going to have different accounts for everybody. So we'll do some basic stuff uses SSL certificates. So stuff that's not that difficult to implement and is good hygiene, but isn't everything that we have to get to eventually. And then once we've kind of got that up and running and we're getting feedback and we're learning stuff and learning how to use the cloud more effectively, you could get again, this is an organization which is just getting started with cloud. Then we can move to say, OK, let's get active directories stood up in the cloud. But we're not going to federate it yet because one of the things that gets us is we don't have to worry about one of the concerns with federating from the on premise active directory was that's all of our kind of companies user data was being synchronized to the cloud. Again, not very mature with the cloud, quite worried about the security implications. So if we can kind of take this as an interim step, we don't have that it's much less data. And so we can kind of not have getting to this point in terms of all the security stories which aren't shown here, but there's a whole bunch of security stories we did have in place to get to this point to have data being synchronized from our data center into the cloud. And it's saying, if we take some interim steps, we don't have to do everything quite yet, right? We can take a subset of things. This just helps us kind of evolve. And so for all of the kind of different aspects of the capabilities of the system, you can build them up incrementally as we go. And so then kind of like the final point on this is that the way we then write our stories for those things which are a bit more technically oriented and a bit more kind of cross-functional, we have to get quite good about articulating the business value of those stories. We can have conversations. So this is kind of a typical thing that we used to do. And a lot of people do, which is just to say, something like as a DBA, the database to be replicated. So it's fault tolerant. And then you take this to the product owner and like, I have no idea. Why do I care about this, right? I need to have this user interface stuff going on in my application. I need to have these things that I understand. That's not something I understand. So I'm not going to prioritize it very high. So we typically have that kind of tension between those kind of business people who want to get stuff done and the technical people who are quite worried about the shortcuts that we're taking, the things that we don't have in place. And so we need to get better at articulating the business value. So one of the things in terms of like availability and backups and that kind of stuff, I find useful is to describe the event that happens. So when there's a region failure in the cloud, what happens? Well, how much time is it going to take to trust and bring it back up? How much data might get lost in that scenario? Another thing is like, who has to be involved in actually making that recovery happen? Who do you have to have on call or what have you in order to make that happen? And when you start articulating it this way, this also helps with doing that more incremental thing. So we can have, again, for that availability question, we can have, okay, the ultimate thing we would like to get through is we want to have a continuous replication across our regions and we want to have it automatically recover. But that's, you know, a fair bit of work to do. So we can say we can have some interim things. We can have conversations with our business stakeholders about how much time you want to spend on this. So like the simplest thing is to say, we just have no guarantee, right? If the region fails, we're just going to have to wait if Amazon or our cloud provider has kind of the data's been lost and we don't make best efforts and whenever we get back up, we'll get back up. And that can actually be acceptable in some circumstances where it's not a business critical system and particularly where it's in development and all of that. But then you might want to have things where we say, okay, we're going to put some basic things in place. We'll put in some nightly backups. We'll have a documented recovery process and that'll mean that we can give you the next business day recovery, you know? And probably we'll do better than that. We think we can probably do better than that but that's what we'll kind of say to you is that what might happen is it'll take us to the day to get the system back up again. We might lose a day's worth of data because we only have back up some last night. You know, is that acceptable? Or do you want to go to, we're going to take snapshots every 10 minutes and have a scripted recovery process. So again, this is kind of because it's put in terms where the impacts to the business in a non-technical sense is easy to understand and where we've mapped that to what's the work we're going to have to do. We can put estimates on this and say, if we really want this, that's fine but it's going to take us this amount of time and you're going to have to prioritize that over getting other work that you might do. So you can have that conversation. So that's pretty much what I've got to say. I'm happy to kind of take questions and have conversations about some of this. And I will say tomorrow I'm going to go into a bit more detail in my talk tomorrow around things like pipeline for infrastructure and some of the patterns around implementing some of the stuff that's underneath what I talked about. It's a little opaque from a dev team perspective or from a deployment but isn't it still happening? So how is that factored into what we are talking here? So it becomes a different conversation or a different process and it kind of depends on who you're getting it from. Public cloud it's fairly easy because it's like that's Amazon's problem, right? So I want to be able to provision servers and I expect Amazon will have the resources available that I can just do it. Internally, if you've got a private cloud it becomes a planning and capacity management exercise. And so yeah, somebody's having to do that. Development team, hopefully if the people running that internal cloud are able to get to a good place and being able to do that well then you kind of don't notice as long as you don't have some kind of crazy, we're now building a system where we're ramping up for an enormous amount that nobody knew about. It's like where you talked about the governance and some of those things built in as a pipeline or built in into pipeline. What's been your experience of having worked with any of your companies which were in a regulated environment. For example, I come from an insurance major. It's fairly regulated and you know the reason why it would be. Getting that into your pipeline and getting somebody to trust the system to say that I don't need to have a physical compliance and it can go based on a rule engine. What's your take of larger industry experience Yeah, so we've implemented that at several banks including some very large global banks so as well as in other industries that are regulated so it's definitely a doable thing. It's one of those things, as you say, it takes a little bit of time for people to get the confidence in that but conceptually it's quite strong because if you don't have the automation you're relying on humans, right? So what we've often done is so first of all we try to engage very early with security and anybody else. One of the techniques we use is what we call a tracer bullet pipeline and that's what we say. We're gonna get something very simple maybe it's just hello world but we're gonna get that into a production environment something that is classified as production as early as possible and we'll start out saying we wanna do this within the first week if we can, right? Of course we can't do it in the first week in that kind of environment. What happens is that kind of sets the trip wires so it's not always easy to find out up front who has to sign off on things before they can go live but as soon as you're trying to push something live that's when they all jump up and say wait, you can't do that, you can't do that and say, okay, that's fine, why not? What do we need to do in order to be able to do that? And then what will happen is that so in the early stages we'll implement the pipeline but you have manual gates in this process so you can say at one of those stages in the pipeline security person can come in and be the one to you can have like credentials put in your credentials to say that I have approved this so they can review the changes and we've done things where we like package up the change logs, we did this on one project with databases where we were using it was DB deploy at the time and that managed, automatically managed schema changes and the DBAs weren't comfortable with that so we said we can spit you out on the pipeline here's the actual schema changes that will be applied and you can review that and then click the button when you're happy with that and so you can give those kind of things where they have the ability to review logs, review tests, whatever it may be and then over time they start to get comfortable and they say I kind of don't want to bother doing that anymore every single time we'll kind of relax the process a bit and still come back and do kind of reviews of to make sure in audits of the pipeline and the testing to make sure that people aren't turning things off or whatever but yeah, it's a journey but definitely it's been done. Thanks. So my question is in regards to the pipelines and how do you manage? How do you get started with those pipelines and where have you seen actually systems and production that it's not as simple as the five blocks from left to right? Oh yeah, it's never as simple as those five blocks, yeah. It's more like you have your reporting engine, you have your payments gateway, you have your front end, you have like mobile applications, it's back to that system diagrams that is like a level one zoom with hundreds of blocks so how can you manage a pipeline? So my talk tomorrow actually goes into more detail on that but it's just a matter of having essentially you end up with multiple pipelines for different parts of your system. Essentially, you'll test like each component in isolation before you kind of release that and then another pipeline will pick it up and integrate it with other things. So that's kind of a very quick summary of how that works. Okay, then a more, how do you version control it and how do you approve changes to the pipelines? I mean, as you evolve the systems, as you bring more workloads, how do you do those changes? So I'll show one diagram here which was this one. So pipeline definitions, it should be in code as well so that when you're managing changes to the pipeline, you're doing it the same way, your deployment scripts, all the kind of things should be kind of managed as code so you can review and see them. In terms of versioning, so there's two main strategies. One is the kind of fan in approach where you take all the different components, you basically bundle them together or maybe have a descriptor file which lists all of the components and version numbers and say these have all been deployed together into this environment and tested. And so for the downstream environments, it's that same collection of the same versions of components that are kind of deployed together and tested together so you have confidence. That doesn't necessarily scale once you get to a very large organization and many things, it becomes a bottleneck so you can move towards a more kind of federated model, so kind of microservices approach where each thing can be pushed into production on its own but there you need to get more mature in terms of your automated testing. Things like contract driven testing and API management where you're saying, okay, my component is relied on by other people's things so I need to make sure that my API doesn't change or is backwards compatible and I can even take tests from other teams and run them against my code to make sure I'm not breaking them but I still have the independence to release and also things like kind of the deployment patterns doing things like canary releases and blue-green deployments where you deploy a change into production but it doesn't necessarily get used by everything and you can kind of test things there, get higher confidence basically in your actual release to production process as well as obviously strong monitoring. So that kind of thing is doable but it takes a lot of mature to make it work. Is that the kind of thing you're looking for? Yeah, exactly. Because I want to move away from the change approval boards and it's even if you actually store it in one git or it's still up to a couple of reviewers and it still takes weeks and you need that level of assurance in order to do it as code. Yeah, what we've done some time and this goes back to that kind of more interim thing is having pipeline stages that bundle up the evidence and the documentation and then push it into like a change management system. So something like ServiceNow, it uses the API and says, okay, here's the version, here's all the documentation, all available bundled up and packaged and then whoever needs to approve it can go and look at it and push the button which then allows the pipeline to make it happen. Then that's hopefully something that over time people get more comfortable with making that a bit more efficient. Yeah, as a follow-up question, how do you specify them? Are these DSLs or like some kind of standard languages? Like how do you specify all these compliance processes, definitions? It all depends, it's usually like various different things. So for the compliance and things like that, it's automated tests is what it comes down to and so you have things like, so Chef InSpec, I think HashiCorp has come out with something compliance as code and so it's different DSLs depending on which tools you select but they're basically making assertions so you can have things like saying that no ports are open except for disallowed port or is those kind of things or there's no user accounts on the machine or no accounts with passwords or whatever kind of policies you want you should be able to assert a code. Thank you. Quick question on the security and vulnerability aspect of cloud applications. There's a lot of hype around the security and vulnerability when we talk about cloud applications. So do you think it's a little too overhyped? So security in the cloud, I guess I wouldn't quite say it's overhyped. I think it's what's overhyped maybe is the idea that data centers are secure. A lot of the high profile attacks that we've seen and leakages and stuff have been in data centers. So there's nothing inherently secure about data centers and in a lot of ways cloud providers and cloud vendors are a lot better at security than most of us are gonna be able to do. So I remember talking with somebody at a bank who was saying we've invested this money in a distributed denial of service attack prevention tool and we're really good at this and obviously Amazon is not gonna be as good as us and I was like, are you kidding me? They've spent billions on this kind of thing. What I think is important though is security is different on the cloud. There are some different risks and different concerns and so you need to be conscious of them and you need to understand them and really take steps. So just for example, there's the cloud account. You have an API key on your AWS account which can be used to destroy everything in your AWS account. And so that means you need to think carefully about your account structure, obviously the kind of accounts that you give out to different people, principles of least privilege, but also that kind of what happens if that does happen, if that account gets compromised, is our source code also is infrastructure is code. One of the great things is that you've got the source you can reproduce and rebuild everything. But if your source code is also sitting in that AWS account and an attacker trashes that then you're just gone, right? So how do you, there are strategies to manage that but you just have to be aware of those and do the right thing. Thank you. So in terms of banks, so there's, is it square in the US? It's payment processing that a lot of people use in that source of the cloud. Certainly a lot of, you know, e-commerce retailers who are doing credit card payments or are doing that in the crowd, in the cloud. I have worked in the UK with a company doing credit cards, payment processing. We have not with them yet gotten to the point of putting their production in the cloud. Although that's on the road map, it's more their development environment at the moment. Yeah, yeah, I think, so I can't speak authoritatively to the square folks what they're doing but I know they're doing, you know, proper credit card payment processing. I've got that right. It's very familiar with the, I'll say square or four square. Now, four square was the location company. It is square, okay, it is square. I was, okay, yeah. So yeah, they're based in Silicon Valley, I think, and yeah, they're doing credit card payment processing. Whether they're connected to the Swift network and all that, I'm not sure. I know that SunCorp and ING are both doing stuff in the cloud. Again, I can't say to what extent, you know, what activities they actually have going on in the cloud. I'd be pretty surprised if somebody's not doing it. But yeah, like I said, I can't point to somebody that I know 100% is doing that level. Sorry, there was a, okay. All right, well, thanks a lot. Yeah, I'll be around. Cheers.