 In terms of the program, it's on our website, but just to refresh your mind, it's actually a single track conference. So we only have one track, that means that we're all in together. We don't do Q&A. What we'll do instead, or obviously we'll have about 200 people who have registered with this event, we have open spaces. So that's our way of bringing the community together, with the speakers, with the sponsors, everyone in a big circle, discussing whatever questions they have during the presentation. And to make it more easier, if you have a mobile phone, the Android iPhone, maybe a Nokia, please point it to this URL, and you can actually ask questions on it. And then you can vote on the questions that actually are to interest for you. So this will help us better organize for the open spaces. Obviously we'll go through the mechanism of open spaces later at about 2 p.m., explaining how we'll split the room and we'll have multiple open spaces work groups. So please, if you have, it's very easy, pigeonhole.at slash .sg, demo stays as g. As well, if you are a social fan, if you are a Twitter, please tweet what you find interesting. We will have a wall and we'll be constantly updating the wall in terms of how every session has passed and it will be good for people who are attending. And as well, those who are attending via live stream or will want to look at this event in the future. It will be a good memory. So keep tweeting, use the hash, tweet hash demo stays as g. Give us feedback and that's about all. I think logistics, we have Matt who's over there in the corner. So he's helping us on the menu. Restaurants, if you need there on the right side, you just do a circle. And as well, if you go by the main entrance, through that, after the elevators, there's a door. So you'll find plenty of restaurants as well. With the help of the view, thanks again for coming. It's our pleasure to host you here and please give us the opening keynote. Thanks a lot. Thank you. Thanks for having me. How many folks is this your first DevOps days event? Quite a few, quite a few. I've been to a number, it's my first time in Singapore, much less my first DevOps day Singapore, but I've been to a few in Europe in a few different cities. And I quite like this event compared to other conferences because it's kind of more closer in a way. And also the way the open spaces works is quite nice because we get a chance to have conversations. So rather than kind of doing Q&A back and forth, it's kind of a lot of interactions between people to have conversations about things. So I found it a really good experience. I've really enjoyed them. So what I'm going to talk about today is around automation. So one of the common themes you hear in DevOps days and other DevOps things is that DevOps is around culture, it's not around tools. And yet automation is still one of the kind of key enablers, one of the things that helps us to work together. So I think it's interesting to explore a little bit. Why do we do automation? So first a little bit about myself. As you've heard, I work for ThoughtWorks out of London. I've been working for ThoughtWorks for about seven years before that. I guess overall, I've been in about kind of 20 or so years in IT and working across development and operations roles. ThoughtWorks, what I do is we tend to go into organizations to help them learn how to deliver software more effectively using agile approaches, lean, continuous delivery, these kinds of things. And I've tended to get involved in engaging with operations teams. And one of the things I found was that people were using the technologies for automation and cloud, but they weren't necessarily adopting the practices that a lot of the more effective organizations were doing. So that was kind of one of the more interesting topics to me, which is why I wrote the book to kind of share what I'd learned, mostly what I've learned from other people and what I've kind of tried out from learning from other people. So as far as cloud and all these kind of automation things go, the question is what benefit are we expecting to get from these? Now, in some cases, particularly management, there's this idea that maybe we're going to save money by having fewer people or that we're going to have people who maybe need to be less experienced, less skilled, and so we can save money and save costs that way. I'll talk a little bit at the end about my kind of thoughts on that. Other things people think about is to be able to scale more quickly, especially if we need to scale the organization in terms of the amount of different things that we're doing, different teams using systems, or just the size of things that automation can help us do that, maybe to make fewer mistakes. So if you have fewer manual tasks that people are doing by hand, the less mistakes people are going to make and the smoother things are going to go. But I think, well, I'll tell a story back in the 1980s from automobile manufacturing. So the American automobile manufacturers back then, General Motors, Ford, Chrysler, they were really struggling, they were really suffering, and they were being overtaken in the market by Japanese manufacturers. And so this is a big concern for them. The Japanese companies were making cars more cheaply. They were able to respond to the market more quickly, so they were making cars that people liked better and were buying more of. And so there was a big crisis for the American automobile industry. And so what they did is managers went to, from the American companies, went to Japan and did tours of factories and have a look. So what is it that the Japanese companies are doing that's so effective? And what they saw when they first went was what was automation, right? So they had all these factories, and they said, oh, look, they have these machines and robots doing all the work and fewer people. So that's how they're doing it. That's how they're saving the cost, and that's what we need to do. So they went and invested billions and billions of dollars in automating their own factories. I'm thinking that would help them out. And it turned out that didn't work. They actually lost even more money because they had spent even more money. And so they had to go back, and over the years since, I think we've explored more about what companies like Toyota have done with the Toyota production system, and we've come around to the whole lean manufacturing thing and realized it goes a bit deeper. So what had happened to the American companies was the way that they were manufacturing cars before automating was they would basically produce like a batch of say 20,000 cars, all the kind of same model, the same configuration, and then they would retool, and they would produce another batch of some thousands and thousands of cars, and they would kind of produce these batches and put them in lots and wait to see if they sold, and if they didn't sell, then they lost loads and loads of money. And so what they did when they automated was exactly the same thing. So what the Japanese companies were doing, what the Lexi Toyota were doing, using automation to be more flexible to say that actually we don't need to produce massive batches of cars that are all the same, we can actually kind of retool as we go along so that each car that comes off the line could be a different configuration from the car that was before it. And so this means we don't need to build up those big inventories which are a risk of losing money if we can't sell them. And so essentially I see this kind of parallel to what's going on with automation today with cloud, with things like Puppet and Chef and all these kind of tools, that companies are adopting the tools thinking it's going to get them something, but they're not changing how they work, and so they're not seeing the benefit because it really is about using the automation to enable continuous change. And so this is kind of my thesis is that the reason we automate is to enable change rather than to save costs or anything else. There might be some things that come as a side benefit, but this is the key point. Because organizations want to go more quickly. They want to get time to market more quickly, respond to needs particularly. So a lot of the clients I work with in the UK are retailers, media companies, increasingly finance companies. We're feeling threatened by these kind of digital native companies, so they're threatened by Amazon, by Google, and so on. And they're trying to figure out how can we adapt, how can we do what those companies do. Again, we're seeing this repeat of what happened with the automobile industry, of people looking at the trappings, the technologies, and saying we need to adopt the technologies in order to be able to compete effectively. But the problem is that there's still risks. So if we're trying to go very fast, even with these technologies, we run the risk of having errors or having problems. And so we need to figure out how do we balance these. Because there is this perception that we have, the kind of intuitive idea that you have to choose between either going quickly or going safely and correctly and properly. And so to a lot of companies who are adopting Agile and DevOps and these kind of things, they kind of assume these mean abandoning quality, right? That Agile means you don't test, you just kind of rush things as fast as you can into production and don't worry about things, you know, move fast and break things. But the reality is that companies that are very effective at IT actually manage to do both. How many people are familiar with the state of the DevOps report? So I hope with this conference, yeah, okay, quite a few. So I think that what the really interesting finding from this for me was that those companies which are most effective in terms of financial performance and also in terms of, let's say, fewer failures, fewer failed releases and less time that their staff spend on remediation, on fixing things and dealing with issues. Companies that were really good at that were also the ones that were releasing the most frequently, which is kind of counterintuitive. We assume that you have to go slowly and carefully in order to get that level of performance, but it turns out the opposite is true. So I think about it as the difference between Iron Age and Cloud Age of managing our IT. Iron Age obviously was when we had physical hardware and that was the basis that we worked with. So in terms of provisioning or implementing a change, when I was managing based on physical servers, if somebody asked me and said, I need a new server, I need an environment, that meant I had to go out and do purchasing, I had to go out and order a bunch of hardware, wait for it to be shipped to me, then we'd have to go and we'd have to assemble it, manually install the operating system and other software onto it, take it down to the data center, put it in the racks, cable it up. So it was weeks, right? It took weeks in order to provision a server and if we needed to make a change, it was a very long time as well, whereas with the cloud, but I want to be clear, when I talk about the cloud age, it doesn't require the literal kind of public cloud, Azure, Amazon, these kind of things. You can also do these things with virtualization. You can even do them with physical servers using things like form and to automatically provision hardware. You can get the same kind of effect and the same benefits. So it's that kind of more of the mentality in how you're using the technology. And so with these technologies, you can provision new servers in a very, very, you know, quick amount of time. And so this changes how we think about change. In the iron age, if we had to make a change, so if I went through all that process of ordering the equipment and getting it into the racks down on the data center, and it turned out, we spin it up and it turned out actually, I didn't have enough RAM in those servers or the hard drives that I'd ordered weren't fast enough. This was a mistake. This was a problem that I would have to go and explain to my management why I now need to go and ask for more money and have more delay in getting these things ready for them. And so change is a negative thing. We need to change management. We need to change control. We need to change boards. So change is a negative thing and a sign of a mistake, if you can move very quickly to correct things and tweak things, it actually becomes an opportunity. It becomes a learning. So if it turns out that I needed more RAM on those servers that I provisioned in the cloud, well, that's fine. I can correct it and, you know, just go and make a change and maybe at the worst, read Buddha's server. And there we go. We have it, right? So that becomes lower cost. And it becomes a good thing. We've learned how much RAM we need. Actually, maybe we need less. In the Iron Age, if it turned out that I'd ordered too much RAM, you know, that was it, right? I mean, I didn't tell anybody. You know, we had too much RAM. But now we can kind of tweak it down and save some costs. We have an opportunity there that we didn't used to have. And so our processes that we use to manage change in the Iron Age is the focus is on the mean time between failure. How can we prevent anything from going wrong if at all possible? Whereas with the cloud age, we can think about the mean time to recovery. And that's what makes repair. Now, it's not to say that we're choosing one over the other that we no longer care about failures and we're not going to do anything to prevent them. It's just that kind of prioritization in the kind of old style process is to say that, you know, oftentimes the mean time to recovery can be quite long because our process to fix things is so long. And so going back to that state of the DevOps reports of why is it that the effective organizations are fast and more effective. I think it's because they can fix things more quickly. So if you have a change process that means you know something that's wrong with your systems. It's not critical, it's not broken right now but it's maybe it's technical debt or it's something that has a risk of going wrong later on. If it takes you several weeks and a lot of work in order to make the fix to that, if you have to go through filling out forms and raising tickets and going to change advisory boards and explaining why you want to make this, you don't bother to make those fixes as much. You just kind of put them on the list of our known issues, we know the things that are wrong goes on to the backlog of technical debt. Whereas if you have a very effective process that can very quickly, I'm just going to go in and tweak that and run it through and it gets tested and goes into production quite quickly and safely. You fix things as a kind of a matter of routine and so the overall quality of your system goes up and up and everything becomes more reliable. To the ideal, what we're aiming for when we adopt these technologies is to lower our cost for change and the risk and the time that it takes to make a change from the Iron Age down to the Cloud Age very, very fast. What we find in a lot of organizations is that actually we still end up with it taking a really long time. We don't get all that order of magnitude of benefits. So for example I'll work, I've talked with engineers who they've gone and implemented a private cloud with say open stack and the engineer tells me it used to take us quite a lot of work to provision servers but now I can provision a server in 15 minutes and it's really quick, that's great. But then I go and talk to the users, the people who actually need environments and servers created and they say it still takes six weeks for me to get a server even though it's a virtual machine and it's an open stack and that's because of the process we're still using the old processes we used before so in order to get that virtual server on open stack I still need to fill out a bunch of forms justifying why I need it what I'm going to use it for and so on and so on and also maybe the processes that take 15 minutes to that server image but then somebody else has to come in and do configuration on it so they have to set up the user accounts and security somebody else has to install packages application servers and so all of this kind of means that we don't actually get the benefits so it's the process that we haven't changed to take advantage of the tools and often as well there's risk again going back to those companies who are trying to do this in order to adopt Agile and DevOps and become digital sometimes either the processes they have actually don't manage the risks that come with the technology very well or they kind of abandon some of the processes and let people take shortcuts without something to kind of replace that so there are actual risks right so when we talk about these things with the iron age and how slow it was it's not that people were stupid right it's that there are actual risks and actual costs and you still need to manage those risks and those costs in some way so this is kind of you're not really getting the cloud age you're getting more of a kind of failed attempt at cloud age when you do this sort of thing so what we want to get to is where we really are in a cloud age way where we're reducing the technical impediments and costs in order to do work but we're also finding ways to have processes that can effectively manage things that are also very quickly and how do we do that we do that by leveraging those technologies those cloud age technologies and it's the automated testing and the continuous integration, continuous delivery we can leverage those in order to get good change control processes then we're going to benefit right and this is what agile DevOps and these kind of things are really all about their kind of modern processes for managing risk and change so some of the kind of practices and things that we can bring to bear on this are obviously it's defining everything as code our applications are defined as code already our infrastructure if we're using things like Chef and Ansible and what have you then our infrastructure is now defined as code maybe Terraform, CloudFormation and then our tests themselves should be treated as code rather than things that people have to carry out manually for the routine and repeated tests and the processes so how do we deploy software how do we control so those things where we have gates, particularly in regulated industries where we have to have auditing and so on how can we leverage the tools and make that defined as code so we can say here are the requirements here are the kind of acceptance criteria here are the kind of fitness functions here are the security rules that have to be adhered to how can we define that as code so that that can be automatically managed and so this lets us automatically test every change so when we have things I'm sorry I'm giving you a hard time here aren't I with the video yes if you're all of these things are defined as code that means every time you make a change to them that is a kind of a discrete visible thing that you can manage it goes into a commit into source control and therefore you can take action it becomes an event and again it's not just the applications and the infrastructure but if somebody changes the process that becomes an event there's a new change to our process a new requirement we have we can trigger our continuous integration to check our existing system against the new requirement and prove that it's good before we move on and start making more changes and so we want to have our our our changes applied by our system so for using tools like Chef Ransable or Terraform or the kind of you know resource management these kind of things we want to have the tools we could use them from our local kind of workstations and desktops and each of us runs them but that runs the risk of A of inconsistency you and I might have different versions of some of the tools on our laptops and therefore we get different results when we run the process but also there's the the issue that we might not do all the things that we need to do so we might not run tests we might skip some things because we think I don't need to bother with that for this change and so that leads to not kind of enforcing overall our regime and our process that we've automated as thoroughly so the idea is to use a continuous integration server, a continuous delivery server and have changes, have the actual tools run on those so you commit your change to source control and then that triggers the server to actually run and execute the change on whatever environments test environments, production environments they may be and then we can promote those changes through environments through a pipeline so this is that delivery continuous delivery concept and it's important to note that this doesn't mean so I think there's a common perception that continuous delivery means every time somebody pushes a change a developer makes a change to the code it kind of immediately goes through to production without any kind of oversight which isn't necessarily the case so you can design your pipeline to have kind of stages where somebody reviews changes before they go through, somebody approves changes you record so that you get for those kind of compliance regimes where you need to have multiple people reviewing each change, you can actually have your pipeline enforce that and record that and make that auditable and provable so then how do we keep things consistent with our provisioning servers and infrastructure as code how do we make sure once it's out there and running over time and we're getting more and more servers running how do we make sure that things keep consistent and keep well managed and updated configuration drift so we want to avoid the situation where we've created a couple of servers and they've started out identical but then over time people have made changes here and there on different servers maybe manually maybe they fixed an issue they found on one server and they haven't made the change on another server because that one didn't need the fix and so they end up with servers that are inconsistent that are not the same and so when they try to run their automation tools again on these existing servers they fail or they work on some servers and not others and that increases our confidence in our automation which then leads us to more break our habits and say I'm going to go and make changes manually because the automation tool tends to break and so then it kind of becomes this downward spiral where we don't have the confidence to use our automation tools as thoroughly and consistently as we should so a lot of this comes down to automation lag so how long is the gap between running your automation tool on the kind of same environment so if you only use your automation tool in order to make a specific change so I need to upgrade Java on those servers so I'm going to tweak my Chef cookbook and then run Chef as a kind of almost a manual activity or a manually managed activity against those servers that means that it's the next time around that I run it but things might have changed you want to be running them continuously so Chef for example and Puppet are designed to be run on a schedule every half an hour every hour whatever it may be even if changes haven't come in so that it kind of reinforces whatever is happening so this idea of continuously synchronizing your configuration regardless of whether it's changed just to make sure that if something has gone on in the system somebody's made a change to the system or somebody's made a change to the automation that you found out right away rather than maybe a week down the road two weeks down the road where things might have accumulated to become to the point where things break and it takes too much work to correct them so all of this stuff lets us do some interesting stuff and change our mindset towards design and architecture to say that rather than designing with an end state in mind so this idea of saying we know where we are now over there on the left we know we've come up with this design for the end all be all final state of where we want to reach and it becomes this big kind of project to get there we can actually take things in kind of small chunks and so we can say okay we know what our target state is what we think it is we can make one change to get us closer to there and another change and another change and another change and along the way we may learn some things we may learn that some of the assumptions that we made at the beginning actually weren't true so maybe we didn't need some of the optimizations that we thought we might need some of the caches or things like that in order to optimize what we assumed we would need to do to optimize the performance of the system actually we find out as we go along we do performance testing as we go along and actually we didn't need that after all so we can keep our architecture simpler and also by doing this it forces us to design in a way that makes it easier to change so this is the whole idea behind test driven development which is that if you're writing your tests or your applications as you write your infrastructure rather than saying we'll write some tests later on once we finished it it actually affects your design it helps you to design stuff that is easy to test so it's easy to make changes to it because in reality you don't really get to an end point with your architecture and system design because as soon as you've gone live with there this is our big release you need changes to it right you find that either there's things that you postpone that you didn't quite manage to fit into the release or there's issues that you found that may be non-critical but we want to go back or you just start getting feedback from your users as they start to use it and say can you add this can you change this so really there is no end state for our system design and so we need to embrace that mentality that things are constantly going to change and that's a good thing so I'll talk a little bit about the processes this is kind of around agile and this kind of thing and continuous leverage and where we get to because again this is applied this is kind of around the change management process and so this is somewhat familiar for development most organizations are doing some kind of things around agile for their software development but fewer are doing it around infrastructure infrastructure architecture and how we change our infrastructure so it's going away from that kind of waterfall type process which is based on the idea of a release that we're going to again define this end state we want to get to and that we're going to wait until we've kind of implemented most of it and then we're ready to release which then becomes a large operation with high risk that we're going to find things that are incorrect and it's going to take too much work to correct so we kind of move to the sprint based process right this is a classic kind of scrum what I call traditional agile which is the idea that you do some amount of planning at the beginning but then you have your sprints your iterations and each of those becomes kind of a time boxed so even if you don't actually deliver to production on each one of those sprints you're still doing a lot of the activities that you would normally have left or you traditionally would have left towards the end of the release say we're going to be testing as we go along so that we have less work to do towards the end and we have higher confidence in what we've produced before we get there and so that's nice and that's an incremental sorry was that a question or sorry so that's a good kind of progress but kind of the next thing that we see people moving on to is more of a canvan or story based process and that's the idea that each story so we assume that we kind of try to break down our work into stories that are going to take some number of days not too long in order to complete each of these we're going to make sure rather than waiting for the end of a sprint and a batch of stories to be finished we're going to do it story by story each story we're going to make sure the code the whole code base at that point isn't a good quality and good state that we could potentially release it and so in this as a kind of a different organizations go to kind of different I guess degrees of this the kind of stronger version of canvan is where after every story you release that into production pretty much right away after having tested it after having done whatever kinds of validation and maybe governance that needs to be applied to it and so that way you don't build up this big batch of integration work of testing work that has to be done at the end you're just doing that continuously which is great and that can be moved on to actually commit base process and this is the kind of continuous delivery and the true sense of continuous delivery is the idea that you make sure as you commit your changes you take your story that you're trying to build and you make changes that work towards that story and you make sure that each change leaves the code base in a good condition to where it is usable if the story that you're working on isn't finished and isn't suitable for people to see you do things like feature hiding or feature toggling so that the code base although code is there it's just that maybe it's not necessarily made available to users and so what this does is this really decouples the release process from the development process so rather than telling the business the people who are funding the development work and have the requirements rather than telling them that we will release based on our development process our operations schedule our technical concerns no longer drive when we release we can release anytime you want and it's actually not disruptive to us as a team we don't need to stop development work we don't need to allocate a bunch of people from operations to spend a couple of weeks doing all this work to make your release it's just fine we'll just push the button because it's the same button in our pipeline that we've pushed for the previous environments we know how it'll work and we're very confident and so just go ahead and do it and call us if something goes wrong with it and we'll come and have a look at it so the kind of final thing I want to talk about is going back to this idea of the people and can we use automation to cut costs to reduce our staff or whatever it may be and I think to my mind the answer to that is rarely so I think this idea that so we often get asked by companies to come in you guys are experts in automation automated testing and automated deployments and infrastructure and so on so can you come in and build us an automated system and then we'll hire some people maybe some graduates recently graduated students who will they can just push buttons right you can build a system so all they have to do is push some buttons and it will run itself so that'll be much cheaper for us or the Hara vendor to come in and install something like say a Paz platform Cloud Foundry from the pivotal folks out there and again it's like have the vendor come in and install the software for us and then they can go away and it will just kind of run itself and the reality of things in the current at least state of the industry is that doesn't work right it actually takes a lot of expertise and understanding of the system in order to maintain it and to keep it upgraded to keep it patched and to keep it to the point so if you like leave it you install software and then leave it for a year and then you want to add some feature to it chances are you can't because it now has things which are no longer supported are known to have security holes and that kind of thing and so you're going to have to have a fairly major effort to upgrade and so you really need to have in your in-house an understanding of your automated systems so that you can carry out maintenance without having to rely on vendors like us or the product vendors or whoever it may be sure bring us in if you have some particular difficult change or interesting thing that you need help with but you need to have that in-house so even when it comes down to again if you have somebody coming and installing a pass or some kind of automated system for you a vendor to do that I tend to recommend people not to have the vendor do the installation but have the vendor tell your people how to do the installation and to support them in doing that and give them the help they need so that they can carry it up themselves and then that will put them in a much better position to be able to support and maintain the system going on so really the purpose of automation is to empower your people when you think about a system the system is not an autonomous system it might be self healing to some degree but really the ingredient you need in order to have a really effective system is the humans so you think about how to use automation in order to make people's jobs easier to make it so that activities that go on whether it's debugging things or doing releases become less about carrying out rote routines so they're spending less time on things like carrying out the same tasks over and over again provisioning servers and they can spend so going back to those kind of teams a lot of times I see organizations where they have a team of people who provision servers like the Linux team, Linux team's job is they take tickets and they create servers for people and configure them for people but what you really want those people to do is to maintain the automation system so that servers can be provisioned automatically and all the right things happen and so rather than having somebody go through a checklist and installing and configuring things just right they look at the scripts to say the packer scripts or what have you that build the server images and so that they know that every server we create is compliant and secure and has the right packages the right versions of things on it without having to actually manually turn a handle on everything themselves so that's pretty much what I've got to say on this for now I'm happy to talk about things in the kind of open spaces this afternoon but I've just got some books to recommend these are things that some of these are things that inspired me when I was writing my book or before working along and things that have come out since that I think are quite good will I think try to publish this probably on slideshare so that you can have a look at it later on if you want to have a look at some of these books thanks a lot and enjoy DevOps days Thank you Keith for giving us the opening you know let's do the wireless quick zip here I guess we'll go to this side really loading up here alright good morning and wasn't that a good talk from Keith that was some good stuff so welcome to DevOps days Singapore my name is Steve Morowski and as mentioned I'm a developer advocate at Microsoft and I followed just a little bit about me I followed a traditional path into IT I started as I owned a garden center and flower shop landscape business worked as a police clerk then I went into a bunch of different IT related roles most recently before coming to Microsoft I worked at Chef as a principal engineer on the community engineering team working on Chef and Test Kitchen a handful of other stuff today we're going to be talking a little bit about story mapping and as Sergio said he asked why would a DevOps audience care about this topic what's important here and the idea for this talk came to me this summer when I was at a camp we had a small group of folks in working on desired state configuration and DevOps initiatives in their organizations it's kind of like a working group where people were exchanging ideas and talking about things that they'd been doing and one of the challenges that I noted was that hey we've been working on our DevOps initiative or our config management initiative for like a year and I look back and we did a lot of work and I really can't verbalize what we've really accomplished and one of the techniques I learned while working at Chef actually was the concept of story mapping and story mapping is going to help us in a couple of different ways and to kind of illustrate what problem we're solving I've got a little scenario here we've got some executives and they come down to IT management and they're like we want the DevOps I read in my in-flight magazine all about the DevOps and your job is to now go make it happen so that we can experience these awesome values we've saw the state of DevOps report and it says hey guess what if you do the DevOps your stock values go up so I want that so IT management hustles off and they google DevOps vendors they're looking eventually they get a little bored and they turn to you and your team alright you go implement the DevOps I want that now either find it in a box and buy it or do what you need so you run off and you read the phoenix project as everybody does that is the kind of canonical entry into most people's exposure to DevOps concepts I think you start working on because you're following the phoenix project you realize oh monitoring is important and CICD and config management are important concepts so you start off and a number of different efforts to do all this stuff now we come back a year later the executives come down IT management what did you actually deliver what value have you achieved for this year long effort where is our DevOps why is my stock value not up yet because you guys are performing awesome so one of the things that story mapping helps us do is to focus on the outcomes now as a technical person I really enjoy focusing on the technical aspects when I worked at chef I worked on the chef code base test kitchen, bunch of other tools I like tools I like tools because that means I don't have to go have hard conversations with people about how to process and this that and the other thing well unfortunately in order to have effective outcomes we need to interact with other folks and story mapping gives us a tool to help have some of those conversations and helps by focusing on the outcomes of things that we want to achieve why do we want the DevOps why do we want config management why do we want automation why do we want monitoring it helps us share the vision of the tools that we have for our endeavor and in a way that's easy to represent and very low overhead and it helps a lot of different audiences it helps us it helps us get concrete about the efforts that we're going to undertake it helps us share that message with our team so our team is all aligned on what are the steps that we're going to take to achieve the goal that we want what are we going to tell our management about what we're doing why are we doing step A, B and C and not doing step D yet well hey in order to get to the point where we can deliver D we need to do these first few things and partner teams we don't work in isolation we want to communicate why we're investing in a particular effort or why we are focusing on a particular technology or why we're focusing on a particular skill set and be able to share that with teams in a place that they can go look and see hey oh yeah this is why we're doing that so talked about why we want story mapping but at the end of the day what is a story map well before we get in there user stories you probably have heard about them we heard about them briefly in Keith's talk user stories are a huge topic in and of themselves and we could go on for a couple of different sections we could spend all day talking about user stories hey if you want to propose an open space and talk about user stories later for the purposes of today's for this this half hour user stories are going to be tasks that people do no more no less that's it we're going to be very very high level and so when I talk about user stories in the context of story mapping it's just things people do alright let's get back to the story mapping step one we need to identify the outcome of what we want to accomplish before we can build a map of stories and things what do we really want to do so I have a little contrived scenario here to kind of illustrate what a story map what building a story map is going to look like and we're going to start out very very basic we've got some automation in our environment we're not doing a lot of we're not doing config management we're not doing a whole lot of automation yet but we've started writing some scripts we've started using them to automate some various things and we're seeing a number of different teams that have been re-implementing the same behavior in different ways and this has led to a little bit of conflict because people have run old versions of scripts, different versions of scripts and things have went a little haywire so we want to reduce the number of outages that we see by 50% that's the goal that's been kind of handed down it's going to increase stability in our environments and we need to go about figuring out how we're going to get there what are we going to do to deliver this now this is just one little scenario you can use this technique for a lot of different scenarios this allowed me to do a very small subset to talk about here you can have very broad story maps you can have very narrow story maps this one's going to be a little more on the narrow side so the first step that we do as we get into the idea of story mapping is to identify the tasks the things that people are going to do the stories task is a super loaded word again we're going to go really really high level here we want to have kind of a functional thing so this morning as I was getting ready I and you're welcome for this I took a shower I got dressed and got ready to come here now taking a shower is a task I could have broke it down to I turned on the water to let it heat up or I got the soap and everything stayed I got my towel out, got my shaving kit all set up no let's just kind of we're going to wrap this up into one little concept of taking a shower it's a high enough I'm not going to go through each step of did I lather up my hair rinse it repeat I don't take that much pride in my hair so you can see the results there there is less and less of this every year so when we talk about tasks for building our story map we want to think breath we want to think wide there will be time later to go deep so don't worry we're not going to lose detail we will get there and the idea of this part of the exercise is we've got a number of folks in the room any of the stakeholders who are going to be working on this it's going to vary based on what you're actually trying to accomplish but you get a really handy tool and I like to use I like to use sticky notes I carry many of them with me all over the place all the time and if you're working in person in a room sticky notes are great you can stick them up on the wall you can stick them out on the table and you write down one task per card you can use the next card sticky notes one task per and you just start putting them all down and everybody starts to think of what are all the things that we need to do all the tasks all the stories that we need to do to accomplish our goal we don't have to worry about ordering we don't have to worry about dependencies or things like that we just want to brain dump all of the possible ideas we'll get to ordering them and we'll get to detail and all that kind of stuff and we want to kind of time box this we don't want to spend we could spend days and days planning out the most detailed thing that we want to do we just want to get a very good grasp of the tasks that we need to do we don't need it doesn't need to be perfect it's okay not to have steps we can add things later but it's okay it it's mainly the things that we want to think people do so some examples that we might see are things like hey we want a CICD pipeline because guess what every problem is solved by a CICD pipeline well well I like to think so but maybe we need a realistic test environment if we're writing a lot of PowerShell we want to lint our scripts with PS Script Analyzer we want to write some unit tests get everything into Git we want to have an internal module repository and start building modules rather than scripts and we want to have admin workstations that are equipped with the latest and greatest tooling in each of our environments so we have known places to go for these things right and so if I'm building on a story map it might start to look something like this my handwriting is pretty horrible so I kept the number of words per card down to a minimum but we build out just a bunch of cards and anybody can contribute I did this by myself but you can do this in groups it actually works much better in groups because one cannot dev up alone now after you've aggregated all your tasks we've gotten to the point where now ordering is going to matter we want to start lining these things up in time order in which they need to happen we also want to group things together that happen about the same period of time and so what that's going to look like is now I start shuffling things around and we've got we want to get things in source control first we need to do that before we really start our CICD pipeline also early on that could happen is and this isn't the final order of things folks so you don't need to worry about is this the perfect order either right these things aren't sticky notes we can move them around it's not that hard we'll see how sticky the wall is here but I put a sticky note here and a sticky note here if I don't like the order it's not that big of an investment to make a change right now now we're early enough on that it's very very easy to move things around and to play with the order of things so we don't have to make it perfect the first time very low cost to change to move a sticky note it's a lot harder to move things around once we've actually got our plan moving in in stone so I've also lined up hey guess what linting scripts and writing unit tests that's kind of similar actions or similar capabilities we'll put that in after our CDICD pipeline and we really don't need a module repository until we actually have modules to share so we can put that towards the end so it's all making sense right we get a list of tasks we put them all in a line in time order now we move on to a portion of the activity call this is probably one of them a little more opaque terms we're going to build a backbone and building the backbone is really about identifying the activities that these groups of tasks kind of talk about and you can think of them as themes to me it's really just putting some nice labels on groups of tasks there's definitely a lot deeper you can go into identifying what these things are so I went through and I started categorizing so I started building out and I've got my timeline very often you'll see in representations of story maps like blue painter's tape laid down across to separate I just used the magic of powerpoint to run a little line across but I put some little themes down so for source control that's when we get all of our automation in one place let's start with that right let's get everything together then we want to have a place to play with those scripts so that's where we categorize the new test environment and making sure we have admin workstations in each of those environments so we have a place from which to practice one place for all of our process that's our CI CD pipeline as we add steps like unit testing and linting and integration tests we want to have one place to enforce those across all of our infrastructure then linting and unit tests that's kind of a quality concept so call that that and last but not least we want to be a reusable packages and that's all the stuff about modules and module repository so I'm just applying themes to the various concepts that are in this story map once we have our backbone so we've got our end-to-end flow we've got a bunch of tasks we've placed them in kind of a time-ordered structure we've given them some labels for some of the themes of things that we're going to be working on across that now it's time to create a slice a slice is merely a selection of those tasks that we're going to work on first or first and second, third and how we divvy up this slice really depends on what what we want to do to show the overall goal of what we're trying to accomplish rather than go and do all the stuff on the left which is the stuff that we need to do earliest right if we go and do all that stuff we haven't shown the value of what we're trying to accomplish if I go and implement source control and build out a new test environment and that eats up the time what do I have to show for that month right if we go and we talk about I want to start taking active steps to achieving the goal of reducing failures due to automation what are the first few things that I need to do and so for this example I have identified we want to get source control in place we want to make sure that we have an admin workstation that our automation is going to run from in each of our environments so that we're only going to take the stuff from source control and get it to those admin workstations and we're going to use a CICD pipeline to go from source control to those admin workstations that is going to be our very first step into implementing CICD for our to bring in control to our scripts and our automation in our environment here and I can see I don't have to take a task from every single theme I only have to take a task for things that matter for that particular segment of functionality that I want to bring this is where we start bringing things down to working in small batches right we want to be able to deliver some value for the efforts that we're putting in and get feedback that works well what doesn't work well because guess what you're going to then have a little bit of time before we move on to the second slice maybe when we hit to that second slice we've determined hey there's a couple more things that we need to do before we can actually retrieve some value from this maybe the admin workstation thing isn't working out maybe we need to have some kind of orchestration launched from that we're not actually moding into some admin workstation maybe we're connecting to a web portal and launching some automation from an interface there right you're going to have a chance to receive some feedback based on the work that you've done but since you've done work across the whole flow not just the stuff that's early in the pipeline you're going to get valuable information next and that's the important thing while I may have started working on defining slice number 2 and slice number 3 those things are not locked in stone those things are malleable they're changeable again from the purpose of our planning perspective they're on sticky notes we can move these things around we can go through this process again when we've achieved our first slice or maybe, hey everything looks good let's move right on to our second slice and maybe we'll revisit then we have a chance for a retrospective or postmortem to take a look at hey we've achieved this first bit of functionality now what do we do next and to me this is one of the most powerful parts of story mapping because it forces you to think about what outcomes am I delivering not just what tasks am I accomplishing when we start breaking things up into user stories and we're moving stuff across our Kanban board we're feeling great because we're doing stuff right what the story map then brings us back to is how do those various tasks come together and then how do they help achieve the goal or the outcome that we're trying to accomplish so to me I find story mapping to be a very handy tool and it looks like I'm running a little fast so we might actually get a little bit extra break for further reading I strongly recommend user story mapping by Jeff Patton he will do the topic much more justice than I have in this short little introduction but if you'd like to talk more about story mapping today we've got open spaces let's do it I'm totally down for user story mapping open space and if you want to find some more from me my blog is stevenmarowski.com and my twitter is down at the bottom of every single slide there and you can find me out on github at smarowski with that thank you very much and let's have a great rest of the devops days we'll take a short pause for about maybe 10-15 minutes I'll come back and we'll meet at Chesna presenting the next talk so when do you want to meet you? I'll watch that can you give me one second I just want to hit the bell yeah I'll be here all day probably most tomorrow probably most tomorrow and then we'll leave for the power show conference yeah yeah mary yeah work with him for about 3 years yeah yeah yeah every day I'm from the US for life that also happens in private life yes yes yeah yeah yeah yeah yeah yeah yeah yeah yeah ah sorry sorry ok ok yeah If you have any questions or comments, I'd love to hear it again. But that's the one that's going to be in the future for the next few months. So we're going to deal with Ray-Ran's and we're going to have a meeting process. And then we're going to move back into the issue. It's okay. That would be great. Yeah. Can you tell us a little bit about this kind of federal law or something? It's just people who are running it for a year or a month. I don't know if there's a court action. No. Okay. I'm sure you lose a little exercise before you lose calories, but I'm carrying that around. It's good to see you! Hey! I know, you're a guy who wants to talk to you. He's on API. Good to see you. Shot. I'll be hoping to get out of the car. It's for like 10 minutes. It's good to see you. I'm always supposed to talk to you. So, I'll just do it. Good to go. Hey, while we're waiting to start up, has anyone heard of All Night DevOps? Nobody? Couple? So, while you're sitting here waiting, you should register for that. There are 100 speakers. They started yesterday. It's a 24-hour event now. There's like 28,000 people registered for it. There's a couple of people, including myself, that are going to speak at it later today from here. So, if you want access to a lot of cutting-edge material, you should register for that. They're recording everything and they'll be rebroadcasting. You'll be able to download the decks. You'll be able to look at the talks. So, awesome opportunity to learn more about DevOps that way. So, I think it's alldaydevops.org. Still I'm missing some people, but I think we will go with our existing schedule. Again, how many of you have mobile phones? Go into a browser and type pigeonhole.ed. slash dodsg. And you will see a form. Just join the conversations. You can, as you are watching and seeing these great speakers in the talks, you can make notes. You can ask questions. You can vote. I'll vote your questions that you think share the same experience. So, we want to start these conversations. And, as you know, we have about 200 people that attend this conference today. So, it will be very hard for us to do the open spaces to come up with a topic. So, let's build that knowledge base as we go through the day and see all the talks. So, without further ado, I would like to introduce Peter Czesna, which has a very interesting title, Full Spectrum Engineering. When we first saw that, it was actually very funny, because it's cheesy. You wouldn't expect this for a DevOps, like what it means, Full Spectrum. We already have multi-star, full-star, all the various front-end, full-star, back-end, full-star. So, definitely looking forward to hear what Peter has to talk and how he can explain it quite good and of interest to DevOps days. Thanks. Thank you. Thank you. I love doing these. I was talking to one of the other speakers. I think this is like my 10th DevOps days this year. Love the format. Love the thought leaders coming up and sharing their ideas. Some great talks this morning as well. So, a little bit about myself. Long-time engineer, 25 years, a software developer. I'm in my 12th year as a security practitioner as well. So, one of the great things about what I do now is I can speak at software conferences, because I know a lot about writing software, and now I can speak a lot at security conferences explaining to them about developers. I've done a lot of stuff not only in my career, but at Veracode specifically. So, moved from Waterfall through to Agilin and DevOps. Did monolith to microservice transformations as well. I love whiskeys, so if there's some great Singapore whiskeys, I'd love to hear about it and tell me where to drink them, because I only got one night in Singapore, and that's tonight. So, there's no tomorrow for me. At least not here, not this time. This is how long I've been doing web application development. Does anyone remember the days when you had to pay for your browsers? I can see a lot of young faces out there like, what's that? Shrinkwrapped on floppy disk. First day of my job, I show up and there's Microsoft Internet Explorer and Netscape Navigator in a box. Crazy. Yeah, so I've been doing web application development since the mid-90s. Ship my first one in 96. This was an intranet application, but it was web based. It was running on IIS, etc. So, know a lot about that space as well. So, this is the fun thing. I created this slide while people were talking because it's interesting. I was thinking about the story of when this talk came to me, and if you see way over there in San Francisco, the little home icon. At RSA this year, in February, I said, I got this really cool idea. I'm looking at the landscape, looking at how development gets done, and I see this shift happening, and I want to tell people about it. And it's funny you said the cheesy name. I was trying to come up with a name for it. I couldn't think of a good name for it. Now, fortunately, I've had the excellent fortune of speaking about this in many different locations. I premiered this at DevOps Day's Stockholm. I've given it in both an ignite format and in a long format. So, I've kind of worked both sides of that. When Stockholm came back to me, I initially submitted it as an ignite. They're like, hey, this is really interesting. Can you do it as a long format? And so I've been able to travel the world talking not only about this, but other topics as well, and you're going to hear it tonight from me later today. But, like I said, an interesting story. So I want to be very, very clear, because you see a lot of definitions around DevOps, right? And what a DevOps team is. So usually when I talk to companies that are just starting on the journey, this is what they mean. It's the people that do the tooling. It's the people that write the automation and the guys that do the operations. It's the ops side and maybe some fancy automation side, right? What I mean when I say DevOps team, and that really needs to be clear because it's pertinent to this talk, they are responsible for everything, soup to nuts, cradle to grave, for everything that that team does. And the reason that we do this is because we want our developers to learn to write better software by operating the software. You will think differently if you get woken up a couple of times at 2 in the morning because no one can figure out how this stuff works and it's broken. So your telemetry gets better, your monitoring gets better, your quality gets better, you just get in that habit of thinking about that all the time. So changing job market too. So my premise was that the way the architectures are being built today, the change to a small cross-functional team from what used to be a 25-50 person team, the DevOps methodology, and of course this need for speed. We need to go fast. And one of the biggest inhibitors to speed are people. You put people in a process, you put a handoff in a process, it's going to slow it down. So if you want to get as fast as you can, you need to think about how do I make it have less handoffs. So the effect of this is changing the way I think about hiring engineers. So I moved from an individual contributor to an engineering manager at Veracode. And actually, are there any customers of Veracode in the room? Ooh, a room full of prospects, I love it. Okay, awesome. And the fact that developers need to know a hell of a lot more today than they did even a year ago. So as you think about and walk through this with me, think about all the things that you now need to learn that were other people's jobs two years ago. And what does that mean for their jobs moving forward? So who do we have in the audience as a developer? Most people? All right, hold on. So this happens to me all the time. I ask a question and I get a couple of arms and then maybe something like this. Can everyone just raise one arm, like really high? I just want to make sure your arms are working. Come on now, I'm not moving on until we go. Please, help me out here. All right, who in the room is a developer? All right, who in the room is quality? A couple? Security? Anybody? Oh, awesome, my brothers. Operations? Excellent, excellent. All right, so who thinks it's their job to do security? See, all your hands should have gone up. It's everybody's job. All right, so these cross-functional teams need to have accountability for what they do. Otherwise that DevOps model that I put up before doesn't work. So let's talk about architectures. You know, whether it's a web browser, an application, a thick client type of thing, you have these end-tier architectures. That's what I grew up doing. That's what I cut my teeth on. Now the problem with this is we started to create these specializations. Right, who was a front-end developer and middle aware and I only do business logic. It's like, what the hell does that mean? I only do database. It's like, well, okay, great, good for you. That was okay to have when I've got a team of 50 people and I can have a couple of DBAs and a bunch of people that only do middleware. But that doesn't work as we're trying to build this software really quickly, right? So we moved from this monolithic end-tier architecture into this microservice where all these things are communicating to each other and they all have embedded UI inside of them. So I've got a SPA sitting inside of my microservice that when it's time to go and do a configuration, it comes, right? Who do you think writes that? Is it a UI guy? I don't think so. I don't have time and I don't have the space on my teams to have people that just do UI. If we look at web technology, so back when I started, you click on a link and it goes to the server and it confetshes a whole new HTML page back and repaints the whole screen. It was really ugly. You know, we moved to that Ajax model and now we're into rest, right? So if you look at Amazon.com, it's one web application, but how many microservices are sitting behind that? How many microservices are actually serving UI into that portal? Right? We don't have these portlets anymore. We've got a bunch of microservices and they all say, here's my UI and if it doesn't work, that's okay. Everything else keeps ticking along. If one service is down, the rest of the experience still works. So this is what we used to hire or this is what we hire today. This is kind of the state of the art. We don't have a full stack developer, right? So I talk about technology. I talk about, hey, do you know how to do Angular? Are you really proficient in JavaScript? Can you do Node? Can you do all of those things? Can you go soup to nuts from front end to back end? That's the way we used to think about developers, but actually, we're all pretty smart and one of the things that happened to me in my career when I came to Veracode, I had done some Java programming but I'd never shipped anything doing Java. It's hard to get a job in those days to say, yeah, I'm really proficient in .NET and C++. I can do this Java thing. I'm a smart guy. I can do this. And it was hard to get that first job. In fact, I went to interview at a company. I was an expert in storage and I had done it in ASP, that first job. They said, well, we can't hire you because you don't know Java. I'm like, how long is it going to take you to build an expert in the storage space? So you're going to go hire someone in those Java and then try to learn the space? That's stupid. It's backwards. So let's talk about methodologies and team sizes now. So here is our waterfall, right? Long time, maybe a year, maybe once a year, maybe four times a year, we're releasing our software. Now, one of the things that I often talk to security professionals and people that are non-technical about, about the difference between waterfall and agile and DevOps. It's like, if you look at what happens with agile, it's not different. I still have requirements. I still have to code it. I still have to test it. I still have to release it. All of those waterfall-type steps still occur. They just all happen with me, right? Smaller chunks, less risk. It's just trimming it down. I'm not doing a one-year horizon and trying to schedule that. I'm actually saying, let's just do a couple of small things. Let's commit to it as a team and let's go, right? So that's agile and DevOps, it's exactly the same thing. I'm taking an even smaller piece. Maybe I'm doing single-piece flow like we talked about before. One developer's work goes right out to production. And when you're doing it at scale, it looks like this, right? It could be happening anywhere and everywhere. If we look at the team sizes now, again, who has done waterfall? I can say it proudly. I'm good with that. 50-person team. Full of specialists. Full of handoffs. Again, one to four releases a year. Now you get to agile, and now we're talking a couple times a month and we've trimmed the team down, six to 12 people. But we're still building applications. So what does that mean for all these specialists? What are we going to do with knowledge that used to be in 50 people's heads I need in six to 12 people's heads? We need to become more cross-functional. This is what happened with agile, right? We started to think about, hey, this test-driven development looks pretty cool because I can't claim I'm done until it's tested. So we started to get serious about writing unit tests. There was people talking about unit tests forever. But when you get to agile and you're like, you know what? You don't get any credit for your velocity unless it's tested and functional and ready to go. I need a shippable unit by the end of that sprint. It's like, okay, I'm going to change the way I think. Of course, the other thing that people don't understand typically is that DevOps and agile, you usually run DevOps on top of Agile. Whether it's Kenban or Agile Scrum or Scrumban or something like that, it's some flavor of that. I've seen people saying, well, you know, I've got these waterfall themes and one of them is going to Agile and one of them is going to DevOps. Scratch my head. I'm like, I don't think that word means what you think it means. They don't understand that all of the lessons that you need to learn, all of the pain that you have to go through to do all the automated testing that you never did when you were doing waterfall, you do in your agile transformation. You have to build up that muscle to say, automate, automate, automate. Now, you don't go as far as DevOps, but that's where you make those learnings and then after that it's just like more, more, more. So the way we ship software today, and we're kind of on the cusp, this is really changing the way things are done, but typically in a physical environment for a company, if I've got a physical infrastructure, developers are going to build the software that I'm going to have quality guy come in and tell me whether it's good or not. We're going to have the security guys validate it. Now, of course, they do that at the end and I can talk to you all day about how apps that can be done a lot better. In fact, that's my ignite today. Management does approvals. Boy, this gets in the way, right? Again, more people, emails with rubber stamps that come back say, hey, I like the change, approved. Like, whoa, how'd that happen? It's like, do they even look at the change request? Again, we talked about this this morning with Iron Age, talking about racking and stacking, cabling up hardware, and then I've got to have someone come in and install operating systems on it and build it up into something that I can actually use, add all the supporting software, so am I running on Tomcat, am I running on JBoss? And then the ops guys, they're the ones that keep it all humming, right? So this was the thing that I saw in the industry, this pivot, kind of taking that tower of full stack and laying it on its side, thinking about what we actually do now. An individual builds a software, they test it, they're responsible for testing. Now sometimes they're going to have help from people that are more astute in quality, but they're going to do it themselves and then they're going to validate their security, so this whole shift left movement in application security, I'm responsible for that as well. I've got a pipeline now that releases the software, it's not a scripting anymore, it's not a person, it's software. The software terraforms the environment, I don't even need hardware anymore. I can build it out of nothing, I can build it in the cloud. And then we configure and install it, whether that be containers or something else, we set something up to run the software for us. I do not need to have a person go into a closet and do some magic. And then I have the pipeline that actually delivers it into production. So I don't see a lot of different things going on here from an individual basis. Now the team monitors it and manages it. So this is the DevOps thing, right? I drive behavior by making you responsible for running what you write. So that's where I came to this conclusion we're really talking not about multi-technology but multi-discipline. It's not, you know what, that languages are going to change. We're going to go to Golang, we're going to do this, we're going to try that, we had Node, it'll be something different tomorrow. There'll always be some new technology that you need to go and learn and some new framework, it struts yesterday and it'll be something else tomorrow. You're going to have to go and learn that. But this part is not really going to change. All of these aspects of software creation are the same and we're being asked to do them now. So it starts with being a great engineer, right? I have to be a good developer. I have to understand open source, I have to understand and stay on top of standards. Next step, again, this was part of our agile transformation, I have to understand quality, I have to build stuff that functions. Now whether I do TDD or I do unit tests or I do something else, I have to understand the quality of the software that I write. This is the really crazy one. This used to be systems engineering and IT. If I'm doing containers now, I'm picking the operating system, I'm picking what's running on top of the operating system. That is actually, my deployment is not an application anymore. It's the whole thing, it's the machine and all into some virtual environment. So think of all the things that you never had to think about. What version of the operating system are you going to run on? Which versions of open source do you have installed in there? Do you have a bill of materials for that? Another talk I do on open source, just to give you a little scare factor, who knows whether or not they ship with struts? You know it or not, right? Everyone knows whether or not they ship with struts. What if I ask the room, who ships with Apache Commons Collection 3.2.1? Does anyone in here know? Now I know, but if you don't know, this is what we as developers really do poorly. We integrate it and then we abandon it. So this has got to change. So how are you going to operate the thing? You need to build in telemetry, you need to build in monitoring. This was the job of the operations people. Now we're going to have operations people on our team or people that are more inclined to do that. But if I want someone to write a piece of code and get it quickly to production, I don't need to have 28 handoffs. They don't have to do like the brute force of setting it up but adding to it as we go. Security, huge topic for me. Love it to death. It needs to be part of your definition of done. If it's not, you should ask why. We need to be educated on it because as developers, I don't think anyone in the room graduated with a secure coding degree. I didn't. And this is, again, this is one of the things that we keep learning in the industry. When we did the agile transformation, it was like, geez, we can fix things quicker and faster if we shift all the quality left into the team. It's the same thing with security. Security is quality. So by testing it earlier, we're actually finding a fixing it and able to move faster. All right, so what do we do about our specialists? We've got all these people, we've got DBAs. I've got people that are experts in operations. I've got people that are experts in architecture. So what I want you to do is think about it as guilds, practices inside of your company that are led by those people. Let them help point the direction, right? We want them to communicate best practices and teach. Hold birds of a feather meeting so you guys can learn from one another. Like I said, you can't be an expert in everything, but as software engineers, don't you know a little about everything? So we're asking to ramp that up just a little bit more because what I'm looking to hire is someone that can say, hey, I know how to change the pipeline. I know how to add stuff into the container. I know how to do all of those things. Maybe I can't do it with a white sheet of paper, but I can certainly work with what we have. All right, so in conclusion, the way software is being written has fundamentally changed. We don't build software the same. We don't test it the same. We don't deploy it the same anymore. The physical infrastructure is going away. If it hasn't for you, it will. We're moving to the cloud, and that means somebody else's computers, but we're actually deploying a different kind of artifact. We're not deploying a war file anymore. We're deploying a container image that has our application inside of it. With team sizes shrinking, we have to become cross-functional in nature, right? So I can't have a quality person. All I do is write tests. Well, that's good, but I really don't need you here. I need you to do more. I need you to take on other job responsibilities. The velocity expectations are not going to go down. Who thinks that the company tomorrow is going to be like, oh, you know that's fast enough. Don't worry about it. We're good. We're making enough money right now. So this is another thing that I've been talking about a lot lately, accountability. I've heard people use it the wrong way, which is I'm going to hold you accountable. What we want is for us as developers in quality and et cetera, to take accountability. I want to be accountable for the product of my work. I take a lot of pride in what I write. I take pride that when I check it in, my buddy in the next cube is going to be like, oh, don't you see that? That's awesome. That's what we like to hear. That's what floats our boat. So this is the next level of that. So to do that, we need to think about those multiple disciplines and not just blindly go and say, hey, I'm going to just pick this container from Docker and say, that's my image. What's in that image? What are you actually installing? What are you running on because the risks that arise because of that? Everyone heard of the Equifax breach? Yeah? Open source. It's not that open source is bad. We don't actually keep track of it. We're like, eh, struts, I'm done. And then structural nobody comes out and no one knows where it is. No one knows how to patch it. We don't have an MTTR that's any good around this. So we have to build out those practices. We have to understand those various aspects and how they affect us as engineers. Thank you. Thanks, Pete. Great conversation. Great presentation. Thank you. I've seen more and more questions being asked in our pigeonhole. So it looks like people get a hand of it. Again, I think the full spectrum is going to, we'll see more. This is how the conversation looks online. So what we need to do is go to this link, which is pigeonhole.at, e-i-g-e-o-n-hole.at. It will ask you for an event passcode. So the passcode is DODSG. So you can enter it here, like that, or you can use it in the slash here. So it becomes like a URL. So if I click DODSG, I enter this question and answers. I can see all the questions that have been asked before me. I can upvote. Any questions that I feel it's of interest, or I can ask you questions depending on the new talk. So without further ado, I would like to introduce our next speaker, which is Dave Kerr, which is going to talk about monolid to microservices. He's going to give us practical tips for CI and continuous delivery. And when we asked him how this topic fit into the DevOps base, he told us that he's going to share a lot of practical lessons from Southeast Asian Bank. So please. Thank you. Hey there, everyone. Can you hear me okay? Does this work? Yes. Awesome. Okay. I'm very nervous, and when I'm nervous, I walk around, so sorry in advance to the camera guys. I tried to calm myself down with a few coffees, I felt like six. So it might be a little bit hectic. I also do this a lot when I'm nervous. Okay. So monolid to microservices, for better or for worse, we are moving more and more into a world of microservices. It's very much a buzzword, and I was very, very excited about it when I started doing microservices. And now the more I'm doing, the more it's like, oh, it's so much pain. But here we go. I'm going to try and share today some practical tips and tricks for working with microservices, particularly in a more enterprise environment. This is going to be fairly brief, lots and lots of tips over a short space of time. So do come and grab me, find me, ask any other questions. So quick introduction first of all. Who am I? I am a technology consultant. I work for McKinsey & Company, working with clients about technology. So something like this. Beautiful money. I don't really actually make that much money. The reality is probably a little bit closer to this. So I spend a lot of time in airports doing deployments and doing coding in really, really ridiculous environments. So that was me just the other day, probably trying to finish off this presentation. So this is more the reality, hunched up in an airplane, desperately trying to code with no Wi-Fi. It's a great way of working actually, not having any internet. Back in the day when I first started on C, I had my big textbooks and it's kind of a bit like that again. So I'm going to talk about microservices. I'm going to talk about implementing microservices in a banking environment, which is obviously a challenging environment because it's heavily regulated. There's a lot of enterprise stuff there. No matter what cool, sexy stuff you're doing on the front end, you probably still have something like an AS400 mainframe sitting in the basement in a couple of data centers. So let me introduce first of all the project. This is the kind of context for what I'm doing at the moment or one of the things I'm doing at the moment. And this kind of gives a bit of a shape of what exactly was going on. I'm just going to also open up my phone with my talk timer so that I don't spend too long or too little time talking about the same stuff. So just one second. Sorry guys, just chill out for a while. Talk amongst yourselves. Go grab a coffee. All right, cool. Here we go. So yeah, this is the project. Or as I like to think of it, now a comeback web sphere always for a given. You weren't that bad really. At least it was only one thing to manage instead of like 25. So what were we doing? In 2016, our client launched a very, very, very sophisticated digital banking application. It's a market leading application, incredibly sophisticated features, huge accolades from the market, very, very sophisticated, very powerful. And it was wonderful. And McKinsey were helping them in that stage of the project as well. The client have a kind of digital garage setup. So the kind of startup style environment, lots of open spaces, lots of great technologists, lots of enthusiasm about technology, lots of meetups and everything like this. A very experienced product team and UI development team. And for a while, everything seemed wonderful. New features were coming out. People were very excited about product. Now, one of the things that we often see is that when organizations start to go on this journey to becoming more like digital natives, you get these product teams that start to move very, very quickly and they rapidly surface systemic or endemic issues in the organization. So for example, if you've got a product team who are trying to build features that are going to be released within a matter of weeks, but you have maybe enterprise, IT, governance issues, which mean you can't release for say three to six months. These issues get surfaced quickly and it creates a lot of tension in the organization. You get different business units kind of fighting each other. You have a product team who are desperate to release. You have an IT team who are like, okay, sure, but this is a bank. This is people's money. We need security. We need governance. We need regulation. And we can make these things work together, but it's hard. So 2017, we were looking at some challenges that we were facing with this product. And a lot of the challenges we realized came down to the way that it was architected on the back end. This is a picture of my colleague, Sanjeev, kind of freaking out when he asked for a print-out of the data model for the single Oracle database that it all runs on. And there he is. I think he had a stroke. He's, you know, not quite recovered ever since. So a big, big monolithic system and the sorts of challenges we were facing are the typical challenges you will get from monolithic systems. It doesn't scale very well. Changes are very, very complicated. Not only that, this monolithic system, the back end, it was actually built by a third party. It was proprietary. It was black-blocked box. It was hidden away somewhere else. The source code wasn't available. So changes to the front end could happen very, very quickly. But changes to this kind of fat middle tier were incredibly difficult. So you have this middle tier which you can't really move, but then you've also got all the enterprise IT stuff at the back. And actually the enterprise IT guys are quite fast at changing things in this organization. It was the middle tier which was really slow. So we spent some time sort of the beginning of this year talking with this client about what we could potentially do when they put out a tender. And what we ended up doing or what we're on the journey of doing is realizing that this is actually a huge, huge thing we need to solve. And it's not just a case of technology. I'm going to focus on technology today, but as lots of other people have said, this is really about culture and it's about people. So this is a big project. We're talking over 100 people in one area doing lots of stuff. A complete rebuild of the technology stack, front end, middle tier and some back end components as well. A huge amount of hiring and bringing in lots of new people. I think there's a bit of a perception that McKinsey is about operational efficiency which often people think means reducing people. Actually not at all. At the moment, what we're doing here is we're desperately trying to find people, the best technologists we can. And lots and lots of training, coaching, capability building, pair programming, everything like this. Not just in the development side but also in the design side, in the governance side, in the PMO side, kind of everything. So it's a big, big project which is challenging, but it's also great because you get the top level sponsorship. You get a change. You can bring that case to the leaders. So here's the kind of boring architecture diagram. Oh wow, it's so boring. I've actually broken the screen. Wow. Sorry about that, everyone. Maybe I kicked something. Okay. You can take a coffee... Oh, even the voice is gone. You can take coffee break number two now. Normally I get these problems when I'm doing the live coding. I don't get them so much when I show the PowerPoint slides. But there you go. At least I can add that to my list of experiences. If we've got a whiteboard I could try and, like, live slide really quickly. Just draw lots of little boxes. Microservices. Big box. Interpretive dance. Okay, yeah. I could huddle in the corner and shake a little bit. That's what it feels like most days. The screen is coming down. I don't know if everyone's super excited to see that page again. Shall I try and plug it in again? No, let's try this, okay? Do we have it back? Hooray! Okay. So, old world on the left. Monolithic system built on WebSphere. Huge Oracle database talking to monolithic and very complicated integrator system that this integrator system is actually talking directly to the core mainframe banking system as well as to the ESB. Beautiful new world on the right. It's not beautiful and cool unless you use hexagons. If you use hexagons then it's microservices. It doesn't matter what's in them or what you have there. But if you use hexagons then yeah, that's it. You're sorted. So lots and lots of small components. Little databases. For certain things which are genuinely isolated data domains we can use specialized databases. It might be a MongoDB database or it could be something like Redis. It doesn't really matter. We can use the right database for the job which is one of the great things about this kind of entire pattern. One thing I actually forgot to put on the diagram which I just quickly whipped up is GraphQL as well. We have another microservice here at the top which is a GraphQL layer and that has been a huge, huge help and in fact it's probably one of the biggest tips. We got a lot of the data modeling wrong. We had a lot of teams building individual services where their domains were overlapping or there were gaps in between the domains and we had lots of people looking at different swagger specs and getting confused about where things were. As soon as we created a single unified GraphQL schema it allowed us to fit it all into place. The UI developers talked to the GraphQL schema. We will mock the data in GraphQL while the back-end developers connect the two dots together and then we have a few cross-functional developers in the intersection. Anyway, this is the kind of diagram or the vision of this and then there's a lot of other stuff going on as well source control, CICD, et cetera, et cetera. Obviously people are nervous about these kind of changes. It's a big challenge. We're talking about going from a quite simple technical domain well complicated components but not too many components to all of this. I mean there is a lot of stuff here and every other thing we introduced introduced anxiety because people are saying well our developers don't necessarily know Node.js let alone Node.js, Mongo, Redis, Kubernetes, everything like this there's a lot of stuff to learn and this makes people worried obviously there's a lot of anxiety around doing things like this. So being able to take these technologies and break them into small components not everyone needs to know about Cassandra only the people who are doing the service where they're working with Cassandra necessarily need to know about that and as long as we define our services well with good boundaries actually we can isolate some of this complexity. So that's where we are building lots and lots and lots of stuff. So here we go Thereby Dragons you can see this on a very old map I think it's in a museum in the Netherlands Thereby Dragons so microservices it all looks good on the PowerPoint but when you get down to the reality it's very very challenging. So here are some of the tips that kind of I've collected over the the year or so of doing this and previous projects as well. So the first tip and I think this is a huge one is keep the pipeline simple it's very easy to sort of think to yourself well I've got all these different environments and I've got all these different tags and I've got all these different semantic releases and I've got all this complicated stuff and build these colossal sprawling complicated pipelines that become monoliths and black holes of complexity in themselves. Now the reality is in pretty much every scenario you have it's always going to be the same thing I verify I build my artifact I verify it somehow and I push my artifact somewhere that's your commodity step it's always going to be the same and you can run that anywhere anywhere you've got CPU memory you can do this you can run this on the cloud you're just generating artifacts then your more complicated stuff will be when I've generated my artifacts I need to put them somewhere okay fine that's where you want to invest your time and your thinking in doing this but this step here keep it simple and don't rely on CICD tools I mean I love Jenkins I love Circle I love all these different tools but what I don't love is when a developer doesn't know how to actually run each individual step themselves so one of the ways that we solve this is we've got all these sexy new tools but we go back to something which has been around for like 37 years now which is GNU make everything you can do every individual step of a pipeline can be run through a make command and that means on anyone's machine anyone can work on any stage of the pipeline tweak it, modify it, adjust it or even run it if there's some kind of problems and understand and reason about what's going on here's an example of this kind of pipeline for a Docker image and I'll go into a bit more detail but when I'm working on any kind of project with Docker I know that I can do make build and I'm going to get an image I know that I can do make test and I'm going to verify it it doesn't matter whether it's Java it doesn't matter whether it's Go it doesn't matter whether it's Node I don't really care if I'm just kind of jumping in those make files are there so what we try and do is take away a lot of this complex code which is kind of hidden away in Jenkins or Circle or wherever it might be all this stuff here which you can't see what's going on until you dive in and actually put it into scripts in the source code and then the whole idea is that these big complicated things the complexity is still there but your pipelines look like this and if I want to tweak a part of the pipeline I can check out the code I can run that section of the pipeline I can see what's going on and I can reason about it and I can adjust it so let's actually jump into some examples here now I've spoken a little bit about microservices but the example I'm going to use first is actually from mobile apps because builds for mobile apps are just absolutely nightmarish I don't know how many people here are used to having to build iOS apps signing and all that kind of stuff it's horrendous like building a Docker file is easy building mobile apps is awful if I take a look at this project I've got here and I will open it up first of all this project is an example of applying this pattern so what I put together was a React native app an Ionic Cordova app a Jammerin app and a pure native app and for every one of these four apps all you need to do is go into the folder make build and you get your IPA and your APK you want to test it, make test you want to push it to body build or test ferry or whatever your system is make publish for every one of these apps it's exactly the same thing it doesn't matter whether it's using Gradle it doesn't matter whether it's using Cordova it doesn't matter whether it's using anything else at least your entry point is there and that's really important because if you actually look at the individual steps within the make files let's take a look at something like the React native app and go down to the make file there is still quite a bit of complexity in there although this is only a few lines of code really some of this stuff is quite complicated so for example doing the bundling and the code signing that's hard anyone who's done it this takes a long time to get right being able to run this on your local machine makes things a lot easier you can test it and debug it and tweak it labelling this is kind of funky you add a label to the icons on your app I've got a little tool for that as well again you can do all of this locally so it means that when you start to want to run this on your CI system it's really easy make label, make build, make deploy you can work through that and you can reason about what's going on so I would really encourage that as a pattern it's something that's worked very well and you can see this project on my github page and see how you can apply it to mobile apps there's now a little bit more on server side code that's what most of us are here for so the next tip is to really craft your docker files to help you I've got enough time for a little story so this is Singapore docker files, I love docker files why docker, why containers there's a nice little anecdote around this back in the day 1600s, 1700s when people were moving freight and cargo around it would be in barrels it would be in boxes, all these different things in every different harbour they had to have different types of cranes that could lift things up and move things around and as more and more international trade was happening it became harder and harder to move things around every environment had its own physical infrastructure for this now if you look at somewhere like Singapore or any one of the big harbours the one thing that stands out universally is that everything is in a container the same size and this is why docker is called docker and this is why we use things like containers, even kubernetes comes from harbormaster or helmsmaster or something like this a container can contain really just about anything it can contain dry goods it can contain refrigerators refrigerated units, it can even contain data centres the lorries that carry a container are all the same, they can all just carry a container the ships that can carry a container all exactly the same, the cranes are all the same and this is exactly the pattern we're applying to our projects but the other thing we can do with containers is we can use containers to hide away pain from our external environments so I'm in a banking environment and we have to use a proxy so I can use my docker container or my docker file to save me some pain here this docker file handles the fact that in some cases I might be using a proxy on a developer's machine they don't pass in any proxy environment variables and it just works without a proxy on the CI server I have the environment variables set and it uses the proxy the developer never needs to care as far as they're interested they just do make build which under the hood runs docker build so we can use this to hide away proxies to hide away things like artifact repositories on my local developers machine I'll use NPMJS and docker hub but when I'm running my kind of gold-plated builds in the banking environment it will use an on-prem NPM instance and it will use an on-prem docker registry but again the docker file is actually hiding all of that away from us so there's lots of simple tricks for docker files that can help you get around some of your environmental problems and if you find that you're having to set environment variables in reverse, have one docker file for here and one docker file for there then that's a point where you need to kind of go back to square one and think no something's wrong because the reality is you should be able to build from a docker file and have a gold-plated artifact generated from anywhere it should never matter that artifact should be able to go into production if you can't do that there's probably a mistake somewhere the other thing that you can do is hide away things like your baseline image so the from here is is masked away because it's based on another baseline image which has some of the more banking specific stuff in it and just like everything else docker files can be managed and maintained using things like make files it doesn't really matter so if I go to something like docker dynamo which is a little project that I run which all it does is it provides dynamo db in a container so it just lets you quickly run up dynamo db for this go back here I don't want to build the docker file but I can't remember what the tag should be or anything like this so I've got my docker file I need to test it I'm super lazy I've kind of forgotten what braid all java and everything else is so I just want to do make test now in this case for this project this is going to run instead of shell scripts and these shell scripts are going to do things like talk to the container try talking to it without a mounted volume and see if it's ephemeral and from the point of view of me the developer or the consumer I have three entry points and my build pipelines are just ridiculously simple if I take a look at the code again and take a look at something like let's see my yaml file what do we have in the yaml file we have a little bit of the environment specific stuff so pulling down a certain version of circle I have a build number which I want to pop in there and then beyond that just make test and make deploy so every one of these individual steps I can work through quite easily the linux foundation give me five dollars every time I say use make that's why I'm really going for it at the moment and if we take a look at the make file itself again it's very very simple all the complexity if you find your spiles sprawling out of control you can put them into scripts there's actually a really nice thing about doing this again sorry guys I'm doing the camera just keep on walking around the nice thing about running your scripts from make files is it solves a little problem which is when I've got a script in a folder do I run it with my root as that folder or do I run it from the root of the project and do I pass in parameters or do I call it as a function and that kind of stuff using these files as your entry points or your index into your logic can also help solve some of those problems instead of having lots and lots of sprawling scripts that you kind of have to remember where they are you have a single entry point okay here's a big tip use the sandbox and also share your toys in the sandbox so I was talking to some colleagues the other day and say we were looking into setting up Splunk and I was saying I've done all of this on our AWS environment now obviously they're a bank they're not using AWS at the moment for the majority of their workloads so someone was saying well we can't use AWS it doesn't really matter just use it anyway even if you're in a banking environment get yourself an AWS account you'll have the same environment on AWS because no one in the bank is going to let me go down to the data center and change the network they might I suppose if I ask them but they're reasonably security conscious but I can change the network on AWS and I can experiment and I can try things like different arrangements of subnets and I can try things like different sizes of machines so I can say for example well what about if we change our Kubernetes master from a 16 gigabyte machine to a 32 gigabyte machine it's going to cost five or six dollars and it's much easier to test that out on the cloud than it is to try and test it out in the data center so you can use the cloud to create your living specification of your software and your infrastructure and then you can pass it over so if you're using a more traditional data center or something like this it doesn't matter by the time I've set my cloud up I know exactly what the infrastructure I need is and an example of that and this is all on my GitHub page is a project called Terraform AWS OpenShift so I set this up to give myself a sandbox again Linux Foundation are going to give me some money if I do this when I want to create my infrastructure just to make infrastructure wildly excited and I'm using Terraform under the hood for that so this is now building up the entire infrastructure and this will take I don't know five minutes to run when I want to install OpenShift I do make OpenShift when I want to add Splunk to the picture I do make Splunk and anyone who wants to go deeper can just go into the code and see it but because I'm using because I'm using this cloud environment even though the clients are not on the cloud and because I'm using infrastructure as code which is something that I'm sure we're all very excited about because we're DevOps folks I know what kind of VPC setup I need I know what kind of firewall rules I need because all of these have been set up I know for example that I'm going to need 80, 80, 443 and 8443 so there's no surprises there so now I'm making the network teams life a lot easier because I'm not saying hey can you just set me up some stuff and I'm going to bug you like 10 times a day from now on asking for ports to be open I can come to them with a specification and say this is how I want it to look and as they start to use more things like VMware and everything like this they can start to use tools like Terraform to do similar things themselves on their own premises I've got a couple of colleagues dotted around who are looking at me thinking you do bother the network team 10 times a day so I do for other things as well so by doing infrastructure as code it creates a living specification and a very dynamic evolving specification even down to the infrastructure level doesn't matter that I'm still running on a traditional data center I can still use that set of tools I can use the sandbox at low cost the other great thing is sharing your toys this is available publicly I've got about 3 people who are now collaborating and they're testing it on their enterprise environment they're using it for the same thing so we're working out things like how does this work when you're using Rails 7 so we're actually getting some help there's no confidential data being shared we're really just looking at networking, infrastructure and software setups but we're getting support from other people on doing that and people are collaborating they're saying why not try this hey maybe if you try this kind of backing for your block store or whatever you might get a better result so it's not like we're exposing anything confidential we're just collaborating on a network with Kubernetes installed but we're getting help for that sharing the toys is a really important thing to do if you can so here we can see that it only took a few minutes for this to run up but I've got the same setup now which was shown in the picture I've got a bastion I've got a master node I've got some worker nodes I've got my Splunk console setup and this is the infrastructure ready to go and then what I can work on is setting up the actual software again, because I can tear all this down in a few minutes it means I can do all sorts of interesting things when we were looking at setting up Splunk we realised we needed to change some of the configuration for Kubernetes on the cluster so I can now share with the team the changes I've made for example doesn't matter too much what the code is here but I can share with the team these are exactly the changes I had to make to the vanilla open shift installation I needed to do this because I needed to change the way the log driver works there's no classification in code now it's not anecdotal, it's not tribal knowledge it's actually there for people to see even try out themselves and they can try that out maybe changing the base image from Amazon Linux to perhaps REL7 so they can see whether it works there so it's a hugely important thing to do and it also introduces that rigor and discipline you might have a traditional data centre but still use these tools anyway yes Terraform don't give me any money when I say this another micro tip just give Terraform a try because it will change the way you think about infrastructure so the way I test the Terraform code is how I test it normally something like a very basic smoke test so what I do is have a pipeline where when I make a change to the code it runs the commands to spin everything up in a brand new VPC and then I run a test which hits something that I expect so for example for a project like this the test would be once it's all set up and running can I SSH into a bastion and come back can I do an HTTP request to the console and get some kind of 200 class response and typically that's about as far as I've gone with a project like this but in theory there's nothing stopping you from doing quite sophisticated tests where you say okay if I set everything up do I still have DNS working does my DNS resolution still work can I still SSH into this box do I still get this version of the software out of it those pipelines will take a bit longer to run but it's very automatable it's quite automatable as well because your pipelines are going to do the same thing as everything else which is they're going to run like make build and then they're going to run make test and stuff like this so again I'd have it all working locally first of all and then just dump that into a build system and at that stage it doesn't matter whether the build system is Jenkins Circle or whatever because they're all going to run roughly the same code actually don't so you can do dry runs and you can do a plan to see exactly what is the infrastructure change that's going to happen so you can just run the Terraform plan command and it will give you a manifest of everything that's going to change and it's very important to do that because one of the things you'll often find is that even though you think you've made a small change it might decide when it looks at the plan that it needs to rebuild the entire VPC or things like this and you might not want that so adding that as a step before anyone does anything in production is pretty important but the other thing you can do as well as you've got all this automation is you can do the equivalent of like blue-green deployments even at the infrastructure level so a blue and green production environment run some things on the blue environment for a while and if it looks like it's healthy retire the green one or just load balance onto the blue one another big tip or a big learning that we had and I'm not going to be able to show the exact code only some sort of sanitized code is make a micro service manifest we just use a basic JSON file we only introduced this about two months ago and it's already been a lifesaver because it's a large project about 70 developers spread across six scrum teams literally getting to the stage where we're forgetting is this service still there? do we still use this service? I mean it's still in the repository but I'm not sure if we're even deploying that anymore which sounds like a terrible thing to say but the reality is when you've got lots of independent teams working you can actually lose track of the services you're running so the simple way that we've started to deal with this problem is we just have a manifest which says we have our channel apps or our client apps and we have our services which run in certain repositories this manifest is now the entry point to a whole load of different logic so for example one of the things I can do now through a shell script is I can say give me an inventory of my dev environment and it will run over to the dev environment and it will check the versions of everything and it will check what's running and check what's healthy so dev looks like this but the SIT environment looks like this and the tags are out I can create a more explicit inventory when I say I want to run this tag, this tag, this tag, this tag and this sort of suite of tags is associated with a certain version of the mobile app the theory behind microservices is that you shouldn't need to do that everything should just work magically because your domains are so well defined I've never seen that work yet if anyone's actually got that working then I'd love to see it particularly in the development phases when you're changing quite big things a big change to a microservice might mean a set of changes to other microservices and might mean a new version of the mobile app and if you've just spent all that time getting your UAT done on this version and then you realise that two weeks later I don't know how to get back to this version because I don't know which version of each different thing I had then that's kind of a disaster so you can pull down a manifest stick it in version control and I know this week we had this collection of services and you can do all sorts of other cool things with it once you've got that so one of the things that we started doing is putting together a set of scripts that work with the manifest the first easiest one just clones all the code to your local machine I mean that's super useful for new joiners the next one builds an inventory of the current environment the next one will run make release on every different version of the microservice even things as simple as trying to say tweak a make file or a docker file across 12 or 14 or 20 different services it actually has become easier to be able to just say I just want to run said across every different service so often you can share the files often you can have a single module where you keep everything but what we've found is that every microservice at the very least needs a make file and a docker file and probably some build files like a groovy script or something and we're ending up having to write tooling to move all these things around so it's really important to have a way to keep track of everything and then once you've got this manifest it's like the magic key from that point you can actually start to run things and different stuff and it opens up a whole world of things you can do another tip is the developers machine comes first so if there's something which kind of violates the effectiveness of a developer you need to stop and pause and think so that might mean if a developer can't use a certain downstream system for testing that might mean you need to invest the time in creating a mocked version of that system which runs in a container which allows them to get their job done if a developer is not allowed to access your private gold-plated artifact repository for day-to-day development activities maybe you need to use a cloud one for the kind of low sensitivity stuff and the gold-plated one for the more sensitive stuff but when you've got 70 different people trying to build stuff super, super quickly their machine really is king and they're not spent on their individual boxes and that kind of comes back to the docker files as well if a developer is building a docker file which is fundamentally different from the docker file that the CI machine is building then there's a mistake there we're kind of violating the principles of docker and what we're trying to do which is create immutable chunks or blocks of logic that can be built anywhere because they're independent of the environment that they were built in and the environment they run in the network connections, file systems and basic infrastructure another quick tip I think we're starting to get there time-wise so I'll speed up a little bit we made another big mistake which is we created a monolithic project which contained all of our, in this case Kubernetes files for config and at first that seemed kind of sensible like all of my routes are here and all of my services are here and then we realized that we'd kind of violated the principle of these microservices so you couldn't do anything with a service until you also went and changed this other project and people were forgetting about the two so what we now have is a much more sane approach which is that every project will have at the bare minimum some Jenkins stuff so it knows how to build itself some Kubernetes stuff so that it knows exactly what its running environment is like the source and the tests the docker file and the make file so this is a unit which you can kind of take this and push it pretty much anywhere push it to your own GKE environment if you want to or you can just use the docker file to build a docker image and run it with docker compose but we discovered that spreading stuff out in lots of different places was really actually quite an anti-pattern so the only monolith project we have now is the manifest and the manifest is just links to other things and a few scripts for working with other things so this was a mistake we made that we're kind of still rolling back from now the other reason that we discovered this execution environments and the code are really inextricably linked so an example of this is if I change the logic in my code for a health check is that going to change the health check that I'm running say on Kubernetes actually a lot of the time yes it is maybe I don't want to call it every 5 seconds maybe I need to call it in a different way and these two things are often fairly inextricably linked so having two pull requests going on in parallel that one team have to deploy this and one team have to deploy this in a different sense so if you realize you've got like two windows open all the time then it's probably that kind of anti-pattern the final tip I'm going to try and leave everyone with something useful 5 minutes so I'll leave you with something useful I hope and a couple of minutes spare is love the shell and spend all your time in the shell with so many components the days of being able to use the mouse in web spheres interface or IIS interface they're just gone you have to be able to pretty much script everything manually doing stuff across 10 services is hard but doing it across 30 is just impossible so find ways to script everything spend as much time in the shell as you can and get really really good at it and treat your scripts with more importance than your code the thing that I dislike the most when I'm seeing projects that I just jump into is if there's a whole bunch of scripts with no comments, with loads of stuff scripts are probably in many cases just as important as your code, if not more important so treat your scripts with the same attention you treat your code test them, verify them, code review them document them, you've just got to document the cracker of everything so this is pretty much the end of the talk I'm going to leave you with one thing because I don't know whether anyone found this interesting or not but whether you find it interesting or not at least hopefully you'll have one take away that you can use and think that was a crappy talk I've never learned this little thing so the other day I was trying to replace a file replace a line in one of my make files and I wanted to replace it across 20 different projects a colleague was sitting with me trying to remember the sed command to do a regex across a file and if anyone's ever seen the man page for sed how do I do a regex in that file anyone, can I have an example please I mean look at the size of this so a little take away is if you're spending more time in the shell TLDR sed it always fits on one screen it gives you the basics that you need to know about so what's on the top of the list how do I replace something in a file how do I match certain things and TLDR is an open source project you can add you can add your own man pages and send them out and it's got it for everything so anyone forget how to do cherry pick or squash TLDR git squash oh page not found TLDR git DR text replace all of these different things if you're spending more time in the shell forget about man pages and try TLDR it will run locally if it doesn't find something it will try to download it to see if someone's added an update recently yeah that's right so the first time you run it it gets it for you this is literally one of those tools since starting to use it about four or five times a day I find myself going thank you so whether the talk was good or not I'm going to do this away and you think rubbish talk but nice little tool there I'm also on my blog writing a series on shell programming an effective shell programming so you can see that as well was there a question there sorry oh yes yes oh can I share is that it a little bit of a shameless self advertising name so thanks everyone please come and find me if you want to know any more I write about a lot of this stuff on my blog and I try and open source as much as I can from what I'm doing come and talk to me about tech or microservice or whatever come and talk to me if you're interested in learning more about McKinsey we don't just do PowerPoint we do some other stuff as well and thanks very much so we are going on the launch rate so it will be for about an hour in terms of logistics we will do the give myself a little bit of space for that it's good what time is it after lunch yeah after lunch oh I I I I I I I I I I I I I I Yeah I would say I'll catch you later It's just I've been doing production I responded to all the messages this morning It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy It's just been happy I'm always looking for new things to automate. There's a lot of interesting things with security, with automated compliance and highly regulated environments. Happy to talk about that stuff afterwards. Thank you very much. I'm happy to dive deeper into this with any of you who are around. And this is how to get a hold of me. Thank you very much. I cheated for a bit. Please forgive me. But this is what happens when the slides are sent. Hello. Are you ready? Sure. Everyone had a fail start. You got this all automated? Yes. I noticed you were automated before that was cool. All right. So you guys already know who I am. I'm here to talk to you about this. This is my love. I am really passionate about this space because I don't think we do a good enough job about it. So I would love to talk about this more afterwards. The software that we write is full of holes. It's full of problems. Really stupid stuff when you look at it like this. Hard coded passwords, bad crypto. I mean we're just not doing a good job of securing our applications. Most attacks happen through the application layer, right? It's not necessarily software we build. It's software we buy or maybe it's open source that we stand up on the edges of our network. There is risk everywhere. And largely we get in through the application. So here's some key ones and there's obviously one missing the recent breach. But you hear people talk about I'm going to secure the most important applications. Well, Target got breached through an HVAC contractor's billing website. Equifax. Again, I don't think they knew where to patch. I don't think that they knew that they were vulnerable until afterwards when they came back into the forensics and said, oh my God. But look, you know, CIO, CEO, CISO all fired. As we talk about technologies, we do things the hard way in waterfall because we only do it once a year and we never get that payback. As you get towards DevOps, you have to automate the world. I love this quote from Nathan Harvey. This is the one that I've glued myself to. It's about changing your entire organization, right? If my support guy picks up the phone and the customer says, hey, the thing that you just shipped is broken. They say, what thing? We have to train everybody. Security's often sitting on the outside of the scrum team looking in going, jeez, I hope they're doing the right thing. I really can't tell. I don't understand the process. They don't invite me and they end up fighting for budget. Here's 10 things that need to be fixed. Maybe they get one or two. Appsec programs look like gates or look like the sea snow, right? A guy standing on the bridge saying, you guys can't ship. It doesn't lead to good results and it doesn't make us feel good about security. They're not being a partner with us and they need to be. So the outcomes are either, you know, there's a gate but I just drive around it or I call my boss who tells me I can drive around it. Or there really is a gate and we crash into it. So you talk about your highly regulated environments. This is what happens there. Neither of those is a good outcome. So the strategy, and you're going to hear me talk about this again, it's mostly about the people. The problem with software is the people that write it and the people that test it. Do you know if you're a developer? Do you know you appear in security? If you're in security, do you know you appear in development? If you don't, then you should. You need to build those relationships. Those are key. It's not, hey, those security people, no, it's Lisa in security, you're Bob in security. You need to make that relationship because what we want is accountability. We want to be measured as a team, security and development together about the software that we build. Make it part of our goals. People don't do it because they're not goal to do it. They're goal to release software quickly. So if you look at an automated pipeline, we need to have tools that we run before check-in that are exactly the same as what we run after check-in. So if you've got a CI CD pipeline and you've got testing in there and the developers can't take the test before they ship, then the pipeline is always broken. Now, one thing that we've measured in our state of software security is around a couple of key things. One of them is training. If we give them and give the developers tools to train on six times better and six times faster, for you, it's about doing some reading. You have to understand how software is built today if you are a security professional, right? So when I talk to security people, it's about reading the Phoenix project and if anyone in the room hasn't read that yet, it should be required reading. You have to do that. Helping developers fix what they find. So as developers, we have never been trained in this. We need to get trained and we need someone to help us. If they do that, we're going to be faster because of it. We shouldn't be brailing around the Internet trying to figure it out. This is the talk I just gave, by the way, Security Champions Program. Eyes and ears of security, getting some specialized training, being the people in the room during the grooming sessions to talk about security, to make sure it's front and center, that we're talking about it while we're planning it. So training throughout, running static analysis the second you start writing code in your IDE, running dynamic testing all the way into production because of the configuration drift, which we heard about today. That's a real thing. In the end, it's about us making a partnership, right? It's about us taking training, us getting training, the security team coming in being a partner and not a boss, right? Because at the end of the day, our business cards all say the same company name on them. And that's what we need to react to, the fact that we're doing right by the company. But until our bosses take it seriously and make it part of our goals, we won't do the right things. Thank you very much. Actually demos all the Ignite speakers. So I need three bullet ears and we'll do the Ignite karaoke. So can I have three brave hands raised? I have one. Two more. Two more. There's three liters of beer, cropped beer. I'll do three if you want to. Can we have two more maybe? Okay. Hey Graham. And one more maybe. And I'll nominate if you don't volunteer. Okay. I'll get the beer. Yeah. So we can have four if you also, Fabian, you also want? Okay. Yeah. Please come to the stage. So what we have is a service for Ignite karaoke. So it's karaoke as a service. We have preceded some topics. The goal for it is you talk for each slide 10 seconds. You have no idea what is on that slide, but we give you a topic. So you need to role play and get into the game and talk about, okay, let's, let's take some pre-order. You worked first, we're the second. Fabian, you were the fourth and then you were the third. So can we have the names? Hi, my name is David Sager. Hi, I'm Graham. I'm going to hold it. I'll be excited for five minutes. I mean, very soon. My name is Nicole. Oh, my name is Fabian. Okay, awesome. So you get to go first. I'll give you the microphone. So I hit play and you just start presenting. Okay, for like five minutes. No, it's going to be two minutes. Okay. Ten seconds per slide. So it's going to be a total of 12 slides. Okay. Cool. Yep. Not going to die, but you're fighting for the beer. So you need to put this on. Okay, ready? Okay, this is a slide to young women playing. They are future. They're going to be the next SREs. How do you support them in school? Well, the secret is the... The secret is to make it as easy as possible for them to do whatever it is they need to do when I'm doing my SREs. And we need to make sure that they have the tools to do the job and the tools are robust and not easily breakable, especially if you hold them up. I think this slide is for itself. I'm just going to give myself a minute. Ten seconds for myself. Ten seconds. Bingo. Victory. Yeah, we are like shit gambling in the game. Thanks for the topic, it's change management. What do I name this slide? Change management. Change management. See, the reason I'm having things so good about this is because really, we should all be practicing immutable infrastructure. So there should be no changes. That's not good. Let's not stay stuck in the past. Immutable infrastructure for everyone. Bots. Bots. Even bots for robots. Only bots. Who were the questions asked before? Bingo. The dictionary. The entire media. Your mother. And these are the answers you get. Fails. All the way through. Parents. Family. Mothers. Yes, old fathers. And there is a lot of data out there. A lot of it's false as well. And such a guy kept increasing, further and further and further. And now we start drinking coffee too. The more coffee, the repeatability of getting a coffee jar in it. The drinking of the coffee, these are the important things in life. The water's a messy, messy place. The coffee doesn't help with this. And we have robots to get us coffee. And we're, you know, we're distilling coffee, it looks like. It's probably whiskey, but let's call it coffee. This is a business environment. And it triggers on, and on, and on, and on, and on, and on, and on, and on, and on, and on, and on. But we will have a freedom. And we get a freedom here. Obviously it's my slaughter in the beach. I mean, but now it's sad. Because the robots are not talking to us anymore. This is a big problem. And we just found out Santa's dead. Life is hard. We're all very sad. And we should try and scream at the robots because they don't care. This is thinking, right? This is one of those fair ideas that I think we need in the office. Like tiny little phone booths. Conversation in quiet. Walls skating from the shops. Thank you. Continuous delivery. Okay. Cut off. We don't continuous delivery. Continuous delivery. Because security has cut the wire. Okay. You ready? Continuous delivery. Ooh. So despite continuous delivery, you begin the moment you have to still meet the origin. Continuous delivery. No delivery is the best delivery. So we have to make sure that when you give your stuff, it's sent up in the right case by its own. So that's probably the concept behind continuous delivery. Bottle. I pass. I have a dream. Products are built and end up in the right place all by themselves. Some of the biggest things with continuous delivery is being able to put it across your organization, discipline, right? Leaching to the right audience at the right time. So complex parts, ideas. So continuous delivery is also about making sure our organization has transparency. You can have your breakfast peacefully. You know, it's not always the case because you wake up in the morning, you have to run through office, or that's it. So continuous delivery is about solving problems, of course, and it has to be gemmulistic as you can, right? And you build continuous delivery as DevOps engineers on a day-to-day basis, but your tools and systems that you build decides how you succeed inside your organization. Thank you. Fabian, you go last. Sure. As digital transformation, my favorite topic. Digital transformati. Transformati? Yeah. All right, you ready? I'm ready. Okay, digital transformation. So how do we get from letters to storing into huge hard drives into USBs and now we're littering Earth with all these USBs? We want to do away with all these USBs. So we serve one another data by... We used to be, you know, serving it by hand. Now we're not. We're using emails. We're using Slack. And that does not look like a digital transformation. It looks like Autobots actually landed there. Yeah, that actually killed a lot of people. And bananas. So a fun fact, if you eat a lot of bananas in a day, you actually die of radiation. But that's kind of a digital transformation by itself. You also want to not have black and white photographs now because they're still awesome. Remember Instagram? They kind of transform how we look at cats. And they are now looking into Instagram and they have Instagram profiles now. So, wow, they're all digitized. So how much? You can do much more with a catabell. A digital catabell. You don't actually have to live the catabell to do exercise now. You can ask the app to do an exercise for you. So we have over 2.7 millions that just perform on Google each month. That's more than the questions you can ask your wife or your mom. Okay? So this person... Oh, awesome. Now, using robots and digital transformation, you don't have enough humans. So you do away with injuries. And you think big, you dream big. Because with all the DevOps tools and skills, there's no limit, right? And of course, now you can bring yourself outside of the world and look at Australia. Cool. Yeah, you never get to work on time. It's okay, because digital transformation. So I hope everyone enjoyed. It's a boost of energy. Now, they all are winners. So what shall we do? Shall we vote? Or we... I'll just give it to all four of you. There's three liters and you share it. That's four. Here you go. So now, to thank you again, guys, for doing the Ignite program. I love the lot. So I hope everyone else in the audience has a good mood. So if we get back into the program, we have three more open spaces. So at three o'clock, we will start the first open space. So that means we have about 25 minutes for bioregues and deciding on the topics that we want to discuss as a group. So what we will do is, as agreed, we'll pick up the top 12... Oh, that's interesting. So we'll pick the 12. And then we will work on the schedule. So give me five minutes. I'll write it on the sticky notes. Then we'll go to the white board, which is in the right corner. And then we will decide who wants to speak, who wants to lead any of these topics, who proposed it. And then we'll build a grid to make sure that everyone gets a chance to participate in those conversations. And I will call also the facilities to split the room. So we'll have four groups. We'll have four groups. So we'll split it in half. And then I will need your help to rearrange the chairs in circles.