 Felly, ydych chi'n edrych, mi gweithio'r wybodaeth cerddordeb yn ddechrau'r deltaeth, a'r cerddordeb yn ddechrau'n gweithio'r cyflonddau. Felly, rydyn ni'n cael ei fath oherwydd wedi'i wneud yn gweithio arall, ym mwyn fmouth a'r cyflodd yn y dda'r ddechrau cyflodd. A'r cyflodd yn dda'r dda'r cyflodd? Roedden i'n bwysig i mewn gwahanol. Roedden i'n bwysig i bwysig i gydag prosesodd gyda own gan mynd yn dweud y dy wybodaeth. Felly, mae'n cael rhaid o'r sixi wahanol sy'n ei wneud o'r periwyr o'r ymddangos dwy fyrdiadau a'r etoedd yma yn ei bod yn gweithio'r dynosor, oedd yna'r ysgol yn ymddangos i'r baratau yn ymddangos, dyfodol ddweud bod wrth gael ychydig, ond mae'n rhaid o'r hyn o'r ysgol. Felly, mae'n gweithio'n oedd unrhyw ar y baratau mewn Arsa P yw'r unrhyw i'r ysgol, mae'n gallu i'r eisiau gweld arna i ddefnyddio'r pryd, ien oedd y ddechrau Fyryddio. Mae rhai gallai Majesty-ddedig yma ar y dda'r newydd yma, yma'r ymgyrch, ymgyrch, i eich mwyaf ar y ddechrau Gwymorol, a ymgyrch. Mae rhai gwneud yn ymgyrch, yn gwybod wedi'i ddweud y cwrtog ymlaen, cyrnyddu drwy adroddau, cyrnyddu 60,000, mae'r grwrs cyd-degwydd yn ymgyrch yn ymgyrch ar gyfer gyda'r trofynu'r cyd-degwydd, yn ymgyrch ar gyfer, dyn ni'n gwybod i gael gweithio'r gweithio'r gwaith i gael gweithio'r sap a'r ddif yn gweithio'r gweithio'r disoedd. Mae'n gweithio'r gweithio'r gweithio'r gweithio'r gweithio'r gweithio. Yn y cyflech sydd o'r ddweud y gweithio, yna'n rhan oedd yn 40 yma, mae'r sefydliadau eich gydigiaeth. Mae'r debyg yma wedi'u ddweud y gweithio'r ddweud. The database schema for that is more than 30,000 tables. That was a shock to me in my software career, to come across a piece of software that was completely beyond the possibility of one person to comprehend. You have to divide and conquer this stuff. Our vision is to help the world run better. One of the ways we're helping to achieve that is our customers produce, as it says, a large proportion of the world's chocolate and beer. I'm not going to talk about SAP as a whole and the entire business suite side, I'm going to talk about my little corner where I work in SAP's IT division. I'm part of the group that's responsible for putting out a lot of our websites, especially the more collaborative websites where people register. A ffew years ago, we found ourselves working on a platform delivering software using a version of Java that had gone out of support two years earlier. This big proprietary application server where just performing a restart took maybe 20 minutes to see if any code had worked. We weren't entirely bad. We thought we were agile. We had monthly release, for example, which isn't too bad for back then. Obviously, it's a lot better now. For example, we were working with physical hardware. If we wanted to do a new project, you're talking about lead time for hardware of several months of talking to vendors, getting the infrastructure team to plug it in, the operating system team to install an operating system, and so on. It just took a long time. Also, on each and monthly release, someone would write a release script that says you need to deploy these artifacts here, then tweak these configuration parameters. It took most of a day to do a deployment and obviously it's prone to errors as well. We had a mostly manual QA cycle. Within that monthly cycle, we'd be writing code. On paper, it said we were writing code for a month. In practice, we were handing the code over to QA after two weeks. In theory, we would start on the next month cycle. What really happened, of course, is in those two weeks, QA would be finding bugs, handing them back to developers. It actually turned into two weeks of development and two weeks of bug fixing. Perhaps one of the more damaging things from a continuous delivery point of view is that the developers, the ops guys doing the actual deployment of the artifacts, and the guys who were responsible for keeping the lights on, were in three completely different business units. Obviously, in a large organisation, you try not to have politics, but some of it's inevitable. There's a bit of a not-invented here syndrome and elements of that did affect us having to basically throw things over organisational boundaries to get things done. We got to a point where in 2010, our units of about 100 people, that equates to about 20,000 person days of available effort in one year. We found ourselves handed a portfolio that when we stuck the estimates on it, showed it was going to take three times as much effort as we had available to deliver that portfolio. We had to really rapidly find a way of eliminating a large amount of waste from our process. That took us into the next stage where we had a new project starting off and that's the project that I work on. Again, a little bit of background on that. SAP, the big enterprise resource software, part of the idea of that is it does everything from human resources, invoising, purchasing, everything in the back end of a company. The point of that is there's one version of the truth in this big system. There's just one database backing everything. At the same time, if you went to our website on the marketing side, you could create an account for that. If you were a developer wanting to do extensions, there was another developer website and you could register for that. Between all of the various websites we had in this big enterprise, some people perhaps needed to create ten different accounts to register and talk to this company that was selling them the concept of one version of the truth. It was a little bit embarrassing. The project I work on, the ID service, was set up to unify all of those. The business still had the agility where it could do lots of different websites. We had a common user database so people signed up once and they could use those account credentials to get into any part of the system. It then looks a lot more like one SAP to the outside world. Putting all of the existing systems together gave us about four million users. Then, of course, we've acquired a couple of companies in the last few years that are cloud-based companies. That's given us another 20 million or so. The CEOs have given us a helpful target of 2015 when you have a billion users in that system. That's the challenge within our project as well. We've got a scale to almost Facebook scale within that period of time. We've done that. We've gone the agile way. We've got a cross-functional team. All the usual roles. Unusually, the company standard is when we're doing scrum-type stuff to try and have co-located teams. For historical reasons within IT, because of all the various acquisitions and combining IT departments from the acquisitions, we have people all around. Our particular team is composed of people from five countries in Germany, Bulgaria, UK, Russia and Israel. We do have that challenge, although the one helpful thing that mitigates that is we're all within two hours of each other time zone-wise. That's one thing that we found very helpful compared to other outsourcing-type things I've done in the past. Having more or less the same office hours is very helpful indeed. For this new project, we decided not to use the big monolithic application server, because SAP, starting to get into the cloud game, developed a much lighter weights server based on Tomcat and RSGI. We called it the Lean Java server internally and you can restart that in a minute or two. We decided to use that to try and decrease our cycle time for development. Otherwise, we carried on using our existing toolkits, so Jira and Bamboo for issues and continuous integration, perforce, version control, eclipse and all that kind of thing. That was our starting point. We just really changed the platform to make it a bit quicker. Also, a developer could run their own Tomcat server rather than having a shared server, which is what we had before. Really, we still had a lot of waste to get rid of. That maybe took care of about 10-15% of our waste. Faced with still a large task to get all of this portfolio delivered, my boss, the chief architect in our group, he'd read Jesse's book. He promoted these principles to the team that everything has to be automated. We have to do automated testing. Everything has to be in version control. In fact, all of his team, he put it in our performance review objectives that we had to read Jesse's book, which is really good for Jesse's royalties. My project was adopted as the pilot project for putting all of this into practice. At the same time, his boss trusted him to do that. That's one of the things that was critical. We didn't do a big top-down enterprise agile thing. This was done almost as a guerrilla thing. With the lower two levels of management, one guy actually driving the vision and the guy above him providing air cover that enabled us just to get on and do what we needed to do. How that was done was essentially within all of that, 10% of any given sub-project budget was given over to putting continuous delivery technology in place. A lot of the tools you'll see in the rest of the talk were done by people on this 10% time. It wasn't really a Google-like Friday working. It was more like the 10% of the people who were really into doing this sort of thing formed a bit of a Black Ops team and put it all into practice. The first phase of that was really making sure that we got rid of this lead time for getting a hold of hardware. The one hardware purchase we did was for a big ESX server that gave us the ability to have lots of virtual instances. That finally gave the ability for each developer to have their own server to run everything on, to be able to run all of the tests and everything else. That meant that if you wanted a server to run a project on, at the time you emailed one of the Ops team, they ran a bit of a script. They allocated something in ESX server, told you the DNS to go into, gave you credentials and they installed the chef client on it. Is everyone here familiar with Chef? Yep. The idea there is we set up a Chef server in our IT area and that had all of the recipes on. Any given developer just said, I need MySQL, I need Java, I need Tomcat, install these recipes. You can see there there's a sample bit of Chef Ruby dialect. It's just a DSL that says I want to install these packages. I might put some conditional logic in to say, depending on the platform, do it differently. The idea is abstracting something that could be done with a simple shell script, but it's done in such a way that it can work across multiple platforms and it's repeatable. From the developer's point of view, some developers were into the Ruby level and creating recipes, but for those that weren't into that level, they were just able to drag and drop things. They went on to the overall Chef server, they clicked on their node and then they can just drag and drop roles and recipes. Then the next time they run Chef Client on their machine, it installs everything they need. That was the automation of one layer of actually getting rid of the waste of waiting to provision hardware. Then we needed to be able to automate the testing. Half of our effort was going into manual testing. We had no culture of developer-created tests at all, JUnit, all that sort of thing. We all knew about it, we all wanted to do it. We didn't feel before that we'd have the space in which to create that kind of thing. The QA team had created some automated tests, but that was using some of the very expensive seat license tools that were available at the time. It just wasn't economically possible to give developers access to that level of technology. Even so, those automated tests were just run as part of this monthly cycle. It was causing us a lot of trouble. We discovered Selenium, and that's one of the early tools that helped us along a lot. It's just being able to record scripts in a browser and play them back, and developers having access to be able to do that made things a lot quicker. Developers got instant feedback. We didn't have to wait for that monthly cycle. The only thing we did have to do, of course, is create test scripts. Going from zero to covering everything was a challenge. The strategy we adopted for doing that was just to say, well, the only thing we really need to cover first is the happy path. For everything we're doing in our system, let's just look at the way it's meant to work, put a quick recording together for saying, if you click A, B and C, it comes out with the right screen at the end. Let's not worry too much about the error cases or the edge cases. Just make sure the basics are covered. Then what we did is, as we gradually went along, a manual testing may find a particular burger or an edge case that we haven't covered, then we'd do that on a, you know, creating a test on a failure basis. A sort of on-demand test creation that meant over time we evolved to a situation where all of those edge cases got covered. By this point, we've done a fairly good job. We've noticed that although we've got rid of a lot of the previous waste, we've still got manual processes here to operate these new tools. You have to email an ops guy to get him to create you a VMware server. Chef Client has to be installed. One of our bright guys actually pulled all of this stuff together and automated that into a script that we called Cocktail. That really gave us our first level of automation where it turned from maybe a couple of hours of emailing people and them doing things fairly quickly into just running a script to achieve the same job. Then we decided to ramp up a notch more. We had some test coverage but it was all very internal focused on the team, very technical level. We decided to move towards Cucumber because this is where we started getting a bit more agile and more able to talk to the business. At the same time, I'll talk about it a bit more later, we've got heavily into scrum. The product owner now can start talking in terms of these guirkin scripts which are just in English language. They're a little bit technical. We're trying to do what Matt Winn says as much as possible and make them business focused, but that's one of our areas of continuing to improve. As you can see, it's fairly straightforward. They've got a scenario description given when then. Now we've defined our definition of done. When we actually want to complete a user story, just having the code there isn't enough. It won't actually count as done unless there's a test. For every JIRA ticket we create for a story now, we actually create a sub-ticket for the Cucumber test. Then the JIRA is set up so that the master ticket can't actually be set to done unless the sub-ticket is completed. From a developer's point of view, Cucumber is really nice because it's very easy to map Cucumber steps onto Java code to perform the testing. As you can see there, that one line when I'm using my valid credentials easily maps onto some code that says get me a username, get me a password and try and do a login. At the same time, something we've developed recently that's also making a big difference is on each story within Cucumber we've got this annotation at user story and that contains the JIRA ID of the actual user story itself. We've written a reporting tool now that you can see on the bottom right there gives us a report that tells us all about the test coverage so Product Owner now can see not only here's the user story that I wrote but here are all of the automated tests that cover it that I can look at in Gherkin in plain language and also I can see when they're running how long they take to run so we're getting a bit of an overview of performance as well. From an audit point of view we can see if you've only got one test for a particular user story that's okay but maybe it's better to have three or four covering some different edge cases. So then we decided to ramp up our continuous delivery technology in another stage. So from the cocktail tool where we were able to run some basic scripts but we've got the central chef's server the problem we had with the central chef's server is we had by then six or seven projects all using this chef's server with this one combined set of recipes. So someone on one team decides that the recipe for installing Tomcat needs a bit of a tweak for their project but they don't really think about the impact on all the other projects of that tweak. So in fact having one chef's server works okay within a team but not so well across multiple teams. So we need to now set up one chef's server for every project. Also we want developer self-service it's nice sending an email and something comes back a couple of hours later but what would be really good is to get a distant virtual machine created. And we want everything under version control. We automate everything everything's under version control. If you're having to email people to set things up that's not very versiony. So we developed a next level tool called BarKeeper which manages all of this. And that gives us this YAML text file that describes an entire project. So you can see though we've got the project description ID of the git repository and then towards the bottom half you can see in this case test and prod it's a sample of a file as other for dev and private developer stuff but in each case we're actually describing here are the servers we need here are the initial recipes to run and so an entire project its entire landscape is captured in one text file that's now under version control. And that text file is read by the BarKeeper tool so now the there's a web UI on top of that reading all of the files of all of the projects and a developer who wants to work on a new project just goes to the list on the left says that's all of the project that BarKeeper is managing clicks on one of them clicks create me a virtual machine and it will just go off and within a few minutes that developer then has all of the servers he needs to work on that project. So with that tool and its ability to be automated via REST as well this is now how our bamboo pipeline looks so we've got an initial actual compiling the code step that's installed onto the development server and then we have all of the test suites running the reason there's so many suites there is because bamboo runs all of the stages in a particular group in parallel so they all run at the same time once all of the tests are complete then we know it's okay to install onto the QA and test landscapes where we do a quick smoke test and then run another test suite and then also you'll see we've got the blue-green deployment there so once it's into QA we check that it's basically okay and then deploy it to blue and green so it's available to everyone and the reason is that it's semi-productive in the sense of the project that we're building is used by other projects for their sign-on so what for us is our QA system is also the system other projects use to do their development so for them it's productive if there's a big problem in QA they can't develop anymore so we have blue-green even at the QA level and then as you can see there the grayed out section on the right is for production so every build doesn't yet go to production but it's very simple if we want to do a release to production then bamboo just enable those steps click build and that's a deployment to production completed so that gives us some lessons the major one of which is the cycle time is critical the amount of time from a developer hitting commit to getting feedback is one of the main bottlenecks in doing development so as you can see from the graph on the right that's from bamboo showing the build duration and you can see it's gradually increasing here so we've just added an extra five minutes and then it drops down again and that's because we've got a rule that when we see that starting to creep up beyond a certain threshold you saw all of the test suites on the previous page running in parallel each of those is composed of lots of steps and scenarios as you can see we have over 700 scenarios at the moment so every time we see one of those steps creeping over three minutes or if you see the overall build creeping over 15 minutes for that stage then we look and go in and split up so we get even more parallelism and then it drops down again and so those are our current targets is for the actual test stages to complete we want that done within 15 minutes we want the overall build and deploy to QA done within 40 minutes so that the impact of a red build is minimised as well so we know about a red build in under an hour and if that's not fixed within 30 minutes or if a developer hasn't said I've got the fix ready to deploy in 30 minutes then they know that it's time to issue a revert commit to get it back to green as you can see very recently there was a build that took almost two hours that was just a normally infrastructure problem that's making the graph look weird so a quick recap on what we've got so far that's mostly technology stuff a bit of culture but we've gone from monthly releases and now we release to production on average twice a week the QA cycle is now well under a day essentially the manual QA portion is now is looking at the tickets for that sprint and just quickly going through making sure the developers didn't miss any corner cases the regression is taking care of from the automated test suite before in our old system if there's any kind of production error it was the classic logging in at 2am and that's really not the best frame of mind to be fixing a bug especially when there's a stress of it being an important bug now because I have blue-green deployment in QA and in production if there's any kind of problem like that all we do is just switch back to the other release when it was working fine and then go and get some good sleep and then come in the next morning think about it properly, fix it properly and there's no panic before it took six months to get a new idea into live production because of ordering the hardware and that actually stifled innovation when you've got a six month cycle time just to see something live there's a lot of ideas that just don't even get done whereas now with the barkeeper someone writes a text file you can create all of your servers in dev QA production write some code, deploy it, have the tests if you can write the code quickly enough to have a new project live in production in a week and that's mostly for internal stuff but that's created some interesting internal applications that people have written and as a result the business is obviously a lot happier with what we're doing even more so with SAPs doing more in the cloud area of things what we've found is these techniques and technologies that we've been developing within IT we're now actually working with the development side of the company to make them part of our cloud product so it's great it's one of the big revolutions we've had in SAP in IT in the last few years is rather than being a cost to the business we're now a net benefit to the business through working on technology and also being through our own reference customer we're trying to use our own software wherever possible and then we can help the sales guy sell it as a result but that is technology and there's still a lot more about the team and really what enabled us to build the technology was the transformation that we had as a team and I've already spoken about the initial part of that which was around the management cover and the vision of the chief architect so we decided to do Scrum at quite an early point in time but what we decided to do was the Scrum that we'd read about by reading web pages and articles so that ended up as a daily stand-up pretty much all it was in fact it turned into a daily argument it was just the argument of the day so we did get some better communication out of that but not a huge amount of extra productivity what really made the difference is deciding to invest in Scrum training and it was actually not just from a Scrum training company but from an agile coaching company as well so we spent an entire week we got the whole distributed team together in Berlin which is where most of the team members were and we had deep learning about Agile and Scrum so actually doing the hands-on exercises together experiencing what waste felt like with Lego blocks and post-its really learning how it went to work but also because this was agile coaching the coaches themselves adapted to our unique situation of being distributed so in fact one of the guys on the team couldn't make it from London to Berlin his passport was running out and he couldn't make the flight so he was left back in London so we did the whole coaching with him we were all round a desk in a workshop in Berlin we had a laptop with Skype and he was on the camera on the laptop as a virtual member of the team and that actually helped us work in our distributed mode as well so in some of the exercises where I needed someone to be a product owner we decided that he would be the product owner so we had this sort of head on a screen telling us what he wanted to do for that exercise and similarly for the pair programming tasks in how we learnt we didn't just sit round a desk together and pair because we're a distributed team we can't do that so within the building we actually sent some developers to other offices in the building and we did the exercises over phone on the extension we had sets on or with remote video so we actually really worked with what we had and as a result we really managed to make being a distributed team work very well for us so we've got this culture now where we're always trying to improve what we do ourselves and we try new things all of the time we had to introduce code review for example now being an ID service we need to make sure no one's bringing in any security problems so initially the code review was just to ensure that security problems weren't there but what we found was that using Garrett for code review was another way that enabled pair programming in a distributed way so when you want to swap the keyboard what we found was all you need to do in eclipse is to click push to Garrett and the other person does fetch from Garrett in 10 seconds they've got all the latest code updates and that's working just off a patch together so it's not even committed to the main code base so the two people can collaborate on a patch using Garrett and it makes it very easy just to swap over the keyboard similarly we've been working with Pomodoro Technique a little bit that one didn't work out so well but it was a cheap experiment to do and the latest one we're doing is something we're calling products so as you saw on the continuous integration pipeline the production part was greyed out and there's a bit of permission control around that so only the ops guys can actually send it to production and at the moment the product owner sends an email to the ops guy as you can see there can you send this deployment to production now we've figured out that actually because this was being done in bamboo and bamboo has a REST API this can be automated so this is what we're working on at the moment we've got a physical device which the product owner has and all the product owner does is he can use the buttons to select the right build number that he wants to say to production inserts the key, unlocks it, flips up the switch and presses the button and that just uses a raspberry pi inside the box to call the REST API and actually perform as a production deployment but you have to have the key to be able to do that so I'm out of time, slightly over time in fact so thank you very much for listening I hope it was useful