 OK. So, gwaith yn fawr, rwy'n gweithio. Roeddwn i'n ddiwedd i'r Gweithio Downhague gyda SAP. A'r gweithio'n gweld i'n meddwl o'r gweithio'r adeg yn ymlaen gan SAP yn y piliad yng Nghymru, sy'n meddwl a'r defops yn ddioghau. A'r gweithio'n meddwl yma wedi y cyfnod gyda'r platform defops o'r platform o'r ddaffan ac mae'n gweithio'n ddechrau a'r dweud y gweithio'n meddwl o'r ddechrau ac yna ydych chi'n ceisio byddaeth gyda'u cymhannu o'r fyddion geyledd. Yn ystafello â'r SAAP, gwirionedd a'r ysgrifennu cyflod yw meddwl â'r pwysig, y dyma yn mynd o gyflym o'r trailng. Mae gweithio y stymes yma o feithio y gallun heddiw yn beth thyth. Mae gweithio iawn i ceisio araloeth, bod yna gweithio 41 adynhyrchu sydd yn y ddechrau. Mae'r awfyr wrthynach gan ein ffordd o gwirionedd gwyrdd yma. A'r dweud o gwasbwynt yn ei hunain o'r dweud yn gweithio'r cyd-fawr ac mae'r ddweud o'r cymhwylion yn oes cyfwylio'r ddweud, mae'r ddweud o'n 23 rydyn ni'n cyflwyffind ychydig o'r cyd-fawr. Felly mae'n gwybod yw hynny'n ei wneud o'r ddweud o'r ddweud o'r cyd-fawr. Rydyn ni'n gweithio'r ddweud i gael i'r ddweud o'r gweithio'r dyma. That's how we used to do things back then. We thought it was quite good and we are reasonable at developing the software in the IT department. We had good source control, issue tracking, build automation, and these are all with the Atlassian suite that worked very well for us at the time. We were doing monthly releases of our software. On the negative side, we had a month's lead time for new hardware so if someone wanted to try a new project they could just try it out. maen nhw'n ei ddweud o'r ffordd iawn, byddwch yn bwyd yn cyflwyfr ac yn y dweud o'r ffordd iawn, mae'r hordau yn y ddefnyddio yn y ddedig ac mae'n ymdweud o'r ffordd iawn. A'r wneud hynny'n ddyn nhw'n ei wneud yn cynyddu gwagol. A'r wneud hynny'n ddiddordeb o'r gwaith o'r gwasanaeth o'r ddifad yw'r gwaith. A'r ddweud yw'r cyffredin o'r web-fagus o'r gwaith i ddechrau ond rwy'n cael ei fod yn cyfael i ddim yn ei rhoi'r prynhau ymlaen. Wrth gwrs, rwy'n cael y tîm opirau, ar y tîm QA, ac every time we did a release we would do a typical job of throwing the code over the wall to the operations team and then they would look up in their word documents, their Excel spreadsheets, of all the things they had to do to deploy that software and put in the right configuration values. That was also prone to quite a lot of manual error. Byddwch yn ymdw i wneud. Mae'r team ymddiannau, ymdw i'r team ymdweud, ac ymdweud yn ymdweud. Mae'n fwyaf ar gyfer busnes yn ymddangos. Mae gyda'r cyflwyno ymdweud ymdweud. Mae'n dweud yn ymdweud yn ymdweud, ac mae'r ffocws yn ymdweud. Mae'r cyflwyno yn oed yn cyflwyno. Mae'r cyflwyno'n cyflwyno yn ymdweud, ac mae'n cefnio i wneud wrth gweithio. Mae'r cyflwno erbyn ymddangos. y cyfnodd ymlaen, dyma'r hyn sy'n cyd-flaen ymlaen yw'r ysgol, rwy'n gyd, dyma'r cyfnodd ymlaen ymlaen, rwy'n gwybod ydych chi'n gweld chi'n cyfnodd. Yn gweithio, rydym eich cyfnodd, mae'n golygiad yma, ar draws, gysylltu, a'r wneud, rwy'n gweithio, mae'n gweithio i'r ysgol. Yn y dyflaen, rydym eich cyfnodd, fy mhagorau cyfeiri a'r cyflwyno a then release to the operations team for deployments of production. Meanwhile, while that testing deployment is going on, the developers are carrying on with the next four weeks. So, we're getting a good monthly deployment of four weeks of solid work. That's the idea. What we actually had was this. So, in the first four weeks of a project, it's great. You do your development, hand it over to QA. Then QA start doing annoying things like finding bugs. A ble mae'r cymdeithas yn rhaglen i'w cyflwyno i'r ysgolion a'r cyflwyno'n gweithio i'w gwbl'r sicrhau. Ac oes yn unig o'r cyflwyno, mae'n meddwl yma'r cyflwyno a'r cyflwyno'n ymddugol. Mae'n meddwl, mae'n meddwl, mae'n meddwl o'r cyflwyno i'r ymddydd i'r ymddydd i'r cyflwyno, mae hi'n meddwl o'r cyflwyno i'w cyflwyno i'r ymddydd, ac nid oes ydych chi wedi'i gweithio'r edrychwch, ychydig iawn, yn ystyrwch am ychydig i'r wych wedi'i gweithio'r rwyf, a'r byddwn yn gyflawn i'r meddwl yn gweithio'r cymdeithasol. Felly, hwnnw'n gweithio ddweud honno, yna bod yw'ch cymdeithasol o'r ddoch chi'n gweithio'r gyflawn. Yn y cwrs fy mwyloedd, roeddwn ni'n meddwl wedi ydweud yn cael cael ei wneud, maybe just not as often or as much as we'd hoped. But we came to a point where we looked at the demand forecast in the IT department for the coming year, and we realised that, with about 100 staff, that's roughly 20,000 person days of effort, and the demand coming in was for 60,000 person days of effort. And there's no way we're going to be able to get budget to hire another 200 staff. We couldn't even ramp up 200 staff in the time it would take. Felly, mae'n ddweud â'r edrych o gywir yw'r fforddi'r ac mae'n teimlo'n ddweud â'r ddweud am y bydd y gallwn. Felly, mae'n hyn syddi'n dda'r ffordd sydd yn gwybod yn fwy o'r pethau ac mae'n dda i'r proses a'r ddechrau'n ymddangosol, mae'n helpu, ac mae'n ddweud â'r ddweud â'r ddweud ac mae'n ddweud â'r ddweud â'r ddweud a'r ddweud â'r ddweud. Mae'n rhan o'r cyfrifoedd yng nghyrchu, Mixer heddiw'r bobl addyniad. Roeddo'n cael ei hyn yn dod yn ddif i wedi hynny, byddwn i'r bobl addyniad yma. Dydyn ni'n gwneud unrhyw gyda'r bobl addyniad. Felly, nid oed yn ddenim ni'n cryn i'ch heddiw i ni. Am wneud o'r bobl addyniad yma lle wneud ein hynny, ond genna ddod beth yny fel mae'r bobl addyniad yn cael ei rai. Rwy'n gandarwch ein bobl edrych yn cael ei defnyrwyr o'r cael y llunio. gyda'n ddweud yr oedd ymddych chi arna i, yn ymgyrch o'r progectau a'n ei gael ei wneud. Felly, ymlaenon hyn ymddych chi'n gawr i'r gyfeirio. Felly, yma'n gweithio i'r rhanol iawn i'ch gael ymddych chi'n gweithio yma i'r 10% a'r progectau ar ymddych chi'n gandol i'r gael. Felly, ymddych chi'n gawr i'r gwaith i'ch gael eich gweithio i'ch gael eich gweithio i'r gweithio i'ch gael. Yna yw'r projekty ar quotes, mae'n封uniait â'r tornado Fabian yna dyn righto'r synod Soul Shiad 먹 De gyll Menedig. Al maen chi plo'r unr alleine a b erioedd yn gweithio pwylliadum trwy teimlo. Mae fans unrhyw deiml ni'n eu swydd录 draws ac so awakening ni canoliad wil maen nhw. A e gyd am bryddoedd byd~! Trafawr iawn o'r swydd yku. toggle. We made it a customer-facing product as part of our cloud portfolio, so it's the single sign-on hub for all of SAP's cloud software, so it really has to be reliable. If this thing goes down, not only can you go and get to SAP's website, but they can't even get to their own software in the cloud. The first thing we did is we got rid of these silos, so instead of having these three different business units, for this pilot a we've got everyone into one team. We've got a product owner together and a scrum master. We've got some user experience people, Java developers. We've got the QA guys in there and the ops guys. They were technically in different units at the time, but we made sure they were in the same room working as part of the same team. I said the same room. It's a virtual room. As you can see there, we were also spread all around the world. Luckily, roughly within the same time zone. That is one thing that I've found to be a success factor for doing this kind of work. Remote working isn't a problem so long as you're all within a couple of hours of each other. Once you start getting out to five, eight hours, then you have issues like someone asks a question and it's next business day before they get an answer to it. That's the people side of it. Here's the first thing we did with DevOps. Getting everything tested. Basically, DevOps boils down to two things. You test everything and you automate everything. We started with the testing. What I hadn't mentioned earlier is one of the things we didn't really have in our previous process was any kind of culture of automated testing, certainly not from the developer's point of view. We worked with a tool called Cucumber, which is an open source tool. Most of the things I'm talking about here are open source. Cucumber works first of all with this language called Gherkin. The QA person works with the product owner and they work on a fairly natural language description of the various things that the product needs to do. In this case you can see the scenario is we want to log on to access someone's user profile and here's what you need to do to log on. Given that you've actually registered at some point, when you try to access your profile, then you should see a login overlay and then when you log in with the right credentials, you get logged in and you see your profile page. The idea of that is that that's a pretty non-technical explanation that the QA team and the product owner can discuss and agree that that's what they want the system to do. There's no code, no technical stuff involved at this stage at all. It's just a language-based description of behaviour. When that's agreed, then the QA team go back to the developers and what they work on is this, and this is where the magic comes in with Cucumber, which line that you've got in the Gherkin, you create a method, it can be currently in Ruby or Java. It doesn't have to be in the language that the system is written in. This is the language that you're testing in. As you can see here, when I log in with my valid credentials, so it gets the login name, it gets the password and then here it's calling Selenium to go to the web page and actually do the login with that username and password. In this case we're writing some Java code on a web browser simulator to actually do the thing that that line of Gherkin talks about. What that gives you is an executable specification which is a really powerful idea. Basically, you've got the thing that you've agreed with the business you're going to do and you can end up running that as a piece of code to check that the system does what you're expected to do. One of the lessons we learned from this is that the cycle time is a critical factor, so how long it takes to run your suite of tests. What you're looking to do is to minimise the amount of time it takes from a developer committing the code to then getting feedback from the system that either works or doesn't work. Obviously, in terms of the developer themselves sitting at their desk writing their individual piece of code, they're going to be writing a test that goes on with it and they're going to get feedback in seconds or minutes at most. What we're looking for here is before they commit, they want to run a test suite to make sure that they haven't broken anything along the way. There we're looking to try and get feedback in about 15 minutes or so to a developer. In other words, they can commit their code, start the test running, go and grab a coffee or something, come back and then they'll get an answer whether it's green or red. Once they're happy with that, then what we want is less than 40 minutes for a full integration suite. In other words, the point where it spits out the end, you're completely happy to deploy it, run all the systems that need to run and it's not broken anything in the wider landscape. So those targets, when you're starting a project are quite easy. You've just got a few tests, they're running seconds, it's no problem. But as you build up, we got to a point where we had over 1,000 scenarios, 10,000 test steps. If each of those is taking a second or two to do something in a web browser, that really adds up. So what we've found is each test suite, which is running a bunch of tests serially, we split them up and split them up so that each test suite would only take around three minutes. Once it got to longer than that, it would split it up again. So we're always getting into this kind of two to three minute domain for running a test suite. And then with those two to three minute suites, we then run that in parallel on the hardware. So obviously if you've got a cloud here, that's really helpful. You can just keep on spawning more and more VMs as you need them to make those suites run in parallel and make sure you get to these 15 and 40 minute targets. So that's the technology side. But of course it is about the people. So what we found was that it was actually a challenge taking a bunch of developers who never really had much experience or enthusiasm for writing automated tests before and getting them to do it. Especially when you've got an existing body of code, the last thing anyone wants to do with this big chunk of code is to go and retrofit a million tests to it. So what we did was to focus not so much on getting every single case possible tested. We focused on making sure that the happy path worked. So in other words, when you log in with the username and password, that works, you do get logged in. We're not going to worry too much about whether if you put the password that's 15 lines too long it causes some stack overflow error. We just want the main working thing to work. So if there's any obvious corner cases a developer can catch at the time they can test for those as well. But then we still have the QA guys doing their exploratory manual testing. We're not automated yet. What happens is, where the magic comes in is every time the QA team now find a bug, the first responsibility of the developer is to write a test for that bug. So that means that that now goes into the test pool and you've automated that failure. So over time you get more and more automated testing and the QA team have got less and less work to do because all of this gets automated. Even so it may take a little bit of encouragement to the developers to do this. I should have deleted shouting really. There wasn't much. So having automated our testing the next stage is to automate the deployment of this product that we've got that now we're confident to release it because it's passing our tests. So in our case we used Chef. Now other tools are available Salt, Ansible, Puppet, they're all good tools. In our case we tried Chef first, it worked well for us so we didn't really have a good reason to go and explore everything else. So I'll talk about Chef here but these principles apply to any of the other frameworks. So beforehand we were tossing this code the release over the wall to the operations guys they were looking at their Excel spreadsheet of configuration values looking through their Word document of what to do how to deploy it. With Chef instead that's coded up using a Ruby script and the data is stored in what's called a data bag in Chef and then you can have a data bag for QA that stores those values and their under version control. So you can actually review configuration before it gets deployed the manual scripts go away completely it's automated in Ruby if you do it well then you've got this property called idempotence of these recipes written in Ruby and what that means is if you run the same script multiple times it just ends up as the same result so ends up with you having deployed software with a configured database and so on. It doesn't change things once they've been done correctly. The other thing that's useful here is deploying to a different pool of servers. So again the cloud is a good thing you can just spin up another pool of VMs deploy the new release to that while the old release is still running you can run tests on that new release and once that's all happy it's all been deployed correctly it's passing all of the tests then you can just instruct the load balancer to now start using the new pool old pool running for a couple of days as well because then if anyone finds a critical bug in the first couple of days of deployment you just flip the switch on the load balancer and go back to the old pool that has a really really good property which means if you get an alarm call in the middle of the night to say it's a serious problem the only thing you have to know how to do is to switch the load balancer the rest of it you can come in tomorrow morning have a good night's sleep and work the problem properly knowing that the old release is still running you can fix bugs in production in the middle of the night because that is a really bad idea so that's the process and the technology naturally it's about the people so you do need to get these skills into your team so we've got a cross functional team we've got ops, developers, QA everyone working together everybody in the team needs to learn about Chef and they need to learn some Ruby not enough to be a wizard programmer but enough to get by and to be able to see what's happening in the recipe the ops guys need to be able to learn just enough coding to write some recipes and they can sit with the developers to help make that happen a lot of these things are really good opportunities for pairing and knowledge sharing so just like the testing was good for QA and developers pairing and QA and the product owner pairing this is a really good activity for the developers and the ops guys to pair up so that's the testing we've done the deployment that's more or less it except that what you need to do then is you need to keep evolving this stuff doesn't stand still even on a pilot you can keep improving and keep improving so in our case when we started off back in 2010 a long time ago now so we didn't have to so much request hardware and wait 6 months but even so at the start if we wanted a new VM we had to send in a request to the operations team and it might be a couple of days so we didn't have a machine spun up that then had to be registered with the chef's server for all of that stuff to work we had to run the chef client manually then check that it looked okay on the machine and then run QCumber and these are all manual things we were doing that is much less complicated than the previous process but it's still a manual process so naturally one of our team members wrote a command line script that automated all of this so given a pool of VMs that we could use this script would allow us to take a bunch of those and say you're the development web server you're the development app server you're the development database hook them together into a coherent deployment actually deploy stuff so this script really allowed us to automate what was taking several hours into a few minutes so finally in 2010 we got to this point where we really did achieve that vision of the four weekly deployment because the QA time was so short but we were getting a genuine four week development per four weeks but even with that command line tool there was still a major people element so I mentioned in the earlier slides that we had the scrum master, we had the product owner but we were doing agile as practised by people who've read about agile on some websites which we learned was a little bit different to really doing agile so it made a big difference to us to actually get some coaching and people who'd been there really done it and taught us how to do agile and scrum properly importantly they weren't afraid to tell people in the team that they were behaving like children that we all just had to grow up and accept some realities about software development the other thing we discovered is that within a given team not everyone necessarily works together in the same way so no one's at fault it's just different personalities work together in different ways so it's useful sometimes just to swap a few people between teams so that you actually get teams who gel and will work together what happens then is you get a team that's interested in its own continuous improvement so you're looking at new tools all of the time the spikes you might take in a given sprint you might take a week for one person to look at a new tool and see if that works out okay and also at the end of each sprint you're doing a retrospective so there you're not looking so much at things to develop next you're looking at how did we do in this last few weeks is there anything about how we did it that could be improved maybe we needed a different collaboration tool maybe we need more meetings maybe we need fewer meetings so you talk about these in the retrospective and typically pick the top two or three of those to actually do something about in the next sprint so you're actually improving how you work as a team at the people level as well as looking at the tools and the technology so we kept improving and the next year we've produced this tool called Barkeeper so in server a command line tool we've now got a web interface we're not having to use a pool of virtual machines anymore we're actually talking directly to the vSphere API and critically we're running this out now not just for one team we're expanding this out to the whole department that we're in over multiple teams and multiple projects so we actually introduced a concept of project self-service so instead of having just one tool running our project for us this web-based interface allows someone to create a new project and then they can define a landscape set up the automation and have their own version of the ID service or whatever project it is they're working on so this DevOps platform starts to scale throughout the department so by now we've reduced our cycle time to two weeks instead of four so we had to schedule deployment every two weeks but we had so much confidence from the automated testing and the automated deployment that even within that two week cycle deploying two or three times a week was a relatively common occurrence sometimes it would just be we'd find a minor bug with an easy fix and we'd roll that fix out it passes all the tests and we can deploy it within a few minutes so it really gave us a lot more power a lot more agility in the non-capital A world to respond to business requirements of course with every tool we need to look at the people impact as well so in spreading this out to the whole department we had to talk to different IT teams this agile coaching we had to make sure that that program of coaching was rolled out across all of those other teams and we started to dissolve these silos so having ops people embedded in the development teams became the normal thing to do so we also encouraged regular team learning days as well so people would have web sessions where if they learned anything particularly cool on their project or anyone else so everyone piles in and we all learn more from each other and it keeps on improving in fact we improved so much that this got noticed throughout the whole company so this is a company of 70-80,000 people about half of them are developers and our little team now we've got responsibility to create this cloud and this DevOps platform for the entire company so that took us to 2013 so we started another new project this time called Monsoon so this is now our cloud and DevOps platform it's custom developed we used microservices, Ruby on Rails the infrastructure as a service layer is roughly equivalent to OpenStack at that point we were aware of OpenStack at the time but it was never quite mature enough for what we needed at the time that we were doing it so we ended up building something that was a lot like OpenStack that just in our case talked to VMware and we used Chef as before and the end collective is a tool to treat a bunch of machines the same way so in other words with end collective and Chef we can run Chef on many systems at once just with a single click we used a tool within Ruby on Rails called VCR a videotape recorder to help with microservices testing so what that allows you to do is to do some testing manually to build the HTTP traffic from your component to the other components so that then if you want to automate that test in the future you know that if you talk to another component with X and it returns Y it should do that in the future when VCR does it sets itself up as a bunch of proxies around your microservice so you don't actually need the other services to exist once you've done the recording just when it gets X it returns Y and that allows you to effectively unit an integration test without having to have the others present so yeah by then as I say we're using VMware we're using F5 load balancers we've got this two week development cycle but by now we've actually got continuous integration and testing multiple times per day and even though the development cycle is two weeks we actually have every single morning someone gets up a little bit early and physically deploys things by pressing the deploy button so every single day we're deploying code to production of course once again there's a people impact to this technology so this time we've grown a bit so this DevOps platform team is now 15 to 20 people two or three people working dedicated on the infrastructure side of it interestingly what you'll see there now is for all of this we're down to one part time person for QA so that tells you how much effort we've saved compared to where we were a few years ago we're adopting chat ops now so using IRC quite commonly for people to talk to each other to keep real time information flow between the whole team because of where we are now restricted we're just down to UK and Germany for our international operations but we're still working essentially in remote mode so even within Germany some people are working at home some people working in the office so we're keeping this idea of a distributed virtual team in roughly the same time zone having said that another really important people lesson here is distributed working works very well but over a period of months the human relationships start to decay I mean if you've seen trolling on the internet that's what happens when you get people who've never met talking to each other they insult each other without consequence now we never had anything remotely like that but it illustrates the point that people have to meet face to face from time to time when you meet face to face you get a new level of trust and new level of efficiency and that can last for several months and eventually decays a bit and you need to get people together just every three to six months you can have a workshop that's helpful but really the important thing is just people sit down and have a meal together and talk as human beings so what that gave us as a result is towards the end of last year we've spread this cloud and DevOps culture throughout the whole company in SAP we've got hundreds of applications running a large part of SAP's cloud portfolio is running on this system that we've developed we've got thousands of developers running it 10,000s of virtual machines 10,000s of storage volumes we're in six regions and 12 availability zones so that's all looking pretty cool of course at this point you're asking what's next and more importantly what to do with OpenStack so slightly before we get to the OpenStack part we discovered Kubernetes so a lot of the properties you want of cloud software and containerisation come with Kubernetes so you've got this automatic scheduling of containers across a cluster you've got great self-healing properties you can orchestrate rollouts I'm not going to go through the laundry list there but it gives you lots of cool stuff that you really want from a kind of DevOps system based on containers the other thing we have is this is one of the earlier slides this sentence here we have a custom developed cloud automation system with an IS layer roughly equivalent to OpenStack so by 2016 OpenStack has really caught up we're sitting on a pile of custom developed code that we're having to keep developing and maintaining that's looking increasingly like what OpenStack does so what we decided to do is changing a throwaway the old platform we're still running it at the moment as a legacy platform but what we're building at the moment is a containerised version of OpenStack running on Kubernetes so all of the OpenStack services are running as individual Kubernetes services in containers that means we can do constant releases there's very little operations overhead the core OS background to Kubernetes self updates so it works very well for us critical point is in how we've organised our OpenStack system so over on the right hand side there that's our OpenStack on the left hand side is the things that OpenStack is managing so networking, storage compute resources and the critical thing here is that we've taken out the actual networking part we don't use the inbuilt parts of Neutron for doing networking within OpenStack instead we use Neutron to control devices from other vendors most typically Cisco there's a talk later on in the conference from SAP and Cisco that goes into more detail how we've done that but the important property of that is if you take all of your cloud out of OpenStack and leave OpenStack in place managing your cloud then if that blue box goes down every single customer virtual machine keeps on running so SAP's cloud portfolio can keep running our customers can keep using their software even if OpenStack goes down that makes it really really easy to think about upgrading OpenStack because we can actually take down the cluster we don't even necessarily need to we can just take down containers and redeploy them and only suffer a few seconds of outage on the blue part where we can take down the blue side and the only thing that stops anyone doing is putting new virtual machines in place or using OpenStack to do stuff but any existing software keeps running also we've got analytics and billing on the platform so we've got Prometheus there sending monitoring events out we've got Sentry for capturing crash data and feeding that back to developers and getting user feedback the ELK stack elastic search LogStash and Kibana are logging and logshipping and searching of it then we have Manasca taking care of the alerting Silometer is busy there collecting all of the events and metrics for billing and we send all of that data from Silometer through to SAP's HANA in memory database where it can be analysed and can do all of the billing and look at what we need to do to improve the platform from the analysis so that's just bare OpenStack what we're also adding into the mix is a couple of tools that were developed called Lyra and Arc so Arc is an agent that sits on the actual virtual machines themselves and can run tasks a little bit like M Collective and Lyra is a high level service that runs alongside other OpenStack services and allows us to schedule automations so basically we're adding this kind of automation and DevOps module into OpenStack where we can use Lyra to say run a bunch of chef recipes on these machines run a bunch of shell scripts on the other machines run some ansible scripts on some other machines so we're taking these properties that we had on our other platform that was independent of the infrastructure layer and bringing them into OpenStack so these are under development at the moment the plan is to be open sourcing them probably during this year and the other thing we've introduced on top is a new dashboard for OpenStack called Electra so Horizon as a dashboard is okay but it's focused really at very technical administrator types what we're trying to do with Electra is get across what I mentioned earlier that we had with this project self service making this something that's much more consumable by a typical developer to be able to set up a project create a bunch of virtual machines and run automations across those machines and we've got the first few pilot projects using this at the moment our new DevOps platform so, yeah, this is essentially what we've built so now we've got container based OpenStack we're running that across 13 regions tens of data centres, xx is the secrets at the moment exactly how many there are we have this control plane which is controlling this is the OpenStack part of it control plane we can get that from a bare metal in a data centre to a scaled OpenStack cluster in a matter of hours we're applying DevOps to building our DevOps platform so we can actually get a new data centre up and running in hours with automated deployment using all of these same tools the control plane controls our data plane so we've got a choice of VMware or KVM at the moment we're looking at adding other cloud providers as well we can deploy bare metal machines for SAP's HANA database we can deploy virtual machines containers as a service is coming shortly as well so this then becomes the platform that SAP uses for delivering its cloud portfolio and also for supporting internal innovation so if someone wants to create a new project or a new product they can use the self-service features here to quickly spin up a few machines try things out, close them down again and they're costing a tiny amount of money to their cost centre we've got all of the monitoring logging, alerting and billing that I showed on the previous slide automation engine we've got the dashboard so we've got this really cool platform which is the foundation of our future success as a software company and process wise we're building this on a four week review cycle so I'm not calling it a sprint anymore we just get together every four weeks to check point and see what we've done but we're really continuously bringing in new features and deploying them we're working more of a Kanban mode of just looking at the number of tickets on the board so on the other side we pull another one in from the left Importantly in all of the previous stuff we were doing almost everything we were doing was our own code, lived within our own walls on the new platform 90% of the code we're producing is open source the 10% that's not open source there's no point open sourcing it's about things like how we do the billing to an SAP cost centre that's not of interest to anyone in the outside world so the SAP specific configuration of SAP specific processes is the 10% that we're developing in-house 90% of what we do is out there on github and yeah you can see the projects that we're contributing to we're contributing to OpenStack itself we're putting stuff into Kubernetes into Docker, Grafana also in several of the consumer libraries so OpenStack4j and Fog which is the Ruby library we're major contributors in those spaces of course it is about the people and this is where we're at now compared to the previous monsoon project it's not a massive change it's maybe 30-40% more people still only half a person doing QA we do have more complexity as we grow this platform as we introduce it to more and more people across the company so instead of having one person on operations or night call duty we now have three sub teams at any point in time therefore we've got two people in each sub team that makes people on call for anything that might go wrong but typically they're handling tickets during working hours there's not really any out of hours cause that go to them we do have a global distributed first level support team that takes care of any out of hours stuff and we've been learning a lot from how the open source community does things actually forcing ourselves to put everything on github and work with the OpenStack community is changing how we develop code in many ways for the better so the fact is in a code that may have been good enough before now when we put it out of the community goes through a review process and it may be a bit annoying it may take a week or two to get a patch set through but by the time that's done it comes back a much better code than we would have written in the first place so the quality of the platform is better for being open source than it would have written it internally highlighted in yellow there you can see github.com slash SAP CC cloud that's the project in github putting all of our open source components so not everything there is necessarily current or finished or rosy but several of our bits are being worked on live there other components will be published there throughout the year so that's the place to keep an eye on for some of the tools like Electra, Lyra and Arc that you can add to an existing OpenStack deployment to give it DevOps capability so that's it thank you reminder it's all about the people there's some really cool tools and technologies it's really important not to be distracted by that they help make it work but it cannot work unless you really focus on your people on your teams getting people to work together work effectively treating each other as human beings eating meals together and improving as a team so thank you any questions I see what he's saying so at that point they cease to become elastic for the part where the control plane is down but the point is they don't completely disappear they're not offline it's a very good point and it's an excellent point no more questions thank you very much and enjoy the rest of the summit