 My name is Roger Meyer At Siemens living in Switzerland Did some computer system engineering in the past like embedded computer hacking and so on and with him my department We started with githlap Hey all Fabio Hoeser Software architect at Siemens mostly doing frontend stuff and you know as a side hobby maintaining the githlap platform at Siemens so We heard a lot about githlap how it's used in big companies and really cool scenarios today already So we kind of thought hey How can we bring the idea and kind of the DevOps culture as we rolled it out with and Siemens a bit better to you So we decided let's pack that into a story who is familiar with Gauls law in this room Some people yeah, that's lovely Gauls law quiet old but for us it's still kind of through so you'll see it here Basically to summarize a complex system will never work You always have to start small and kind of scale it up from there and today We're going to show you why this is The truth for so many things and hopefully you can learn one or the other thing from there as well cool, so just one Slide about Siemens how we started out in the past Siemens wasn't like this multi Million-dollar company since the beginning we started actually are quite small so there are two guys We're not from Siemens. That's kind of where the name is coming from You know in a backyard machine shop, so it was pretty much a early-day German startup already and the other funny thing is the first patent kind of shows you that seems all about innovation and you know researchy thing was actually Filed in the jail, so Vernon from Siemens Invented the whole electro government galvanization in jail and it kind of start from there So small small startup, but you see 1847 so quite a while back And we've built electrical cars before it has But they haven't been that successful during that time Exactly so you see small company and I would say we had more or less success So Siemens still existing nowadays, but of course it's a bit more complex at this point So you see a ton of stuff what we are building. So for example the tube here was also Built by by Siemens at least some of them and the other really nice thing you see here Those are in fact silos. So Siemens exist as so many different smaller companies As anyone we really love to not talk to each other So a ton of silos over there and you know, it's mainly focused on business to business So it's really hard to kind of grasp the connection to the customers You know all this big company mumbo jumbo as I think most people here are familiar with that kind of thing So you see also in terms of number and two people that was kind of cool now It's a 380,000 people so you actually see the scale just got much bigger and the whole complexity You know kind of really showed himself. So that's kind of the you know Outcoming situation we have and you know for us it was kind of the challenge How do we build a DevOps culture around this really fractured federalistic kind of company structure? Yeah, you see also many different cultures are involved 190 countries so also in a like human to human interaction level the challenges are certainly there and you know If you're interested we have also couple open shops there Cool so That's kind of the major part of the story we're going to tell you today and we try to answer the question What is this code? So this kind of code is our lovely baby code Siemens come That's our adaption of github, which we're using within Siemens and we really try to you know brand this idea So it's not only github yet another instance. It's really a community It's a brand which is behind of that. So now you could quite easily say hey We take a github we add a little bit of Siemens sauce and we have code Siemens come but Roger kind of showed you how this whole off story started Yeah at the early beginning as you see in 2013 we got In love with github the UI looked a little bit different. We had that nice fox logo And if you look at the application architecture, there were just three little pieces So the cool thing was it was a MIT license. So as we are addicted to open source software especially within from the embedded system development and so on I'm my origin in and Developer self-service. So this was one of the of the major topics for us So that people we don't need an administrator creating kind of users admins and so on and creating repos So having the people we're being able to just use it in 2013 Yeah, you can imagine we have probably all the version controls that exists on the planet in use within the company and of course then also all the build tools and We have co-paces with over 20 years of history. So a Lot of those. Yeah, and there was no company white source code hosting during that time So then we started Preparing that in-house for us BT GitLab at the early days. It was called the plan was to have it available for about a hundred people so for the embedded CPU module platform users and so on and We we are heavily using internal social network networks during that time. This was Socialcast and I was suggesting hey, of course, it is nice. I have a solution for my team But guys we need a common platform code Siemens come my colleague just registered the name and a few months later. I Told my boss I made of our 100 people GitLab instance code Siemens come and we will have other people joining us on the platform and They were okay So you see already here, it's quite an entrepreneurial kind of spirit, you know, which which we try to to go forward I mean it was a hefty discussion You can imagine you have like a small server and everyone is kind of honky-dory and suddenly of like people you don't even know on the Platform playing around and of course the computer cost is gonna rise and rise and rise But even back then we kind of thought hey this are you ideal logistic kind of idea? We had and the route you want to take is the right thing to do and all the details will figure out later on the way One important topic. Yeah, this was and during the early days. I made that had we didn't had the platform up and running But later on as we had the platform up and running just took a picture put that as a logo and Then one of our community members created that fancy logo so people are identifying themselves So we have now stickers people putting that on their notebooks and so on We were focusing on Not we did not focus on complainers. So this was an important topic for us So we had a lot of people that were addicted to on the way of working with kid Hey, we need kid. We don't like use that version control and so on and we focused on the people that were on fire with us for that common Coding platform and so we've been we've established an ambassador circle. So we have monthly calls and all those people the power users Contributing to our documentations and so on And of course now a day People do really identify themselves also with the platform. So if you meet Siemens developer Ask him. Do you know code Siemens come? I'm pretty sure he will know it. Yeah I mean the platform is sadly just the in-house thing, but I would say that logo is quite cool I mean if you're a developer, you kind of can relate to code and we brought some stickers with us So if you're interested in code and stickers, feel free to reach out to us At the very early beginning as we are all developers To make this platform a success and to bring this out to the whole company. It was key for us to have a Great team that is doing this and So at the very early beginning we defined the team charter So we really as we have enough silos within the company already the goal was to really have Cross silo thinker people To have people with a strong open-source mindsets that are that have worked within communities as maintainers as Contributors and so on so to bring in that Colorative way of working into the company So it's not like an administrative crew that is looking at a green lamp So we we have a team of eight people that is doing development and operations all the things For that whole platform. Yeah, and also the vision. It's quite all but still valid Colaborate on code and share within minutes You can imagine in the past. There were a lot of if you had two teams have to work together Which kind of version control to use which servers and setting all those things up took a long long time in the past Nowadays, it's just code teams come create group at your members bank. I Mean, you know the stuff which Russia just showed you kind of sounds benign, right? I mean, it's quite clear Everyone needs to have a vision everyone needs to have girls But we've seen it in the past. It's you kind of get really stuck with technology, right? I mean, there are so many great tools out there the open source world You know comes up with new tools every week about that the end of a day You know, we really try to solve it like a human issue We want to collaborate and the tool is just a secondary thing after all and you know all those things to give up We really found a tool which facilitates this ideology. It's all about the people behind it And you know to kind of maintain this idea and also have this community spirit within Siemens You really need to establish such community You need to find the right people in the open source world. You would say hey We want to have all maintainer personality Within Siemens kind of the same story So we're really always on the hunt for those maintainer people which really also can lead this cultural change Cool. So now we heard a lot about vision strategy kind of this matter thing and it's really important to kind of start from there So I think that was around 2015 or so where we had the platform. We were already in the thousands of user area You will see some current numbers a bit later in the presentation But this talk is all about the DevOps CICD It's not about what the Siemens folks have, you know in terms of vision already. So 2015 was kind of the day where we started with CICD We already heard a lot about You know see our runner in the restricted environment and the company and normally It's a thing that you have to go to your local IT department. You order a server You have to wait a couple of months and for us. This is not really, you know Inline with DevOps and CICD culture. So we kind of had the urge to change this However, we have not the capabilities to change it and of course you could nowadays Make beautiful fancy terraform setups or play around with Ansible and AWS and in the early days This wasn't possible. We also didn't really had the monetary fund to you know with four people Offer CI capabilities for a whole company with 300,000 people So that's why we kind of said hey, let's start small like as all things should be and you actually see it there That's not just Photograph we downloaded from the internet. No those were indeed the CI server, which means kind of you know Backed our whole platform for more than 80 percent over. I think more than a year We actually came up with a name for this kind of pattern and you know at least we're here from Switzerland this kind of You know saying already spread itself. It's junkyard computing Thanks to the ease of use of the key lab runner You can set up new machines in a matter of minutes if you have old machines laying around and you have a Good enough set up in terms of network and everything you can Literally on a minute base set up in your runners new capabilities and it's quiet cost effective So that was the idea where we really started out had those old runners, but people loved it They joined our platform. They could build Free CI stuff without maintaining their own server. So it was all there It was was already based on Docker So people could actually really bring their own Docker image and it this was something which you know never exists That's even some people literally about writing us an email like hey, is this possible? I don't have to pay anything It's like yes sure. I mean you're working for this company and that's That's benign as it sound and nowadays it's like usual thing But back in the past it was a huge paradigm change and you know it kind of sells itself, right? You get sia for free. You get a enormously stable service You get a state-of-the-art Docker image all based on you the problem So this really took off and I think not sure how many builds do we have per month? 1.5 million Per month so it's about you said 44k per day. Yeah So during our talk they're building a lot And and today We have our infrastructure on AWS So there's no junker compute anymore So this was really at the early day to give the whole topic a strong push and and to increase the adoption across the company Yeah, and that's really something we can recommend you a ton So if you have a github installation somewhere in your like a company and you know want to advertise that Having something like shared CRN There is a really really awesome way to argue with your department head or with the Definements department because you suddenly see how well everything works and how much the saving can be You don't have like your part-time administrators anymore, which you know after our Maintain a server and have to build it to the company. You have a really nice Centralized and cool maintained github runners set up. So I would say we only do doctor bills on our on our shared runners and We went through all the Linux distros We went through a lot of kernel box and stuff as we have a quite a high load on those things So now at the moment we are with the nixOS and for the runners Exactly So this was indeed a huge milestone for us and the really sad part of the whole story a lot of part-time Jenkins administrator They you know they have had to take care of more time to do important things So Support is always a really important question, right? I mean now we have this big platform everyone uses it I think it's kind of time to hire tons of tons of telephone supporters But that's not something we really want right good lab is a tool for developers and if you're familiar with the github community if you Already created an issue in the github issue tracker and to be honest I can highly recommend you to interact with the community. You noticed they have top-notch people So if you have an issue and you ask them they can really help you and that's kind of something We also want to do I mean roger is a developer I myself mainly or a developer and we also want to create a platform and a community which helps other developers So why should we hire support folks which didn't have the capabilities and like the rights to change anything? So we kind of said hey Let's create a community which educates themselves So that's kind of a help yourself ethos. We try to establish and in the same role mass github You know we started out with a documentation portal sounds easy and thanks to github pages. It is indeed really easy But getting the community Being part of that is a bit harder. So you also see the phone. That's a real thing. That's our support phone It has no connection at all. So so far we could probably say with zero support requests Right and you know, there are many many other ways how you can solve this whole support topic You know without having huge support stuff. I mean In our case, we're still eight people maintaining and supporting all the people Of course, we have the whole community supporting the people as well. So that's really lovely to see So people have for example if the if there is a question on the in on the internet social network Usually they get an answer within half an hour From other developers or from core team members and so on. So the people is really Greatly engaged And we receive also merge requests for our dog's page and our road map is fully transparent for all the people So all the people can see what we're working on what our next step. So we're discussing and you can imagine the time You have those it tickets I want to have this this And for us this was kind of a super important thing to don't have those Classical help desk topic and it's not only about the support So, you know, we we didn't just show you that this slide to say hey, you know Look at us. We don't have any support work to do is something even more important because having Not a help desk but really give power to the community enabled us to have this DevOps culture within Siemens So, you know rather than support ticket and you get a response like oh, you should do it like this You really have the folks teaching each other a I have this maven spring boot script, you know to automate this with gilab See I and then we'll do some automated security tests Stuff like that is really part of our documentation also part of our culture. So, you know people really educate each other We more or less have know how bubbles, you know We have the maven bubble and you have the go guy bubble and you know people can really go into those communities and you know Just exchange with each other and that helped us rolling out this Tremendous CI movement which happened at Siemens. I mean in the first half year We maybe had like 10,000 builds per month and as Roger just said we have more than 40 K Builds per day at this point. So without having this distributed knowledge how this would not be possible So it's not only about giving the people the right tool and giving them access It's also about the mindset and the collaboration of the culture And having a team that is able to handle also all those questions from experts. I guess that's that's Another topic super important to really have people that are able to answer all the question and giving direction on the issue tracker. No Being able to say no Yeah, so now this is kind of the inner circle and how we collaborate with and Siemens But I think more important is it going to be how do we collaborate with the bigger circle meaning the whole github community? And I think Roger can say some words upstream first so this is I've learned a lot also during the Linux Linux and a bootload of kernel hacking and all those things Usually you don't like to have patches and maintain patches and port them to the next version and so on and for us It was absolutely clear. We go without patches. We only deploy upstream Nothing else if you want to have new features we contribute them upstream We do not patch our instance As soon as they're merged upstream we're gonna deploy the next version so we shape every month We do about for production deployments per month Of course also configuration change security patch and so on and This was one of the main reasons why we why we've started using githl app It's open source and you can imagine as a company's like Siemens you have special needs and That's why we want to be able to Make that happen if you have a Special configuration we want to have in place or we did GPG code signing everything githl app the open ID connect The provider Bunch of security features like to FA enablement per group and and Our session list you have within your user profile and many many many other things and this is um Yeah, so We're also a little bit proud that we are one of the the biggest Contributor to githl app itself. So we're not only using Open source software and we're heavily contributing to several pieces also maintaining add-on libraries like the Python githl app maintained by max and So those are the things we're doing with the team of these a people and with Diego working on upstream topics Primarily for us on githl app, but maybe already disclaimer here, right? So we filtered out githl app on purpose because Of course the biggest contributor and there are a ton of I would even consider them githl app heroes You know like private people working on githl app like crazy as well But it's still cool and you see many many companies there and the list is by far not finished There's so many con companies contributing to githl app And you know, it's not only about contributing to the code base itself for us. It's also the cultural aspect It's exchanging yourself with the githl app folks with the community around githl app because there's so many things we can of course learn as Well, and you know, we will love this every day to kind of get new ideas how to do CI Which we can then you know take into our culture and adapt it within Siemens again So this whole exchange with the broader community and with githl app itself It's so valuable and you know, we can also highly recommend you to try contribute as githl app Famously says everything starts with a merge request. It's actually really easy. You don't have to be a programming hero They're really welcoming and I can just highly recommend it and as Roger said for us It was a huge cost saving as well to not maintain local patches or having to talk to 30,000 people and saying them like oh, no, sorry, we cannot accept your patch But you know this patch we want to have that would have been a possible with the team size we have so Ops room first is certainly the way to go and we would highly recommend it to you as well. Yeah Save money and time and get rid of that maintenance in the nightmare you will face Another topic as we've mentioned call slow Start with a simple system and we started a simple system and at the early beginning We just had one server on Bare metal later on we moved to AWS Now we have we had one server until 20,000 people so we reached the end of that drop-down so to increase the machine size and For us it was The most important topic was reliability of the service as we are developers as well so we were heavily focusing on Availability reliability on security crash reporting lock metrics and so on so all the non-functional requirements as you can imagine in the business Businesses we are at Siemens Those things are running for 20 years Super-reliable management systems for building or airports or whatever thing and and so the quality aspects are Super important for us and of course for all the users. So those were our primary focus and then We already had the plan to split the things out and as we know the github architecture quite well as we are Contributing to several parts on github itself, of course We had to plan ready to scale up nowadays we have about two github is For front-end nodes three sidekicks one pages and a gatekeeper For the internet access so we provide several service also via the internet so we can have contractors and so on on board and of course Kubernetes is nice and We might do this one day But at the end your our customers our developers they don't care about the fence technology We are going to use they want to have a reliable service that is running all the time and I guess in many cases you see people or projects building so sophisticated super nice fantastic architectures for what Because they are addicted to it to technology, but don't forget about your customer and the people that what are their expectations and Yeah, we like to use Kubernetes in the future, but We are focusing on happy developers. That's the most important thing And the people they know us they know the people within the core team So we are visible we are persons are behind the service. So if we if you break things We do incident incident Blockposts internally tell the people hey, what what happens? What was wrong? We're talking to the people and explaining The good things as well, of course Oh Right, so, you know, this is pretty much yet another story why goals law in our case worked out Just beautifully and we certainly also want to show you where we are now where github is now So you see github in? 2019 if you can remember on the previous slide you had this really nice architecture diagram of github in the 2013's that's the current github architecture diagrams. So I will consider it a little bit more complex So also they heavily moved into a more complex area because you know you have to if you want to have more feature and more speed The company itself grew as well And on Siemens on the other hand you see the service is established So, you know if you say hey, where's your code the answer is always it's a code Siemens come It's kind of the place to be because people like it there. We don't have to force them It's just a thing Everyone can contribute so you know we have just one big instance for the whole company people can make the repository internally That's all up to the each individual person and the product of course and you know if you're part of Siemens You have this huge You know like of different repositories who can collaborate with and you know We really try to bring the open source culture in and so far we really succeeded and then of course like I said CICD with one and a half million builds every month I would really say yes the whole culture thing also kind of changed and you know It's now completely normal if you create a project and code Siemens come you also have CI enabled because again, it's free It's easy And welcome good lab to the club of complex application architectures So we have tons of them So just maybe a rundown about the numbers so code Siemens come what's the scale you see it's 32,000 people and I think we are already on 33 more or less now. So, you know, even though We're already on a really high number. It still grows if I would be you know on the stock market I would certainly invest into this growing curve There is no end in sight and the funny thing is Siemens is around 20,000 software developers. So we're already kind of beat this market and Now have you know people from finance on there because they really love the issue board and the Kanban, you know board feature So it's not only developers which love the tool. It's also all the folks around You see those people actually do work on code Siemens come so they have not just registered They also have a ton of projects. So every person around two projects and you know for us a really important metric good lab is all about collaboration and Just having multiple projects. That's not really an indication for anything at all for us Also, the nodes are important. So every, you know comment you write in an issue or in a merge request It's counted as a note. So if more than three million nodes people actually indeed collaborate with each other I mean at least I myself don't write comments just to myself. So that's kind of an indication that people really talk to each other Then of course, you know, this year builds now. We're actually this week. We beat just 17 million I said it still goes rapidly and you know, we see that the exponential curve still goes on and As mentioned a lot of different countries. So the culture on code Siemens come the whole exchange is certainly really interesting and Beneath all of that you have eight people keeping this thing running more or less on the side. So that's quite a challenge Beautifully so Roger some famous last words It's all about people I would say that's That's the most important topic. It's not just Picking some people together and say hey, that's your goal. Go ahead. So having them being Have the technical skills and the people skills To drive the direction to to bring that adoption into the company and of course being transparent in all the things you're doing so Since the beginning we were talking about all our ideas and stuff and and have our roadmap visible for all the people Within the company. So it's not like We're cool. You have to walk the talk That's Key of course focus on your customers for developers from developers and that's why it's super important to have Developers within such a team. We are able to answer any question from the Colonel hacker To the Java spring boot guy to the Node.js guy and so on. We are able to handle this That's why we need no help desk. That's why we have documentation and so on and of course give and take it's Contributing stuff to open source projects to in-house projects or to the documentation and so on Living that spirit Within the company so the people see out there not just within their little island Here with the team We are collaborating with all the people if you see for a conference for example having a There the speaker The conference schedule and website is also in code teams come I see an issue there. I Create a merge request since in the past people did send emails, but the living that spirit of collaboration and In in the in-house world, but also in the open source world. That's that's the thing exactly So marks in the Roger will in the upcoming live session show you how this really looks like behind the scenes How one of the Siemens DevOps project actually do look like We are certainly available afterwards in the break if you have any question and with this kind of guideline Roger just showed you can by yourself kind of do the same thing as we we at least hope so thank you very much and See you later. Thank you