 Are we ready to go? Thumbs up. Okay, fantastic. Hi guys, how you doing? Everybody with me here? Okay, it's 4.30 on Wednesday afternoon. I know you're all extremely tired, thirsty. I know my place. But we're going to have a lot of fun. So come with me for the next 30 minutes and we'll have a lot of fun. So today, we're going to be talking about your CICD infrastructure as a monolith too. So we're going to have a lot of fun talking about this. It's going to be very interactive. So my name is Lachlan Evenson and if you've ever seen me talk before, I love dialogue. So please interject. I want to be here to make this interactive. I want to help everybody on their journey. So this is supposed to be something that's good fun. I hope you learn something and shoot your best questions at me. Okay? We don't have to wait till the end. You with me? Fantastic, fantastic. Before I get started today, I just want to say a massive thank you to my team. So I wouldn't be on this stage showing you what I'm about to show you without my team. So I want to call those guys out by name. I like a little bit of giving thanks to the people that have helped elevate me into this position. So Joseph Sandoval, Mike Tujeron, Mike Leong and Mark Ward are my team and the stuff you're about to see will blow your mind. And those guys are behind this. So I just want to give them a little thank you. Okay. Our philosophy. Everything as code. Let that sink in a minute. This is our aspiration to get to this. We're not there yet, but we want to shoot towards this. So keep this end goal in mind as this session goes through. What does everything as code mean? To us, it means all of our infrastructure is codified, versioned, repeatable, composable. Does anybody else want this dream? Okay, you're kind of the right place. Gonna have some fun with this. Okay. Jenkins is an app. I actually want to go to a demo right now. We're going to mix it up. And I want to show you our CI CD infrastructure and how we actually deploy it. You guys ready for this? Let's go into a demo. So I'm going to keep it nice and raw in real time. I like demos. I hope you all can come with me. So I'm just going to show you some Kubernetes infrastructure that we have. And I'm going to deploy Jenkins on top of it very quickly. So here we have an environment. I'm just going to show you what we've got right now. We have an environment called Jenkins. There's nothing in it. Let's go create something. I'm going to create a Jenkins master. Okay, I have a Jenkins master. Okay, so let's just have a look at what we've got here. Jenkins master running. Oh, wow. Gee, that was quick. Okay, so let's actually just go and take a look at it so you can have some proof here. So here's the cluster IP. Let's go and pick that up. Let's throw that into a browser like we always do. Jenkins is starting. Oh, not quite fast enough. But while we're waiting here, let's take a look at what's going on inside Jenkins. We can see it starting. So take a look. So I'm going to grab the logs. Logs. And I want the container Jenkins. Oh, Jenkins is fully up and running. I'm way too slow, obviously. I have a Jenkins instance. Okay, so I've just told Kubernetes to give me a Jenkins instance. That's not the fun bit. The fun bit really is when it comes to actually spinning up slaves. And this is where horizontally scaled infrastructure really showing. So let's take a look at what we've got one more time before we go into this. Okay, so I've got a Jenkins master. Let me go ahead and create a Jenkins client. Okay, so I've created a Jenkins client. Woo. Okay, so Jenkins client zero of one running. We should actually be able to come over here. And if you know Jenkins at all, if anybody's ever used Jenkins, you should see down the bottom there on the right. See that guide that build executed just pop up there? I just created a slave on the bottom left. Okay, but I don't want one slave. I want three. So let's go and create three. So let me just scale. Jenkins client replicas equals three because I have one replica. Okay, I can't do that because I need to tell it I want to scale a replication controller. So again, I've scaled it to three. What you should see is these Jenkins client come online. Okay, so all I'm illustrating is that with this framework, I can actually I've defined Jenkins as code, and I can deploy Jenkins quickly and easily. Okay, so this is the foundation for the rest of this talk. So how many people have deployed Jenkins that quickly before? Good. Good. This is a great basis. And when you start with this, you actually have horizontal auto scaling as well. We can, based on load, spin more of these or less of these up. So I just wanted to get in with the demo. You guys with me on this? Okay, fantastic. So the big surprise here is that Jenkins is just an app as well. So why not treat Jenkins like an app? Alex Paul V said it yesterday. OpenStack is a series of apps. So let's treat them like that. So what have I created actually on one slide too far? What have we done here? We've just devopsed the heck out of Jenkins. We've made Jenkins on tap. But why would we want to do that? And we've just created Jenkins as a containerized microservice. So now we have infrastructure at our fingertips to be able to create applications and deploy them standardly. Pipeline is code. Okay, so let's take this up a notch. So what I'm going to put into Jenkins right now is pipeline as code. And what is pipeline as code? It's just another microservice. So let's make our pipeline a microservice, version it, build, test, deploy. We want to make this really lightweight. The state of our Jenkins build infrastructure. So hopefully you can all sympathize with this. Okay, your first day in a job, where's the build server? Nobody knows. The guy who built it left. Nobody knows how it's built. All I know is you can't touch it. Or the business will fall flat on its face for months as we rebuild it. Okay, you find the Jenkins server, right? It's a battle tank. It can never go down. Because if it does go down, business stops. So you've put all this infrastructure, you've made it the biggest thing in the world. You've built moats around it. You've put sticks on top of it. It is something that you never want to go down. Okay, you found it. You see that it's a tank. What else? It is the combination of thousands of clicks that nobody can figure out. Right? So there's so much knowledge that's baked into the Jenkins infrastructure that you cannot extract. What's the value of that? Does anybody sympathize with this? Okay, great, great. So I want to propose that we move away from the click generation. We don't want to click things. We want to automate things. If you're at OpenStack Summit, you like automation. Or I hope you do. I hope you don't like clicks because you're in for a rude awakening if you do. I heard something that scared me yesterday. We need service discovery for our pipelines in the build system to figure out how things are built. Is that not scare people? What are we making here? What are we preaching to the application developers? What do we tell on them? Keep it simple, repeatable, composable, don't make these massive monoliths that nobody can untangle or want to work with. And anyway, when you go to run jobs, they get stuck. There's not enough resources. A build's hung on one teams. Another team's job doesn't get built. Nobody can figure out how to unwind it. So, you know, I think this is common. What I'm proposing here today is that we try and leave that and go to pipeline as code. And I'm going to show you how we did that. Any questions at this point? No? Cool. Yeah, come on, man. I'm going to give it all. What do you think I'm just going to say you can look at all this, but you can't have it? I'll show you all that. So, this is what we were shooting for. The microservice experience. Has anybody heard of what they make people do at the first day at Airbnb? You have to deploy the running production code to update your profile as an employee, right? So, this is what we're shooting for. Day one experience as an application developer. You walk in, you get your coffee, you say hello to your new team, you sit down, you deploy an app, and you have an end-to-end pipeline and you start iterating. You're not worried about where it's running. You're worried about writing your app. I think this is a nice dream to have. This is what we're shooting for here. Okay. So, you want to go through that with me? Do you want me to be the guinea pig here? Do you want me to show you how we're doing that? Yeah? Can I get a yeah? Okay, come on, come on. Got to keep me awake up here, too. Okay, so let's go ahead and do that. So, I want to show you the metadata and how everything actually hangs together, but I just want to walk through the experience right now with you guys. So, you know, here is the Jenkins server I spun up a little bit earlier before my demo. So, let me see if I can actually increase the front. So, this is a Jenkins server that's running that I spun up the same way as I spun up the Jenkins server five minutes ago. Now, it has this little hello demo, right? Anybody heard of the hello world app? No? Okay, great. So, I just want to walk you through an example here. So, this is what I bring to the plate as a developer. The GitHub. Nothing else. All the metadata for the build is inside that GitHub. I'm just telling Jenkins where the repo is. We'll go and dig into that in a minute, but let's just stay with that now, okay? So, configuring, rebuilding my branch, I just need a Jenkins server, spin up another one, and point it at my repo. My build information's in the repo. I can move this to another server. I can do whatever I like with it. Everything's self-contained. I can version the build pipeline. Okay. Okay. So, here we have my app. Okay? You can see I have a dev branch and a master branch. Let's go change some code. Does anybody know Java here? Need some help. Okay, so let's say I want to, so this is just an environment here. I have the repo checked out. I'm in the source code, right? So, this is my little app. It's just a web server that says hello world. Let's say I want to version this. Let's say I want to roll out, I'm going to do something basic and make it print 39.2. Okay? And I'm going to edit my POM XML because I don't have Maven Jenkins modifying that. So, all I've done right now is updated my app. So, all I'm going to do is push that into the repo right now. Let's just say bumped version. Never put that as your commit message. Disclaimer. Okay. So, let's go back. All I've done is update my app, presumably put something a little bit more meaningful than that, but for purposes of this demo. So, what we should see in a minute is this get picked up by the Jenkins master. And the Jenkins master will then go and build it. It's on a timer right now. It's every one or two minutes. So, I can either trigger it or let it be triggered. So, let's just assume that it's going to be automatically triggered and I'll talk about it. So, if you want to take a look at how the environment is actually set up, if you go and take a look at this repo that we have, while this is kicking off here. So, here's my hello world source code. The developer experience here is they fork a repo and add that source directory. Everything else is generated. What we came to find that most applications were being generated the same way. So, for the 90th percentile, we could actually get most of the data in there and make abstract it. So, if I want to go to production or stage, all I need to do is come and change these four variables. What namespace do I want it? How many do I want of them? And that defines how it's deployed. So, as a developer, all I need to do is actually change those values. Planet is a Kubernetes cluster in an environment. So, it's prod Kubernetes in San Jose, this particular cluster. So, it's the destination cluster. Okay. So, we should, typical. So, let's go and just say build now. So, the power that we have here is an elastic Jenkins infrastructure and our pipeline is code. Now, this isn't all what we actually went and did is put everything in Docker containers as well. So, as we spin up this infrastructure, you can watch this go but what it's going to do is compile the app, stuff it in a container, publish it in a registry, pull it down and deploy it in a Kubernetes. So, that whole process is done in containers, on containers. So, ask me about inception a little bit later. And I'll quite gladly tell you about it. So, you can see that actually two builds kicked off here because one was automatic and one was manual that I just kicked off. But what should happen is I should be able to go to my app once it's pushed out and see that it's just had a rolling update. Let's go and check out the app right now. Okay. So, excuse me. I've got to look at this screen to see what you guys are seeing. So, I'm going to go and just grab the namespaces that I have. Spaces. And just grab my name because from that metadata file, I can actually tell you that there should be a Jenkins prod. I checked into master so it's going to be under this guy, right? So, let's just set my namespace. Let's go and take a look at what we have in this namespace. So, right now I have build six out in the environment. Let's take a look at build six and it's been running for five days just to give you guys. So, that pipeline is still running there. And what I should see here is the running version of my app. It's on 8080. Okay. Hello world 3.9.1. So, once that pipeline runs, we should get .2. The other beauty of this is as a developer I can branch and work on different branches and have different branches go to different destinations because that's a typical developer workflow. So, while this thing is rolling out, let's go and flip this to the dev branch. Okay. So, let's have a look at what I got in dev. Oh, I'm at version 4 on a snapshot. So, let's go just bump this to version 4.1 and we'll bump the app to the app version so we can see. So, all I'm illustrating here is that I'm working on a different branch and it will generate the pipeline off a different branch. And in dev. Okay. So, here we go. Let's check this in. Okay. Bump version. So, let's see if this is, that hasn't actually finished yet. Let's go and take a look. I get this thing to switch window on me. Okay. So, this thing we can go and take a look. Right. So, at the moment it's actually doing a rolling update of that pod as we speak. So, if we go and take a look at the infrastructure, we should have a look, we should be able to see that that's indeed what's going on there. So, you can see that I have release 6 and release 7. I have release 7 running as a pod and release 6 will spin out and I should have release 7. So, all I've done as a developer is changed the code and this is actually going out of Git commit right now. So, what should happen is that I shouldn't have with a new running environment once that, once that rolling update is completed. Okay. Do you guys want to dig into how that all hangs together? Yeah. You've seen the end result. So, let's dig into how it hangs together. So, we're using something called the Jenkins workflow plugin and if you pre Jenkins 2.0, which I think landed last week, pipeline as code is something that's fairly new. But actually what it looks like is there's some Jenkins, sorry, some Java groovy libraries that you can create functions around and functionize all this. So, we just have a repo of common functions that we use and we've just created functions that we use to do rolling updates, check things into repositories. So, let's take a look. If I go here, Jenkins is looking for this. This Jenkins file and this Jenkins file defines exactly what is going to happen in that build pipeline. So, it's broken out into stages. So, we have a preparation stage and that's just grabbing some of the source repo and checking it out. Let's go down to build and publish. So, we're creating a list of tags and then we just go down, set some variables and then we just map these variables into a pipeline and this pipeline is all abstracted. If we go all the way to the deployment, rolling update. So, we've created functions called pipeline rolling update. We pass the arguments into rolling update. All rolling updates and Kubernetes are identical. So, we're able to functionize that and that's just a library that we wrote. So, we have a standard library. Everything's in source control. You pull this down. The developer experience is actually that they don't need to look at this. This is just for your guys' information, right? They can certainly come in and change this but we're giving them that out of the box and that's really the value prop. So, let me just go and take a look at how it hangs together here. If I take a look at the Jenkins. So, this is all we're calling in. This is all version control. So, Docker with registry. Push the artifact up into a Docker registry. So, there's a function there that we've created. Find last replication controller. So, you're able to actually object orientate your code and pull these functions in and use them in a flexible way. Is there any questions? Yes. Yes. So, the deployment of this into Jenkins. So, what Jenkins is doing is we're telling in the Jenkins file of the repository to pull out these artifacts out of a Git repository. Yep, these workflow modules, the workflow modules. Pull them in and then you have the functions in the modules. So, in the preparation stage, there's actually go and grab the modules from master. So, we could version these modules too and have them. Now, what we're creating here is a collaborative workflow that people can modify, add, and we get the visibility and the versioning. Right, so the question was when Jenkins goes and pulls that file, can you rephrase? Uh-huh. Right. Yeah, so the Jenkins file in the repo has all the build instructions. Inside that build file, we've functioned out all these workflow steps that are common to us. So, all you need to do is bring in that YAML file that defines that I want one, I want to name it this and there and we render that into the build and release. I can dig into it a little bit more. What calls the Jenkins file? Jenkins knows to look for Jenkins file in the repo or you can explicitly say if you have a different name for it, please find this file and you can have different Jenkins file per branch or one that's for the whole repo irregardless of branch. Yes. Yes, yes, absolutely you can. So, can you rephrase your question? Can you? Yeah. Yes. Yes. Yeah, you can absolutely do that. So, can you actually make it have the repo in there when you spin it up? So, pass it in as a runtime variable to Jenkins. Yes, you can. So, you could pass that up and boot that and say create a build pipeline on that. It's hard for me. Is there any questions at the back? It's hard for me to see. Okay, thanks. So, let's just go back to take a look at the final outcome here. So, I mean what I should end up with is obviously hello world 3.9.2 went out, right? Can I mic drop now? This mic drop, pick it back up and drop it again. So, this looks incredibly easy, right? And this is from a developer's perspective, we all know that there's a lot of hard work going on on the campus here, right? But what we've tried to do is create something that's actually really easy for developers to onboard and give a really lightweight pipeline. Now, the benefits of that is people want to write code. People actually code will get better if it starts going through CI CD pipelines. You don't have all your intellectual property or all your mind share inside the Jenkins server. It's in your repo with your code. So, I can take that code, put it anywhere and the system will know how to build it. I think that's really valuable. Okay, that's all I had for today. I'm free for questions. Yep, so can you extend it to Garrett? So, whatever the flexibility is inside of the groovy workflow, there's a tremendous amount of it's Java, right? So, you can do whatever you need to do as long as you can pass it in and tell Jenkins to make a decision on that. I'm sure you could hook it into a Garrett system, too. Garrett nodes, yeah, new Garrett node, yes. So, this was Stash, but it's all Git, right? That particular repo was in Stash, but it's all Git workflow, yes? Absolutely, absolutely. So, anything that's an app, Kubernetes is really good at spinning up apps. So, you could spin up a whole independent piece of infrastructure on Kubernetes and it could either last a long time or be temporal. Spin up, do what it needs to do, produce an artifact, publish it and disappear. Any other questions? One in the back, gentleman in the back, yes? So, the question was Jenkins needs persistence, how are we handling persistence? So, because everything's on OpenStack, I have storage at an API. So, we are actually pinning the Jenkins master to specific nodes and then we have host mounts on those nodes right now. So, it's not, it won't ever get scheduled to another node, we're pinning it so that the persistence is there. If you did notice, I could go back, but we actually created a backup sidecar container and it basically grabs all the persistence and puts it somewhere else should we ever need it. But what do you need? The only thing we actually need now is the log files because everything else is in the repo, right? So, persistence in Jenkins is less important. You needed Jenkins persistence before because if you lost the Jenkins server, you'd have to recover all the pipelines. Yes? How did they get there? So, the question was when we spun up the Jenkins server, it came pre-configured with jobs. So, actually we build Jenkins, the Jenkins master and slave containers that I'm calling in another pipeline that's defined on another Jenkins server the same way. So, we're eating our own dog food to create Jenkins. Now, what we actually do in there is say, hey, we want new plugins. So, when I spin that up, it's already pre-loaded with all the plugins and we also load in some default jobs in there. So, you could pre-cut containers with jobs configured, which is what he was saying. Now, that was just a vanilla install for us and that container had some pre-configured pipelines in there. But you could do whatever you want. If you wanted everybody to get a standard set of tools that were baked into Jenkins with a standard set of plugins, you can do that. Yes, in the front here. Can I show how I made it easy for Jenkins to talk to Kubernetes? Yes? Sorry, I'm not getting there. Clean, also Kubernetes. Right. Right, so as part of the build setup, so remember we have everything in containers here and when you go into those build serps, this was a Java app, so I needed Maven. Right, so I have a, here's where Inception starts to start. So, I have a Jenkins slave in a container. I have Docker running inside the Jenkins slave, so I have Docker in Docker. I pull down a Maven container to compile the app and produce the artifact. I stuff that artifact into our container repository and then I actually call Terraform and what Terraform does, we actually render Kubernetes templates. So that metadata, you were seeing that YAML file that defined what planet, what was the name of the namespace and how many you wanted. That gets plugged into Terraform and we have Terraform in a container. So Terraform runs, pulls that information down, renders the Kubernetes templates and submits them to the Kubernetes API. So it's actually Terraform rendering templates that we submit as part of the upgrade. Anything else? Yes, it would be easy. So the beauty of Kubernetes is I'm exposing a service record under a namespace. So Jenkins.zone, all we need to tell, all we need to tell the slaves is where the master is. So you have access to that master from your VM infrastructure as well. Laying down your containers outside of Kubernetes if you want it containerized or a VM or something is simple. And in fact, the slaves inside Kubernetes are using that same infrastructure to reach the master. So, okay, well that's it. Thank you very much. Oh, we got one more. So the question was are we scaling Kubernetes nodes or Jenkins slaves? Yes, the compute nodes underneath. So the question was are we auto scaling compute nodes? Right now we're not, but it absolutely would be possible. The way you would do it is you would create an auto scaling group with heat, have salometer run there and have a condition if I hit 50% across my whole fleet of Kubernetes nodes, then trigger spinning up another flavor of this type and run the same cloud and then lay down a Kubernetes node. So depending on, so the question is how do we then scale down? So this is why we moved a lot of this into containers because I could actually have better conditionals around scaling up and down, the Kubernetes. So you need two triggers, right, and up and down. You can't say when up, not met, then scale down. You actually need a trigger when you wanna take resources away, right? So if you put some logic around a scale up and a scale down event inside heat, you get a webhook. Write some logic around what you wanna do there, fire the webhook, scale down. You could do it. We're not at the moment. Anything else? Fantastic. Thank you for listening and thanks for coming. Happy to feel any questions.