 Okay, it's 10 after 5. I want you all to detect problems in your code before that code becomes part of your code base. That's the one thing I want you to take away from this presentation today. Detecting problems in your code before it actually becomes part of your larger code base. Hi there, my name is Evertaves. I'm a developer advocate at Rackspace. Now I get the question enough, just what is it a developer advocate do? What do you do all day? I get the question enough that I could spend a minute at the start of the presentation just kind of saying what it is I do. So first and foremost, I am a developer. I write code for some portion of my time. Sometimes it's 20%, sometimes it's 80%. Just kind of depends week to week, month to month. Whatever is going on. To that end, I'm a project management committee member and committer on a open source Apache project known as JClouds. This is a multi-cloud toolkit that works with many clouds out there, all the big clouds out there, including OpenStack and Rackspace and HP and all the OpenStack providers and Google and Microsoft and Amazon as well. The advocate person. He's the guy doing the song and dance before you today. He's the guy that goes out into the world wherever the developers are, conferences, user groups, meetups. And not only am I advocating for our technologies, you know, JClouds, OpenStack, Rackspace, et cetera, et cetera, I'm advocating for you, our users. I wanna hear about the experiences you're having using our developer tools and especially if that experience is bad because then we can improve it. I can take that feedback and feed it into our product teams or take it into the tools that we're responsible for directly. And this is me giving you a presentation right here, right now at the OpenStack Summit. This is me, okay. I've also done a bit of operations at my time. In a previous job, I helped build the first OpenStack cloud in Canada with one region in Alberta and one region in Quebec connected by a high-speed research network. And I later found out that this was actually the first OpenStack cloud built outside of NASA. We adopted it very early. And now I work at Rackspace. During the course of a week, five days, I along with some other co-authors helped write the OpenStack operations guide. And, you know, we spent the last day, day and a half editing it down, trying to get something readable out of it. And that's the version here on the left. And then O'Reilly got wind of it and they kind of grabbed it and turned it into a book and actually edited it down so it really made sense. And now I very much consider myself just a contributor to it because we've released it to the community. Anyone can participate in the creation of this book and make it better. You can download it here or buy the dead tree version, whatever floats your boat. And there's really no need to write down URLs or anything here. I'm gonna make these slides available after. And there will just be one URL you need to remember. Okay, so continuous integration. How many people here are practicing continuous integration right now? So half. So those people can kind of keep their hands up. How many people here know what continuous integration is? So just about everybody. Okay, so I'm just gonna quickly zoom through a definition so we're all on the same page. So it's really part of your software development life cycle or SDLC. It's really a very fundamental and crucial part and if it isn't, it should be. And hopefully you can take away some tips today to make it so. It's all about integrating code from many developers working on the same project, bringing their code together in the same base and integrating it. Now the definitions are always kind of fuzzy on just how frequently you should be. Like they say, I don't know, every few minutes, every few hours, a few times a day. So I would like to advocate that you do it with every single change to your software. Every single change that happens in your software should go through the continuous integration process, no matter how small. Naturally, this is all brought together in a shared code repository. Subversion, Git, Mercurial, what have you. The same place where everybody's bringing their code together to work on it, to create, say, a web application. Now you need somewhere to automate the build of your software to actually bring it together, to continuously integrate it and build that software. So you have servers out there that do this sort of thing like Jenkins or Travis or Shipit, I think is another one. One of the core aspects of doing continuous integration and running that automated build is running your tests. Your unit tests, possibly even your integration tests, if you can automate those as well. But it's really important to know that your test run, that gives you a much, much better indication of the quality of that change that has been made. So again, coming back to detecting problems before they become part of your code base. This to me is the fundamental core principle of continuous integration. If you can prevent a problem before it becomes part of your code base, you're much, much better off. You have much higher quality software, which is something that we all desperately want. So now I'm gonna talk about continuous integration for applications on OpenStack. Now there's a huge, gigantic, sweet continuous integration suite that OpenStack itself goes through in order to become OpenStack and in order for people to make changes to OpenStack. That is not what I'm talking about. I'm talking about the developers who are building applications on top of OpenStack, who are deploying their applications on OpenStack Clouds and doing continuous integration on OpenStack Clouds, not OpenStack itself. So who are the players in this story? It is Jenkins. It's gonna be our continuous integration server that we'll be using. We'll be using a plugin for Jenkins, the JClouds Jenkins plugin. GitHub will be our shared code repository. And of course, our cloud is going to be OpenStack. This is where it all happens. So let's take a look at this from the developer's perspective. That is the developer who is actually creating that application on OpenStack. Say it's a web application for doing event registry, you know, like event bread or something. People are building that on your OpenStack Cloud. Okay, so here we've got some of our players. We have our developer, GitHub, and a cloud. The developer, eh, you know, maybe they don't even really care what cloud they're running on. They just know they're running on a cloud. They're getting resources and using that to build their application and deploy it. So that developer wants to make a change. So they submit a pull request to GitHub. And then, we're not even really too sure. Something else happens that triggers some sort of magic. This magic goes on for a little while. Not really sure what's happening there. And then at the end of the magic, there's some result. And you get notified in GitHub of what that result is. So again, you know, continuing with the developer's perspective, they issue a pull request. And how many people here are familiar with GitHub and pull requests? Okay, pretty much everybody by now. That's good to hear. So as we know, a pull request is kind of the, what I consider the basic currency of contribution in a shared code repository. So they go ahead and they make their change and they create their pull request. My change, I'm trying to commit it to master. I've got my branch, my change at the very top. And I'm trying to commit it to master with this pull request. And then some happens, something else happens, magic at this point. And at the end of it, I see in my pull request that it's waiting to hear about this commit I made. Merge build started. Okay, sounds good. That sounds like some magic that's happening somewhere that I don't know where. So after a little while, you get the result and it looks something like this. Blah! You know, damn it. My build failed. And I get this message at the bottom, you know, failed in red. You could still conceivably merge this pull request, but there's a problem with it, right? So at this point, the code that I submitted, the change that I submitted has been merged with the larger code base, not made part of the shared code repository yet, but, you know, merged within our continuous integration server. The test ran, you know, the code was checked for code style problems, something failed, something went wrong. So now I just need to click on the details link, and that takes me to the console output of Jenkins. And so I can immediately, wow, after scrolling for much, much output, I can find out exactly what it is that went wrong. And once I know what went wrong, I can go back and fix it. So I add another commit to my pull request, and that's this commit here. So I fixed it, whatever it was that was wrong. And now the continuous integration ran again, and all is well. Good stuff. Now I can merge that pull request, make that code part of my code base with much more confidence. There's no guarantee it's perfect. There's no guarantee it's right, but you have much more confidence now that this is a quality change being introduced into your code base. And of course, as part of the whole pull request procedure in GitHub, whether you're using GitHub public or private, is the code review. You know, that's an aspect that you don't want to ignore either as part of your changes. So as you're going along, not only are you looking for the result of your continuous integration, you need more eyes on that code too. You need more people looking at it to making sure it is a good change and appropriate for your project, for the web application that you're building. And this is an example of a good change where you kind of knocked it out of the park on the first try and everything passed, just to contrast it with that initial failed build change. Okay, thank you for sticking with me for this long. Let's take a bit of a brain break for a bad slide. This is a terrible slide. I start with Comic Sans, obscured text that you can't see. Old outdated internet memes and terms that nobody uses anymore. Video with no sound. This is me. I live in Austin, Texas now. I went to a Maker Faire with my family. And you go to a Maker Faire expecting to play with Lego or Raspberry Pis or computers and soldering and putting things together. The first thing that we saw there was a 60-foot scaffold and people jumping off it. A stunt camp had set up shop and we're having people jump off this thing. So to my family's credit, my six-year-old, my eight-year-old, and my wife-year-old all jumped off this thing, including myself, and it was just a really bizarre way to start a Maker Faire with this jolt of adrenaline. Okay, so now that we've taken a little break, let's look at this from the operator's perspective. So this may very well be you. So who here is kind of responsible for administrating the resources on their cloud? Okay, so, Han, so this certainly applies to you and it's also good knowledge for developers to have as well, to have some idea of what's going on in the background. This is also a role that can be fulfilled by, say, if you wanna call them a build master, somebody who's responsible for creating all the artifacts that make up your software project. They can also fulfill this role. So again, we have the same players, but now for sure we know that we're using an open-stat cloud. That we know for sure, because that's why we're all here. So we have, again, our developer creative pull request, but now, because we're under the hood, we know that a webhook is being fired from GitHub to our Jenkins instance. And we know that Jenkins is our continuous integration server. And at step three, a build is happening on one of our worker machines. We're not necessarily building on the Jenkins master itself. We've created a pool of workers to pull from or dynamically create. And once that's done, Jenkins feeds back the result to GitHub. Okay, so the reality of this is that it is really very complicated to actually set those steps up. And it's really more detail and more complexity than I can kind of reasonably deliver in a presentation like this. It's really more of a workshop, a hands-on kind of workshop thing. So what this is here is a link to this blog post on the developer.rackspace.com page. And this is a very, you probably can't quite see the scroll bar here, but quite a lengthy blog post on how to set up Jenkins on an open-stat cloud. The example happens to be on Rackspace private cloud, but that's just open-stack. And this works on vanilla open-stack anywhere you can find it. The reason I put this here like this is because when I encounter a blog post or a knowledge-based article or whatever it is, I like to see everything at once and put things side by side. So as I'm working through the blog post or whatever it is, the instructions, I have the things that I'm working with on the left-hand side. And you can see I have them in the tabs up there. Either this is all on one screen or you've got multiple monitors. And this is probably common sense, but I've seen enough people not work like this where they flick between screens that it kind of makes me crazy. So just a little tip when you encounter a lengthy kind of one, two, three instruction-style blog post. So again, this is a link that takes you to this blog post. What I'm going to give you here is kind of the high-level overview of the steps you need to take to set this up on open-stack. Okay, so first off, you need a server. You need to create a server for your Jenkins master. Then on that server, you need to install Jenkins. And this is just a matter of setting up a specific repo and getting the latest and greatest Jenkins rather than installing whatever the packaged version is for your Linux distro. You definitely want to secure Jenkins. Out of the box, it comes unsecured. If you're working through the UI, you just want to go out and do it directly first thing. You can script that as well, which is also recommended. Then you need to install a handful of Jenkins plugins for working with GitHub. And they're all specified in the blog post. It's the GitHub plugin, the GitHub OAuth plugin. So you can use OAuth tokens and GitHub and the GitHub pull request builder. There's a special plugin just for working with pull requests. Then you create a Jenkins job. This is the job that's configured to actually build your software project. Then you need to kind of fiddle a bit in GitHub itself to allow access to Jenkins and allow Jenkins to access GitHub. And there's definitely a fair amount of fiddling going on here. And in this case, I think the complexity is warranted because you're establishing trust between two systems. And you need to make sure that all the bits and pieces are in place so you don't get hacked, basically. Now, if you're running all of this behind your firewall, if you have your private open stack deployment, then that sort of thing might not be as much a concern because you're running behind your firewall and you've got a whole team worrying about security just keeping people outside of your firewall. But you know what? It's not a bad idea to also pay attention to security for your internal systems. You always want to consider security in depth. You want layers of security in your organizations. Okay, so I've been going on and on about pull requests and building pull requests. But the reality of it is at some point, people are just gonna commit directly to master and skip the pull request process. It's true. It's not a good idea, but people are going to do it. And maybe sometimes it's warranted. Maybe it's a trivial change. So they just wanna go, boom, I'm fixing a spelling mistake or whatever it is. They're just gonna commit directly to master. So you do need to configure your job to also build on commit. But this is really what you wanna do. This is what you wanna focus your energy and effort on. And that's building whenever a pull request is issued to that shared code repository. So now we've got Jenkins and we've got GitHub and we've got kind of all the machinations completed there. But what if we wanna be a little more dynamic about how we create the workers that actually build our software? What if we wanna be a little more intelligent about how many jobs we're running and the capacity that we're using in our cloud? If we can run our jobs in parallel, then developers don't have to wait as long to see the results of their commits, of their pull requests to that repository. So if you can fire up a worker for every single build, say your build takes, I don't know, 10 minutes to run. It's a big complicated system. Lots of tests, lots of dependencies. You don't want all those builds coming into a queue as developers are making changes. Ideally, you could run all of those builds in parallel rather than running them one after another. So here's how you create Jenkins workers using the Jclouds plugin on Rackspace private cloud. Again, just open stack. This is another link to a blog post that I literally finished writing last night and got published today. So it's hot off the press and this is how you configure Jenkins to create workers for you. First, I show you how to do it manually and then dynamically so it kind of helps you understand it a bit. Again, a little too much detail to get into right now. So I'll take us through kind of the high level steps. First of all, you need to install the Jclouds plugin. You need to add a new cloud as Jenkins terms it. You go into a configuration page and you add a new cloud and it's the Jclouds cloud. It's kind of a funny terminology. Then you create an instance template. This is a template that is responsible for building your application. It's how your instance starts up and this instance is responsible for building your application and running the tests that are part of your test suite doing the continuous integration. So once you have that template, you want to test it out because you have to know that it can actually build your project that it actually works properly. So you do a manual instance creation. You go to this manage nodes page and you manage and you provision it manually. And if it blows up, you get a stack trace and an error page and you can see what went wrong. So now that you've got this instance, this one lone instance, you want to test it and make sure it can actually build your job. You've created it. Now you actually want to run a build on it so you know that it's capable of building your job. After you've determined that, that this template is good and it can actually build your job, then you can take a step back and delete the thing, say, okay, I know this works. And you go and you configure dynamic instance creation. So this is where, as build jobs are coming in, you're creating workers for those build jobs. Developer makes a change, webhook fires to Jenkins. It creates a worker and runs a build on it, runs all your tests and everything and then the result comes back. So let's see what that looks like now and fill in the blanks. So again, here's what I just described. This whole process of the continuous integration life cycle in your software project, putting it all together, the pull request that we saw before. So now we know that it's actually a webhook firing from GitHub to Jenkins saying, oh look, here's a pull request. In your repo in GitHub, if you click on settings and then go to, on the left hand side, you'll see a menu for webhooks and applications or something like that. You can actually then drill down into the requests that GitHub has sent to your Jenkins server and you can see here at the top all of the headers that are sent in that request and then the payload, the body of the request. So there's just a ton of information in this payload. It's all the information that any, practically any client continuous integration server would require to build your project. But really the key here is that there was a pull request and now I'm firing this webhook off to Jenkins so you can do something about it. And build your project. Step three. So this is kind of the status page of Jenkins for the jobs. And as my example, I actually was building J Clouds itself. I forked the project and submitted a change to my own fork of the repo. Did the whole pull request process. And then I had an instant spin up dynamically to run the tests in J Clouds itself. So it's like I'm using J Clouds to build J Clouds here. Some nice inception going on. And you can see the progress of the build. And finally, from the developer's perspective as we saw before, you get the result. Whether or not it was a good change. Whether or not there were any problems, attempting being introduced into your code base. So now that you're doing continuous integration for your applications on OpenStack like this, what's next? What are some of the next steps you can take after you've gone through all of this stuff? And there's quite a bit there to do. Well, you could conceivably instead of, you know, you create that server and then you install Jenkins on it. You could actually turn that into a heat template. And in fact, lots of these heat templates already exist out there. So use heat to install Jenkins and you cut out a whole bunch of those manual steps. The creation of your continuous integration server becomes much more reproducible, much more repeatable. Now here are a couple of my favorites. Again, these are links. Hijacking the build pipeline. This is the slide deck from a colleague of mine, Kyle Kelly, talking about security. He's very much into security and he came up with this great talk that he did at DEF CON about how if someone knows kind of your continuous integration pipeline, there are ways to insert themselves into it and actually hijack it and take it over. Or they can just spill your secrets to the world, you know, by doing simple things like printing out environment variables as the build is running. They introduced the code that they change is simply printing out environment variables. And they're phishing for things that could be printing out the environment variable password and then they'll do a bad change so that they see that Jenkins error page. And at the top is gonna be, oh, here's what the value of that environment variable was. And there's lots of ways that you can secure against this, but I just want to impress upon you that if you're doing this stuff in public or if you're doing it in private, you wanna think about security and find ways to prevent people from hijacking your build pipeline. Another good one from a colleague of mine in the J Clouds community, Andrew Bayer. He actually works for Cloudera. He's responsible for building the Hadoop distribution that Cloudera produces. And he's pretty much a build master over there. And he has a great talk called, hello, AV guy? Is that me? Hello? Okay. Little glitch. Okay, well, we're almost done here so let's get through it. Okay, so anyway, he's got this great talk, seven habits of highly effective Jenkins users. Lots of good advice in here. Just tons of good advice in here. And he gives this talk at Jenkins conferences and when CloudBees goes out to present at conferences, they fly them out there. So the key part, this one, the key takeaway for me for this one is automating a lot of the Jenkins dials and knobs and levers and pulleys. There's so many of these things on Jenkins you wouldn't believe. There's an UI element for each one of these things. So if you can, you can use the Jenkins API, the application programming interface or the Jenkins command line interface to automate a lot of this stuff. So again, that's speaking to making the creation of your continuous integration server a lot more repeatable, a lot more reproducible. So once again, the whole point of this, all of this work is to improve the quality of your software projects that you're producing for OpenStack. These web applications that you're creating on OpenStack, all of these different things that you wanna do on OpenStack, they can be higher quality when you introduce continuous integration into your software development lifecycle. So detect problems before those problems become part of your code base. I will be making this presentation available at that short URL right there. That's the one and only URL that you need to remember from this presentation. Does not work right now because there's nothing up there. So, but I'm hoping that this evening I'll have a chance to upload the presentation and put it up there. You can find it at that URL, that short URL. If you wanna try this stuff out on a public cloud, massively scalable OpenStack public cloud, check us out on developer.rackspace.com where you can sign up for our Developer Plus program and that lets you kick the tires for free on our public cloud for $50 per month for 12 months. Thank you very much. I'll take some questions now. Just a question. If you have bills that take, say, 10 minutes to do, what happens when you get one commit, you start doing the build and you get another commit? Do you start doing the second build immediately with all data or counting the first commit? So there's a configuration piece in the, I believe it's the job configuration page called Single Use Slave or Single Use Worker and the terminology I was using here and that ensures that you only ever use a worker once and then you discard it. So that's one way to avoid that situation. I'd have to toy around with it a bit more but it should spin up. I was thinking about conflicting commits. Oh, conflicting commits. Yeah, so that the first one would work but the second one applied to base will work but the second one and the first one will not work. So you'd still have to have it bisecting, something to do bisecting afterwards. Oh, so like from different developers? Oh, from different, yeah, okay. So I mean, in that case, like if they don't have any knowledge of each other at that point, then they won't actually see the change. The continuous integration won't see the change but I think I know what you're saying in that, this kind of comes back to how you manage your software development. If you were to have something like, if you were managing your development, your integration with like feature flags and everyone's just constantly committing into master rather than everybody having their own feature branches. Like when everybody has their own feature branches, yes, you get into this situation that you described but if you were to do feature flags where everybody's constantly committing to master, constantly all putting into the master branch, then that'll make the developers aware of each other's changes a lot earlier on. So that's one way to avoid that scenario. Do you mind asking into the mic if you would? We're using OpenStack. So why do you need the Ajay Clouds abstraction layer except for the fact that you're a contributor? Yeah, no, it's just one way to spin up workers is the answer. It's one way to spin up workers for your builds. There are other ways to do it. You could manually provision a whole bunch and not even use Ajay Clouds at all. I just find that it's a convenient way to go about it. And yes, that was the next point, thank you. You can use it across different clouds. You can build on any cloud that Ajay Cloud supports. I'm not using any hosted services, so I'm using GitLab, not GitHub, and everything's internal. Why should I run Jenkins in the cloud instead of just deploying it on bare metal? What are my motivators for deploying in the cloud and dynamically allocating sleeves? So in this particular case, it is that you said it right there, you hit the nail on the head is the dynamic nature of it. It's being able to create workers as you need them. And I kind of forgot to mention, it also removes those workers when they're not used anymore. There's a retention time dial in there. So if you only have five bare metal servers for your continuous integration cluster, then you can only ever have four builds running at once. And honestly, they're probably using some small portion of the CPU, or you could dynamically create many virtual machines in order to run your builds in parallel. Does that answer your question? Sort of? Yeah? Is the dynamic nature in being able to parallelize your builds? In this case, pretty much so. Does anybody have anything else that can help? Okay. Oh, I see. Is JCloud's plugin specific for Jenkins or it works with, let's say, Bamboo? This particular plugin's, yes, specific to Jenkins. The question was, is this used in Cloud Foundry? And the answer is no, not to my knowledge. Okay, thank you very much, everyone, and have a great time for the rest of the summit.