 Okay, everybody. So, let's start. Let's move on. Let's welcome our next speaker, Arele Vingy, who is the principal software engineer at Red Hat, who worked with DevOps on CI workflows, and he's going to tell us more about how to build all the things with Dev CI CD pipeline. So welcome. Great. Thank you. So we're going to talk about developing everything, all the things with Dev CI CD pipeline. I'm going to walk through basically how to set up all of this in a local environment. Your laptop, your workstation, instead of, you know, having all these things remote like we do in production or stage environments, we need all this for developers to be successful locally in their own box. Oops. I think I hit the wrong. There we go. Thank you. So we're going to talk about what's the problem here. So obviously, we can push code changes up into, you know, for PRs or check-in code, but what about actually testing it locally and doing it more efficiently that way? This way we can find bugs sooner. We can get things solved right at the developer's screen instead of it being pushed into the environment as a bug possibly or something that, you know, peers are reviewing. So I'm going to go with the solution really. The Dev CI CD pipeline is composed of an OpenShift environment. OpenShift already has Jenkins integrated into it as an application, so it's very easy to just spin these things up. And I'm going to talk also about source to image, which is kind of, I call it Docker on steroids, really, is what OpenShift has done with that feature for images. So the OpenShift apps, obviously, Jenkins, everyone either loves or hates it, but it's already integrated into the environment. And what I'm also adding is an application into the environment is a local Git server. And I'm going to talk more details. And I'm going to go over two specific use cases, one that OpenShift already has out of the box. So they have a sample Jenkins and OpenShift pipeline that builds and then deploys applications. And I'm going to talk about specifically a tool, a Python tool that we develop in our group, actually by that man, Clint Savage over there, who it's called Lynchpin. And he'll actually be doing a talk tomorrow at noon on that. So then we'll do a demo and just talk future improvements and open it up for some Q and A. I did it again. There we go. Oh, my God. We're constantly, development and operations are coming together. We're continuously delivering. It's very scary. Dogs and cats living together, mass hysteria. So as I said before, the problem is we really want to enable developers to be more efficient, develop higher quality code quickly as they possibly can and not overburden them with a CICD process. Developers like I don't want to have to set up Jenkins and do all these things. What if it was early there pretty much with a couple commands to have everything set up and running? And then that just allows them to make changes quicker. So a solution to this is that we want to constantly develop. We're going to build. We're going to test. And we're going to keep integrating and actually deploy these possibly into stage or production environments within your own system here that matches up with what stage and production would look like in your environments that are outside of your little laptop, your little world here. And that will help us to identify issues sooner. We instead of issues getting into a PR or God forbid into the code base, they can identify those issues sooner. And we just really need to do this, right? So as Nike says, we just have to do this. This is something that's going to make us better as developers and write and have better products and better code at the end of the day. So how do we build this? So we're going to provision a local pass and a CICD infrastructure. I know that sounds like a lot, but OpenShift has made that pretty seamless here at this point. OpenShift is composed of a router registry, HA proxy. They already have a concept of pipelines that integrates and matches up with Jenkins pipelines. So that work has already been done. I'm going to show that in the demo. And then we have the Jenkins master, which is just an application that runs like any other application or microservice inside of OpenShift and a Git server, a local Git server. And this allows us to then to do all these other tasks of building, testing, and delivering, deploying, whether it's Docker images or PIP modules or RPMs or other things. It allows us to do that and deploy that in the different environments. So basically by doing some simple commands, all you need is OC client, one three and above. You can get that, set the path on your system. Another thing that I'll also add on here that's extremely helpful is it's called OC cluster wrapper. And it allows you, it's actually written by Jorge Morales on the OpenShift evangelist team. And what allows you to do is instead of just having one cluster as developer that you're bringing up and down, maybe you have one cluster that's for QA testing or you're testing something or you have a customer issue and you have a different environment here. It allows you to bring different clusters up and down and test them as you want. And really, this is all you need to set it up is OC cluster up, whatever you want to call it, log in and you're ready to go. So source to image, like I said, is Docker images on steroids. You build Docker images from source code and you can actually keep, you can make changes in your source code and that will automatically cause you to build a new image in the OpenShift environment. And then you have a running, you have ready to run images and it's injecting that source code into the container. And the container does all the work, does all the preparation and then you can version and control your build environment. I'm not going to go too much more in detail on S2I. There's definitely tons of documentation out there and I recommend looking into it. It's extremely powerful. So we all know what Jenkins is. In this case, we're taking this Jenkins master, also slaves. We're running them off the Kubernetes pods that are in OpenShift. We're spinning up pods to run them. And in this case, we can have Jenkins and ephemeral image they offer out of the box or if you have persistent volume set up, you can use that as well. And then you can, it's all based on, you can make your own custom Jenkins image out of that. So if you wanted to actually enhance what OpenShift has already done or add your own plug-ins or make configuration changes, you can do that all with the source to image and it will build that way inside of OpenShift. The slave master is actually running as a, is configured inside of the master configuration. And you just, it's another S2I image that gets loaded. Again, OpenShift offers you a slave base image, a Node.js and a Maven image. But you can basically, what I did with ours is I took their slave base and I made a, I used that as my base image and then I made another one to set up my dependencies for Lenspin. The other cool thing is the slave isn't even there spun up as a pod until you're actually kicking off a job in Jenkins. So it's completely tied into that. So you're not having, even you're not having bare metal slaves lay around or even as how we use OpenStack of having slaves in OpenStack that are kind of sitting there and have to rejoin. These are really on-demand slaves. Then there's Pipeline and this is kind of Cloudby's new method of describing Jenkins jobs. It's all written in groovy DSL. It is getting better. It's a little bit, if you don't know groovy, initially it was a little cumbersome, but it definitely, they're basically making more of declarative language for it. And it's all contained in this Jenkins file. So it's very similar. If anyone knows Jenkins job builder, it's very similar. That's more of a YAML format. The difference here that's pretty cool is if you put this Jenkins file as another component in your source code repository, so you have your source code, you have tests and now you have a Jenkins file, that automatically gets put into Jenkins. You add your SCM to Jenkins and it says, oh, there's a Jenkins file here. Now I'm going to set up these jobs for you as a result of that, which is pretty neat. This is basically what a Jenkins file looks like. So in this case, I'm running on Node that Jenkins slave. I have these two different stages, a build and a test. I'm setting up a virtual environment in the first stage and I'm pulling from that local Git repository that's already in my OpenShift environment. And in the second stage, I'm doing testing and I'm just doing a basic nose test that's in there. Some integration tests that we have. And then I'm archiving those artifacts and I'm publishing them. And we'll show what that looks like when we get to the demo part. And then we have this local Git server and the cool thing here is this image already has some web hooks already tied into OpenShift. So as soon as I take my code base and I make changes and I push it to this local Git server, it automatically has a hook that if it sees a Jenkins file will automatically put it into that Jenkins instance and start running the pipeline for you. So that's also pretty neat that you can put this Jenkins file together, could have your tests all set and your source code and your change that is the development code that you're really concerned about and have that change all pushed up to that local Git server and have it running. So like I said, the two use cases is kind of the general out-of-the-box OpenShift one where they build applications and deploy them. And use case two which is dealing with linchpin that we're going to build and test that and that Jenkins file referred to that use case scenario. The OpenShift one you can get and all the references at the end of the slide deck you can have it has a list of all the places to get all the information and there's also a blog post on how to do what I just did for the demo. So let's get to that. So here like I said we're just going to get the OC command. I went there, downloaded it, set the path and I also going to get OC cluster wrapper. It's just a GitHub project that you can check out. There's links to that as well. So now I'm just doing the OC cluster up. So now all I had was a command line utility and the wrapper going to bring up a cluster. I called it some name. Call it CICD Sandbox and it's going to start to bring up that cluster. So that's locally here in my environment. Obviously you need to be on, you know, have network connectivity when you're going to pull images down to set all this stuff up. But once you actually have the images downloaded then you no longer have to be tied to the internet. So you can run this all locally as a standalone environment. I'm going to jump around just to speed things up a bit. There it goes. So at the end of this it's going to give a URL back and you can log in as developer, developer which will also be your same login credentials to the Jenkins master when we set that up. So you can just forward basically it allows you to authenticate through OpenShift and not have to have a separate login set up for you there. And that's going to be your local URL that you're going to go to in the cluster. It comes already with this My Project setup. Obviously you got to get past the certificate part and then here the cluster is going to be up and running. Nothing in the no apps in the cluster yet once we once we log in. So now we're going to add a project. All can do that through the command line. You don't have to add a new project but I did in this case which allows you also keep things separately by project if you wanted to do that. So now I made the new project called CI CD pipeline. When I go in there I have no nothing inside of it, not no applications. The Jenkins master is not in there yet. But then all I'm going to do is I'm going to say OC new app and Jenkins dash ephemeral. So this is already keyed off tight into OpenShift very closely to set up a Jenkins master. And I did an OC status and automatically it's already starting to build. It's pulling it pulled down the image. Obviously I fast forward this a bit so that we're not waiting for the poll to happen. And then the image gets gets put into the OpenShift environment automatically. You also could use the persistent volume one if you wanted to. In this case I'm not. Now you have the route already set up for your Jenkins master. You approve the certificate for that. It's going to say is it okay if I log in with your OpenShift credentials? So you're going to put in your OpenShift credentials there. It's going to ask you this one time about you sure you basically want to do this after this login happens. You say yes I am cool with that. And then you have a Jenkins master. You have a sample job already to go. It's already in the environment. I didn't have to do any crazy configurations of my Jenkins master to get this. I really just did an OC command to bring this into the environment. And then this is just a sample job. The first one failed because I forgot in this demo to actually add the application into OpenShift. So that's why the first one was read. But now I'm going to go back to the command line. And I'm going to add that application. This by the way is all in the documentation that you'll you can find for the Jenkins master example that's out there. So I'm going to add the application. And now I want to go back to Jenkins and I go to click build. Now it's going to build it for me. There's the app. Let's skip to this part. So now the second build is running. This is a standard build. It's not a pipeline build. But I wanted to show the difference with if you're used to Jenkins of what a regular build in Jenkins looks like and what a pipeline will look like. It's just basically a Node.js like Hello World app that's running. And it's going to have a front end. And by the way, the only thing that this is doing here is there's also inside side is Jenkins image. There's an OpenShift pipeline plugin that is already integrated into Jenkins so it can talk directly to OpenShift. So it's doing OpenShift builds, OpenShift deploy as actual build steps. So it's a first class citizen instead of you having to do OC commands as shell steps. You can actually just run those as build steps triggers and post build actions. And there's the app. So now I'm going to jump to the pipeline. Let's go to that. So now it also, let me just pause here for one sec. So this here, sorry about that. So this here, the other cool thing is if I showed you on the command line how I added the Jenkins master, you could also use the UI to add either of the Jenkins types of masters that they have here. Or you could create your own also and have your own template here. And here is also a Jenkins pipeline already integrated into OpenShift for you. It's going to come up with this template and you can change values here of where maybe you're getting the pipeline from and all that. I took the default here, but it's completely flexible to you to pull from your own source code to make this if you wanted to. And now that I've added that, you'll see that it automatically loaded that pipeline now into my Jenkins master for me. So the same application we just built doing the standard way in Jenkins of the old, you know, Jenkins job and that sort of thing. Now we're building a pipeline and we're going to see the difference in the UI. There's also, I don't have it demoed here. And if there's time, maybe I'll show at the end. There is the new Jenkins UI, which is called Blue Ocean that has even a nicer look than the old Jenkins UI. And now the cool thing here too is what we're seeing in the Jenkins master for pipeline. It also reflects here in OpenShift, the exact same pipeline. You can also see more comprehensive logs that you usually with Jenkins, you get like a double of entire console logs. It really doesn't have a breakup of the different steps. But if you have those broken out in your Jenkins file, it will show up as separate log notations where you can just click and say, okay, this is what happened during that specific build step. And there's we're building the application. And here's the logs. So I did a trigger. So I'm just looking at the log for that. Now I'm looking at the log for the actual build part. And here's the deploy. And the idea will also just like with the other example, we'll have a application at the end of it that we can look at. There's a log, the same log that you'll see in OpenShift is also here in the console log. And there's the pipeline as well. You can see the log here to same log that was in Jenkins. And there's the build. Let me just back up. So there's the again, the same same pipeline in both places. Obviously, if you're prefer to see how it looks in Jenkins, you can do that and just stay into Jenkins world or you can also look what the visually what it looks like inside of OpenShift. Really, with the three, four release of OpenShift is where this full integration happened. I think they're even they're making more kind of ties to Jenkins as they go along. Right now, we're ready to really set up our final piece of the development environment and we're going to deploy our Git server also. Yeah, OpenShift. Yeah. Yeah. Yes. Yeah, it automatically because of the plugins that were written by the OpenShift team, the Jenkins plugins, they have a sync plugin which is used for the credential part that you saw at the beginning. And the OpenShift pipeline plugin is what basically it's like you have build steps today to do Python or do any other things. It's a first-class citizen, so you don't have to do a shell step and say OC whatever. It already has these concepts built in to Jenkins automatically by doing that and that plugins automatically in that image. So if you take a derivative of that image or you make a new one, instantiate that, you would automatically get that already for yourself. And then, yeah, the idea is that it can just automatically talk to OpenShift in your environment and it's already connected and there's no added twisting and turning to get that to work. It's already kind of there. I'll talk about the one twist and turn that you have to do or you have to at least, I do it through the UI now, but I'm going to come up with more of a command line way to do it, but there's only one piece of that which is nice. So right now I just deployed that local Git server. I created this role for the Git user. Again, all this is in my blog post and documentation, so you don't have to memorize it. So now I want to make sure that I have the route set up or I want my Git server a variable so I know what it is. And you just basically do an OC command to get that. And now I'm going to make my, I actually, yeah, one other bug that I found while doing this and I have to talk to Cesar Wong and put it together is, it doesn't happen with a Jenkins master, but with the Git server for some reason, you have to put the URL or the host name in your Etsy host file, which is kind of annoying, but it's a one-time thing if, you know, or you can just add to the line, it's always going to be the same IP address and you add it there. So it was one of the things that I want to improve upon. So now we have our full environment set up, right? We have, we have the OpenShift cluster. We have our Jenkins master. We have also the slave base images being called inside the configuration Jenkins master. I'm going to talk about my specific image and then we have the Git server set up to now actually do our development work and really with a small set of commands to do that. Let's see. Okay. So now at this point I'm ready. I've checked out my repository. I have LynchPin checked out. I went and went and you can check out something on GitHub or any Git server that you, you know, have code on whether it's internal like to Red Hat or some company and you can grab that code. And then I add my Jenkins file that I showed you guys earlier, same Jenkins file. I'm going to make a quick change to the, to the, to the code here. And now I'm going to commit this. I'll also show you a mistake that I made in the demo. But you get, add, you have to add that remote repository. I would say you have to add the local Git server now to your, yeah. So now it's going to give me the Sarasail. You don't have that Git server in there. I don't know what you're talking about. But again, it's just a command to add that like you would any remote. And now I'm pushing my changes. Since it had that Jenkins file in there, we're automatically, bam, there it is. Jenkins file showed up. It's already started my build. You can see the same pipeline in OpenShift that was in Jenkins. And again, you can look at the logs. So the same shell step I was doing some virtual environment setup and then installing. And I'm running my test. I'll also get my test results since I, in my Jenkins file to find I wanted to archive the test results. And there's the read me with the change that I made. And here's the tests that I wanted to run. And they're here in Jenkins. I can look at them here. And I have the file archived as well. And you can see all the tests ran there. And there's the Jenkins file again. And that's basically it for the demo. That works, yeah. So some things that I wanted to add. The one thing is, I didn't show it here, but you have to, from my specific slave image, I had to go into Jenkins in the global config and add the name of that image. In this case, I had one on Docker Hub that I have up there. I had the name, some other basic parameters. And it automatically, that allows me to spin it up dynamically as I wish whenever that job runs. And then decommission it and destroy it when it's done. So that, that part I would like to automate in some way, probably maybe a command line or add it, maybe a part of the OC command or something that we could do there to make it a little bit easier for developers. They don't have to go into the clunky UI that Jenkins is. The new UI looks pretty, pretty cool. But again, developers don't really want to deal with UIs. They rather do stuff on the command line. So that's one thing. Consolidate any steps where possible. I think I've limited them pretty well, but there may be some areas to fine tune. And really have more examples of creating custom Jenkins images and slave images. Openshift does a great job of that. But they also may not want a lot of plugins or they don't have a lot of ways to show all the different kind of facets of different installations or different configuration changes that you might want to make. So I'm also would like to add more use case examples. So maybe if anyone has them or has ideas, I'd love to hear them and add them, add them to the list. So here's all the references. All the slide, slide decks will get posted, but it has everything about the blogs, Openshift sites, Jenkins stuff, the world, develop all things. So that's, and here's reference to the different plugins that were used here for Jenkins that are already integrated. So Openshift actively owns these plugins. It's up for the Jenkins pipeline. Obviously, CloudBeez owns that, and it's part of the community. But the other two plugins are also upstream, but they are maintained by Gabe Montero and the Openshift team. Thanks. And questions. Yes. So that would, you could probably then use one Git server and have the repos called out in that Jenkins file to have them brought in together at the same time and then built. Yeah, I think that's how you could do it. And then you can build an application. Yes. So you're saying on the deployment part? Yes. Yep. Sure. So what you can do is you can have like obviously we had the one project here, the CI CD pipeline project in Openshift. Let's say we called that stage and we had another one called production. We could then in the Jenkins part of the Jenkins file would be I deployed this into stage. Maybe I ran some more integration testing. I was happy with the results. It gets promoted into that another namespace in Openshift called production. That's how you would do that. Did I finish your question? Sorry. Okay. Yeah. Oh, sorry. Thank you. So Mike asked how do we get basically I guess specific configuration information into that image? Is that a correct way to say it? In a source image. In a source image. So the way they allow you to do that like for Jenkins or even for, they have like a run an assembly that's part of that sits along with a Docker file that if you put stuff in that folder automatically so you can keep it within that one image repo or one project repo you could have it there and it would just be to like I want to install this as part of the execution of building the Docker file and deploying it. It would copy that over automatically for you. That's what the source to image stuff allows you to do. So let's say you had some like specific like an example is what they do with the Jenkins master. They have like a list of plugins. But if you want to add your own plugins you could have a plugins directory and it would say oh you you're looking we have to bring this plugins directory because it's automatically noted in our assembly to put it in here. So there is obviously as developers you may want to dive into that more but it there's more work to do that. That's why I kind of show the basic example but it's definitely feasible to add your own custom you can customize it but you can also keep it in one repo you don't have to have like five different repos to pull from this location to that location to get the information that you want. Any other questions? I'm sorry I can't hear you can you just say look you you you could you could do that you're saying like you push a change up to GitHub. Let me repeat the question. You're saying push it into OpenShift without a Git server is that? Yeah so repeat the question. He want to know if we could just without doing a push to the local Git server is there another way that we could do it from our local workspace to get the code into to start the process. So currently there is not. That's why I put the Git server in the setup. I asked that question earlier on to some of the evangelist guys in OpenShift and members of the OpenShift team and they said this is the way to do it now it's not to say that it will only be the way but right now this is the limitation that you have to push you have to have that Git server there and do it that push it into that and then it has already the hooks built in to do that. There may be some other mechanism that comes down the road that you could just do it from your workspace a lot easier. Yeah I guess you could do that too. That's a good idea. Any other questions. So OpenShift has. So the question is can I use something else besides Jenkins like Travis for example. Currently OpenShift is extremely tied to Jenkins. They actually propose this to all customers that come through. So this is actually a solution that they put out there. So it's not to say that you couldn't come up with your own solution that maybe does a source to image of some something with Travis that you could do that. But definitely OpenShift is is seeing the value in Jenkins coupled with the pipeline stuff to leverage that. I think they kind of look at that as the in talking to some of the folks in OpenShift is kind of the it's like 80 percent of the market is geared towards Jenkins with their with OpenShift. So that's kind of what they tackle. It's not to say it's the only solution out there. Any other questions. Yes. Yes. On the master. So I may not understand in the question. Yes. Yeah. Oh I see and then you're doing Java on top of that. Yeah. Yeah the question was basically could you run if you're running Java and you connect with it with the slave base image and it's already Java on that. Aren't you going to have a collision running test. I think is what you were getting at. So I think the Docker and Docker is one solution which is is pretty scary but but I'm sure there's you could also probably spin up another another node from that slave and maybe run and execute and talk to it and run some tests. You probably do something like that. It's the only thing I can think of right now. Any other questions. Yeah Mike. So the question is is there a limitation with the pipelines that OpenShift can understand. So if we write a complex one in Jenkins you know that Jenkins file gets pretty big and we have a whole bunch of stages I would say you know are we limited. And the answer is no. Right now it is very heavily integrated with Jenkins file and that pipeline. They spent a lot of time working on not just the under the covers mechanism into OpenShift but also the UI work to to plumb that to the top so that you could see all the pipeline. I think it's just leveraging what's coming out of Jenkins into OpenShift. Yeah I think all it I think actually what it's doing is basically reading the it's reading the same API that from Jenkins I believe. Yeah. Yes. So the question was can we have a multi-micro service and launch it from Jenkins basically. And yet that's exactly right. You could do it all couple together. The idea is you could have a like let's say a Node.js with its own database or something else and you can all have it set up as a Jenkins pipeline that would then talk to OpenShift to then deploy that in the different environments as you want. Yeah one of the examples that I did not show has that actually like a it was a Node.js front end and a MongoDB back end and it's building and testing both of those and then deploying them. It was one thing I was going to say on that too. So yeah there's there's no that was kind of why they went that route with using Jenkins coupled with OpenShift for that for that very reason. Any other questions. Right well thank you very much. Appreciate your time.