 Ricky Elrod and we're talking about the use of OpenShift within Fedora's infrastructure and I'm going to talk about what we have set up, what we're thinking about doing in the future, some open-ended questions that we are going to be looking at going forward. So quickly a little about myself, again my name is Ricky Elrod. I work at Red Hat. I'm on the Fedora engineering team. I've been involved in the project for about seven years now. I've been interning at Red Hat since 2012 and it just became full-time back in May of this year and these are some of my research interests. Okay, so OpenShift, why are we looking at OpenShift? Well, we wanted to start kind of offering a more self- service platform for people who are running applications on the infrastructure to have more control over over deploying the app without necessarily always depending on the core sysadmin team to handle scaling, handle a lot of the maintenance that comes along with the deployment of these apps. So along with this app owners can kind of scale and deploy, redeploy their own apps without necessarily having to ping the core sysadmin main team to do a lot of this manual work for them. Another advantage of OpenShift for people who own these apps is that deployments of these apps can be triggered based on any number of external sources. So when you push to a Git repository you can trigger a rebuild to happen automatically. When you run a test suite somewhere it can tie into the OpenShift API and cause a deployment to run or not. Any number of configurations can cause a deployment to run. Another thing is that the packaging problem can potentially largely go away and Java people got a little bit. Yeah, so what do I mean by the packaging problem? Well, a quick meme. App owner, I want to deploy to that, right? You know, we have this all the time get RFR requests where you know somebody comes up with this this cool thing that is going to make everything better they want to deploy it in the infrastructure but there's a problem where you know it has all these external dependencies and you know they all need to be packaged into RPMs, they all need to go through the review process and it's a pain. It can take a very long time. In some cases I've seen it can take days, weeks, sometimes even longer to get all that work done. So a container based system means that there's less concern about packaging where we can you know just say you know this app comes from Git or this app comes from you know some source that is not an RPM package. The only concern or one of the biggest concerns is remaining is the is that we still need to be concerned about which license the app is or its dependencies are. So when a package is reviewed as part of the review process that we get for free we know that it's free software we know that the license is valid for Fedora. If we skip the packaging process and just containerize everything we have to be more aware of all the licenses of the software that is getting pulled in. Some advantages for the application developer again OpenShift allows us to tie into existing continuous integration continuous deployment pipelines. So you know if you have say Jenkins that ties into your software that you're developing Jenkins can trigger an OpenShift deployment of that software or you know if you push to get that can trigger a redeployment. There's many different ways that deployment can be triggered. There's less to worry about with regards to deployment of scaling. So I know more recently I've seen more requests come in for you know app owners say you know hey we need another front-end box you know we want to scale more we want redundancy and they have to depend on the courses that I'm in teams who spin up a VM configure the Ansible variables and you know do all this manual labor whereas with OpenShift you know it's just a command line you know you run the OC and you know you give the parameters to scale up the application for you or you go into the web panel and you click the up arrow that says hey I want I want more redundancy for this application and that scaling can also be based on a number of triggers so if you're running an app and you know for some reason the app you know gets to the reddit front page or something you know we get a lot of traffic coming in. OpenShift can automatically scale up the application to handle the load as necessary. So again this to some extent this avoids relying on sysadmins to spin up VMs to do you know a lot of manual labor. So it's quicker for app developers you're not waiting on us to handle your request. On the sysadmins side again it's easier we don't have to to spin up VMs for you know providing redundancy we don't have to do as much work and hand-holding with the app maintainers to to scale our app. Security concerns go away a little bit because the app is is containerized but this does ignore things which I'll talk about a little bit later such as you know when when apps need to access to other hosts in the infrastructure for example database servers which we don't currently run in OpenShift. So how are we deploying OpenShift apps so far? We wrote a kind of domain specific language DSL like set of roles in Ansible and this is independent I know OpenShift 3.6 now comes with some Ansible integration built in. We have not looked at that yet but we probably should at some point right now both our staging and our production OpenShift are still on 3.5 but going forward we should look at this and see how what we've done with Ansible compared to what the upstream is done. But the idea is the idea with what we have so far and our current use of Ansible is you write the OpenShift .eml files like you normally would for you know if you wanted to deploy your app on any other OpenShift cluster you write your objects your routes your deployment config your config maps etc etc you stick those into an Ansible role and you write a very small playbook that just includes those roles with various configuration parameters I'll show an example on the next slide and you run the playbook and the playbook is Ida Potin you can run it as many times as you want and it will update the configuration automatically. Behind the scenes it's really just calling ocapply-f on the the master's and applying the Yeml configuration for you. So this might be a little bit hard to see but this is an example playbook this is from one of the apps that we've deployed in our staging infrastructure on OpenShift and the beginning of I mean the stuff the top is fairly straightforward most of our playbooks have that. The more interesting part is the the second half the roles and you see you know you just you you include through the OpenShift for example the OpenShift object role you give it the app name which is kind of acts as a unique identifier within OpenShift and you can pass it either a template or a file if you pass it a template it will it'll execute the template like a normal Ansible template if you pass it a file it'll not and again behind the scenes it's a really just calling ocapply-f commands on the master's and the roles that we have like a OpenShift project, OpenShift object, OpenShift start build we've just been adding these as necessary so there are probably more oc commands that we're going to need to add as projects start making more use of this but we're just adding them as necessary. So what is not already what do we have set up already? We have the Ansible roles that I just talked about. We have a staging deployment of OpenShift and we have three apps deployed in it currently. We have modern based we have waiverDB and GreenWave. We have a production instance that we set up literally two or three days before flop and in there we have GreenWave and waiverDB set up and again this is all using our Ansible set up that I talked about. And so what do we have to do? Well again like I said we're running 3.5 in both staging and production right now we need to upgrade to 3.6 see what new features are there look at the Ansible integration look at some of the other. We want to deploy an instance of OpenShift inside of our infrastructure cloud and the idea there is that if you are an app developer if you have this cool idea you just want to experiment this gives you a place to play with OpenShift as a contributor to Fedora that is not connected to our stage or production environment. So that would be not necessarily a free-for-all but it would be you know more you know just playing around getting a feel for what OpenShift is you know seeing how your own apps work with it and so on without necessarily touching our staging or production environment. So that's one idea that we want to do. To decide what policies are going to come along with the use of OpenShift this is all very new so there are parts of our RFR process that are probably going to change. I'll talk a slide on you know some open-ended questions this is something that it's going to have to be discussed going forward as we start making more use of the OpenShift cluster. And we want to move over more of our production web apps to this once we know that it's working well and you know once we are more familiar with it we want to get modern pace the production modern-paced moved over a couple other small apps like Ipslon was FedoCal, Elections, maybe NDAPI. You know just some of our smaller apps just at least for now just to see you know what happens and take advantage of OpenShift what we know is working well. So a couple of open-ended questions. Right now there is no persistent storage in our instance so for example databases they are not part of the OpenShift system they're not in their own containers all of our apps that are both in staging and in the production instance are using our external database servers. When last we looked the storage options for network storage were less than ideal they were complicated I'm not sure if 3.6 might have more options there but that is something that we need to work on as something if anyone has input on that we'd love to hear it. Again which policy is there between RFR processes you know how do we enforce what would we do with the whole if we don't allow or if we allow things that are not RPMs to be running inside of the canors you know how can we enforce the the license that is yes. What is RFR? Request for resources so when someone wants to run an app in the infrastructure they file an RFR it's a ticket on the infrastructure tracker and they kind of explain you know how are they going to maintain it who is going to maintain it what is the purpose etc. Who has access to which parts of the web panel if any versus the command line tooling do we want to allow build configs in other words we want people to be able to kind of use their own Docker files and containers or do we want them to have to pick from a pre-approved list do we want to allow auto deploying when get pushes happen or do we want everything to again have to be packaged and if we if we do allow things that are not packaged we deployed do we want to start being more liberal in terms of other languages other frameworks that we're less familiar with as part of the system there's still a maintenance cost there but deploying it would be potentially easier so these are some things to consider so these are a couple of resources if you're interested in learning more about OpenShift the first link is a YouTube channel by the Red Hat OpenShift team that has a whole plethora of videos where you can learn more about how it works the second page is a weekly link and it's fairly outdated but it has you know our initial idea for why we're considering OpenShift why we're using it the third link is the OpenShift docs which have been very very useful for me personally and the docs the docs are very good for OpenShift so yeah I will take questions it's yeah so we have two instances right now we have a staging instance and a production instance they're both running 3.5 right now there is a 3.6 that exists we have not set it up yet so we need to test upgraded to it and see what new features it provides you mean like the the web panel or you want to deploy an asset if you're looking to just kind of experiment like you were saying that's we do want to set up a an instance of OpenShift in our in our fedora infrastructure cloud and that'll be more for like that's not this that's what I was trying to get right that's a separate thing yeah that'll be for that purpose just getting in playing around and getting a deal for it if we don't require the things are packaged you know how do we push people how do we push the app owners to keep keep the app and all its dependencies up to date and you know security updates come out you know how do we tell people hey like keep your stuff up to date that is an open question I don't really necessarily have the answer that that would be on the app owners it is yeah it's no it is a concern it's something we need to discuss that's Kevin can probably answer that better than I can he set up the