 As is indicated there, my name is Kevin for anyone who doesn't know me. I work on Fedora infrastructure So this talk is going to be about our new communi-shift instance, which I'm going to talk about How it's set up how you can use it? What it consists of how we expect it to be used all that kind of stuff But I thought I'd give a little brief history of the the kind of things that we were That this is replacing essentially We have right now a private cloud instance that's running OpenStack 5 5.0. If anyone remembers that long ago And it's of course having lots of problems because it's so ancient and We have a number of instances there that are Community-oriented things we have maintainer test machines Coper run some of its stuff there. We have Just a lot of development instances for our application developers and We we kind of envisioned that cloud network as being a place where people could Spin up things experiment to figure out how their their applications were working You know test things get things to a state where they wanted to have wider usage things like that and It wasn't really that successful and I think we've learned a lot of lessons from that and Hopefully this this replacement will will be much more effective My presentation console is out of sync So we have this communi-shift instance. It's a OpenShift 4.1 instance and It's using a lot of a lot of hardware. So it's got a lot of capacity for things It is One of the reasons we went with OpenShift 4 as opposed to OpenShift 3 or another OpenStack something like that You may have seen some of the other talks at this flock about how the CPE team which I'm part of Or the Fedora engineering team as we used to be Are overloaded we have a lot of stuff to do. We don't have that much time We were trying to cut back on the things we're doing we're trying to focus on the things that will give us the most Bang for the buck. So one of the things we really wanted out of this is for it to be very self-service so people were not blocked waiting for us or You know asking us for things they could just dive right in and go do what they needed to do without any further Interaction and so OpenShift 4 for those of you who have not used it yet Is very hands-off as far as managing If we have time here, I'll show you because there's an update pending But basically when there's up an update you just click a button that says okay. Go ahead and update my cluster It does everything in the background. It moves your pods off of the node It updates the node it reboots the node and make sure it's working. It moves pods back onto it It does all of that and so that's gonna It it does work. It does. It's amazing It is magic. So what one of the things We really wanted was to have something that was low maintenance for us self-service for the people who are using it And so that people could actually get a lot of use out of it without us having to babysit it So how many people here know what OpenShift is have used it quite a few people good So I won't go too much into this but if you have never used OpenShift it's just basically a platform for managing your containers and You can stuff all kinds of things in containers as you might imagine your application can run in a container OpenShift has a concept called pods, which is multiple containers that are sort of able to talk to each other and perform different functions OpenShift gives you a platform that is Opinionated and is the same from a cluster to cluster you can export from one cluster import to another cluster. It's the same stuff And it's it's really nice because it's you can get started really quickly You can have it manage a lot of the heavy lifting stuff that the infrastructure That you would normally have to do yourself if you were if you were just deploying this to a VM or something like that So it's it's very handy for this sort of So here here are the use cases we kind of imagine for this cluster Proof of concept applications So you might have an idea for something and go hmm, you know I I think this is a great idea But I'd like to make sure it works before I ask people to help me with it that kind of thing applications to get your work done more efficiently at a good example of this is There's some folks who use Who build modules right now for Koji and do a lot of their work in their own Koji instances that they run elsewhere This is certainly a place that you could run that and Do test module builds things like that very easily application supported by The community that we are trying to not support ourselves anymore because we're trying to focus on things This is a great place to have the resources to stand up that application and Get a team or a group to work on it and make sure that it continues to to function and provide value to the community and It's a generally a nice way to for people to learn about OpenShift if you don't have experience with it or you want to to gain experience with it or See how it works There's any level of detail you can get to here sure and that's actually something that we've we've had in the past where Local communities have come to us and said, you know, oh, we want a website and you know place for gathering And we didn't have any way to do that, you know, we didn't have a general infrastructure for that kind of thing But yeah, that's absolutely something that they could run in in this in this instance So what do we support in this instance we support the OpenShift platform and that's it We want things to be as self-service as possible There will be outages at downtime, but we will try and keep that to a minimum as a minimum as possible I mentioned upgrades earlier OpenShift 4 is like I said has been really good at upgrades our Clusters running Red Hat Core OS as all the nodes So even the even the nodes are managed by the cluster manager by Kubernetes. So it does a really pretty good job about Processing through the updates and and just handling that for us So I don't foresee there being downtime for updates or things like that There might be up downtime for you know data center issues or network issues or things like that but there really shouldn't be that much in the way of maintenance and whatnot and So what does that leave for the people running things to support? Well, there's a wiki page that I have at the end that has a lot of this mention more in more detail, but It's up to you to back up your application which when you're using OpenShift That's pretty easy. You can actually export the the definition and if you have a database or something like that You'll obviously need to back up that database But it's actually pretty easy to do that kind of thing with OpenShift We definitely recommend that you put your Management of your application into Git you can use any Git forage you want Pager, GitHub, GitLab, your own personal Git server Whatever you want to use But if you use Git for your changes to your application that way you keep track of them You can roll back easily. Other people can contribute new pull requests, etc, etc And OpenShift is really built around that sort of workflow. Your application can build out of your Git repo You can actually set things up very Automatically so you don't actually need to mess with the OpenShift side of things anymore You know you just do a commit to your Git repo it OpenShift notices that there's a change rebuilds your source image Rolls out your your new application and there it is and you actually all all you had to do was commit that Git commit Another thing that's very important with this setup is your application needs to have a way For people to know who it is that they can contact if they start using your application and it goes down We're not going to be able to do anything about that. We need to know who to refer them to so We would definitely recommend you have some way of Referring them to you email address, etc, etc. Yeah, Tim Yes Yes, so that the question was is there persistent storage and the answer is yes and The second question was if you're responsible for backing up that persistent storage and the answer again is yes And we do have it. It's an NFS storage and we do have it set to not Reclaim so like if you got if you accidentally deleted your app or something like that That storage would still be there for a while until we manually went and reclaimed that drive that claim but You know, you shouldn't count on that. You should actually back up that information. So, right Right, correct. And the reason we thought about it You know providing a backup service or we even thought about providing database services things like that but then we get back to us doing a lot of work and We're already overloaded. So it's like, okay Do you want rawhide composed to be fixed today or do you want me to restore the data for this application? So we wanted it to be pretty clear that it's it's up to the user to to back up that data or not I mean if you don't want to back up your data and if you want to reproduce it If it depends on what your data is I suppose Huh No, it's it's just NFS right now, although I haven't run into that problem with that, but maybe it's still there. It's Yeah, yeah, absolutely we can we can look into that I think it depends on what you're storing I mean if it's a database then obviously you just connect to your database and do a dump but Right absolutely and Yeah, we can look into that and see what the best best practice is I think There may be a way to Get it exported or right You could maybe do it with another container that does stuff or possibly even just Some sort of OCR shell script thing. I don't I'd have to look but you're right. You're right. That's that's a good point Yeah So the question was would would we provide a platform for the backups and I don't think we plan to do that now I mean, I guess if there was a great demand we could look into doing that Correct. Yeah, and I was having a later slide about that But yeah, one of the other things is we're thinking of this as sort of a an incubator so like if somebody brings up an app and they get a team and they start making this app great and You know They It gets used by a lot of people it becomes central to whatever and it's it something that is in our mission So it's like something that helps with the build system or something like that Then yeah, absolutely. We could look at you know promoting it into a different area Yeah, open ship the question was logging shared logging and OpenShift does provide For itself a elk stack type of thing But that is just for the sort of the infrastructure so it would like log That your pod was crashing or something like that. It wouldn't be your application, but you can instantiate your own Logging stack, obviously you would have to store that data somewhere Okay, so the question is is the is the cluster manager view available We haven't talked about that, but I don't see any reason why we wouldn't make that available Are you seems? Good others question back there Yeah, so I've got a slide on on quota actually right right coming up. So let me let me get to that So this is just briefly talking about access right now or what our plan is is to have a fast operator or an operator that syncs fast groups and We'll just add people to a fast group and then it'll just Create your account or give you privileges to self-provision Right now that is not done yet. So I would ask anyone to who it wants access Let me know send me an email I will write a list and then after flock I will sit down and just add everybody and then you can log in We're using the Fedora IDP So it's it will use your Fedora credentials to log in and it will by default everyone will be a self-managed or a Whatever it's app creator or whatever that role is so you can create your own apps. You could create your your own pods if you need for some reason if you want a shared app if there's some app that You know multiple people are wanting to work on you don't want it in a particular your particular namespace something like that Just talk to us and we can set that up fairly fairly easily or move an app after the fact if if need be We're hoping or I'm hoping to do more widespread access down the road like Packager or QA or you know just open it up to all those groups and just you know see what what people want to use it for So quotas this is kind of the the low ball quota that we were thinking of but We could certainly change it. It's five projects ten pods five volumes something like that Like I said, we're open to changing this it just depends on how many people are interested What kind of apps people are doing that kind of stuff? We could also Either look at bumping this Totally or on-demand here Right, it's multi. It's multi project. These are multi project quotas. So No, it's ten pods total. Yeah, right. Yeah, we could We could put those restrictions in place but I was just going to start out at least not doing that and just You know if somebody we reserve the right if you're disrupting things to destroy your app, but Hopefully people are good about that and just you know use as much resources as they actually need also, I think that a lot of the well the CPU and and Some of the other resources are are kind of strange in OpenShift to deal with anyway, so So the future I've actually got a Cubevert CNV 2.0 installed and I've been playing around with it For those not familiar cube vert is a way to use OpenShift to manage VMs instead of containers Unfortunately, I hit a bug in 2.0 that makes it not very useful for anyone So I'm hoping there's a bug fix release out soon that will fix that as soon as a that's useful enough to where we can We can use it. I'm going to try and move all the stuff from our old cloud off So that we can decommission it And most of those things don't need a whole lot. They need you know SSH access or at web access things like that So as soon as they fix the the masquerade issue that I hit Hopefully we can move those. Yeah, Neil. I Don't know if it does nested or not, but basically what it does is it? Creates a pod that runs Libvert and runs your VM with the definition that you gave it So it may be possible to just to define that in the in the config I don't know Yeah, yeah, I'm not sure Right, I think the only thing that prevents that is you may need to get permissions to the kbm device or something like that Yeah, right Right. Yeah. Yeah, right, right and it's The VM stuff is is very flexible to there's different sizes and you can you can pick your image And you can say what the storage is that you want associated with it and blah blah blah you can use cloud in it Etc. If if you really want Wide availability as I was saying earlier I hope to open this to like some general Fedora groups so we can get people just testing it out and Playing with their apps So this is something that we have discussed but we never came to a conclusion on and we probably need to figure out it's been suggested that we we have some kind of heartbeat or Periodic check type thing so that We know you know if you start an app and you run it and then you leave That this app is you know, not really maintained anymore. Nobody's using it Etc. Etc. But it's kind of difficult to know what the best way to do this is so if anyone has suggestions on On how we should do this. We're all ears. So Yeah, that's Right for the recording that the suggestion is that we leverage the alert manager built into OpenShift to require app owners to do something or Prove that they still exist Periodically. Yeah, that's a good idea. That's yeah, I mean we thought about Looking to see if it had any traffic or you know Mailing the app owner but again, we want to avoid any kind of like manual stuff I don't want to oh, it's six months. I've got to go through these 50 apps and mail these people. No bad here Well, the the Kind of the top-level requirement there is that it's Fedora related. So People should not run their personal email server. No crypto mining No, you know, none of that stuff. So No So here's some links, there's our wiki page with more information We also have a hackfest Sunday morning So if you want to catch me and ask me questions about it or say that you're interested and want to log in Just you can definitely find me there and if you have any ideas or feedback or whatever The Fedora info list would be the best place or tickets if there's there's something you want to change. Yes Yes, actually, that's a good point. We we have set up the So open here for has operators that can do all kinds of fun things and we've we've set up a SSL cert operator so it's actually got let's encrypt certs for all of its stuff and Any application that wants to be under that domain, you know apps dot OS dot Fedora and for cloud Whatever will just be under those Those certs if you're doing a VM or you're doing something with an external IP or something like that You can actually call that operator in your app. You could say hey, I want a TLS cert for this route And it will it'll handle it. It'll get the cert Renew it keep make sure everything's great. So yeah operators are Awesome, right right now. It's a flat domain. So it's it's the typical there the top part of it is OS dot Fedora and for cloud dot org and So there's apps and then username and projects On top of that. So it is kind of gets kind of unwieldy But if there's demand for different domains, we could look at that but I Don't know if that matters that much But there are there are external IPs that will probably be get used with the VMs. So we could put Things that want other Other domains on some other external IP and do a different for Yep Yep. Yep, absolutely. Yeah, and if there was like say there's a popular app that a lot of people use might want a different Easier to use URL So actually we have three clusters because we have a staging cluster a production cluster and then this cluster But but yeah, it depends Really our staging in our production clusters We're keeping for things that we're actually maintaining as a team the infrastructure team and The community one is for things that so it's a matter of who's maintaining it or who's who's the upstream for it or who? you know that that sort of thing so if Yeah, it's very easy to move it would be very easy to move stuff through especially if it's stateless Right, right and and like I said at the beginning I mean our our vision for this is that it it pretty much self-manages itself and barring You know some sort of hardware catastrophe. It's just gonna run. We're not gonna it doesn't need updates It doesn't need or it updates itself. So yeah, I am it I am lying with that. Yes, it does need updates So yeah, it's running for one eight right now and actually do we have a little time left? Maybe a minute or two 419 is out and it's pending so I could click the little button that says update cluster if you want Yeah Which of course it will since I'm the network might be not too great here though And I wait it just see there cluster update is available Update now it's not done yet so as part of this update it like updates the red hat core OS on all of the nodes reboots them all and Updates all the containers of all the stuff that is the cluster so it like pulls the new containers Does stuff reboots it make sure that comes back up does the next one etc etc Right it evacuates all the pods off of the nodes as it reboots them and moves them back just in the background Right, so so the question is do we have any particular way of deploying the production open-shift stuff? So that you can look to doing that later in the in the future Yeah, we do and it's really opinionated. So it is all in Ansible. It's in our Ansible repo. You can look at it The way it works is it does everything in Ansible. So Ansible is responsible for creating all the the project all the objects Everything so if the cluster burned to the ground tomorrow, we could run Ansible the and the app would Be redeployed if it did not have any persistent data We we don't story ammo files we use Ansible to to make the individual objects So if you look it's it's not that it's not the ammo. It's like the Describes how the gamel should be I guess. I don't know how to describe that But you can look at examples on our on our Ansible repo Those are not but those are not the bra if you dump that out of open shift That's not what you would get you you wouldn't get you know Ansible roll this with this. Yeah Right Right, right on our production cluster. That's not true App owners have only permission to their app and Ansible runs the playbooks and everything they do not have For this one. Yes, absolutely you you can use OC you can define your app Whatever you want to do there, but again, it's up to you to back up that application or manage it However, you want to manage it You know, I'm I would suggest to get repo and you know have some management there But if that's not what you want to do that's fine, but you know if something happens Then you'll have to recreate it Yeah, yeah, I believe the registry should be available Yeah, so again the the thought here is also that anything in here is open source You should not never put anything in this that is not What is that 18% complete? So if you want to look at more detail you can look at the cluster operators here and it'll actually show you eventually It takes like 20 minutes or something usually Three masters and eight workers eight nodes Right. Yeah, it does them. I think one at a time. So you can see what what things are already 419 And it shows you that So it's pretty cool and it's pretty nice that I don't have to do anything our production clusters is OpenShift 3 and there you have to and their rel nodes not coro s So you have to evacuate them update them reboot them run ansible OpenShift ansible It's actually a Dell what's to Dell FX boxes. They're a blade type system. It's just Yeah, yeah It's it's in our cloud network though at the same network that our our private cloud is in which is completely Separated from anything else there. It's it's like directly connected to the outside so that Section of the network could not conceivably It's physically impossible for that network to get back into the inside Which also means if you need resources from from Fedora project your Request go out and then back in and use the public IP space. So there's no there's no like Back entrance to like coji packages or anything there, but you can go around to the public IP and and access that stuff Yeah, yeah, let's wrap this up any other questions. All right