 Please to have with us Adam Miller who's one of the key folks on the OpenShift online team. He's an OpenShift release engineer on top of doing all kinds of good work on the behind-the-scenes keeping OpenShift up and running and being part of the OpenShift operations team and what I asked him to talk about a little bit today is the idea and the way we do DevOps at OpenShift online and I'm going to let him take it away and we'll record this and if you need to we'll be you'll be able to play it back And at any time that you're leaving here. So Adam would that you want to take it away? Yeah, great. Thank you So with the OpenShift comments, we're you know, we're very excited to be a part of the community and be sharing some of our lessons and hopefully You know hear from others and kind of collaborate on those kind of things one of the things we want to discuss was the process by which we work with the developers and work with upstream and how Everything makes its way into OpenShift online So what we'll go and cover in this session is how OpenShift online consumes origin How the online team contributes to origin and how code goes from development environments into production kind of the process but by which that occurs and Just kind of So however, we you know before we get down that road that you know the title of the talk had mentioned DevOps and DevOps is one of those terms that are very overloaded. It will change depending on who you ask I tend to like to default to the definition that is found on Wikipedia Which I paraphrased here It's a software development method that stresses communication collaboration and integration between the dev team and the operations team and We believe that in a lot of ways that increases Efficiency it helps your workflow and it can really kind of tear down barriers I wouldn't call it classic environments a lot of times the dev team would just write something We kind of wrote over the time the operations team had to figure out how to run it And if you know if the two teams are collaborating along the way and discussing and voicing one of those concerns So they can be taken into consideration It it helps in a lot of ways. So Moving from there how code flows In OpenShift, this is probably a diagram most people have seen if not this is how the OpenShift Project and products are structured. Object origin is upstream. All the code is upstream first Anything that goes into either the online or the enterprise product is upstream first Open source is very near and dear to our hearts on our team We you know, we live and breathe it and it's very near and dear to Red Hat We continue that tradition in the OpenShift line So all the code is upstream first and anything that comes downstream from there Is essentially shared back and forth from the online enterprise. We have different release cadences Online we released a production every three weeks. Whereas enterprise, I believe Average is about every six months ish release cycle enterprise releases, you know when they're ready With online we have planned releases and if features aren't ready, they just get cut So we kind of have different life cycle Cadences but all the code is always upstream first and you'll probably hear me iterate on that point and kind of hammer that As I talked through some other funds So I wanted to go over the architecture of OpenShift Because as we move through The rest of the presentation Development environments, which we refer to as dev amps and a dev amp is a single host. It's a one machine all inclusive OpenShift environment. It's running every component of OpenShift on a single virtual machine that's run For our purposes for online. It's run out in EC2, but it could be run anywhere really So inside of that development environment or that dev amp we have Our our Mongo storage. We have our authentication component We have our DNS and then the broker which you'll see At the top the broker is kind of the orchestration component. It is the point in which everything is keyed off of so When the client whether they're using the RHC command line tools or ID integrations or web Makes a REST API call into the broker the broker then will go through and Check authentication make sure that you are who you say you are and your position Want to do I will then Send a message over the M-collective message messaging interface to Schedule whatever task you need done and that will communicate over to the node the The actual task to be accomplished in the nodes in a traditional environment You would normally have a lot of nodes, but for the sake of development You can run that component on the same host and just run your gears there locally So over M-collective it'll tell the node to let's just say for example create a gear it will then Send that message over M-collective the node will receive that message and it will then go ahead and Set up that gear as it was requested and from there the node who can accept Get pushes from the client and then also web requests from the rest of the world and As that application is being created the broker also goes through and stores metadata Mongo database and also sets up your DNS entry so all of these components all of these different pieces all together on one machine is The dev amp and and it's for You know the use case of having a fully functional OpenShift environment on a single host that you can use as a point To start development on top of if you want to hack on OpenShift the platform itself Or it's also useful for developing cartridges and buying things that nature Okay So We we create these automatically we have a Jenkins Server out there that will actually orchestrate all this develop all of the Dell dev and creation It we queue them off based on you get commits to master that have been merged We queue them off of new updates to the base image so we have a clean OS that's relevant or Fedora that we go ahead and Well, actually we've also since Since since us has joined the you know greater red hat family. We've also added sent us to the mix So we have our base a mi base a mi and for those who don't know a mi is the image format or Amazon EC2 so we have our a mi and We register that with easy to launch instances from there Jenkins will we'll spawn a task and actually run a build and the build will go out and it will build the entire OpenShift platform from source or through our piano Do that installation Onto the environment and then we have a large number of post install Procedures that are done to actually set up the environments and register things and set up a demo like a demo user with some authentication things like that Set up some quotas for that user so they can actually go in and start creating gears And then that image is then re-registered again So that at any point in time one of the developers would like to they can spawn a new development environment And get the latest build with all the latest updates so the build workflow is Jenkins will clone the master branch from OpenShift They will then launch the base AMI And we use that base AMI to build the new dev app and then Like I said earlier it will do the I'm sorry It will then synchronize the code up to the dev app and then as I mentioned before will build the RPMs from source Create a local yum repository and it will yum install all the RPMs from there and then do the post install tasks very similarly to how somebody would install OpenShift themselves from a yum repository However from there it will also run through a number of Unit tasks rate tasks cucumber famous motor with a cucumber testing framework And we'll run all these tests and report results. So that way only known good Images are registered for developers to launch as development environments That is mostly because if in the event one of these builds is kicked off from a merge into master and somehow somebody got a bad commit in Because as much as we have bots and testing if if one of the few people who has the permissions to just merge things does so or Maybe some something new got merged in that it doesn't have a test case in the merge testing Just if it sneaks in and something happens these tests are done to make sure that the development environments We're in a known good state so that when developers launch one and start their tech their development We're not giving you time. It's already broken From there how development happens is a developer will go ahead and clone onto their local machine the code from GitHub from open to the origin and then they will launch a development environment and synchronize the code from their local Branch and we say local branch because our development workflow is if you want to add a feature or create a topic I'm sorry adding you User story or something like that you create a topic branch. We call them topic branches and You do your development there locally in that branch and get well, you can synchronize your local branches up to the dev amp that you've launched and then the the dev amp will on Synchronization actually run an rpm build and create that local young repository and do the install with your new packages and do an upgrade task And then from there The developer can run all of these rate tests or they can run a limited set Let's say you're working on a node feature and you want it to just run the node tests And there's also a set of node extended tests So that would be kind of the developer workflow from there the developer would put in a whole request upstream on to github and Then somebody would tag it with the literal open bar open bracket The word merge close bracket that would be the comment and we have to be by somebody who has the permissions to kick off a merge task and Developers who've been with the ownership project long enough to have been granted that permission and are you know trusted to be to be Doing that sort of thing so after it's been through code review somebody will tag it with merge and then the OpenShift bot which we call Rosie and culture why there's a pick up Rosie Jetsons who remembers that? So Rosie will go out and grab that commit Clone it locally in a Jenkins job merge That new patch locally Synchronize it up to a completely different disparate brand new pristine development environment And then we'll run it through the same round of testing that It should have gone through on the developer side, but we we run it through those tests again just to ensure that or To do our due diligence that back when it's done master So from there it will go through and if it passes those tests it will be merged into upstream and is now in so Once all the the code has been committed into master We have all our features in we need to get into deployment So for the operations side of the house what we do is Every day After the first week of the sprint we do I'm sorry after the first four days of the sprint we do Deployment for RPM package sets. We will actually farm out RPM builds into our internal build system if anybody's familiar with Koji Our internal build system is is based on Koji Well, we'll send into our build system and then we'll create our our young repository and we'll do deployments We just full deployments with all of our our ansible and puppet Tooling and I try to mention both of them because we do use both of them. They are both Very useful public is great for configuration management Ansible is great for certain deployment in order task execution So we use them together. I know a lot of people try to kind of Well in a lot of conversations. I have a lot of people try to compare them Directly to one another or consider them as competing technologies. We find that using them together is very useful and and just using The I guess for us, it's tool for the job kind of thing. So we have we do have public configuration out in Origin project we have a Puppet manifests and things that people can actually use to deploy object origin themselves We have some some ansible orchestration stuff that we're also working on that's out on GitHub as well If anybody wants to look at that A lot of a lot of the ansible stuff. I will admit those for I should be three which is Not yet, right? But I'm it's all up there. So please, you know, check it out and give us feedback that sort of thing But anyways, so we will do these deployments every day through integration And then our QE team watch it go through and test it every day and provide feedback and the developers you know they're they're bugs and they're they're just testing reports and things like that from QE team they'll fix everything and then every sprint cycle we have a stage day and we cut to stage and we actually do a branch in in the get our product GitHub repositories and The one thing I like to mention is this diagram Down at the bottom that has ownership origin and has a line and then there's Kind of a line that breaks off and that's our stage branch and that's a show that at some point we Basically clump the code and deviate now from what is happening master. However, any time that any Stage it is upstream into origin first because I'm sorry upstreamed into master first because master is supposed to never stop That is where continued development is going to always happen It's just that we have to we have to hit pause at some point and make sure that what we have is good before we do deployments So what we will do is we have these rpm package sets that once they've made it into stage And once we've done these deployments there They they're no longer like fresh build They instead of what we call graduate just for lack of a better term So they graduate from integration to staging and then again they graduate from staging to production once they've passed all the Labeled as such we do all of our once once they graduated in production We do all of our rpm signing to you know once we do our deployments We could do audits and things like that and production to verify that all the code out there and you know is Number one from us number two exactly what we expect to be there and then once we get deployed That's what's running in production. That's what's that open ship calm. That's what you know when you do a deployment out there using the you know the platform itself to launch your web app your development environment for Your I don't know PHP or no JS application pick your poison. It's that's what's running out there so What does our environment look like? In In easy to we have our rest API that hides behind actually to two layers of low balancers And we do that for a number of reasons, but oh it is It's automation but so we have We mentioned before that we were hosted in easy to so we use an easy to elastic load balancer and we have Multiple h.a. Proxy instances that sit behind the elastic load balancer and then we can actually rotate those in and out as we need to and One thing to note is actually we have load balancers the mix that are running rel seven Atomic hoax to beta we because the fact that it's all hiding behind the ELB and we can load balance across them we wanted to Start demoing some of that newer technology and and that's how we do it as we find we find points in the system That are highly redundant and we can we can rotate in a more experimental Component and if we need to we'll move it out. So I just I like to mention that because that's kind of interesting topic That's that's been coming up lately with the atomic host and doctor everything so we one of one of our load balancers in In production is actually running through docker on rel seven atomic beta Just the fun fact of the day so From there we we have a set of brokers. I have two in the diagram. We actually run for in production From there Mongo a lot of people will ask when when we're presenting our conferences or attack lead ups What are we doing to tune Mongo? How do we handle the load or the transactions that we get in a production environment as big as open-shift online? with Mongo and you know the question about tuning and and sharding and and all those things The answer is we don't that we have a standard Torino replica replica set It has been performant enough for what we do with it that we haven't we haven't had to we do some indexing of certain Collections, but that's about it. We don't you know, we don't have any sharding. We don't have a very Exotic configuration, it's it's pretty standard So just bring me you know running this internally or or you know on if there have any plans to Deploy open shift to three no replica set serves serves you very well So In online from there our brokers will call out to we have a DNS cloud provider We actually don't use our own DNS. We have a third-party provider And one thing I'd like to note is with online all of our code That we use it is either open source or we work on getting open source So one of the things our DNS playing that one is now up in GitHub for a while it was internal only because of I don't I honestly don't know I'm sure legal or something of that sort, but you know The work was done to actually get that in the open source. So it is now Everything online that is not it's only extensions. We it's only a plug-in or an extension It's never a core component that we add features to and just keep it inside It's it's always you know, it's always just kind of an add-on component that Is is that the way it is for now and then you know, so a prime example that is single sign-on So internally red hat has a science service. It was written in-house. It was homegrown There there's not a whole lot of motivation to go through the You know the process of getting the approval to open source the plug-in for that single sign-on That was written in-house because nobody else is gonna be writing that so it wouldn't be very useful for anybody else but It is just a plug-in for anybody who who is familiar with the authentication component of OpenShift It's it's just in the plug-ins directory. We throw it in there We add a config and it loads if we were to take it out. It would be the same So so our DNS is a cloud provider our single sign-on as I mentioned was in in-house From there the broker will talk m-collective and plug-in talks Online our back-end for m-collective is the active MQ message queue So we have a active active highly available active MQ cluster So when the message comes from the brokers it hits that it goes up to our nodes and and I won't talk about exactly how many nodes we have but there's quite a quite a lot of them and and from there the nodes will talk back and forth through active MQ and And That that is effectively what our production environment looks like and hopefully for anybody who's you know considering running this in-house themselves for on-premise has requirements or potentially for you know hosting another public class at Paz such as you know some of some of the members of of the Double-Shift commons community and the original community have done which is great You know we're considering those kinds of things. Hopefully this will Provide some amount of guideline of you know what what we have set up and how we handle how we handle production scale scale load and kind of things So at this point, I'd like to take questions if there are any I would be happy to answer them as I am able All right. Well, thank you so much Adam This that was great and a great overview of what's actually going on behind the scenes and like you said we have a couple of other Folks that are using Cisco's doing some stuff with the inner cloud and hosting OpenShift online Get up cloud is another community member So there's a lot of other people who are doing slightly different variations on this team, too So we're gonna put this slide show up But we'll also probably try and ask those folks to give us their Configurations and their feedback and best practices as well. So thanks again Adam for your You're taking your time today, and we will put this up and make it available shortly for everybody And if you have questions feel free to post them on the IRC channel at OpenShift or OpenShift.dev and Adam or one of the other members from the operations team will Make sure we we get back to you as quickly as possible and you can also post any questions you might have into the OpenShift Commons mailing list and we'll make sure that they get directed to the correct technical resources. All right. Thanks again, Adam Great. Thank you