 Well, welcome everybody to this OpenShift Commons briefing with Dan McPherson, one of my favorite folks on the OpenShift engineering team. And this, as I was saying earlier to Dan, this is something we should probably repeat every quarter as new people come on and old people forget some of the tips and tricks for using the different development tool chains for OpenShift. And it's really key, I think, to participating in the OpenShift community to understand all the different tools that we use in the development process and life cycles. So I'm really pleased to have Dan McPherson here today to walk us through some of them. If you have questions, put them in the chat room. We'll try, if it does, if there's a pause in Dan's talking, I'll try and read them off. And then we'll save most of them for the end of the conversation and I'll unmute folks and we'll have a conversation. Without further ado, here's Dan McPherson and let's start navigating. Okay, thanks, Dan. Yeah, so I'm one of the lead engineers with OpenShift. I've actually been working on the product pretty much since its inception. I'm going to try to give you guys a rundown here of basically what it means to be a contributor to OpenShift, both from a how do we take requirements as well as how we build the code and actually in the end deliver the product. So we're going to start with planning. I will warn you guys, I'm going to be jumping all the real place here going back and forth between charts and various, showing you various of the artifacts and pieces that we use to build everything. And if I get too fast, just yell, I'm going to try and slow down. Okay, so again, starting with planning, we use Trello for our planning. We picked Trello a couple of years ago, largely because we use an agile process where we use three big sprints. Trello is a pretty generic tool and fit nicely for that. But primarily Trello enabled us to be able to share everything we do externally. We're an open source company. We like to make everything we do public, and so it was a good fit for us. So actually, I'm going to actually show you guys that now. So there's a few tools here in Trello that are worth talking about. So when everyone can actually see our Trello publicly, and it's just that Trello.com slash Atomic OpenShift. And this is actually what it looks like whenever you are coming externally even without an ID, I'm not logged in here. And there's a couple of boards here, a bunch of boards that are interesting. There's a couple of concepts though. So we first have a roadmap concept. That's what I'm showing here. And this roadmap concept has basically two main things. There's an epic backlog of all of the big concepts that we're working on, as well as a number of lists of what we previously have accomplished for both the 2x time frame and the 3x time frame. And as far as the backlog goes, we try and maintain this in a rough priority order, although I will get the caveat of, given the major rewrite we just did for 3.x, the priorities was pretty much a matter of time, we had tons of very, very high priorities. So you will see this every time. Go back to more of a here are the next five or six things that are at the top of the list that are most important. And so for each one of these items, it has two main lists in it. One list of scenarios and one list of future scenarios. These lists tend to match up to things that we are working on for the current release versus things that are at some point in the future we will work on them. And there's two types of things in this list. So you have links to other cards. And these cards are on other boards that a given team is working on. And then as well as things that are added in here directly. And at some point we will, with the items that are added directly, we will come in and convert these to a card and move them to a particular team board. As far as the ones that are linked to team boards already, those are actually automated so that whenever they are finished, on a team board, they will be automatically checked off on this list. So we don't actually add the items in here. They're added in automatically with a set of scripting that we've built around Trello. Okay, so that's the roadmap board. Now I'm gonna show you one of the team boards here. The team boards we have a lot of teams and each of the teams run through roughly the same process. This happens to be our user interface team. And the way this process works, again, we do sprinting and agile. And so we basically move everything in here from left to right as things go through the process. So as new requirements come in, they'll be added to the new list. These requirements can come from engineers. They can come from product managers. They can come from someone getting an external request and coming and adding it into Trello directly. And so again, they start on the new list. We do backlog grooming once every sprint for each of the teams. At which point the teams will go through the items in the new list, decide whether or not they are well-formed enough. And if so, move them into the backlog. And the backlog for each team we maintain in a rough priori order. The next list is basically the top of that backlog of the things that we're planning on working on for the next milestone. And then once we get past the next, we start working on them to move into in progress, complete and accepted are various stages of whether or not it's gone through QB and then everything after that is a list of previous sprints. So that's a team board. I'm going to go back to the presentation. Actually, I will mention before I do that, we do have a summary of this as well. We host a summary off of our CI environment. We built this basically for an easy way to get a high level summary of everything that's going on across all the teams. And so there's a few pages here that are interesting. The roadmap overview is essentially a list of what is on the roadmap board. I just showed you just in a slightly more verbose format that lets you see everything all on one page. And then if you want to get an idea of what's happening at a team level, going back in time as well, we have the sprints overview, which for each team here, there's a list of essentially what's currently happening, what has happened over the last few sprints. The previous sprints overview is basically the same thing except for for going back further in time, but it's not refreshing, right? Oh, never mind, it's just because that team hasn't been around quite as long. So you can see here, the team goes back as far as December of last year to get a long-term history. Actually, the last one I'll show here is the sprint schedule. So if you want to get an idea of when our sprint boundaries are, you can always refer to this page. And again, they're three-week sprints. This basically just shows projecting out through time when those sprints will end. Okay, I want to get back to the presentation. Okay, so next up is the development process. We have a lot of options for actually developing an open shift. The primary two that people actually use for doing development are either using local development, in which case you have to have a machine which can run, get, go, and docker. And there's a link here to the specific instructions to get that set up. And then you can also use Vagrant. We have Vagrant with VirtualBox and Libbert images published. And that basically lets you do development against any local system. And I would say even amongst our development team, we probably have a roughly 50-50 split between using either pure local environment running docker locally or using Vagrant. And actually, I'm going to go back now and do a little more demoing. So actually, what I'm going to show here is a really quick example of how simple it is to get up and start it with the Vagrant. Can you make your view much, much bigger? Yeah, sure. Thanks. That looks great. OK. So yeah, so I am inside of the open shift origin Git repo here. And we have a Vagrant file checked into that. There are actually more advanced versions of this using a plugin we created called Vagrant OpenShift. It's not a requirement. So this is going to do a Vagrant up, which is why do you fucking use Fedora? And we also have CentOS images published as well. And so what this does is it will pull down the images if you haven't already pulled it down, which I have for purposes of this demo. And it will actually melt in your source locally if you have a source locally. If not, you can use source that's on the VM directly. Most people tend to prefer local. Just tends to be how most developers want to work. And so yeah, so in this case, it's already started up here. It's actually showing here it has melted in the source. And so what I'm going to do is SSH to that instance now. And I'm actually going to start up OpenShift. I'm just going to run status on that to make sure it started. OK. Does appear that it's running. And at that point, we actually do port mapping, which it showed whenever it started up, port mapping of all of the interesting ports from the instance to your local instance. And so if I go back to the browser here, it'll look closed. It will actually have a console running. And so, and this is just OpenShift running inside of that VM, port map back to my local host. So pretty simple. As far as making changes, like I said, most people would mount the source in, make the changes, and that point can run builds and everything is all local. So it's a pretty simple setup to get going. OK, so a few more things as far as source code and why we're talking about it. So again, our primary repo here is the origin repo. There are a number of secondary repos for building images and sourced image and a few other external secondary projects to origin. The origin is the primary one. The origin is what we embed Kubernetes into. Most of our actual code is under package within the origin repo. And as far as Kubernetes goes, like I said, we embed Kubernetes in here. I mean, it's used in two ways. So we expose Kubernetes directly. So basically everything that's in Kubernetes is exposed directly through OpenShift. And it's also the case that we use Kubernetes as a set of libraries to build our code on top of. And so, yeah, so if you look under Workspace here, you can see the actual. That's the entire Kubernetes code base, which is embedded inside of origin here. And we tend to rebase on or basically pull in that dependency roughly once a sprint. So roughly over three weeks. Now, I mentioned the two main ways of developing here using either local or using the vagrant image. There's a third way, which depending on what kind of development you're doing may be relevant, which is we also publish a Docker image of origin to Docker Hub. And so you can always just do a Docker run on the origin image. I guess it's Docker run OpenShift slash origin and be able to do development with OpenShift. So for example, if you're building images to run on top of OpenShift, that's a very, very simple way to do it. All of them are still valid if you want to build images. But running the Docker image is probably the simplest. And actually one last thing while I'm in here, as far as the code base goes, we do also do releases of origin. So basically every time that we are spinning a release of origin, there's usually an enterprise release that follows it fairly soon after. The version numbers don't match here, but you can get a rough idea of what's going to land in the next enterprise release by seeing what's happening in the origin release notes. Actually the presentation. Okay, so next up, we certainly accept pull request. We have a number of external contributors now to the origin project. And it's the case that you can always submit a pull request for anything you want to see changed. It's also the case that you can submit issues for things that you want to see changed, but don't necessarily have the time or skills to make the change yourself. Generally, I would recommend that if you want to make a change, then you probably should ask yourself, are you willing to invest the time to make the change first? Or do you want to discuss it? If it's a really small change, then you're probably fine just submitting the pull request with a change and discussing it as part of that submission. If it's a really big change or some sort of re-architecturing or restructuring of the code, rather than then go through all the work to make a change first, I would probably recommend submitting an issue to start against the origin project and discussing it there. And usually that will wind up in a solution that everyone is happy with. So next up here is actually what happens once you do submit a pull request. Whenever we started the OpenShift project, we started with a relatively small group of five or six people, roughly five years ago now. And at that point in time, we did what a lot of folks get away with as far as continuous integration, which is those five people work pretty closely together and they submitted code. And after the code was submitted, a bunch of test cases got kicked off. And every once in a while, the test cases would break and it would be pretty obvious what changed and what we would need to go fix to fix it. And at some point in time, we realized that once we got to 10 or 15 or so people, it was the case that it was always broken because amongst those 10 or 15 well-intended people, someone at some point in time was always going to be breaking the code base. And so we were always in a state basically of the CI system was broken and we had to debug which of the 10 to 15 people making changes all day long broke it, which changed whether and how do we undo it or fix it. And that became a pretty tedious process and was a pretty big effort on everyone's part to maintain it. And so what we sort of do that is continuous integration 1.0. And so we needed to move beyond that. And so there are some systems now which are relatively prevalent like Travis CI, which have also sort of moved beyond that. And that's sort of what we consider CI 2.0, which is being able to test the code changes against the pull request without actually having to merge it. And so we ourselves developed a system to do this very closely integrated with GitHub. There are other systems as well like Garrett that give you a way to do this. However, whenever you start using the system like Garrett, you somewhat have to divorce yourself from a nice GitHub pull request model. So what we did was we developed a relatively simple system. And I will say you do not need to read this entire diagram here. There's a lot on the page, which all of this is still exactly what we do. But for the purposes of getting involved, you do not need to understand that much complexity. So again, there's systems like Travis CI, which have this model of you submit port request and they go test your code whenever you submit the port request. There are some downsides with Travis CI in addition to you can't run everything. So we can't run Docker actually with Travis CI yet, which limits us on what we can actually test. The other big issue with Travis is that it does not handle a queuing mechanism. And so there's no guarantee that you won't have two people submit code changes that conflict. It also doesn't give you anything more than a point in time. So if someone submits a port request and it sits there for a week, someone might come along a week later and say, OK, it passed the test. I'm going to merge it. But that was a week ago and stuff may have changed since then. So that if you actually were to run those tests again, they would not pass again. So to be fair, though, we actually did develop this even before Travis CI was totally in place. We actually do use Travis CI now, which you'll see on some of the coming pages, has sort of a first pass. But we do more than that to guarantee that code actually makes it into master is actually still correct and valid. And so again, the way we do that is we have a merge queue. That merge queue test emerges one thing at a time. It guarantees that all of the test cases actually pass so that you know that there was no delta other than that one pull request. And we completely tested that pull request with what was previously on master. And basically you have to wait your turn in the queue before you actually get merged. Now, we don't have a list of trusted users who have merged permission. And we'll talk a little bit later about testing as well, which is a more expanded list. We actually make it so that almost no one actually has real push permission to our repos, which means basically it's difficult for someone to even shoot themselves in the foot and accidentally push something to master. So it's a system basically that we develop to make sure that we do keep a stable code base at all times, which means that whenever we expanded even beyond the 10 to 15 people, we always know that the code that's on master is still valid, which keeps all of those people productive. We didn't build a system with Jenkins as well as a decent bit of scripting that ties together with Jenkins to provide the functionality. Okay, so this is an example of pull request, or at least the bits of the pull request that are relevant here. And this pull request is relative to the common, right? So you'll have someone come in and review a change. And whenever they review the change, they will first go through a series of comments, making sure that things are okay. And once they agree that everything is good, then they will add this merge tag. So you can see the merge in square brackets here. That indicates to the bot that the bot should go and test the change. And then you see the bot commenting back. So the bot will first comment and say it's running the test cases. And then at the end of it, if it succeeds, it will actually merge the code. And you see here, in this case, it did succeed. And the bot actually merged that commit in. There's actually two things here that are printed. One is from Travis, and the other one is from our bot. And actually the bot hasn't actually updated this yet. But eventually the bot would check mark here as well. And I mentioned we also have test. Test is something which quite often people will add before they add merge. So a common use case might be a person would see a change come in, or pull request come in. And before they go, take the time to fully review the changes and make sure that everything's okay. They'll first add a test tag just to know like does it still function. And before they go spend all the time to review it, it's useful just to make sure that the person submitting the code actually has a valid change before it's reviewed. Or in actuality, our own developers even will often just use this to validate their own code. It's easier in some cases to submit the pull request and have Jenkins run the test cases than it is for them to run all the test cases locally. And in this example actually that you're looking at, it's actually the case that the bot actually asked for it to be tested at the time. So that's a way for while you're waiting in the merge queue for something to actually get merged, the bot will, if there is a queue, meaning there's more than one person currently in it, the bot will ask that we run tests against it before we even get to the merge queue. That way, if you are fourth or fifth in line, you'll get early feedback to know wherever you get to the top of the queue, do you actually have a chance of passing? Okay, so I mentioned before we do use Jenkins for this. You don't necessarily have to be too aware of this. It is a public Jenkins, so all the links that we put in the pull request will show up as something you can click on. All of them run through a couple of jobs here. So they either run through merge pull request origin or test pull request origin here. And so for each one of these, you can see that it's going to report success or failure. If it does succeed, it's going to put a link here to the pull request afterwards. And for each one, you can click on it and see the logs of what actually happened. Again, this one passed. So very last thing it did was update the particular pull request. So if we actually click on that pull request here, you can see the bot actually reported back that it was successful. And in this case, someone six minutes ago actually just added a merge tag. So now it's actually in the process of getting merged. All right, let's go back to the presentation. Okay, this is sort of extra credit, but just so everyone's aware, we do also have a set of extended test cases. Extended test cases are, let me actually start with, it is the case that test emerge would try to be pretty rigid about how long they take. So we generally like for it to be less than 30 minutes. We tend to get pretty angry when it's over an hour because no one wants to wait very long for their code to get merged. And so generally it stays somewhere in between those two boundaries because of that. And so currently we're in the process of trying to get it down and trying to add more test cases to it, am I adding more parallelism to it? But it's always gonna be the case that we have some set of test cases that take too long to run. So a good example of that would be a reliability test case. So a test case that says, I'm gonna drive the system for two hours and make sure that it stays running constantly in a good state for those two hours or take a longer time period. There technically is no way you can run that in less than two hours. And we can't wait on that for every single merge to happen. And so we fit those into additional buckets of test cases that are our extended buckets. And it is possible to request those extended buckets to run as part of tests, but not merge. And so the general flow would be, someone would submit a pull request and we would say, okay, this is a risky enough change or we know that we have test cases, but they are in extended. And the reviewer would add an additional command here, directive of extended colon and then the list of buckets. You can add multiple buckets, comma, separated. In this case, they actually ask for all of the test cases to be run, all the extended test cases to be run. And like I said, we don't block on these for merge, but you can ask for them with test. Your test will then take longer to actually happen. And then it's the case that after each merge, we kick off an additional build. I mean, the official build basically for the QE team to look at it. And that official build is after official build finishes, we then run the extended test cases to make sure that those are continuously run just after the fact. So it's sort of the model that we don't prefer that we originally had that I described before of, they could break because we're not guaranteeing that we run them ahead of time. Unfortunately, we just don't have a way to make every single test case run instantaneously. So it's just reality, but at least we limit the amount of test cases that fit into those buckets as much as possible. Okay, we also do our documentation publicly, so I'll show that real quick here. So this is the same documentation that is generated from the same code that ends up for the origin and enterprise and actually only as well documentation. And so this is the origin version of it. It's docs.openship.com for the enterprise version of it. And all of that documentation is generated from this Openship Docs repo. So it's something that we like being public because it means that it's very easy for our developers as well as external people to open issues or poor bust or whatever they, wherever they see changes that need to be made. It's very easy for anyone to do it rather than hit them behind some wall. Lastly here, actually I will show a little bit more Jenkins. But the last slide here at least, I wanted to mention was we do use two mechanisms I mentioned GitHub issues before for being able to open bugs. And we also have bugzilla. So both of these are valid. We treat them relatively equally. So this is our issues list in origin. There's a lot here obviously. And if you want to open bugs against bugzilla, it's under the OpenShift Origin or OpenShift Enterprise product. And just make sure specified version, we will change it if you get wrong, but try to specify the version you're appropriate to which one you're asking for a reporting issue or asking for a feature against. And we have time here. So I will go back and show one quick thing here in Jenkins, which is one of the things actually in that flow chart that I showed before is we also have the ability to do something that we call 4KMIs. So 4KMIs are one more variation on us being able to run our test against code before it gets merged into master. And so a very typical use case for 4KMI is we've made a major change to the code base and we may be able to pass all of our test cases. But things like the reliability test, things like performance testing, more rigorous testing that our QE team may do. We don't have a way to get to those except for 4KMI. And so we basically developed the 4KMI system as a means to be able to be able to submit your code to your branch or your fork and GitHub. And then we can create a 4KMI which creates an official build that our QE team is able to do a full regression against at that point. And so no matter how big the change is, it's something that we never want to risk putting risky stuff into master. And so if you ever do submit a change and it's something where it does have some amount of risk to it, you may see us go through a 4KMI process with it. You say we do this ourselves on a pretty regular basis. And so it basically just means that the change is risky enough that we do want to allow our QE team to do a full regression, essentially, against it before it does go into master. That's probably the last major piece that really is relevant to external contributors. Okay, so I think that's the majority of what I wanted to show. So I guess we can open up before here for questions now. All right, thank you, Dan. That was awesome and a lot in one sitting. So but I think it's a great overview and a great way to get started. I'm looking to see if there's any questions. If there are, if you can type into chat and I'll unmute you if you'd like to speak rather than talk, if not, then I expect you're all going to be experts on this and be doing pull requests, logging bugs and fixing them very shortly. We did get one request to update the OpenShift.org site from Chuck. And I will definitely say that we are working on a new version of the OpenShift.org site that incorporates the atomic pieces as well. So that's happening as we speak. Okay, so there's a question here about when should you open a bug against GitHub or get up issue versus a bugzilla bug? And the answer there is both are OK. We internally, our QE team tends to use bugzilla. Extra only more people tend to use GitHub. We treat them both equally from a triage perspective. So every single sprint, we go through the bugs in both lists to figure out what things need to be fixed versus and if they don't need to fix immediately, then they get prioritized into a list. So it's really a matter of whatever you're more comfortable with is is the right answer. And if you're very used to the OpenShift and OpenShift origin, Dan did point out that OpenShift and that is a great place to even start extending the documentation or asking questions to some more clarifications on pieces of the documentation that aren't up to snow. And we'll probably ask you for feedback on that. And get that going because the documentation is one of the key pieces of the project. So if you're looking for some place to start, join us in fixing up some of the documentation for the week. And if there's no other questions, then I'm going to really say a big thank you to Dan for doing this. It's very thorough. It'll be up recorded. So if you want to play it back and look over some of the things or see slides, the URLs, it'll be up probably first thing tomorrow morning on the commons.openshift.org site. So thank you again, Dan, for doing this. And I think I will try and make this a semi quarterly event because we keep having new members coming in the commons and we need to get up to speed. It's great. Thanks very much. Yeah, thanks, everyone. Take care.