 So, hi, my name is Dominic and today I want to show you how Cockpit works as a modular system. Right, Fedora is trying to modularize and that means different things, obviously. But I'm going to show you how Cockpit works with dealing with a bunch of different technologies and techs and APIs and how we survive as a project even though we're relatively small. So, our protagonist is Cockpit, I work on the Cockpit team for Red Hat and I'm only using as a case study to show you how we think projects like that can work. Mostly because I'm familiar with it, obviously, but it's always good to have a solid application when discussing abstract concepts. So instead of just talking about how it should be in general, I want to show you how we do it and then we can get into a dialogue of how that can apply to other different things. So as an intermission, I'm just going to start with a little demo right away, I think that makes sense. But we have, this is what you see when you log into a system, right, you collect your system with a web browser, you can just type in the username and this is the username from the system. You might have seen the log on the face already and as you can see you can log in. So what our goal is, is to make Linux and REL discoverable to a broader ad than audience and should make tasks in Linux more discoverable. And we can't look at all the features today, obviously, but we can see what the system can do. You can join in the main, you have system time, power options, you can look at services that are running, you can look at Docker containers, start Docker, it's not started, system logs, you can do storage, set of raids, logical volumes, networking, just a bunch of things. And all these little pieces are most of what you see are actually modules inside Pocket. So we have Shell, Pocket itself, and there are modules that perform a specific task. And each of those modules interface with different parts of the system. So we have modules working with modules basically. So that's why I think it's good to take away a few lessons from this. So back to the presentation, because in Pocket is a remote interface, a remote interactive admin interface. So what that means is, it runs in a browser, you can connect your system for anywhere, it's zero footprint, and you use it to configure a system. You don't configure Pocket, right? So if you log into your system via Pocket, and that's the system you see, Pocket doesn't have its own configuration. It doesn't have an extra state. That's just a system. Yeah, use a system user, and if you set up your system and remove the Pocket, it will work the same way it did before, although you can't log into the Pocket anymore. But it doesn't, I mean, some of you may know previous attempts to configure everything and control the whole stack. That's not what Pocket does. There are really great tools out there that configure a system. Some are usable via the CLI. Some have a diva interface or REST APIs. And what Pocket does is bring those things together. So what we do is we call system APIs directly from your browser. And I'm stating this again, just to drive the point home, you see it says clearly in the presentation tier. So you have really the lower level, the current level, you have LBMC groups. So you can say where the rubber meets the rubber, right? That's the name there. That's the nose level. Then you have the logic stuff that the sign was going to have. You have the system, the network manager, Kubernetes, all those useful tools. And then you have the presentation tier, Pocket, CLI, and other presentations you have. And those should work with each other and augment each other. And if possible, no logic should be inside Pocket. Because then it's not accessible by other means. And that means you're limited in how you work. And when you work, you want to achieve a specific goal. You don't want to work for a tool, but just use it to achieve something else. And to explain this a bit further, a bit more technical, we have the web interface on the client side. So that's a browser running somewhere on the operating system of your choice. We have the Pocket web service, that's PocketWS, that's network stuff, that interfaces with a browser. And that communicates with the Cockpit bridge on the system. And that bridge runs as the user that you log in as. So there's no demon that runs in the background, no other service. When you log in, that bridge spawns, it communicates, transits all the stuff. And when you log out, that goes away again. The web service can run in a container. And it doesn't even have to run around the systems. You can use one system as a gateway and connect to other machines. But that's more detail. And then the bridge forwards all the communication to the system APIs. And the bridge doesn't have to be aware of all those APIs. So that logic is actually on the client side. And the main means of communication, obviously, is SSH. You can't use SSH from the browser. It would be nice, but it would have bring its own problems. So what we have is a web socket. That's what we use. Our supporting cast, stuff we work with. This is by no means a complete list. And I'll spare you reading all of that. It's just, for example, Docker, atomic, all the CLI tools like sudo, all kinds of things that we interface with to make the web interface actually usable for the user and bring the stuff together. Some of the stuff we've contributed to, to make them to bring them together. Some of the other teams have helped shape their APIs so we can consume them. But the important thing is that we don't try to take their logic. We trust the other tool to do their job and bring them together to do what's necessary. So it's just a crazy amount of APIs to keep track of. So that's one of the big points here. And if we map this out, this is just a family tree so you don't have to read the text. If we would map out the dependencies, it would probably look something like this. We've never tried mapping it out because that would just be insane. So now I want to show you a specific example of how we interface via divas. Because divas is very important for us. It's asynchronous. And you can actually open a diva session in the browser. This is a short video. For example, we have the host name here. And if we open a diva session in the browser, we can actually connect. We can use JavaScript to connect to divas. So that object will have completion. For example, you can see, OK, I want to see the trailer release. You can use this to debunk. Actually, it's very nice. It makes divas accessible. Then we can say, OK, let's see set host name. We do that. And that interfaces with the divas process on the server. And you see right away, it changed. And if we go to the terminal and do the change in a different way, you can see it took effect. If we use a CLI to change the host name again, then the browser will receive the notification via divas. And it will change right away. So it's really interchangeable. Because in the end, when you want to do something on a system, you don't really want to care about how it's done as long as it's done properly. And what you really don't want is the second state that you have to keep synchronized. So the system is the state. And also, from personal experience, I can say that debugging or developing with divas is a lot more comfortable in this setting and trying it out on the CLI. So to give you a better picture of what happened in the background there is we have the web interface, the JavaScript, where we instantiated our little helper library that's part of the cockpit API that interface with the web socket, the web service. And that sends those commands to call a divas API with certain parameters and receive an answer. You can do that with a payment, for example. You can receive text back. You can say you set the host name. And those are translated from the bridge as the user you're logged in as to the system APIs. And this API of cockpit is stable, so to spawn processes, interface with divas, open files. It explicitly does not contain anything that's API specific, so it's a very generic and lightweight API. So what drives cockpit? We have, it's obviously based on the open source principles, on the agile principles. What that means, I'll go into a bit further later. We don't pull things in, we don't own the other things. We use them as they fit. We try to keep our work distributed, try to keep our work open, and deliver early, deliver often. And the most important thing is, with all the things that you saw, the APIs, we have to test a lot. So we can have lots of ideas and lots of good ideas, but if it doesn't work, then it's not really much used to us. So we have to test to make sure that everything works. So let's bring it all together. We have all the APIs with Docker API, we have different system APIs, user account management, disk management rates, network interfaces. And luckily for us, upstream APIs never change. They always stay stable. They're always durly tested to capture every single bug, so we never have to worry about that. All the distributions are the same. So we can just implement it once and we're happy, right? So what could possibly go wrong? Well, and trying out part of the globe in our faces, since we have a common tutorial explosion, if we say we have all these system APIs running on different operating systems, maybe even different versions, then you can't test it manually. It's just way too much. So it's safe to say that the project lives and dies by its testing. Testing is not always as fun as creating a new feature, but going back and having to fix features all the time isn't fun either, so testing is actually preferable. So due to the way it's set up, that we consume so many APIs, the testing is very vital to us as a project. And rather than paying the price for having this huge abstraction layer and trying to encompass everything, we think of every possible expansion of the future. What we do is we do a lot of testing, just to stay on top of things. And so right now we have about 10,000 testing instances a day. That means actual virtual machines with Fedora 24, 23, 22, Fedora testing being spun up, running a new pull request and being spun down again. And you can see the results in GitHub. So each pull request we have upstream gets really run hundreds of thousands of times before it's actually merged. But the key is that it is tested before it's merged because you couldn't do it manually. So it's very important that this is done upstream. Obviously we need robots for that because it's just way too much to do manually. So continuous integration is the key word here. If we want to release earlier release often and we have to make sure that the quality matches that. Open source has typically and historically take the hit exactly there. Say you have a certain cut for futures and then you start testing. And then it turns out you actually find bugs. You have to fix them. It takes longer to fix than the pushback release dates. So these cycles keep getting longer and longer. So in order to make this work you have to test before merging. You have to test continuously. This is what the tests look like typically on a pull request on GitHub for us. We test on real operating systems. We have Fedora, 24, Atomic. We have different browser tests with Chrome and Firefox. We have CentOS 7, RAL 7, RAL Atomic. This is not just unit testing. Unit tests are masqueraded as the Travis tests there. There are a couple of thousand unit tests that run also. But the integration tests are really the thing that catch the most issues that users will have because it opens an actual browser and interacts with cockpit. So here usually a pull request gets merged with all the tests to pass. So if a test fails, either the test is broken, which does happen, especially race conditions are our favorite, or the code broke something that wasn't supposed to break. And with all these operating systems it's really difficult to keep track of all the side effects of coding. For example, our master branch will also test against Fedora testing so we can see stuff that covers. We can see errors coming before they actually hit us in production. From an infrastructure point of view our virtual machines, they easily vert. You can run our test infrastructure from a developer machine. You can run it from a Docker container that you can deploy in some machine that we have. Or you can run it via Jenkins. It doesn't really matter. And this is reflected in our testing architecture. So we have GitHub which uses as the... we call it a single point of failure which does fail occasionally because that's where our source code lives, upstream. So that's where we have the status and the pull requests. We have fedorapeople.org where we store images, our virtual machines. Not every developer should have to create their own testing images of Fedora 23. And we have to make sure that we use the same testing images. We store logs, follow our tests on the servers. And then, if you have... if you're a developer and you see on the left side here you have your git checkout. You can run a test of your machine. You can run all the tests. You can run a single test if it's your choice. And we have verified machines that are publicly visible. They have...they check GitHub. They see if there's no pull requests to test those. And they can run multiple tests at the same time. Maybe they can scale. And we maybe...we have testing machines behind the firewall. It doesn't matter. As long as they push the results into the visible space, it doesn't matter where the tests are run. So this is a big important aspect. It doesn't matter to us where the tests are run as long as everyone can see the results. If you can't see test results, it didn't happen. So what we have there is we have... we look at testing tasks that have to be run like create a new testing image, for example. We try to create a new image every week for Dora, for example, to get new packages and see if the API has changed. We see the testing status on GitHub. We can pull images, push images. And we have, sadly, known issues like test load that we know that fail currently because of issues upstream from us. But, yeah, we keep track of those as well. So I want to show you a quick demo of how that testing works in practice. It wouldn't be productive day if I didn't open a pull request. Holding a talk is no excuse. Yes? We will see some of the best industries. Okay, yeah, I can... I think it's... It's not necessary right now about... The idea is that for us, we don't use a testing framework, right? So we have... I'll get more to that later, but it's like peek ahead. We have tests that use avocado. We have tests that use selenium. We have tests that use phantom.js and Python. We have tests that compile binaries. It doesn't really matter. The tests are really modular. So whatever you want to run as a test, our framework, all our framework does is build cockpit for a target system, say Debian, Fedora, whatever you want, start up a VM, install those packages, and run your test. So I can... We can go into more detail on specific tests that you think are useful, but in general it shouldn't matter which language or which framework the tests are in. So... Yeah, so I looked at the readme for testing and noticed that there's some parts in there that aren't really necessary anymore. And... So we can commit my changes. We don't have a certain script anymore. So I committed this to my fork. Then we have a cockpit project. A local request. It's really big here. You should see our test first. Start to take this up. Travis picked this up right away. And... The other test runners, they periodically scan Github. And they should pick this up. So I can also do this locally. Since, like I said, it doesn't matter where tests run, right? For Github scan the D is just for dry run. So it doesn't change anything. And that should scan Github for new tasks. And that's exactly what the test runners, we call them, actually do. There's no separate... There are no separate scripts for infrastructure. So what you do as a developer is also the test machines do. And this shows we're making this a bit smaller. Yeah, so you can see that different... Let's go. Yeah. The different pull requests, this should be mine, I think. Should be tested for different operating systems and specific image versions. And also in a couple of days the images will be refreshed. For example, we'll have a new Fidora testing image in 2.5 days. That's like a 7 day task. This is not picking up for some reason. Our lab with the test runners is moving today. In the end, once that is updated it should look like the other pull requests. Like this one. And you'll see a list of operating systems. The tests that passed, tests that failed. And you can look at the results. For example, you have... This container is actually a container test. It creates a new Docker container. You can see that Fidora tests failed. You can look at the details if it opens. The important part is that the test results are publicly visible. Right? You can work with those. It doesn't matter where they run as long as you can see the results. And the connection is not really running. Luckily I have one open. This is an example of a test where a rebase failed. And the test runner says that it couldn't automatically rebase your pull request onto the master branch so please fix this. We can't merge it as it is. And in other cases you'll see a list of tests and the ones that failed and the ones that succeeded. But apparently not right now. So I'll just go back to the presentation. So the quick review, we have the different parts of the testing system. We have Github that we use. That choose what to run and they are very distributed. We have the test artifacts. We have the testing images. We have the containers. We have packages. We have a log sync. So places where you push logs. It's just a Python script. You drop in there and connect the SSH. And we have the different test suites. Those should actually be plural. Different sets of tests. Single tests or individual tests. So like I said, distributed is really the key here. You don't want to bind yourself to single testing framework. Because frameworks come and go, especially in the JavaScript world, things change very rapidly. Other languages. And sometimes you have different parts of a software that require different types of tests. Or maybe even have existing tests. That's the reason to convert working tests into a different framework just to have a different framework. That's usually wasted time. And by making the results public, making accessible, you really reduce the infrastructure load. Steward also scales a lot better. Because if you have a bunch of test runners running somewhere, obviously, then you don't have to run all tests on your own machine, which would take a really long time. So if you need more test runners, your Jenkins, provision them in the Ansible, it doesn't really matter. And just use them when you need to. So one could argue that one of the reasons that open source is successful is that it's distributed. So that's what makes it work. A lot of people collaborating to produce something greater than what they can do on their own. And tests are part of the software. So we can't hoard our integration testing. It's not an advantage you have. It's something you need to share. If you hoard it, you lose part of the advantage you have with it. It's the same mindset as you should hoard source code. Because it just doesn't benefit you in the long run. So in order to benefit from the tests, you need to be upstream and open source. So one could say, to sum it up, the framework is not king. You shouldn't write tests to a certain framework. That shouldn't be the goal. One project isn't better than another just because you use a different framework. They're tested better if they have better tests. If the coverage is better. The framework should be secondary. Instead, we can say the framework is king. This is based on the agile principle you'll find out in there. So we have feedback from the tests from users. That's the whole reason why we want to release early and often is to get that feedback. If we have long cycles, then it becomes more difficult as a developer to react, to changes, to wishes, to change your mission, so what you want is to really reduce those feedback loops down to a minimum. As an example of that, so if we work for Red Hat, we had the Red Hat QE testing, so we might be familiar with that, so what you do is you develop your software and it goes into testing and you get back a bunch of bugs and they say, please fix these. And maybe upstream, you've already gone three, four versions past that, so you have to go back, you have to do a lot of work. So what we did was we moved all those tests upstream. So for each pull request that we have, those tests are actually run. That's a lot easier for QE and a lot easier for us. The same with packaging. If you're going to package things differently than what you test on, something will break. Everything you don't test is going to break eventually. So what we do is we actually pack it in our tests, we have the test VMs, we package our software, we install them as those packages and test them as they would be if we had released that software. It also makes automating releases easier. She tests it all the time. And of course QE still runs tests and they have some very specific ones they also run, but the core of the tests are run all the time upstream. So one could argue that tests are like microservices. We have different tests for different software. When you're testing, you always test something that you change, you get something that didn't change. And when you have a test, you want to reuse that test. For example, cockpit tests could be used to test a new version of cockpit against a known version of Fedora. On the other hand, if you're working on changing the Docker API, for example, you might want to test, you could just use our tests with a fixed version of cockpit and run a new build of Fedora and see if the cockpit stuff breaks. You wouldn't have to write new tests for that, you could just reuse them. You don't have to care how they run, just run them and see if they break. And then at least you're aware of something breaking. So what our goal is is to make testing work more like the way open source already does. It's not a different thing. You want to have the open source, you want to apply the open source principles to testing. So to use a famous Lego blocks example, it doesn't have to be an unordered pile. If you know which parts to pick out, you can save yourselves a lot of work. There's no need to always start from scratch. So that's what I mean with the open source principle. You reuse things that make sense to reuse and only write new things when you have to. So like I said, testables from upstream can be invariance, downstream. It really, the terminology upstream and downstream becomes a bit muddy if you think about the feedback loops. What is upstream, downstream, sidestream, mainstream, I don't know. It's just, the key is to reuse work. So the same way you would reuse a library, you just reuse a test. And to reiterate what I said earlier, the real magic of continuous integration really happens before you merge. So that's when you want to catch the bugs before something gets merged. Then it's still easy to fix. It's still scoped, and then you can deal with them. The other aspect is continuous delivery. If you automate all the tests and do the packaging anyway, then it's not that big of a leap to say, to adhere to the principle of releasing early and often. And that also makes it easier to listen to your customers. It's one of the core ideas in open source. And one of the key factors in the success of Linux. So it's good to keep that in mind. If a user contributes something and you say, okay, this is merged, this will be pushed to Fedora next week, then it's a lot more fun for them than to say, please check out our next release in six months, and you'll see the results of what you did. So that actually helps to drive things forward. The agile manifesto, if you look at the first three principles in that manifesto, they're actually about delivering rapidly. There's a good reason for that. It helps to keep that loop and making sure that you stay close to what you're trying to do and think about what you did. So you want to reduce those cycles and do whatever it takes to make that happen. In Cockpit's case, we release weekly. So every end month is not nearly enough to do that. So delivering, of course depends on the project that you're working on. It could be a testable run of a package. It could be a complete release. It's what you decide it to be, right? And to make sure that the community grows, the source community, you need the feedback loop that's going to be short. So our conclusion is open source is distributed. I think it's difficult to argue against that. Distributive scales, we've noticed that with container and everything. And open source development is distributed. People all over the world contributing to projects make them work. Open source communication is also distributed. Different means of communication, different modes, different languages, different places. And integration testing should also be distributed. Since that's part of the development, part of the project. And action and delivery should also be distributed. So to sum it up, we have the code that's upstream. Discussion happens upstream where it's visible and transparent. Reviews are upstream. Integration testing is upstream. Packaging is upstream. Delivery is upstream. Open source is upstream. You might see a pattern here. I'd be lying if I said this was easy. And of course, it's a goal to work towards. And it needs constant care and attention. And with everything in open source, you don't have to do it on your own. If you don't hoard what you did, but share it, then you can share and maintain it together. It's not as much of a burden. So I'm at the end of my presentation and I'll open to questions. There was one question earlier about the actual testing scripts. Yes. I think I'll show you on Vitha if it's accessible. It's not right now. So no internet connection. I can show you source code. Is this big enough to read? So what we have is something that's not maximized here. So in our testing framework, we have different test suites. We have our core testing suite. It's called verify. We have these Python tests that work with phantom.js. So what we do is we boot up test machines. We simulate a browser that interacts with a JavaScript and clicks on things and checks to see what are the variables. So you fill out text fields, see if the variable is correct, wait for text, that kind of thing. We have known issues with known failures. For example, for door 24 there's a known issue with Stalin GitHub for mounting fails due to Nestling failure, for example. So we retract those. That's one more complex thing. We have a free IPA test that pulls up my free IPA instance and lets the machine join the domain and test that. So it brings up multiple virtual machines of infrastructure. We have open shift tests that bring up open shift instances. We have avocado tests that use avocado framework or Selenium browser tests, for example. We test that a login works with Firefox and Chrome with different versions. Those kind of things. And we have unit tests that compile binaries or just run shell scripts. Does this answer your question? So in the background we have a set of common scripts. So with the core of what this does we have scripts that run a virtual machine. It uses Livert and makes sure that the machine is up. In the background we make sure that the testing network is up and running. We have tools to download virtual machines. So right now we have the checksums for images in the GitHub re-repo. So each checkout has a specific image it tests against. So if you create a new image then you change that checksum and you make sure that everyone can share it. And tasks, specific scripts in the case of GitHub but the tests are pretty separate right now and I think we're on the verge of spilling those out from Cockpit. So if there is a project that says they can use our testing framework then we'd be happy to help make that happen. Recently I did some work to make our tests work on Jenkins. So Jenkins provisions a machine and that runs with tests. I can also if I run the test locally so if I'm here for example I can say I want to run the check accounts test for door 24 that will spawn the machine in the background run the test and they give me the result. So if I'm developing I can just do a single test. If I want to do them all I would usually push to GitHub so our infrastructure can do the testing. Should be done in a little bit. So what it does is it really spawns the machine starts Cockpit interfaces with that, really clicks on login and all that. Are there any other questions comments, tests passed, always good? Okay, thank you for your attention. And if you have questions don't hesitate to contact us on the RSE or visit our website and if you have something that you think should be in Cockpit then don't hesitate to talk to us. You can try it, we have some example modules, example plugins that you can work with and the hurdle is really not that high to get something running. And we have a lot of the scripts in place to work with the system so you can basically just start out with whatever you want. So the individual modules they don't have to be in the same JavaScript framework they don't even have to say have the same build system. So all they have to be is installed to the same target directory on the system. In this case for example you have different modules that are installed one part that deals with Docker one that's the basic shell the system so depending on your system you might have different modules here you can write a custom one and that will then be present when you log in. I see what modules are do you look for ADCs or also the others like we interface with SE Troubleshoot so everything that SE Troubleshoot can reward and you can also if the version is new enough then you can also fix issues so I think apparently I have no alerts right now but for example you can set the enforce policy to off a very controversial feature but sometimes it's very helpful and if you had an error then here you would be able to see the details you could see the logs and if there are potential fixes you can apply them and terminal to log in this is the real session for the system so you can you can create a ceiling denial in the terminal and be able to see any of the others we can do that afterwards if you want I don't know often how to cause one so I'll only cause them when I'm not intending to cause one that's a fairly recent feature the SE Troubleshoot team did a lot to work with us on that to make that happen but also works via debus mostly via debus API and like I said the modules they don't have to be JavaScript is a fast-moving thing so we have parts that are written in Angular parts that are in React some that use jQuery and each module can choose what they use if they use ES6 or not if they use webpack or just make it doesn't really matter as long as it interfaces with a stable cockpit API everything works yeah so thank you again