 So we here at the colonelci.org so what's going on here? Well, this is a project that we've started as a community project to help the upstream colonel maintainers Validate their changes across to arm arm 64 and x86 platforms So what you're seeing here is the jobs detail page for the mainline colonel Now we do mainline next arm SOC and a few of the other maintainers trees Essentially what we do is we build every single deaf config for arm arm 64 and x86 And then we boot test them and provide the results either in an email report form that goes out to the mailing list Or we provide this website where you can dig down into different bits of information if there's failures So when you see this here, this is all the various builds of mainline So here's our get described which is our kernel version here What branch was built from and the exact commit and then at a glance you can see how many deaf configs We built and how many failed and how many platforms rebooted and how many failed and then the dates right here It's all sequential. So this is the last 14 days and you can see quite a quite a lot of activity happens on mainline So when we look at maybe what one of these build reports looks like for a particular build You can see how many deaf configs we built how many failed and how many were Unknown and so it's quite a few. We have the architectures listed here So you see a arm arm 64 and then the actual deaf config name listed on left So as you scroll through you see quite a few of them So if we take a look if you click on any of these what you can see is what images are produced Links to all the binaries and if you click the build log here, you'll take it directly to to the log for the build On the other hand, here's what our boot reporting looks like. So currently right now We have five labs distributed all over the world providing the hardware for this So we have about 260 boots every time a tree changes upstream and The boards here are listed As their upstream name So if you're ever interested in does my board have support upstream you can simply come to this website here and look at the Details of how we're booting the board all of the load addresses in the u-boot configuration or the boot loader configurations are Present here. So is this a new new solution? It is. Yeah, this is the last six months. We've been working on this and really at This connect we kind of made it public and started to bring in wider audiences for feedback So is the idea that Anybody doing anything to Linux gets tested on everything every time. Is that what happens? Yeah, and what we find is we can catch the breakages before they're merged into the upstream trees such as like Linus's tree and that's good because then the community can always count on boards booting on mainline And we can work out what what the failures were before we actually Merge them into trees people care about but this is a big deal. No, oh, this is a huge deal And it's a very big effort and it's a lot of work to do But I think we're providing a lot of value so I can give you an example of a failure that we've detected So here's a recent failure It was you can see here that get described was next to was built on the next tree We were having a little bit of problems with with some of the clock patches that have been merged and what you see here is the kernel error and The reason why you click this html link here, and it'll take you to the actual boot log And if you can scroll through here, you can see how the platform was configured and what you'll see is oh, there's lots of Stack traces from the kernel. Well, if we go back One kernel developer might want to know You know how how can I bisect this issue and what we give is a table here with the good commit and the bad commits And then basically a little summary for bisection and you can even click this here And it will give you a little starting script for the bisection. So we want to provide this information to the The maintainers developers that are patching the Linux kernel so that they're able to figure out what change broke it And how to get a fix and quickly so that the trees aren't broken for very long So what's the response by the those guys? Well, it depends so if we catch it in like RMSOC or or next Typically, it's the same day that we're reporting failure failures We found an interesting failure with some of the memory maps of the subsystem And we were able to report it and have it fixed in two days, which was great But what's the like the the enthusiasm or the other people happy about this? Well, yeah, I think this there's there hasn't been a system like this before That's that's public and it's focused on the upstream kernel There might have been other systems that that were developed as proprietary solutions And so Intel has has a similar thing called zero day. However Some of that stuff is close source to them and they they they're not willing to share it So we had to go ahead and build our own system for this Do they test all the on boards? I don't believe they do no But they mainly do it in virtual machines and I'm being Intel They have large amounts of hardware so it's easy for them to scale and to do some really good testing which they do And it's it's very it's not just for Intel. They do report issues. They find To arm developers. So they're working pretty good in the community. So before today There was a bunch of guys with a bunch of boards each testing manually separately different things and very slowly It took months to see if there was a bug exactly Yeah, and all of this testing is happening However, it's spread out throughout the community And so this is kernel CI.org aims to bring all of the testing together and unify it and provide unified reports about What's working and what's not working and so that we can get these issues fixed quickly Now this is just doing boot testing and very very minimal. I mean we're tracking stuff like how long it takes the kernel to boot Can you run basic users based tests? Ideally, we want to actually expand this to running performance benchmarks tests, you know Performance for catch performance regressions as they happen upstream. So that's the end goal But we're not there yet. So do the kernel developers have to opt-in for this server. So they just all get the email Automatically. Yeah, so they get the emails the public trees. So mainline next arm SOC Stable trees you can sign up on our kernel build report mailing list and you can receive these notifications without having to do anything special Some of the developer trees we have a private mailing list where you can have your trees subscribed and then we can build and boot test your tree as well so So that's some of the it was was this one of the ideas that came out of lava stuff Yeah, so actually all behind the scenes your lava is driving a lot of this There's two automation frameworks. There's pie boot developed by Kevin Hillman and then obviously there's lava And so here's my lava server. It's running and you can see here's all the jobs that have already ran for for a mainline kernel and There's quite a few of them. And so what lava can do. What's really nice is that we go to my reporting framework here We're just starting to play around with this But this will allow us to trend boot time over over time So as the kernel mainline progresses this way we can make sure that the The boot time stay within the range that they're at So this is a delta reporting view right here where it shows the variances between each boot now if we Tick this what you can see is here's here's the raw boot times and right underneath it So with multi v7 on the Arndale octa platform It's about five seconds five and a half seconds boot time for the kernel on the external It's Exynos config. Excuse me It's a little about two and a half seconds and so you can kind of trend this over time And what we're looking for with this view is to see large variances If the kernel all of a sudden takes much longer to boot now that we can catch that and report it and we'll if you actually Now so far you can see the kernel tag that was built So we know what version caused the regression And so this is the kind of things that we're building with lava and the kernel CI.org to provide this this information back to the developers This boot time. What else right now? We have this is kind of neat. I can show you here I'm looking for kernel exceptions and kernel warnings and so Here this is a filter that you can oops That lava provides that will catch look for certain test cases that fail And so in this case this is looking for kernel exceptions and what you can see here is the kernel exception It was captured and failed by lava So we're also tracking the amount of kernel warnings and exceptions for every boot So this is just the beginning of where we want to go. We'd like to see you know More depth in our testing Really doing performance Benchmarking whether that's you know some some IO based tests or it's a networking performance test and sort of trend those over time So that eventually we have the entire Linux kernel subsystems covered with with functional tests So this is speeding up Linux development. Yes, and it's making it safer And it's making sure that it's ensuring the changes that are being made are resulting in Better performance and not worse performance and that's what we want to do And the idea is if we can test it upstream then everybody that uses a Linux distribution is going to benefit because the distros don't run on bleeding edge Kernels a lot of times they pick it up every so often So if we can ensure that there's a good base to start from then the distros have to spend less times fixing fixing bugs And they can work on other things to improve their distribution That's really cool. So 260 boots that means 260 Boards no because there's there's different kernel configurations. So we just multi v7 with lpae and virtualization configurations. So it's not just One one boot to one deaf config. There's multiple deaf configs that boot many boards. So Yeah, 260 boots happen every time a tree changes upstream. We're looking to increase that If you want your board if you go to colonelci.org and don't see your board on there Send info at colonelci.org and email and we'll see if we can we can help you out Maybe they should do some work to get it in there. Oh, yeah And so that's that's the great part is that there's an actual API. I can show you so if you go to api That colonelci.org There's there's an extensive documentation on how to get your boot reports your build reports Published in the system so that labs around the world can use this and it doesn't have to be in a single lab And that's that's what I find is that the hardware spread out all over the world But we can still have it unified in one place and that's the goal of colonelci.org Nice. So these guys like rock chip for example. Yeah, and all winner. Yeah, they were kind enough to give me their Pre-production runs for the QB board for which is the all-inner a80 chip and then this is the radix rock rock 2 Which is the RK3288 chip So we're gonna put these into colonelci.org And make sure that any of the upstream work that happens on these platforms is not causing any regressions But I can imagine lots of different solutions in 2015. They will all want to be there. Yeah, absolutely Well, and some of the enterprise server platforms from ARM, you know All that all the 48 64 cores 96 cores and all these guys and that's when the regression testing becomes very very critical Because with that amount of cores, we have to make sure that the the apis inside the colonel We have our scale properly and if there's not issues just doing basic boot testing but ideally we'd like to do performance measurements as well and Stretch goal would be power measurements making sure that we can not only Increase performance, but then also to decrease power consumption. So this is nocta core big little a15 a7 This is a quad core a17. Yeah, so many different arm SSC's right and it's pretty cool that Linux just works on all of them Oh, that's the great part about Linux and the ARM ecosystem is that you can have so many different designs that Work in different ways for different use cases and essentially they all use, you know similar technology So, yeah, that's why I really enjoy working on these projects I'd like to see the broad ecosystem on ARM improve with as well as being good citizens You know, you notice I mentioned we're testing x86 is because any of the arm changes to the arm trees We want to make sure that they're they don't break other architecture So we are looking to expand, you know to maybe power PC and MIPS at some point just to make sure that we're being Good citizens because that's what we want to do But arm is a pretty cool ecosystem. Oh arm is one of the best ecosystems the best it could be the best Yes, I you know from what I've seen that I would agree that is that it is the best the innovation around the arm ships You saw that the high keyboard This week here at Lonaro connect and just looking at a credit cards credit card sized arm v8 board is is Quite amazing. All right, it's cool