 Hello, thank you very much Candice for the introduction and welcome everyone for this Colonel CI webinar I called it travel guide 2024 to basically go through all the different parts of Colonel CI and try to Get oriented funds of orientation around it So I'll start with not really a personal introduction but more about my involvement in the in the Colonel CI project over the past few years So I started in 2017 as a contributor so I Implemented the automated bisections as support for boot testing So whenever a Platform fail started failing to boot That was detected as a regression And typically with Linux next it was very difficult to find Why that happens? So I worked on Implementing an automated bisection for that And gradually starting from there I got more involved with a project. It was still quite early days for the project So lots of things were not really in place So I had to kind of fix things myself and then I started taking on more responsibilities So then I started implementing support for functional testing So that's more than you know, not just booting but doing more things after boot it So basically running things like a self-test LTP All the you know IGT all these classic things Then the project became Colonel CI became an explanation project in 2019 and I was the chair of the technical steering committee In 2019 and Kevin Hellman was Chair of the board at the time and then the second year I was became Chair of the board as well instead of you know to take over from Kevin at that point No, it was quite interesting also because it was very much around setting things up the project has been had been Still very small and now all of a sudden it became part of the Linux foundation But I'll get into the more details of that a bit later And one of the things after that was to basically evolve. How do we? transition from the rather small project that it was before to something that will actually Deliver on what you know its mission from from an explanation project That's where the the new PI concepts came from by you know consulting different people in the Colonel community and all around Then after that I got really busy with from where support Adding support basically to run Colonel CI tests on Cromberg hardware and doing From where it's builds with the full user space and Colonel testing of course, so that took a lot of time and kind of diverted a bit in the Hocus from the you know the new EPI and the core features to really get in the Cromberg stuff Up to speed and Then yeah in 2023 So last year was really more about bootstrapping Things to actually get the new PI in place get the new web dashboard developed To go with it and created and creating a roadmap and setting things up for that to happen And then at the end of 2023 I decided to leave Colabora, so I didn't mention that but all that all these years before I've been doing this as a Collabora employee And I decided to resign from Colabora and then now I'm also Moving out of Colonel CI gradually if you want so that's the theme for me this year Well, at least the beginning of 2024 is to Plant for the succession for these things to weather the people So that's just an overview I'd like to say also if you have any questions Please ask them at any point and then at the end of each slide You know we can have a small break and answer any questions if it makes more sense to answer them While we're looking at something relevant Okay, so you have made a small drawing basically that the map For this webinar. So first we'll do a history path. I've already explained a bit of the history, but we'll kind of fast forward ten years in few minutes then places of interest that's The important things to know about Colonel CI now and then more like a and the third part It's more like a viewpoint about the future like well this year and Potential plans for the longer term So I've explained it in three parts is more to keep track keep better track of the structure and the the the webinar So this is the you know time arrow. So as you can see it started in 2014 So three years before I joined the project That's actually When the first comets in GitHub can be found or Colonel CI maybe some ideas were there before but it's basically when it started Which is exactly ten years ago Then I would say around so initially it was very very basic and It was really around the arm arm community and there were lots of things it took a long time basically to to start really Producing something useful because the it was all done by by my colon maintainers on that, you know this bad time So it took about maybe two years. I'd say until you get to 2016 and that's when there's a more like Reliable system that's doing boot testing regularly and sending emails to public mailing lists and building boots And so it keeps growing in 2016 17 18 and then in 2019. That's when it becomes a Linux foundation project And that's where there's a new initiative called KC IDB Mostly by a couple of people from red hat initially and now it's really carried by the whole project And that's where the scope of the project changes So that's where the start of the redesign comes from and then 2022 onwards. That's when the implementation of the new design if you want is Is actually taking place Okay, so Now I'll go through all these phases a bit a little bit more So initially there was Kevin in a Kevin Hillman and all of your hands on who are still maintainers around the D arm So see Subsystems in the kernel and At that time you have to Well, 10 years 10 years ago. They were still they were already some Android phones and things I'm not sure about Android actually It was becoming a bit more popular, but it's nothing like now. So the arm support like Kernel bills on the out for the out platform kept kept breaking on the time So that was one of the first things to do to fix if you want with kernel CI And there was real big incentive to have a CI to continuously do bills with mainline So that's what they started Really as independent kernel maintainers And then they were joined by a few people from linear although it was not a linear project It's just a few people from linear happen to give them a hand if you want So Tyler who provided some of the original kernel builders in his garage And Milo who developed the original backhand and front-end like they're at the web stack Which is still online right now So it was mostly about doing kernel bills and then boot testing on embedded our platforms So here you can see some of the original labs. Well, this picture is probably from 2016 17 You can always see a bunch of Chromebooks. No one on the left is in Krabra's lab No one on the right was Kevin Elman's original lab with lots of small dev boards everywhere So, yes, we're doing this 2016 17 18 is about three years. That's where the project really Grew in terms of the original I call it the classic system if you want So you have lots of CI systems for the kernel some are some are more public than others And currency I was really Designed to do that so to do them, you know arm builds booting on our platforms And try to do that pretty well So that's what happened during that time and then in 2019 Well, mostly Kevin Hillman still, you know, he started the project basically and then he also did a lot of legwork to find members to launch the Connors CI project so that was officially done at open source submit 2019 and One of the reasons why this happened is Greg Croy Hartman was receiving lots of different reports For stable reviews, you know every time there's a And you release candidate for a stable kernel All the fixes are sent to the mailing list for review And some people would reply to say this works. There's a problem and he would get very patchy Set of answers from very different people With different CI systems during different kinds of things. It was a bit difficult to Have a full picture of the quality of the kernel basically So The main reason for Connors CI to become an explanation project was that there was a need for at least one project to become the one CI system if you want to become them the main one few the main one for all four main line And I think I don't see I probably got chosen because it was not specific to a particular product like it was not really owned by a company or You know, it was not just for one particular use case. So you could then grow to cover the whole kernel And then eventually it was launched with a five members which will go through a bit later So it's kind of like having one CI to rule them all if but it's really a collaborative effort and Here you can see the current members which are basically the same as the founding ones And there was also foundries.io as member the first year And Yeah, I think they left it. Why is it basically now? We still have Bay Libre, which is a company where Kevin Hillman works civil infrastructure platform Which is another Linux foundation project and it came about very long-term stable kernels to be used in civil infrastructure where you come a great kernels and Collabra Well worked for for seven years and Google that's also one of the reasons why there's a lot of support for Chromoresis because Google is one of the members Microsoft Who really Had a big interest in stable kernels and at the time we had such such a Levine who was working at Microsoft and a crew maintainer stable with Greg Raul and and red hat who Have their own CI system called CKI and it worked The provided Well spent a lot of effort to help with what's called KCIDB Which will go through a bit later to have a central database. So they all had like a slightly different reason for being a member which is Very nice way to start if you want and It was still looking for new members now we have you know each member provides a yearly contribution To to the budget for the project so we have we have a budget to do things I didn't detail them here, but like you know, we can pay for contractors or infrastructure and stuff like that But having more Variety I think in the members and showing that new members can join the project now. I think would be very positive. So there are a few potential ones There's also Of course, there's an email you can well, that's not the link here It's a screenshot if you go on the colonelci.org website on a home page You'll see this at the bottom and you can send an email if you're interested if you want to discuss How you know if your company would like to join Of course, we can also reach out in other ways now, yeah, so the members also provide Cloud compute services like, you know, Microsoft Azure and Google Cloud Which which is used to build kernels and Also people from member companies You know have become Maintainers of the project and a treasurer and chair and other rules It's really important for the sustainability of the project series if one member decided to leave for example It'd be difficult after after a while, you know to Sustain a project to having three more members. I think would would be really a way to secure it Okay, then back to the drawing board in 2020 2021 So because it became because a project became part of the Linux Foundation and its mission got bigger There was a need to take a look again at What it was doing what do you need to do and how to get from where it was to a would need to be So basically going from arm embedded focus system to scalable Flexible framework if you want for any subsystem of the any use case of the kernel to be to be covered potentially So I Mean initially more More test reads gone enabled such as a case L test in IGT and LTP And what we call baseline checks That's like some scripts to verify that drivers and devices have initialized correctly and things like this Um Then yeah, then there was a lot of effort put in place to add from rest test support. There's you and I like a slightly separate Instance instance of kernel CI Just to show the chrome rest results With a dedicated set of builders folk romance kernels And KCI DB got launched at the same time, so I've mentioned it a couple of times Um, so this is really just to gather Uh results from lots of different CI systems such as C spot and zero day Um, and gen 2 kernel CI initially. I mean, there's there's a we go through this again In more detail soon a bit later in this webinar, but there's a whole list and it keeps moving and anybody can Even like individual maintainers can send when you're individual developers If you have if you happen to be running tests automatically, you can Sign up if you want and send your data to KCI DB And yeah, then the new api which also implies a new web dashboard so basically the new web stack Needed that needed to be a bit to be defined So the the first step was to gather requirements from the community Uh, so that's why we did a community survey That was in 2020. Yes. Uh, this slide is from a presentation at linux plumber's conference in 2020 It's also Available on an ocea.org website in a blog post So this is a summary. I mean we had like quite a lot of questions and there's a link here on the blog post Um, so the three main takeaways if you want from from that were That it's really important to test patches. That's also called as also known as uh pre-merge testing Because initially Colonel say I was only doing was still kind of is only Looking at get trees. So once patches have been applied. So it bypasses only review stage on mailing lists and that's really critical Um, it's really difficult to do as well because there's lots of volume of emails And many technical challenges around this. Um, but this came out as one really important thing That kernel developers and maintainers would really like to see Um, and then extending long running Cover ages it's called. Um, typically especially for stable kernels. I think that's where it was really important to Have maybe, you know, one or, you know, 24 hours or 48 hours of like testing envelopes so you can run Things like fs tests that can take a long time to cover all the possibilities Some performance benchmarks can take a long time to If you wanted to run all of ltb, maybe that's not very So yeah, it's it's also difficult to find the best test because it principle you could run forever But people express the the need for Doing deeper testing because these are things that are really difficult to do by hand Say also, if you know Even if you have Even if you have the time to run lots of tests, you can maybe just run it on one platform Or it's hard to run things in parallel if you don't have a bigger infrastructure So that's really something that people would, you know, express As um As a requirement if you want to be able to rely on on bigger infrastructure To find bugs that are really hard to find by hand And improving the web dashboard I think about three quarters of people Said they would be happy to use a web dashboard if there was one Um, a small amount of people maybe like 20% I think said they would not use the web dashboard at all They would just use email and common line tools um, but a majority of people would Be really happy to use a web dashboard if you provided the right features for the workflow And that means a fair amount of customization like being able to set up your queries to look at the data in a particular way um And that was like the starting point to say, okay, we need to uh define this better So that was yes, would you be able to show the dashboard? I think I put that in there that might actually set the context for a lot of people Yes, uh, however speaking. Yes Can you see this now? This is a kernel ci website Yes, okay Here there's a link to dashboard. This is the old dashboard It was initially created 10 years ago and got upgraded a little bit to show test results um, but um What i'm talking about is there's a need for a better web dashboard than that So this solves some of the issues But it doesn't have the level of um flexibility that you would normally need Maybe we can see what the limitations are. So for example, if Here you have a list of trees. It's actually possible to look to sort the data by hardware type But it's really clunky. There's there's a ssc page here Which kind of worked well enough in the beginning with a few types of armboards But now I'm not even gonna through it now. You can play with this if you want But it can take a long time to load and it's a bit quirky to use Typically, for example, if you want to look at mainline test results, you can click on on these Numbers here That shows you all the test results Uh for the different test that were run When there's a red triangle, it's when there was a regression Um, so let's see if there's like a can also a case l test few text regression So you can see on the platforms done regressed um And with this you can see, you know, you can see the regressions This one is a new one. It will tell you if it regressed for a long time or not If you also do anything that the website doesn't provide out of the box, then It's really difficult to To add to add new features to this. Um, so sometimes people have said Uh, as a maintainer, I would like to be able to see regressions that on my branch and not on mainline So for example, um, here we have armed the um bergman's tree And mark brown, you know, if you maintainers like this here They base their branch on top of mainline like a rc2 tag, for example And if you look here, there's like 20 regressions on arms branch And mainline has 46 Uh, well, maybe because they got introduced later all coverage is different, but What maintainers don't really know is if you open, um I'm just giving you an example of the limitations of this kind of web dashboard if you look here, uh Well, first of all, they can take a while to load um, if you look at Regressions for a particular test, you don't really know if these regressions were Um, created on, you know, started to happen with that, uh With the commits on your maintainer branch, right? They were already there in mainline Okay, well So could you be able to show the builds part? I think, um, I don't I I understand that we have to improve this and then also I think it would be good to show what we are currently what it's being done currently And then you have an impressive list of socks So you probably want to show that as well Yeah, well, there's lots of performance issues with the socks tab Actually, I have, um, I have a slide a little bit later to talk about the the old web dashboards So maybe we can go through it there Um, at this point, I really wanted to highlight the fact that this this is the old one There's a big need as part of Yes, uh as part of the community survey we Found that it was there was a big need for an improvement from that Yeah Okay The that's a slide from fosdem 2021 with the first year as Uh, connoisseur as with first year with connoisseur is a Linux foundation project So the first three months, I guess, you know, you know, to Vanuver December 2022 Uh, sorry, 2019 um, were Basically, yeah finding out How to set up a project more than anything else At the end of it, we had our mission statement, which you can find on the website That's basically now now we know basically What the project is all about in terms of long term mission and goal Then Yeah, kcidb started having a web dashboard in in 2020 Um, which is currently kind of offline. There's uh, there's some issues with that Well, we'll hope to have the new web dashboard for a kcidb data set So I can't really show it to you right now. There's a slide a bit later that shows this snapshot But I can't show you a live version of the web dashboard right now. Um, then we started having Um More resources from microsoft to jewel Since microsoft was a founding member one of the things that they also did was to provide the project with infrastructure, which is really helpful Um, and google also did that Uh, but it happened before Um, yeah functional testing really happened after that too I guess it became more important Um, as the project was growing to really cover all the on the classic test feats Then community survey, which I just showed uh was done around that time frame as well beginning of 2020 summer Then we started adding support for communities to run kernel to to do kernel builds And then xprom as is mentioned here, uh, that's where there were lots of talks about kernel ci Um, I'm gonna can go through that in a bit more detail But also where it was I think it was a real milestone in terms of discussing things with a kernel community to say Now the project is a Linux foundation project. It's been maybe useful until now, but from now on it needs to Really think itself So that's why it was really important to discuss with lots of people. So we had um buzz of earth Heather above above session at Plumbers in 2020 to really discuss these kinds of things with lots of kernel people in the room So it was kind of related to the survey if you want but more in our lives Uh context at plumbers And yeah, google sysbot started sending data to kcidb And got a bit closer to kernel ci So quite a lot of interest and things started really taking up taking off at that point And then Uh, yeah, well that was in 2021 was basically the you know ramping up from there Uh, but then in 2022 and 2023 That's when we really identified world a new api and pipeline. I like to call it Would need to look like so One of the key things is that it really has to be flexible So what came out? As a result is that we can't really just use an off-the-shelf ci system somebody Paul said How about uh build but how about this uh one called concourse ci And previously colonel ci. I've been using Jenkins, which was very fashionable 10 years ago And there's other things like it lapsi i and get hub with So all these things can be used But none of them is really Well suited for testing the whole kernel in the way that kernel ci was aiming to be able to do and also providing a way for And kernel developers to gradually gradually Change their workflow from Manual testing to automation that needed some very specific Very specific Tools if you want I think one of the reasons also why we decided that Why why we established that we needed a new type of framework is if you look at the history of the kernel is It's a very special type of a pencils project because it's arguably one of the biggest truly open source project as One big repository if you want And I like to take the example of get for example, you know initially Kernel development was done with only turbos and patch files Then I think it started using something called bitkeeper for a while But that didn't really work out So what happened then is a special tool called a new tool called git was created Precisely to answer the needs of the kernel community And it also happened to be very useful for lots of other things and and then get have happened and all that um But I think it showed that The kernel is a very special Type of project. So we needed special types of tools. So git solve that Um, and I think from a kernel ci point of view, maybe it's less of a less generic. Let's less universal thing to solve But it is a special tool as well to be able to test it because every part of the kernel is almost like a small community with different ways of testing so it needs that You know git is very distributed. It has it's very flexible And I think that's what also what we need in terms of a ci Framework a ci system So if you have to use one server if you have to use one framework Then you'll soon or later you hit the hits of limitations and some people will not be able to use it Um, so how do we do do that? Well, I didn't You know, I didn't want to go into too many details about the new api design But maybe one of the key things is to have a database of course with an api to send data read data. That's kind of classic But also made it make it quite um Adjust, you know tune it for the type of use cases for for kernel testing um, and on top of that have um Events mechanism so that you can receive an event whenever for example a new kernel build is available so if you have but A test form you can subscribe to receive events. So Or you can filter to receive a particular type of event. So if you have um, I don't know like a A test form with our platforms or x86 platforms You might want to only receive builds for that particular type or even with a particular kernel config Once you receive the event to say there's a build available Then you can download it in a automatically tested you know platform and send the results to the to the api so that way anybody can Set up their own client for the api anywhere in the even in the in the private private networks receive events run some tests and send the results to the api There's a typical use case for that we work we discussed it with um Several companies who have their own um test systems that are not necessarily Public or maybe they are public but very very specific Um, I mean, yeah, pangatronics have something called lab grid and that's open source Uh, but mostly only pangatronics Use it and so can also I didn't support it negatively having this event Mechanism means they could Receive events and start running things in in lab grid. So that's one of the things that started looking into It hasn't finished hasn't been completely implemented yet, but it's You know one of the steps And it was like through some more details if you Uh, maybe in the q&a if you have particular questions some of it is explained on the unknown website Some of it is still not completely set in stone But that's the kind of issues that was That the new api is trying to address so being really flexible in order to be able to Kind of run anything anywhere And let people submit data And that'll that'll also Uh, clarify a lot of things about what the new web dashboard would look like. So in terms of being flexible now We had an idea of the type of data that would be there the type of Um workflows on user flows as they're called On the web dashboard And yeah, that took a bit longer So coming soon in 2024, that's where we are now So since I left and it was Tsc chair as well as board chair now and there's new people who've come in or taken up these rules Uh, well, I'd like to say also sure. I just joined a connoisseur at tsc last week. So congratulations. Sure. It's great. You have you part of the team and then Yeah, the roadmap adjustments. So in 2023 We made some red map about how to get a new api developed and a new web dashboard and a few things Are being readjusted now And the x analysis is something that's happening right now. So we did um Rhp and request more proposals to have people who are more skilled into more skilled in web development to really write Specs and maybe make a prototype or demo if you want for in for the new web dashboard because that's like web development is is typically skill that was missing from the connoisseur team So that you know, that's the things we know so far in 2024 At the end of the presentation, we can go through the future a bit more So what we have now. So yeah, that's the legacy system for the cobweb in the corner the So it's still online like I showed a bit earlier and It definitely does a lot of bills for uh linux next it does maybe about 200 200 bills for For each revision every day, you know, there's a new linux next tag every day and there's about 200 bills from connoisseur for that Um with gcc and clang and actually two versions of clang now and rust support And it has lots of architectures some are even not listed here, but it does, you know, all the main ones and even slightly less than the main ones And yeah, it's doing a lot a growing number of tests, uh, so I guess the LTP IGT I've mentioned them before there's um Specific ones like cross cc is listed on this on the snapshot here. That's a chrome rice embedded controller basically any test that Is open source and tests uh part of the kernel And can be run on mainline is eligible. So if you want to if you have your own test Or if you want to work on adding an existing test suite if you want to kernel ci that can be done um as long as they meet these criteria the um nearly downside nearly nearly kind of um heard or right now is the um The legacy system this one is meant to be retired. Um, so it's a bit difficult to justify Uh putting lots of effort into trying to make it do more things when we know it's going to be retired so if If this is like a stepping stone to get the tests running with a new api and the new and the new system Then that's that's good. But if it's a lot of work That's going to be deleted if you want in a few months time or maybe one year Uh, that's where you know, we need to be a bit careful It's probably one of the reasons why some things have been kind of put on pause as well Um Anyway, it does regression tracking. So that's all the red numbers you can see on the screenshot Um, so the green is for passing yellow is for Failing like things that have always failed or things that didn't run And red is for things that were passing and have started failing at some point Um And bisections are not mentioned on the web dashboard. Uh, but they run by the legacy system. So for every regression typically not quite every regression But in principle every test, uh regression can be bisected and then an email will be sent Um, so this is a slide I've used in previous presentations. So the idea is to have There's only one in a kernel mainline And the idea is to have only one ci system to cover it Of course, that's not just like one monolithic system It's an umbrella of systems if you want because there's one kernel with lots of subsystems So we need one ci system with lots of sub modules if you want to cover them And let the subsystem maintainers be able to um Own these different parts like they're in the code Ideally, they would be able to also in the um system that runs the tests And connoisseur is not focused on any particular product or any target of architecture So, um, that's why also Is eligible for being it makes sense for it to be the no one the ci system And it's based on the what we've called for a little while now on the um open testing philosophy So it's a bit like open source philosophy where you share the code, but here's about sharing test related Things so there's um the test software course itself, so Sharing just the test suites I'm also sharing hardware pools So sharing test farms so that for example, you can let other people run their tests on your hardware And you can run your tests on other people's hardware Uh and grouping the results into a single database so we can compare all these things or you can um see, you know patterns and Yeah, I have this collaboration to see all the results together in one place in the same way that you Um have code contributions in one place. You can have the data from the test results And it's community driven. Um, so maybe every Um Every test contributor will have very specific needs specific needs as some people will be running tests on their own products and their own farm But if you contribute if you contribute all these results together Then that's how you create a community based on on different individuals and different specific use cases And then the project makes decisions That makes sense for the whole community Exactly like for open source if you want to send a change for example, that maybe because you really care about that like you want to Add support for a particular device or you want to change a feature in a kernel that will You know, that's something you need for your own use private use case. I would want to say Um as long as that works for everyone, it doesn't break other things. It adds value to everyone Then it will eventually be accepted. It's the same with with testing. So everybody has their own particularly integrated reasons for doing testing Um, and when you combine all of them, then you have this community Uh, so this image is like an eclipse between like, you know, the the kernel developments It's happening and the idea is to align testing to have coverage of all of it It's not really to remove the light but more There's the limit of the metaphor if you want, but it's trying to align things Because there's still the other side of the of the moon that's in the sun Okay, so yeah, case the IDB I wanted to spend a little bit more time on this. So here's a screenshot from The proof of concept web dashboard that it's based on graph earner So it just loads the data directly from the database into graphs Uh, it's used to basically have some kind of insight into what is in the database And you can see it has you know tens of thousands of test results sent every day Um, and you can see it has even more, uh hardware CP architectures being built and it can be thousands of test results and build results So it's quite big. It can scale pretty well So it's receiving results from red hat, uh, cki from the main kernel ci Pipelines from cspot, talk sweet, which is a linear service Anybody using it can tick a box and all the tests or the bills and tests that they're running The results that will be sent to kcidb Um, there's one issue With this web dashboard if if it's public then it sends too many queries to the database Uh, and then I call this some infrastructure limitation So initially it was public and then we hit some problems, so now it's private until there's Effects for that like a basically a cache to each of these um And as soon as that's resolved the next bit is to add support for issue tracking um Which I won't go into too much detail here, but basically Every ci system or lots of ci systems have a way to track problems Like we talked about the kernel ci dashboard. It can track regressions That's kind of very basic. You don't know when it's resolved and all that But some ci systems have more information around around that And having all this public issue tracking in kcidb will also really help Okay, um new api it's still In what we call the early access data set. It's like beta beta testing So he does very small very small number of things It does only x86 bills as far as I'm aware. Maybe it started doing a bit more, but I think it's basically um x86 bills boot testing and k unit And you can see this Um graph here, which is also taken directly from the database. So it's not like a proper web dashboard um, you can see the number of um A tester is very small a number like, you know tens of results um normally the original roadmap was that it would be um The new production system the new system in production for kernel ci beginning of march But that didn't happen. Lots of things got slowed down and put on hold. So So it's still at the early early stage and um Yeah, the x analysis work is to design a new web dashboard to actually show this data a bit better um Now, yeah, I like to mention this quote from Linus Torval's about a growing ecosystem. So um To say, you know, at some point in the interview he said it was the kernel the social project That's really interesting because It's like actually the the next slide if you only have all the different parts of the ecosystem interacting with each other And now we have more and more Actors if you want in in this ecosystem so more kernel developers are starting to look at kernel ci more, you know, different ways of testing also have started to happen Um, so that's really growing. I mean we can look into this into more detail but the idea is the idea is to have um A full loop so between people who write the code and write the tests at some point to get some feedback via a web dashboard or email notifications and right now this These two things happen kind of asynchronously so the kernel development will carry on regardless of the test results but as test results become more uh more stable and more more useful People will start relying on them more and at some point the hope is that well the the idea would be to require test results to be good enough before making um Before for example adding a patch to a branch or before releasing a new kernel version Um, so here's an example of um fairly recent addition to the Landscape which is a red spot And now some kernel scenario regressions are sent to a red spot. It's not very clear How often and how and when but he have put an example from about a year ago um I think it's still pretty manual But it's that's really something that is a potential um thing to work on And github ci, I think we have helen here in the attendees so helen sent um A patch series on the mailing list, uh I think it was a couple of weeks ago three weeks ago, maybe about adding initial support for testing Mainline kernel with github ci There's a demo video which i've put the link here And it's not a kernel ci system not a kernel ci tool But it's sort of related to kernel ci. It's like a a cousin project if you want. It's not very clear to me right now How all these two things relate? um, but it's definitely at least at the level of Something like, you know, like regsbot or cqi or zero day all these things that are part of the part of the ecosystem because they They are relevant to kernel ci And um, yeah part of another category if you want of members of the ecosystem or organizations So some of the big companies who rely a lot on on the lindex kernel like, you know meta For example, I've started working a bit closer to kernel ci, especially for bpf ci Of course, um, no meta care a lot about networking and they have lots of kernel people who Work on bpf and all sorts of network related things Um, sort of created a patch work based system for or bpf um, and now they've uh started helping They started contributing to a kernel ci to the new system to to have pre-emerge testing or patch testing, uh, which is something i mentioned earlier as a result of the um community survey we did several years ago, well three years ago So that's a really good example, um of a new New type of input if you want from the community Uh, pangatronics i mentioned a while ago in this in this webinar, um, because they have, uh, the lab grid Test farms they wanted to integrate A bit similar with ti Um, there's been a few public mentions of, um, you know them wanting to run some tests An arm of course as well. I've been running, you know things with uh with kernel ci for a while Um, you know, we've been discussing things with them um There's normally sending results from tux suites I think it would be interesting to see if we can do more collaboration between their tux suites system and kernel ci um And but yeah, some of these things are also currently Slowed down a bit because they depend on a new api to be available and if the new api takes longer Then it takes longer for these things to actually come to fruition And from the course kernel ci tests the kernel kind of kernel has lots of different domains and subsystems um So these play of course a really important part in in the ecosystem is I'd like to say it's almost like the core of the ecosystem Um, so you've been we've been doing Uh, clang builds for quite a long time now I'd like to quote this email from nick dizonio who's one of um, uh clang build linux maintenance He says he's always really really happy to see all these, um Email reports, um Some emails are not very easy to read some are a bit noisy Um, especially for test results, but for the bill ones a few people have really said that You know, even if they're not perfect at least it provides some really useful information for them and apparently it has worked quite well for for clang We've worked also with um, um miguel to add support for rust And I think miguel basically owns This part of kernel ci now, uh, at least you know, I wanted to really make sure that Uh, he would be able to add new versions of the rust compiler uh Change which branches Needed to be tested and you know the type of builds and all that So I think that's working now, uh, which is a really good precedent initially Canals the kernel ci team had to do all that which doesn't really scale like if That's one of the requirements that came out from the survey and discussing with people people need to be able to um join the join kernel ci and Use use kernel ci if you want without having to be directly involved with with a core team all the time Um, I don't think we're quite there yet with v4l2. It's being tested But that's I think still kind of more or less managed by the kernel ci core team And dii was tested for a while and then kind of stopped and nothing is coming back There's lots of changes in dii and and they have their own, uh, git lab ci, but it's somewhere on the on the radar BPF I just mentioned for patch testing And individual maintainers like we said before there's hon bergman and and mark brown and mark evin inman and there's maybe like 15 or 20 different kernel maintainers who have their own trees So they play a key part as well in the ecosystem How much time have we got um Yes, it should be about time to do q&a, but I still have new slides to go through So I go through them quite quickly now, uh, yeah stable reviews This is something, um That started a few months ago. Yes, sorry. There is one question. Um, oh, yeah What do you mean by kernel ci is not stable enough? Not stable enough Yeah, somebody I think they probably have gotten the impression that it's not stable enough Okay. Yeah. Well the new api is not finished Well, these two things there's the legacy system Which is not reliable enough because it's kind of all technology and it's not scaling well So like you said like you should like you saw a bit earlier I was trying to show you some results and it was taking a long time to load for example And some things keep crashing and it's using Jenkins and So that's one of the issues if you want the solution is to use to have um the new system in place And the new system is not finished So I don't Don't really want to say it's not stable. Uh, because you can only Judge whether it's stable or not. What's in syn? Like production ready state, but it's still in kind of beta early um every stage There's a number of things that need to be done for it to be really production ready Uh, I don't know if that's the kind of thing we want to go through right now Is that there is one follow-up question? I think you mentioned, um, I think the clarification On the stable enough question. You mentioned it's not stable to be used as a blocker for the release of the kernel Oh, yeah, so that's not just kernel ci. That's the whole kernel workflow in general like right now uh Well, if you look at, you know, I was mentioning stable, um stable releases and stable reviews of it earlier so for every, uh Stable release candidate there would be a list of all the patches sent to the mailing list as a thread To say here's for the new 6.1.123 whatever There's a list of like 50 um fixes Uh, and then people will reply to say that they look okay or not if If there was an issue that was not reported or if basically if only one person replied to say it looks okay Then I think the Um, the stable release would happen at that point. Anyway, like there's no way there's nothing to Completely verify that things are actually passing There's no documented set of tests and set of things that need to be passing There's no quality. There's no control quality level if you want It's more like if nobody complains nobody reports a new issue. That's um deal breaker Then the release happens and lots of testing happens separately from from that release phase because actually, um Especially that's that's for stable that happens, you know, several times every week if you look at mainline Um, there was a release. I think a couple of years ago. I like the um, I should take this one as an example I think it was 5.4. But yeah, when it got released So it went through all the rcs and people tested and all that and replied and then it got 5.4 And then in the store that said, okay, now it's released. Please go and test it Go and try this new kernel. So that shows basically that people the the kernel mainline release Is just a code, you know, it doesn't come with any um quality level you don't know whether it's actually working or not if If nobody has reported a problem, it doesn't mean that there's no problem There's there's no requirement to say we need to have these things really working um So testing happens kind of in a different universe from developments If in a real in a real Yes, yeah, if I may let me clarify that um The when the really kernel release comes out. Yes, you're right Linus asks people to test So it is not that none of the tests are run Various maintainers make sure that they have run Their tests and they have done their regression test before they send a pull request If you were to send a pull request to Linus and Linus finds a problem And that's not a situation any maintainer wants to be in So a lot of the code stays in next and gets integrated over a period of almost four to six weeks After the merge window, we don't put any new features into the kernel after rc1 comes out And after that maintainers are submitting their code into the next next and it is being integrated It is being tested by various Um test strings and then also users So that test So I am I think I'm questioning the statement you're making that it is just code. It's not tested. It is tested Uh, we do not have access to all of the systems out there all of the use cases out there So is it tested on all of the use cases and all of the available systems out there because Linux obviously runs on 30 some architectures and then maybe thousands of or if not tens tens of thousands of systems out there So, um, the so the to clarify what you're saying. I'm putting a context around it that What we are saying is we haven't tested the entire universe that Linux runs on Please tell us if we broke anything Because we humanly is not possible to test. That's where kernel ci and Leonardo test rings are all very important to the ecosystem to make the quality releases quality releases Yeah, I think my statement was more like if you get a tag They release, you know, what is being delivered mainline? Is really just a code You know that people it's like best effort for testing things and people have tested it and I've tried to run all the tests that they think really matters But it's not documented really with the release, you know to say this kernel has been tested this way and you know There isn't a particular List of things that the kernel needs to be passing before the release happens So, um, I wanted to mention this if you compare it with a I want to say like a full modern closed loop ci system Where you have like, you know, GitLab ci for example, you can block changes until they pass a number of tests and I could be just like Starting just even with like a build. Can you build the x86 dev config? It's kind of assumed that it does But there's nothing anywhere that says it has to It's it's it's um, testing is also just like the project Code writing and testing is also open source responsibility. Yes, you are right that there is no closed um, it doesn't happen on like a closed source system where you Say release notes comes out and says, okay, these are all the things we very quite Yeah, so it's a different, uh, testing philosophy that it's also users responsible to test the code Yeah, and I think that was um, it's a social project Yeah, but I think, you know, I wanted to start talking about this based on the question that was raised about why, um What is not, uh, Complete if you want in in the kernel Testing and I think that's what it is because you know when kernel development started ci didn't really exist Now if you create a new project today, you would have ci from the from the day one if you want And very easily especially especially if it's a small project like an apple website You can say it has to pass the smoke test. Maybe pass some vm test or whatever If some tests don't work don't don't don't pass then the changes don't get merged um, we don't have that for the kernel now and it's Basically why i'm saying that testing happens in a separate Uh space if you want to compare it from development So if for example only um, so even if you know some people might report a problem At the end of the day the maintainer says this problem is okay because we know, uh It's only on a particular type of hardware that nobody has or it Won't be fixed by another branch coming from someone else Even if things are reported as problems to maintainers at the end of the day can decide to still apply the patch and still have it part of the release That's something you can't have with a real automated closed lube ci system And we don't have it's not it's not one so somebody I think just described it better saying kind of distributed cure response. Yes, it is It is the responsibility of the community to make sure the release is good and when problems are reported you are right maintainers will come and say hey, yes, it we know this is a problem However, it it can be fixed and they get fixed Soon enough. So we don't have a ci system cannot block a release. You're absolutely right Give enough Here is one more question. Uh, would you like to address that before you move on? I don't want to disrupt your flow here No, that's fine The test page does not show the latest test results. What makes it wait so long wouldn't be it'd be better At least the values from a database from a while ago be what it displayed I guess they're asking about the test dashboard Yeah, and then I guess the dashboard basically I'd like to say it's broken Um, it works Because we're lucky. I think you know, it's very old technology and has lots of issues If it's not showing things it's Yeah, I mean, I'm really Kind of almost astonished. It's still online and people still rely on it Um, so improvements on the old web dashboard. That's basically, you know, not gonna happen It's only like if something is really breaking it in because completely offline Then maybe a fix will be made for that But now, you know, it really makes more sense now to focus on the new web dashboard Which should be online or the plan was to have it this summer. They're actually a written plan And for that so I think that one of the best things to To do at this stage is if you have an idea about How the web dashboard, how a good web dashboard Would look like or what type of feature you want to see on the web dashboard Then you can, you know, send an email to explain it or whatever communication channel you might use on the IRC even Explain your idea and then it Will have a chance to be factored in and be actually implemented for for the new web dashboard I hope that this is the question Yes, I did a little disappointing Well, it's work in progress once you put in a dashboard and you're using it It needs it needs to move forward and that's what Colonel CL project is trying to do. So that's all good Okay, so no more questions Okay, so I'll go quickly through no more questions here. I'll go quickly quickly through those two more reviews So that's something that started When initially it started several years ago Um, Colonel CI used to send replies automatically to a stable RC Email threads Then it wasn't really well maintained and it started breaking it was disabled and then some people like me myself included at some point try to um Reply manually to these Threads by looking at the results from Colonel CI so like There's a new stable RC review thread That gets tested by Colonel CI. You can go and see the results on the web dashboard in email reports So the idea would be to kind of compile that a compiler summary by hand You know come up with a list of regressions and things that seem really important to report and reply manually with that Having in keeping in mind that it would be ultimately something to automate because it doesn't really scale to have Lots of people doing like this and doing that all the time um And then I think that got phrased as that got captured as a requirement for the new epi and the new web dashboard and the new system No notification system to be able to reply to uh to emails with specific results like that um And recently at the end of last year some people decided to have a look again and try to do this again Which is good. I think from the point of view of Engaging again with with uh with the colonel community. So in this case, it means there's a bit more relevant activity Going on between colonel ci and and the stable reviews of mainly greco art man, but also every day also cares about stable quality um However, it won't be completely fixed until we have automation for it, I believe because Otherwise, it's not really sustainable in the long term Uh, so here I've made a snapshot of A summary of the versions that are being tested now. It's only doing if Well, it's looking at the colonel ci builds and looking at boot Boot testing only it's not looking at any more test results And here it's showing the list of regressions that were identified and sent So that's kind of also work in progress Uh, this is the email from greco art man one of the one of the first reports like this was sent in december Uh, so basically it's been like, you know, positive feedback. So, um We know we know that we'll always know there was a need for that. Um, hopefully we'll have um An automated process for doing that soon And you know part of the um Like I said before, you know, one of the real key things for a true Um, so I system for the whole kernel community is to empower kernel developers and maintainers to be able to um To own their little part of well, maybe a big part of kernel ci and how And and how it's testing their part of the the part of the kernel that they care about Like I said, you know for rust, for example, it's really good to have the rust maintainer able to configure how kernel ci is testing Is doing rust builds in the kernel Having this kind of principle applied to everything every part of the kernel would be really ideal And yeah, I'm not going to go through all that different details, but Of how the new system is meant to work But the idea was to basically have an abstraction between the runtime environments Where builds and tests or any kind of job is being run and the test definition That means you can use the kernel ci infrastructure. You can use your own infrastructure You can create a new type of infrastructure So you can run a kernel ci job ideally like on your own on your laptop on a vm that you're in Or in um kernel ci owned part of the infrastructure With just a slight different command line option to say like your runtime is molecule shell Runtime is this Google vm or this has no vm or whatever you can create your own environment for running tests if you want That's really useful for people who want to start Automating gradually So you can sudo to make something By using kernel ci to run things just in your in your local shell and gradually go Further than that to scale and run more tests in in vm's in the cloud and all that It's also useful to come back from that. So If there's lots of tests being run in the cloud and one of them is failing for example Being able to rerun exactly the same test in your local shell means You can work on it and debug it more because then you can Run it slightly differently to you know get some debug output Run it again with a slightly different build with more debug options turn on and things like that So that's kind of these two-way process Which I think is really a key thing here and it's something you don't have in or you don't easily have in normal ci systems So for that the idea was to have a common line tool called kci you would have things like kci run this job and kci get a result and it decides any result and so far And yeah, so Empowering developers from manual test to full automation and I'd like to say and back from automation back to to their local manual systems And Um, yeah, I've also highlighted here the main ways to get in touch with the kernel ci team. So there's a mailing list Which is hosted as a kernel mailing list. So it's also archived on lore Um, there's regular meetings. There's one every Thursday. That's open to everyone Uh, well, let's see one of them. There's open open hours as well And uh, I'll see channel on libera dot chat These you know really ways for anyone who wants, you know, whether you're a kernel developer or test developer or product product manufacturer, you know anybody from the Like from the ecosystem picture should be for anybody who has an interest in in The upstream kernel quality can just, you know, show up and and start discussing things and that's how That's how the project grows Uh, yeah, in a string. That's basically um a summary of the the mission of kernel ci is to um automate factor out the, um repetitive and difficult work Of a maintainer a lot of things can be automated um And if we can rely more on on machines if you want on the infrastructure to do all these things then People especially maintainers who can be really have the I don't want to say the I don't want to call them bottlenecks, but it could be really at the junction of a lot of Code coming in and pressure to have things released Um, so that's a huge burden if automation makes this work easier then it can just focus on the real things that make sense for for maintainer as a person to deal with so like critical change the critical choices Um, and you know connecting with other people and and building the community That's something you can't you can't really do with a automated system So that's the ultimate, um, I think goal for for kernel ci and what it stands for Um, and I'm saying I'm here. I mentioned the idea of uh, synchronous ci loop That's what I explained a bit before so I'm not going to go through that again But you know the idea of gating releases based on ci results um Is something actually I talked about at kernel recipes a year and a half ago Um, and I wanted to introduce something like a new trailer a new Tag if you want in in comets Um, like you have link for link to mailing is discussion discussion Yeah, uh, tested by and the name of someone who tested it Um, what I proposed at the time was to add a test link So it's like a link to a mailing list, but it points to A static page with a documentation about which tests were run Um For this particular A kernel revision typically you would have that on on a release tag So on a stable tag or mainline type maybe Or maybe subsystem maintainers could do that or they want when they send a poll request or for for their own their own management And I got some positive reviews and Greg Rodman said that would be very useful as well for People who build products based on stable because that would give them already some paperwork about how stable the kernel is How it's been tested before they add their own patches on top So anyway, you know related to synchronous ci loop Um, now I think we have there's a bit of time left Yeah, in principle, we have like 15 minutes left So feel free to ask me, you know more questions at this stage Um, but yeah going forward 2024. So now, uh, Don Zikus from red hat has been elected as new Chair of the board of members of the project Uh, so he's been doing this since January 24. So it's still quite recent Um, and Nikolai Kondrashov just got elected as new TSC chair And that will be effective First of April next week now in two weeks time. Hang on. Yeah Beginning of well first of April and she has joined the TSC Um as a TSC member in in march So these are Actually, I mentioned sure here As because it's bought nothing to highlight when kernel retainers Join join the TSC all people who Were not necessarily Core kernel ci developers Well, we've also had Two or two or three people joining the TSC recently who had been The kernel ci developers for for a few years And basically got more established in a project if you want a more like Taking parts in in votes and maintainers so you can see that there's even the core team as well is growing And that's actually documented on the on the kernel ci website I can just quickly show you if you go in documentation You have the organization part Well, mission statement, but you can see the um advisory board who's on the board here and TSC technical steering committee You can see all the members and The votes by the TSC are listed here. So you can see when you know I've all joined the TSC and uh, Jenny joined the TSC and You can see there's put put a lot of things happening and Have been happening in the past few months Okay Yes, so it's kind of a bit of a new beginning in a way for the project to have a bit of New new people joining Of course, it means also That it requires transition phase when you have a new mall chair as well as a new TSC chair Um And so it has kind of side effects in a short term So I guess it will take a while maybe a very maybe not very long But maybe it will take a you know, a couple of months or something to start seeing the the results from it So that I think it's kind of related to why we have um, so that's why we have Some delay with a new api and and I wanted to mention To have something more concrete to show about the new web dashboard in this webinar Like I actually put it in the abstract when I wrote it. I thought I would have something more concrete We don't have it. We don't have it right now, but maybe in a few months. Maybe in one or two months time there'll be like What we call a clickable demo So something that looks like a web dashboard And you can click on it to open different pages and stuff like that But it's not with a real live database. Hopefully we'll have something like this In a near future so I've good I've got a few Words from uh, donzikus is the new board chair. So Um, yeah, I asked him if he had if if you wanted me to if you wanted to to have A particular message added to this uh, webinar because it's really about The project and not just my own story for In kernel ci. Nothing is main message is that there's no Um, there isn't going to be any major disruption At least not in a short term. The idea is to carry on with the same A momentum of the project and keep growing it basically And also a quote from uh, Nikolai Kondrashov who's the new tsc chair And yeah, I think it's basically You know make the new kernel ci api useful a sap that's basically catching up with the delay that's happened We're deploying the new and completing the new api um So I think that's also kind of a line kind of about, um Continuing in the same in the same direction And then uh, yeah, take me slow down. So that's one of the challenges we have now This is the Sydney opera house, which is very famous. I mean taken about four times longer than a plan to be uh Finished of course with kernel ci and um Community projects in general you can't really completely be sure of how long things are going to take because you don't have like one overarching entity Completely owns it. I mean you have maintainers of course, but it's not quite the same. Um, it's probably True even for non-community projects. There's always like of course risks and Someone said the best way to make sure something doesn't happen is to make a plan for it Hands very rarely actually roll, um, you know unfold as as they were As they were created So it's kind of you know part of life if you want Now, uh, the new api has been delayed a few times So I was really hoping that this new work map with some specific milestones, which I don't wear reasonable timescale Um, what's going to happen? But there's been some uh, also some reshuffling around and that's taking more time So maybe Mostly a bit later this year um, but yeah, the legacy system is really overloaded and outdated um, and the new new web dashboard needs ideally the new web api to be Uh stabilized Before you can have actually a web dashboard to show the data from it So for um, I think people that might not be familiar with the new api and how would you would like to elaborate on that? What would new api the role api plays? And how it connects dashboard and uh the data I'm assuming people could uh, I mean what what does this new api do? Okay, um I think I've touched on this before Um, I can explain it here, but there's also a blog post What is this new api? Um Yeah, that's good. I mean people can read the blog post Yeah, I wanted to put the link here Can you guys see the link I've posted? Yeah But in in a few words basically, I mean there's eight databases you would expect with all the test results With a web api on top to be able to send results and read results So the web dashboard would be using that to show the the results on you know as a web dashboard Or maybe bypass it and directly read from the database, but it's more like implementation detail That's the very basic Thing which you would not see with any ci system with a with a with a web interface Now also, uh, well, I think this new system would have would be uh user accounts so you can you know, you Be able to see things anonymously as You know all the data set would be public But also by having a user account you can save your own searches. You can Opt in for specific notifications So you would only get any emails that you care about to say only send me a regression when uh blank 13 build on arm 64 happens Happens to change for example That's the kind of use case you would want to have uh want to be able to do and then Also The new api as event mechanism. So it's not just passively reading and writing data is also um Acting as a way to orchestrate different things. That's something you normally, you know in other cell systems. It's kind of built in here you have there's very low level way of orchestrating things just by sending events to say distant data just just came in And that's all the api does. It doesn't say now you need to run this beyond the unit around this desk It doesn't say that it just just says This new type of this new piece of data just arrived and then something else on the client side can receive it You might have lots of different services receiving these events Uh and deciding to do something with it So as a user you could say Okay, I want to get notified with an event from the api every time Uh a case self test is fading for example or it's a case self test regression has been found And then you can have your own service that will do something locally on your even on your own laptop You know, you can receive that event and do something with it to for example, um Run exactly the same case cell test, but with a different kernel build that has more debug options turned on um You know, that's just a random example here That's something you can do as even as a maintainer to automatically have your own particular type of debug being run even without watching um when particular set of um the particular type of event is sent So that's just an example and then these results can be sent to the web dashboard as well. You can see this maybe um as part of your own data and having events also means we can have more like Real-time live updates on the web dashboard. So you can see this revision is being tested. This result just came in This one is waiting for something. Uh, you know Uh, you can see that tests that are that haven't started started yet. So you could anticipate What is going to be run? That's the type of thing you would see with the new api in principle say I uh Maintain of run test before they send it up. I I run my test before I send my full request Could I somehow say feed it in using this new api saying? um This these tests passed on this full request for Our next for example uh Yes, um, I'm not entirely sure. So you run some tests. Yeah, you can if you run some tests locally Manually, you can have these results sent to the api. Definitely. Yes. Okay. That's that's a question. Yeah Okay, so that that would be a lot of value add if we can um, if if this api can enable maintainers to Um and contributors to report their tests what they have done before they send a full request So that kind of shows um, then they could see potentially That before they send a full request to Linus They there some of these tests are done. We may mention that in our emails, but it would be nice To reflect that on a dashboard Yeah, or you could even have um, yeah, or you can make it part of your workflow if you Write it in email in a particular way and have a service that listens for emails And then when it sees your email it forward, it's the same Yeah nothing additional need to be done. You just send email Okay, so that sounds great. Yeah Maybe that would be maybe slightly clunky Like if you have the information firsthand, you could just send it to the api and also send any But maybe that it's up to you. Um, like, you know, it's definitely something a maintainer could set up to not have to really rewrite their own workflow or whatever You can create your own small python script or something that will listen for iMac females and then pause it and send that stuff to the api if you wanted to And then make it then you can make it part of the main kernel CI tools if you want to as well so Okay, Candice is reminding us. It's five minutes. Oh, I think I think we nearly so yeah, I mean there's lots of ideas about what we're going to do after that So adding performance testing stats based As difficult to say stats based by section Um, I made a talk about that at primers a couple of years ago and dynamic scheduling is about Running fewer tests first and then deciding on the spot which other tests to run Based like if something starts failing you run more tests around that which is kind of like post regression checks If something starts failing you want to run more tests around that Automatically to see how often it's failing if it's a stable failure Which is kind of related to performance testing and stats based so that's kind of ideas that We've known for a while that it would be really useful. We've discussed with people at Intel and others about their systems and that's A set of features that would really take it to the next level But it was decided to you know, there's no point trying to implement that with the legacy system Um, so these are things also that are waiting for the new api to be implemented Um, and so yeah, now the project is basically trying to get engaged more with the kernel community again um And Yeah, ci CD has come a long way With the pencils projects that's what I said a while ago, you know, the when the kernel first started There there was no real ci and now every project you create basically you have ci for free and it's very easy to set up Um, and let's do the last slide. I think yes and brainstorming A few more ideas about where kernel ci might go in the future Some things people have said before is what about testing more than just the kernel? That's complicated, of course. Um One example for that is currently chroma s. Um, it's not testing user space as moving target, but it's regularly upgrading to a new version of chroma s as a full user site stack But in future we can imagine like these things things like I don't know an android snapshot or an embedded snapshot and you have instead of What's it called the product and the test if you want instead of being Just a kernel you could have a whole os image. You need to have one way to identify the version of that Um, that's something maybe that could happen. Another thing that's really important is in 2025 There's a european union directive about um basically reducing um What extending coverage? It's like extending support from vendors product vendors For at least five years for the operating system, and I think it's maybe like seven years for the hardware It's to avoid having lots of phones That come out with one release maybe going to get a os upgrade six months later Maybe a year later and then they stop and then you have a lot of lots of devices with security random relatives this whole ecosystem based on Trying to get people to buy new furniture basically. So that's the kind of thing that european union is trying to change and if you look at um, apple does it an apple is not particularly Big actor in the open source world or they use like free free bsd and things um apple have fewer products and support them for a very long time and chrome rice is a bit like that as well with chrome books um, but apple is really doing it for a huge um much bigger volume. Um, well Android is really really the big big big volume here And then apple is it's pretty big volume with with iPhones too and has support for several for several years so I think following this example, I think we would basically It's basically what this is all about and having a good ci system for that will help to improve, you know How long stable will be stable? Okay, um, so yeah, thank you for joining us today. Do we have like maybe one more minute for any last question? Do we have any more questions? Slide 38. Oh, that's just the link. Yep. Okay So, um, thanks for sure for realizing this in kandis as well and you're already at the other Great It's been great to be able to go through this. It's been an interesting exercise for me as well to kind of look back look forward That project. I hope it was beneficial for people here Thank you for doing this kind of back to you Awesome. Thank you. Yeah, minchua for your time today and thank you everyone for joining us as a reminder This recording will be on the linux foundations youtube page later today and a copy of the presentation slides will be added to the linux foundation website We hope you are able to join us for future mentorship sessions. Have a wonderful day