 First of all, thank you for coming. There's a huge offering at TLC normally, so Yeah, you have plenty of offer of possibilities. Thank you for coming to this one my colleague Ben Hutchinson and myself Will be talking a little bit about Some of the ideas that a company called Copethink is bringing into the kernel maintenance within CIP My colleague Ben, it's a well-known kernel maintainer Davian kernel maintainer Also, he works upstream as a maintainer and currently he's maintaining the the CIP kernel based on 4.4 and I work as a consultant in the same company and Basically, I I represent the company at CIP Copethink is an open-source embedded company 10 years old based in Manchester and we have a What I believe is a strong background in working in the open and also working in customers. We are quite strong in automotive and Those of you who were this morning at the CIP talk, I'm not gonna repeat most of what was said there I would simply say that are basically six companies The platinum members among the four they have more than a million employees Which is more than many countries out there and then we have Copethink and Plathome Doing kind of a specific word plathome in the IoT space and Copethink providing These we are working on the kernel side of things for in the CIP And we have basically four four topics to comment. First of all, we're gonna give a very brief Picture of what maintenance is today in different industries And then I would go for very briefly also for different strategies that we could have chosen to work on the kernel within CIP and then Some considerations from Ben related with the limits of the maintenance lifetime of the kernel and finally He will provide us some ideas of how can we achieve? Longer lifetime in terms of the kernel so Yeah, this is just a picture that is approximating and you can make tons of considerations here But basically what I'm trying to show here is that on the enterprise world the site the maintenance Well, the lifetime cycle Of the the life cycle of the products that are out there in their enterprise world Depending on how you consider it not around 15 years 15 17 13 depending on which company how do you do you consider that the life cycle of a kernel? But it's run about that time Then non-commercial distros like Davian. They are around eight If you count all the life cycle of the product Or project if we go to the mobile space we see around five six years since they started and until the Lifetime of a product finished And focusing a little bit more on the kernel Well, depending on how you count or you want to think about how the kernel is developed if Some might say that you have feature stabilization in next and then in the main line You do the whole product stabilization that could be around four months somebody will claim that it takes years to Change a core feature. So depending on how you look at that We are talking about up to six years with this new announcement for for the LTS branch Two two and a half. It's if we consider Other branches and then on top of the LTS you can have LSK or LTS I they just try to extend in different ways that period so Again, the specific dates are not very relevant When you think about it from the industrial perspective This this slide was pretty much shown this morning. So I'm not gonna stop on it too much I'll just gonna say that Think about a Railway control system and for example each country has its own rules and way of doing things So just the the the specifications for us for a customer's extensions might take up to four years These are data provided by by some of the members, right? So Or just the initial certification of that system might take one year and we are thinking about Think about those systems maintain it for a very long time if we go to power plants, for instance, then the year This is completely crazy You take out about five year up to five years to develop and then just simply Specific customers takes extensions might take up to four years And if you think about the deployment time or the supply Time, then it's a whole bunch of years are putting a power plan up to working up to work It's it's insanely costly. So and Then you associate hardware with software in these cases and those two You have to provide hardware Maintenance and support and software so if we can be talking about I mean Urs van Siemens was talking about Systems being developed Provided in this in the late 60s or in the 70s. Sorry and I'm still working, right? So It's a huge challenge. So when you think about those timescales In terms of the mind of the kernel of the Linux kernel if we are aiming to put Linux And that's the whole point of where these industries heading little by little and you compare that with what we know today It doesn't matter how you count it. We are thinking about in the best scenarios about extending the current Maintenance Cycles in in four three four If you compare it with enterprise, but if you compare it with embedded, I don't know buy a lot so For a company a small company like Like co-thing that the following question is and why what are we doing here? working in in in CIP so For us it was very interesting from a company that comes from embedded and is getting into a strong and automotive Think about the strategies that we have been following lately are very close to what upstream is telling everybody, right? which is basically Reduce the delta between upstream and and your product and that will help you and keep your product alive and Then we have these use cases and for us. It was a challenge that we want to learn because maybe maybe Things might change in the future and the current strategies Are not working that are currently Challenged by these kind of use cases will bring us to something new and different or no Or maybe just the use cases will need to change if they want to adopt open source or Something new comes right so As I said the first strategy that we can follow It's like in as a as a company that is let's say expert in In in embedded world and lately in automotive and approaches these big customers is look The current model that you are that you're following is completely wrong. You need to update so you and when I mean to update I mean having updates as a core strategy towards When you think about your maintenance cycle so your business model also around it That is one strategy which is in the embedded world is what we Know that's fast as we would want to but is what we are seeing and we are seeing also that pushing the Automotive industry The second strategy would be like well, you know, we are doing things on the maintenance side of things on the kernel in some specific ways So what we are going to do is just extend Try to extend and push as hard and as much as we can our current approach and see where we get right and and Let's find out the limits of that technically speaking but also business wise economic wise also and process wise, let's find out the limits where it take us and Assuming that over time tooling get better and we are improving right Five or six years ago having The announcement that we got a few weeks ago that the LTS branch was going to be extended for six to six years Would probably have been a completely crazy thing And Then the following one the following strategy is just sit down Assume that the current model doesn't work. So we would sit down until somebody finds a solution that we can use Or we can apply with some guarantee and So those companies that are members of CIP would keep Would would keep doing what they are currently doing which is doing everything in-house their own way and And and and trying to make a good business out of that without killing anybody So obviously we went for the second option which is we are trying to push the the current The current strategy. Let's see what where it take us assuming that we are getting better and better at this and Yeah, so now Benny is going to explain some of those Some of the actions and considerations that we are taking With this second strategy in mind within CIP so firstly going to describe some of the Probable practical limits to the maintenance lifetime of a kernel branch So unfortunately, there is a what seems to be a hard limit on supporting the the current CIP kernel branch based on Linux 4.4 and that's the year 2038 It's kind of like the year 2000 problem which required a whole lot of changes in a whole lot of different pieces of software Linux systems like Unix represent absolute time or walk-alike time as a number of seconds since 1970 and If they you where this is represented in a 32-bit signed value the maximum possible time you can represent is sometime early in 2038 When Linux is running on a 32-bit architecture The system called API is that you use a space users to talk to the kernel and many of the internal interfaces in the kernel Use those two bit signed values so that will stop working and malfunction in unpredictable ways after 2038 And unfortunately even on 64 bit architectures there are some places in the kernel Where those two bit values are used? So to fix this there need to be changes in the kernel There needs to be changes in Knewlip C or any other C library That's being used Other libraries that use that have API's that involve time types mainly to be updated and applications that store time in binary files or use them in network protocols may also need to be updated The EPS work underway to fix this in the kernel Things have got better since 4.4 But they're not complete I am hoping that I and possibly other CIP member developers will be able to work on finishing that Having that in place when we start the next CIP kernel branch That work is probably not going to be possible to back port onto 4.4. So that's Lightly to be an absolute limit on the the lifetime of this branch second issue is going to be hardware the commercial support lifetime of most CPUs SOC's and some of the other Important ASICs used in these embedded systems is going to be less than 30 years And over time some of those Some of that hardware is going to break and it's going to be harder and harder to get hold of spares The only the most recent CIP kernel branch is going to be receiving back ported support for new hardware And after say 10 15 years, it's going to be practically impossible to add support for a new SOC So I would expect that some of the long-lived systems that are using the CIP kernel Are going to need to have at least as an option a Replacement for the Linux based components that would update both the hardware and the kernel And the third limiting factor is in the kernel software itself As you all know the kernel is is being developed and and changing very rapidly So internal API's and their implementation details that may need to be fixed change over time Times in quite dramatic ways And any bug fix that that is made upstream and ought to be back ported Might actually depend on some of the interface or implementation changes that have happened since the stable or CIP branch was created So back porting the bug fix Can become very difficult The longer the stable or CIP branch lasts The harder back porting any given bug fixes likely to become So how how can CIP achieve a longer lifetime than Current long-term support branches promise How how do we how can we possibly maintain a branch? for decades One factor that should make this Help to make this tractable is that the scope of maintenance is based on the needs of members The The CIP only makes releases The kernel source rather than back binary packages, so members can use whatever configuration they want But we're collecting together the configurations that actually use And using that to determine which features of the kernel are Are actually important to members For example on the current branch that's based on 4.4 The members are only using the arm arm 64 and x86 architectures So any bugs in other That affect other architectures are simply irrelevant. They don't need to be reviewed. They won't need to be back ported and Most drivers most file systems and all the different network protocols in the kernel only a small subset of those Would need to be maintained in the CIP branch So this should greatly reduce the overall effort required in back porting and in reviewing fixes compared to a regular stable or long-term stable branch secondly any any long-term stable branch if you look at the Get history and compare how many changes are going into each update you can you can easily see that the rate of fixes that's needed reduces over time of course, there's a finite number of bugs in any given release and So More of them get fixed the fewer of them are left to fix pretty simple arithmetic So a lot of bugs get fixed within just one release cycle Most of the remainder within a year or the major bugs get fixed within a year after five or six years at the end of a Long-term support branch There will be a much much smaller number of bugs number of major bugs to fix Upstream kernel developers can and mostly do help Stable maintainers to filter out in applicable fixes By including information in their commit message Which commit or kernel version introduce the bug that they're fixing? So although Bug fixes will get harder to back port over the years. There will be a lot fewer of them to deal with and The bugs that get found later will tend to be more obscure things Some of those obscure things are unfortunately security issues that quite probably do need to be fixed But obscure issues that don't have security significance and don't have a safety significance Might not be worth back porting to a to an older branch and third major Point is This is a collaboration Although I'm currently maintaining the CRP kernel I'm going to be handing it over to developers from the various members over the next year or so So the members of CRP are not going to be Customers who have to negotiate or plead with their vendor to keep supporting an operating system When the vendor and its developers really like to move on to something else It's an open-source project and Developers from all members can take responsibility for fixing the the things that their members need to get fixed While also pooling and sharing the the work they do on common areas of the kernel So ultimately the lifetime of CRP kernel branch Will be determined by the the interest and the capability and the will of its members To keep to keep it going So it's yeah, that's basically about it what we have to say. We don't need to spend the time if you don't want to right No, we we we deliberately since we know this is a Interesting topic for many we have about 15 minutes for for questions. I think that was the Most relevant interesting part for for us. So any question Should I hand over the oh I repeat the question anybody Okay, the question is if we do we have a strategy for testing the kernel? Okay Beside taking care of the kernel maintenance, we We also take care of the testing the kernel testing for now within CIP Yes, we do have a strategy We have some actions. We just released an implementation of kernel ci.org To try to test the kernel locally As a first step towards having a share common a common share service that everybody In CIP can use So in terms of service architecture for testing, yes, we do have a strategy in terms of tooling also now the hard part comes Most of what it's out there is very focused on in terms of testing the kernel in development Not so much on maintenance So that's an open question that we will need to figure out by ourself But well not by ourself with with others also because this is becoming a topic also for a GL and other groups out there But it's we are just taking the first few steps. So we don't have any real conclusion on that part yet The question is if it would be a better strategy to sorry To a new Sunnier kernel and then forward for the drivers If it if they are missing I leave that to you That may well be easier from a development point of view but CIP members have to consider That's a much larger code change and that likely means that Regulations will Mandate that a larger a much Longer period of testing and and read defecation So that's something that That that is is not not necessarily so easy in the areas where where CRP members are working. I would add to that that we are currently focused on one kernel 4.4, but obviously over time we will have more In I mean the the members are developing products in a continuous basis So at some point they will we will have to start At a point in which it makes no sense to start by porting because the hardware support will not allow it So we will have to have another another kernel in two three years. That's still undetermined So CIP will maintain more than one kernel over time Any other question the question is if we expect to solve the The year 2038 issues also in user space It's a big issue for members We are We we have identified year 2038 as a big topic within CIP and we have Clearly identified that as an action point from our side also We cannot just wait for the community to do it at their own schedule on best effort basis And there are some already ongoing activities from members in that in that particular area Within CIP something that we have discussed for Sometime in this event and I we are expecting some actions taking in that direction sooner than later currently Yeah, it just identified that as a priority with these big companies is is a huge step And I think that acting on the on it. It's sometimes easier than Understanding and realizing within the group that that's that is a problem and that we prioritize that as a need So I'm very very positive about CIP helping The more wider community in this regard because obviously it's an issue for for some of the companies. Yes It has to happen in in Mainline Right and we have to backport it. It's Well, if you were this morning in the talk it was upstream first policy and and also We have our policies in the current that clearly state that if it's not a mainline We should not backport it obviously for hardware support. We have some I would call it exceptions that are not really exceptions but but yeah, yeah, it's a We are not gonna do this for ourself and then Offer it to the community. We are gonna do it Within the kernel community and then figure out how we're gonna consume it So it can be the case for instance in which we fix issues for 2038 and cannot be backported He mentioned that risk. So hey, it's it's how it works in any case. We will be benefiting the members yes, you can use it and The reason why you can use it is not just because we have a credible maintainer behind it But the current members have plans to ship it on products Well more than plans. I will say sorry. I'm not hearing you. Sorry the question if if there is an estimation of Companies that might need this in order to estimate the development effort that will be put into this No, I don't but if you think about the three companies that are labeled there Toshiba Hitachi and Siemens If they as they are doing clearly state in public that they are shipping The the kernel and they keep working in the open for the core parts. I think that it's a quite strong Big development effort behind that It's a matter of wait and see obviously like everything in the open, but so far I'm I'm quite positive about that Right. So he started with 4.4 because that was at the time the the latest Stable kernel that had long-term support, which at that time we expected meant two years For now the CIP branch is is tracking the 4.4 long-term stable branch and adds Some back ported hardware support and a few back ported enhancements from the kernel self-protection project so all the fixes from 4.4 stable are going to 4.4 CIP Yes, it is Sorry, the question that he was asked asking is if everything from the LTS it has been taken Into the CIP kernel and the answer was yes. It has we have taken up the same approach than LTS at this point We are also he has been reviewing the configurations of members in order to understand What kind of features we will have to support over time I mean the idea of 4.4 being six years. It's just pretty recent. So we were expecting this year to take up Ownership basically of the con of the kernel, right? So it's it give us some more time to To adapt ourselves and learn in this strategy of pushing the current LTS model farther and farther and let's see how how it goes and until when and how much we learn out of it So you're asking you're saying that sometimes customers want the last kernel plus some small number some selected fixes As I said, CIP is not a vendor to customers. So in effect the the Members are going to be their own customers and if that they're developers if they want can can cherry-pick fixes in the short term for an internal kernel tree, but The expectation is that in the In the longer term they're going to be up. They're going to be taking everything from the CIP branch Which for the moment means everything from the long-term stable branch Any other question we have five minutes. Oh, yes, we have one The question is what are the main difference between the deviant kernel maintenance policies and the CIP kernel maintenance policies The main the main difference then would be that Debin is trying to debut firstly Debin is building is is shipping binary packages. So it's selecting a configuration which in general turns on pretty much all the features as long as they're not going to have a Huge performance impact for little for an obscure feature and Debin also tries to avoid a bi breaks which that it's not a not a concern for CIP at all But there is a quite a lot of commonality in the Debin says upstream first Debin does Does allow for back port at hardware support Does that answer your question Do we have time for one last question probably any final comment from your side then No, thank you very much for coming. We're done. Thank you