 So welcome everyone. This is a huge room and Decent audience Thank you for coming. So my name is Jan Kisker. I'm working for Siemens corp technology usually I'm talking about low-level stuff like like kernel development embedded systems real-time virtualization I would actually like to attend a talk in parallel to this one, but today. I'm going to talk about something more high-level About a collaboration project that we started this year together with other member companies On a topic. Yeah, it's called about civil infrastructure platforms And I would like to introduce you to what we are doing what we are heading for and then how the first steps Already look like so as we heard in the keynotes as we heard in various talks Linux is everywhere It's not just in the cloud. It's our daily lives also all around us in various systems We have an industrial systems for controlling machinery for organizing transportation systems transportation infrastructure for distributing and generating energy and Many other systems in healthcare systems in building management systems And on and so forth. So Linux is pervasive in all these systems and Yeah, basically you will find Linux systems in all these devices more and more you will find also some other things Still these days, but it's a consolidation going on in this area So we are talking about these domain. We are clustering together as the the civil infrastructure systems. So what are they? And these are technical systems responsible for the supervision control management of infrastructure supporting human activities Including all the domains I mentioned before and These networks are essential service for quite eventual services Provide shelter support for social interaction economic development. They are the society's lifeline. So we all depend on it More than we probably realize because they are just working and if they wouldn't work. We had a problem a massive problem and Well, they are running Linux So what is the difference of all these systems compared to what we usually know on our mobiles or on our servers or in the cloud? Well, there are some core characteristics make those systems different from other ones and We call them industrial great. What does it mean? Well, they have reliability requirements, which are all probably similar to other systems as well, but for a different purpose and They have functional safety requirements not all of them, but many You want to expect that the train breaks when it has to break And that's for sure if your cloud service fails. Well, you may have other problems but this thing Rules over a life of people and over high assets to be protected They also have security requirements Specifically more than these days will see this and they have real-time requirements means they have to react on certain actions in time And there is no chance or there's no option for failure in this But what makes them really different from a typical server or specifically from your mobile devices? They have a very long lifetime You're not throwing away those systems after they've been built after a couple of years replacing them continuously We have them in in operation for decades literally It doesn't necessarily mean that the software which have been written 40 years ago is still in use sometimes there is but It definitely means that those systems have to be replaced in their Defined architecture and defined environment that has been defined decades ago and it will like look like this in the future and That brings in a very conservative update strategies already of systems today. So The providers and also the users are very careful in updating any part of these systems and They want to minimize the risk of any regressions because if they fail as I said high asset is in danger and And the updates involve sometimes also a lot of formal regulations to ensure that the result is still safe is still secure and these Formal regulations require a lot of additional non-technical efforts and thus make the updating cost extremely higher So the business behind it the business drivers behind it Are about reducing of course the cost like everywhere and specifically the maintenance cost Not just on the software or on the physical system itself on the whole You don't want to go out for some remote system and and actually attach to it and do some physical maintenance work because this is extremely costly and So whatever we do it has to maintain the remote maintainability the poor example of those systems and There are often masses of those systems automatically deployed so You have to really look into the cost and introducing this and Then there is of course the need to reduce the development cost on this system. You will see where this come from We don't want to reinvent the wheels And the development time assistant as an aspect very important So while the system lives that long they tend to have a long development cycle at front As well often defined by the hardware involved by the large mechanics involved in physics involved But also during the due to the long development time used to be in the software world So there are some changes now While those System stacks used to be pretty preparatory in the past. There's commodization going on already in the past Yes, but now really for sure the differentiating level moved up significantly and Companies involved in this business. They no longer differentiate about the lower level features Over the whole system and the value is much higher than it used to be That means more and more standard components come in While they have to maintain of course these properties are mentioned on the left and Another thing which comes in is the connectivity So we're talking about IOT also industrial of course And that means that these systems would used to be pretty isolated running in and yeah Physical boxes without any connection except maybe power to the outside are now suddenly connected to the cloud So they have to fulfill the same security requirement even of not harder than all the networks appliances and all the services you have out there as well and That of course breaks some of the assumption previously made like you develop once you ship and then you are done No, actually the hard work then only starts so the maintenance of those systems over that long lifetime is now really essential also for security aspects and therefore We founded this civil infrastructure platform project as a collaboration project under the roof of the Linux Foundation simply to bring in This requirements and bring in the members we have already and work on a on a super long term Maintain industrial great embedded Linux platform for this domain So what has to be done? first of all the joining the forces on the commodity components That means there are a lot of already available. You can just ticket But sometimes we have to add certain qualities as I mentioned before those requirements from the dust will be Industrial domain extend the existing frameworks and existing components and that has to be done and best done in a collaborative manner and Well, we also don't want to go through these forks and Branches that many other domains already went through and these things have to be done upstream That means we have to search for the collaboration with Respective open source communities and work with them together Ensuring the quality already in the mainline project and not only in some in some forks and special versions But then actually the hard work as I said starts the maintenance the long-term maintenance and this is significant costs These are costs that we definitely need to share in order to yeah Keep the overall cost of these systems reasonable and yeah, as I said, there is no differentiating actually on this part And then there are a lot of things coming up the space of the development is increasing and And the best way to adopt to this is to share also the innovation on this So work together on the core components on the core frameworks that build those systems now and in the future Regarding IOT architectures regarding machine to machine connectivity and things like this so Our base layer that we want to define in this civil infrastructure platform We are working on as I said is a is an industrial grade software stack Fulfilling safety reliability security and maintainability requirements There are some gaps to be filled regarding existing open source projects and that's where we come in that we insist on providing these kind of qualities Support the existing projects with adding these qualities to the baselines That may consist of specific specifications of the on-device Software stakes so a vision behind is basically it in the end. You just say okay. I'm going to develop an industrial system I will pick the standard open source deck for this Which is well defined like an Android system for the mobile device like in a carrier grade Linux For the cali-com community or like the automotive people are doing with their systems There is no differentiating in between these systems anymore It makes easier to develop on these platforms together that involves the kernel that involves the file systems the base system Forselected reference hardware of course it also involves the tooling and around so that we will have The ability to rebuild those systems to change those systems not only now when they are being deployed But also after decades while they are being maintained and upgraded continuously Testing involves this testing frameworks Intensively currently done with a whole system by each member themselves for each product But of course the base layer is easily shared and also regarding the test efforts and Yeah, SDKs and APIs around this So basically it's the idea to trigger in an ecosystem development For industrial base systems around these core components There's a lot of things to do and as we are still a small project We have to focus on something and this focus is specifically right now on the long-term maintenance Maintaining components for a period which is beyond what's currently being done specifically for the domain or the embedded industrial domain so Again, this is the list of of Topics and components which is broad Where we see potential activities where we haven't said okay, we will do this and this definitely and Well, except for some aspect actually it starts on the device side with the base system like the kernel with base quality that this kernel has to have like real-time support like certain virtualization features we need and It goes on to use a space component a set of base component we need for these systems And up to the middlewares and like standards we have out there OPC or are things like this where we want to move to on the side again the the tool aspects of it and Built environments which have to be available for this long period may it be your base may be other solutions based test automation tracing reporting Configuration and so on there are also aspects beyond this non-technical where we can collaborate on Concept the safety for example the safety domain is a broad field Applied this on on open source component is a hot topic Not just for us, but of course is a consolidation point for us to bring this together But also aspects like license clearing which is important for our domain specifically we are shipping devices We are shipping software with open source license on top and yeah They have to be a fulfill the requirements and the quality that you expect from today to clarify just briefly the scope of what systems we are targeting and the field Goes from the well kind of server like environments high-end device device environments Down to everything which is a capable of running Linux. So we definitely exclude so far The smaller systems and micro controller systems which are still in use in certain areas But for our domain the specific focus on the more complex systems Where Linux is capable of running on so this may change of course as Linux is improving, but That's currently the focus on this As a project is currently set up. Well, we have a number of member companies Providing the budget of our project and from this budget we primarily Finance our developer set. So we have a certain number of fixed developers See more about it later Plus each member is invited and is participating with own developers and in the activities around it and That leads to contributions Other to existing collaborative projects like the real-time Linux project for example or distribution projects Or to the creation of new projects on the long run. So currently there is no concrete project being launched by us But this is something we definitely keep in mind and All these contributions to upstream open source projects again have to be Not just done once but have to be maintained over a longer period So all these projects have some kind of maintenance already Some only on one version others really with a complex system behind it So where there's a gap where there is no long-term maintenance of 10 years plus And this is the most case is the case and we come in Either by supporting directly the communities and extending their existing lifetime or by taking over when they move on to the next major versions Most currently is this of course been done. This is the most prominent topic right now on the on the next girl very important This is not a Disable goal of our project is not the fork of existing project. The goal is Working upstream so we have the policy set to ourselves that if whatever we adopt and and work on and maintain for a long period We don't want to maintain something that we meant ourselves that comes in from the side way We only maintain what is accepted by the response a respective open source project as as upstream component That of course means that yeah Some of the BSP like approaches we find out the industry is not really the way we want to go For long-term period and you can imagine easily that this is also not technically and a quality point of view the right way to go So looking at the kernel how it looks like in this key component for us right now We already have of course long-term maintenance there So there are the LTS trees the LTS versions with a lifetime of a couple of years Some longer some shorter, but none of them in the range that we are heading for There's also another initiative to call the LTS I long-term stable initiative, which is not just Maintaining regarding security fixes and and functional fixes, but which are also initially Using their their kernel version in order to broaden the support of an older kernel version so they they include back ports of of Board support packages of Soc support which is upstream, but not in the version they want to maintain over the long period So they have meant basically what LTS is for That exists and this being used in existing product a lot But right now every company does its own so every company picks a certain version from a third party from its own development Defines that this version to be in the product X and then maintains it for this product in ideal place Maybe for a product series or for a broader scope, but it's not really a collaborative work as it's right now what we want to Do is to replace this lower part basically to offer a replacement Where we collaborate on these work and for the kernel that means concretely we extend the lifetime of LTS Not every version That's probably not dealable not manageable from the resource point of view at least not right now But for a well-defined set of versions and and we also implement something which extends The support of a specific base version by additional features additional hardware support So I will see the later on there is a merge window basically which enables the back porting of features the back porting of Of board support into our base version and then and eventually we close this window and go for for the long-term maintenance Regarding security fixes regarding functional fixes and that while we always talk about this 10-year thing So 10 year is the lower bound basically this is currently what you get from many of the long-term available SOCs as lifetime so Products are being made with the same SOC for this phase basically then they have to be redesigned because the silicon vendors provide some new Version and no longer ship the old one And that's possibly the point where you also have to rethink about upgrading your base version But as long as those hardware replacements are identically available the software stack should not change in most cases That's basically the area we want to work in with a kernel specifically but also with other components So the concrete plans going on regarding the super long-term stable kernel Well, we have to establish the development process for us looking a lot at what existing project are doing like LTS I with this merge window thing. I mentioned it already and with a certain validation behind it, of course We will repeat these releases naturally to be defined period And we will have to establish a strong validation on this Kernel test infrastructure will be required But also more testing on the device is additional feature than it's currently been done and The result shall be shared so in order to start off with this we are very happy to announce that we Just created or are creating and it's a super long-term stable team for the CIP project so Ben Hutchings Employed by by code thing will join us and support us as the super long-term kernel maintainer Ben Hutchings is well known for its debbing contribution and Package maintenance and he's currently already maintaining several LTS kernel versions for more than the couple of two or three years actually And he will extend this work for the CIP kernel to come And he will additionally be supported by another developer which is to be defined And start his work in September So very soon first of all with a setup of the development environment We need and the validation process that comes first naturally and we will then prepare and perform the first else SLTS kernel releases and further on there are The support for super long-term will have to be extended to other component other core packages And his expertise in this area of maintenance long-term maintenance We will reuse of course in order to extend our support to other Areas of the core system not just the kernel So which version of the current will be a good question as was all Long-term kernel stable releases always a miracle which version to pick and we also thought a lot about it It's no official announcement yet, but I would like to make a bit more transparent what we are thinking about and how We thought what the criteria are for us. So of course it makes a lot of sense It is essential that we base our own version of super long-term maintenance on an existing Maintenance long-term maintenance kernel. Otherwise, we would replicate too much work And of course, you also want to synchronize with LTS is so work being done there Should influence and and benefit we beneficial for our work and vice versa we are working for the same targets for the same goals and There's overlap a lot. There might be some difference or there are surely some difference in certain aspects But in general the more we can synchronize the more we can benefit from each other the better Key criteria, of course is the broad usage of a specific kernel version in civil infrastructure platforms So we were thinking about what version to pick and initially one option would be taking an older version, which is currently being deployed in the majority of products of the members and Which is also being Going out of maintenance rather soon and basically yeah, right on do the job on this version And all the alternative is pick something which is currently being under preparation to come up in products So which is currently? Yeah in the selection process so to say for for products to come of the members And this is the second part actually we chose to go because it's way more critical right now that we have Perspective for our upcoming projects products and where which version we can really rely on to go for So We are still in discussions. We have an internal idea on this It's not yet completely settled and we are open for further input and ideas and suggestion on this We will announce as soon as it's ready and where we are feel confident about that we can really Provide this very long-term maintenance with a reasonable low risk The final decision of course is with our student committee This is how the project works, but again, we are open for discussions and input on this essential for ensuring that These long-term kernel when they pull in updates fixes changes are still stable still in all regards Major enough is testing So one of the actual concrete tasks to do before Releasing a kernel is to ensure that the test infrastructure is working properly So the goal behind it is and yeah performing tests on real hardware, of course Because this is what is specifically different our domain the various of industry hardware And we will define certain reference platforms based on the input that we have on the existing members and the products we see as relevant for this and Yeah ensure that this is covered by the tests and We have to have a short round trip time of the tests So within hours and not days or weeks To react quickly on on security issues and be quickly ready to ensure that a new version is ready to be released So not in the focus like others are doing right now is the long-term Continuous testing for example like the OS ADL doings with a real-time kernel. We don't want to replicate these efforts But of course what is required in our domain is availability of the results You have to sometimes prove it to certain authorities what you've done And so it's an archiving feature and the insurance of the regression freeness of older versions versus newer versions Has to be sometimes provided. So we have to have these results long-term available And of course no replications of existing activities So we try to we will try to align the activities with other communities like for example the AGL Who have similar requirements and similar goals but different target systems and but yeah similar testing infrastructures So what are the currently looking into and is we're using the the kernel CI infrastructure for this also going to see I Framework for this. So we have currently already set up a private instance of this kernel CI framework and So that member companies can run on their in their own labs their own platforms on it Not just a reference board like we have everyone but also the more interesting things Which actually are the products that we want to use the specific version on so that's also the idea of this framework Having a distributed test infrastructure where everyone can bring in their own systems without having to hand them out Maybe prior to the release Including what we defined a standard hardware rack where members would members can replicate and build on and yeah at their own tools to this So it's it's purely locally operation and we will share the results of these tests in the public on the public website and Yeah That's that's the the base systems or kernel CI if you don't know it It's basically testing if a kernel is still capable of booting a specific board not much more tests are performed So if the system is up and running and reacting test is passed Of course, it's not enough and specifically you look into things like real-time requirements and other things you have to do More testing on the target. There are other frameworks for this for example for ego is One candidate which have to be combined with this in order to ensure that yeah Also the functional features of a more complex system is still fulfilled when we update and Yeah, as I mentioned the goal is to feedback the results plug into existing Kernel CI deployment possibly deliver in the same platform also our results for our The kernel is one thing but of course the system is more than just the kernel And there are also user space packages involved and essentially you also need to support for these packages So user space is a vast area and if you think about how many System any components are in your embedded systems possibly there are quite a few but usually they are much less than you have in the big machines so essential for us therefore isn't the first step to really focus on the on a minimal bootable and System with basic functionality available the minimal viable system for embedded industrial control system for example Is the criteria to selecting packages into this? It shall be components which are commonly used in these systems nothing too obscure nothing too unusual and Primarily interesting of course are those packages which are security sensitive which either have some external interfaces or our processing data Coming in and yeah have security Requirements not only but primarily also important of course and even this is component as well established and We have to assess if these components stay like this It's available for a longer period because we have to maintain it for decades for a decade at least And so it's not that easy to pick the right one and some of the component we find in in the other domains are attractive but also frequently changing and frequently Moving in their directions moving in their relevance possibly so picking the right one Which will stay more or less like this on a long period is of course another challenge and the Korean to put into a mind Again, we are open for discussions Taking inputs taking suggestions taking criterias So you're welcome Just to give you an idea how this could look like it's really small on the device side What is essential for the system to boot up besides the kernel with its special real-time features on top? and The base system you always have is things like the busybox or similar tools in order to run something and control something We have a base library and You have some essential components for for security for connecting to the devices Secure SSL libraries and open SSH are things like this on the one side on the device and On this in the same side, of course You also have to ensure that what you are building for the device can be rebuilt after a long period So you have to maintain also the development environment in one way or the other That's not the same approach you're applying of course for the on-device components But in the end you have to be able to keep the same results reproducible after decade So whatever comes on the development side has to be considered at least and possibly also tuned to a certain degree In order to ensure this reproducibility so as we are still rather small project and We give us ourselves some time to grow Focusing first of all on on the kernel and on a small number of core packages in the first phase Ensuring for those that these super long-term stable is capable is possible with these and would be Provide the releases on a regular base on these components and Then slowly grow in scope Regarding adding more packages to this Broadening the user's user base and on the long run also adding additional Features on top, maybe creating new frameworks new components new middlewares specifically Missing so far in our domain or not being available as an open source and a solution yet The milestones we set ourselves For this year's look pretty good. So we launched the project in April We established the the requirements on the base layer and We are currently working on yeah realizing them. So we have Defined the common component for us which are relevant in feature size so The the feature of the kernel basically What is now to come basically is to bring the the maintenance into operation Announcement of the maintainer is one step, but of course we have to deliver also On on bringing up the development process on bringing up the test infrastructure and then on releasing the first version on this That's the next thing to come. So realize the plans that we made and then the next but one year basically extend on the features and The number of components we want to have so this is of course an advertisement talk to join the activity In one way or the other we have most interest of course the members but not be also interested in discussion and exchange the currently member set the founding members Hitachi Siemens and Toshi bar together with code thing and platform We are in constant discussion with other interested party just before this Presentation we had an interesting talk with another candidate on this And we see there is a big chance to join on this topic In an industry domain which used to be pretty separate pitchy isolated eyes and sinking in silos On this platform So there is a big chance behind it and coming together on this way Why join? Well, we ship our results in public. It's open source. It will be available There is no desire to join you can just use it of course, but you can't influence it The project decision to be made of course up to the members Regarding which version will come which version to pick where to focus on which boards which SOC's which components to include That is the added value of joining Besides demonstrating to your customers to your users. You are compatible with this infrastructure with these CIP base layer And there are additional values on top of course But this is the cure actually we are looking for we are providing for the membership So just for reference for the contacts But I'm also around talk to me talk to many of these people here in the front rows who are part of the project supporting us Go to our website look at our mailing list again It's a it's a still a fresh project, but you have to keep in mind our domain is slow by nature But once we are moving we are moving for a long time So don't expect a hip active yet Development as you may know from the colonel or from other communities other collaboration projects But we are growing we are growing with a long breath, and that's the good thing about it So thank you for attentions, and I'm open to take questions. I'm also open to ask questions So who of you here in the audience is involved in this domain or is delivering embedded systems or Is you are you just basically interested though? Okay there at least two Three four well, I know of the members Yeah, yeah, it's an interesting domain as well true Yeah, I think there is a lot of overlap so we defined as a social infrastructure platform But one of the some of the requirements maybe not all are shared with other domains so a lot of Interest comes also from automotive, and I'm pretty sure also automotive will share and benefit from our work while as we are Benefiting from bear activities to standardize on Linux systems They have a little bit different requirements in some areas and a little bit different lifetime sometimes But in the end it's the same so another interesting aspect actually yeah Well, we are in the big business so to say we provided solution for the large deployment specifically But this doesn't exclude Medium-sized or even small sized once we can standardize on the platform beneficial for all Yeah, that's the idea cool well Thank you all for coming. I know this is an untypical event for this kind of topic So I'm very pleased to see you here nevertheless and Wow, if you have further questions follow up don't hesitate Directly or we are our electronic channels Thanks again