 Welcome back. The next talk will be very young. Can you see a kiss car kiss car kiss car on getting more deviant into our civil infrastructure? Yeah, thank you Michael Yeah, so my name is young kiss car You may not know me. I'm not deviant developer. I'm not deviantainer. I'm just an upstream hacker I'm looking for Siemens and Part of a Linux team there for now 10 years actually More than 10 years We are supporting our business units in getting Linux into the products successfully for that long time and even long actually And today I'm representing a collaborative project that has some relationship with Debian and even more soon I'm pretty sure so first of all Maybe surprised to some of you. Our civilization is heavily running on Linux and you may now think about These kind of devices where some kind of Linux inside or you may think of all the cloud service running Linux inside But actually this is about devices More closer to us in all of our infrastructure There are control systems. There are management systems included and and many many many of them run Linux inside So maybe if you are traveling with a Deutsche Bahn To this event these days There was some Linux system on the train as well as they were on the wayside so on the control side Energy generation so power plants. They are also run with Linux in very interesting ways and positive ways Industry automation so the factories they have control systems inside and quite a few are running Linux inside and Also other systems like healthcare diagnostic systems. So these Big bows up there the magnetic resonance Imaging systems. They're running on Linux for over a decade now Building automation so not at home, but in the professional building an area so actually As I said the train systems, they are going to be even more on Debian soon We have Debian for quite a while and power generation. So we in this case Siemens We have the box underneath on the third row and the nostril switch there is running Debian And the health care device is still on the window, but soon we Debian as well Just to give some examples and yeah, these are The areas where we as a group and we as Siemens are active But there are some problems with this just take an example From a railway systems usually these kind of devices installations. They have a lifetime of 25 50 years Well, it used to be quite simple with these old devices simple in the sense that it was mechanic It was pretty robust and you can I was once told that one of these locking systems They were basically left in a booth in a box out there for 50 years and no one entered the boss No one touched the whole thing for 50 years These times a little bit over nowadays. We have more electronic systems in these systems and They contain of course software And what does it mean just to give you an idea how these kind of development looks like in this domain? So it starts that well development takes quite a long time Until the product is ready three to five years Then in the railway domain It's mostly about customizing the systems for specific installations of the railway systems Not only in Europe there are kind of messy regarding the differences So you have specific requirements of the customer means the railway operators to adjust these systems for their needs and you see by then After five years already a Debian version would be out of maintenance and if you add another you yes You can start over again So in the development time you may change to the system But later on it's getting hard to change the system fundamentally because then the interesting part starts in this domain Not only in this domain, that's safety and security Assessment and and approval for these systems and it also takes time and So for example in Germany you go for the Eisenbahn Bundesamt and you ask to get the permission to run that train on the track And if they say I'm not happy with it You do it over again And it takes time and if you change something in the system It becomes interesting because then some of these certification aspects become invalid and you have to redo it and Then of course these trains and the installations they have a long life and as I mentioned before So how do you deal with this in electronic devices and software driven devices over this long phase? that's our challenge and it's just one example and there are more in this area at the same time what we see now is these fancy buzzwords from cloud business entering our conservative slowly moving domain and we talk about IoT Industrial IoT so connected devices we talk about edge computing means getting the power of the cloud To the device in the field closer to where the real thing happened So networking becomes a topic so in the past you basically build a system you locked it up Physically you never touched it again except the customer complained that there was some bug inside these days the customer asked us to Do a frequent update and actually not only customers also the regulatory bodies asked for this So you have to have some security maintenance concept in this which means regular updates regular fixes And and that is of course challenging for these kind of doing where you have a slow running and long running support cycles so to summarize there's a very long time we have to maintain our devices in the field and So far this was mostly done individually So each company and sometimes and not that sometimes quite frequently also inside the company each Product group development corner did it individually So everyone has having its own kernel everyone is having its own base system It was easy to build up so it should be easy to maintain of course. That's not Yeah, so this was one thing or this is one important thing and then of course we Not always are completely happy with what the free software gives us There are some needs to out to make things more robust to make things more secure reliable So we have to work with these components and improve them mostly upstream And that of course is another challenge we have to address in this area and Catch up with the trend that coming in from the service base from the cloud space So with these challenge oops It was the point where where we in this case a number of of big users of industrial open-source systems came together and created a new collaborative project as you do in the open-source area and This project is called civil infrastructure platform It's among the or it's under the umbrella of the Linux foundation So there are many project Linux foundation you may have seen but most of them are more in the area of cloud computing Or in the area of maybe a Automotive computing this one actually is even more conservative than the other ones and it's it's also comparably small and our goal is to build these Yeah, this open-source base layer for these application scenarios Based on free software based on on Linux We started now two years ago And that's basically our structure. It gives you an idea. So member companies. So the three on the top are the founding platinum companies Hitachi, Toshiba and Siemens We have code thing and a platform on board. We had an onboard from the first time as well And where does that's joined us and just recently also moxa So if you compare this with other cooperation courageous project, it's pretty small one Very small ones or a budget is also limited. It's still it's decent enough, but well, we are growing and Based on this budget. We have some developers being paid. So part of Ben is paid this way. You will see later on why and we have People working from the companies in the communities and we are ramping up on on working with communities Yeah, to improve the base layers for our needs Everything is open source We have a kid leprepo as well and you can look up there. What's going on there? so the main areas activities where we are Working on right now. So four areas Colonel maintenance So we started with declaring one colonel as the CIP colonel to have an extended Support phase for this kernel of of 10 years. So this is what we are aiming for Which is feasible already for some enterprise distros in a specific area But here we are talking about an industrial area an embedded area. So there is some some challenge Well, I'm saying 10 years. They are sometimes within 15 years. We will see after 10 years if we prolong to this Along with this of course comes a need for for real-time support. So we are also currently it's a separate branch But it's going to be integrate eventually have Premda T branch colleague of mine is doing this and as I mentioned before Ben is currently our 4.4 CIP colonel maintainer So, yeah, this is the core basically where we started activities We continued in extending this on on test infrastructure. So we invested a bit in Improving on on lover infrastructure. We are now ramping up an internal lava lab Just to enable the colonel testing, of course and then that's actually what I'm going to talk about today a bit more There's a CIP core. So the colonel alone doesn't make a systems You need a user space and you need a user land and and that's basically Where we are now focusing on or you're now ramping up Our activities is to define this CIP core means a base system user space base system Which you want to maintain as long as the colonel. So another 10 years thing So and our group had a couple of members which were already familiar with them in before And so it was pretty easy for that group to decide on Choosing Debian as the base as a base source for our core CIP core package package set. So why was Debian chosen? Well, it has an outstanding maturity And and focus on stability. So you are pretty much aligned regarding how conservative we see certain things Which is a positive thing for us. It has very professional security Properties, but we also rely on heavily and also another interesting aspect for us is License hygiene that you are after to ensure that there is yeah only free software in these packages and that is properly documented We when we are using and redistributing software well in contrast to for example the The service base where you don't usually redistribute things We are redistributing devices. So we are redistributing software We have to take care of the licenses that we are redistributing and that we are compliant With all these licenses included so it's very important for us that this is a consistent picture We get from the package someone took a look at this already And we are still looking ourself on this, but that's a very important thing So with these characters we chose Debian as the base system So what does it mean right now? We are currently in the process. I said to select the core packages from Debian packages There is to be some little bit of stripped down obviously So we are already working with Debian on certain long-term support aspects Just to mention two activities. So we were sponsoring already the the staging repo for security master actually, I'm personally not aware of the current status of this project, but We've got the feedback that is apparently a valuable thing for LTS activity. We just joined LTS project platinum sponsoring and We are now involved in the discussion to for this extended LTS activities so anything beyond five years And well in the end that's what we committed to our users We want to ensure that for the base system the 10 years is reached of course ideally in the community Not only based on our own personal activities, but in the end we have to fill the gap And that's basically our commitment on this Don't take this literally what is written here This is basically to reflect the package set we are discussing and there are well There are some 30 to 300 packages under discussion. So to say right now Which is we are condensing basically the input from our users from our members What they are using already and there's a difference We will see later on where these come from in the amount of packages in the way they're using so the kernel currently is Not part of the Debian thing we import although some of our users will directly use a Debian kernel But as I said when there is a need for additional activities That's why our CRP kernel comes in and then we have a set of base packages And then of course we also have to have a certain set of packages that we need to keep In a usable way to ensure the reproducibility of this base set because if we want to fix something After nine years in the field on the base system produced in the past We have to ensure if you can come up with the same result plus the Delta so there are different ways how to build a system and Compared to the classic installation you may know from a desktop on a server. You're not installing we are Prebuilding images and then deploy these images on the device either in the factory or out there in the field So the challenge for us is if we have these package list how to get to the to the device image So just to give you a brief idea. So, of course, there is some input from the CRP kernel in source form Then we are using predominantly pre-built binary packages from Debian and or source package or the source feed from Debian So the the Debian source the upstream source plus the Debian patches as input feed and that comes Down to a minimum base system to be generated and we are currently working on this There is no Defined way of producing this image within CRP at this point. We are basically following two paths One of them is the path which is dominated by the idea. Okay, we have to ensure we in this case The corporate environments have to ensure to reproduce the image ourselves the binaries ourselves. So we take the Maintained sources from a Debian community But we rebuild and then generate a new binary. So to see out of these That's one way and that's an activity, which if you heard about it meta Debian a project prominently predominantly driven by Toshiba which uses The Yachto like bit back like way of producing a base system but out of Debian sources so that you have a maintained Yeah source input feed for this production That's one path the other path is using predominantly binary packages and Personally and specifically also at Siemens. We are more following this path here So there is for example the ESA project Barjan is one of the developers here as well. We are working on this path means that 95 or 99% of your image consists originally of binaries Debian binaries as they are shipped as they are released and then there is often the need to modify a little bit Maybe the kernel may be the bootloader may be a specially patched package for whatever reason hopefully good ones and You have an infrastructure to assemble the binary images and to produce the source packages on demand and Install that into an image that you then can flash on the device. That's the second path. We are following As I said, that's just to describe the workflows and the technology behind it is not yet standardizing the CIP For us at Siemens currently the main strategy as I said is the ESA path So it's also bit big based Yachto like production, but based on the Debian binaries Producing ready to install Yeah device image so if we look at the Situation so what is Debian providing well a large set of packages a nice level of support three plus two years LTS mostly That's already great. I mean there's everything available almost everything in the world. It's for free software. We can get via Debian The build well it's a port native build. That's also advantage because well fighting after 10 years 15 years with cross build. There is always a problem with cross building Even a little bit. So this is a good strategy to go to go Although you're also working on cross builds that may be interesting for certain scenarios as well for us And we're all discussing this these days reproducible build is also very important for us Because we also have to prove that the Delta is really only on the Delta that has to be changed and not anything else And we have to rebuild something for whatever reason. We don't want to produce a completely different image in the end So it's a very important topic. I mentioned already before the license compliance topics I'm not really the the deep expert in all the licensing thing except when I have to be because some customer ask us internally How to be compliant and how to solve certain? compliance findings Colleague of mine my colleague for example who's maintaining the phosology project is way more in this Because we have our also infrastructure to ensure license compliance And and identify packages package ideas and the idea to as far as I heard is that to combine these kind of activities So that Debian can also use the information that these kind of scanners produce like spdx formats and and built it into the Debian 5 next generation and and in turn we can extract these information and ensure that they are still valid when we When we take a package, so there's a lot of activity already great in this area And of course testing not to mention so what we need to require here as I said one thing is what we need longer support face the number of packages fortunately is then much lower so Yeah, as I said so something like a couple of hundreds at most is what we are currently heading for for most of our devices We have the need to both build natively and cross build cross build predominantly in the development phase But there might be also cases where it might be useful for a productive image But predominantly it's for development phase. You want a quick turnaround time when you're building on xd6 for arm for example Yeah, the binary source packages should be managed and reproducible the last compliance already mentioned and Yeah, the testing activity is also something that we want to improve on further. So where we see the collaboration Yeah, already mentioned longer term maintenance for packages That's definitely an area where we we are reaching out and we are already in discussion Contributing to Debian cross there's activities going on this area reproducible we had some discussion hugger and Chris these days where we could possibly support you on this It's not our topmost priority at this point, but it's obvious that it will become in the future And also way possibly interesting for you. I think there's a good chance that these activities also open up more Adoption in the bedded space of Debian So because we also discussing these kind of things with our suppliers means the silicon vendors Pushing them to be more upstream in order to have it easier for us to integrate their work in our systems And eventually also enabling them to well use the same magnetic mechanism that we are using for building our images to build their our customer SDKs or however they call them and that can create a larger ecosystem and We have a discussing already with some of these vendors and someone actually I've been interested in in Debian as well as a Default image to replace those not so successful Source-built approaches that are out there in the field Eventually with something more easy to use Another area I really like to see that we have collaboration on is the regarding the license results So we at Siemens currently are running through with these subset package set to Phosology run and I would like to see the result of these run comparing it to what Debian is currently reporting in the metadata If there's any gaps anything that our experts say, okay, you should Document it more in that way or there is something missing and of course report these issues upstream because eventually I don't want to rescan every single security update package internally again if you did already That should just run through and we should have the trust that these information is accurate and and we can rely on them That's the vision behind it and Yeah test cases would be also an area where we see The chance to contribute something Further things we are discussing might be not that interesting for Debian But it's interesting in general functional safety activities You will be surprised how many people are asking for functional safe Linux these days may it be for automotive but also for industrial purposes Worth mentioning actually is the security standard this way So even if you're not involved in all these I see whatever stuff It's interesting because this is pushing us in industry to do things like update strategies even more system Consistently and ensuring that the image that we ship is in taker. So that is really the original image Up to the questions how to secure the boot and and how to secure this the system is running so that helps us to argue internally and externally for consolidation and That helps us currently to push a lot of these users and towards the Debian solution So one of our units did once a survey Recently actually about how many Linux systems they have out there and they counted 99 balloons and Linux systems Actually, and of course you can imagine it's pretty hard to maintain 99 variants in the field out there So they are one of the most prominent drivers Inside our company to consolidate the systems and we are currently consolidating over Debian not everything but most of it Yeah, and then there is this doomsday Date as well, which is crazy and creating an increasing concern because you can imagine that if you are building a device today Maybe it's out of business in 10 years. Okay, you're lucky Maybe it's still running in 20 years and it's not yet ready for 20 38 and then we have a problem So that's things that's going on currently already So one of our units for example is sponsoring activities in gdpc to prove the topic and as a Consortium as a CIP group. We are also looking into this We will not jump in on things which are already being happening But if they are gaps up there, then we will possibly jump in here as well So to summarize We believe I personally as well is very strong that our infrastructure as infrastructure is way too critical to run arbitrary software on it Which is happening not everywhere fortunately and we can improve on this together because There's a strong interest in our group to enable and preserve an open source base layer for this environment We chose Debian as a solid foundation because we believe that this is Technically a good solution and it's also a good solution because it's a community approach that we are also following We see that we don't differentiate Over these base layer. We differentiate between Our competitors on the higher functionality on the integration But not what is in detail is running underneath and this is a very great point to or collaborate and to work together And see a piece really looking forward to deepness support Off and the collaboration with Debian community So if you have any ideas or if you have any Suggestions how we should approach certain things, but we should also think about If you're free to approach me to free to approach the community There are some resources. I think I put it somewhere here No, it's gone and what Lee There's a meeting that we have It's currently dominated by by kernel patch reviews the back ports that we are doing for the 4.4 kernel But there are also other toys being discussed. So this is one of the channels. We have our Gitlabs Yeah, and we are persons around Meeting us maybe once in a while on some conference or whatever Free free to approach us and talk to us would be very much appreciated and with this now Thank you and I'm picking questions Thanks for the talk. Are there questions? the video team One one question you said you have like 100 different linux Variations, do you plan to replace them with Debian or is is the plan only for the new devices you your chip to use? Debian so unless you are really forced to for whatever reason you usually do not replace Since something that has been deployed to the field That usually causes more turn than to keep with it and and Biden the dust and try to prolong the support So this is more about the new things coming up It's a step Moving forward so currently we're discussing maybe a handful of in this specific domain where there's 99 variants come from a handful of applications ranging from virtual machine images On on embedded servers to the better servers themselves and the smallest thing I currently working on is a is a box for Being a data diode So it's a device which ensures that the critical network is not infiltrated from the outside physically But it still enables certain kind of logging functionality for this thing So these are the range that we are currently dealing with in that domain only And there's more coming Including hopefully at least for our company even for some of our platforms Which are open platforms means you get a hardware and you get a software stack on top Which might be windows or might be linux and and that Linux might be Debian Thank you for this really insightful talk. I'm totally excited about Such big companies putting some pressure Behind linux and Debian and the free software in general in industrial applications And I remember my heart jumping when I saw a Debian spill to two weeks ago on one of those info displays in the S Bond bringing you home from work Actually Even if that device was malfunctioning It was displaying a Debian My question is What is your vision about the SSL situation in in this very long-term? Maintenance, I mean we still have the vast majority of software using open SSL with its Problematic license and we have a gazillion of alternatives. Where are things going on the SSL side of industrial applications? That's a very good question. I don't have a definite answer to this as well So currently there is quite a bit of open SSL being Deployed it. It's not really new. It's has happened. I think for 10 years already And that's the state of the art so to say with all the pros and cons involved We are a little bit involved in in working certain certain features and upstream but in the end It's a question. You will have to see Hopefully these libraries become more compatible. So eventually you may be able to even replace Certain aspects of them in the field But if all goes wrong, we really have to invest also in the maintenance of these things So while we are currently also Facing this with the distribution. So hopefully the distributions will work on this as well and where there is a need We can collaborate on it Technically I would say it would be great to have something clean shiny and target it But then again, it's a pretty hard challenge to solve this and to work on this So I don't think that the major push will come from industry at this point our domain is still rather small compared to others But if we can Individually support something without having to initiative have to initiative for to write a call thing or goes a different way That would be great. Yeah, I said that I don't have a definite answer on this right now There was a question over there. So my questions are a little bit of licensing The GPS 3 is mostly a concern in the industry What is the policy or how is CIP handling this? special license So we don't have a definitive license exclusion list except it has to be free software We have a preferred license for our own activities actually, which is a permissive license But we also do GPL licensing in our group But then the question is what do we include in our package set which is already there So just these days I had the discussion with an internal user Who came around with ah these GPL B3 this evil stuff? That's it because in the context of we want to ensure that our device is really original So we have to lock down. We have to Encrypted and we have to Yeah, who's a skill boot thing? I said, okay Yes, the license says explicitly that there is the need to unencrypt the device And that has of course Complications when you think about the liability that we have for our devices So we run through all the certification processes and we stand for that and it now the someone changes this and How can we prove that this has been changed? So they were very concerned about GPL B3 and and I said, okay It might be more explicit in the license, but GPL B2 had the same idea and That you are at least in the in the gray area as well in this is license if you think about locking things down Fortunately, there are patterns if they will work something we have to see and that you can enable an end customer to replace the Software be compliant with the license But of course then avoid your warranty warranty on the device and on all these Yeah legal obligation that be fulfilled this hopefully this pattern it will work If it doesn't work, but we are not alone with this problem by far But we are very accurate in this that can be sure so if we do this we do it accurately Yeah, the concern I hear often is also with functional safety in that concern. Yeah, so Yeah, how how to handle that? So I'm not a lawyer so I can answer this definitely But I think the discussion I had a couple of days ago Triggered some ideas and there was one lawyer on the table as well So the units are thinking about these patterns and if they can evolve a legal construct from this idea, okay? Enabling the end user who ever has the device to replace the software and install their own thing without Granting the same liability on the outcome of this thing We will see so I'm really interested in that outcome So there's lack for that Any further questions? I just wanted to add two small notes you mentioned voiding of warranty if the user changes the Firm we need to be careful with that because that's actually not legal in the EU You cannot void warranty exclusively because the user has changed the software have to prove that the change of the software yeah Made something wrong with the device basically so and another note At collaborate develop a few tools which you may or may not find useful for Debian derivatives for like creating Images for example, it's called debo s So I see you use a slightly different workflow than we do because we have an OBS Which builds the binary packages out of the Debian sources? And then we like use the apt feeds and we build images out of that But still you may find the the the tool useful That's it. Yeah Yeah, we definitely look into what Debian tools being used in the community We don't try to devoid deviate from this wherever possible, but yeah, there are some special requirements that we are dealing with That sometimes make it easier in the other way, yeah Any further questions if not then let's thank you again