 So hello, everyone. Nice that you made it and even left the snacks alone and come here to listen to our talk. So my name is Stephan Ivers. And this is Philip. Hi, nice to meet you. And we will tell you a little bit about our story, trying to find our path from embedded to Edge and trying to introduce product lines in our company. So we are both from Bosch. I am from Bosch IO. That is 100% subsidiary of Robert Bosch GmbH. Yeah, and I am directly hired at the Rob Bosch GmbH or the company. Yeah, so I don't know. Typically when I'm coming to the US and I'm talking about Bosch, many people don't even know exactly what this company is about. So I want to give you a little bit of background for Bosch. So we are not only producing dryers and dishwashers and things like that. We also are the biggest or one of the biggest car parts manufacturers, so a so-called tier one. Besides from that, we also do drilling machines. We do things like on the ceilings, this fire detectors, and all these things. So we are producing actually a lot of things, many, many different kinds of things in different domains. So also when you try to produce something, we are also doing a lot of industrial manufacturing devices. So we are therefore working in many different areas and we are producing many different things in these different areas. And that in the last 10 years, when we're observing what is happening in the technology, many of these things are getting more and more connected. And that's what they call more or less the internet of things. So getting all these devices connected is changing a lot. So that's not the only thing. We'll talk about other things later. But changing, connecting these things means that actually they are also somehow, let's say, exposed for any kinds of attacks and things like that. You hear a lot of stories about these things. So it means, on the one side, that you need to protect the device against these attacks. On the other hand, it also means that, for example, you have the opportunity typically to update. And when you have the opportunity to update more or less customers and everyone around it, it's expecting that these things are also getting, let's say, more modern over time. So all this brings a lot of change. And this change that is coming with the internet of things is affecting these companies that are producing devices when we are there to a large extent. So what does this have to do with Linux? Of course, we use Linux in many, many devices and we did a little research internally. And we are expecting that in 2025 roughly, we will produce around 50 million devices based on Linux per year. And when you do a little bit of calculation, so the prices of these devices vary a lot. We have some devices that are rather inexpensive and we have other devices that are really expensive, like even up to $100,000. But in the middle, when you say, let's say, roughly about $100 per device, then $50 million. And you say 1% is only invested in the Linux part. You already have $50 million only for the Linux part. But we've talked to many different of our business units and the different business units, some of them are even saying $50 million. That's what we are spending on our own on that stuff. So it's even more. So that's in a certain extent, it's like a lower limit what a company like ours are expecting that they spend on Linux, one way or the other. Like maintaining it, developing it, making sure that it works, all the things together, at least, so it's like a lower limit. But I think when we are at that point, it will be significantly more. So Linux is important for us, like for, I guess, for nearly every industrial manufacturer. And when you look at the things that happened in these 10 years that I'm looking at, I already said, because they are connected, things are changing. And the main thing is, before it was more or less, you develop the device. And when it's finished, you try not to touch it. So you focus a lot on the building part, a lot on everything that happens before start of production. That is your main focus. But with the changes that I described, you also have to look a lot in the phases afterwards. So you have to look at the time when actually the maintenance takes place. And that also brings a lot of changes into the entire development process and how you treat everything. And that's also what we can see when we talk to all the different projects that are doing Linux devices inside our company. One thing is that when you look at the problems that they are having, so as when you're really focused on trying to get the device finished according to the specifications that has been given to you or making your customer happy, so you have a very clear goal and you have a more or less like a deadline. And when you look at the maintenance time, it's very different. It's much more open and you don't know exactly what to expect. So collaboration before was maybe not so important. But when you're doing maintenance, you have to do it in a very different way because you have to see what is happening. You need to know what are the problems. You need to know how to fix it over a long time. And so you need to establish something much more sustainable than when you're looking at single devices and producing them. So that is something that we are learning. So maybe you haven't seen Bosch or the name in kernel development or anything like that. That's typically because at least in the past, often what we are doing, we are paying third parties to do these things. So it's not that we are not doing anything. We are just consuming. We are doing a lot of things, but it's actually through a third party. But when this becomes such an important element as we had it, then it more or less becomes a crucial part of the business, that you understand how to do the maintenance over the long time and how do you do the collaboration with all the projects with upstream, with corresponding open source projects, with distributions, with partners that have the expertise, with manufacturers, service providers, with suppliers. So the entire ecosystem is changing, and that's what we are seeing. So in all this, what we started was we were trying to identify a way to change things for us. And so we started an internal project where we're trying to do as many as possible, looking at all these different topics, trying to understand how it is, like seeing what is the structure all over our company, what can be consolidated in that, and what is the interesting part of that where we can focus on to make things better for us. So it's actually very easy. So we have the core element that is more or less like a Bosch derivate of any kind of Linux that we can use in as many devices as we could. That was the idea that we had at the beginning. It turned out not to be so simple. So you have the Linux part that you focus on, but then you realize there's a lot of things around it. You have an entire system around that. And so we have focused on many different areas. And one thing was, OK, so we have one area that is automotive. That is a central part of it. And we developed a Debian derivate that was another company. And that was something that we called then a purchase. So that is for, in particular, it was meant for infotainment systems. But we will see later that it's also now used for other things. But we also have things like the communication with the cloud side. Of course, that is in IoT an important part. For that, we have different open source projects started at the Eclipse Foundation. And one of them is Eclipse Kanto. That is another important part of that element. Also the open source management that we are focusing on, that is compliance is important in such a scenario for us. And we've also done a lot in the open source review toolkit. We also tried to do other areas. So we have, for example, Eclipse Cuxar, which is a project where we've been active that was trying to get the IoT stuff more used and more usable in the automotive area. That was originally like a publicly funded project. And it became an open source project. So that is like what we have been doing in the past. But we realized this is still not enough. And there's a lot of things to do, a lot of things that we need to do in the future. And we realized it's also a cultural change inside the company. And that is maybe even the biggest struggle. And anyone who knows how to organize change, those cultural changes are maybe the biggest challenge in something like that. So it is actually not so easy to do these changes. So it takes its time. But we are moving, and that's helping us a lot. And we are also really thankful that it's working with many different people and open source projects and open source communities, foundations like the Linux Foundation. And it's really great that these things are getting forward. So we have many other things that we now doing in the future. And Philip will tell us much more about the story in these areas that we have not so much investigated in the past and tell us more about it. But before I do that, before I'm handing that over, I want to give you the focus areas what we are currently concentrating on. And that is actually what you see here in the right side. It's container software management. So that's mainly open source compliance. And like mainlining the SSC drivers and things like that. So getting the things in the mainline kernel is something that we heard about it today. For example, Tim Burt mentioned that this would be a great thing that is happening more. And that is also something that we would like to see because it's helping us a lot when things are working with the mainline kernel itself. But also security, we have also heard a lot about security. That's also, of course, as I mentioned, a very important topic for us. So that's something that we are focusing on as well. And virtualization in containers is something that is keeping us busy because we see that this is going into the embedded space so that all the container stuff and the virtualization that we've seen in the cloud side is now becoming more and more also like a topic. How can we use this cool technology in the embedded world? And that is a topic that is also keeping us busy right now. So now Philip will tell us a little bit more about the things that we are doing here. Yeah, thanks a lot, Stefan. So first of all, maybe from the functional areas which we've just mentioned, I will not touch virtualization as much in this call, actually, not any further. And I'll also leave out the security parts for security. We will actually submit a talk for the embedded Linux conference in Dublin and see if it's get accepted. And then we have a full session just about security measures, which we have done. And now I'll just start with the reference system elements, one elements container where talk more about the USS compliance. I will have two slides, and these are rather short. Also there will be a proposal to be hopefully accepted at COS as summit in Dublin. And I spent a few more words on a purchase for the SOC up, driver, mainlining, and so on. And well, this is the embedded Linux conference, so it should be a longer Linux part in there. Right. I'll start with the container work in there. We originally had a large set of different container elements. We looked for what there are embedded container frameworks also, which the guys from Foundry's are all there. So I guess you're also providing elements. We have Balena, Toradex, and others. A lot of these frameworks actually work either based on LXC or on Docker. Flatback may be an example for a different kind. It's not exactly like this. And what we also consider was Pottman. I judge Pottman as a little bit more lightweight than the full Docker, but it's having the same API. So basically, you could just replay the things. And we would like to see how the influence is there. Some others are mentioned here, which we just put into conservation. We do some benchmark just to get a better understanding how things are on this. And what we were looking for is we wanted to get an understanding. As Stefan told you, we worked a lot with experts, external support, but it's only worse if you also get a basic understanding because you need to describe your requirements. You need to understand what it means when you talk about security mechanisms, about updates, and so on. So we want to get hand on experience and share this with developers so that also Bosch developers can go outside. First message, of course, there is nothing as such like the best container framework. There are benefits and drawbacks, depending on your use case. And just to show you some results, which we did. First, an important part, maybe CPU memory usage. We tried some stress and G testing. Shouldn't go too close to the speaker. Here, as they basically all use the same APIs on the knees because if you end up in same system calls and heavily depends just on the CPU which you have, how much utilization it is in there. Similar for the disk IO performance, well, there you are either limited by the MMC storage. If you put it SSD, it looks different. So you could really see that all the frameworks basically show the similar performance on this. So it doesn't have an impact. So if you want to evaluate on your own, don't spend too much time or do it in a more clever way than we did. Then some measurements, which were really interesting, were more on the round trip times, if you have TCP network communication from host to the container framework or to the containers as such. Here, important to mention, if you try to get a container to container communication or inside the system, of course, sometimes it goes over to virtual network devices and you suddenly experience awesome network transfer speed like 12, 13, 14 gigabit per second while you've just a one gigabit interface, you wonder what happens? Well, it never ends up the physical and to the fire. So therefore, I just took your TCP round trip elements and well, we are then certain latency and startup. Of course, if you start up something, it depends on the system size, which you have, of course, they're also started with loads more. So this is just here. You may think that, as I said, there is no such perfect solution, but it looks a lot like flat pack would be the best choice, right? If we just compare the numbers, what we figured out from the corporate usage for flat back, it's not as widespread as Docker and LXC. So the documentation is also not as extensive. And especially if you go into corporate use, you need to make sure that the different developer can also understand it, get fast training and so on. And we had to learn a lot and maintain a lot to get this flat back up and running. So we said, this is something for special use cases. Right, this is a key message here. Even don't trust only the performance numbers, also make up your mind on the use cases and what comes on top was documentation, which for example, also with the LXC, which is more lightweight, but there is no OCI compliance fully. So it's much nicer to have something which is OCI compliant if you have a widespread product use. Although there it helps to maybe exchange some framework extend or so. So this is something for the container, but what we then figured out, we need to orchestrate the whole thing. And there are a few solutions on there. So there's micro chaos or QB, you just keep edge. So there are solutions out there. Depending on your embedded use case, they may still be too powerful because they have too many features in there which you may not need. And then they are also not fully made for embedded. If you think about the reliability of your system and the network reliability, so typically when you come to orchestration, you have a whole server data center, then you communicate, but your embedded device may not be always reachable. So it needs to have certain autonomy when you use this, then you're not firing to service, but you have very limited resources and if you just prepare something where you see ICD pipeline, you may have a small device, a large device. So also how you orchestrate the containers on the device, how you deploy them makes a whole lot of difference. For this, we just started our work. We just thought there are a lot of elements, but similar like we had for the benchmarking, we see there's not such perfect solution yet. We're actually looking on potential cooperation and partnership to figure out who's interested in this topic, where to bring this forward. And then as we have a very spread use of technologies, our idea is to bring this very much forward into a mainline. So this is something very important for this. Also very interesting, I need to get a good hand over to my next slide, is the S-bomb generation within container. So because containers get modified, you don't always know what's in there, how to generate and so on. While you generate your container, that you may have things in place. And for this, I briefly want to mention the OSS review toolkit. There are good talks about this much better than when I have Maser Kurzman and Bosch colleague gave good presentation last year, December, on the OSS Summit or Compliance Summit in Japan, who will continue this in Dublin. Here what I like to stress is, it's very interesting to see that even if you do certain S-bomb generation, if you have compliance, you may have special licenses in there. You may have a corporate policy, what you're allowed to use, what you're not allowed to use. And here, a good framework, a very flexible framework is there and needed to generate an output. And you could see from the last slide, which Stefan showed, there is the S-pdx, this could be a good input and output powers for such frameworks. So I just shortly show the OSS review toolkit there, provides a lot of elements in there. There's typically what I like about it, that it's really modules, so you can just say, I need a certain module, which I haven't made use of yet. So I grabbed this one, I'll take the analyzer, take the report generation, it heavily relies on APIs between, which are quite stable or handle wire configurations. And so it's possible to interact with other projects and open source projects. And this tool actually gets extended, so we're currently working also that can handle not only Yachter, Debian, but also Albu, Ezar and other flavors of build tools on this. Apertis is the one which is supported in there, we haven't said that much about Apertis yet, which is the embedded Linux part of it, so we'll jump into this. As Stefan said, Apertis originally came from automotive. Why? Because there is powerful devices, some don't call it embedded even anymore because it's multi-core CPUs and the Raspberry Pi 4 partially has lower performance than the typical instrument cluster and navigation devices. So they're actually more powerful on this. But as we see, there's so much uses from the high numbers in there and everybody just starts and just say, well, I start my project here, I'm working in building technology, power tools, home appliances, industrial. Let me check, oh, I have an SOC here, so I grab the next Yachter release and I just start and I don't make my mind up yet in the beginning on what does that mean for security, what does it mean for maintenance and what I get a quick start. And for this, we said we need to have something in place, which is the full blown toolkit and this differentiates Apertis a little bit from just taking a pure Debian distro because we need to put it into a company-wide focus. But for this, we already provide pre-built images for different architectures. We have different reference hardware. We also have Raspberry Pi 4 supported and our old Minoboard, several so NXP devices are in there, so you get a formal set and we're continuously also increasing the number of devices in there. Documentation is important as well, so these are all the elements which are part of the purchase and this also enables that we come from an automotive that we can make use of this in further devices for like bicycle where you can see also there is an embedded Linux on the most powerful ones and those which have navigation enabled. There are robots which getting more powerful where it gets connected and for this also just real appliance came in. As this is quite, their extents use are a wide product range and we're talking about product lines. We can see that, well, we see the big question mark. I guess this is the most prominent part in there. We currently have a purchase and we try to make it more open source, get more people around it and why? Because we believe in an embedded Debian which formed the base which you can take as from the clearance of the upstream part that you say we have a package selection here. Keep this in mind, there's more resource elements and already now we bring everything back upstream which is done in our purchase but we still need a vehicle where we can just have short time purchase and until they are upstreamed. So basically there is upstream first policy but there are the demands of customers where you need to hook in and with a public reference, there's a chance to make use also from divisions if you don't need to really work in a product environment yet. On top there is then here like the product where it comes in which we have here a purchase pro which enables people in a certain environment so maybe you are creating your product and you don't want to have everything visible yet you're in an early production space, you customize your thing so this gets more. I like to compare it a little bit with the embedded Debian which is similar maybe to Fedora and then you have a send us like the purchase and then a rail for the purchase pro so just from the business model element in there and this typically goes into the products. It looks like many steps where we try to bring and push everything down like in a tree that we said it don't make much effect if you do it at the leaf but if you go down to the root and change thing there it's all the whole tree benefits from the work in there. How we do so also is that we enable also building source packages because we see there are a lot of packages coming and if you're Debian based more think about binary packages so we have a good set of binary packages but we include the building from source in there we put things together and create images we have a virtual file system in which we create an image to get a deployable part we have for S3 involved we may also change forward to Rook and there's something in there along we figured out they also need to have some issue tracking with it and also for our usage we have started with a fabricator we're now migrating further to Gitlib to have a consolidation on the tools so that we don't have too many things in there and there's lava for testing involved we have test farms because if you're using existing things there's much more beneficial than reinventing the wheel ever now and then again. Right. Some practical things to get in I would like to talk like for the handling of GPL S3 I may not make too many friends because there is a stream which says we should have also GPL S3 enabled but unfortunately I have to say there are sometimes demands which well the most restrictive license which I have seen more or less was on the benchmarking part and this benchmarking tool prohibited to share the results of the benchmark to another company or customer if this customer was not licensing exactly the same version of this benchmark and if you need to handle this kind of constraint somewhere and you have maybe a potential binary somewhere in there which you would like to also give to suppliers so then GPL S3 handling and the wrong handling and there could cause quite some cost extensive results and well GPL S3 is still something like poisonous in the automotive because also car projection with Apple CarPlay and Google Elements and there also causes to think how can we handle this anyway we publish things but we cannot have full GPL S3 part in there one pain point many talk about bash because there's a bash shall begin but Debbie and long time ago I guess it was with Ubuntu 604 they already changed to Dash as default but still there was always bash and there were still scripts in there there were script calls in there so we did a lot of clean up there with our external partner and so this was quite heavy to do a clean up word and to have things more sorted out another one was to change when core utils changed to the GPL S3 so we did a long evaluation I put the links down there so if you the materials available as public documentation uploaded slides also can just simply click on the links and here we went for rust version and in there while we're using the rust core utils which was well on MMIT license not my favorite one but it's very nice for the industrial use there you can also see upstream patches from our partners then what's still open which is not nice that we are on an old GNU PG GPL 2 version we have different elements in there we were looking for RP, PGP and also for other options but we are not using too much of it it's a lot of work is from the development and not really late on in the product use so the effort we're at the gain which we have currently does not make a rational to go forward there where the effort was really of use often which is also nice for a company perspective is the upstreaming work so we did some upstreaming of an in-house hardware which first of all sounds a little bit weird at least inside the company because the people know that it's a proprietary hardware they say why should I bring something which is a proprietary open source well there are certain benefits because if it goes into main and of course we have the testing responsibility if updates get in we still need to test at our site but at least we are aware when patches come in we don't we see it directly we get interactions and so for this we took the system master how it's called from the home appliance into the main line we had work for U-Boot arm trusted firmware some Debian work was needed on these device trees and we were lucky that there has already been a good support for the I.MX8 series so therefore there was a lot of SSC support and our hardware design was very close to reference design already supported so the network was less effort there pain was causing the U-Boot because the U-Boot support was more intent there was no direct board support for this we wanted to have network boot in there so that you can put a network file system in this was causing some pain for us and many iterations but this pain actually is the gain because you get the reviews upfront and if you have an engineer working in the company there's typically not too many working on the bootloader so they really get the feedback by others just compared to in-house reviews so the quality really increase what could cause pain of course if you hit IP modules which are not yet supported then you start looking into data sheets right drivers and some of them are actually not even known by the silicon vendor because they also just saw certain party IP for this we're just starting the work we want to get more public in there you saw the CIP project we wanted to also approach the CIP project so that we get more public work on mainlining activities and to see that all the silicon vendor may go away from the Wendell kernel right I surely want to mention I was we had Amarula who were involved there in this upstream process so they were not directly source from our project but they did the work in parallel while we were doing it and there was also a collaborator who did a lot of work so if you follow the page they are also mentioned here in the list you'll find the upstream works from Amarula and a collaborator on this so big thanks also to them what we would like to achieve is that we learn these things which now currently other partners do to do on our own that we start with a cultural change within Bosch so that work more together and also get more the folks together and I guess there are some further words on this and clothing words from Stefan so we'll hand over to you again um yeah so you've got a little bit of a glimpse into what happened or happens right now inside Bosch so we're investigating different areas that Philip described really nicely and it's it takes a lot to do this shift from the as I said like from the focusing on getting a device working and the way it should to the point that it's actually working in a like open source way that you actually develop it this way that you maintain it this way and that you phase it out this way it's a long way and it's it involves many areas and we described few of them but actually it's many more and also what we what you've seen for example that you when you take a device a hardware device that is not supposed to get in anywhere else but it makes a lot of sense still that you put in mainline for example another example why it was useful is different parts of the company are using different Linux flavors and when we have a mainline it actually like everyone hopefully can get the version that where it's already working out of the box and so that's one actually had a recent example on this where the one question business unit in Bosch was already using a certain device family and by all work which we did they're figuring out that another business unit also uses the same and they just starting a project so by this we have established a contact and they can now exchange information on this really on this level and we said why limit this to the company this should be much more on a wider spread community funding partners doing work together invest here because in the end we're all doing the same thing again and again and again and what all the business units in Bosch are doing other companies do as well and it's nothing which is differentiating we all would like to have a safer more secure proper world with a base ecosystem and make our money in the upper layer application and that the people do business and constantly they wear it there's the most benefit to concentrate on here as we are running shortly running out of time we have I guess I think another three minutes so we should give you like the opportunity to ask some questions if you have them do you have any questions if yes then please wait until the microphone is coming to you and also the upfront information that wins Friday they're still a BF session uh... from Tim Burt on the corporate usage of Linux so I think there's also a good forum to exchange and uh... to talk about things because actually for us the core element is that we have the opportunity to discuss it with you with other companies to go more into this direction collaboration onto this like let's say corporate level okay so questions otherwise we can fill the last two minutes easily nothing okay so we are here uh... until friday we uh... we would love if you would come to the birthday feather session from from Tim to discuss it or otherwise just approach us whenever you see us maybe it's not so easy with the mask on but yeah we would love to talk about all these things yeah thanks a lot for listening and uh... looking forward to talk to you thanks so much thank you