 Hello everyone. Welcome to our case study. My name is Stefan Ebers. I am working for Bosch IO. And I'm Sven Erich Joszeski, also working for Bosch IO. We both participate in a study inside Bosch and also in a more like a project that is trying to find the way forward for Bosch in the context of embedded Knudel Linux. So we want to give you a little bit of a background what the situation is with Bosch so you understand what is behind this project. So the internet of things is actually something that's really important for Bosch because we have a lot of things that we are producing one way or the other in the different domains in mobility, in industry, in building and kind of consumer goods. So that is all important for us and all these devices are becoming these days connected, which is actually a big change. As part of this project we have done some polls inside Bosch and to identify what actually the situation is and what the amount of devices is that we are talking about. And it turned out that in 2025 we can expect that Bosch is actually producing more than 15 million devices per year based on Linux, which is quite a big number. And all these different devices are actually right now they're not done with one way or one type of Linux distribution or anything like that. Many of them are done in different ways. So now we have the big challenge that we have to find out how should we go into the future and how should we do that in the future. And we realized talking to many other companies we are not the only one. Many have exactly this challenge, a rising number of devices based on Linux and also the connection of the devices to the internet. So like let's first have a look at how the approach is with embedded software creation. So for that Sven will tell us a little bit about it. Sven. Thank you Stefan for this nice introduction. So what I will tell you in the next couple of minutes is the situation that we perceived during our project in a couple of different embedded software projects like the creation of software for those devices. So of course since this is a case study this is more like a generalization of the different topics but since this is what we're aiming for with this talk. So the first thing that you typically come across when you have these embedded software projects is you have six different layers. So the first layer that you can see on the left hand side of the slide is the hardware support. So this starts with the hardware and also how is the hardware supported in software. So mainly drivers are included in the mainline kernel. How do I do that? Do I need to write my own drivers? All the things. Next level would be the software base system. So the question is what is the actual base of the foundation of my software running on the system? If you go up to the next layer you sometimes have an on software which is still more like a non-infrastructural level for instance integrating this different software distribution mechanisms but it's a bit above the base layer. So this way we differentiate here with the add-on software. The thing that's now coming more into this project is also the layer of runtime services. So with the approach of IoT or Internet of Things devices you have to take more care of integrating your devices with overall architecture, be it in the cloud or some other systems. So you need some clients on your devices to integrate with those backends and this can be substituted with runtime. And a really important thing in which sometimes tends to be forgotten is the question of tooling. So to enable developers to do all the other layers that I just mentioned in a really nice and efficient and maybe even fun manner. So looking at the embedded software project now but the task of the developers in this project and also the managers and product designers and everyone else is to deliver a specific device and a specific solution to the customer that somehow differentiates on the market and somehow has benefit for the customer in the end. And the way they do it they have this embedded software project trying to take all the different components from third parties, self-developed, maybe open source, maybe not open source and try to integrate this in the embedded software project. So it's a huge task to actually come up with this whole integration and selection and making sure that everything fits together to then come up with the device software. So the software that's actually running on the device beats the firmware. Nowadays also included with an update mechanism which is really important how to integrate it on software, backing clients and obviously also how to provide the extra functionality, the actual value to the customer which then works on top of all these layers that I just mentioned here. Another thing which sometimes tends to be forgotten a bit when you just talk about technology, this also needs to be more in a compliant manner so that you really know which components are in your system that you can track so that you can react to vulnerabilities but you also know that all the license requirements are met in your software so this is also that you have to take care of. So going to the next slide, this is a situation that those development teams typically tend to find themselves in. So they have to focus on something of course every time you need to focus but the question is what is the thing you focus on and in a typical project that we've seen you tend to focus on the build phase and this is the phase of actually creating the thing or the device or what you try to create in a more short-term way with a specific deadline in mind. So in our case often this deadline has been called SOP or start of production. So by that day it's the day you start producing your devices and you also start sending them out to your customers. You have to make sure that the software is up and running in a stable way. So this is one of your main short-term goals so you start maybe again with the whole project you try to control your costs also for your hardware that you get into so then you also have to try to somehow differentiate with your competitors to bring more value to your customers. So this way you also focus on these features and also I mean many developers love to develop features. They love to develop new things so they tend to also go into this feature development scene. So one way to achieve this during the whole build thing is that you try to make use of your local flexibility so you try to avoid external dependencies because you don't want to talk to external to align with external communities be it really in the open world be it with another department be it with another division it's always more flexible to just do it on your own machine like works on my machine so works on my device works on my concrete example or specific example but this is perfectly fine if you build something up but you end up with a couple of risks and these risks many of those can be summarized with one term which would be maintenance. So once you have your product in place you still need to focus and need another focus which would be now the maintenance and this sometimes maybe not that much fun than building something because we already have it but you still need to spend a lot of effort in developing your software technology and also the software on your device so you otherwise you might end up in a situation where you plan something where you build something like a house and it completely changes the purpose or its behavior or how it looks like over time and this is something that you have to react on or maybe in some cases also you have to prevent this from happening because otherwise you have a lot of effort to follow this. So one approach to do this would be reusing results to also share this maintenance and also to make sure that new specific components fit in the overall evolving ecosystem so this is kind of difficult sometimes because as I said earlier to gain flexibility it's often easier to solve your problems locally so this might end in the situation that you saw some redundancy so there's another benefit in actually upstreaming these changes or these developments that you made to get more in a core development mode which is obviously hard if you don't focus on the maintenance but if you actually tend to focus on the build phase because after your first device you just continue building the next device and the next device and over years you have a larger stack of things that you want to or would need to work on but you kind of miss a focus to go into this maintenance track and actually do the things in a proper upstreamed manner. So maybe coming into one reason or situation why this is difficult to shift this focus is on our next slide just one slide here you go it's get out in the shoes of a product team starting a new production and what they typically have they go into a room and decide okay we do things like this using these features and we estimate a certain number or capacity or cost or time or however you want to measure that for specific things and often you tend to underestimate certain efforts if you're more focused on the build phase one thing is the effort for the actual system integration and system development so every team every developer is developing a single component but now you don't have enough time or resources to do the actual integration the nitty gritty details of getting everything run on the device again also in a compliant manner. This even becomes a harder topic if you go to the Internet of Things because with the Internet of Things you definitely increase the attack surface to your devices by a lot because now basically everyone somewhere has access to the internet has greater access to your device so you really want to make sure that these people cannot get into your device because no one wants your light bulb taking part in a DDoS attack while you sit in a during room so that's something that sometimes can be underestimated especially even in the long run if you talk about the maintenance again over the whole product life cycle which is also one of the another risks in general that you tend to neglect these long-term effects because you first want to come up with some features and some value in the first place and then it's also often underestimated how much effort you have actually to spend into maintaining those proprietary elements or those differentiating features or those features that you consider differentiating but in the long run you maybe come up with the observation that all the effort that you have to spend to adapt to the changing system environment around you is maybe not the benefit that you get out of that but this is something that you just realize when it's maybe too late or later on so maybe to come up with a comparison it's few sometimes it's people in this room plan for an 800 meters run they know the goal it's some revisable we need a lot of things to do but it's you can do it in a fast pace and once they reach the finish line they realize actually the finish line is 10k so it's almost 10 times the distance so we have to go all the maintenance we have to shift our focus to being more in an endurance and reliable mode and once you come to this realization you might come up with and no one wants people to become frustrated but so we need to find a way to avoid such a situation we just drown in all these different issues of keeping up with updates of all these different dependencies be it something that you got from an external distributor be it just be it an open source project be it something that you bought somewhere be it some outdated components be it that you maybe underestimated the effort for system integration that I just mentioned and this even has increased this feeling or the situation because you missed out on the opportunity to share this topics like to share this complexity to share this effort or to go in a mode of co-innovation because early on you lost your focus on the maintenance or did not have the time and the maybe also the reasons to shift in this maintenance mode and this is definitely a risk that you can run into again I'm not saying that every project is always going this path but there's the possibility and we have seen it sometimes in some ways so this was enough for us to come to this conclusion that we might need to change things in this front and in this regard and to come up with a project on how to change this how the system development is done and how to enable the developers from these really specific teams again they really focus on their domain and their device to come on this more collaborative mode and to enable them in their work and the the progress that we have made during that process that's something that Stefan will talk about in the next minute. Yeah thank you Sen that was a very good and detailed description of like all the problems that we identified that I think not only Bosch but the typical industrial companies are facing these days was embedded devices. So you have to keep in mind like if you're someone that have done a lot of open source development that still a lot of this embedded software development for specific devices is done behind closed doors like on this slide on this picture it's because they're very focused on getting things done before the actual production of the device starts. So and we have been thinking about how can we actually face this shift that it's becoming more and more important that the maintenance is done in an efficient way and not only focus on the build part of the device with this very complex situation coming up as described and for us it become more and more aware looking at that and opening up this process and do it more in a collaborative way inside and also outside of the company might be a good way out and so we were thinking about like what how can you actually foster this collaboration and what does it actually need? That was like one of our intermediate results and we would love of course to hear your feedback about all this but yeah we just want to show you what now for us at the moment is the current result. So internal needs to actually foster the collaboration how can we do that and what does it need? We came to the conclusion that a cross-divisional service organization would actually help because it's bringing these different projects closer together and helping them to find synodity, she's between shared parts so meaning if one project is using something and the other project is using the same thing if there is like a service organization that would help either even external or internal one both is possible and also it would help if such a service organization would be there that a new project can start on all the things that already have been understood that it needs to be done like one way or the other or in particular for the corresponding situation so that they can just start faster and also all the tooling that you need which is quite a lot if you just look at testing or like all the different ways to store the code or also do the integration and all these things it's quite a bit already and also to maintain that and set that up is actually it's quite some work so that would also help if there would be an infrastructure as long as it fits of course we always have to make sure that each individual project are not let's say put too much in a box meaning that they lose the ability to just use and set up what they need and so that is very much important that you keep that in mind. A second thing is that there is already in these big companies a lot of knowledge and know-how around these topics but it's really hard to get that coordinated that actually this is shared across all the different parts of a big organization and spread and that would help to do that to do a coordinated approach to make that more like help this Linux community inside a company like ours to do that. A third measure or a third me is actually what we think the more coordinated participation in the Linux ecosystem so there needs to be ways or we need to find ways to foster that to push that and we are also looking at that externally we were looking at three different things so externally the goal I think of not only of us but of many is now how to improve this IIT support for Linux devices and this is like there are these three dimensions that always come up it's hardware support and software and tooling and the goal that would be great is actually to get in different areas an improvement that is wonderful one is like transparency for example through certification that it's becoming obvious like when you have for example a certain service provider for example in which way he is making sure that all the results all the changes that he's doing to the corresponding open source software is actually getting upstream or how he is providing for example if it is a hardware vendor how he is providing the corresponding drivers in the main line kernel or anything like that so that it's becoming easy to maintain that over time another thing of course is standardization so the entire embedded Linux area there is still a lot of room to standardize the things so it's becoming easy on introducing the complexity for the individual projects of course it would also be great if many of the things that are done redundantly currently for many reasons if we find ways to share that and make it more efficient the same is true for testing one thing that is turned out to be really important is traceability so that you know exactly what component is on which device or where this is coming from and that that you can do that and handle it on a large scale that is helping a lot for if you for example use a linux distribution or instead of like doing it very customized for each individual and the eyes the other thing is when you do that then you have components and you can share the metadata for the corresponding components and like handle that in a larger scale manner for example with licenses and vulnerabilities and the third one is the for future devices so when something new is coming up it would be great that it's easier for an out of the box solution that are like already following and doing all these things with Bosch we use something and develop something that's called a purchase that is to a certain extent fulfilling our needs and we already also published it on a website like a purchase.org but we don't expect that everyone like is like happy with that one but like anyone who is interested of course can use it on the other hand we think that is something that would be useful in general and what we heard from many other companies are doing actually something like that they have developed something like that that is helpful and doing it for new devices is like a great way to do it. A second part of the results is that we that all this what I just discussed opening up more doing more collaboration in this way what we think what is needed is actually for a company like ours and cultural change in many areas and if you want to do it this way it's has certain requirements you need strategic goals so you get the corresponding funding and the corresponding support management attention and also the corresponding time so like open source goals need to become the strategic goals to support the corresponding business goals and the developers need to have the time to learn the skills and do the corresponding things that are enabling them to do this new to do these things in this new kind of manner in this new culture and of course they also have to struggle with certain barriers that they need to overcome so it's also important that they give this that they get this time and that they get the corresponding support to overcome that because it means doing that in this different kind of way working more out there in the open in this open source communities that they have to leave through their comfort zone for something unknown so far and also it is because it's new somehow uncertain what does the corresponding actions have for consequences for them and also for the company so it is like it takes some times to do these changes so this cultural change is something that is really difficult in general like every cultural change in the big company is something that takes times and also takes energy and patience I think from what we've seen it it would be worth it to do that so how can we do that concrete in detail so what would be first steps towards that we also thought about that and establishing the participation in the open source community as a new normal it would take these concrete measures here train the open source skills encourage open source activities like incentive wise and also like yeah instead of making it hard to go into corresponding conflicts and giving talks and visiting the events making it easy and also providing the corresponding experimenters from externals or when you have them hire them and also have them internally and of course removing the various that we just talked about the third thing that we came to that we think would help is actually what we call something like an industrially great minox at least that's what we think meaning introducing something like two additional layers between these upstream software that is there and the embedded software project we already have in many cases in many projects already a service provider that is helping us to do many things like for example in the open source space and so that is already there but in many cases what we see is actually the things that we are doing are actually not really um getting to the corresponding upstream projects or it's getting somehow lost or the next project has to start in you that's the reason why we think it might be really helpful to have something like another open source community that is focusing very much on the embedded linux area for example we see something like civil infrastructure project which is already a great thing and there are also many other activities but we think there's still something missing and it would be interesting to talk about how if this is true and how if this is true and many people think like that how we can close this gap yeah so that is more or less like what we have learned and I hope this is helpful for other people and we would love to discuss that thing and we thank you a lot for giving you giving us your attention and so if you have any feedback and if you want to discuss we would love to hear so actually we also asked some of our colleagues if they could give us like a little statement or like what they think like they're sharing their experience so that it's all not only us we are just representing some people out of this context and so we will add this to this video two statements that we got we like them we hope you like it too and yeah Sven if you have anything to add you can also like bring that out now anything thank you very much as the only thing I would like to add is I'm really looking forward to making developing linux based devices even more fun and also maybe even more efficient so looking forward to the journey yeah and again I already said it but I cannot emphasize it this enough we would really love to hear from you we would love to discuss this with you if hearing your insights your understanding maybe what we said you don't share this you it would be interesting to hear what you see differently but also of course when you say think yes this is true this is also an interesting feedback so please contact us and have a nice open source summit thank you bye hi I'm Chris from Bosch Power Tools I'm software architect here and I want to share some experience that I made during the last years of linux development within Bosch so some years ago we started as an organization to shift from microcontroller based software development to linux processor based software development while this was quite new to us we tried in the beginning to create our own images from scratch and we had to build up all this build infrastructure and all this open source disclosure stuff and everything and we recognized after I think one or two years we recognized it is much more than just hacking some software together because there's so much process and everything that you need and we recognized that we need some additional help and we tried to get in contact with external partners and to contract something but this was also not maybe the right thing for us so we tried to join forces within Bosch itself and this was definitely a game changer at that time because on on one hand of course it was a cost sharing that we had but what we actually did was we were jumping on a platform of a bigger part inside Bosch who had had the same experience in the past and they were very open to help us and what was nice about their platform approach was the part that being Debian system which helped us to get very quick and rapid prototyping a part of that was that they had this open source mindset so it was not this typical industry style platform where you have lots of requirements engineering in the beginning and then you form a platform and everybody has to take it and it was the other way around it was true open source it was like there is something you can take it you can adapt it you can upstream it back you are always able to deliver you don't have to wait for anything and this was a true game changer this change in mind from this stiff platform development approach one fits all to this idea of being open for multiple different requirements or multiple different units having completely different work styles but still having one aggregate where you can upstream and this was really great so we after half a year we were able to get really into production mode and to do our products and now we have our first products in market working on this platform and this is great so the key message of this is be open for open source even if you are industry and industry and open source fits very well together but it's don't try to adapt open source to your industry style working but try to adapt your industry style working to open source style it's much much more agile you can read so much thank you hi i'm dick as a project manager i'm responsible for an ecosystem for iot development based on embedded linux and now i'm going to show you where i live i live close to a forest i love to live here and this this is my fence it is important to protect my property by a fence and additionally i have nero my german shepherd at my property nero and me we have clear rules for the family and the guests outside in the forest it is different the forest is open and you never know who you meet or what you will discover there for the all the forest is an opportunity to meet other dogs or to hunt deer and i like to meet people of my community people who love dogs and love the forest in early june i crossed this fence and met a neighbor she told me about the doctor where a family vaccination was possible on short notice my family got vaccinated after two days and we enjoyed our summer holidays with full covid protection what would have happened or not happened if i was not in touch with my neighbor behind the fence okay behind the fence it might be uncertain have you ever heard about wild boars wild pigs living in the forest however i see a lot of chances more chances than risks and this is the reason why we are now going to cross the fence nero come cross the fence