 Hello and welcome everybody. This is our talk. Plan your learnings. Can friendly user testing after SOP become a common practice. These are considerations by my colleague Nishi Gutsich-San and by myself Philipp Armann. We are both engineers at ARDIT. Our form is Advanced Driver Information Technology. It's a joint venture at Robert Bosch GmbH in Germany and Densu-San to the cooperation in Japan. We are very happy that we got the chance to talk here at the Automotive Linux Summit. It's our first trial to make a pre-recorded session and I hope you will enjoy what we have to say today. Let me also not forget to say thanks to all the sponsors of this Automotive Linux Summit, especially also to the Linux Foundation and to the Automotive Great Linux Project. At the beginning of the presentation, let me first of all give you an idea on how this whole talk is structured. I split this talk in three major groups. First one is called BASE. The basic ideas of IVI systems is presented, how software for IVI system looks like, then how safety ECUs are still treated today and how both of them are influenced by updates getting into the field, into the common sense. The second part is the change area. It tells you a little bit on how nowadays cars get connected, how influence into the automotive market can get over from other industries, like the web service industry or also mobile industry, and this will have impact on how our products look into the future. So for this, the consequent third part will start with giving you an idea on triggered learnings, validated learnings, a first use case on how to provide log information and then a very detailed view on the ELISA project use case of telltale handling in a Linux-based cluster. At the end, we will give a summary and have hopefully some room for discussions. But first, let us have a few words on who we are. As said in the beginning, my name is Philipp Armann. I'm currently manager at the ARTIC GmbH, the German entity of the ARTIC Corporation, and as said, it's joint venture company of the Bosch GmbH and Anzo Corporation in Japan. In the last five years, my main work was in the field of project leader before I had three years as a test team leader. Also, I spent four years as hardware and software engineer. This provides me with an overall experience for more than 12 years in the field of automotive. And, yeah, taking my time before I joined ARTIC, also a lot of Linux expertise from university and so on, which gives me over 15 years in this area. Since last year and very beginning, I'm also active member of the ELISA project, which also influence a little bit of the talk here today, and I'm former member of Autosign and Linaro. Additionally, I'm an open source enthusiast. I try to use open source wherever possible. And, yeah, due to my background and the many years was added, I really loved Japan. And as I'm always interested in new technologies and methodologies, I get to know Kaizen and they believe this also influence the talk here today a bit. Now, I want to hand over to my colleague Nishiguchi-san, so that he also gets a chance to introduce himself. Thank you, Philip. Hello. Let me introduce myself. I'm Na-Filo Nishiguchi belonging to the AD-80 Corporation at Japan. And currently, I'm working on Linux project leader. So far, I have developed IVI system, especially audio part from low-layer, like a sound driver to upper layer like UI application. Through such a development, I spent 12 years as a developer, two years as an architect. And some part of myself is still architect in AD-80 Japan site. I have been in the automotive industry for about 14 years, developing on Linux for about 8 years. And recently, I'm working for the functional processor adaptation in AD-80. Now, I'm an AGL, automotive-grade Linux and ERISA member. Okay, then I will hand over to Philip again. Okay, so then let's start our journey. So the base of our journey would be to get the knowledge on how IVI systems typically still look like, also how recent ECUs in the field of safety get in and also how updates after SOP influence these products a lot. As I bring a lot of experience from the field of in-vehicle infotainment, so IVI products, I always had the impression that we treat these products really in the same way as we treat engine or brake systems. So we have a very heavy process. We spent all effort with highest quality and all possible measures to reach the point where we have no reset, no freeze, full stability and we got rid of any kind of issues. Of course, we are way towards perfection as needed because the car should work properly until the end of the life. And this can be 10 or more years, much more than a smartphone would give us or television software. And also here we are already used to see resets. An IVI product should not reset and nevertheless it could happen because also we target perfection, we never be perfect completely. To give you an example, I observed the first reset where I see that the IV system turned off and on again only 6 years after operation. And funny enough, a week after I experienced it again and after this no longer. And it was totally clear that this reset was there but I never reported. So never an engineer will get the feeling that there was something wrong, something could be missed and something which could be improved for the future. So I think it's valid to ask the question how many more perfect IVI systems will be out there on the street having a reset and no engineer will ever get to know it. And even if we just think about perfection in an IVI system, we have even stronger demands of course in functional safe operation for safety ECUs. Here the safety and integrity standards already tells us that the system is functionally safe, maybe not intrinsically safe and even standards will be argued that we do not expect failures as we took all possible failure measures with safe state and other options. The strict process with a lot of evidences also assumed that there is no failure can happen. But can we really say that these assumptions are still correct? The world is getting more complex, so the past there were split ECUs and nowadays these ECUs get centralized. Bayer Metal software stack with a thousand lines of code suddenly gets into Autosar from Autosar to Autosar adaptive and may end up in a full Linux stack. We use hypervisors, OS cells, containers, much more than the traditional artists and this is really a change to what happened in the last years. Nevertheless a safety standard or safety integrity standard still treat these products in the same way. I also wanted to give an example outside of the IVI or outside the automotive world which shows how small design floors can make a fundamental change. So Boeing wanted to be competitive with Airbus when bringing in the 737 Mac series. They followed their processors, they made considerations, but there was a little peace which didn't behave expected and really caused a lot of incidents. Even if a standard is followed, even if complexity is treated and even if perfection was also in mind. So this is something also where updates were needed and just these days after a long time Boeing was able to get the information and to get an update and let the Boeing 737 Macs fly again. Another part, even if you had functional safety, you still may have connected ECUs, you may have still access. And here I wanted to give an example on newspaper article which says that hackers found ways to exploit automotive software and were able to overtake cars. So this means security updates are necessary because security exploits may force OTA updates of cars, especially for connected ECUs as I just said. There are much more ideas where you can have field updates. So one part where I read their long way and it was a nice article for Hyundai how they enabled update over the air to bring in new functionality for able to deliver security updates, also demand for it. But also if you are not doing and you have a connected car, you can see this from TechCrunch. There have been bugs in there that hackers remotely control and Mercedes-Benz car. A more positive example is what a hacker could find in the software. So this hacker was not actually hacking the full car for doing harm, but he was looking into what he could get as information out of the car. And Tesla was providing hints that they introduced 5G hotspot software update parts in their latest software. So this gives you an idea and a summary on the update after SAP section and how this influence our modern connected devices, centralized ECUs. Of course the one top item are the security vulnerabilities. These should be there in no time and much faster and it may not be sufficient to go into workshop anymore to get a regular software update. Operating system updates are another field. So there could be someone forcing you to do an update. There could be something that you just want to address new functionality. You want to have for example some bug fixes in which were not crucial issues, but you figure out that the user experience is much more relived and improved. And of course one traditional part is always a map update. I guess this is something which 10 years back you may have done with CD or DVD, then later on maybe by USB storage or SD card storage. And recently you most likely connect the ECU and may be able to update over the air. And this is an update of your device after SAP. And even if you have checked carefully, you don't know if new issues come up. Last part of course would be the most brilliant one when you really want to deliver new features as we could just slide back on what Tesla did where they introduced a new feature on 5G hotspot by a software which is not usable yet, but still already deployed. So this also brings us directly to the second part where we say what's the change which comes in. Where nowadays cars get connected and where we should also take a glimpse to other industries, other industries which we focus on our web services and mobile industries. So let's get a look into this. I saw what's very nice to look at is what we already find in the car. Emergency location services or e-call is something most of you will most likely know in case you have a crash or so. There is a call and this call will go to service center and they will ask you everything is fine. And due to the location services they know where you are. So even if you're not talk they can inform the ambulance or other parts. This works already with the mobile connection so there need to be a modem in there which can call. This directly goes along with data plans and if you already have a data plan maybe you think about apps and connected apps is nothing new. I put here a link to if you scan the QR code what it worked on in 2014. So six years back was our first connected product and from connected apps. So next step to also provide the chance to install apps so you go for the app stores which you can also find in the first cars. Then if you anyway are in the connection fields where you have bluetooth connection, you have Wi-Fi connection, you have a lot of sensors in the car. It's just natural also to connect to other cars and to the infrastructure. So you have car to car, car to x communication. You may provide information on traffic signs detected, communicate with other cars, one euro traffic jams or other. And hard thing this comes with cloud computing and processing where you really upload data and receive data also during your way in the car. And then as you have the mobile services as we could see also again with the Tesla power there is the Wi-Fi and access points. These can also just be used to share media across the car from a device to the IVI system, from the IVI system to other parts and devices. So this is also what comes in. And looking into this feature set, it sounds a lot like smartphones and they also started small. So I took the first smartphone kind of thing where not everybody was talking about smartphone with a Nokia communicator. This was my first experience for something more than a feature phone back in 1996. I was really impressed to see this what I called more a computer at this time in a pocket size format. And then it took a long away until we really became to something which is called smartphone. And well the first iPhone where Apple calls itself that Apple reinvent the phone in 2008 was most and best iPod ever seen. They really made it true that they said this is only the beginning. And what a phone involved over the time, you can see when you just think that you may have taken up your smartphone and just check the QR code which is up there. Here. I didn't want to spend much more on the parts how cars get connected. I once more want to stress we cannot deny field updates of the car. We have cars already connected and much heavier data exchange will come over the air. And I said we should take a look and glimpse to other industries. They have a much better way already today in dealing with updates and also how they plan their learnings. I took a very minimal slide, but I found it impressive enough to explain a little bit about the web services and how all things get in there, especially because you typically don't even see one feature which I really love is this split or a B testing. So basic of concept of this is to make a change. For example, to web page and some users will see what's rendered web page and the other users will see a slightly different page. So I put a link here also so that you can read a little more afterwards. Basic idea here was say comparing click rates. So here the green icon was giving much more clicks with 72% compared to just the blue learn more with 52%. This is also related to microchanges, faster iterations and before making a full user interface with all the considerations and so on. So before you bring your product to perfection, you start your learnings. So you come with an idea on the feature. You bring this micro feature in and you already try it with a friendly user testing or a very limited number of customers. Then also to get hidden features in. It's not really hidden features. It's some software residing on a server, but it can also be operated. It can operate in the background so you don't see it. And just at a certain point of time, if you gained a lot of confidence that it's not causing any other side effect on the system, you put it live, you make it go live. And here I guess that's a little bit also how Hacker found the Tesla 5G thing. It's a web service, but most likely Tesla has learned from this. Next slide I get into the mobile industry. And you also spend only one major slide. So what do we see there? Mobile industry is much more familiar to most of us. We know about beta testing. You can, for example, sign up and make your Pixel phone a beta phone. You see that stability is already quite good at an early point of time. And you can also use beta tests to be on the tip when it comes to app development by migrating an Android version. So this is all the learning and the ideas to just speed up development. You get feedback from users in an early stage. You help them to develop new things and you can make your product better. Then the simpler way without active positive patient or signing up by users is just feedback requests. This can be, for example, just the ratings. I put one screenshot here on how the call quality is. You may see this on a lot of your services, which you also operate where you work with tools. And it's very similar to also rating system where you say, yeah, this app is good. You look into crash reports, which is coming in and you can also provide surveys. And for us, I guess this part is getting normal. And here's thinking on how normal this is on your car. So I never heard about someone who was offering beta testing to users within the car. We just said, well, you know, you want to get the beta of this or how was the call quality. I want to update this one in the car also selling that you can rate this product because they come with your car. It will be more indirect reviews or something like this. One use part, which I also wanted to bring is from nine to five Google. This I found quite interesting to read was that also developers should disclose crash reports and that they also inform the users so that if you're not doing a proper way of crash reports and bringing this in and also providing spec to Google to a certain extent, you may be even removed from Play Store. So you can see that here crash reports is a fundamental part of the work. That may also mean that androids gets very strong. And talking about Android, Android is also entering the automotive market. So mobile industry took its chance, took its knowledge from all the smartphones and try to get into new market with automotive. So these new players enter the car and I took a screenshot here or a picture from the Polestar. Polestar 2, maybe I'm not sure. This includes Android at the first cooperation by Google where they also bring a full Android stack in there. This hopefully doesn't sound surprising. I think a lot more Android products are ramping up here. And here I'm not sure if Google planned their learnings, but they got a lot of experience upfront. So taking a look way back five to six years, we see that Android entered the car already with Android Auto. And I think this is something which we all integrated with our IVI products. So CarPlay, Android Auto are used in a lot of products. It already provides an entry device or smaller device also made higher premium devices with Google look and feel, usage of apps and it already enables people to see Android devices on their IVI system. Even if there is an underlying other rest and it make them used to expect also an Android in their car. So if you are these days surprised how strong Android grows, you may have considered this also four or five years back when the first Android Auto was getting into market. I made a statement here. I said which learnings have been planned since this announcement. So have you prepared to see this change of Android? What Android means to your products? How you can compete? How your software will compete? Also what infrastructure Android brings, what demands will come? And if you haven't done, maybe now would be a good time to also say what are your assumptions on how the story will go on and how to prove these assumptions. Now we successfully covered the base part, the changes which we see to the base of products and we can enter the consequence which is the details on the learnings which we can trigger and how we should share the learnings which we gain to remain competitive. And this also includes the part where we got the main idea from and this was actually not from the traditional software product but from a completely different market. So plant learning or also validated learning is a strong element in the book The Lean Startup by Eric Rees. It's very interesting because it gives you an idea how to get faster products and that whenever you take business, so not software but business that's why I said it's different area where the idea comes from is that you shouldn't wait until you learn something by the hard way at the end. It's basically often taken as a good story into something where you want to hide your failures. So from failure in this case means from what you originally considered as a good product where you never saw that it may fail and typically you just do a post mortem analysis so you wait until something happened. You were surprised and they say yeah but at least we learned something out of it and here you should trigger and see that you made progress, that you validate whatever your learnings you plan and I think this becomes a little bit more obvious when we go to further use cases. But before we can look into use case we should first of all see which levels of information and potential friendly user testing exist. The most obvious one with the lowest impact to a product with not even being connected can be the inspection during the workshop. So if you don't want to spend a modem into your car you can still read error memory information that you can just put some base logs in there and this is just common. So here you could collect data and inform that this should be forwarded from the workshop to the OE down the road to the BSP developer. If you don't want to wait for the workshop you could give it a try with the QR code. You've seen two or three already in this presentation and I guess you're so much used to QR code scanning so this could be something which you can also place into an IVI system and just say well I found this bar, here's the code, scan it and get in log information. Then you could also consider direct communication if you already have a modem in their data plan so you could use some bytes of this data plan to have even seamless upload of logs without user interaction. If things are hidden this may not always be the thing which people like. So for this it may be worse to ask also for approval so that people become active participants in this and are really friendly users testing because they signed to be there part in this area is also to respect data privacy statements and data privacy of some certain countries. I know Germany is very strict on data privacy, a lot of rules apply here so be careful and watch information you upload and what profiles you create if you don't ask users for their acceptance on this. And one important part is also this all implies there's something where you need to learn and this says your product is not perfect as we may have seen it in the first slides. So we need to get a culture to say we still target to make a product as good before but we should accept an imperfect world and here some way to accept imperfect world would be to announce the try features for free if you don't pay for something your acceptance on smaller issues may be much higher than if you paid a lot of money for it. Another way how to motivate people could also be that you ask for surveys and to have a motivation in there is that you can win a voucher that could be something small like a cap or just some stickers or you get a car cleaning voucher or something like this. And last element here is also to see if we saw the Android Auto getting into the car if we see how people are used to certain elements maybe IVI system will on the long run anyway be much more similar to smartphones than to brake or engine control. All this said I guess it's time to rephrase our initial fire and forget policy and perfection at the point of release. I guess before SOP IVI products will no longer be treated as engine and brake systems so we can rephrase it maybe to IVI products can be more seen as a spice level phone and I guess we still will dedicate our efforts to perfection until SOP but now as we know updates will come in we will not stop there but as we have the connected cars we also want to overcome the last parts which we say in the no idea on when the reset happens so we should change the last statement in here and say we plan our learnings to figure out the cause of the issues for example resets and are able to receive logs. However in order to be able to receive logs we will need a fundamental change in the way we collaborate and the way we share information and this is the part not about trigger learnings but we need to share them so we should make data available along the line meaning if we have an OE information should not remain there issue logs trace logs and information which can be there should go for service provider centrally stored and then be forwarded for issue analysis and user preference analysis and so on so all these potential learnings you can make a B testing and others to the respective supplier or whoever will need it I want to show it to you in the field of a Linux product also in the next slide so if we take this card from before there will be four main elements I come more from the lowest layer I embedded guy originally so I see a BSP is very important and I know we are trying to build platforms we try to use kernel wherever we can to be the same for multiple products for multiple releases and to maybe migrate to an aversion but take our learnings from one generation to the other similar we would do with middleware updates would say basically the octo layer and we take also here update after update and it's more reuse and a completely new development from one generation to the other so this means potential issues which we find in one generation may be a learning for the next one also technologies like USB and smartphone elements will not change too much of course the smartphone itself yes but the interface is how they used not so this may also help you for one generation to the other here this is typically maybe with already with two teams so the embedded guy, the base platform team not being directly the same one as the ones working really on the BSP the BSP may also come from silicon vendor and then we are at the tier one project team which will take a base platform and develop the first applications on top so here is a third party involved and it could even be different suppliers different tier ones and so on the full product will end up at the OE with a lot of customizations maybe some of them will also be the product team but at least the final product will get out on the OE side with a full customization and the OE is also the one who will receive the log information if any even the debug message which we build and also so it's good that we have them we could figure out things before as a people we need to get away how the logs will end up here with the embedded guys because they also take a lot of work and stability at the learnings so this said I want to conclude my part you got a good view on how products have to change what demands we put to car connection that maybe we will reach a point of time where information is available in a collaborative style from OE down to a SOC vendor and how important this could be will be explained a little more in the last part of the presentation which Nishikuchi-san will take over here and explain you for the Lisa use case about the next let me explain example of a Linux based safety product the Eliza community and the Asia community are currently collaborating on the realization of the instrument cluster on pure Linux this collaboration is discussed in the Eliza automotive working group in this session I will use the materials shown on this page which were used at the September 2020 Eliza workshop both materials can be found at the links so if you are interested could you please refer to them from this link at first I would briefly explain the use case in instrument cluster a telltale is a lamp typically found on the instrument panel of the vehicle as shown in the picture in right hand side such lamp warns of a sheet of unknown use doors open, lack of engine oil and gasoline and so on and Eliza telltale has a following characteristics it's not extremely safety critical typically telltale is A through B in ISO 26262 terms timing requirements are not as severe as with engine control software it's interesting content for tier 1 and OEMs and an opportunity to cooperate between Eliza and AGL and AGL has a reference hardware and telltale demonstration software these can be used at the starting point of discussion and activity in Eliza Hansen, I'd like to talk about the original architecture design of the AGL telltale here is an expert of the AGL architecture design functional safety and non-functional safety are isolated by the hardware the blue part on the left is achieved on the non-safety side using rhinance the green part on the right side is on the safety side using the RDOS on the rhinox side the telltale is drawn based on the status or information of the other ECU received via the CAN communication and this is monitored on the safety side if the safety monitor detects an abnormality in the telltale graphic part in the screen it is processed to shift to a safety state this safety state is depend on the OEM specification for the fundamental understanding the essence of the telltale system here is a diagram with only what is necessary for essence again, the design concept of the system is to check and monitor the drawing on the unsafe rhinance side for any anomalies by the safety side in summary, the setups of the AGLs are here rhinance will do a lot of rendering in the typical workloads this typical work include the general IVI system features such as the navigation guidance, music playback and carplay or android art as well as the meter drawing rhinance is treated as unsafe but general quality assurance and the product development is carried out currently, there is no probing use or safety argumentation for rhinance at least in the automotive world therefore, the system requires the safety monitor at the outside of the rhinance to accomplish AGL qualification the safety monitor is based on the AGL-qualified RTOS running in the functional safety context and will check that the rhinance result are correct or consistent finally, in case the safety monitor identifies improper behavior safety monitor or safety side software will take over and continue rendering or it will bring the system into a safe state now, I'd like to look back on the interruption once we talked about how IVI and other in vehicle systems are currently being treated to be perfect as described here as mentioned before, perfection means that nothing happened to any issue if this can be realized, a safety monitor is not required but there are other uses for the safety monitor our learning activities as preparation for the next generation product include the following things as per our assumption and as per our ARGT quality level in case that the rhinance implementation works correctly safety monitor will never be triggered for a long renderer image this information needs to be captured and processed as a learning information information on when the safety monitor has been triggered or how long it has not been triggered can be collected by OEMs and suppliers via the cloud it can indicate or demonstrate how stable the Linux system is this is a learning that can be applied to the next product development and that this is a seed for the improvement the concept is very simple but unfortunately the real world is much more complex so the following should be considered in case that the safety monitor detect failure or abnormal maybe Rhinoxide cannot be credible therefore, RTOS site could store error logs in non-polar type memory in RTOS site after a system moved to safe state RTOS leads logs and send it to Rhinoxide sometimes Rhinoxide was restarted in order to move safe state at that time Rhinoxide can credible because of safe state then Rhinoxide can store rows from RTOS site and send to cloud but there are some problems, risks or questions first point is about how do we send the data from RTOS to Linux while keeping RTOS is safe depending on the mechanism we may not be able to keep the safety to increase the robustness of RTOS site it is a communication better in one direction this means that RTOS to Linux only but in this case it may not be possible to receive rows correctly it depends on the state of the Linux finally, I will explain the summary key messages and takeaways from this session are shown this slide at first, cards are becoming connected it's not just for the end user and customer feature but also for issue analysis and new market opportunities by analyzing preferences and making them more safer, more higher quality and more secure use validated learning to avoid surprises from the field automotive industry learns from larger industries such as web services and mobile phones apply the practices even in non-standard areas it may even enter functional safety use cases this is the last message from our end what will be the next learning you want to plan up front if you have any ideas could you please write it down into chat that's all for our presentation thank you very much