 Hi, hope everybody can hear me fine. Thanks for joining this session about the web applications and user interface on the AGL demo platform. My name is Lorenzo Tilbe. I'm very happy that I'm able to be here again on site after a couple of years only being virtual this event. During the presentation today, I'm going to be just briefly introducing me and my company and the work we have been doing in collaboration with automotive-grade Linux. And I will be explaining which were like the main motivations for having the AGL web runtime. I will be explaining which is like a technical solution that is being used for providing this functionality based on Chromium and the web application manager or one. And I will be going a bit more in detail about how to create and debug web applications running on top of AGL demo platform. And also following up on what is according to status, ongoing work and future plans. So very briefly about me, I am Lorenzo Tilbe. I work for Igalia. Igalia is an open source consultancy which was founded 21 years ago in Galicia in the northwest of Spain. We are currently have over 130 employees and we are in 28 countries. We are very distributed all over the world. And our main area of experience is working on web rendering and browsers. So we have a lot of very specific technical knowledge on Chromium and WebKit and also WPE and Firefox which are like the main open source web engines in the world. We are like the second biggest contributor to Chromium and WebKit after Google and Apple. So we are like the biggest independent consultancy working on them for several years already. And we have also teams that complement that work related to work, to compilers, graphics, multimedia and stuff. This is a picture from our headquarters in Spain. So due to that experience working on web engines, we were the ones that we could be helping AGL to put in place a web runtime for automotive demo platform. So the motivations for doing that is that, OK, besides the initial UI that was available for AGL that was based on Qt, it was important to be able to provide full web platform support into AGL. So what this means that besides having given the capacity to build solutions on top of the rest of the open source stack that AGL demo platform provides, it was important to be able to have like, the goal was to have an HTML5 only version of the image that you can just rely on web technologies exclusively on standard features. So basically something that was important is that it should be framework agnostic. So different integrators here one might have their own preferred like UI framework. They may be using whatever web solution to develop their own applications. So it was important that it was not connected to any specific front end. So it should provide out-of-the-box compatibility with all standard web APIs. When working on the web, like you get, like out of the box, all the functionalities that are available on web engines, meaning from multimedia support, from rendering support, I mean everything you can have access in the web engine, you are going to have it like directly integrated and portability of course. People already have their own applications. So it's important for AGL to make them available directly in the web. Something important was also that the experience was similar to native like. This means that we were focusing these on performance. So the experience interacting with the web version only base of the AGL demo platform should be smooth. I mean normally when working on web pure cases, like you need to speed things up. So the friction when switching applications is small. So this was also some of the goals to provide integration with other vehicle to cloud services. I mean, directly by having that using a web UI, you have that already integrated. And I was mentioning having an AGL web runtime gives already a huge access to a big community of developers. So creating an application and putting into AGL should be very easy. So they can, I mean it's easier to test applications and run them into AGL. Interoperability using any web API that can be used. So the solution that was defined for this approach was using an approach based on top of Chromium and WAM. WAM is the acronym for the web application manager. That is a piece of software that was initially developed by LG for web OS. So it's used already on millions of products worldwide for isolating applications that run under web OS. Then it was open source as part of the web OS open source edition. So we took that piece of software that is built on top of Chromium and we adapted that work into AGL as the scenario was good benefit of using that open source project. I mean the goal of having a web runtime is that you can run separate applications into AGL demo platform. They are using the, in this case, the Chromium unboxing model, but they also have additional levels of security because applications should be able to talk to some specific APIs that other shouldn't. They have integration of lifecycle specifically in the case of AGL. So there are many things that were determining that the solution could be like beneficial for this ecosystem. Something else we were putting on top of Chromium and WAM for AGL was we were using the AppStream OS on Wayland back in implementation that we did for AppStream Chromium. This was a work sponsored by Vanessa's. So we were able to have Chromium AppStream using Wayland instead of the former X11 implementation. We just made it possible to use the GPU capabilities of more modern hardware, embedded hardware platforms. So we were able to speed up the state of the art of GPU acceleration. It was also a solution tested in many scenarios and connectivity was provided by this solution. So as I was mentioning, specifically one is allowing to optimize memory, optimize application launching time, separate the permissions, isolate better applications. So they, I mean, when running on top of a web runtime, you should be able to give the browser all the permissions to have access to all the JavaScript APIs that are exposed. This could be a security problem. So this is something that WAM fixes and also provides lifecycle control of applications and integration. I will go briefly to explain how to build the AGL demo platform image. So all the documentation about the hardware that can be used in the context of AGL is available on the documentation for needle fees that is the current stable version. Besides, people can still use master or which will become the next octopus version. So the documentation is there. Different hardware target architectures can be used so people can use, can build the platform entirely from the scratch using Yocto for renaissance boards for emulation with QMU or for Raspberry Pi. So there are like additional documentation there on how to use those, those different build targets. And the way to use that is like, OK, just using Depot tools and Yocto is as simple as checking out the Garrett repository from automotive Linux. Download that, configure with the flags that are relevant for building specifically the IBI demo platform HTML5, which is the web only one that pops up the web layout and basically build it with big bags. There are also images available. I mean, everything you can follow the list and all the documentation in the automotive Linux website about more details about building for other platforms or for other targets. Well, the code where one and Chromium is, or the recipes to build them are using the meta AGL demo Yocto layer. For those not familiar with building in Yocto is basically like a set of recipes for compilation rules to include in the building of the entire system, the web application manager and Chromium. And they are linking to code that is also, everything is public is upstream so one and Chromium are also in GitHub available from the build recipes in Garrett. And then the images as mentioned can be directly flashed using those commands. By the way, all the PDF with this documentation is already uploaded to my presentation session so you can download it and check the details and copy like the commands from there if needed. So you can generate the image and flash it into an SD card and then boot it into the device directly. What you would be getting when starting the AGL demo image, you are getting this UI which has like separate areas. And I can be explaining you a bit more on how the HTML interface is composed. So basically this is showing two web applications. One application that is visible is like the home screen, which is the one that is filling all the UI, I mean all the screen available on the device that is being run, which is showing some shortcuts, some widgets that show like the status of connectivity, connecting to the service, to the time and whether etc. So this is a web application, the home screen area and also in other way and surface that is handled by the AGL compositor that is also the graphic element that is deployed on top of. There's another web application that is the applications area, this is like a launcher which is a separate web app. So we are seeing there everything you see in the web UI is like all of things web and there are like several web applications running and visible at the same time. So when clicking on the shortcuts or clicking on some of the list of applications you will be like popping up those applications in the screen area. And when switching back in the home screen application you are restoring and bringing them back and out. We have like a demo setup in the exhibit or showcase if you are more interested you can come by and we can be happy to explain it and you can be playing with the demos there. So now going a bit more in detail on what web application for AGL is. So a web app can be something as simple as basically three files or two files. So it could be like an app infogation document which is like the one that defines the metadata the application uses. So it should be like each web app should have like the basic main information like the title, the application type, the icon etc. So we can have like the reference to the license etc. And can even wrap an external URL as we are using a web runtime. Actually what you pop up when running a web application is a URL that brings up a Chromium application inside there. A web view that shows like the content for your service. So it should work as you are running inside a Chromium web view, your application configurator. So as mentioned you can like use any web technology you want to run. So a web app can be like just a plain HTML and JavaScript. It can be a Wasm file, it can be a WebGL element, it can be a Node application that has supported. You can like make a NPM bundle and you can run it from a web app inside AGL with any front-end library. And this also applies to the home screen. Like the home screen that I was showing was like again an application that could be any integrator or tier one can use their own version of the UI just changing the styles, creating a new application with a new composition. And you could be getting like your own HTML view inside AGL demo platform. So in order to make them in the upstream repositories of automatic Linux or even for working them locally, you just create that main HTML file with all the resources. So when you are bundling an application, like the code of the application that you run locally, you add the application configuration file and then you can create like the Jotter recipes to add it to the AGL tree. So you can see that there are like the examples of the already existing web applications inside the demo platform, which are like the system ones. I will be talking a bit about those but applications like the mixer, the HVAC control, like multimedia applications, like settings. All those apps are already upstream in the garbage repository of AGL, a dashboard. Some of them are like placeholders or are like concepts of applications that are not connected to all the services are kind of, some of them are mocks. But you can see some of them are made with nodes. But you can see examples of how to create applications and integrate them into AGL. Also in order to explain like the easiest web app that one could integrate into AGL, I'm using this simple example that is creating a Jamendo application. Jamendo is a free music streaming service that just has one URL. We can just use that as a kind of a PWA application. So it just consumes something from a service and just brings it and puts it into AGL. So the example of this would be as easy as defining in the application JSON file. So which is like the idea of the application, like the title that it would be, like the description, which is like the entry point for this. This is in this case an index HTML file, which would be like very simple because it would just, will open just a remote location file. So if your service is already available remotely, you can use an index HTML file that just connect to an external service. If it's not talking to services inside the car. I will be explaining cases that are more complex that talk to the services inside the vehicle in the next example. And an icon and that's it. So in this case, as I was mentioning, the index is as simple as opening an external service. I mean for testing something very basic, this is like the boilerplate for creating applications. If your web app is local, which is like it's going to be like the majority of the cases. Here this entry point will have like your game or whatever bundled as like all the resources that are used are going to be packed together with this, not just this index file. To create the Yocto recipe for this, you just add it to the recipes demo. And in this case you just link to the ID of the application. You list the license that is going to be used. Those information are all available in the Yocto guides about the licensing that is going to be used and you can see the rest of them. This would just configure the URL where the new application is. And as mentioned, this is, once it's available, but for testing in local repositories you can use some local file. So this is going to be when building the entire system, it's going to pack that extension and put it inside the demo image. So to make that easier, it's also possible to use a local dev file. I don't know if you are familiar with building in Yocto, but it's also possible to skip using the source and using that local compilation. So in this case, for this simple example, you just put that in your AGL root build config file and you do the modifications locally. If you are developing specifically changes inside the AGL version of the web app, I insist. For everything else, you can work on your usual web development environment. You are just working on your application. When the changes are available, you just can package and also zip it. This is like for getting the application installed for the very first time. Then it's just when you have your virtual image running on QMU, you just can update with the changes to sync them with any version of the file. So then it's just compiled and this is going to be building already the application that you included. So in this case, there are some additional flags that are using the Yocto RCP. You will see all of them are the same and the build infrastructure is the same for all the services in the web applications. You just add the application into the list of existing apps that are going to be available in the AGL demo platform. And then when creating the whole AGL from the scratch, it will be including this sample of application. As mentioned, probably the easiest way to set up and to start testing AGL demo apps is using the emulator. So then you just run a QMU64 for instance and you just directly launch it and can connect to that using a VNC client. Then it would be just looking like that. The steps might look not trivial for the first time but this is like in order just to have the application available from the scratch. When someone builds an HTML image to get directly it integrated into the demo platform. So then it gets activated by the launcher so the list of applications that it was listed will be already showing that if we do that. And then you just click on the icon and you get it launched. Also something possible when working on web apps in AGL is using remote dev tools. So when using with the AGL devil flag, not in production mode, it's possible to connect to the port 9998 to inspect with Chromium developer tools. The code application you can detect any issue, you can trigger JavaScript events, you can inspect the status, etc. So everything can be done directly. So it might be necessary in some case so just sharing here the tip to enable port forwarding if needed to connect to the service with exporting QB option. Again and this way you could be like running AGL in your emulated QEMU or in a physical device. And you can use your local Chromium browser to connect to the IP of the virtual machine or where the Rinesas board or Raspberry is working. Debug the application there with dev tools. So this was like I mean the majority of the some of the applications that people might be willing to incorporate to the demo platform. Might not need to talk to specific services inside the car. So this case of the application was something very simple that use opens a remote service and provides delivers multimedia content. You get that done because you are using the Chromium web runtime so it's like opening up a web service of that and you are getting all the functionality by that. You get that with games, you get that with I mean a lot of applications that don't need to talk to the sensors or the hardware that is specifically in the vehicle. But this is not the case for all the cases. So some applications I will be explaining here the case of for instance the HVAC application that is the air condition application in the car. Okay they need of course to know the status of okay what's the fun speed in my system what's like the temperature I want to set to the left passenger. So this requires talking to services that are in the car. The way that we have been doing this is using with WebSockets. So web applications can use JavaScript APIs and use WebSockets to talk to the can service to the Bluetooth to the settings to everything. There's like documentation on the API of the services available on the car. This was using a component that was called the application framework that is being replaced by other more standard ways to do that in open source projects as this application framework was not being maintained. So the idea is also to make possible to communicate these services communication with other front ends as could be also the case of Flutter. So the idea is to be able to use gRPC for inter protocol communication. It's still possible from the web apps to talk using WebSockets. But for implementing the communication with the layer with the lower layer sorry. We are going to use we are using already cooks. So this is a way to wrap the inter process communication. Working within being SQL vehicles and is modeled by a by VSS data model so it's a way to communicate services. So again we can still have services in the car to talk to those APIs. Which are services I mean the easy the most clear example is this HVAC application. I will be using some cases that explain how this communication is happening there. So in order to do that we define a new WebSocket that is able to communicate with that. And then it's necessary to authorize that that application is going to be able to talk to the to the cooks up dot bal server. So again all all of this code is you can check how the HVAC application works and you can see that you can use it as a boilerplate. For anyone willing to integrate services that are talking to to APIs inside the car can use that as a reference. So what it does is this it opens a socket and the socket is authorized. Which provides that a token is brought to the client JSON WebToken. So each application has their own tokens for security. So as mentioned the different web apps have the different tokens so that allow us to have isolation between permissions of the different web applications. Specifically for development there's it's possible to use directly cooks up dot bal keys so they are already available so this can be simplified. And there's documentation there to how to use that but basically then a request is sent the communication is done and then there's a WebSocket. To make the communication so from our HVAC web application that is OK you you just have to subscribe to signals that will be notified when the value of some system in the car changes. So when I mean every service in the car will be triggering a signal that is going to be exported using these these APIs. So if the I mean when doors are open with the fuel system everything is going to be triggering messages in this case. The fan speed that can be modified with any UI or the physical buttons that trigger that AC is turned on or off will be sending messages to a certain method names. So by the application being subscribed it's going to be receiving the updates so they can be notified and then and trigger back the response. The same to send message I mean in some cases we want to do the information going down so it's like OK I want to switch or to put to a hundred percent the fan speed in my car. So you just send a message that get gets notified. Into the into the into the path to retrieve and set that as I was mentioning is the same so again you just have getters and set their methods with the path of the object to be used and the value to be used. And this then provides sorry that it's a bit dark but you can test this with the age back application. So when you are clicking on the buttons of the heating system or on the fan speed on setting configurations you get like the message in the canvas being notified and getting the like the listener actions received on the age back test again. Feel free to come by the booth and we can be explained more about this in detail. So there's already ongoing work on this regard so basically we are like working some of these communication API to the rest of the demos. So the demo web apps I mean some of them are still ongoing because the interfaces are being defined. And the idea is to provide a protobuf interface so it's easier to make connection with all with all of them. Again this is in order to generate the code you can see that in the in the example of the web apps that are available in the system. About anything the request I mean this again just has to be made once for application that need to talk to services in the car and then is just standard JavaScript work is like OK. If I want to use a method available I want to get access to the multimedia volume status or for do some create some some notification wherever you just do once you did that you just use every service in the car. So yeah there are additional documents about how to work with these documents on the state of of your PC in the browser. But again like doing a small recap to create a simple web app you just bundle or you just have an application file that you put once in the building of the application and that connects that to all the tooling that makes application available in in a year. And after that you just scan copy your applications in that directory to get them tested. And for more complex things that need to talk to the car with you scoops or as a rapper for your PC. So the very simple examples for this application are here for this application and the cooks and the H back once so you can take a look and come back with more complex questions if you have them. So here I also linked in the video a demo of the that you can see here is your physically at Yokohama this time or in this video that is showing how this runs this is showing also. Like 3D acceleration working on the renaissance H3 board this demo of the H back application and again coming back so part of the goal for all these infrastructure is like again any web technology can be run into a year right now. So if you have a web app you can just build it and it's going to be running in a year. This also extends to more use cases as for instance there's already native I mean there's already web support for flutter web bundles there's there's some going work to make to provide flutter capacity as another UI as another possible UI for a year. Which might not be the case for all use cases some people might be having the choice to use like different UI technologies. So yeah for flutter apps this is possible to to bundle them and they are running as you see here this is like the gallery app that is running as a web app as the same example that I did for the for the. You just bundle that you put the index file you put the manifest and then you can run it here I am linking here to a specific talk we deliver about how to do this if you are interested in knowing more about this case. So again like recapping about how which is like the current status on our work into into into this area of working on the web runtime with one. We still are working on updating the rest of the dem application to use the cook sabal server services apis the good thing also is that. All the methods are going to be compatible with all the different UI so right now the cute demo that is using them the HTML and any flutter application will be using them. But yeah for for cases of people just using. Only standard HTML technologies they can use directly the web image and put their applications there. We are also working on back fixing and improvements on needle fish which is the current version and octopus that will be like the next release. So yeah there are some additional system applications and demo apps that we are working on making available to have like a. Like a more complete experience using the web application there are like a huge lot of use cases that are interesting some of them we were also showing them in the in the booth is like. When you are using playing. HTML anything you can do with any web example you can run it out of the box on on a GL. Some demos for instance are using JavaScript libraries that detect faces and status so it's very easy to say OK. If I'm a web developer and I want to do an application that detects that you are tired and you get some notification you can do that. If you want to do anything that connects to external services you can do that too so everything like. The web doesn't have any limit about that you can just connect a great communication application that uses WebRTC whatever very easily. So this is kind of the ongoing work and future plans that we have we are also working on every bump of the home screen layout to make more capacity. That people can follow up and know also how to do that there are different flavors they can put their own UI or CSS styling. We keep also updating the Chromium WebRound time to be it's currently using 91 and it will be using 94 in like very soon which is like the latest version that LG releases for WebOS and next. And yeah and also the plan is to provide a better full feature browser user experience so we are shipping like standard Chromium that is embedded into the into the demo. We have a full complete browser there but I mean there's margin for improvement to make it more like touch gesture friendly there are features that you don't need. When working on on it on a full feature browser that you may have in the car but you don't need like all the functionalities or a small book whatever there are many things that can be simplified there so that's kind of the ongoing work in that regard so. I want to thank everyone my colleagues that have been working on this that has been Roger and Jose specifically and yes also me you can reach to any of us. Here if you have doubts or if we can help with with any topic on that we are we're covering on this presentation or related in general to how Chromium is embedded here or whatever. I was not getting to be in detail about the architecture itself. And that was all so thank you. And I think now I have time for some questions. So how can I debug the application using log tracing metrics or there's these. Standard app web application framework using the for example APM application purpose monitoring something so that can you use that those traditional tool to debug or monitor application on the vehicle. So I was explaining a bit how using remote web inspector to debug the application one is running. And if the vehicle doesn't guarantee to be connected to the Internet. For example the undersea tunnel or parking lot. So basement so how those situation going to be handled. Okay that's that's a good question. So I the question I was getting is how to debug the behavior of the of the applications when working on offline scenario where we don't have capacity to use our remote inspector. Well part of the way to do that is like. As you kind of would be working on any other offline application when you're running an app on an embedded system and you don't have the capacity to connect to that. I understand that. You might add some additional logic in the application to add like additional logging mechanisms or something like that you could be like connecting to them later. It's actually a use case I we didn't have any example of that. But yeah that's something interesting that what I could think of right now this like OK if I want to get like status of what happened during the car. I think it would be necessary to be keeping track locally about that. And then use I mean when you have connectivity then you can you can use any of the storage API you have in the browser. For instance you can use any any of the existing internal database storage procedures. Yeah right now right now should be responsibility of the application. Thanks. So you're using Chrome as Chromium as the client of the WebSocket. But I'm trying to understand where the server of this WebSocket is that's taking the request and then actually talking to the cars internal systems. OK so your question was about who is the server of the. Is there a Demian running in the background when you kick it off or. Yeah in this case the server of those services is Cuxa dotbal service. So there's a demon that is running. So it's providing the server to connect to those API. So you just consume that with your application WebSocket. But the server is Cuxa dotbal service. So then I have a car with a new feature. A new API sends into that whole socket pipeline. OK yeah I mean you can also add additional service that are exported. You can add additional services that are using in the Cuxa area. So you can also add services to be exported from that. I think I'm afraid I'm running out of time. So we can we can also continue talking about that outside of line. OK so thank you everyone.