 Hello, everyone. Welcome to this presentation about GraphQL user interface in embedded systems. My name is Hidenori Matsubashi from Japan. This is my first talk at Embedded Renux Conference Euro, so I hope you enjoyed the talk. Today in my presentation, I'm going to talk about ZUI Toolkit. The title is GraphQL user interface using Flutter in embedded systems. Okay, now first let me introduce myself. I've been working for Sony as a software engineer and I've been developing the wide range of embedded software for over 10 years. I've experienced both bring-ups such as Qualcomm, NXP and NVIDIA, and developing middleware of graphics and video. As a whole low-level radio, I've also experienced developing firmware, device driver and etc. Now, I mainly use C++. Here is today's agenda. First, I'll talk about the background of a GUI in embedded systems. Next, I'll talk about the property of the new approach using Flutter. And then, I'll show you the demo video first. Finally, I'll talk about details on our approach and demo video. First of all, I'll talk about the background. So far, we've been searching SW GUI Toolkit or Fireman for embedded systems. There are several options including OSS, commercial software and Toolkit from SOC vendors. Naturally, we want to use a Toolkit that exists in OSS. However, there aren't a lot of GUI Toolkit available for especially our consumer embedded products in OSS. Why not? There are two main reasons. First, there aren't a lot of Toolkit that meet our requirement. The other one is typical challenges on embedded platforms. And these are our requirements. Naturally, modern and high design, easy of development and good performance are required. And then, it is also important that it is easy to port to another platform even if the SOC or platform is changed. Software lessons is also sometimes important to develop from priority software. I think this is common. Now, these are examples of GUI Toolkit, but it doesn't include all that are limited to a specific platform such as Android and iOS. QT and WebView, such as WebKit and Chrome, are often used in embedded systems. As for QT, some of its tools are only available when you use commercial license. On the other hand, we think Flutter is one of the best Toolkit for embedded systems as a new approach. Next, I will explain about WebView. We can generally develop GUI easily by using WebKit or Chrome as a WebView. However, there are also some cons. For example, WebKit is huge source code and needs a lot of dependent packages for libraries. Therefore, generally, introduction costs it higher, and WebView also needs more performance than other Toolkit. Especially, it is difficult to access directly local files or hardware resources from web apps because of security. So, when you use WebView, you need to use internal web server in an embedded system. In addition, in some cases, security measures may be required. Next, I will explain about some native Toolkits such as 3DK, QT and SDL in this time. 3DK is a major Toolkit in Linux. It supports cross-platform-like desktops. However, it doesn't support modern UI like smartphone, so 3DK is suitable for desktop apps. Next, QT is very popular and it is often used in embedded products. It also supports already-always Android and iOS platforms, but its software license is your license and the commercial and open-source licenses. Finally, SDL Simple Toolkit or Graphic Library. When you use Simple New or Create Game Apps, because it has no widget for creating UI, so it's not suitable for normal UI. Now, let me build on Flutter we choose. Flutter is Google's new UI Toolkit released in 2017. Features are building beautiful UI like smartphone, natively compiled apps and support cross-platform from a single source code. And then, Program Rangleties. Here is Flutter's architecture overview. Flutter mainly consists of three layers. First, Flutter Framework is libraries for user applications. Developers usually create UI by using widgets or APRs provided by this framework. Next, Flutter Engine is the original rendering engine like web browser's rendering engine. It is written in C++ and user.source code works on that virtual machine and draw by using Scea Vector Graphics Library. Finally, Emulator is the protein layer for specific platforms like Android, iOS, Linux, etc. Basically, you can plot Flutter engine to any platforms. So, I will explain why we choose Flutter later. Now, let me move on to the second reason. Generally, when we develop embedded product, there are some challenges. As you know, development is usually done on different architectures and resources such as CPU performance and memory size are limited. Especially, supported display server by BDSP such as X11 and Warrant is important for GUI Toolkit. So, we need GUI Toolkit that are more lightweight as possible and independent of hardware and architectures. Next, I will explain about BDSP trends of supporting display servers. Here is the example list of some resources for a platform. There are links, video, protocol, protocol, or Raspberry Pi. Many vendors support Warrant and it may be gradually shifting to Warrant. NXP doesn't support exhibiting on some latest boards. Warrant is suitable display server for embedded system. However, a lot of GUI Toolkit and enough support will Warrant because they are based on X11 and still developing. For example, this is example app of CTK. And then, when I run this app in a Warrant environment, the drawing size is incorrect like this. So, it is unlikely that GUI Toolkit will work perfectly in a Warrant environment. Therefore, we need to evaluate firmly before using. As you can see, there is an issue with the combination of Warrant and GUI Toolkit. Flapping up the background. We were looking for Toolkit that can be suitable for Warrant and embedded system and has high functionality and designability like WebView. Let's move on to next topic. I will introduce and explain about new approach in embedded system using Flutter and Warrant to resolve the issues I told. I think Flutter plus Warrant is a best practice in embedded systems using Linux. And why is it Flutter? Why is the combination of Flutter and Warrant good? I think you feel doubt. Next is the reasons why we choose Flutter. First, creating a modern UI like a mobile app is very easy and Flutter supports all platforms such as mobiles, desktops, and web. Therefore, Flutter is a popular OSS and there are a lot of information in the internet. Here is Flutter's total number of stars in GitHub. It is in top 20 in the GitHub ranking. And next, Flutter provides the custom embedded API layer for porting to specific platforms. Therefore, the porting is very easy. In addition, Flutter needs few dependent libraries. Basically, it needs only OpenGL or EGL library. Finally, software license is BSD3. And next, I will explain why the combination of Flutter and Warrant is good. Flutter, especially the raw layout, it usually draws graphics by using OpenGL or Valkan. Other APIs, Flutter, Invitator, or Communication API as an input, output, API, and so on. Basically, as a Warrant client, all you have to do is creating an EGL communicating with Flutter framework and supporting the input, output, such as a mouse, touch, and keyboard. So it can be ported as a Warrant client very simply. And therefore, as for graphics, Flutter uses only EGL and doesn't use a Warrant protocol. So there are no described problems like other toolkit have. Here is some APIs of Flutter, Invitator. The APIs are very simple. If you are interested, please check the source code. Let's move on to the next topic. In this time, we created the demo application at the UI like a tablet using Flutter. I will show you its demo video. In this time, I used NVIDIA Stats on our board. Let's start. As you can see, it has the clock display and the app launcher like a tablet. Naturally, you can develop Flutter orbs and run it on our Linux host desktop. This is a typical Flutter sample app. Then, a variant for ARM64 on the target device. And then install it from remote bridge code to target device. The same app works in the ARM environment. You can also easily create an app like a music player. This is a mock music player app. You can also install the app from remote bridge code. That's all my demo video. For your information, I will show the chart of Flutter's performance. This chart shows performances when running the demo app's show individual from booting on JetSunderer. Each Flutter app uses around 100 megabytes of primary and usage rate of CPU and GPU are not too high. Let's move on to next topic. I will explain about details on our approach. Here, read the architecture overview of the demo app. The inside of the red frame is our developed software. In application framework layer, there are mainly two modules. This is the graphic chart for Flutter module written in C++. The other is system UI written in dot and C++. These modules provide system menus for users. The user application layer is for the user's application to be launched from the system UI. These are Flutter app or other programs. System UI is the Flutter app written in dot and C++. It's currently only for demo, so it has no enough functionality now. The future of the application launcher is showing or ensuring user apps and croc. So you can easily create an UI like this using Flutter. Next, I will share the graphic chart for Flutter. As I mentioned, graphic chart has mainly three functions. The first is managing the Flutter engine. It uses the Flutter embedder APIs to manage a Flutter application project running on the Flutter engine. The second is simple window manager. It runs as a desktop shape client on Western and creates easier surface for Flutter engine. In addition, it sends the event of user's input such as a mouse, touch, and keyboard to Flutter engine. The others are communicating between dot and platform's native programs. The system, local, update, and so on. As for this module, we've been using the Flutter official source code as the platform-independent port. Currently, we've been using Western as a variant compositor. Here is default Western desktop shell, so to replace it to our graphic shell, we use Western in Confirmation file. Here is the example of Western in Confirmation, but using client field in the section field like this, you can already use it as your own desktop shell. In addition, you can use and create custom field like this. Besides Western in, you will need to return ready flag when a variant client here is ready to show using the Western API. And you can also get information from your Western in. By the way, I will explain about the official Flutter Linux desktop specification. The official Flutter for Linux uses CTK as the graphic shell. Because Flutter doesn't have traditional desktop widget like open directories, choose files, source files, and so on. So, using CTK, the developers can create traditional desktop apps and modern apps like mobile. I think this approach is reasonable. In addition, Western CTK has already supported WebRAM. However, in embedded systems, it doesn't require traditional desktop UI and other dependent libraries like X11. So, as for embedded systems, I think Western plus Flutter is simpler and better than official version. Next, I will explain about IPC and language binding in Flutter. You have some options. The first is using the Flutter API. APIs called method channel, event channel, and basic message channel are provided. These are syncs communication protocol between dot and native programs like C++. The second is the Dart FFI. Developers can use the Dart FFI library to call native C APIs. FFI standards for foreign function interface. The third is the Unix domain socket in Dart I.O. Unix sockets are writer-than-TCP IP sockets. Developers can use Unix domain socket in Dart. The others are using third-party libraries such as Diverse, GRPC. So, next, I will explain about this GRPC library. GRPC. is the GRPC library in Dart implementation. The owner is also Google. You can easily use GRPC in Dart by using this library. And then, we will be contributing to support Unix domain socket in GRPC. Third, this function or pull request uses Unix domain socket for IPC and GRPC for external process communication. Unix sockets are writer-than-TCP IP sockets when you use an IPC. Next, I will share about Flutter Engine. As I mentioned, Flutter Engine is the original rendering engine with memory scale and Dart virtual machine. We are using the OSS Flutter Engine as is. However, currently, Flutter Community doesn't support Linux R64 host. So, we've been contributing to support Linux R64 host. And then, you can use the Flutter in R64 embedded systems now. Finally, I will share about Flutter SDK. Flutter provides the Flutter SDK to develop and debug Flutter Arc. However, like the Flutter Engine, it doesn't support R64 host. Therefore, we've also been contributing to support multi-architecture host. We were especially focusing to support working on Linux R64 host and cross-boarding for R64 target. Let's move on to the final topic. Here is the summary. I introduced the new GUI approach using Flutter embedded systems. Our motivation and purpose is using Flutter in R64 and Linux embedded systems. Going forward, we will continue to contribute to Flutter's Linux support because Flutter for Linux is still our version now. Thank you for listening.