 Let's now put aside the STM32 family to focus on the graphical user interface aspects. First, we will see how the graphical user interface evolution drives the MCU evolution and lead to this workshop. Then, some basic graphics terms will be presented to set up a common language. Then, we will describe the many points that must be taken care of when dealing with graphical user interface management, such as memory or screen refresh. And finally, we'll get an idea of how Graphics Library can help in this task. The release of the iPhone in 2007 changed the user's expectations regarding graphical user interface on small devices. The result is that as soon as a 5-inch screen is involved, user expects a smartphone-like graphical user interface with full colors, touch, animations, etc. In terms of graphical quality, user expects bigger display size, better display resolution, nice icons, and state-of-the-art animations, such as the cover floor, for example. The user expects the same experience as on smartphones. But this means a huge resource on the microcontroller side, also some hardware acceleration becomes mandatory, and the content creation and design also becomes a key aspect. However, even on smartphone interfaces, 3D rendering is not required. 2D graphics is sufficient. Speaking about display on everyday devices, the trend is now moving from optional monochrome displays, such as seven-segment displays that we can see on Alarm Clock, for example, to dot-matrix displays that can be seen on also Alarm Clock, but on elevator screens or toaster screens, for example. And with limited interactivity and most of the time physical keyboard buttons and Knob. On the next generation devices, users will expect color display with touch-sensing and even some virtual keyboard and Knob displayed directly on the screen with which the user can interact. And of course, some richer content, such as integrated user manual as video or animations of how-to, as you already have on some printers that can help to diagnose an issue or to remove some paper jam. Since no specific knowledge was required for this workshop, we must at least present some basic 2D graphics terms that will be used during this workshop. If this vocabulary does not make you graphics experts, at least we will share the same language. A pixel is the smallest element of a digital image. Pixels are usually rectangles, but are only associated with a color value, not a size. The reason is that the size of a pixel may vary among the same screen, so it makes no sense to describe them with their size. It is defined thus with a combination of primary colors that finally define its color value. A bitmap or raster image is an array or grid of pixels that forms a digital image. A bitmap is defined by its widths, its height, and a pixel format, such as RGB 888 or RGB 565. We will come to later in the slides to the pixel format. Just know that a pixel format can also be called color model or color mode. In today's graphics, a differentiation must be done between raster image and vector image. A raster image or a bitmap contains a color value for each of its pixels. On the contrary, a vector graphics or vector image contains a description of the image content. Regarding digital images, a differentiation must be done between raster image, that is called bitmap, and vector image. A raster image contains a color value for each pixels that compose the image. It takes a lot of room in memory, in ROM, usually in flash, because the pixels are stored in C tables. But it does not require any MCU usage, except if there is compression involved, to display the image on the screen. On the contrary, the vector graphics are a description of the content of an image. So instead of containing all the pixel values that describe a circle, we will have a description of there is a circle at this position, with this diameter and the color is this one. So the main advantage of a vector graphics image is their size, because they require only some space to contain the description of the content. But on the contrary, they require some MCU computation to actually display the image on the screen. In this case, a GPU or a graphical processing unit is mandatory. Regarding geometrical transformation, of course, the bitmap, as you can see on the small images that are at the bottom of the screen, on the left side you have a zoomed bitmap. As you can see, we can discover the pixels, because we are actually just making bigger the pixels that are described in the bitmap. On vector graphics, when you zoom in an image, you just add the description in the image to zoom it, and then it's computed when the image is displayed on the screen. So it's direct geometrical transformation, but with a lot of processing involved. As far as graphical interface and microcontroller are concerned, we will go on the left part, so on bitmaps. We will use only bitmaps, because we cannot afford to add a GPU, because a GPU is large. It requires a lot of RAM and of course some complex software to be driven correctly. So it is not required for most HMI, so it is not required for most HMI. We will then only compute bitmaps. Other terms regarding bitmaps are the definition and the resolution. These terms are sometimes confused. But to remember, just think about your television. You bought an HD television, and the HD stands for High Definition, and the HD means that you have 1080 lines on the screen. So it's the number of lines, which implies the number of columns that compose an image. So the definition in general is the number of pixels composing an image. You have several definitions that are very well known, such as VGA, SVGA, etc. The resolution is the number of points forming a given length. It's usually in dot per inch. In display domain, it's pixel per inch. So for example, a laptop usually has today 1600 by 900, which means on a 14 inches screen, 131 pixel per inch. The ISO iPhone 7, for example, has a resolution of 386 pixel per inch. And for the SDN32F746 discovery board, we are at a similar resolution as a laptop, which is 128 pixel per inch. Remember also that the screen size is usually in inch, and is the diagonal of the screen, not the width or the height. So this is an example of a star, a red star, at various resolutions. As you can see, the higher the resolution, the clearer is the star. What we will call a framber phone during this session is the portion of memory that will be directly displayed on the screen. It's a portion of contiguous memory because the display, depending on this type, may access directly this memory. It consists of color values for each and every pixel to be shown on the display as we see in the bitmap definition. Of course, it needs a fast access memory storage, such as RAM, SRAM, or SD RAM. We will see why later. And this frame buffer, depending on the color format, can be very large. Once again, we will describe the color format in the next slide, but just to have an idea, the size of the frame buffer is the width of the bitmap, multiplied by the height, multiplied by a pixel size. And the pixel size, which is one aspect of the color format, can be on one bit, 16 bit, 24 bit, or 32 bits, depending on the display. And as you can see, depending on the pixel size, the size of the frame buffer can grow very fast. And sometimes, trade-offs are necessary between the color image quality and the memory usage. For example, a 16 bits frame buffer can be stored on the internal RAM of the STM32F746G, but not a 24 bits one. The display cycle is the set of operation that leads to the display of one image of a graphical user interface. First, we start with the graphical element of the graphical user interface, such as button, text box, images on an ST logo. In the first step, we fill the frame buffer by composing graphical primitives. A large memory area is needed to store the input bitmaps, but not necessarily a fast one. So the flash in this case can be used to store the input images. The frame buffer is refreshed at what is called the rendering rate. In the second step of the display cycle, the frame buffer is sent to the display. In this case, the refresh rate is fixed by the display itself. Most of the displays are required to be refreshed 60 times per second. So keep in mind that the refresh rate, which is here, for example, 16 hertz, is higher most of the time than the rendering rate. As we saw, the frame buffer may not be refreshed as frequently as the screen needs to be. That's why usually we introduce the notion of double buffering. The display is refreshed at a constant rate, let's say every 16 milliseconds. But a complex frame may take more than 16 milliseconds to be rendered. The idea is then to hide the next frame construction using a second buffer while the current one is sent to the display. The buffer that is sent to the display is called a front buffer. It's sent whatever it contains every 16 milliseconds. And no writing is allowed in this buffer while it is displayed. On the contrary, the back buffer can be filled during the screen display to prepare the next frame. Let's see an animation to understand why double buffering is needed. Let's take an example of an image viewer example. So the idea of the graphical user interface is to first display a loading box. And once the image is loaded to memory and ready to be displayed, as well as some button, for example, next image, previous image, and an ST logo. Then the frame buffer will be sent to the display. At the frame end, the back buffer contains the loading message. The front buffer also contains the loading message and is displayed on the screen. At the frame end plus 1, 16 milliseconds later, the back buffer is being filled with the image to be displayed. But the filling is not completed yet. So the front buffer still displays the loading message. And finally, at frame end plus 2, 16 milliseconds later, the back buffer contains a ready to be displayed image containing all the buttons, the image that will be displayed that is finally loaded and an ST logo. The front buffer is for a few instant still the loading box. But then they are swapped so that the image is displayed on the screen. Without a double buffering, the result would have been a halfly rendered image with a part of a loading box. And we do have created some flickering on the screen. We already talked about pixel format or color model. Let's go deeper in this topic now. So the pixel format is defined by three things. First, the color space on the left image is displayed only a grayscale image. And on the right part, you have a three component color image. Then the color depths, which is the number of bits, required to code the color value. And the color depths define the color precision. Here you have an image with a color depth of one bit. So you have only two possible values, which is white and black. On the second image, the color depth is on four bits, which gives 16 possible values for each pixel. And finally, the last image is on 24 bit per pixel, which gives you more than 16 million value for the pixel color. The RGB888 color format is the most widely used color format. And finally, the data mapping of the values in memory that we will see in further slides. In terms of pixel color coding, a differentiation must be done between direct coding and indirect coding. In direct coding, a pixel value contains the information for each color component it contains. On the other side, for indirect coding, a color lookup table is needed so that for each pixel is coded only an index value, which corresponds to a real value that is stored in a color lookup table. So the image size using indirect coding is smaller than direct coding, but requires an additional element, which is the color lookup table, so that a pixel index is converted in a pixel color value. This is an example of a color lookup table. So indirect color coding only on 8 bit value. You could expect that on 8 bit value, as I mentioned previously, that the image will look quite ugly. But in fact, as you can see, it's perfectly clean. The reason is that only the required color are coded into the lookup table, which is displayed on the right. But this color lookup table contains only 256 colors, so it's quite small, but it's dedicated for this picture. You cannot use it for any picture. The alpha channel defines the level of transparency of a pixel. The alpha channel is the A that stands in the RGB 8888. So our RGB 8888 means that each component, transparency, red, green, and blue are coded on 8 bits. Each color component ranges from 0 to 2 power 8 minus 1, which is 255. So you have 256 values for each component, including the alpha channel. Usually the maximum value for the alpha channel means complete opacity, and a null value means full transparency. For example, this red square has each of its pixel alpha values set to the maximum value, which is 255. The second box is half-transparent, and this one is 75% transparent. With the alpha value, or transparency, comes the notion of alpha blending. Alpha blending consists in mixing a source image with a destination one, taking into account the alpha information of each pixel. For example, let's take this bulb image as a source. Note that the chessboard in gray and white behind the bulb is actually how transparency is represented in an image. So you can assume that all that is outside this bulb is transparent. Each pixel has a zero value for the alpha. The destination is this background image, which is fully opaque, and of course the bulb in itself is composed of pixels that have alpha value as its maximum, so it's fully opaque also. For the computation during the blending, the alpha value, which is comprised in between 0 and 255, is converted to a floating value between 0 and 1. The reason is the formula required to compute the output color, which is displayed at the bottom of the screen. In theory, this formula should also take into account the alpha value of the destination, but to make it more simple and easier to understand, we remove this one. So the output color of each pixel is computed from the input color of the source multiplied by transparency, added to the pixel color value of the background, multiplied by 1 minus the transparency of the source. The result is the bulb mixed with the background without the square around the bulb. If the source image didn't have any transparency information, we would have got a simple mixing result containing a square around the bulb. Anti-aliasing. The aliasing is a staircase effect observed on oblique ages of digital images, as represented on the right side of the screen, but also and more specifically on fonts. Anti-aliasing consists in adapting the edge pixel alpha value to mix with the background. For bitmaps with non-square edges, as you can see on the anti-aliased version of the image, the alpha value of the pixel on the edges of the edge forms are adapted so that the staircase effect do not appear anymore on oblique lines nor on a circle. This is even more important on text. Here is the version of the ALater with anti-aliasing. Finally, as mentioned before, the pixel format is also determined by the data mapping of the each pixel. Here is the list of all the pixel formats that are supported by the internal DMA2D accelerator that we will see in the next session. The idea is to show that depending on the pixel format, the data will be stored differently and will take less space for each pixel. For RGB 8888, you need 32 bits to represent one pixel, but for 16 bits representation such as RGB 565s, only two bytes are required because the green value is split on two different bytes. And finally, the L format at the bottom of the screen are lookup table ones, so they are very compact. But as we saw before, they only contain an index for each pixel, but needs an external lookup table to determine the actual color value of each pixel. In this section, we will describe the graphical user interface, composition and update processes. First, the graphical user interface is composed of graphical elements. These elements can be basic geometrical forms, such as rectangles, circles, images, text, buttons, and more generally advanced graphical controls that are also called widgets. Widgets is a shortcut for Windows gadgets and is the generic word used in graphical user interface domain to describe an advanced control. So a widget can be a progress bar, a checkbox, a slider, or a customized one. Actually, we are talking about widgets as soon as there is a state machine behind a control and an associated action. Then, these graphical elements must be managed. First of all, the graphical elements appearance must be managed through a set of primitives that compose the elements. For example, this button is composed of several forms, basic ones, such as rectangles with transparency information in it, but also some text. And the different text allows to draw some shadow on the button and the various basic form allows to make some 3D aspects. Also associated to a graphical element is the element states. For example, a button is either pressed or released. But we must also handle their purpose in the overall graphical user interface. Their purpose means their relation with other elements. Thus, a button is in front of the background or in front of another control. So that is called the Z-ordering information. It is the third dimension of a graphical user interface, the first two ones being the X and Y information. Their purpose is also dependent on user interaction through the two screens, for example, or even through physical buttons. And also dependent on system events such as sensor input, for example, for a widget that would display the temperature of a room, or a timer also for a clock. And finally, the actual refresh of the frame buffer must be handled by the graphical user interface manager. Regarding the frame buffer refresh in itself, let's consider these two clock widgets. The left one has a second hand, which is moving, of course, every second, and requires a refresh every second. On the opposite, the right one has only minutes information, but has a temperature also information displayed on it. So the right one must be updated every minute or maybe more if the temperature changes. But still, the display needs to be updated 60 times per second. The frame buffer does not need and must not be updated each time something moves in the graphical user interface. That's why we are talking about partial rendering to update only the required area. These areas are called dirty area or invalidated area. For example, for the clock widget, when the second hand moves, only the part that contains this hand will be updated, not the rest of the clock. It will save some flash read operations and thus reduce the overall memory bandwidth. Further, objects that are hidden by other objects do not need to be refreshed in the frame buffer. This is called occlusion culling and strongly depends on the Z-ordering information. For example, here with the background on the clock widget, the background does not need to be read entirely from the memory since it does not need to be updated while it's hidden by the clock widget. So from the graphical user interface logic to the frame rendering, there is a lot to take care of when managing a graphical user interface. Fortunately, GraphicsLibrary handle most of them almost seamlessly and efficiently. ST has a partnership with three solutions, M-Win from SIGUR, EmbeddedWizard from TARA, and TouchFX from DROPNER. These three solutions propose first a framework which is composed of ready-to-use advanced graphical control elements that we call the widgets. An environment which is usually object-oriented, this environment allows to handle the position and size of each graphical element of the graphical user interface, but also to handle the graphical element's connection to other graphical elements or to user-based interactions. And finally, they provide some additional tools such as graphical user interface visual editor, but also some build and prototype environment. At least the GCC tool chain is provided. Some tools such as TouchFX also provide some ready-to-use kill project or IR project. And usually they also propose some bitmap and font importation tools to convert bitmaps and fonts into C tables. But the core of a GraphicsLibrary solution is Graphic Engine. The Graphic Engine allows to save MCU usage so that the application can still perform some action while the graphical user interface is updated. It can save the MCU usage using a deep knowledge of the platform architecture. For example, on the M7-based platform, a floating-point unit is available and can be used to leverage the computation processes. Further, some hardware accelerator such as the Chromart is available on some STM32 products in case save also a lot of computation process from the MCU. It also saves some memory bandwidth usage by compressing the bitmap or using indirect color mode that we saw previously. They propose advanced occlusion calling algorithm that can skip the reading of hidden objects and finally, the dirty-region detection algorithm allows partial rendering to save also the memory bandwidth usage. All this processing runs in software, of course, but usually has a low memory footprint on the solution selected by ST. It's around 20 or 40 kilobytes of RAM and they use deterministic memory allocation as much as possible. In conclusion, at the end of this session you should now understand the vocabulary to deal with graphical user interface. You know what is a bitmap, a frame buffer, what is alpha blending, what is a pixel format. You also understand the difference between composition rendering versus the actual screen refresh, what is doubled buffering, what the display cycle involves. You know now what needs to be watched to handle efficiently a graphical user interface which is composed of graphical elements and should perform occlusion calling and partial rendering. And you have some hints on how Graphics Library solution can help in this process through a graphical user interface framework, the graphic engine and some hardware optimizations.