 Okay, so hi everybody, I'm Gupta Chirizavi and I will share with you some of the experiences I had in putting Linux on embedded devices and in a band of people on the vendor. I'm an embedded Linux engineer at AI Sportcon and I'm designing an exploration of the developers, I've worked at the cameras of Sportcars. I mostly take care of the low-level stack, the load loader, the kernel device drivers and so on. There is software integration in the system. But the things I'm talking about today are not related to this company because of previous experience. I'm also a developer that will contribute to some progress through the big workload system. So, introducing the topic. Every system chip manufacturer typically provides a sort of BSP with software you will need to use it for Linux. This is the idea of BSP that I would love to receive, which is just three things. It's a main line of external, main line of boot or barebox, and very good hardware documentation, easily accessible. Nothing more and nothing less than that. And I will explain at the top why I would like to receive this. But unfortunately, this is not what I would see. So, just a few words about the system chip that I'm talking about. It is called Duraton N32-926. It's based on an R926 for 240MHz. It has a very normal set of peripherals inside. Most not to be an H264 encoder and decoder. It's a net audio codec. It is quite a low cost chip and it has a pretty unique solution. It has 64MB of video up to around inside the package. So, that means this is not accessible outside. You have a package with both the system chip and RAM. There are two different types. Two different parts of silicon, one and internal together. So, you don't have access to the RAM bus on the outside. This allows for a locking count and several other advantages on the outside. And it also has an entry of the package with several cost settings. So, overall, it is suitable for low cost IT cameras and other video applications. So, we have really set this quite nicely. The processor is not very fast but still far okay for several applications. So, my quest started with the software because I am software engineer and this is where my issues started. The main chapters that we covered are documentation, the use kernel, booting and development tools and customer support. So, the software documentation, as I said, is very important. It would be nice if it were easily accessible. Unfortunately, on the website for the product, there is a data sheet which is actually only a main page document with a list of features. So, it's not a real data sheet. You cannot decide your product with that. That's all I found on the website. If you are a customer, you have a documentation and all the BSP but it suffers a GAs so I cannot cover that. You can access something by buying this relatively low cost development kit which is available on Chinese online stores. It contains a pretty normal development board and a DVD with a subset of the BSP first customers. It also contains this document, the M5292X design guide which is what other manufacturers call the hardware reference model or similar. So, it is what documents all the peripherals inside the system on G which is what you mostly work with. For example, it has a list of registers, defeats in each register, and with a combination. In most cases, there is also a general description of the device which works. Overall, the document is lower quality with respect to other vendors could see. But if you're used to this kind of document and using it, you probably can fill in the gaps with your experience. It was quite enough for me in my job. But let's go to something more technical and let's have a look at the Linux program that Vendor provides. Vendor provides a kernel based on Linux 2.6.35.4. 2.6.35 was released in the beginning of 2010 so the feature set is quite old now, almost 80 years. The .400.4 is not even the latest in the 2.6.35 range because that is .14. In the middle, between the two, there are 11 months and 1,400 costs. So chances are some of these bugs affect your product. But luckily it is very easy to merge the .14 version with minimal conflicts. So that's why I've been using that. Things are a lot more different if you think about using mailer Linux because the difference is huge, it's totally uncountable. New features, boot speed improvements, new drivers and so on. Unfortunately, migrating is not fully doable in the short time. One thing that happened in the meantime between 2.6.35 and current models is the by-stream which was a big revolution in the architecture. So if you want to use new drivers, maybe you will have, well, definitely will have lots of work to pull back for that. On one note on security, you read on the news from time to time that buffments have been created out of internet connected devices. In old kernels, as in every old software, security flows. Many of these have been discovered fixed in regular kernels. They aren't recorded in a while. So if you're using something on kernel, you have vulnerabilities and everybody knows a lot of things and you don't have access to those processes. So the main issues with using an old kernel the vendor changes amount to 170,000 lines changed, which is a lot, but maybe not necessarily that much for some of the new systems. They are provided as a set of patches. There is no big rainbow, so you cannot see why this change was made. You cannot take long or play the main thing. But you have, I think in the latest PSP there are five or six patches. There are a few five patches. The first is 3.6 megabytes. So it's really a huge good drop that you put there for comparison. You cannot really see what was done in the middle. But in case you are unable to apply a patch, you can just speak the problem with that. So let's see some of the main issues I faced with this kernel, which are mainly bugs, missing features, and good old issues. The first time I tried to record from the microphone, I did a record of my 5.1 megabyte kernel in Gresh because there is a public reference if you use the default settings with ASA. That was quite easy to work around. This took more time with the 8.64 decoder. It has a lot of non-pointed references. It works very well. We perfectly included the trims for both of them. But if you are doing real-time trims on the network, you will not be using it too much. So you will have packet loss with normal. With packet loss, that trim is not perfect, and this will trigger lots of non-pointed references to the rival. So it is actually a tiny piece. Talking about missing features, most of all the devices of the whole are implemented. That's something we needed to understand. In FGPios, you can save and get GBAOs. But in E-DRAT, it's not implemented. So if you want to do something with a battery, you have to pull it from the state. Or implement E-DRAT in Europe. Power Patrick is implemented twice. With the proprietary RKI, and with the Linux standard RTI, but that doesn't work. It is incomplete. So it's going to take a lot of time before you get that quality. Finally, good quality. The current additions by the vendor change from 5 to 5, but they are generally quite bad. Just one trivial metric. The additions include more than 500 lines up to a dash of 0, which probably gives you some hint of current quality. But let's see just one example. There are probably no slides, but this one is the most meaningful. As I said, the 8264 driver has bugs, so in order to investigate them, I thought I'd give this a body to be able to test more quickly new modified versions. Because it's not possible. And that's because the device needs a pretty large amount of continuous memory. And the way it works is it has a function getting the buffer size that returns the number of bytes of size for this buffer that needs to be allocated. But the allocation is done in the memory management code in reserve node 0, which is called very early due to the boot process, which allocates memory and calls getABCBufferSize, that one to know how much to allocate. So if you were to build the driver's module, that function would not exist in the kernel, and so it wouldn't even be the only driver. So in order to test the version of this driver, we have to review the reflection of the kernel. Okay, next big chapter here is booting. For booting, in all the better PSPs that I have seen except this one, you get some form of U-boot. So we somehow have mainline support, which is ideal. Some ship a new version of U-boot adapted to the board, to the system achieve, and maybe with bytes, but in this case, there is no U-boot at all, no bootloader that you probably already know about. There is a bunch of four private bootloaders. These bootloaders are provided with sources, but they are not open source. Technically, on the top of the files, it says, well, private reserve, so you are not even allowed to use them in your product, which doesn't make sense to me. But from the technical speaking, the bootloaders are only thought to work on this system achieve or similar achieve of this in families, so they are not generalized. So if you have other products that you have U-boot and you are satisfied with it, it works well, and it's your firmer dates or whatever, and you think about reusing that scheme, you cannot. Let's have a look at how these bootings these bootloaders work. So with the top, you have a very standard booting scheme. Of course, there is no real standard, but this is quite a typical example. The example is for Netflix, but the same principles apply to all media. And the typical system achieve has a bootrom that loads an SPL into the turnaround from storage, that initializes the external run and loads the main bootloader in the external run. The bootloader then loads the kernel maybe from a fine system, or maybe not, and the kernel then mounts the first system to use it. That's it. With these proprietary bootloaders, what we get is very similar at the beginning, because the bootrom loads from storage as a small bootloader, it is called a downloader or an SPI order, depending on your memory. And that one loads from initializes the external memory, which is external, even though it is in package, it is not known to the system achieveer or whatever, so it is initialized in a standard way. And then it loads from storage the main bootloader into external memory. So far it is pretty similar, but this bootloader does very different things then. Normally, for the error going down, it looks for two fast partitions in net, and if they don't exist, it wipes the net and trips them. Then it mounts the first partition, and it looks, therefore, a file called comproblemin, which is nothing but a Linux image with a penalty in front of Fs. So we do look at how the zero executes it. In the init from Fs, the file partition is mounted, and the init string looks for executes a file called start.sh, which in turn can do other things without the files. But during boot, if at this point if you press the button, if you can press the button, it will not do all of that, rather it will expose all the two fast partitions as a USB storage. So you can access them from this easily and change the files in there. The goal that the main advantage this team gives is in deploying demos, the end of provide some demos to the most trained hardware features. For example, this one which is a demo for the video player, the H.264 player, and in that case it has comproblemin, start.sh, and players, and a bunch of beautiful files. So it is very easy even for the hardware designer who maybe has only windows on his PC to just press a button to reboot, mount as a storage, replace the files it does provided in the demo, reboot, and look at the demo working. This is very handy when you are trying to test the features of the system. But this team has a lot of drawbacks for making a real product. The fact is completely unreliable on power loss at least. It is inefficient and it cannot post a real Unix file system. No permissions, no security, and so on. So it just cannot work for that. Also working on-man is itself unreliable. So the demo provides a sort of fat on-man immigration layer which creates a fat protection over an end in the internal FTL which is provided for Linux as a binary module. So you have to really rely that it is working perfectly. You have to trust it. If it has bugs, you can not do anything about that. And the MPT model cannot mount anything but fat. So it cannot mount UDI, FSJ, FS2, or anything. So it is pretty locked. Also this team has no redundancy at all. So if you have failure in your memory, the device will be leaked. And also having exposed the effect of your data if USB and the boot mode buttons are accessible to the device you, the end user of the device, they will be able to grab all the data and modify all the data on your device as well. Other Provex related to using any fromFS. The fromFS is a model for picture in the Linux kernel but it is volatile and it uses RAM to store everything even if you are not going to use. So it is not valued for a real root system for a device that is not super tiny. It really leaves just all this stuff at Alvres Zero and Jamstreet so it cannot pass the kernel line to the kernel. The only option that you have is to have quality in the kernel that is defined for that config that is called online which works. But let's say you want to go NFS booting, you have to change the kernel line so you cannot do that in new boot and NFS booting so it is all time consuming. And you cannot do TFTP load of the kernel because the NFT loader does not know TFTP so you really force to replace the kernel every time. So there are many issues as you have seen and of course I started looking for some alternative option and this is what I have come up to. The first idea was let's keep everything as is except the file to meet and let's have standard usage mount a SquashFS file system and then switch to it. So this is possible, SquashFS is a real Neonix file system. It has all the features, it is efficient, it is compressed and being actually it removes most of the constraints from the NFS but this scheme has also many bugs. SquashFS is read only so you cannot modify it with the NFS, you can modify it but it is volatile, here you cannot even modify it. On the other hand I would not rely on booting any read write file system over FAT over and over and at flash it would be completely crazy. So all the scheme is not very satisfactory and also you cannot atomically upgrade this SquashFS file system because FAT cannot read any. So this is not very satisfactory and I went a step ahead and said okay let's tweak a little bit more, let's tweak the activity loader in addition to the NFS and so let's avoid this feature to create two FAT partitions and let's the only one and leave the rest of the space for a QBI logical volume so in this case you can have a real read file system it can read write if you want it's efficient safe for man and so on so it's a good solution for the read file system it needs some tweaks but not that many on the activity loader as well as the NFS so basically now the whole FAT idea and the FAT partitions are amplified to the very minimum to contain kernel anything from FAS so whose main benefit was to be able to upgrade VLSB storage so now we can upgrade VLSB storage but only the kernel in NFS which is not that super useful and so we lose the big feature of NPT loader and so the next step was quite natural to remove NPT loader and it turned out to be feasible and not even very difficult because what NAND loader does what NAND loader does is just loads images and puts them at an address to specify the header and then jumps to it and just loads our kernel with the NFS at address 0 and jumps to that tweet and it works so this way we completely remove NPT loader we completely remove the FAT and less code, less bugs, less boot time and it is a lot more satisfactory it starts looking like a typical Linux embedded device the drawbacks are the kernel is still on bare hand the protection from UBI but NPT loader can model from there and also it has no nothing similar to a U-boot environment for this last point there is something you can do and it is create another UBI partition UBFS partition the minimum you can do store like text files which are pretty much like the U-boot environment and in this case you can have basically your UBFS has it's own environment which is right it looks pretty much like the U-boot environment and it's also accessible from user space so you can do things that are very similar to U-boot in other words you are kind of using Linux as a U-boot loader in some terms ok this is where my journey stopped we were similar to this actually I didn't go one step further which would have been very nice to support this platform that would be the best thing to do of course because it would mean that users don't have all of the usual U-boot features enabled like network routing, scripting and whatever you know of but unfortunately this would open a development window which is relatively large and difficult to predict in my case so I didn't go on that way but if somebody wants to do it we'll love it ok tools of course you have lots of development tools on your development PC for developing embedded product but there are none in my IDE ESP if you remember correctly actually this is because I would love to use all these kind of tools in most cases system chips do the same things in slightly different ways so there is no real reason to have different tools to different vendors but there is at least one one case where this doesn't happen and it is for flashing a device that has no usable feature so either it's just out of production or it was just produced so it has nothing on storage or it's built so you have to talk to the boot form directly and flash for the first time in the memory and every vendor as far as I can tell implemented their own protocols and there are tools to do that some are good, some are less good but they are all different this is also what happens in this case in this case you have a tool that there is only one tool which is actually pretty powerful because it can write all the types of memory it has several options all of the memory that can boot can also be written with this tool including SD if you need it it can also write it to external RAM and execute from there directly so it works quite well it uses USB as a connection which is very handy in my opinion so it would be quite ok except it has several drawbacks first it is a movie program so you cannot put it in a script and command it to do some more complex thing it's for Windows only so in addition to your development machine you also need a Windows machine to fill it with something it's proprietary there are no sources for this so you cannot like fix the above issues and finally the protocol to speak with the boot from is not meant so unless you want to reverse engineer that you basically have to speak with this tool and there is nothing you can do a strange thing this tool does is when it writes into memory it also stores there a sort of partition table typically in empty devices there is no partition tables here they do that it might be an interesting idea because it might be handy in general but it's unusual that after all you are not used to using it it's not super useful it's not even burning generally because it burns you there is no way to burn it because the flashing tool does write it unconditionally also in an order it uses it to know where to find the next image to load and the address memory where it should be loaded and a few stuff that's okay for the rest I was able to use standard tools not always easily but it was possible some words about customer support you need some degree of customer support at some point with any system achieved even the best supported ones but this gets even more important if the support you get out of the box is not good customer support in this case it's quite responsive the usual respond the next day to good times of issues at least for me where I am you cannot have an answer on the same day but you get an answer on the next day the answer does not always fix your problem this is an example of a conversation I had via email with them with the Windows tool to flash the memories it was not working on my machine so I wrote the background we told the details about that and the next day the answer was it was my PC's which I attached that didn't fix my problem so I wrote back to them and said I think I asked probably I definitely asked if software can log error so I could set that to log and they could diagnose it and the next day I got another answer which was quite polite we are sorry but I didn't log it, it would not be practical this was the end of our conversation so I involved my thieves and trying to wait to get around that so let's try to draw some conclusions hopefully to find some way to proceed better finally the result of this work compared to the world support system which it is it took a much higher development cost we had to support ourselves to find bugs, fix them read back booting with two very similar type and so I could just very very roughly estimate that in like 6 to 12 months of work at least which is higher development cost you might not care about it if you are producing one million pieces till the higher development cost it is going to be like something euro per device but you will care about other things so it is time to market you care, even if you don't care about development cost you care that your device will be 6 months later on the market because it has issues to solve and everybody cares about time to market also the thing that I most care about is the final product quality for this product was a lot lower than it should have been so it has issues of problems that cannot be directly solved due to the limitation I just talked about and it is really a pity because the hardware could do much better than it has so as an open source minded person I started asking what can I do to improve the situation if I can do something and I didn't find much actually that I could do except as an embedded looks engineer so I'm always speaking to many embedded looks engineers here it is very very important that you have any potential problem issue very very early before selecting the system in chief I've seen several cases in which the system in chief is chosen based on several several priorities cost electrical features hardware features but software is often misunderstood by decision makers and so they are often completely not understanding what the difference between having limits which is a huge difference they hear from the segment there is limits for this chip but there is a huge difference and as an embedded looks engineer we should try to communicate this then if you are a hobbyist or hacker you simply not buy goods with the best support if you want to do your advice for example or you can do the opposite thing buy these devices and start improving their implementation and bringing it to the main line that would be a lovely thing to do you lose the gun or whatever but unfortunately it is not a job for a couple of years it is a large job so it takes time but it would be very, very new to have who is really in a position to change the situation of course is the vendor so I would like to suggest to every vendor to first not reinvent the wheel just port your goods and give your goods to your users they will have the environment that they know and they know how to leverage they will be happy with it write good documentation make them easily downloadable so everybody can study the platform without waiting to sign any ATAs or anything and also document your good from protocol so if your tool is bad no matter people you are there and be happy with it of course push your code to my line I could repeat this a couple of times it is definitely not consuming it is expensive it takes time but it is a lot rewarding in the long term it gives a lot of longer life to your device and finally leverage the community which is a powerful tool that works for free for you so let your support engineers reply directly on public mailing lists on stock control or ILC these things maybe exact ILC are stored forever on the internet and when the next engineer will find the same problem they will find it in a few minutes over the internet and not wait for days with customer support and your customer support people will be free to do other works and also maybe other friendly boards can help a lot so we can learn from new group although I am not sure this team would be very much interesting to hackers because there is no CPU but generally speaking I think this would be a lot ok when I gave this this one in February someone from the audience proposed an interesting idea he said just like we have a compatible with the stand logo why can't we have a Linux mainline logo so actually I am not sure this is believable it would have lots of issues to be made but at least it helps us in understanding we probably need some way to communicate very directly very easily between the decision makers this has a lot of support this has just some support which is a lot different so it's a matter of communicating to the people so this is a simple way it might be other ideas but it's probably something that is very important ok that's all for this talk hopefully some time for questions and I have a question for you first of all how many people face a similar situation with a badly supported chip or a very old kernel can you raise your hands ok good good suggestion who has no problems with PSP oh you lucky guys you ok I am not surprised first because if you are here you probably care about the topic and second because last year in this conference in Berlin Wiltscheper gave a similar to this stuck in 2009 how it survived it was a bit different talking only about the kernel but it got me inspired to show my experience so we will and ok so except for the two I am with you I totally understand you I am close to you this will not change your daily job at all but at least you feel understood ok any serious question yeah why I use this chip and now throw with me why do you continue using this chip because I am not the one who chooses the chip and of course there are several reasons besides support and some people care mostly about other reasons I don't want to make it by class but yes it is a suggestion it is what it is a suggestion it is what it is a suggestion for Wiltscheper I don't think I should advertise this one I also don't know all of them of course but I think if you look at the kernel sources and see what supported what is not just take the latest manai kernel you will not find this chip you will find others so that is the first idea of course not equally well supported but it is the first idea also for some chips like Texas or Winner and a few others you can find websites with comparison tables with which peripherals are supported partially supported from which kernel version and so on so you have to do some research on that I'm afraid there is a comprehensive guide in this selection you provide BSPs and you ask what should I provide yeah well it was in one of the first slides I would love to have mainline unit mainline kernel and very good documentation that's it because after that there are two scenarios and this kind of work some use some use YAPO, some use GMC some use muscle so why forcing people to work in a given way some companies provide like a their own build system there is there used to be a DLU from Fisca Italy which comes out there are others many engineers work on different system changes with different vendors so if they have to learn again their job every time so let them use the tools that they know that work well for their use case that's my opinion of course and okay that's all thank you very much