 So, my name is Enrique Vallabó, I'm from Colabora, and as electronic engineer I didn't learn much about Linux in the university, even so I learned as a hobbyist that was 20 years ago. My first job was not really related to electronics, but was related to Linux, which was my other hobby. I started joining in a big company and started programming some drivers in C++, I'm playing with Corva. Anyone remembers Corva? What is it? Wow, one person. It doesn't matter, even I remember nothing about it. But yeah, after a few years later I got fired, I was fired of this company, but I was lucky. I was lucky because then I had the possibility to start in a very small company. We were three people, all dedicated to electronics and engineering, and our first job was to create a computer model based on an ARM processor for our customer. We started selecting the processor and evaluating a bit the evaluation kit that the vendor provided, and we thought that would fit on our requirements and we started developing the new amazing board. We developed the schematics and did the prototypes, and then we needed to bring up the board and it was a pain. We made a lot of bad decisions. The first one that we made is to change the memory and do not use the same that was using the evaluation board, and it was a pain. Even when we tried to put that board, even the board water didn't start. But although we suffered a lot, we also learned a lot, and at the end we succeeded. Yes, we succeeded six months later, but we succeeded. Since then I did the bring up of different boards, and this talk is just my experience trying to explain a little bit how to bring up these boards a bit better. This is a little bit the agenda. First, I want to introduce the concept of the board bring up for those that know nothing about this concept. Then I will try to explain a little bit the basic process and the steps that need to be done to develop a new board. I will try to explain what's wrong with this process and why even can kill a project. And then I will try to explain my experience on how software can help us with the with the different hardware phase to get better a bring up. And finally some conclusions and lessons learned, tips and tricks that I hope that will be helpful. And at the end I will be really happy to hear about your bring up experience if someone wants to explain a little bit. So board bring up is the process of validating both electrically and functionality a new circuit board design including the porting of a boot firmware and the development of a board support package. This process can take sometimes a lot of time and also a product never gets to the to the market because the development files in this phase. So board bring up is a very important step in the development cycle of a new product. In this process usually there are two different teams involved. On one side there is the hardware team and on the other side there is the software team. Continuing with the bring up process of a new board the process starts to the process to create a new board starts with the hardware team. The first step is select the the various integrated circuits, sensors, displays, connectors and all the components needed to ensure that the design is working and fits well with the specifications and with the target retail price. Then the second step is to create the schematic design or the diagram. The schematic shows how every component from microprocessor to sample resistors connects together. Once the schematic is done we need to create the design for the printed circuit board also called the PCB which is the physical board that holds and connects all the electronic components. After that you will create the bill of materials and this is usually created by the schematic design software and the bill of material is the part number, the quantity and all the component specifications. Then you have to create the electronic prototypes. This is usually a two-step process. The first step produces the very printed circuit board and the second step is having all the electronic components soldered onto the board. Once the prototypes are done the hardware team needs to verify that the board has been assembled correctly. There are a number of tools that can help to perform the assembly. For example optical inspection but this is not used in the prototype space. It's used when you go to production. And finally we need to do the basic hardware test to ensure that all the devices on the board are operational and this involves the test the power rails, the clocks, the basic functionality connectivity and what happens here is that usually the hardware team does a very basic test and gives the board to the software team to finish the test. On the software side the software team needs to cover different parts. Some components of the board that have specific functionality might require a firmware programming so a firmware development which means that it has some cost. Then we'll probably need to port the bootloader to the new board and take care of the memory initialization, do a basic board initialization and set up a way to be able to see what is happening. For example set up just a serial port of the book. Then starts the programming of the kernel drivers. If your board is based on a vsp maybe you have some drivers that are already developed but if not you need to develop the drivers that are missing. Then there is the software stack so you need to choose the right software stack for your project and of course there are a lot of reasons you may choose one stack or over another but might be the appropriate for your requirements and finally there is the application development that needs to cover all the user and customer requirements. So let me explain a little bit what's the problem with this process. So the problem or a first problem with this process is that the development is linear so that means that goes from hardware to software. So first the hardware team selects the components that they think will be good and this is the schematic. They assemble the prototypes and does some basic electrical testing and then gives the boards to the software team. The software team can start from zero or not it depends but it's most likely that they start based on a board support package provided by the vendor. Then you will need to start porting the bootloader and after that the kernel and maybe do some firmware modifications and once the kernel runs the software team starts testing that all the board the whole board is functional but at some point what happens is that the software team discovers a hardware back. If it's easy to solve this hardware back maybe with a small modification with a small hardware modification the software team can continue with the testing doesn't matter. If the hardware back can be fixed with a modification of the board we will need to a prototype respin that fix the problem and make another prototype which means more cost. So on every respin you lose money and generally the planet two or three prototypes spins always ended up with four five or even more respins because you always in the software team always found or finds problems with the second prototype that they didn't catch with the first and find problems with the third prototype that they didn't catch before because chasing these kind of bugs can be time consuming and cause missing deadlines and even kill a project. Also when the software team finds a back sometimes you are in the fuzzy lane between the hardware and the software and you don't really know if it's a hardware problem or a software problem. Sometimes the hardware team and the software team discuss about the problem and with the responsible and gets angry and well at the end it doesn't matter who is the responsible the important thing is is catch as much as possible errors with less respins. So catch bugs like catch bugs here is like the the chicken egg dilemma because you need to you need the hardware to test to test it but most tests are done by the software and your tests are progressing while while you are testing the the hardware. So actually the well the number of respins of course depends a little bit on of the hardware complexity and as I told you before what you really want is catch as much as possible bugs in a prototype and avoid as much as possible respins. So as a consequence you lose less money and spend less time. It's for that that one thing that is not very usual but I think that will be good to introduce is that your hardware team and your software team must work together and when I say that they must go work together I would say that they must work together from the beginning because there is a lot of work that the software team can do when the hardware team is under development of a new war. So let's see how the software team can help the hardware team and let's start how software can help us with the the component selection phase. So basically here you must select the main processor the and the other integrated circuits for the main processor you can evaluate the the board support pack the board support package is the first thing to do but it's also interesting evaluate the main line support of this processor this can show you if the vendor is doing things well or or not. If the vendor doesn't take care of the software that's a bad notice for you and trust me you you will have problems in in the future being forced to to use an all or deprecated vsp for the other integrated circuits check that the the driver is is always upstream and if the driver is not upstream but it's in your vsp take care because can be free can be full of horrible hacks and finally and as an extra note I saw a lot of hardware engineers use lots of microcontrollers so my recommendation is don't overuse microcontrollers because microcontrollers will need a kernel driver and a firmware and sometimes there is a big software cost here and that's a little bit an example of of a vendor kernel in this case is TI which has a policy that tries to to upstream things they release a vsp version based on on a long-term kernel as you can see in this graph the number of commits and file changing between the the vsp release and the stable kernel is reduced every time they release a new vsp so you need to consider that not all the processors are supported on on all the vsp's but this indicates a attendance so take a little bit these numbers with with cauchon uh with the schematic how how software can help us with the schematic is a very is a very good practice let the the software team review your your schematic you you will be surprised at the the the bugs that you can find just doing the the device 3d file for example yes i know that this is a a little bit dta specific but it's it's also a good practice uh take a special attention with with the pin control settings and the state of these pins when you go to to suspend to ram for example uh also taking consideration the the enable of the integrated circuits and and the regulators and it's the full state in this is by revising the schematic using the device 3d for example you can also catch crosser lines for example crosser transmit and receive signals for urs or spi and also is a good thing have a list of all the device all the device addresses of a of a boss to to avoid avoid conflicts and this is a this is an example to try to explain that the hardware decisions has direct influence to the software in this case here the problem is that we need to control two displays but the the sock is only is only capable to handle one the first prototype implemented a solution that was put in the middle between the sock and the displays a microcontroller that acts like like a bridge the sock uh first selects which display to send the video data sends the data and the microcontroller sends the video data to the to the corresponding display in this case was implemented all via i2c which means is a bit slow for for video data and from software point of view that means that uh we have the application that needs to write to the device to the video device zero or one we we need to create also a specific kernel driver that talks with a microcontroller the kernel driver can be the same but you need two two instance of the driver and finally you need to develop also a custom firmware for for the microcontroller when when we receive it the the board with with this with this solution we did some software effort to make it run properly in the kernel site and in the application site but didn't work well because the the performance was was was really bad so we worked with the hardware team and decided to to change this and then implemented another solution uh that's in this case there is a FPGA between the sock and the and the displays and the idea is that the sock sends twice the width per the height and the FPGA just splits the the data and sends it to the proper display from the software point of view we needed to create the the bit stream of the FPGA to split between the two displays but the kernel driver in this case is is the display controller of the sock so the only thing that we needed here is adjust the the timings so from the application point of view and the only thing that needs to know is that if it writes to the first half of the of the height resolution will go to the first display but if it writes to the second half of the height it will go to the second display uh that solution had less development cost and worked a lot better in the next phase it which is the the bill of materials uh yeah to be honest i'm not sure i'm not sure how the software team can help here so i'm open to your suggestions uh the only thing that comes to to my mind is that sometimes an alternative part number might have different software support um as you example is when the identifier of the device is not recognized by by the driver but this is a simple feature because you need only to add the identifier in the in the driver but i also i remember i remember some some alnands that were pin to pin compatible but the driver was was was really different so needed a different driver uh in this in the next phase which is the the the PCB prototypes uh you in this phase you can start planning your test so prepare a stress test and performance test for every device that that you can think is in your board prepare outs to the connectivity test and how you will connect to to your board via serial port via ethernet etc and think a functional test for every device in the board a functional a functional for example a functional test for your real time clock or a functional test for for a light sensor think also uh about how you can test this you can develop a software to test this or you can simply add some test points and check that some signals are are right and and finally the the big question here is if you do automated or manual testing of course yeah automatic testing is is better but sometimes you can do to test the the hardware you can do automated automated testing or you need to to develop extra hardware to do this and finally there is with assembly verification a step it's important to do the stress test and the throughput test test the RAM the non-volatile memory for example the the different buses at the maximum speed and it's important don't forget to test everything everything that is wired in your board so everything that is in the schematic even the driver is not using it for example you can have a chip that has a wired IR cool but to the shock but for some reason the driver does not use it but even though uh think a test for it and test it because maybe you you will need this later uh yeah and finally in this phase you can you can do more high-level tests functional testing against the the business requirements of of the of the application run user acceptance testing to check that the the the solution works for the the an user and do regression testing every time a hardware modification is done because might solve the problem that you are working might might solve the problem that something is working but you need a you do a modification and then you you break the one thing that you are already tested some some conclusions well to to both successfully for the first time with mainline there is there is no secret because what you only need is that your hardware must be supported in mainline so for that selecting the the right components when you design your new hardware is is very is very important and one of the recommendations that i can do i i suggest to do is is up follow the rule well upstream fierce even before you have the the fierce prototypes for example you can have you can use evaluation boards and and a breadboard with a specific device to develop the driver before you get the the real prototypes that will really help you help you later and finally some some three three lessons to learn it yeah in my experience review the schematic from the software point of view can help you you to catch a lot of errors sometimes in the review of the schematic we found fortunately a lot lots of errors another thing is try to develop the software as much as possible even before you have the fierce prototypes uh even i recommend to develop the of course the the drivers that are on your board but also the the final the final application that because that gives you a good overall of of what do you need to to exactly test um and important is that before doing a hardware spin which has some cost always run all the test again and test for for the the different regressions because fixing fixing something can break something that you already fix it and well that's all it's a bit of of my experience trying to to bring up different boards in different especially for small companies that that doesn't have a lot of a lot of resources but that just looking at at the software fears can help or we can help them to to to develop a better hardware and have the possibility to run to remain line or an extreme version more likely and well that's all thank you very much