 Hello, and thanks for attending our talk on from zero to AB swimming upstream the after project barebox and rock So for future watchers of this video. We have put here some abstract that we had submitted A little about us. My name is Ahmed Fattoum. I am holding this lecture with my colleague Roland Heba We are embedded Linux developers with pingotronics We are a German Linux consulting company and we help customers with deploying Linux on their embedded systems We do driver development kernel development bootloader development. We do system integration for all these low-level system Operating system components and we also do a lot of graphic infrastructure work. So across the stack from the kernel into the user space And yeah, some customers say start with us new projects Some others come to us later in the product life cycle and as such We see a great variety of systems each with its own history And many of these systems use downstream BSPs So downstream PSPs are BSPs often provided by the silicon vendor or by a module vendor and They are quite attractive because they work out of the box on the evaluation hardware You don't have any hardware it so you want something to try and run demos on they come with demo with demos They are hopefully supported by the vendor for some time into the future and it's usually available earlier So like these two cups here that are hunting downstream. They will catch some young fish because the old fish is not upstream yet But priorities will often diverge so the vendor will shift focus on your hardware They already started with an old PSP along with the hardware development and by the time they release the SOC The PSP is already a bit outdated. They will keep updating it best case But now your hardware comes and the priorities shift, but you have your embedded product and you need to update so Yeah, your product the priorities diverge and now you are on your own and you want to do an update So it can be a lot of reasons for that. You want to patch security issues or you just playing bug fixes You want to interoperate with new devices So these embedded systems can have quite like quite long life spans like 10 15 years And you will want to interface with new devices and protocols which while you Implements the protocols they need quick handling because they don't implement the protocol exactly and Yeah components will get discontinued. You will need to support new components. They are not software compatible And yeah, you just want to support new features. So are no tasks with doing an update So in isolation, how would you update a downstream component? Normally you would start by defining test cases for update verification So you can verify that your update didn't break the stuff that you depend on then you start Looking at these patches that separate you from the versions that you were forked off from and Then you look through some patches you can drop because you did everything, right? You upstream that it's upstream functionality You don't need that patch on your side anymore Some are rebased because well, you didn't have enough time So you rebase the patch perhaps give it some love send it out to upstream so in the next update iteration You can get rid of it But some you need to rework because you depend it on functionality. That's not there anymore And yeah, that takes a lot of effort and at worst case must completely replace it because the API is you are you used are not there anymore or you were depending on behavior that wasn't guaranteed to you But still it's quite straightforward what to do you test debug fix and keep iterating on that But then come real-world complications as is usual with the real world for one you have knowledge loss so people leave the company and this downstream this basing on downstream BSPs invites like Development workflow where you don't write good documentation because you know what the problem is your reviewer knows What's the problem is you don't need to explain it to somebody else and then you end up with patches like Optimize optimize locking and it just deletes a spin lock out of the corner And now you are tasked with updating that and the colleague isn't there anymore and now you wonder What does that mean? So what was the problem? How can I reproduce this problem? Did she did they check that this that this is a safe thing to do just delete two locking lines out of the corner you don't know that and it's a two-line patch, but it takes considerable time and That's a good side of the knowledge loss because you know there is a problem Then you have these implicit assumptions that you don't know about and that will eventually break because you depended on something behaving this undocumented way or having this specific order and This brings you to technical depth so like this bear trap Canyon You have this technical depth lying around just out there to get you and it's not just limited to your own patches It's also the patches of the vendor where you are now the maintainer because you want to update them So if you have 200 patches, that's a lot for you now you go to the main You need to look at the patches of the maintainer and you have 5000 and Yes, there are fixes there that will go away when moving to a new versions. So our features there because yeah Sock support for example that you didn't have before and there are reverts in there because They also just have limited time and then when they moved to the old kernel version They said, okay, we will just revert these upstream changes and replay our old state And we will get the same result and it's a problem for future them but future them is now now you because you are the one maintaining this and Especially Harry our custom interfaces to use their space because it takes time to upstream stuff and for good reasons So for example in upstream we have infrastructure like DMA buffs Where in a G-streamer pipeline for example, you can pass around just a handle to describe Memory memory range in colonel space that can be moved between devices So you save memory copies, which is very important on embedded systems that just have not said much resources You have this infrastructure in the colonel, but it wasn't there yet when the vendor wrote his software and what did the vendor do? Well, they just pass around physical addresses and now your Video decoding pipeline has root access and it's an end safe thing. It's an unsecure thing It's a very bad idea and you are now married to this interface because it's it's just like a cancer everywhere You have to use special G-streamer elements. You have maybe to use a special sink You need to write your application against that which brings us to the last point You have some sort of soft vendor login. So it's open source software. You can see the sources, but you are still locked in into this ecosystem and It works as long you have these approved use cases which most customers of the vendor have and which for some time They will keep maintaining but once you go away from that and try to do other things you run into other problems So for example, I had a problem where on an older Raspberry Pi colonel for now for nine I think the system was locking up after a few weeks and Yeah, you want to debug that normally you would just write a bug report to the pre-empt RT developers and tell them Yeah, I have a problem here and that's how I reproduce it But you are not using upstream by fire. It's completely different colonel that you are using. It's old It has vendor patches on top says use patches on top and the problem is probably in your part that they won't maintain for you So where will you go the Raspberry Pi foundation? They do good work. For example on Unlike lip camera, but preempt RT is not and priority to them and it wasn't something they did and Yeah, so you just had host a branch for the preempt RT integration So despite it being a big ecosystem with many people using it that Concrete use case of yours is not covered. We're very well And now it costs you a lot of effort to try to understand why this happens Spoiler it was the fast interrupts in their downstream USB driver You turn it off, but then it's too slow and you can't just move to upstream to try upstream had great rework in that area that you can't go to upstream and compare what's the performance now and now you are in a very Unfortunate situation because elsewhere you are using all these Raspberry Pi specific APIs and you yeah And you just can't use these normal API that you use on the desktop So you so you would benefit greatly from the rest using the same stuff you use on the desktop on your embedded system Yes, it needs more work at first, but it pays off in the longer in the long run So well, what what if you had a clean slate if you were back at the drawing board and You are doing a new product What could you do differently to not find yourself in this situation that you start with something and then down the line and Down the line is quite a long line when you have 15 years So you don't want to end up there. What can you do? You should choose hardware that can be supported to it So modern SOCs have such a degree of complexity that you need many thousands of Pages just to describe the peripherals not even the CPU and it would and it's not even enough then and there will be missing parts But at least that gives you a chance to support a system for that long Because knowledge loss doesn't just affect you it affects the vendor as well They lose source code all the time people go away and if you are left without documentation It's not a nice thing, but that's just one part of the equation If if you have documentation is there upstream engagement around the platform If it's attractive to you, it should be attractive to other people as well So yeah, look at look upstream look at the mailing lists Do people what do people write about this platform? Is your board supported main line? Is there a device tree for it? If not the board is a device tree for your SOC. What kind of devices are written there? Are there drivers for this? If someone reports a buck who yeah, does he gets does they get replies to their issue? And is the vendor themselves engaged upstream? so do they want to have a sustainable model where they upstream stuff and or do they just live in their downstream world with 6,000 patches on top and yeah with great efforts they do an update every two years and that's this kind of stuff you want to look at and There are actually alternatives that you can use an upstream kernel on and not miss out on too much Here are some examples for on NXP the IMX and the layerscape series you must Yeah, marketing speak in fact set a bit So the IMX 8 QM and the IMX 8 MQ there are completely different SOCs completely different upstream support and Watch out for that. So search actually for that what you want to use From TI like these this SOC family where the Beagle bonus from from microchip rock chip These are SOCs that we that I had Some experience with at work They work well you have documentation for them. So is upstream support It's not of course a complete list, but I can just talk about the things I know about And that's one part the other part is you choose hardware to use upstream solution with yeah Use upstream solutions to that's the point developed based on the mainline kernel instead of the vendor fork try to use The interfaces that are provided to you by the mainline kernel and avoid these board specific one-offs for yeah We are embedded every one is special, but we share a lot of stuff And there is no need to keep reinventing the wheel and have Stuff that's only specific to us which we could use in a future project or someone else could use it Try to upstream that stuff not everything at once It's normal that you have pastures, but at least have pastures where they are Documented where other people can work with which in your update iterations can be part by part be smaller Yeah, and that's yeah collaborate with a community. So to sum up You have this juicy upstream fish, that's the one you want and Yeah, it's harder at first you have these upstream currents of sorts Which I can imagine are harder to hunt in than the downstream currents But you saw the two cups at first they didn't look very full It's a big burst hunting upstream they have more of that juicy upstream goodies and Yes, that's where you want to go in series. It sounds all nice, but yeah, how do you go? Previously you had this downstream BSP which had all things integrated, but now if you want to use mainline Probably you don't have a BSP where everything is there So my colleague Roland will now tell you a bit about how it could look like to integrate such an upstream based BSP and yeah, we will use yokto for that rogue and bear box and That will be his part. Oh Well, well, well hello Yeah, I'm gonna introduce me already We are gonna look at a clean plate and how can this look like so you have nothing you start from scratch you want to end up somewhere with a functional system that can be updated and in this example, we're using the yokto project as a BSP and built on it using rock as an update framework which is failsafe and can image you can do image-based updating with it and on under the hood we're using bear box for as a boot loader and also as we as we will see as Integration with the update framework So the board we're looking at is From one of our customers, it's you can you can buy it if you want if you Talk to intercom. They will tell it to you. So it's readily available and on the market. It's Yeah, it's a song with an I mix 8m processor with a bit of rum and a bit of Ememcy on top of it on the song. Yeah, and We will look at our example BSP that we just coupled together You can have a look at it on our git on our github page there If you want to dive deeper into it, but I will also share a lot of the source code on the slides here But let's first look at the architecture We have some kind of boot loader In our case as I said, it's bear box It has this Do we have something like a pointer here? I don't know I don't know I'll just use this Sorry for the people on the stream that won't see my laser pointer. Yeah, it has some kind of safe information that is used to Switch to one of two system partitions so In one system Partition there is a full root of s and in the other there's also a full root of s and we are using one In the one partition. We are using rock install to install an image into the inactive partition and If we then boot into the other partition by evaluating this state in the boot loader Which rock uses to switch the active partition then We can boot into the new partition and if everything works well some if the system comes up basically and you have Some sort of I don't know system deservice and our case. It's usually that calls this rock status market command and This basically cements this this boot loader switch. So the boot loader knows that this partition is can be booted and There's nothing wrong with it so This Babers day. There's a lot of code here now but all I want to have you look at is that it's basically a divestree blob and You have a state node here and a SD partition here Which you can use as a back end for your oh, I have nice Arrows for that. So the state partition refers to the sorry this state node refers to the state partition as a back end and The rest of this is basically the variables that you want to have in this day So we have one variable for the remaining attempts of the system partition one for the priority of the system partition and The same for the other system partition, which I have spared you here because it didn't fit on the slide And then we have a flag for a Babers to know which partition was recently booted So it can know what to switch to or in case of user space Rock knows which partition it needs to write on to And then you have this alias node. So you bless this divestree node as a state node From user space it looks like This you can use Babox states util from our DT utils to dump the state and Yeah, this is on top here is the Babox side and on the bottom is the user space side So you basically see this same variables Okay, so we need port support for Babox and Linux. We need a device tree with Babox state node We need on the other hand in our BSP two layers one machine layer with an image with two partitions Babox recipe and a Linux recipe and a small very small distro layer Which contains our rock recipes and a bundle? So you start from scratch, right? So what do we do? Well, you Just build on pokey. That's the easiest thing to do instead of using some pre-configured thing Which you don't understand which has 5,000 patches on top of it. So you just check out pokey you Grab your meter layers as for example get sub modules For this case here, we really need the open embedded core meter layer the meter PTX layer which contains the DT utils and the Babox and Then we also need a meter rock for rock support Yeah, and then you just source your built-in and add your layers to be the layers The board support layer is Very simple you can just if you don't know it you can just use a bit fake layer tool to create it And it will create you a nice reasonable nice recipe's example to which you can just delete But yeah, that's just a quick reference slide if you didn't know that and so we need some kind of machine and On this slide here. I've shown you a minimal set of variables that you need to Yeah, need to set for your machine to be workable. So you said basically here this this first block is you would define which tool chain you're using which So see and therefore processor you're compiling for you can just use this predefined include files from pokey and Set the right variables. So these includes to the right thing Then you have these machine features. Yeah, you can Pre-define basically a set of software that lands in your device in your image on your device and Yeah, on the end we also have an image configuration file, which we'll later use to Define our two partitions and Continued we will just set our preferred kernels and DTB and bootlada packages we also need to define on the dependencies for our bootlada as we will see later on the next slide and Yeah, that's basically everything so You Were using this nice vendor BSP, which does everything for you As it turns out our I mix 8m needs some kind of food firmware to start and trusted firmware To In this case the trusted firmware basically starts the processor on the I mix 8m and The firmware is used to train the DDR for link training and these are Basically, yeah downstream binary blobs that you just have to use There's currently no alternative for it It's a questionable if you want to build an open source alternative for it or if you even can so Yeah, you are stuck with something from the vendor BSP But it's only basically these two recipes that you can Fork from the meter free-scale layer and it has also the advantage that you can control Basically stay on an old version if the new version version doesn't work for you yet Yeah, so our barebox recipe We have on the left here I've printed this state of the Of the directory what is which is already what is all in there? We have these BB appends here which we use to Basically Put our patch in our single patch So yeah, the BB append just appends this patch to the source your eye and Well, you need this patch. This is From the mailing list a colleague of mine did the work before supporting this this piece this stock upstream If you didn't know you can just use before on the URL it's a Yeah, it's a lower instance like the kernel users and Most of the board code is basically auto generated and really really simple boilerplate so you can if you are using this yourself if you are doing this yourself you can really You can Just use the boards that are already supported in barebox and yeah use them as boilerplate And we have some environment variables too You can just put them in the device in the directory here and the barebox recipe will pick them up Here we are just copying this firmware image to our barebox Devastries so barebox can pick it up and start the machine Then we need a device tree to The device tree was already Already existing we had a device tree for for this board so the device tree the device tree Recipe for that is really simple. You just Include or inherit the device tree BB class and everything works as intended You Have a kernel too as I'm it already said everything is already supported mainline. We have a device tree already So the Linux kernel can just use the device tree and initialize our hardware. Yeah, and Therefore our kernel recipe is really simple too. It just inherits the kernel BB class sets the version Yeah, and Sets the compatible machine. So Yachter knows which machine it is completely perform Okay, so we basically have Half of it. I think we have a barebox recipe Linux recipe our board support is there and the device tree with Barok state node so let's see the Image with a be partitioning partitioning is also really simple You just have two partitions in this conflict file. Yeah That's it. So we need rock support to We are using meta rock which has already has the recipes for rock itself and a bit of support for yeah copying the system conflict you see and There is also this meta rock community layer which we Have on our GitHub Organization which contains support for other platforms as well So if you're using this for your new book for your new port, then you can just use it as an example And see what you need to do in this example, we want to do it ourselves, so we will see what to do and we're basically creating a new layer as I said before you can use the big big layers tool and Really minimal distrocon for it. So you Set the distro name and the distro version and also Use rock as the distro features Okay, we need a bundle to bundle recipe is also fairly simple You inherit the bundle class that comes from meta rock Then you define basically your your slots which you want to have in this case We only have two slots which are both named root if s so root if s a root if s1 and root if s2 are both Kind of the same image. So you have these root if s as a rock bundle slot And you define which image you want to put in these slots in this case it's just the usual core image minimal from pokey and This next part is Configuration for our cryptography There is a nice script which I Think I saw had on one slide, but I don't know where it is. Well If you know open SSL you can generate these on the fly, but there's also a script in the meta rock community GitHub which generates these keys for you and of course you don't want to put your private key into git But that's just what I did here to show you how to do it Right, and then we have our system con which basically Defines what we want to have which lots we want to have which key ring we want to check against if we want to update the the image so Rock won't install anything yet It will only install things that are signed by the right Yeah, by the right search basically Yeah, and then you say where your slot lives slot a lives and where your slot B lives in Which block device? Yeah so we Basically have everything we need we can do it big update bundle and be very happy about it and I think that's all we have other questions, I think we have a Second micro Well, sorry Why not include the IMX layer because potentially you might need to change the arm trusted firmware, which we've had to do Yeah, we don't use I am x recipe for as I am x ATF recipe from the meta free-scale layer That's a downstream trusted firmware. We use the upstream one. So it's not actually a blob It's built from the upstream repository and the meta free-scale layer wouldn't help us there So the only dependency we have is just the firmware and the meta free-scale layer has Questionable stuff they do in their layer conf with like sock overrides and it's just a dependency I don't want for a single recipe. Yes, it's something It's one thing you need to take care of when you do updates But I would rather import one recipe that just downloads Shades, it's basically a shared script installer that's downloaded. It doesn't build anything from source this You can change the source for we've had to we've had to patch. No, I I am not talking about the ATF I am talking about the firmware mix. That's the only thing that we imported from the meter free-scale layer So trusted firmware we built from the upstream repository not from the next piece sources And for secure boot Does barebox support HAB? Yes, the box supports HAB. I have seen it running It's on the IMX 6 platforms on the IMX 8mq. I haven't myself did it done it on an IMX 8mm But your infrastructure there Thank you, and it's integrated into barebox. You don't actually have to mess with it in the BSP. It's nice and as I said previously You sometimes you want to be able to control what goes into your image and which firmware you are actually using so for example in in one of our customers case we had The case that their user space application didn't work with the new firmware with the new other the ATF firmware or the new DDR firmware image, firmware blob so when you when we were updating from An old BSP to the new Kyrgyzstan release they changed the firmware to a new version and our user Their user space application broke. So yeah, we basically ended up doing the same thing forking the recipes to a layer. Yeah Any other questions? Well, that's it. I thank you for being here and yeah check out the Github repository give it a try It's not complete. We hacked it here in the speaker lounge a few rooms Next to this room. So feel free to send pull requests or if you have questions, just contact us. Thank you. Thank you very much