 So this is being recorded today and in fact it's being recorded in two fashions, it's not only being recorded by scale but we're also recording it for the project itself. So please if you are asking questions can you put up your hand or otherwise indicate that you'd like to get the mic because we need to get the audio in order for questions not to sound weird like the speaker is basically talking thin air. So please ask for the mic for questions. Remind me, off the cover. Great, thanks Behan. Thanks all for coming. Normally twice a year there is a Linux Foundation sponsored embedded Linux conference that happens usually in the spring and in the fall and this year for various reasons the spring one was pushed back to the end of August so there was sort of a gap usually these sorts of talks are given at the embedded Linux conference and so there was a bit of a gap the open embedded developers like to get together and there was no sort of obvious opportunity to do that in the spring and most developers would have a hard time going to conference at the end of August and then another one in October so we thought scales the timing for scale is perfect the location is perfect so we thought we would have an inaugural open embedded summit give it a whirl here and see how it how it how it goes anyway my name is Trevor I'll be talking about using open embedded I work for Togan Labs I'm a senior software developer there we specialize in open embedded but we do general embedded Linux consulting we have a strong open source focus and so I guess the first question I ask that people will ask me is why would I use open embedded what is it open embedded as a tool that allows you to create your own embedded Linux distribution so why would you want to create your own Linux embedded distributions well for a casual user I don't really have a good answer for that I have no way of answering that in a way that's going to satisfy anyone but certainly if you're interested in distributions you know if it's a hobby or something but certainly if you're putting together products for a company if you're creating a line of products instead of just creating images transitioning over to distributions is something that at some point you will probably end up discovering and the great thing is that when you discover this you'll find that there's already a tool that's going to help you along having a distribution allows you to have tight control over the installed software across all your products so you're not just targeting one image for one device and one release you're looking at the lifetime of the product lifetime of your company in fact and and it allows you of course to so if you have multiple devices because over the lifetime so sees don't last the way they used to I mean the 8051 been around since the 70s it's still actively being produced whereas so many of our more modern SOC's you know they have a two-year lifespan so if you're trying to put out a product for 15 years and support it for a long time you're almost never going to find anything modernish that's still going to be available your your product is going to live on multiple devices that's a fact and so having an idea instead of just creating images oh here's a release here's a release having a distribution where you're controlling that over all the products even something like raspberry pi we think of raspberry pi perhaps as one thing but raspberry pi is two or three things there's the original pi ones and the pi zeros which are arm 11's their older technology and then of course you've got the latest and greatest raspberry pi 3 which is a modern a 53 cortex a 53 so 64 bit versus 32 bit I mean vastly different devices under the under the covers but of course you want to try and treat them as sort of one one idea having a distribution also when you get into distributions you're probably creating packages and package feeds and that allows you to do security you can have somewhere set up on the internet a server serving packages and all your devices can phone home essentially and update themselves so thinking more in terms of distributions as opposed to just single releases is something that will hit you at some point if you're thinking of long devices long-lived devices so how would you put together a distribution how would you put together an image there are in the open embedded world in the new linux ecosystem there are a lot of code choices and there are a lot of code choices that every step you know you press the power button and there's a dozen things or more several dozen things that happen many pieces of software that got run before you see that command line come up you've got bootloaders and bootloaders from the SOC and then bootloaders that you install you got multi-stage bootloaders and then the kernel comes up and then the last thing the kernel does is run an init system that's the only thing it does it runs an init system at the end and that's the init system that spawns everything else if you're doing tech space stuff then you'll have some sort of login system and then once you've logged in then you have a shell if you're doing graphics stuff then you have a display server protocol and then on top of that is a display manager and so and then a window manager and a desktop environment just for clarity if you just say a linux system then technically all you're talking about is this one component of this multi faceted environment so it's always nicer and safer to say new linux instead of just linux because sometimes when you speak to a technical person you say linux then they'll often spend the whole conversation thinking you're talking about this one thing when you actually mean everything but anyway for bootloaders I mean half a dozen choices there and then even choosing a kernel do you want to use the latest and greatest from kernel.org maybe the vendor of your device has their own evil vendor kernel or if you want to be more politically correct maybe it's a benevolent distro kernel do you want to use preempt RT or is NMI do you want to use something based on I should update these for five now but do you want to use a five or do you want to use a two six no but it exists do you want to use a LTS a long term or you know they're just choosing a kernel is an exercise in itself in its systems I mean there's an explosion of in its systems nowadays display servers do you want to stick with X or do you want to use Wayland or a mere or direct SB I mean it's dying but a lot of embedded devices love it still display managers for logging in GDM XDM KDM I mean there's dozens of those and then the environment gnome KDE and then based on that you can have whatever window managers you want and then that that's just you know the start I mean there's still shells to choose there's term while terminal I'm in there's which are kind of the same thing there is you know a dozen web servers you can choose from there's browsers there's C libraries which underline your whole system SSH servers you can choose to compile your entire image with GCC or you can use LLVM for the basic command line utilities you can use core utilities which are what you have in US of LS cat more all that sort of stuff or you can use busy box or toy box you know there's you have your choice of LS you know this is a great world we live in you know numerous databases to choose from languages your developers can write and go Rust CC plus us Pearl Python Python 2 Python 3 no JS Earl Lang Ada there's virtualization there's all sorts of security stuff you can add there's all sorts of versions of these they don't all play well together sometimes you need you know grub two of this and whatever version three of that because you know two and two don't work together or something and then of course overlaid on top of all this you got dependencies they don't you know you pretty much have to stay if you if you want to use something from five years ago it's best if everything you're using is from five years ago it's hard to use a modern component with old stuff and vice versa so choosing wading through all of this is not trivial obviously and it doesn't it doesn't all work you can't just go and pick and choose whatever you want and so as an aside just a very quick aside you might be thinking well why do we have all this isn't this horrible this is you know it's not code forks will happen due to disagreements sometimes they're technical sometimes they're legal you know we wouldn't well we probably wouldn't have known today if it wasn't for the original licensing of the KD system you know sometimes the leadership there's a disagreement in leadership or direction or the community you know sometimes you want to redesign something I mean Python 3 Python 2 they wanted to experiment with stuff but they didn't want to drop to so it wasn't a natural progression it was within its own project they forked and yeah because they wanted to fundamentally change how various things work and and and this is a good thing I mean back in the day we're going to be talking about cross-compiling it's what we need to do in the embedded world a lot and back in the day just trying to get a cross-compiler was a major I mean I've taken multiple days you couldn't just pick any version of GCC and hope that you could target your device back then developers had to work a long time on one version get it to work and then in the community you would know if you were targeting a PowerPC 405 that you would go to this website this area and you would get this specific GCC 27 with these patches and that's the only way that you could work with that well if you didn't know that you know if you were trying to use a more modern or an older it just wouldn't work I think it's safe to say that GCC now does an excellent job but I also think it's safe to say I don't know if they would have focused so much there was a time where they officially and literally said we don't care about cross-compiling we are so busy with trying to get GCC to work and all the new versions of C and C++ and getting all this supported and all the front-end portraits and we just don't have time to worry about then LLVM came out and whoa all of a sudden GCC does a great job of cross-compiling because that was one of the things that LLVM decided that they were going to so competition is good sometimes you're not aware of existing projects I mean this happens a lot maybe with younger people perhaps you know they get all excited they learn about this new Linux system and they're like I'm gonna write my own window manager because I don't like known well there's a thousand to choose from why not just choose one of those you know what happens a lot as an original project will get abandoned that had it happened to me many times and many people have this experience you're using some old piece of software you're still using it the original developers long since moved on they're not working at the same company anymore it doesn't work with the latest GCC you create a patch you send it upstream you send a push request pull request on github you never hear back from that person so you fork and you move on and if you are active enough then suddenly you become the new project for whatever that is sometimes it's just good to start fresh and if you want to learn anything if you want to learn how bootloaders work write your own boot loader I mean you know it doesn't have to be popular but even if it never succeeds you'll learn a lot anyway so there's always good reason to do multiple things and to have multiple things going on but anyway that was an aside getting back to you know versions and all the different stuff to choose from and trying to create a distribution as we all know the new Linux system is not controlled by some one place or any company or organization it's all a bunch of individuals working mostly autonomously there's no overall direction for the project people do whatever they want and one simply doesn't slap together cohesive working set by accident it's quite the opposite getting all that stuff to work together is non-trivial and and is something that is planned I guess the main issue summing up is that there is no one correct way to write open-source software or software in general developers choose what languages they like they choose what repot well hopefully they choose a repository system but you know they'll choose whatever repository system they'll store it on github git lab bit bucket wherever they want and they'll choose their own licensing and Linus's rant aside there's still I don't think are enough developers considering cross compilation it certainly got a lot better it's certainly not perfect there are a lot of developers they write something on their x86 machine they compile it for it they run it on their x86 machine it works great and they move on but when you try and compile it for a mip64 little endian well you know that's where things go south and and sadly they sometimes don't care about that but so the solution is to if you are creating images then there are embedded Linux build systems of which I think open embedded is great there are others but please you know it if open embedded is not your bag that's fine please use one of them please don't roll your own embedded development is becoming well is very popular I mean they're more and more jobs more and more companies more and more people more and more hobbyists more and more people are doing embedded development and if we all go in our own directions it's just not a good thing it's not good for you if every company in the world that was doing embedded stuff focused on these two or three different projects then you know companies could hire people easier because they could hire someone and say hey do you have open-bed experience because I will be using yeah of course I've been using that for 10 years no problem and and the more people that work with a project the better it gets you're gonna be using it in some way that no one else is gonna be using it and this other person's gonna use it in their own special way and if we all contribute to few projects then they all move forward so getting code to run of course requires cross compilation even if you're running something interpreted you know that interpreted thing has to be interpreted by something and that has to be cross compiled so at the end of the day you're cross compiling you can't get away with it so how do you compile code for some other device well you have to fetch it you have to unpack it you often have to patch it you configure it you build it and then you stage it and package it fetching it's not always obvious where you get your code from or from where you get your code many projects like Firefox or Maria DB all these things they have website they have cheerleaders they have communities they have IRC channels we all know where to get that stuff Firefox that are there there's my source that's great in fact they'll even provide in many cases pre-compiled sources so you just grab the Raspberry Pi version that they've already compiled for you but where do you get LS where's the website where you know the LS is stored where's the cat or more you know there are hundreds of little utilities that we rely on every day that are almost too small to have their own you know cheerleaders their own websites so where do you get all this stuff I mean just finding the stuff there are many projects that are core you use them a hundred times every day and they've essentially been abandoned and to find the source code for them you have to go to Red Hat or Ubuntu or Debian and each one of these projects sort of has their own version of of these sources because no one's actually sort of coordinating it's not really enough effort there's not really enough interest in sort of having its own one place and it hasn't really changed in many years and so just finding source code and then of course once you find it then you have to know how to get it you know is it stored in CVS or Mercurial or Git or whatever so oh right I used CVS 30 years ago how do I CVS something again I gotta look that up and then yeah interacting with the site there might be authentication required and then how do you retrieve a branch or a tag and then verifying checksums of tar balls and then unpacking it oh I remember the first time I saw an XZ file I was like what on earth is this I don't know what this is so gotta go look it up every couple years there's a new compression or decompression scheme coming out and then the moment you do something then you have to patch your code the moment you do something slightly different using maybe using muscle instead of G-Lib C or maybe using the latest compiler for some code that was written two years ago or something like that the moment you do something different than what the original developer did you frequently will run into an issue where you need a patch I mean it just happens all the time that there's some little tweak there's some little header file that was changed some definition was moved to a different header file now you got to include something else as soon as you change the dependencies slightly you'll find little things are needed to be added so you need patches but sometimes upstream doesn't exist or sometimes they don't accept your patches they're like we don't care about that we're just focused on this you know so you need some sort of place to store all these patches where other people can find them and then you have to configure and build so you have to you know there's there's been make and auto-config for decades but in the last couple years we've seen an explosion of Jip and Jin and Ninja Waf, Messon and of course Aunt and Maven been around for Java for years but there's the auto tools there's Cmake all sorts of different ways and then you actually have to cross compile creating a cross compiler as I said earlier is not trivial it's a multi-step process it's a bit involved a bit sophisticated you know you're not gonna do it by accident and then using a cross compiler is not trivial either because you have to be very careful to not be contaminated by your host you have to make sure that the header files that your cross compiler is looking for the header files that come from your staged area not the build host and if you don't do things explicitly then implicitly it will look at the host header files and stuff so cross compiling is you know a landmine is landmines everywhere here there be dragons and then staging when I went through that list the first time you might not have recognized staging but it's obvious in retrospect when you build software a lot of software will have man pages and help files and all that sort of stuff documents include files on your target you don't might not need the man pages you don't need the include files if you're not building on your target so you stage you sort of package up well here's the binaries here's the configuration files I need those but I don't need the man pages but maybe someone does so I'm not gonna throw them away I'm gonna just kind of package them up and leave them over here and if someone wants them they can add it to their image and then signing all that stuff and then if we are gonna have package management on the device then obviously we have to have some sort of package manager we're probably all familiar with Debian and RPMs but there's also the IPK packaging which was a package manager and a package format that was created for embedded systems it's not as heavyweight or as sophisticated as the big players but as the big ones but it certainly does the job okay so this is how you create an image for your target device how do you actually get it to run because there are more dragons around this corner too every device has its own way of doing things has a oh this one needs two partitions and the you know the MLO has to be out of specific offset and oh my god all winners need sex files whatever those are which predate you know DTB's and they're kind of DTB-ish but they're not and yeah chip needs a UBI image and the board has to be put in cell mode and an RPI needs a config.txt in order to run traditionally Beagle board although my understanding is this change is traditionally Beagle board needed a U evt.txt file for U-boot to work and stuff all of these boards have these they're tweaks you know and it would be nice to get away from that it would be nice if your system would kind of take care of most of that for you creating how many partitions putting stuff in the right place for you so that you can focus on the high-level stuff this is my application this isn't where you make money you don't make money knowing all this stuff you make money writing an application and selling devices you know this stuff is just a waste in a sense but of course you know none of these manufacturers will ever get together and agree on anyone standard so you have to know them wouldn't it be great if your build system took care of this for you so putting it all together creating a distro is an immense amount of work a lot of things to know a lot of things to do a lot of tools you have to be familiar with a lot of concepts and all sorts of weird esoteric stuff you wouldn't think about and so I'm proposing hey let's use open embedded isn't this great open embedded does solve a lot of these problems it does have mechanisms and things in place that take care of all the things that I've discussed and of course it targets multiple devices this isn't it's vendor neutral it's not a system that was put together by well there were a lot of people who got together to to help bring it along and a lot of different companies have contributed to it for sure but it does remain vendor neutral and it does target a lot of different devices it provides recipes it provides metadata if you wanted to install a certain application onto your target device you have to know where to get it how to unpack it what patches you need how to compile it how to configure and all that stuff all of that metadata is contained in these things called recipes and it has recipes for thousands and thousands of different packages so you know I'm sure all of us could go out and find some package that doesn't have a recipe for it but most of what you're most likely going to want or need is going to be there and in many cases it's in the base system alone it prepares your cross-development tools for you it does all the building all those steps fetching unpacking patching all that stuff it creates the image for you in the proper format whether it needs partitions or not it puts things in the right places and with in many cases a very very minor small change the exact same versions of software and image that you are running on one device can be created and run on a completely different device with very small changes and then you have to recompile so the recompile takes four hours but the actual your work involved is a very minor in many cases it has you know most of us interact with open embedded using text files we use emacs or vi whatever most comfortable with the most of us are comfortable editing config files and stuff but if you are more comfortable with the graphical environment there is a tool called toaster that you could use it also can integrate into eclipse although I don't use that myself so but another very important thing that it does is within your organization you'll have one or maybe a couple of people involved with generating the image and working with the device itself but the whole point of what you're doing is to create this application that's the main focus so you might have five developers you might have 20 developers the developers don't care about images are creating images they write their software all they care about is the SDK where's my SDK and so open embedded has mechanisms in it that will if you've created your image you then with one more command it will then go ahead and create a relocatable retargetable SDK that you then give to your developers and that SDK can be used obviously on Linux but you can also use it on when it was in Mac using a project called crops which was sort of a sub-project that falls under the old embedded development and and and it also targets multiple architectures arm power pc x86 little indian big indian 64 bit 32 bit risk 5 all that stuff such as the mental board which is a 64 bit x86 the begobone cortex a or cortex a 15 if it's the newer ones and for like an open embedded also has of course a layer system I mentioned that a lot of the a lot of the stuff you're going to need every day is going to be in the base system but there is a layered system on top of open embedded where you can add additional functionalities or BSPs in this case what we're looking at here is BSPs so in the base system you wouldn't necessarily use that or be able to use that to build an image for your minnow board but by adding the meta Intel layer then you that gives you what you need to create image for that board for Beagle board which cortex a8 you would use meta TI for example if you were targeting the dragon board you'd use meta qualcomm which is a 64 bit cortex a 53 for example and uses the adrino 306 the or freedrino which is a open source graphic stack which is very very exciting if you wanted to use the chip board that's an all-winner cortex a8 uses Mali 400 use use meta chip for example there's the MIPS devices odroid amlogic s805 s905 devices the odroids the Samsung's metodroid xilinx has their SOC's which are also supported by meta xilinx very active actively developed but and then pine you've got the all winners you've got fire rock chips you've got the free scale imx6 and now imx7s for example which are a9s and stuff this also has open source graphics in the in the project and at Naviv look at stash and bunch of people from from over in Europe are doing an amazing job bringing reverse engineering the vivante graphics processors and producing open source graphics for these sorts of devices of course we have the raspberry pi that everyone knows about and as I mentioned before that comes in multiple flavors but they're all supported by one BSP layer banana pies orange pies and dozens and dozens of others the great thing to another amazing thing too about these embedded Linux build systems is again back in the day you would have someone working on embedded build system they would get a bunch of stuff running and they would say oh I getting tired of typing all these commands in all time I'll just create a shell script I'll put in all my commands and then I can do these things automatically and then of course someone on the other side of the office hears that this person over here has you know is doing this embedded development and they've got this script though send me your script that's great use the script that fails instantly and it's like well it works on my machine what's going on and after two days of banging your head against the wall you realize that one person is using fedora and the other person is using open to say or something and there's enough differences in the host system that what works for one person doesn't even though both of them would be very modern what you might happen have happened is that one person's distribution will have a more modern version of git for example and they happen to discover a flag that they think is great they'll use that flag and then when the other person tries to run that same script the git will fail because that flag doesn't exist type thing and so that is a very common problem and of course as we know there are dozens of different linux distributions and trying to get developers all around the world to be able to create the same image gets very difficult and so these embedded build systems actually have this really neat feature where before they start doing anything whatsoever before they even think about your device before they even look at your configuration what you want to build they don't care about the first thing they do is they download and compile configure and build tools that it'll use itself so in many cases it will download you know its own versions of git and stuff like that so that everyone using the same release of open embedded will have the same git functionality whether they're using fedora from three years ago or the latest say leap or whatever tumbleweed or something you know and that is a very fantastic functionality they all do that now and that has alleviated enormous amounts of problems so they're mostly host independent open embedded also has a very extensive test framework you can have tests that are installed to the device that are run to on the device when it boots up automatically so you have a test image you can just add a line to your configuration and now you give that to your test department and when they install that on the device as it comes up it'll run whatever test you want to run there are tests for the compiled you can look at a compiled binary and make sure that it's compiled for the right architecture for example so why not test that you can look at the binary and make sure that the library files that it's pulling from aren't the system library files for example so there there are tests that can be done on the host machine there are tests that can be done on the target there's tests that can be done for the packaging and all that sort of stuff and there's been a lot of work in the past couple years putting all this extensive test framework testing if you can think of something to test open embedded tries to test it another thing that's really important too and for a while open embedded was one of the few ones I think one of the only ones that did it although most of the others are following suit something that's becoming very important nowadays is licensing you know in the cowboy days of yesteryear if we could get something to work we shipped it you know it's like it works it's good nowadays you know just as it's about to go out the door you get that knock on the door from the corporate lawyer and they say hey what licenses are you know what software are we running in here what and these built systems now they have an ability that they that's in that image every version and will also say what licenses each piece of software is and so there's this way of looking at it you generate an image and then generate a manifest file but you can also do it in the reverse to you can say to open embedded we are not allowed to include any gpl3 in our in our product and so these are the things I want you go ahead you know I can't be bothered to do all that work myself you go ahead and start this build if you come across the gpl v3 piece of software you stop and tell me and it will and so you know you can be safe in knowing that as you're building your image because of course there's depend there's the things you want but then there's dependencies of those and then there's dependencies dependencies that you know selecting three things at a high level generates an image with hundreds of packages in it going through that yourself would be very difficult so open embedded has you know they thought of this they have this capability and a really neat thing too is the estate cash which is sort of like ccash I guess if open embedded has built a piece of software already then it generates a check some of that and it stores that and then when it goes to do right before it compiles anything it checks its estate and if it sees that it's already compiled this configuration and it's very clever to I mean it stores not just the compiled but how it was configured and everything so if anything in the config if you want to change a configuration option then I'll say oh you're changing configuration over to recompile this better re re it's very clever actually if you change a configuration option it knows it doesn't have to fetch it it knows it doesn't have to patch it but it knows it has to reconfigure it and it has to rebuild it so it's very clever in that sense that it does and that's a lot of work that again that's not a trivial thing that happens by accident but it can speed up your build your first build will always take a long time especially if I have to download everything but subsequent builds tweaking your image playing with it becomes incredibly fast after that and you don't have to know you don't have to say well you failed in the configures so I need to go back here you just type bit bake and it knows where to start from so how does it work yes it has metadata and recipes but there's this higher level thing called classes you know a lot of C software for example will either use auto tools or CMake or whatever and there's no point having to store in the metadata how to auto tool something so how to auto tool something gets pulled out into a class and then in your recipe you say here's what I want to build here's where you find it here's your your computer here's its license and it's an auto tool package and then bit bake will look at that and say oh it's not a tool package I know what to do I'll go and grab this all tools class and work with it that way open embedded has this concept of layers as I mentioned before you have sort of a base layer which is put out by the project itself and tested as a cohesive set but then you can also add functionality on top of that for BSPs software distros by adding layers to your build a bit bake is the tool itself it's the quote-unquote make it's what you type that runs everything it runs the show and then of course you have a series of configuration files sprinkled throughout the system that configure the various things I looked at this right before starting and realized that a black background was not the greatest so that's why we didn't the lights hopefully it works but anyway this is what a recipe kind of looks like don't get too bogged down in the details it has a bit of preamble which is just you know stuff like a summary there's a one line summary there is a multi-line description so as you're looking through something like the GUI tool toaster as it gives you a list of all the packages you can choose from it'll spit out that little one line summary so as you're looking through you can see oh this is oh I forget what this package does oh that's right it's a free implementation of the OpenGL API it gives you the home page it can optionally you can write down where the bug tracker is so as you're working with it and it fails to compile then you don't have to go searching for where to send bugs to it'll tell you and then every recipe can be sort of sorted as to whether it's you know bootloader or GUI or something in this case it's an x11 and then there's the licensing stuff you know so there's some preamble there's some licensing stuff you have to specify a license with every recipe you can't forget about this bit bake will fail it'll say there's no license you need to tell me the license now of course this is the tricky part because as of today it doesn't have the intelligence to actually look through and figure out what the license is itself the person who puts together this file has to do that manually and there are potential errors involved with that but we certainly expect that anyone putting together one of these files is able to understand what licensing is and can write down the right license but provided that's all done correctly the other thing that's really nice too is that most projects will of course will have a licensing file included with it and so we check some that license file and it happens very frequently that a paragraph will get added to a licensing clause when that happens and you go to compile the next version of some piece of software bit bake will detect that because it's checked something the license file and when it sees the difference it will fail it'll stop and say hey this license change you need to investigate this does that license line need to be changed why did it change we have to figure out we have to figure this out so that's kind of a cool feature there's this idea of provides a long time ago in the talk I talked about how you know there are dozens of shells to choose from where every shell essentially provides the same functionality and what's kind of neat is that you might have a package that relies on a shell but it doesn't really matter if it's s h or bash you know and so you can you can say that so in the recipe for bash you can say it provides this functionality provides a shell so that when you compile this other application it's not going to complain and say oh you didn't include bash it's going to say oh I see you've included a shell okay I'm satisfied because you know again if you're trying to put together an image and there is no shell then you know you need to add a shell but it it's a way of saying I don't really care which shells included I just need some kind of shell functionality so I need something that provides this functionality so you might need something that provides EGL or GL and you don't care if that's provided by binaries like the Mali binaries for example or you don't care if it's provided by the Fridrino project or something you know to you it doesn't matter you just need GL functionality both of those will provide that so my application will be happy because when it runs it'll find GL and then there's inheriting yes this is that you know this is an auto tool package so this is how I tell BitBake that this is an auto tool package so just use auto tools and you know it can inherit there's a whole bunch of lots of classes there's I don't know if there's a hundred of them but there's lots of classes for things like package config auto tools get text you know all that sort of stuff checking for features checking for Python 3 native you know all sorts of stuff and then this is the tricky part but when you configure software let's take the example of VI for example VI can be run on the text on the command line in which case all you need is get text and you know it's dependencies but you can also run it as a completely independent GUI application in which case it has a title bar and all sorts of stuff in which case you need GDK plus but when you're developing for you wouldn't want to have to include GDK and all its dependencies in an embedded device if all you want to do is run in VI from the command line so there's this package config things and that's all handled by in the auto tools for example it's when you run configure dot slash configure and then you go dash dash this dash dash that enable DRI disable this you know enable docs you know all that sort of stuff and so this is this is open embedded's way of allowing a user so in this recipe we're not mandating these sets of you know oh you must build VI with this we're saying well we know that you can build it this way or that way so we're going to provide two package configs we're going to provide you know a GUI package config and a command line package config and at the high level in your this is not the VI example of course so but this is an example of another package but in this case this package this recipe has specifically defined a DRI 3 package config so this software by default I'm guessing would most likely assume DRI 2 but has the option to be configured for DRI 3 so in this case at a high level you can say oh I want to try the DRI 3 stuff so this would be how you'd specify that and this is how open embedded handles package configuration and then dependencies are very important it's it's a lot of what this stuff does dependencies dependencies dependencies so you list the dependencies and here's another neat thing too as open embedded is compiling and linking all your software it can actually I mean it will look through and the way that it builds stuff is if you forgot to add a dependency the build will most likely fail but if by some miracle it actually does succeed it can look at the binary and see what libraries you're linking to and it compares those libraries that your application is trying to link to with what's listed in the dependencies and if there's a mix match it'll actually stop and say hey I think there's something fishy with your dependency list so dependencies it does a very good job of making sure your dependency list is create because in the good old days of using open embedded it wasn't so stringent and so you could forget to list a couple dependencies you would create your image you would deploy it and lo and behold your image forgot a bunch of libraries and then when you went to run your app it would fail so then you have to go back and recreate your own at least it catches it a lot earlier now so you're not wasting your time but dependencies often depend on how you've configured the software like I said before in the vi example if you want to build the GUI version then you need gdk but you need you know get text if you don't so as you're specifying package configs you can optionally specify different sets of dependencies as well so there's there's a way of saying no matter how you use the software you need these dependencies but if you enable DRI 3 well now you need these extra dependencies as well so that's all handled and then down here is where you fetch it from and check sums for various you can check some everything if you download a patch you can check some the patch you can check some it in multiple ways md5 shaw 256 blah blah blah so either you can generate the check some yourself and put it in the recipe or if the people creating the tar ball specify one for you then you just copy and paste that in here and you have the flexibility of you know using whatever they are so that's a recipe file that's the metadata of how you get open embedded to build a package and then there's those layers as we talked about you have to tell it which layers you want to use so in this case I'm building for a raspberry pie I want to use the core open embedded stuff I'm using the extra open embedded libraries gnome python and I'm using meta browser because I'm probably going to include Firefox or Chromium and and I want Qt5 stuff running as well so I've added the Qt5 layers or the Qt5 layers so recipes layer configuration can be simple can be a one-liner if you have a very basic image or they can go on for many many dozens of lines depending on how sophisticated your images and so at a top level you you have to tell open embedded somewhere you know what what you're doing at a high level and so in this case I'm telling it I'm building for a Raspberry Pi 3 the distro I'm not really using a distro for this build where to place my downloads if you work in a company and you've got you know five different people doing builds well you don't need everyone downloading every package from the internet individually you can create a shared space where everyone downloads their stuff too so if your fellow co-worker downloaded something yesterday well your build can use that and doesn't have to waste its time downloading if you're all pointing to the same shared location oh yes and down here like for example I want to use a specific touchscreen and so I want to turn off I want to disable the cursor when I build my image because touchscreen devices you don't see a cursor so I want to tell when I want to tell the system that when it builds the X server I want to turn on this package config this configuration that says no cursor which if you go and download the sources for the X server and you type that slosh configure help and you look at all the help you'll see that there's a no cursor option and so whoever wrote the recipe for X 11 has provided this little package config thing called quote-unquote no cursor and so I want to enable that I want to set that package config when I build my X server so here in my top level configuration I'm saying you know enable that because I don't want to see a cursor and then you know all these other things I want to use IPKs instead of Debian's or RPMs for example you can specify all that and yeah and in this case I'm building a test image so I don't want the root to have to I don't have to install passwords I just want to type root and get into my device so I've enabled debug tweaks which is just a bunch of little cute things it's just a one liner to say and make me a test image so I don't have to worry about all these little details and then of course when you create your production image you would not include that line preferably and then you would have to install a root password and users and all sort of stuff so I like to talk about a case study for why you would want to get into distributions and why you would want to use such a big system and a couple years ago I was helping you know me and the people I worked with a certain company want to put together what we have here is a company that makes set-top boxes and so they have you know hundreds and thousands of customers and they need to give set-top boxes to all these customers now purposefully they didn't want to get their set-top boxes from only one vendor they want to purposefully get it from multiple vendors you don't want to be relying on one company when you're trying to give thousands of these every day to your customers but not only did they not want to rely on one company they also specifically did not want to rely on one architecture because you never know where you're gonna wake up tomorrow and find that someone is claiming a patent on MIPS and oh my god everyone who's selling MIPS has to cease and desist you know so they purposefully had multiple architectures multiple providers but when you unpack your set-top box and install it from company XYZ they all need to look the same you can't have 30 different instruction manuals you know you can't have your QA department fielding calls from it has to look the same for everyone it has to have the same features you can't have one set-top box that is very blocky and chunky and another one that smooth and really cool and everything I mean they all have to be cool they all have to have the same things have to be in the same place and so what you have is but you you rely on your set-top box provider to provide a base system for you you assume that when you're going to add your product to a set-top box and give it to a customer that it has you boot it has some kernel it has it's coming up to at least some point you don't want to have to start from zero with all of these products you want to start from a known position write your software install it on there and have it it works so you need you know you have developers you have to give them SDKs you have hardware manufacturers they need to provide you with some base working set of software and and this system is called the the RDK so they created this distro called the RDK and a purposefully multi-vendor yeah and so and they needed to tightly control you know if you want to use some new fangled feature from Linux well you then have to make sure that all your set-top boxes are using at least that version of Linux so that you know you have that functionality installed so they need to say to their vendors we need at least this and then they need their vendors to provide that but of course how you compile this kernel for that product is is all I mean it's a mess and frankly when we started this project they had their own homegrown solution and they had gotten to the point where they realized that it wasn't working anymore first of all you have to teach all of your ISVs how to use your system which is not easy and every day of course they were discovering oh how do we generate an SDK oh my god we want to add a new you know architecture oh we want to do a MIPS 64 now oh how are we gonna do that you know and and they realized that that's not where they make money they don't make money you know building images that's not their product their product is well actually their product is selling the data that streams through the device the set-top boxes and even their products you know so that that's not where you want to waste your time and your money and so they looked at open embedded and well long story short they loved it and everyone lived happily ever after so but it did solve their it solved their problems and it's been what five years now I think and there's still Rdk is growing more and more companies are coming into the ecosystem and open embedded has been a success for them yes so I've painted this wonderful picture about how open embedded solves all your problems unfortunately it doesn't not all layers work together it is it's very easy to choose stuff that won't work together I mean even though open embedded tries its best to make sure I mean there's only so much testing that can be done the quality of the layer sometimes BSP layers often will frequently not work well together they assume that if you're including meta Raspberry Pi that you're only building for meta for a Raspberry Pi product but you might be trying to create a distribution for multiple products in which case you would have multiple BSPs and so now suddenly you know the Raspberry Pi I'm just using this as an example I'm not saying this is what happens but you know the meta Raspberry Pi layer now is suddenly trying to change something on your device that you know an Intel device doesn't need or care about or might mess up the Intel device if this thing gets changed you know so oh and building needs a big build machine whenever we're introducing open embedded to a company the first thing we say is by by a monster I mean these builds you can't have enough memory you can't have enough drive you can't have enough processors you can't have enough cash I mean the more the merrier for all this stuff and it certainly needs a really good connection to the internet because of course the first thing you need to do is download a whole bunch of stuff so it's not perfect but it's it's certainly gonna help you get there but it still needs you know some some work on your part so anyway with that I've got five minutes or so so questions comments praise money all right well thanks so much so this whole day this room this whole day is gonna be open embedded talks like I said at the start this is the inaugural open embedded summit that we're giving a whirl this year at scale and so later on today we're gonna have talks about setting up Wi-Fi for example which is not as trivial as it sounds we're gonna have a talk about if you're a kernel developer and you're building your image with open embedded then you know how would you go about working with the kernel and stuff like that and then the last talk is John Mason and oh actually the second talk is John Mason from ARM and he is going to be talking about this is yucky oh no it's John that's doing the kernel development that oh yes Alistair we have Alistair Francis from Western Digital and he's been doing a lot of work lately with risk five and of course risk five is growing is really cool and the needle stuff nowadays and so he's gonna come and talk about the work that he's doing on risk five at Western Digital but also how open embedded has been helping them I gave you an example of a company that was using open embedded successfully and he's gonna be talking about how he's using open embedded with his risk five work as I said earlier this was supposed to be someone else low cash unfortunately wasn't able to make it he had a great talk he sent me a slides but I wasn't going to present someone else the slides about how they use open embedded at LG LG has been using open embedded for a very long time they're one of the early companies and they use it in all their web OS projects web OS is a distribution that's built using open embedded and so it's unfortunate he wasn't here I was looking forward to getting all the nitty-gritty details but unfortunately he wasn't so you're stuck with me and said yes no thank you do you know if he's going to be if you can provide those slides or oh see or if he's going to be presenting at another conference I can't speak to that obviously I don't know the plan is to there will be there's open embedded org and when we were planning this at scale we weren't sure scale was going to provide us space on their website to be able to promote this summit and so we had started the the wheels in motion to create an open embedded event site where we would list you know if you go to ELC and give a talk then we're gonna list that there and stuff like that so we have all the input but then at the last minute scale said oh yes you can have web space of course you can you know so then we sort of abandon that and focused on what we know getting our talks ready and stuff but there is an event site that will be coming up the DNS is already in place and we will be listing all these stuff and I'll definitely put his slides on there for sure thanks thank you thank you for everything that you do thanks I have stickers open embedded stickers and to them stickers if anyone's interested all right we'll see you at 1 30 for drew right good afternoon thanks for finding your way back we're gonna continue on now with drew drew works for Mender.io and we'll be talking about configuring open embedded for Wi-Fi deployment I have a bunch of stickers up here open embedded stickers as I mentioned earlier this morning this is the inaugural open embedded summit we're trying a little something new this year normally we would all get together at the Linux Foundation's embedded Linux conference but because that was pushed out till the end of August we found ourselves with this big massive gap between Edinburgh last year and San Diego in August so we thought we would get together at scale the timing was perfect for scale and the location was great I think so can you guys hear me alright very good thank you Trevor yeah so as Trevor said my name is Drew Mosley I work for a company called Northern Tech on a project called Mender.io but before I I'll tell you a little bit more about me in a second but let's talk about a little bit let's get started with you know kind of like why we're here so in the open embedded space a lot of boards don't have Wi-Fi and so out of the box getting Wi-Fi working doesn't necessarily just happen you've got to take a few extra steps there's several different network management packages that you can install that will handle this for you so I'm going to discuss a little bit of the differences between them and so show some runtime examples of how you use each of these network management packages to configure a particular Wi-Fi network at the actual runtime additionally I've got on the bottom you see my github link for recipes and metadata that will actually configure these various network packages at build time and as you see the recipes are there in github and then just briefly mentioned just a few other related considerations that you should things you should think about when you're bringing up these systems this this talk tends to be a little bit more how-to than most of my other things so I've got a lot of recipe samples and things like that I'm not sure how well that's going to going to work in a presentation format but that's why it's all there in github feel free to take a look afterwards and use it to use it as you see fit the main goal here was really to gather all this information into one place all this there are plenty of blog posts that I found and most of them are lost to history I apologize to the original authors I can't quite point to them all because it's been so long ago that I originally started pulling this information together but I didn't find a single canonical location that had them all laid out in one place so that's really my goal here I ultimately want to turn this into a blog post that gets posted somewhere and will hopefully be maintained for posterity there so that was really the objective here is to pull all this information together like I say it's not complicated stuff it's out there but I didn't find a single place to pull it together so a little bit about me I've been in the embedded Linux base about 10 years and quite a bit longer than that in general embedded currently my role is technical solutions architect for the Mender project meaning I'm kind of you know looking at users I'm not actually part of the development team directly but I'm working with users to get requirements and figure out how to adapt to the the Mender IO update or to their to their particular use case so if you have any need for over-the-air updates for your embedded connected devices feel free to find me after the talk and I'll be happy to talk to you all about that so I just wanted to start with a couple of the challenges that you run into when you are configuring Wi-Fi for these these types of embedded devices so the first one that's most obvious is the on target user interface or in most cases the lack thereof a lot of the utilities kind of assume you have a GUI of some sort that you can go in and enter the credentials in a lot of these devices don't necessarily have that they if you if you jump through extra hoops which we which you'll see here in the demo hardware you can get a serial console not all of the devices have a serial console available easily there is a variety of network management packages and and we'll discuss three or I guess four of them what one interesting thing to consider when you're doing an embedded build is the difference between entering the credentials at runtime and entering them at build time in your desktop environment you you boot the system it comes up you say okay connect to Wi-Fi whatever enter the end of the password you're good to go and that's fine for your your your personal use devices but if you're producing a hopefully the devices that you want to have some subset of SSIDs pre pre-configured on you want some of some means to do that at build time and then related to the network management packages which system in it are you using system D or just via knit or any of the other ones out there they it's kind of peripherally related I mentioned it here just because they're they're kind of tightly tied to the network management packages that we use and and similar to you know how do you get the credentials at build time if you do have multiple systems and you want to have a single image for it and you want to be able to get those credentials in the device you want to make sure that they're in that image in a secure fashion and you don't want to store them in GitHub for one thing and then one of the thing to consider is the read all the root file system all of these network management packages they have a particular format for their configuration files generally speaking they're under Etsy if your slash Etsy directory is read only then you might need to take some extra steps to make that read right if you want it read right in your end user device and of course trusted storage TPM things like that I'm not discussing that here that's kind of outside of the scope of what I'm doing but that's a you know something else you might want to consider moving forward so briefly what was my test setup raspberry pi zero zero W rather because we needed Wi-Fi and because I'm a helpful guy that that blue link will take you to the Amazon page or the raspberry pi page where you can doubt buy one of your own if you're unable to use Google similarly the serial console cable this is the one I I've chosen to use there's dozens of them out there you can go to Ali express and buy them for about 75 cents each ship from China I still haven't figured out how that works but you can order 10 of them and have them shipped to your house in six eight weeks and you're good to go and then I was using the GL line at portable router which I found on Amazon it's a pretty handy little little device that has open to be our T built in it's easily set up to be configured to be it will connect on to further Wi-Fi's and act basically as a bridge so that I bring this with me whenever I travel I have one device I have to hook up to the airport Wi-Fi and then all of my other devices just immediately jump on that and I set it up with a demo SSID and everybody's favorite highly secure password monkey one two three so if you happen to see if I power my device up anybody can get on it and you know obviously I wouldn't recommend that you that you use that password in production so the first step is just getting the open embedded source code the set of instructions we see on the screen here take us through and the link at the bottom is the best place to go to find this this detailed setup obviously we're starting in a clean source directory we're going to clone a couple of a couple of get repo repositories the first one obviously is the open embedded base code and then we adding the meta Raspberry Pi layer which is where the platform support is the kernel you boot and that kind of thing that that are necessary for Raspberry Pi bit bake we're cloning version 1.40 because that's what that link told me to do and that's the actual build build executive for the open embedded system and then for now we'll go ahead and check out my my personal metal Wi-Fi credentials repository even though initially we're not going to be using it and then we we simply run the standard build steps for creating your build directory by running the OEM it build in and script and then since we're building for Raspberry Pi we go ahead and add that layer to the system and for the basic configuration this is the basics that I started with in the config file obviously we specify what I do too many buttons there we go so first line obviously specifies what machine we're using and I do generally always add the dot B map version of whatever binary I'm going to be installing if you haven't used B map instead of just the DD utility please do a Google search on it figure it out it's easily an order of magnitude faster to write most of these images you know some of these some of these images can take you know five to ten minutes in using DD and they'll take 30 seconds when using B map so it will save you a lot of time enable you already equals one that's specific to the Raspberry Pi just because of the way those pins are multiplexed with other devices it's not enabled by default distro features of pinned this is basically what tells the open embedded build system that there's something related to Wi-Fi that's going to be part of the build honestly I haven't dug into that to see exactly what packages that directly includes but that's the first step and then couple hardware specific things and of course this is going to be very specific to the Raspberry Pi we've got the firmware and in the third branch is actually from a they've got this our pie distro version of the firmware that's part of the Raspberry Pi layer so you have to actually specify the firmware blobs that are that are corresponding to the onboard Wi-Fi chipset and then also the kernel module so depending on your your platform layer and the hardware you're choosing you may or may not need those kind of lines but depending on what exact hardware you're using you probably have to do a little bit of research just to figure out what is appropriate for your system and finally just because I got tired of waiting for it to generate SSH server keys on every time I booted up I'm just removing the SSH server from the build because for purposes of this demo I didn't really need it and the basic build and deploy step run a bit big build and as I think probably everybody in here is aware at that point you might as well go get a cup of coffee it's going to take a while you're starting from a fresh fresh thing I've got a build VM with I think 32 cores and 64 gigs of RAM or something and it still takes 30 or 45 minutes for that that that initial build so and then from within this directory here the deploy images Raspberry Pi zero Wi-Fi this is the beam app magic that I mentioned and basically this is used this uses a sparse file system stuff that's part of the Linux kernel to greatly speed up the the writing of the block device so in the case obviously DevMMC block zero is the block device on my laptop corresponding to the SD card that I put in be very very careful make sure you choose the right device note here it can really ruin your day if you choose the wrong one not that I would have ever done that or anything so now we move into actual testing of the basic system at this point we've got the SD card we put it in the board we we power it up because the debug tweaks was enabled login is root with no password we log in we run the IP adder command and we verify okay it sees the WLAN zero device without some of those steps that we did before you would not have seen that so at this point we know we have the hardware configured to the point that we can actually start doing things with it so let's talk a little bit about what the network management options that I'm going to dig into are so we've got the two main build systems this vnit and system d with this vnit it's basically just WPA supplicant you're actually putting your credentials in a properly encoded form in the configuration files for the for WPA supplicant with system d you've got three different options that I've tested system d network d which is a package config option for system d and then there's con man and network manager now con man I don't know it's full history but I know it got it it came out of the automotive space or was used quite heavily in the automotive space and then network manager is what you see on your typical Linux desktop Ubuntu Debian system or whatever those two can probably work with this vnit I'd be surprised if they couldn't but I don't normally run a system without system d so I haven't actually tested in that particular environment so to build for that first component that that first option sissy in it and WPA supplicant note in the in the the third branch that is the default you really only need to add the one the one line to your local com to make sure you have the WPA supplicant in your build that might be that might be overkill you might not even need that I haven't tested if that's part of the distro features equals Wi-Fi that's true too but because of the way I do an append and then later do another append I'll put the plus equals in there just just to make sure it's you know Belton suspenders I'd rather have it in there and not need it one thing to note if you're not if you haven't been working with open admitted for a while note specifically there is a space between that double quote and the and the first line that is required when you're using the under underscore a pen syntax I'm sorry yes that that that is required when you use the underbar pin if you just use the plus equals it's not so no if you don't have it what it's going to concatenate that up against whatever was already there and you you generally want that space oh really okay that's news to me I because I could even even if I have multiple ones in the same file okay okay but so and just for the the recording purpose using the plus equals and the underscore append is not supposed to be is considered bad syntax it happens to work in this case is what the the consensus in the room is now let me ask you guys this so if I have two underscore pins in the same file will they both be applied okay then then yeah in that case you do not yeah okay so I was unaware that that was the case so so I will update all my scripts at at some time in the future right they don't know what's the pending next to it if they don't have a space then it'll become WPA supplicant but sure if you put a space right here then you know that this is correct all the time yeah true there we go alright so how do we use this thing at runtime there's a WPA passphrase utility that's built and installed as part of the WPA supplicant package in this case you simply run it with the name of the SS or the SSID and password that you're trying to connect to and redirect that to the Etsy WPA supplicant dot com and we'll see the syntax of that here in a second and if you're using just system just be in it with WPA supplicant the default setup at this point you simply run IF up WLAN 0 and you're good to go note that WLAN 0 is the name of the device for my board and depending on your configuration it might be different so that's another another place where the hardware specifics may come come into play so how do we build this into our image so in that GitHub repo that I pushed out I've actually got several subdirectories one for each of the configuration so all we do is add the layer and in this case you see I've actually got a BB append to the WPA supplicant recipe where we're adding a scale-demo.conf file to the source URI we're adding a do install append which means it's going to run at the end of the do install function and at that point we're simply appending the contents of that file which is here to the existing WPA supplicant.conf so in this case this is essentially the output of that WPA passphrase and when you run it from the command line it'll actually include your clear text password obviously I pulled it out for security reasons here I'm not sure how how properly encrypted this pre-shared key is but presumably if somebody actually has access to this file on your system then all bets for security are off anyway so and the other the the other component we needed to do is make sure that that interface is brought up automatically at boot so with sysvnnet simply append autowlan0 to your Etsy network interfaces file and then next time you you run your full build and deploy the system will come up and that that interface will automatically connect to the Wi-Fi that was provided in the previous screen so moving forward to systemd with systemd network the I think by default when you build a systemd system package config includes networkd so we don't really need to change anything for that I have seen some posts and perhaps you guys are more familiar with the details of making systemd the default to knit system but I don't know how or when that's going to happen so exactly how this is done in newer versions might change a bit you might have to change it to get back to the previous one but the the obviously the first line is the same thing that we had before just to make sure WPA supplicant is there and then the following four lines are the the the the means with which you switch your system to systemd and I honestly couldn't tell you what they all do I just know that if you google how do you switch to systemd every post says put these four lines in and run a bill so once you do that you have a system that you have your system will be properly configured for systemd note that these five the four bottom lines here are required for all systemd base configurations I won't mention them again as we jump through each but this is the default setup for getting into systemd so if we're if we're using networkd the networkd package config of systemd the the method with which that file gets configured is very similar to what we did before running the same basic WPA that passphrase command that we ran before however now we're putting it into a directory a file in a sub directory called WPA supplicant and one thing to note is this in L 802 11 suffix on that file that has to do with some of the user space Wi-Fi driver types or something in L 802 11 I think is the the new hotness in the old the old and busted one is a wireless extensions but you know when you buy the $3 USB Wi-Fi hubs from from from Ali express again most of them are using the older format so that's still around your you will still see it but fortunately for the raspberry pi zero you don't need it additionally you need to create this WLAN network file which basically says anything that the WLAN star anything that shows up like that in the system it's going to use DHCP before and we're not going to do anything with a hostname for purposes for purposes of this demo once this file is there that basically means systemd networkd is going to try to do something with this interface when it comes up so we simply restart systemd networkd and we start the WPA supplicant specifically running on WLAN zero and so that's going to look at all the configuration information that we put in and now we're we're up and running and we're live with systemd networkd there's a button here that I think I hit just hit it again oh there we go okay so to do the the the same thing at the build time obviously there's a another layer in my repository that will add a couple couple things here first we're looking at the systemd configuration where we explicitly add networkd and resolved again I think those are default that may not be necessary here but just to make things consistent and in future proof I want to head and put it in there go ahead and specify that WLAN network file which we see down here I've cleaned up the spacing just to make it fit on the slide a bit better and add a do install a pen snippet that that makes sure that file gets installed under the systemd networkd directory that we wrote to in the previous slide and additionally we have to update the WPA supplicant config this is the the the file that contains all that information very similar to what we did previously persist via knit and then again for WPA supplicant we're adding this file to the source URI and I wish this worked to be able to actually use the systemd enable bits that are part of the open embedded metadata to to create a to automatically start a service file using the the at syntax I couldn't get it to work I don't know if if anybody else has a means to get that to work that would mean we could get rid of some of this ugliness down here these these bottom two lines in my shell script snippet are basically recreating what the systemctl enable command does at the command line so this is really you know kind of out of place here because it assumes certain knowledge of how systemd works and it looks like doing something like this is expected to work although in my case I couldn't get it to work so for now I've just got the manually creating the sim links and that's what tells systemd when it comes up that it's taking over that WLAN zero interface okay so moving on we got two more that we're going to look at but both of them are using systemd and in both cases they're not going to be using systemd networkd so in this case we're going to go ahead and use a package an underbar removed for the package config to make sure that we don't have networkd or resolved because the the other network management package that we're going to install will be taking the place of those so for conman again very simple adding conman and the conman client into our image install and once we boot the system we're going to run conman CTL we're going to enable Wi-Fi we're going to scan the Wi-Fi the available Wi-Fi we're going to turn the agent on which is effectively telling it to register with WPA supplicant we look at the services that it has seen which is basically show me all the Wi-Fi access points you've seen and in this case it sees our scale demo and gives us this big long name here we connect to that big long name and it's going to ask us for a password I think this prompts a little misleading the first couple times I did it I wanted to say yes I need to give you a passphrase but it you actually are typing the passphrase in there and once you type that in you're connected you're up and running you're able to ping the network and all is happy similarly for doing a build doing this configuration at build time we have another layer that does this setup we're pending the conman recipe which is doing effectively the same thing that we just did at the command line we're creating we're telling it to start the service after five seconds we're telling it to enable Wi-Fi and then of course we have this this demo config script down here which is effectively what that conman CTL did we're just creating the file ahead of time and making sure that it gets gets deployed to the target system under the Etsy directory in the appropriate place or excuse me rather in the barlib conman directory as you see here and finally network manager which I have come to decide is probably the easiest one to use at least from my perspective it does require a little bit more setup than some of the previous options network manager itself is in the meta networking layer under the meta open embedded repository and then that has the the dependency that it needs Python and the the general meta oe metadata so we go ahead and we we add those three extra layers we add again our network manager and the text user interface to it we add that to our image install do our build and deploy as always and now we boot the thing and in this case there's a simple command line in curses based utility called an MT UI comes up gives you this nice little text based user interface you get to activate a connection you see all the ones that are there in this case you're good to go now to replicate this at build time again same thing we did before and now we're appending network manager in this case where we are creating this configuration file which is exactly what the MT UI utility generated and we're pre-deploying that into under Etsy network manager system connections and one thing I will say is for all of these utilities you can include multiple configurations so if you have multiple you know when I do my normal builds for my daily work I've got five or six SSIDs that I always deploy into my systems just so they're there in case I happen to be on the road or at the coffee shop or whatever so you know in the case of network manager you just have multiple files called that in connection and network manager will automatically cycle through them depending on which ones are available and so finally just some other considerations that that come into play when you when you're doing these things obviously the security credentials is an issue I do have the monkey one two three password in that github repository I don't recommend you put your actual Wi-Fi credentials in there for my daily builds I actually keep them stored in a directory that's never published anywhere offline my local machine I mentioned the wireless extensions versus the NL 802 11 user land drivers something you'll have to consider based on what particular Wi-Fi chipset you're using multicast DNS so this is the the ability your devices to come up and register with your local DNS server so that in the case of the Raspberry Pi zero board I can always just say you know ping raspberry pi zero dash Wi-Fi local so I believe con man and I believe sorry I believe that network manager is the only one of the options that I considered here that requires an additional setup to enable that multicast DNS I think the others come enabled by default captive portals if you are producing a actual device obviously when you're deploying the device you know a lot of times if you're at an airport or whatever first thing that comes up is that you know login extra login screen that that's something you you will need to handle for a more robust solution access point mode a lot of commercial say consumer devices they'll come up in it in an access point mode so that you can connect to the the access point and then from there you can actually enter the credentials of your home Wi-Fi to allow your newly bought consumer electronics device to get on your home Wi-Fi so that's not that's obviously something that's not handled by the the current solution here and enterprise Wi-Fi I know there are some other Wi-Fi modes that require more complicated schemes things like two-factor authentication things like that so there are certainly other considerations and other modes that might need to be supported depending on your particular use cases for for purposes for my purposes I really was just trying to figure out how to get these devices to function on a regular basis in my my you know my home office Wi-Fi environment which is pretty straightforward so with that I think we've got some time for some questions if there are any like what sort of utilities do you have available to debug Wi-Fi from the app like the driver level like I honestly I don't know the answer to that you know that's the standard you know Linux bring up stuff it's not obviously not going to be specific to an open embedded environment although most of the util any of the utilities like that that you would use in that environment they're likely already available in recipe form in somewhere in an open a bit embedded layer well and those are all different those are all different options right I mean you use one or the other so typically you know when you enable system D if you don't do anything different you're going to get the system D network deconfiguration and you know depending on what distro you you happen to enable the distro can can can make the policy decision of whether it's going to switch off of that or switch to network manager or what whatever but it's the point is that it's easily easy to switch from one to the other based on your particular needs all right very good thank you very much continuing on with our summit so at 3 o'clock we're going to have Alistair he has a huge demo back here set up Alistair works for Western Digital has been doing a lot of risk five work and so he'll be talking to us at 3 about how he uses open embedded to help him with his risk five stuff and he has an awesome demo with risk five boards and lots of hardware and cool stuff so we'll see you at 3 thanks so much all right ladies and gentlemen 3 o'clock so we're about to get started thanks for joining us you're all here for open embedded right and wrist and wrist life if you're not aware we are running an open embedded summit today so I'll be talking this room we'll be all open embedded right now we have Alistair from Western Digital playing around with risk five and Alistair is playing a lot of work with risk five and he's been using open embedded for his work so when I bumped into him at Plummer last year in Vancouver he said hey I got lots of cool stuff to show so I said all right you're on without much further ado Alistair thanks thanks hi so as Trevor said so I'm from Western Digital I'm part of the research organization there so I work on everything risk five and Linux open source is basically what I do so I'm a QMU maintainer for the risk five port as well as other bits so that's a lot of my background is in QMU and QMU development and I also am a big contributor to the meta risk five layer and other things are open embedded Linux and other projects around the place and so today I'm going to talk about risk five for probably the first half what it is why people are using it what Western Digital is doing I'm gonna talk about the Western Digital Swerve core and then I'll talk about why I use open embedded and what I like about open embedded and what I don't like about open embedded and then at the end they have a demo which is kind of half set up here but we'll go into that later so if anyone has any questions at any point feel free to just shout them out it's always hard to know how much people do and don't know about risk five so I and try and cover the general bits but if anything's confusing it doesn't make sense so you want to know more information just let me know so risk five is an open risk based which I guess you should know is a so a lot of confusion normally occurs because people kind of get confused with it being an open ISA and open core risk five is just the ISA so it's the kind of the details of how the instructions that's implemented it doesn't mandate anything about how the core is implemented or how the core is licensed so the ISA itself is BSD so anyone can go download it and use it but you can create a closed source proprietary core with that so it doesn't there's a lot of open ones and it's generally open movement but it doesn't lock you out so which is why a big reason why companies are interested in it because they can still do they don't have to release RTL for everything right so so the base ISA is very small it's basically just addition subtraction and loads and stores and it's works in 32 bit 64 bit and 128 bit but no one's really focusing on that at the moment so and the ISA is designed to be extended so you normally see something like RV 64 GC or IMC or something like that right and those numbers at the end of the extensions that have been enabled so I'm not going to read them all out and everyone can kind of see there but there's different levels of extensions so there's the base ones which is the kind of second and third bullet the bullet points there and they are ones that are frozen or ratified by the RISC 5 organization so they're pretty stable you could implement them in hardware today and they shouldn't change or if they're fully ratified then they're definitely not going to change but the other ones say like virtualization is just a draft so if you want to implement it today it would definitely change by the time that it's finalized so there's different levels of how how easy progressing and it's all managed by the RISC 5 organization which is a company with over a hundred members and companies and individual contributors can become members and they will go through the process of ratifying the ISA and and approving it and finalizing it and and that's kind of the legal part behind it but at the same time the development is all open so the for example the hypervisor extension or the virtualization extension is available today you can go download it the latest the latest draft spec and implement it in hardware if you want nothing stopping you from doing that as it's discussions are public it's on github they're a poor quest there's mailing lists it's all kind of done in the public and we want people to contribute and we want feedback so again the hypervisor of the virtualization extensions have feedback from KVM developers they've given feedback on what hasn't hasn't worked in the past and that's been incorporated into the specs so it's a very open process for people can contribute back to or contribute to there's multiple CPU implementations available not going to go through all of them but you can look around I think the RISC 5 organization has a website dedicated to it and they have a sci-fi have some boom have one Kim you has support and WD has a core as well which I'll talk about in a little bit so any questions on what RISC 5 is before I move on to why RISC 5 so why RISC 5 so in an open ISA allows anyone to modify the ISA so this is this is different to commercial ISA's that are commonly used now where you even if you pay the vendor a lot of money they still give you a black box it's still locked down you can't change it you you kind of have to trust what they gave you is what they said that it is and that it works and it doesn't have problems so open ISA means that you can implement a core and then you can go show that to anyone and release it publicly or show it to your customers and it makes it easier to do security audits it means people can look at it and know what it's doing it allows lower cost of entry right you don't have to pay these licensing fees to get access which gives access to small companies and hobbyists and academia which is why RISC 5 started and so they can get access to a core and everyone can work together on this one thing without having to be locked out to just big companies or just important people and and things like that so the open core behind it has huge benefits there and it also allows people to modify it so you can modify it for your use cases and and change it how you want so if you are interested in a small and better application and you want really high performance single threaded workloads you can design a core implementing the ISA to do that and if you're not interested in that and you want a whole heap of cores as many as you can possibly get you can design cores to do that and everything's open and you can customize it for your work case so what is Western Digital doing at RISC 5 so a lot of people always ask me if this means that because Western Digital people are doing Linux work that are we running Linux on our hard drives and Linux on our SD cards and and no we're not we none of our stuff like that runs Linux um we are here because Western Digital is publicly committed to RISC 5 so we ship a billion cores every year and we are publicly committed to transitioning those to RISC 5 cores and so part of that is we want to encourage the community to work with RISC 5 and support RISC 5 and that's why there's a lot of RISC 5 where every RISC 5 conference is always the Western Digital people there who are doing open source work we're contributing to the Linux kernel we're contributing to Uboot we're contributing to to all these open source projects because we want it to work and we want it to and it benefits us not just directly in our products um so that's why we support an ecosystem and we have other things so I'm not going to talk about it much in detail here because it's probably a kind of off topic but there's an on the extent which has done the bottom there which is a cask of here interconnect so again we're not just we're not just contributing software also contributing hardware and hardware design specifications so we we really see this as a fundamental thing in the future and we're publicly committed to it and putting effort and resources into this so any questions on that all right so this is kind of the sad part now so how can you use RISC 5 today so QMU is a great option and it works um since 2.12 but really that had a lot of problems in it so I would say since 3.0 QMU we've had RISC 5 support um so most modern distros should you should be able to just install it through your package manager and you should have RISC 5 QMU there if not you can use the master branch which always has better support and has new things in there and so the QMU machine has a the QMU RISC 5 has the VRT machine which is a a virtualized machine it doesn't really exist in the real world it's just a QMU specific thing and that has the best support so if you're using QMU I would always recommend to use the VRT machine it has as a 4.0 which will be coming out in two months or so it'll have PCIe support so you can attach virtual graphics cards to it and get the displays running inside QMU and really with PCIe in that you can attach any PCIe device that QMU supports so you can attach USB devices things like that so the VRT machine is generally the way to go the QMU also has support for the sci-fi machines the sci-fi view and the sci-fi E in some support so that's another option if you're interested in those and QMU will generate the device tree for itself and pass that to the kernel so you don't have to worry about making sure you have the right device tree built from the kernel and include that and make sure it matches the QMU will generate it for you and pass it into the kernel so as long as you have a kernel and a boot loader which I'll talk about in a little bit as well you can boot up into Linux and open a bed is a great way to do that and the other option is the high five unleashed so I don't have a laser pointer but it's I don't have the mouse however it's the one on the bottom the little small one it's up here too the one here so it's the only risk five capable board of booting Linux it's the only ASIC that's publicly released that can boot a Linux kernel so it's a four core board and it's it's small one so it has ethernet serial and GPO is pretty much all it's in there so it's unfortunately very expensive so it's it's great it's fine for companies but for individual contributors and hobbyists it's it's not down there with the it's not down there with the Raspberry so it's this one so it's not down there with the Raspberry Pi is in that you can have five in the menu of house because you just keep buying them for some reason so then the other part that is also up here I know it's probably a little hard to see but if this top part this is a micro semi expansion board so this is a giant FPGA in the middle that basically converts PCIe traffic to chip link traffic which is what the sci-fi board understands and this allows us to attach two PCIe cards the graphics card up here which I also have here and a USB to PCIe device here which I also have here which gives us graphics and USB and in this case there's a SATA hard drive attached and there's a you can't see but there's NVMe drive under here the solar state drive I don't have those in my demo today but this setup is unfortunately very big as you can see here and is also quite expensive and hard to get so unfortunately the hardware available today to boot Linux on is is a bit of a struggle and a pain point but there are embedded hardware there's a Vega board and a sci-fi have a Arduino like board that you can get if you're interested in doing any more embedded works or Zephyr and stuff works on those otherwise QMU is a great option anyone have any questions on those the micro semi board was they had a a crowd supply campaign and sold I think 20 or 25 of those you have to talk to micro semi if you're interested in in getting those micro semi have publicly announced that they are basically going to release this top board a new version of this top board with hardened RISC 5 cores in the FPGA so it'll be a hardened for hardened cores and FPGA fabric they announced that last December they don't have timing or pricing on that yet so hopefully when that comes out we'll have a it'll be small on this bill and still have PCIe and and stuff like that so it'll be quite a nice board but I don't have details or timelines on that so mm-hmm oh I think the yeah you used to turn a hardened core hardened RISC 5 core yeah hardened part of that oh hardened just means it's a actual ASIC it's not just implemented FPGA so it another thing you can do today is if you you can buy an FPGA and put a soft core which just means like FPGA generated bitstream core on there um which comes with a huge performance decrease right because it's running an FPGA but it's cheaper and easier and so someone in the earlier talk was asking there are a lot of open implementations as well if anyone's interested in in FPGA development depending what FPGAs you have how big they are you can put smaller 32 bit cores all the way up to 64 bit ones that run Linux and and that's an option too if you if you kind of know how to do that but it does come with performance impact so if you want quick performance Kim you is still the way to go as it it does compete at the moment with hardware in terms of performance speed talking about open cores uh Western Digital recently announced Swerve so Swerve is a open source core that we released so I didn't work on this uh I have some my colleagues did so I won't talk about it that long the people always ask so I figured I'd talk about it a little bit um so you can go download this today from GitHub um it's under the Western Digital GitHub repo and it's a embedded core um targeting 32 bit with the IMC extensions so it's uh very well designed for single threaded um high throughput applications and it's licensed under the Apache license I think so if anyone's interested you can go download it if you have an FPGA lying around or if you're going to buy one you can put it onto that they accept pull requests and issue tracking on the GitHub so it's it's a real open source project and we accept contributions back it's not just like a toggle dump and um so I'm not going to go through everything on it if you understand what it is and are interested definitely look it up there's a lot of information about it and this is some more stuff on it so it's a multi-stage pipeline it is quite an advanced core it's um again I'm not going to go through that if you if you understand that interested I would definitely recommend looking it up and we do have some performance numbers on it though and I think in the earlier session people asking about performance you can see there kind of how it compares with other similar CPUs so this is all scaled per megahertz and per core so it's the swerve one is designed like I said to be very single core and very embedded in specific so um any questions on the swerve core uh this is the the core mark mark the core mark score um which is a like a performance benchmark yeah yeah yeah so I mean it's it is scaled for the threads right so obviously you're going to get a lot more out of a huge server class machine but yeah it's it's pretty cool it's the swerve hardware world is is getting there right because the ISA doesn't specify how your pipeline works and how your how big your caches are and all that stuff so so that is up to you and and as the process improves people are going to get more performance so that was the end of my kind of overview of risk five now until the more open embedded risk five part so if anyone has any last questions before you're really confused and have no idea what risk five is then now's the time to ask so open embedded risk five support is pretty good so ken talked about it this morning in a lot more detail i'm not going to go into into all that detail again if you miss that and interested you should rewatch his talk or you can ask me after this or if you're interested in something you can ask me about that but overall there's a meta risk five layer it is very well maintained and updated all the time and we have linux the octo support in the qmu machines and almost mainline on the hardware so this demo here which i'll get to is the 5.0 kernel with about 20 patches on top and most of those are going to be in the 5.1 release they just kind of just miss the 5.0 merge window so it's very it's very upstream first mentality of working on everything but at the same time not everything can always be upstream straight away right so everything's being bumped across so most packages almost all packages have some master support now so lvm it has experimental support rust has support i heard go lang apparently has support too because limited things aren't there now sometimes you definitely see the support's not in the last release or the release being used so like gdb 8.2 for example only has very basic support and that's what open embedded uses so in the meta risk five layer we're using a much a newer version of that because that's where you get linux application debugging and things like that but it is there and master it just has to wait to propagate to a release so again this morning open sbi came up a little bit so open embedded has full support for open sbi now so in the risk five world the sbi input protocol or implementation is what the linux kernel uses or your your hyper your operating system uses to call down into the firmware below it so on it's similar to the ARM psci calls for everyone is familiar with that and it basically means there's a privileged level of the machine mode where your firmware runs and above that is the s mode where your operating system or your hypervisor eventually could run and that calls down into the machine mode to do certain things and that and for example a common use case in the arm world is or in the risk five world will be to turn cores on or off so you can power control the cores things like that and so at the moment we don't have a specification where the default is what sci-fi have in there hardware because they have the hardware and we don't want to break that and but that has all sorts of issues and and it's not future proof to not specify it and and so the risk five organization is working on writing sbi specifications so it'll be ratified to the normal process of the isa goes through to specify how kernels talk to the firmware below them so you can swap out linux and put bsd on or swap out the firmware below and put a different firmware on and it's all into our freight between them but as part of that open sbi is an implementation of that so the idea is open sbi is a library that you can use to implement these sbi features in your project so if you want to use u-boot sbl then boot to the linux you can put open s link with lib open sb lib sbi and don't have to rewrite the open sbi implementation and you can use that with your u-boot or your core boot or grub or linux or whatever you want to do at the moment we just have a reference firmware implementation in there which is open sbi and that's what we use because no one else core boot actually does have some stuff but no one else at the moment has is implementing this but that's hopefully the plan and that's what we're going for and and we've been talking to u-boot people and core boot people and this is very new so if anyone who's ever done risk five work in the past you would have heard of bbl which was the berkeley bootloader sometimes you also see a caller's risk five dash pk the proxy kernel it was awful and a real pain and so we've gotten rid of that and that was the point um you still have to link in the vm linux into the bootloader and that was the only way we'd boot and qmu and on hardware and so now we're moving we're not there yet and in the open embedded world but we're moving to open sbi to u-boot to linux kernel so we can follow the full standard flow the other embedded systems follow and this means we can get access to the kernel ci as they want new boot support in there and and everything like that so we're moving towards the standard kind of general approach and right now in qmu for example you can specify open sbi and the kernel separately so you don't have to rebuild your kernel or open sbi you don't have to rebuild the other one when you just change one so we're all everything's kind of getting there and boot was a sore point for a long time and it's for anyone who has used in the past it's much better now and so if anyone's interested in open sbi is there again it's public pull requests are welcome issues are welcome testing is welcome everything is welcome it's open we want people to contribute you want people to use it we we just want people to yeah to use it yeah oh yeah the open sbi that kind of takes some of the function of your secure boot on an on-processor and call boot zero that initial boot instead of your main all that kind of stuff before do your authentication as the next level of the boot so kind kind of so open sbi is more targeted as a reference implementation it's probably it doesn't have secure boot support now i mean the hardware wouldn't support it if it does but it doesn't have secure boot support now if someone wanted to add that we would probably that's probably fine but open sbi is not the only option so it might end up in a world where open sbi is the is the the boot loader or in the in the firmware you use in qmu and on hardware you vendors go with uboot spl and and use that instead because it has more features so it's we're not trying it's not the only option right it's not like in the arm world where etf is kind of whatever one uses we the library part of most open sbi is also the interesting part and hopefully everyone can implement their own boot flow as they see fit for different use cases right if you're running in a more desktop centric one you probably want uefi and grub and all that stuff and if you're really embedded you maybe just want to run straight in the end mode and not use this so it's it's not there um we haven't thought about it too much in open sbi but it doesn't mean that you couldn't use it in something else any last questions on open sbi all right so why i use open embedded so my colleagues ask me this sometimes too so fedora and devian both have great support um and open suzer has support and i think alpine's getting support and everything although your more traditional desktop distros are getting there so why do i use open embedded and so what i like about open embedded is it gives you a really good idea of how the ecosystem is doing it's very easy and very clear to look at the meta response layer and see that okay i need these patches in this project to get this to work and you can watch it if you watch the meta response layer over the last year it went from i need these patches and all these weird places and now it's kind of we're just believing them and now okay we're almost there and i need this weird g i need a gdb thing i need a fork of glibc for 32 bits support but otherwise it's all kind of coming together and so open embedded is really easy way to see that so traditional distros i guess you go find wherever fedora keep their stuff and it's there somewhere right but i like it that it's all there in one place where i'm working so it also allows it to easily add packages so in my demo i have some text-to-speech stuff that i'll use and mimic is what what they use in in this demo um and it wasn't there it's probably not it's not super well common not super commonly used but open embedded it's very trivial to add the patch is actually upstream now um and you just everyone's used it it's probably like five or ten lines right it kind of handles it all for you you throw in a url and some license information and that's it it's very easy to add packages and for developing it's really useful because okay i need this package i can bring it in and then you work on it and it's a lot easier than if fedora doesn't have something you have to deal with then installing all the tool chains and making sure they work and cross and compile it natively and and all that it's it's built to add things so um and the cross compiling it compared to native compiling so it's probably not everyone agrees but i think cross compiling ends up being a lot easier most traditional distros always native compile so for dev devian and fedora compile inside qmu or on boards and anyway people can do what they want but i'd like to cross compiling um and when i was doing the the qmu port for risk five hosts so that is qmu will run on a risk five machine and then let you run an x86 or any any architecture application in yeah you like cross compiling because you can run your development environment on a big server and have fast return on times whereas if you're doing cross compiling um well okay let me finish this and if that's definitely helps it definitely helps having a big server but um first of all when i was adding this qmu support the cross compiling meant so i meant i had to compile qmu to run on risk five and then run it on risk five and test it and it would crash and debug it in gdb and everything like that and cross compiling meant i could use my standard flow that i always use i could edit it with all my everything set up i knew what i was doing i had two monitors i was doing you could use those it backs up every night i didn't have to worry about any of the stuff and i used the same flow i use every day and i didn't have to change anything with if i was doing native development i would have had to dealt with setting it up in a different setup and backing that up and committing that and how do it and getting keys in there and all this stuff and the cross compiler just made that easier unless you could do it natively there's no reason not to i mean there's no reason you couldn't but for me it was just so much easier to do it all there i didn't have to change anything i especially did they can open embedded would just build it in for me start it and it was often running and i could see it crash and know that didn't fix the problem yet so it was yeah so that's why i one of the main reasons but also i do have a big server and that helps but um but you also have board issues right so you see especially early hardware is not always reliable and something worse than the hardware then board crashes and did my build break did i build there's something wrong with my build or did the board just crash it was a someone kicked the power supply and that was it it's so yeah i don't want to get too much into the cross compiler it's native compile but i like it so so open embedded images default to being small so one of the main things i also do with open embedded is build tests for running qmu so i want to test that either i didn't break anything or no one else broke anything so today master changed and make sure no one regressed the wrist five build and you can use anything to do that right but the open and better ones are nice because i can build the bare minimal to get a booting system and then i can add the packages i want so i have a i mean core image minimal has almost nothing in it right so i can build core into minimal and pull in stress and g and a few other things that i want and boot that and it's very quick because not starting up a graphic stack and none of that stuff and so boots and i can run my tests and then kill it and i don't have and then the images are small i can put them on a server and back them up somewhere so i know i have these golden images that this works and if this test doesn't work then it's qmu broke and not something else broke and so i like that and i like that they're small and again you can yeah oh um i didn't yeah i guess i didn't say what it is sorry um it's a destroyish thing that it builds you let you let build embedded linux um yes yeah yeah sorry i kind of when i prepared the talk i was like it's a open embedded room so i didn't bother explaining it but um yeah sorry if anyone's confused it yeah it's a build system to build embedded linux destroyers and as well as all things but that's one other thing that does um and so it allows me to build clean images so uh yes yeah it pulls it in yeah but i mean it's similar to if anyone's used build root it's kind of somewhat competing to build root um is that clear up you okay um oh yeah so it allows me to build clean images so if anyone's so again kind of back to this when i was developing with this stuff so you boot up and okay i need new version of gdb because only this guy's fork of gdb has this working thing and so you have to in a normal distro you have to pull that in and build that and it makes you save that and keep or keep the steps around to rebuild it then open embedded you can pull it into a layer and you still have a layer but you get it working and then just commit that or keep it in a branch or stash it or something and you don't have to worry about it and can delete all the images and rebuild overnight and come back in the next day and it's the exact same steps nothing's changed i didn't have to make sure i wrote down the steps that i used to build this weird thing and it's always just committed in there so that's what i really like about it too is it keeps everything bundled together it's easy to manage and easy to maintain um and the packages on that kind of similar note packages are also very recent so gcc9 is already progressing all it's not merged in yet but they're open in better branches now with gcc9 and so we use that for example we caught bugs and open sbi that wouldn't build with gcc9 and open embedded was the only one at the moment who's testing that and they caught the bug and so the recent packages was very nice and if they're not recent they're very easy to update you just basically rename a file on it and change some shards and it updates the package so it's it's a really great tool to kind of always test for this community stuff to see that gcc9 does work and with us great and this version i know doesn't so it can easily bump it up and then contribute it back to everyone that's my last slide on why i use open embedded so um well how to easily patch external projects so as part of this demo i had to build cute or cute 5 and kde and all these other things and it's really easy when it doesn't build to look and go okay why doesn't it build and with open embedded it has external source you can point at an external source tree and do a development and there it has dev tools you can do all of stuff with that and so it's very easy to fix the build or fix a runtime problem get the patch out make sure it works test it and then you have a patch ready to go you just send upstream and hopefully that gets merged or we send it to the upstream to the meta layer and it gets put in there for the time being and while you wait for it to get merged depending on on the upstream project but it's very easy to contribute to other projects from this and so um i'd recommend if anyone and you'd also the advantage of that is you don't have to learn new ways of building things so cute has this very complex way that they build things and i don't have to know what that is and i don't really care because they can open it better to handle it for me so i can test my package in the way i normally do and move on and don't have to learn all these new different ways of building things in every different project so i mean if it's a build problem then you might have to learn it a little bit but you don't have to change your flow every time you encounter a new thing and the last one's not that exciting but open and better will also build tool chains for you to use and so you can build tool chains and then use those and so it's a great way to test if Zephyr works with the latest GCC 8.3 release and instead of some fork that someone had at one point that used to work so so any last questions before i talk about why the problem pain points so it's slow it's really slow even with a even with a big machine it can be it can be really slow so if you if you're smart and organized you can probably build overnight and you get in the next day it will cache things so you don't have to rebuild things that haven't changed it's pretty smart so if you build overnight and you come in the next day you're probably not too bad but normally and i have a problem but building cute is takes forever and then the kde stuff on top of that is also takes forever and when you're doing hardware then it has to download the sd card and you have to swap the sd card out and then the dependencies are a pain so if you change one thing that cute depends on open embedded the file a bit big decides everything from there on has to be built so you spend an hour rebuilding cute and all the cute stuff and all the kde stuff and then the sorry i don't i don't have a solution or it's like oh this is obviously stupid but it's it just works so that's so native compiling i guess would if you use a more traditional disk or app installer dnf installer would be way quicker but it does there are disadvantages um and is there easy access to a package manager although i think i'm just using the wrong distro there and yeah i tried i tried and i couldn't get it to work and then i've read some places that didn't really speed things up because you had a lot of network stuff anyway so oh i don't i don't i mean i don't have a problem i don't have a solution i do oh yeah but they're doing a lot of stuff it's just i mean it's like complaining it takes time to build things it just is what it is so um so i don't yeah i'm not i don't mean to sound too complaining it's just that's the main pain point there and no easy access to a package manager i think this can be fixed if you're smarter than me use a distro um but i don't and so the problem is when you especially when you're trying to get a demo you start you download it to the package start building or start running something on the board and then boom command not found and you have to rebuild the thing and then re-flash the sd card and takes a while so if you have a package manager um that will help you uh yeah yeah are you just are you trying to get a bootable version of the linux on this risk board or are you talking about difficulties you're running into with development for um i'll answer that question when i show you my demo that should explain it so this is my last slide before the demo slides or if anyone else has any more questions uh you had to make changes yeah so the work had definitely been done before i started in this five worlds i wasn't there in 2010 when it started um but generally i'd say most most packages don't have any architecture specific stuff right besides obvious kernel gcc any tools all that stuff most packages will just cross compile maybe the hardest part is they do some hard-coded config check and you just need to add support for that but most things cross compile fairly easily once you have source um then linux and gcc is a whole different and qmu is a whole different problem and that needs people who know what they're doing so gcc port needs someone who knows risk 5 and gcc and that's a lot of work um but and kernel is the same and qmu is the same but the general packages normally kind of just cross compile that's not too much of an issue and my demo kind of shows that too so um okay so my demo is mycroft which is an open source speech assistant so similar to amazon's alexa and and all those things um it uses pycin as a wrapper and calls into these c binaries that it builds on the host so it builds itself on the host which is a pain when working with open embedded but and it's officially supported on the rosary py as well as some other hardware they have but this is a demo of it running on risk five um and it's also the demo of plasma mobile which is the mobile interface version of plasma which is the kde development the kde environment um and it's built on top of qt which is where the qt thing came from and it uses kwin and wailand so there's at the bottom is that's the the risk five meta risk five github link and then in there there's a docs plasma mobile on unleashed which also how to do the plasma mobile part there's no steps on the mycroft part because it needs a python virtual environment and that's the pain so it's not really clean and easy to set up but if anyway is interested you you're welcome to email me and i can tell you how to do it and and everything but so at the same time it's also connected to this thing you know cables can be um but which is obot which is like a kid's toy that you can just control and and make the lips move and the mouth move so this is just to make it even more better for a demo so this is running on um i can put it out but you can come see it afterwards if you want but there's a high five unleashed here which is the high five board the micro semi board which is the micro semi board i talked about earlier then we have a pc i e to usb to give us b input and this is an amd graphics card r600 something so it's just a standard nothing special um like a standard pc i e graphics card hopefully this cable kind of hangs down uh let's do the microphone first i have to reboot to get the thing to it so hey mycroft what time is it hey mycroft what time is it you can ask it a whole heap of different different questions um i'm not sure how interested i just keep asking of things so i might wait till the end if anyone wants to come up and try and see how many stupid things you can ask it until it breaks uh oh actually there's a good one hey mycroft what is risk five okay later but then it does say risk v which is a bit annoying but it will normally answer the risk five one hey mycroft what is risk five that's architecture based on established reviews instructions and computing principles okay so yeah i think the cable problem no i think i'm okay yeah so i can reboot it see what happens all right so while it's coming up which it definitely should um does anyone else have any more questions oh so i think i answered your question before so cute is used to kde and or plasma and that's where that came from i was just thinking you were talking about day to day pain points um well yeah oh yeah not necessarily just for the getting out yeah it's a little slow they don't it has a few minutes still any other questions or we can just wait until it comes up doesn't seem like there's any real time requirements in your application is it just because it's a small footprint that you're using um i mean what do you what do you mean real time like real real time so i guess the focus is not on the real time but on the embedded uh yeah so i mean you're not launching any real time threads are you no no um yeah i mean my this is just a cool demo i it's not kind of tied to any product i'm sure there's people looking more real time systems for the iot stuff um but yeah my focus is linux and linux the linux world and the cool stuff so you can show off with that so it's just kind of just where i focus on instead of oh yeah no sorry this is this is a hardened wrist five core so there you can see but this this um fly five unleashed the high five unleashed board is a four hardened essentially five but four hardened wrist five cores in there and the FPGA part is just to convert PCIe traffic to chip link traffic so these are hardened cores not soft FPGA implementations that's a good question um it's okay it's coming up now yeah so so there's a there's a few reasons so it it boots very quickly um it's not it's if you you can use it like a normal system um in kind of day today like you don't notice it's that slow part of it is a lot of this stuff to do on javascript and and things like that which has no optimization done for wrist five so even though it compiles and runs there's a lot of work that goes into lots of things to make them run faster on different architectures and wrist five at the moment especially with javascript has none of that so it's all it's a lot slower because of that and generally in the whole in other places there's there's a lot less optimization that's being done compared to other architectures so and you can see so this is a a touch based um operating system so pretend you have a touchscreen i do have one here but no one be able to see it so but you can see oh there we are i can't read what that says see i mean it's it's not too slow right it's it's kind of usable um so it is definitely usable as in terms of day-to-day stuff it's not like fpga implementation would be a lot slower was there another question you said i don't know what yellow dog is um but it's just a distro okay no no no um that's okay um so yeah so fedora runs on this devian runs on this um so so open soothal runs on this i think alpine have something it's probably missing some um you can use a lot of distros on this so this is the distro i like for embedded stuff but you could boot fedora and use dnf install and rpms and all that fun stuff and boot devian and use devs and um in this morning's talk kem had a there's a great slide of the the halving packages across compiled and devian and risk five is up there with higher than iatc and itanium and all that which isn't that surprising i guess but they're higher than a lot of other a lot of other cpu's um and a lot of other architectures so the devian and fedora support is great open and bed support's great it's yeah there's a lot of options if you're if you're interested is it possible that we can see a risk five based phone i mean i i don't know i wish it was digital i don't know um you can make one i i mean yeah so i mean the problem is this is cool but it has no power management so it it it's it's plugged into this huge transfer this huge power brick here um we're not there yet right the phones are really hard um we're not there yet what's your digital going to get back into the general cpu business i'm not announcing anything so i don't know have a look at our press releases let john have the mic so the the the obvious question is is western digital generally referred to in in the in its current generation as a storage company you are making a cpu when can we possibly see these delightful cpu's bolted directly to your storage engines and the network plugs that we can stop having our self clusters on anything but other than you know a hard drive plugged into a switch i don't know you're asking the wrong person i don't know since we're talking about you know products and weird stuff okay so how long before we have risk five in that little micro p controller that's in all the emmc devices that nobody knows about the ones that i used to program i don't know i don't know if i don't want to give you a date like i don't know i'm in the research i stay away from what stuff i spend time getting demos broken hey so what's going on here i've never used it on a projector before oh it seems like it's might have crashed but i go to java western digital in the research department um yeah so yeah i mean so western digital is is we have people who adjust job is to just contribute risk five stuff and so this was a great excuse to do that so yeah well with that i think we're out of time so um a huge thanks to all of you guys nice demo and it went well yeah it was all right we have some stickers up here some open embedded stickers if you're interested we are continuing with our open embedded summit at uh four thirty we have john mason from art he's going to be talking about how do you use open embedded to do kernel development so if anyone wants to come and look at this as well you're welcome to come up and have a look so mic check one two yeah okay okay so hello everyone can you hear me day oh still talk yep oh that's loud your voice is not as loud as this yeah but i'm not trevor so um okay it's me and i'm gonna be talking in probably one minute so set your watches you get a countdown um yeah are we good i think i'm the very model of a modern major in general i don't know the rest of it something about a mineral something yes plants yeah i took no uh acting classes you know i'm not a doctor but i play one on tv check check check yes in the united states we do not have one two yeah no we have two we have lines no no no you don't have queues one two one two yeah and we also don't have queues the utility is xz by the way it's not xz i want you guys to talk to me so anyway welcome back this is oh i'm gonna hit record anyway welcome back this is the final talk of our open embedded summit uh thanks so much for joining our inaugural little experiment here i think it's going great so far yes thanks we have stickers up here open embedded stickers if you're interested thank you john they're free you just have to say hi wait or not even say hi just take them this is the last session if they're not there you have to take them home so for our last speaker we have john mason from arm he is actually the yachto project uh liaison for arm i guess something like that yeah okay and um so he definitely is been in around the open embedded sphere for quite a while and um formally uh john was doing a lot of kernel development so let's kernel development work and so when he started at arm and they said do this open embedded stuff he said but i like doing kernel stuff so he's combined those two and it's something that we get asked a lot last year in edinburgh i gave a talk on dev tool and dev tool is a general tool that allows you it's part of open embedded it allows you to um basically work with your sources if you're working if you have a package that's giving you trouble you can work with that one package specifically by using dev tool and number one question everyone just wants to know how do i use dev tool to play with the kernel and because i was sort of a general dev tool person i wasn't like specific on kernel stuff so when i bumped into john at uh fummer's last year in vancouver uh he said hey i want to give a talk and i was like that's awesome that's fantastic what do you want to talk about and he gave me a list he goes i don't know i'll give you a list gave me a list like four or five things and the first one was oe and kernel i said yep that's it so without much further ado thanks john take it away thank you so i appreciate everyone staying to the very uh bloody end here of uh last session of the last day so thanks for bunny um john mason i'm presenting kernel development work flows using open embedded it's a little bit gonna be a little bit more generic than that but uh i think i think everyone will enjoy it so hopefully everyone here knows what open embedded is so i'm gonna go through this a little quickly um we can get out here a little earlier um so everyone knows what a distro is raise your hand if you don't know what a distro is okay without trolling okay um so what's open embedded so open embedded it's a build auto so this is straight from probably wikipedia build automation framework and cross-compile environment used to create linux distributions for me this is mine not just embedded devices you actually can use it for other things and i think that open embedded actually needs to get a lot more credit for being things that are not embedded um but that's kind of outside the scope of this talk but i wanted to point that out um runs on every major linux distro i mean maybe not some of the the smaller ones but um all the ones you want to use it it'll run on um all major cpu architectures including risk five which uh surprised me a little bit but uh congratulations um but it also works on arm and works on arm very well my employer obligates me to say that um and has packages for pretty much everything you can think of and if it's not there it's pretty much trivial to add so how they fit together it's uh recipes contained in layers um you can fetch and build and you make an image out of that so here's a very simple recipe it's it's for east tool i believe yes east tool which you're going to see too much of in the rest of this um so um of note you can see the um the the source that it's pulling it down from the internet it does some md5 sums to make sure that it's actually what you want it to be and then some basic build instructions and layers are essentially how these recipes are organized into logical blocking groups if you will and this enables you to only add the pieces that you need which while maybe not significant if you're looking at a traditional distro for embedded or something that you that you actually care about being tiny that's actually pretty significant but this also enables you to replace layers which i think every customer out there really needs to be pushing for um using open embedded just solely for this like if you're making a set top box for example and you're some vendor i don't want to name name plus uh you they give be litigious but um you can be you can have an x86 board and you're like oh this x86 they want to charge me price a and this arm vendor wants to charge me price b and it might be better your entire stack you can just replace one layer for the other and and you're done everything in theory should be working the same and you saved your company millions of dollars why not um or you can replace one GUI for another or one toolchain for another which is also popular changing gcc for llvm and things like that um and then here's the layer index and and disregard all the no you can't see it how can i show this okay well sorry i'm trying to show the layer index let's just say that there are thousands of of layers there are at least hundreds of layers that you can you can show sorry it's not showing up um these are the two files you you really can mentally modify to add layers and um specify the packages you want in there again if i'm going fast feel free to raise your hand but i have a feeling that most everyone has a pretty good grasp of this um so output so this is the the big difference between a normal distro and uh using open embedded is a normal distro you're going to have an installer you're going to pop it in you're going to go through the steps to configure it you when you use open embedded you're going to have a system image because you know what already is going to go in there um you can also precompile rpms or devs and if you're really crazy you can cherry pick binaries out of the um the build so like if you were doing an entire build and you only care about the kernel you could just copy the kernel binary and it's probably not a great idea but it's but it's possible and how to deploy well it depends on your use case so now here's uh what everyone hopefully came came for so excuse me one second so what are you developing so a lot of kernel developers also have their fingers and pies like boot loaders user space packages that are very closely tied to the kernel and you might even be doing pulling it all together making a distro so why would you use open embedded over just using a simple distro so that you can just hack on the kernel like thousands of other kernel developers do well like trevor was saying dev tool dev tool really is the the hammer that you can beat every nail with um it gives you the ability to replace or upgrade packages quite easily which alice was saying in its previous talk it's it's almost trivial to update a version or downgrade a version um so you could have customers that want a certain version of gcc and you can just bump gcc recompile everything and make your life you see that way um similar to what i was saying before you only have to install what you need and it's um easier to debug out of the box because you're compiling everything locally so what's the downside here's my gratuitous xkcd it takes a long time to compile and you're going to in the demos i'm going to show a little bit later there is no compiling because we would be here until this evening it's because i'll be compiling on a vmmi laptop and it'll be forever you don't want that but i mean if you have more idle time to do other things like answer emails from your manager maybe that's a good thing i mean managers like being conversed with frequently so how do you modify the source of an existing recipe so there's really two ways that i've i've used there might be other esoteric ways but i think these are the a good way and a and a stupid way using dev tool where you can fetch source or you can link to an existing get tree or you know source tree that you have that's not in version control or you can modify an existing recipe to point to some some tar balls so dev tool so you can hook in using an existing get tree which i think is probably the the way that everyone's going to want to do it but you can just use random source that you've downloaded off the internet which can bite you in the butt and um or you can use it to simply just fetch what's currently there and i'll i'll show you how to do that in a little bit and the benefit of linking to the existing code is you can develop and debug what you have locally and you can also like i said earlier use a older or newer version or you can modify an existing recipe and this is the the stupid way i was referring to earlier um you can hard code pass in in a recipe to point to a local tar ball or a local get tree um why would you want to do that you can actually make it more permanent you can modify and instead of pointing to um let's say using e-tool again using e-tool on kernel.org you could use your own modified version that's internal to your organization so you can modify the path that way um or if you don't like get or any other version control system and you just want it to work because you needed it done yesterday and you don't want to learn because learning is hard then you can hard code hard code it this way okay and time for the demo okay so it is showing um all right close that table already okay so this is the the avert your eyes you're seeing under the uh okay so in this uh example here i have an existing linux get tree right it's 5.0 it's it's exciting it's the latest and greatest um so i did a few changes here just to show that i am not just like tricking you in some way so um i added scale to the to the version because can everyone see that by the way is it too tiny blow it up okay uh that didn't help very much at all um and then okay i'll try is that any better is it okay sorry nothing's about it's gonna get um so i modified um the make file to show it um i added a kind of bogus um entry into the vertio device driver to say that we're at scale so it's just a a bogus entry and i also modified the um the header file so it all you're gonna believe everyone believes that this is a modified kernel right and similarly oh dang it there and then i modified each tool as well similarly um i'm modifying the user space application so that you can actually see that the flag is now being set so i think that's one of one of the cool things that you can do with open embedded quite easily is you can modify multiple sources that are hooked together so i've modified both the kernel and the user space app that's talking to the kernel and again it takes a while to compile but you can you can use qmu in this example or you could use a live board um which is probably much faster to to show that i have a modified kernel i've modified each tool and they're both tightly coupled now and you can do the same thing if you're modifying a boot loader in the kernel all these things that are very tightly um coupled together and this is unfortunately going to take a minute because it's inside of vm running qmu and they don't disregard that uh that x error and the other errors that are are coming because this was uh i needed to debug something this week and um yeah that's a little tiny but um so hopefully maybe not in the back but most people hopefully can see that uh the the version is scale and in the kernel config i added oe just to make it even more um proof that i've changed it and let's do a little e-tool here normal e-tool and then dash i scale yes so i was able to do that the changes this morning took one or two minutes the compiling took you know along for me to take a shower and get breakfast again it's inside of vm um okay and um what you'll also want to see is what i changed locally to make these changes um happen as soon as it shuts down and this is within pocky because i didn't want to roll my own distro so you know no use reinventing the wheel okay you don't care about the first one or the second one so what you'll see here is the the e-tool source that um that i was referencing earlier i was able to um essentially modify the source put the tarball locally and instead of having it you can't see the red is that now readable sort of maybe better sorry so i changed it from pointing to the real world to a local file you could point it to a local file system you could point it to your local um however you host things internal your local web server or whatever and to point the kernel there i did a dev tool i did this so essentially what this is doing is um you're choosing to modify an existing recipe because unless you're doing something super unique you're always going to be modifying an existing recipe kernel in this case um and the end is saying that you already have an existing in um get tree and this is the path to it so it's pretty trivial to add and um sorry and to reset and as soon as this command's done it is no longer pointing to it so you could simply let's say your get tree is just not happy and you've screwed it up beyond all repair and you want to get back to a sane state you can say okay i'm not i don't want you to point to this anymore or let me just go back to the last working state and so that this is now pointing to nothing and it still leaves your tree around if you wanted to try to fix it later or whatever yes there's a one second i think for people so i think this was a dev tool is in fixed work with kernel recently so in older releases it didn't work well is that right um i've been using it this way for a few years so um so it's possible that in in the versions prior to that i've used that i've used that it was broken um glibc was broken okay i i haven't modified glibc but but that is one of the the cool things is um glibc you know if you're trying to modify an app that uses glibc you can modify them both together in the exact same way so you know kernel and kernel things are cool and they get they get lots of attention but you can use it the exact same way i mean much the way the alpha was referring to earlier i mean you can modify lots of pieces very very easily and when you're doing board bring up or even um trying to do an entire stack it's it's quite quite beneficial and and then just um to relink it it'll it'll take a few seconds but it's done and it now points back to the the exact same place and conversely if you liked your changes and wanted to keep it a depth tool finish would create patches for you that are then fed back into the build system so those patches are waiting and then you can send those patches upstream if they're you know legit and ready to go exactly it's quite the tool yes yes it it is a tool that needs to get a lot more love and attention because it's extremely useful um okay and then um does that tool finish actually work though in this case where you're using an out of build get check out yeah i was wondering about that use yeah i was wondering about that too usually when one uses depth tool depth tool will create a workspace for you and then you do all your stuff in the workspace you know and that's what i was just about to show so okay yeah it certainly works if you're using depth tools workspace uh out of tree and um kind of globbing on to the point that trevor just made um you can actually just modify it locally you don't have to point to an existing tree you can just check out what's currently up there into your um into your build environment it'll you'll see it here in a second as soon as it finishes it will give you the path that it checks out into you can modify it locally there and then you know have fun that way it's a long 98 percent well how much money you got um no i'm not worried about the error but it's because this is a dirty tree if this is not a dirty tree you wouldn't see that error but because it because i've made some modifications for linking the um and building the previous get tree i had to um i was on a different version of the kernel that it didn't so by i turned off the kernel version checking so that i could build a newer version than what it wanted because i was building 5.0 wow this is taking a very long time um okay so how about in the meantime even because this shouldn't take nearly this long um another vm which hopefully is it still very difficult to read um i'm gonna show here one thing you can do if you're trying to change kernel config so changing a config um you know it can be a little bit of a pain in the butt um well it's not a pain in the butt it's a little less obvious um you simply pass the command hopefully i did that correctly okay well apparently that's the one thing i didn't do because i assumed it just is going to work out of out of the box okay um okay and both of them are well we've had an entire day of open embedded and this is the first time we're seeing open embedded actually you know running so well so i guess while we're waiting and infinitely for these things how about an infinite amount of questions any questions in general because we're getting towards the end of the uh of the show here so what about the deployment workflow when you're when you're doing something like this i know when you're doing user space apps you can use dev tool to actually push the binary without rebooting and stuff but is there anything that can bypass the need to to flash a new nd card well so um you can actually have um oe output as a as a dev or an rpm so that if you're doing some user space thing or even just the kernel you can spit out a new version as a package and then update your package yes the other way of doing it is actually if you do a tfgp kernel and then an nfs root file system it then it doesn't matter and i do that all the time from my actual project build so yeah and also there's um what dev tool there's dev tools deploy and uh touch as well and undeploy so there's many many ways of doing it uh fd card should never be part of your development workflow basically what it comes down to if you're using it an fd card in your workflow you're doing it wrong so well i mean you you need to deploy your final image and theoretically it's on an aspect of the deployment team okay so this one's finally finished um so it's putting it inside workspace sources and in this case one xiocto and just to prove that it's there it's the kernel that you would expect and you can modify it and it'll all be here your build objects are not going to be here just the source the question was how is the other vm doing it's still running oh there you go and uh just a normal menu config it opens a new tab or a new window depending on what your distro chooses to do and i haven't found a rhyme or reason to it but uh it's usually a new tab but for some reason uh is it a local comp that okay yes exactly you can save the config and it will save it just like you normally would um sorry not much more exciting than that i mean you can um i think the only caveat here with this with these commands is they're not all universal so you could be trying to use menu config on on u-boot and it'll work but not everything has a menu config and similarly um not all the commands are there so i'm not sure mr proper work i don't think it does i think it'll blow up yes so you can use the tasks we'll show you all the tasks that are available so it doesn't have a do mr proper or anything like that so not to be confused with the fact you're using minus p for command and you're looking at tasks but you know not what i'm poking fun at oe people in the room or anything other questions sorry no get back to it actually working but this is more or less the end have you used a depth tool for the Linux source and then change configs as well at the same time how does it handle that how does it handle changing the config in an external tree yes yes the this that's not this one the other one um so if you do a cross compiling is fun it helps us make sure you back to where i thought i was my display is too small now no no it should be i'm specifying the architecture for arm so it should be working um menu config needs a bigger window yeah yeah the control should control shift minus is no there he goes okay yeah which makes it into my chart but then you can go start there you go and then maybe i can just do that okay well but yeah so this is the this is the non-stock for some reason it's not loading do it in a slightly harder way then um isn't that team up pre pre-records all of the demos um so it should work it was working earlier today i mean so uh yes i was using menu config through bitbake to just set it so that way it fetched the board config from um the kernel cache and then i was modifying it locally to add it brings up under dev shelving so if for some reason it's not showing the changes well so i mean you can see this is modified but this should still have the config from earlier which doesn't have yako standard should have oe which i put there earlier from the from the u name so sorry okay so just to finish there i'll show my last slide no no no like it's it's a call to arms okay cool you're maybe go through all of them okay i just wanted to kind of do a call to arms here oe is a community um we always need more people more developers more features more testers more everything so if you have idle time or don't have idle time but have interest um please join the mailing list contribute all that fun stuff and then thank you well thanks john and uh i guess that concludes our inaugural oe summit so thanks to everyone for joining thanks for everyone who's been here all day and uh we'll see you next year