 So hello, welcome anyone my name is kunkoi and The title of my talk is our embedded build system still needed and before you say hey, isn't there a law about that? Yes So our embedded build system still needed and there's a multiple choice answer there and hopefully at the end of the presentation you can Fill this in for your specific use case If you've ever attended one of my talks before and you'll notice that I run through the slides in like 10 minutes And then go to the questions section. It would be nice if you can now do the questions earlier so if you have a question Stand up and state your question in the microphone or speak loud and I'll repeat the question and I'll try to answer it and If you want to read along with the slides go to this tiny URL that points to this Google doc So this is how it got suckered into embedded Linux around 2002. I bought a PDA and It was awesome. It had Bluetooth and a touchscreen a touch screen. Yes and it had both Compact flash and SD card slot and and it was fine, but it even had an end flash, but That But it had flash but Windows CE did not use flash that was for the operating system. Everything was stored in The RAM basically Windows CE was a big ram disk. So if you ran out of battery That end of the buzz would basically reinstall Including the oh you have a touchscreen now Let's go practice and you have to go through the setup routine and that was tedious and then someone said well You can install Linux on PDA. So it's like that's pretty cool So I tried to research it and it turns out people were trying to port Linux to HP IPEC 2210 like ah This will be easy, but it turns out that Linux on IPEX worked well for the older models the strong arm models and this one Was work in progress. They said well just Download our kernel from CVS It's like what CVS? Download the kernel and like now what I build it But how do I install it and then someone said well you have to actually cross compile it like cross compiling What are you talking about and along the way? I discovered that arm is really special and you need to do things differently and Then people said we're now going to use this new thing will think open embedded Chris Larson Just created it a month ago. It will be awesome We will switch to it. So I tried it out and I could build things and along the way I bought a second PDA in all the model that could run Linux. So could test things in case this would ever run Linux that's how things start and It's like well, it's working pretty well. I can do my appointments. I can do audio recording and But this hasn't been released The familiar Linux distribution hasn't seen a release for over a year Why they said well if someone would volunteer to be a release manager, we could do a release I said oh, why doesn't someone do that and they said you just did So with your experience I became a release manager, so that's how I got suckered into embedded Linux So this is the early 2000s to PDAs. They had 32 max of RAM 64 max of RAM I think the one just shown even had a 128 Mac version. That was well Wow How much RAM will you ever need it had tiny flash and you could have MMC or compact flash slots the IPack 3600 had no SD card slot You could put in a big jacket and you got big ass PCMCA and you could use micro drives and things like that But it was cumbersome and then you ended up with basically an 80 cell phone so Which groups in this period need a build system the users they don't if they download something they download the firmware in which They flash it over serial and there's not much you really you can do with it You just have enough NAND left to store basically year-wards of appointments in your calendar You don't have a camera or something like that. So There's not not much you can do with your PDA For the people that had the large PDA when SD card you could install Extra packages, but because the screen was tiny. It was a touch screen. You didn't have a keyboard Every application you had needed to get customized and as it turned out every customized application was already built and pre-packaged for you So the users they didn't even know what a build system was. They just used your distribution For application developers, they had to use it. There weren't any SDKs available and no on target tool chains and Well, no just generic tool chains were available. You could do cross tool that might work might not work You could do build route which might work might not work So basically you needed a build system and the distro developers they needed it because Open embedded and build would were the only way to do a distribution and along the way familiar open source and open simpad Decided that they wanted to do a support a bunch of different machines and be support package management. So Build would was a complete fail for that and they created something new open embedded As you might notice in that period it was really hip to call something open And what build systems did we had available? There was an IPAC cluster at HP that was IPACs with the PCMCA sleeves and Debian installed on a hard drive with networking access. You could SSH in and Around 2002 one or two were working. They ran an ancient version of Debian for arm With even all the compilers you all you had was GCC 2.9503 and that's it It didn't run actually any distribution provided by handhelds.org So if you want to develop for different distribution, you just couldn't you were tied to old compilers old C libraries At this point it was basically useless Build route that worked fine except it didn't scale for lots of different targets You ended up with just n copies of your build with config instead of just small changes to your machine description So that didn't scale and Chris Larson said I can do this on company time. I Hate Montevista at work. I hate build route in my spare time I will create something new and that's how open embedded started and Open and better at that time. We had a lively RSC community where we would pass our local comms into pacements and things like that because you needed the magic options to make it work I think Tim Bird can testify that you needed a lot needed a lot of magic options to make it actually work so No one was happy There was closed color cooperation between application developers like GP and OP specific Palm top environments and a distro people the distro people work closely together because Everyone was doing all those roles if you were an application developer You had to be a build system developer and you had to be a distro developer just to Be able to see your application on the screen Which does mean that there is a very tight integration. So your software runs as expected nowadays when you Install a distribution on your computer or your laptop you can say oh I really hate this gnome desktop And then someone will step up and say well in upstream gnome Everything is okay, but the tweaks fedora open sues or whatever due to it Those sort of things you hate and you get the big discussion of distro specific patch and things like that Those didn't exist in that day. So life was good for The very few people who were working on and then that time there were like 20 people So a while later t.i. Released the beagle board. It was massively more powerful than anything on the market. It was the first Cortex a8 device available. It had a floating point processor it even had to you could do factor floating point on it and Sim the instruction then it was cheap it was I forget two hundred dollars on launch or something That was a lot cheaper than the three thousand dollars that you could get. So what did that mean? You got a lot of RAM you had an SD card for storage and you could actually use networking and That was the start of the problems for build systems So users were still like they didn't need a build system they finally had gigabytes of storage They used everything from package feeds because they were like I can finally install things on it They were just so happy with that Application developers got slightly grumpy because they were finally used of having working SDKs But for the beagle board there was a flaky SDK available, which didn't really work It was something that we said oh open and better can generate SDKs. We'll just build the SDK upload it and We'll see what happens. Well It was broken the on target toolchain didn't really work and the external toolchain It turns out only works if you had the exact same distribution as the toolchain was built on Which in my case was everybody was locked to an ancient version of Debian and additional developers They of course needed it because there was still no other way to build a complete Distribution from scratch in a reasonable time frame and The build systems there were a few more available. Well open about it still around the ancient conflict worked you had 32 distribution conflicts and I went back. I said more than 90% Three of the distribution conflicts worked the 29 other conflicts just didn't work for one reason or another So that doesn't inspire you a lot of confidence You add gen to prefix that worked extremely well, but Virtually no one knows about it still don't doesn't Gen to prefix basically means You say I now want to build for arm and gentle will build for arm in a system using the gentle metadata And it just works So you don't have to run gen to on your target the cross compiles for you Why people don't know whether I don't know but at that time it worked very well. The problem was that No toolchain supported ea bi you had to patch your toolchain the code sorcery tool chains Yeah, I've said enough about that, but they didn't work you had to patch them You get had to get bribed someone with a code sorcery subscription to get the patch to make the toolchain actually work So once that patch was out to me open you could basically build it and that's how we got open embedded to work and gen to and Finally you could do arm ebi with all the nice arm Cortex a8 stuff, but but still If you want to do cross compiling was open better the gen to the rest was native Compilation and since the bigger board was new native compilation was on all the hardware For example Debin used nsle twos which is a for no sorry 266 megahertz arm 9 and Anything you wanted to build on it it took at least two weeks because pretty much everything depends on nss or nspr Which is part of firefox Which takes two weeks to build because swap is on our usb hard disk. So anything you did two week timeout Native builds they didn't really scale, but if you just add enough nsle twos and go away in Six months approximately you could build the W in archive. That's what Leonard Bowden hack did He used an angstrom toolchain to build the first W and ebi support and it took him six months on really fast hardware That's how they mean got started. So that isn't really an option for a distribution at that time With the build box on university which had multiple CPUs etc I could build the file system image for the bigger board which got a gnome desktop in approximately three days So that's still quite slow, but that's manageable if you do it from from scratch That's not the six months it would take on Debin with a build form people got unhappier in this situation strangely enough no co-corporation between app developers anymore because Gnome KDE etc only cared about real-life computers like they said where you could just have a GPU with open GL and tons of RAM Distro people and build system people they didn't really work together because Big distros do native builds so you don't have to talk to the build system people You just maintain your package at use one and at one point you all the builder will kick out the package for your architecture and Even worse uses suspect red had enterprise Linux type of stability with gentle style optimization and WN wealth of pre-built packages from your Distribution which is that point was still managed by people in the spare time But the bigger bone had blinking LEDs. So Users were still happy. They were grumbling about. Oh, this doesn't work that doesn't work But at the end of the day they could blink their LEDs and then they were they were happy At that point there wasn't any bit banging going on yet Because you didn't really have the on target tool chain So if you want to develop your own application you had to do it in Python or something and that didn't know about GPIOs yet So fast forward and it's the time of the RG Arduino on Raspberry Pi This is not an Arduino and a Raspberry Pi the big red thing is a sanguino. That's basically an Arduino on steroids people who know about 3d opinion community, that's the controller of the original wooden maker bot and That plexiglass thing that is the plexiglass version of a original wooden maker bot but now driven by a beagle bone So that was my hobby project convert all the arduinos into beagle bones and it worked I can't say it's better than the arduino, but at least it runs Linux now, so That's suddenly a completely different market the original beagle board was sent out to developers who wanted to develop on the Brandspanking new omap 3 Cortex a8 because it was the only way to get your hands on that type of hardware It was a strange that the way situation that TR was actually funding their competitors because people would buy a beagle board implement our software and Then buy the free scale CPU for their design but we would still sell beagle boards and With the beagle bone We wanted to go to the maker community Right at the point that the arduino was very popular and the raspberry pi So the use what the users wanted to do they wanted to interface sensors motors their electronic cat feeder etc to your board and In arduino they were used to just if you need a protocol you bit bang it They get your board and they go like Oh That's weird. You don't have an IDE where I just click upload. They're like, yeah But but you can use a shell script and Cissif has to twiggle up in and they're like, oh, yeah I can do that. I can blink an LED and Then a week later they come back with but if I use a shell script and Cissif entries for GPIO I can only get like two kilohertz out of the pin for my thing and I need 200 kilohertz and we're like What are you doing? That needs a kernel module and I go like, yeah, but we're not programmers so With people moving to scripting languages Hardware becoming available etc. Etc. Etc less and less people actually need to build system And we started getting jokes about ethical compilation Because at one Ubuntu developer Summit one person said cross-compiling is fundamentally broken and another one meant said it was something unethical. So That's what we stuck with so ethical builds In embedded space angstrom and then pokey and open embedded ristors are basically based the only cross-compiled distribution build route is not really used by distributions more for one-off things like metacor Industrial units Ubuntu runs on it fedora now runs an arm open suzie runs an arm gen2 has a native arm port Arch Linux has a native arm port basically any distribution with its all has a native arm port and We'll have a port for the raspberry pi the bigger bone bigger board one board cubie board Arndale board you get the picture. There are a lot of boards around there now. So Trying to get people interested in build systems is really really really hard nowadays We lucked out Yachto Yachto became popular Intel started funding yachto and started saying like oh this Virtualization team in China all 60 people are now build system engineers And that is basically one of the great leap forward that thanks to yachto people are now paid to work on open embedded Because all the hobby users that were there from the start are now paid to work on it We have very very very few volunteers left because All the fun stuff has been done and if you want to get work done Install Ubuntu on your deaf development board and you run into quotes like this. So The first quote is I disabled USB in a kernel to make it boot faster uses that one USB can easily Reconfigure and rebuild the kernel because the kernel is using the yachto kernel tooling so This was for an Intel based board and Intel marketing was saying this is the Intel architecture advantage Stuff just runs. It's not special anymore like that creepy arm So I said people buy that board hook up their monitor or power it up Let's stop on a monitor plug in a keyboard mouse. It doesn't work Strange you go like online like yeah, you need to rebuild the kernel like yeah, we can do that I rebuild the kernel from a laptop and Then it should up like no no no you cannot rebuild the kernel the way you want you have to install this open embedded and then Build everything including a tool chain and on top of that you cannot reconfigure the kernel because we use is yachto kernel tooling And no bad word about yachto kernel tooling. It's great because it scales to a lot of boards But if you're just interested in your board It's a learning curve At that point the marketing lady blew a gasket and said we should stop being stupid and it just turned on USB so We could have lured people into the build system, but we soon realized if we would pull stunts like this People were good Doesn't work Google. Oh, I can install you boom to and it would install you boom to it And we still wouldn't have people only pistol mouth even more and next quote This is from a build root developer in 2006. We don't support x11 because we're embedded and not bloated And then five years later we support 11. We're so awesome Yes No, but then they said the maintainers left and they got new maintainers so it's like I won't insult them so Yes Because well the other problem was that Basically, you need to have a really strong will to be a build system developer and in regular terms You basically needed to be an asshole to be a build system developer and that if not then Yeah, it's and open source developers is like herding cats. You it's just impossible to get something done by consensus and Yeah, the build with people were like usually let's see we're even better than all that open and better They're weird, but nowadays build would support basically anything so more power to them and you get this is not really build system related, but it's Someone said to me. Well, I made this device tree and it makes supporting this platform so much easier So I said, well, it's a derivative of this board So just send me the device tree I include and we'll work out of the box and it goes like But why would I do that the point is it's external to the colonel We should never go upstream It's just so I just grabbed and then added it to my build system So it's not a stream in the colonel, but stuff now works out of the box and This is a question I get very often should I switch from using open embedded to using Yachto and I went to the keynote this morning and It confused the heck out of me because I always learned that Yachto was this umbrella project and open embedded Was the build system and the people on stage kept saying building with Yachto. So I don't know what's going on anymore and that should attract more and more users The answer to this question Open embedded is part of the Yachto project. So if you use open embedded you use Yachto and the real kicker one of the biggest Yachto and open embedded users nowadays thanks software open embedded is just a single script that only supports one thing So like I said people in build systems were jerks So do you have any questions? I'm also open to Yachto and open embedded questions if you want to hear about that Question could you come forward and speak into the the microphone? Do you have a conclusion then there? So the conclusion is It's it's a compound it it's depend but for the last Users don't need it you need to be in a very specific situation that you need an embedded build system Because for most things nowadays your distro takes care of it. You only need to be a package developer The people at arm said oh, we're doing our bsp now for the chromebook Do we need a build system because we have this use case, you know like Yeah, in that case you need a build system because they want it to be ahead of the curve They wanted to test wayland and mirror and things like that. So yeah You need a build system, but the people you're delivering this to They don't because they integrate it in their distribution that it's not needed anymore Disclaimer i'm a build contributor Okay, um, there are cases where an embedded Build system are still useful for example, if I just install raspberry pi It takes roughly 45 second one minute to boot up to the desktop screen, which I do not need I just want to run a single team and tv add-ons that records the tvb usb And streams it in my home Building it with a okay build but we are to or whatever open-vated Boots in three seconds Without optimizations at all. Yes. So we just in need Cases where we can tweak the the fight system the target of this So with with embedded build system that it allows you to build things from the ground up and you have a few edge cases like Guess avr32 Sadly it's a dead architecture now it's been it's been end of line But all the developer board you would get had eight max of ram So you cannot do any native development for it. It had no q emu available for it So anytime someone says just build it natively you would just want to reach out through the internet and punch them because There's no way to do that Could you or pass the microphone back there? That's also an option what One thing to note that to chris, uh, who was originally developed oe was actually fired from ti for spending all this time working It's just a small fact. It's they actually made the mark too often that we get by with 128 makes the frame And they're us for a short period actually a build farm building they've been Get 10 of those Yeah, and that I had almost completed. I had an avr32 where well leon wustenberg kindly replaced Manually replaced the the eight max chip with a 32 max chip So in theory I could have done a native build as well on that but at that point it wasn't really feasible and um arm 64 There's no hardware available yet. So the company that has been the most hostile to Open embedded arm limited they would invent new build system just to afford using open embedded finally went like We have a problem There is no publicly available emulator for it No simulator and no hardware, but we need to bring up the software because we want The server people to be able to use it. So What the hell do we do? And luckily nearly narrow said well, there's this thing called open embedded that can do cross compilation and arm said So and linear we now bringing up The original bring up was with open embedded and there are now simulators available freely downloadable from the arm website So you can just go the gen2 route and demin route and do native compilation But for bring up on a new architecture You need cross-compilation and as soon as you're in that position You need an embedded build system because it's just too hard to do it yourself and I think Tim confessed that his do it yourself build system wasn't completely diy, but heavily leaning on montevista Yeah, well there were several do it yourself build systems of sony And but the one that we recently converted over from was actually a lot of us did Right and and while montevista was great It basically was you have this big file system and you can cut it down to your needs and mostly you want to build it up So that's where open embedded and build would shine you go like i want this and only this and that's what you get So More questions Yeah, another issue with native compilation There is a hard cleaning cell It has a lot of package but no firefox And uh, I asked That told me we can't build it. Why? Guess why not enough ram On the build server Suggest so that that not enough ram on the build server that is a fairly recent Problem that for example the final link stage of webkit requires approximately three gigs of ram So, uh, yeah, you can add a lot of swap that doesn't always work and various architectures don't Allow the complete four gigabyte and you run into a lot of problems. Um, I've heard that llvm will solve this but That's still a bit away The x86 64 bit kernel so nine gig for the final link stage for an x86 kernel. So Well x86 is not really what I would call embedded, but it shows that modern software requires a ton of ram. So Native compilation is not always an option unless you have Oodles and oodles of time and my guess is that a lot of people go like we're consultants. We get paid by the hour We will do native compilation and in sometimes it's it's easier I I must confess nowadays a prototype for native compilation, especially for things like pearl modules Uh, someone said I want to use this piece of software and I looked at it. It uses wx widgets Which are horrible to cross build and even worse it uses the pearl version of the wx widgets So I looked at it and went like you know what I'm going to install pearl on the target first and you see pen to build everything Five hours later, uh, it halted somewhere halfway because I had run out of ram And I spent a few weeks to integrating wx into open embedded and I finally have it working and you can build it in 10 minutes or so so Future generations will be able to build wx in like 10 minutes or install it from the binary package feeds because Still native development is not really an option only are the ARM platforms questions No one has mentioned openGL yet What's the next corner case? Well, there isn't one in the slides, but there are a few um more corner cases Uh, which are coming up again because there's also a big engine version of arm 64 coming And it uses a weird target tablet it ends in be instead of eb So we have to patch a lot of build systems for the big engine detection nowadays because even the linux kernel You say I'm an arm big engine I want to build big engine and it builds and builds and builds and gives you something and uh, well Internally people said it builds success and I integrated the kernel into open embedded and open embedded says That's a little and the binary error So there's a long road to go for weirdly named architectures and With build systems you you are able to spot such error that it's actually building little engine instead of big engine binaries Or what we notice in open embedded after we Finally added architecture checks that for Approximately six years we built the point-to-point daemon pppd as an x86 binary for all architectures and installed it into arm file systems No one ever noticed it So if there are no more questions remarks then Thank you for listening and see you next year