 So, let's start. First of all, who am I? I'm Alexander Kanyavsky. I'm working in Intel as a software architect. I was doing the Linux distributions for quite many years. It's practically, it will be 20 years in a few months. I did the Linux distributions for local markets, like with localization for desktop, for servers, when later on for Nokia, for mobile devices, tablets and phones, when lately in Intel for Tizen, and for a few more years with Yocto. So, I wanted to share some of experience how inside Intel we are using Yocto tools to build distributions for our internal products. The things what we're going to talk today is actually, what it actually means, the platforms, products, what is continuous integration, how to establish continuous delivery for your products, what tools and how to organize with source code. Many people are talking about the content, like what you are putting inside your distro, but few people are talking about actually how you do it. And I want to share that. So, the first thing, platforms and products. Just a quick quiz. So, how many of you are using Yocto and Pocky? Pretty much everyone. Great. So, platform for you it will be easy to understand. It's like Pocky. You have BitBake, you have Recore, you have a bunch of other layers what you want to use. Practically, it's a distribution what you are targeting for multiple devices with multiple functionality, so you develop features over here. In my example, I will be using our reference IoT distribution, which is public, you can look how it's implemented, the CI part of it is also public. Go have a look like how our developers are working with it in real life. The product. This is something what you usually... You take a platform, you have a few configuration file changes, like few tweaks, you probably don't care about multiple devices. You have like one or two which you are targeting to. And in my example, again, it's our Intel reference IoT kit for Intel Jewel product. Again, have a look, see how it's implemented. Practices. Continuous integration, continuous delivery. Buzzwords, right? So, many people know those, think what they know about it. What it means in reality, like in plain words. Continuous integration. Old practice. You need to build everything, you need to test everything, you need to visualize status. You define what is broken state and here's the important thing. You define what is a broken state. And in case if you notice the broken state, it's your highest priority to actually go and fix it. And this is not something what people are actually doing. Obviously, these practices are evolved over the years. So, there is a new thing. It's called to stage continuous integration. Many people are practicing it, but again, without really realizing what it is. The thing is, it's a method or practices to prevent you to get into broken state. So, prevent you from making your code line in the state where development cannot happen. It's like, get your changes in your sandboxes. It can be called differently. Like patch sets or master next or pull request and GitHub doesn't really matter how you call it, but it's a sandbox where you test it and when you're satisfied, you merge it. Next thing, continuous delivery. So, in plain words, it's your process of what happens with your software between the time when developer changes it until you assume the time when you ship the product to the end customers. So, the world is changing nowadays. Our users are expecting what software is updated regularly on your devices. With small iterations, with new features, with small bug fixes, but often. You need to make sure what your releases are rolling. And all of those practices, everything, it all targeted one single motor. So, you need to maximize throughput of your good changes and minimize the breakages. Sorry. So, let's see how to do it. Let's start with source code management. You need to have a system where you actually allow your developers to collaborate. And by collaborate, I mean, like, you need to have sandboxes, you need to have reviews. And for that, you have nowadays a lot of choices. You can have GitHub, you can have Bitbucket, you can have something hosted by your own, like GitLab, Garrett. And to just predict the question, Gitalite patch works probably not the best tools for you at the moment. Why? Because the tools for what you're using for source code management, you need to have a few key features. People need to have easily to create sandboxes. You need to have a good integration with CI systems. So, it means like APIs. You need to have a good status tracking. So, you shouldn't get to the situation when a person sends a patch to a mailing list and waits for months trying to figure out what is the state of his patch. And then suddenly realize what, yeah, it was merged. For us, the choice was obvious. We used GitHub. It gives us whatever we need. It actually gave us a bit more things, like access controls and groups for our developers. So, let's see how it's implemented. Well, we have organized the source code for our platforms. You need to understand. Viopta is about integration. You have two activities, development and integration. Development is happening in layers. Layers, it's a set of reusable components. It's your recipes, it's your features. You might use it in more than one distribution. And you need to have a release repository. A release repository is actually something that you ship. It's one single thing what users should take, download, build, and this is what your release consists of. So, how you do it? A release repository is done by Combo Layer Tool. It's one of the oldest tool in the Yocto project and probably one of the least known tool in the Yocto project. What it is about. It knows about set of repositories, it takes comment out of it and push it to this release repository. You have a configuration file. And this configuration file describes what is inside of it. Like location files, where to put the files and the last track revision of it. How to update with file? Regularly in Pocky it's done by Richard or some other maintainers. We run it manually. In our case, we run it in CI system. So CI is actually controlling what and when we are moving the things between layers into our release repository. That allows us actually to do quality and schedule gate. So for example, you might want to have... Well, the quality gate is obvious. You merge only for things what works. So if BitBake introduces a change which breaks your build, you probably don't want to take it. The same thing. Like if you are trying to ship a product very soon, you probably don't want to accept the changes from OECore. You want to delay it for a while. So for layers, the service is what we want to implement. You have layer repository. You have developers who want to contribute. And the natural way of contributing is obviously it's pull request. So pull request, it's a sandbox of changes. It's built on the latest state of release repository. It tested on the real hardware. And when you provide feedback to the developers. So for example, if your branches changes, if maintainers merge with another change, and GitHub notifies what had changed, you do rebuilds and again, you provide the status to the developer what is going on. With the help of GitHub integration, the maintainers have the ability to use the magic comments and the pull requests which says, please retest that thing or not. This magic things, it actually helps with a few other things. Like if you operate in public, obviously not all the code what comes from unknown developers, you want to take... Oh, you want to build immediately. Because we all know, Yocto is not fully isolated from the host. So we provide also some security measures what maintainers can have a look at code which comes from a person who we don't know and then say like, yes, it's okay to test that thing. So the service for release repository. Again, obviously. Anytime you merge something, you perform a build. Here is one under water stone. So if your platform release actually produce package feeds, you need to have a version going forward for packages what you produce. So for release repository, you might want to limit what you have only one concurrent build for a branch. And another thing what again people not often realize is what you need to promote with builds. You build it once, you test it, you perform with slow test and again, exactly the same set of binary artifacts it's something what you need to publish to your customers. You don't need to rebuild it anymore. Like one single build idea, one set of binary artifacts. What happens with pull request to that repository? We created a thing which is called upstream monitor. What is upstream monitor is about? It monitors everything what is written on combo layer like every single repository. And once it detects the change in one of those repositories, it creates a pull request to update the existing one with new set of changes. So pretty much the maintainer of release repository he always have ability to see okay, what is the pending set of changes what we are coming from all the layers from which we are constructing our repository. This is an automatic part but maintainers have ability to do the things manually. So for example, maintainer can say I want to take this important bit bake fix or I want to take this BSP fix but I want to postpone the rest of the things. Usually we don't accept any pull request from other people. However, there are certain scenarios where those pull requests here might be considered as legitimate. What are those? One and the most important one is when people want to introduce a new layer. So developer just need to update the combo layer configuration file saying this is a new repository I want to include. For rest will be triggered by upstream monitor. So it will pull in the repository you will see how it builds. Another legitimate case is when you need to test big changes. Big changes means what you need to... You have scenarios when you need to synchronize like you need to change something in your BSP you need to change something in your configuration files like image files or in worst case scenario you need to try fix for upstream tools like BitBake or some of OECore classes. That way you send it you see how it works in your release. Quite complex, right? So we need to have a good engine for it. And for engine we use Jenkins. Why? Few things. Obviously it's newcomer friendly so you have a web UI you can configure it there are tons of plugins for it but the most important thing it's scriptable and it develops friendly. It means what you can automate whatever you do there. Implementation of it. So classical set of jobs. You have orchestrator jobs so for every layer you have job which performs for build for a branch or pull request. So you can have unlimited amount of branches but for pull request it's usually one job which handles all of the jobs. So it figures out what is the destination branch and when users will correspond in release repository for it. So if you have like dozens of layers you have dozens of job or two dozens of job but the actual build is pretty much the same all the time. So you have, or at least we have we have a two set of targets which is used for all the layers and all the release repositories. So we have build job which is dependent on each machine so each machine is built on separate server and we have a test hardware. So test hardware it's interesting thing because you might have in your terms you have like three machines but some of the build for particular machine in your term can be tested on multiple different hardware so that's where it is distinguishing between machine and test hardware. And obviously you have a bunch of post-processing job like publishing, promotion, maintenance, download servers, whatever which are my thing. Big, complex, sometimes hard to replicate but it gives ability to produce a platform easily. What happens with products? We try to simplify things and we wanted to use sometimes new technologies sometimes a bit different technologies or different ways. So one of new technology what we wanted to use is Jenkins 2.0 feature, the pipeline and with pipeline it allows you to do one thing. Practically you have a GitHub repository but you can have a multi-branch and inside of this Git repository you have file which describes actually how to build that thing. It's like, you know, like the ground file, docker file, make file, the same thing here you have a Jenkins file and it describes what to do. It's easy scripting, it's easy to do a parallel execution it's easy to like to do a persistent tasks which is in classical Jenkins is not really possible. And how it looks, something like that. You define where the source code like checkout run the shell scripts. Your shell scripts is actually part of your repository so you don't need to maintain it in a separate place. The pipeline, the whole product is one single repository for you and the same thing with build targets. It's a game, it's part of your source code so developers can easily adjust what we built. So, few of our different technologies what we are using. One thing is reproducible environment. Again, Yocto is not fully isolated from the host and we've seen a lot of situations when result on one distribution is not the same result with different compiler on our distribution. We try to isolate that. We provide a docker container. So, user can easily download our product repository he needs to have a docker installed but afterwards he runs one script like local build and the scripts will handle creation of isolated clean environment specifically for the build with nothing more and nothing less what it's needed and perform a build. So, in reality, this actually allows us to do even the Yocto builds on the mark like with docker for mark integration. It's the same set of scripts, it's done transparently. Another thing which we also try to use is Gitsub models instead of combo layer. Again, for product you need to have a very simple thing. You want to take specific release of your platform. You want to override couple of configuration parameters. You might have a bit different set of images what you're trying to build. You might have couple of more additional layers but that's it, your product is often not very different from your platform. And submodels, to that extent, it's a bit easy to introduce. So, we're big difference to approaches. So, this combo layer is obvious. You have a self-hosted repository, all good, you control you see all the changes you can track where it comes from and so on. The bad thing, it clatters your Git history. You see, like, if an open MB that you have 200 commits you will see 200 commits in your Git log. It might be hard to review, but it depends on your people. With Gitsub models it's a bit easier. So, the combo layer approach doesn't prevent you to change some files in some subdirector. With submodels it's a bit more tricky because you are referring to some third-party repository. And this third-party repository unless you have commit rights where you cannot change your file. And this actually goes to Conspart. So, when you are using submodels you are dependent on some other hosting. So, if your upstream part force pushed something or host gone down, you lose your setup. Second thing is actually also it's a bit harder to review the changes because when you do a pull request for submodel it's actually you see you are moving between one commit state to another commit state. You need to go to that repository and do a div between those if you want to see what is actually changing. Infrastructure. How it's done in the backend. We have two parts. One part is publicly-facing like it's obvious like website download site and when we have Jenkins in outside. Practically Jenkins master to outside is for two things. One is to provide UI for developers where we can look for the logs and second to have GitHub integration where you use the hooks for it. But then what is backend? Backend we have obviously set of builders. We have testing harness with a bunch of test of devices and by the way those two clusters can be in different places. In our setup it's funny. Builders are in United States, test devices are in Finland. Few things for builder clusters. You want to have a NAS. You need to have a local storage. Yocto produces a normal amount of data. And that's actually where few of underwater stones are happening. As state. You need to have a few maintenance jobs which taking care of that as state. You have something what is available for the builders but you also have something what you need to upload afterwards to the public site what our developers will be able to benefit. Another thing is coordination of few things. As I mentioned if you have a... If you have a package fit you need to use the service which is called PR server. And if you try to do the distributed builds between like multiple hosts you need to have a single place where you say PR server is over here go and fetch for revision what I need to use for that package. And if you do the same thing for multiple branches for multiple architectures or for pull requests you need to have multiple instances of that PR server. So we are using the host called coordinator for doing that. There are a couple of other maintenance jobs arriving on this coordinator host. Obviously it's hard to maintain all of those things. And we did quite a lot of things what simplify our life. So the initial provisioning. We are using uncivil. We'll define it with DevOps tool. And our set of playbooks how we provision Jenkins workers and all other hosts we are open, have a look and feel free to reuse, feel free to comment and provide suggestions to improve all of those. What happens with the jobs? We are using the thing which is called Jenkins job DSL. It's again, it's scriptable part which allows us to easily maintain those hundreds of jobs what we have for every branch, for every layer repository for every pull request and so on. And we have in this repository where we maintain the Jenkins configurations like two branches like one is production, one is staging. Obviously like we have two setups. We have public and we have staging in private so we can try a few of the changes before actually we go to a production level. In reality, the whole deployment of your scripts of your configuration has become just a git push. We did a couple of interesting tricks here. So one trick is actually the initial, Jenkins initial seed. You configure one thing, one single job which says here's the repository where we have all configurations and this is where script which says how to set up the rest of the things. In our case with initial setup it does few magic. This magic is actually it verifies what all the needed plugin sign installed in all the shape and afterwards it starts to create the jobs like all the maintenance jobs, all the seed jobs. And by seed jobs it's actually it's actually something what helps you to dynamically manage your configuration. So imagine like you have a combo layer configuration file and it describes this is the layers what I want to use. You know exactly which branch, which git repository what you are interested in. So your DSL script can read that configuration file and one create as many jobs as you need. So for us the maintenance become really, really easy. We're just saying here's where our repository the rest is where. We don't create jobs anymore by hand and you are not limited to one branch. You might have like we are mostly using the rolling release model. So for us the master is important. But if you have a maintenance branches you do the same thing for maintenance that it will create few dozens of our jobs for you automatically. So here are the tools. The automatic testing part. You've seen probably today or yesterday or maybe tomorrow it will be few hour presentation. A lot of about automatic testing. People are trying to use linear allow it's good system, big one, it's good community but we want something smaller. We want something what people can easily reproduce. So it's basic set of tools which provides automatic flashing. It provides networking to the test device. It controls power. In some cases it also controls keyboard emulation to select which reboot devices are. In our testing setup we can handle majority of PC like devices with UFI boot, big old bones and bunch of Intel devices to do. The project itself is done out of off-shelf components like normal hubs, normal power cutters what you can buy. You can use Arduino, fast keyboard emulator and pretty much that's it. All ownership or cost ownership is minimal. And we have actually prototype which uses big old bones as keyboard emulator and mass storage device. We even simplified the setup for that. But if you're interested how it is done this project is done by my colleague, Igor Stopor and he did a presentation last year in ELC in the United States for that. Few other things. Build history. When you're producing the distributions you want to understand what is actually in your build. Basic October or open embedded tools we are providing build history but the reason or the initial idea of build history it was primarily to support the package feeds. So you know how your package feeds are rolling. We need something in addition to that. We wanted to know exactly in each build what is part of it. So what files which revisions were used for every single recipe in the build. And we created a class which is called build history extra. Unfortunately it's not yet in OECore but I hope it will be merged anytime soon into it. And we have a set of scripts in our CI system which actually helps to track what is part of your build. So if you look at the OSTRO project build history you will see what every single build produced by our system we have a tag and inside tag if you start the kit you can compare the tags and you see what is actually changing between those builds. As state, for people who are using Yocto you know all the problems with it, right? We tried, we have massive set of jobs. We have massive set of builds what is happening all the time. So we end up in a scenario what each builder actually have its own local as state. And when after the builds is happening the builders are synchronizing as state to the central place to the coordinator to our NAS and when we are publishing that state. The reason for that is just purely performance or sometimes we are bugs and NFS servers. Another thing which again not always obvious is when you do the builds in sandboxes like for the pull requests you don't want to pollute your normal as state what you are using for release builds. So we are creating temporary as states for pull requests and that actually helped us to catch several bugs. So PR server I already mentioned you need to be careful when you are using it for multiple branches and parallel builds. Other thing like performance and disk operations. Here I wanted to say one thing Yocto is trying to help with local development so it tries to see what you have enough disk space you have download directory in a proper place but if you have scenarios when you download the source directories over NFS these nice features for developers actually killing your performance so your bit bake will be constantly trying to check if there is enough disk space and if you look at the NAS monitoring you will see like constant flow of requests from the NFS server it's probably not something what you want to use. So verify your settings what you are actually using and monitor your hardware like how it's happening. Last but not least we integrated to Yocto Core well to open and be that core with tool called BIMAP tools so if you are constantly making device images and you know the scenario it's raw image 4GB you flash it with DD not really efficient especially like if you have a real data on what image is about like 500 megabytes or like 20% of it so which one should we BIMAP tool? BIMAP tool is something what understand what how much of real information is present in the image and when you are flashing to the device you flash only those amount of valuable information have a look maybe it will simplify your life so I think that's it I'm open for questions links to all of the things what I showed it will be in my presentation like the next slide so when you download it have a look Questions? Yes, yes of course and one thing which I wanted also to say is if you have any questions after this feel free to send me mail I will try to answer it so question is we are testing on real hardware using EFT and is it possible to virtualize? let me answer it that way I'm working for hardware company so for us yes in early stages of development virtualization is usable but in the end of the day we want to ship a product which will be working on the real hardware so for us it's most important what it works like obviously you can have a built target switch which runs in QMO if you are just interested in some software features which is not related to hardware for us it's less important but if you look at what happened in public majority of we test what we run it's done in QMO you can use whatever we have yes, rsync, yeah it's more about how we organize it by the directories so like local builders we have local state between the builders it's shared in the builder cluster on the network storage so the builders can fetch the object if it's not present locally and when for the pull request we just put it into separate directories so it doesn't pollute so like different rsync target so how we validate a state what it's still valid because everything what we do is through pull request so practically for pull request you have a cleaner state of a state so you have built inside pull request which is taking the state of official build plus what changes what is part of a pull request when the change is merged it's done against the official state so we can verify what actually the result what we've seen in the pull request build it's not different from what after the merge, that's one thing second thing it's an overall even the official state might be broken we had some time ago so called like nightly build jobs which pretty much done one single thing it took the latest states of a branch wipe out the whole state disabled state mirrors and when do we clean build it's slow but at least once a day it shows us how we were performing luckily in like over one and a half year we've seen just few scenarios where it actually helped it to catch a couple of bucks another thing what we actually are actually using is OE self test for recipes so when the when the orchestration of a build is triggered for platform repository so we start to build for real targets but we have one more additional job which called self test with self test it goes through all the recipes and actually tries to verify what did I lost my mic it tries to verify what recipes or some environment parameters of a host system is not affecting the checksums of a state so if something will try to invalidate a state we will see it self test will be failing we are not trying to prevent it I'm serious it's let's say our image consumes a lot more space than our state so we can afford a few hundred gigabytes of data when probably you need to see what actually changes your checksums that often because what is in state means in reality like it checks what still it sounds a bit suspicious for that I would suggest try to grab with your developers and above I think Richard will be the best person to help with that so first question combo layer sub models and why not to use repo you can use it if it's good for your workflow you are free to use it in our case in our team with developers we are simply not happy about repo so I try to utilize what our teams are doing like combo layer what the other guys are doing second question is about how we maintain jobs there are two sides of that question so platform it's done and separate repository and yes we perform the same thing like it's pull requests we review what is happening where sometimes we try to do the same changes in the staging environment to see if it properly works and only afterwards merge to production system the CI system for product when Jenkins pipeline it's part of the repository and the scripts so it's again all the changes are done through pull requests and we have the same safety measures so if a pull request comes from one of our developers it builds automatically we are not trying to harm us but if it's coming from any random user on github it will not build unless the maintainer says try it, yes only for unknown people courtesy of plug-in what we are using to trigger our builds it's github pull request builder so we handle it in two ways one way is we perform a test request to release the repository where all the changes are combined because it's self-hosted repository regardless from which layer it comes and when we are ready this set of changes are working fine we actually split them by pieces and when they integrate to the layer at that point the maintainer can temporarily ignore the situation like the layer branch building has actually become broken but we know as soon as we merge it into the layers the upstream monitor will kick in combine all changes together and then produce it as a proper pull request again and then again it will bring it to a proper buildable state we ignore the broken state of the layer branch for a very short period of time to synchronize the changes again I mean like we split the test request to each layer and when it temporarily ignore the broken state yes it's manual, it's manual override historical, no it's just a historical reason so this project what I'm talking about with reference IOT Linux distro it happens for a while and pipeline is recent stuff so like six months or something I will probably use for a pipeline it's just a classical thing some people prefer one way some people already moved to another way and I wanted to show and demonstrate what even if you do it in a classical way means how to automate the thing so you don't need to manually maintain it, good question so again one of the reasons why I told promoting all of these things like continuous delivery is when you have ability to easily look like easily monitor the quality of changes what is coming on from upstream you can have in parallel like you can follow master, you can follow stable branches and you can always have multiple branches in your project to see how your stuff works on top of the upstreams so for you the upgrades between major versions it becomes predictable in our case with reference Linux for IOT we don't even bother with following the stable branches we are always on master we know what if something broken in master it will be code by our testing we simply will not merge well stuff until it get to a point or into a shape but it's usable for us and it allows us to actually to work very close to the upstream so if we introduce something which breaks us we immediately like in few hours we can say like guys you are doing something wrong here fix it gentlemen I think we are running out of time I will be happy to answer our questions in the corridor or just grab me and we will