Good morning. Thank you for attending our session.We'd like to introduce some long-term maintenance approaches for the embedded products based on DBM and YOKUTO project.I'm Kazuhiro Hayashi working in Toshiba Corporation in Japan.And usually developing our embedded Linux products in our company.And also I'm a member of the CIP project, which will be described in this session.And could you explain?My name is Jan Kisker. I'm basically doing the same what Kazuhiro is doing, but for Siemens Corp Technology.I'm also involved in the CIP project looking after the ESA path of CIP, where we'll look into later on.And also maintaining and contributing to a number of other open source projects.Thank you.And here's today's agenda.First, I'd like to explain some background in our product development and explain what is the CIP.And the CIP core.And explain some example of the implementation of the CIP core.And how we are thinking about building products on the top of the CIP.Here is a basic steps in industrial product development in the left side.And in the right side, there are several requirements in each step.For example, in the development state, we need to select a base system which supports multiple architectures and embedded boards.Also, the high customizability and the scalability are required to cover the many kind of product developments.Usually, our product developers require the ready to use images to install the systems into the target very easily and quickly.As well as standalone SDK, which is required to develop the application by themselves.And after the released product certification, the products will be in maintenance phase.In this phase, the important thing is that we need to provide many kinds of bug fixes and security fix if needed.And without adding many significant updates to each software installed in the target.And we need to support such kind of things for a long time with the verified automated update.And we also need to do such kind of things very quickly.And also need to share common resources elements required in the multiple products.So, things are what we need to do first is to select appropriate base system.We usually select the one, our Linux distribution for that.And also need to provide some tools that can integrate that base system into our product requirements.Civil infrastructure platform project was launched to cover such kind of requirement in our product development.It provides first industrial grade software, which is required to achieve the features,required features like reliability or high real-time performance and so on.And second, it's also need to provide the sustainability so that the systems of the productsneed to be maintained a very long time, more than 10 years being based on the standards.Also, security.There, we need to apply the many security fixes for a long time by the very secure ways without the regulations.Basically, the software stacks of the product consist of the Linux kernel and core packages,the 100 packages that is provided an open source distribution and the product specific software.CIP is focusing on to establish an open source base layer, which consists of super long-term support kernel plus CIP core packages.Most famous distribution focusing on provide various 100 packages to cover the various requirements in general,but the target of the CIP is bottom two things.There are several work groups in the CIP project.CIP core is one of them and it focuses on user-run software and the tools.One of the goals of the CIP core is to define a list of CIP core packages that should be maintained for a very long time by the CIP.And also to provide the reference implementation, including that core packages.Also, to test the implementation on the CIP reference hardware, which is decided by the CIP project members.Currently, around the six boards, the reference hardware defined in the CIP project.This is the position of the CIP core work group in the CIP project.First, we clearly define the process to decide core packages we need to support and discuss with the other CIP members,then updates the core package list.And other work groups like security and software update request some packages or additional configuration in the system to support the features they are requiring.In that case, each work group request some specific packages to the CIP core.Then, the CIP core provides the reference implementation, including the super long-term supported kernel and the CIP core packages.Then, test the implementation on the CIP reference hardware using the CI environment provided by the testing work groups.So, for the CIP core implementation, currently it's based on Debian, which is one of the major and high-quality distribution in the world.And from a long time ago, it supports many new and old CIP architectures and it's suitable for the value system from small one to the big one.Where there are many various packages need to be installed.And also, it provides a lot of security updates frequently by Debian security team and also Debian LTS and Debian extended LTS project.So, CIP core provides two profiles based on Debian.One of them is the generic profile and another thing is the tiny profile.In generic profile, its approach is the binary based and using the tool either to generate the final image.And in the tiny profile, this depends on the source packages of Debian and using Debian as a build tool.And we would like to explain some features of each upstream project.First one is Debian, repository name is MetaDebian, which is one of the Yoctoproject extension for using the Debian source packages.So, the build system is completely same as Yoctoproject.And its goal is to achieve the stability and long-term support with keeping the Yoctop advantages.Yeah, Yoctoproject is very flexible and extendable by adding our own recipes.And the interesting, one of the interesting thing is we can provide some very small footprint systems using the Debian sources around its root file system size is around two megabytes.And it can also provide various target machine configuration for the various CPU architectures and tuning and also as well as BSP layers are provided by the board vendors.You can check this upstream repository and CEP core tiny profile is provided as bottom URL.This described how the Debian works very simply.In the bottom, there is a pokey that is provided in the Yoctoproject.And on top of this, the MetaDebian working, it contains some recipes to cross build the Debian sources.And also, it provides some common functions to effectively fetch or unpack or patch the Debian sources.And finally, the BitBake automatically generates required images like a kernel and root file system with loader and a standalone SDK for each hardware.Then, from this slide, I'd like to change the speaker to...Yeah, thank you, Karzu.So, I would like to give you a brief introduction about the build tool we are using for the binary-based approach in CIP.That's the ESA build system.It's not part of the CIP project, but it's an external project we support that has the goal to generate you basically a Debian-compliant Debian-compatible image.Out of normal Debian way, out of binary packages, mostly.It's a developer-centric workflow, so you have one command building basically.You have still the need in embedded world to do customizations and extensions to some packages, but not to all.That's the philosophy behind it, and this is purported by this.And yeah, as we are primarily binary-based, it's efficient building.So, the idea behind it is basically to combine the best of world worlds, have the binary distribution as a major source,but have a build system similar to Yocto, which actually we are using BitBake internally,in order to have the customization and have also a similar structure like Karzu described for the MedaDebian approach.And that, of course, also implies reusing knowledge of developers who used to work with Yocto before.Just to give you a brief idea of how the workflow looks like inside ESA,so we start with a reboot strap of the required environments we need for the target as well as for the build environment.So we create a build change route out of this to have a build environment for custom packages for those few we may have,like for example the kernel or some customizations on other packages or not yet there been packages or components.So you can either come with a source package which contains already debianization,or we have some internal recipes which generated ad hoc.Well, of course, a lot of debian standards, but technically working.And then you generate packages out of these normal binary packages and add them to the pool of the pre-existing debian packages,to finally assemble the root file system for your target installed or the needed components.So package centric.And last but not least, generate an image out of this, the image you want to boot on your system from the flash,from the SD card or whatever you have.For that we are reusing Vic from the Yocto project in most cases.Some are done with custom classes, but that is the approach behind it for customizing the image,generating the image, doing the partitioning, bootloader installation and all the other stuff.And then you have your bootable image in the end.These are the two systems on top.We have our layers for CIP.So for the ESA variant, there is the ESA CIP core layer,and this is just to give you an idea how quickly you can generate an image for a concrete board,in this case the BeagleBone board.We are using configuration management tool for that,which is also usable for Yocto and for the tiny profile path.And you just select basically what you want to build,which board, which configuration,and in the end you have this directly bootable images.So to compare the two approaches,the ESA path is you get the compatibility to Debian,so you get an image in the end where you can just,if you like to install further packages on the target,when you are developing still,or you can pull from all the compatible sourcesof Debian packages onto your target,while with a tiny profile you have a systemwhich is confined first of all to a subset of Debian packages,which have been put into recipes,but in addition you can pull from the Yocto Worldadditional components you want to have there.Clearly the goal for the generic profile is,well,a bigger size system,or bigger these days is also relative,so anything starting from 100 megabytes,plus-,plus-,minus,is the normal goal for these systems,while as Caso said,for Debian you can go much smaller.Compateria already mentioned,on the ESA side the binary packages,on the Debian side the Yocto recipes.The skill set is kind of similar,BitBake Yocto is the common sense,but in addition on the ESA side you should have a little bit of knowledgeof Debian packaging to do with things properly,for the non-trivial things at least,for the simple things I said,there are automation for that available.Major difference is of course the build time,as we are generating not too many packages with the ESA pass,you're usually up to a few ten minutes for an image build,while if you have to build everything from sources,you can do the mask that can easily go into our dimension.Regarding the customization,needs for these two paths,clearly if you go the generic profile,the ESA path,that is only needed,the goal should only be to have a few packages customizedand the rest just taking as they are,while with the Debian profile you can,tiny profile you can go much more into customizationswith all the pros and cons you have of this.So yeah,typical systems for the generic profile,IoT gateways,edge devices,industrial controllers,everything of a certain size,while with a tiny profile you can go down to small IoT devices,everything which still runs something like Linux.So of course these paths also need to be tested,so we have set up,not too surprisingly,complex test infrastructure where we push the buildson both sides into GitLab,so CIPproject is hosted on GitLab.comand trigger the necessary builds on them.We have a AWS-based build farm behind it,which is a project of its own by the way,so we can also use it if you like to,where these images are being producedand the artifacts later on being pushedto S3.And from there on,in parallel,the build also triggers a lava test,so a lava master is running in our laband we have a couple of member labshosting a few boardsthat will then take these jobsand execute,pulling the artifactsand executes the test run directly on the targets.So how to apply this now into a product development?So normally if you build a product,you are not happy with just taking what we distributein form of a layer or in form of package list,you want to customize,you need to customize.That means you need to put at least one layer on topwhich describes your productto,well,install your customer applicationto do further adjustments to your target,to add some BSP which is not in scopeor not supported here as is.For that,both approaches,as I said,are based on the BitBakelayering concept,so you canput these layers on topand manage the customizations in this way.You can also manage,of course,morecommodalization,so if you have a product line,define common product layersor even go into your mapping,your corporate structure on this,if you have for your division a commodity,you can also put it this way.And as I said,these passes are supported in both profiles,even if the technology underneathare in details different.Actually,this is currently also the approachthat CIP members are applying,so we have internally in our companiesthese kind of structure.We have some business domain-specificcommodality layers often,which add further internalor open source componentsspecific to the domainand then further on havethe product layer describing theindividual product-to-product line.And that happens with both systems.However,there is now the desireto make this easier integratable CIP,so currently our internal,our CIP layers are primarily targetingthe testability of our infrastructure,so they are in early stage.So what you see on the right in the structureis basically just using the build systemdirectly underneaththe CIP layer.This is what we want to change.So we are heading for makingthe CIP layers relevantand interesting for direct product use,means providing real-life releaseswith tested dependencies,also including the build tools behind itand the board supportsand the things like this.And also providing repositoriesfor mirroring our dependenciesso making them long-term available.I mean by now many of them identicalto the upstream components,but if you think five years ahead,that situation will eventually change.Not all of these will be supportedby upstream communities anymoreand CIP will have to becomethe source and then beprovided in one place.So furthermore,as Kazum mentioned,we are working on these package set,package list,and that of courseare encoded as a reference imagein these layers.Once it has been concluded the list,we will add this to that.It's going to be soon.And last but not least,as multiple workgroupsare working on componentswhich should come into test as well,which should be part of thisone-stop solution,so to say.We want to integrate them,preintegrate them.That is specifically relevantfor the software update mechanism.Which,of course,is a mechanism which will requiresome product specific customization,but the existing patterns could bedemonstrated this way and the existingcomponents we integrate could betested this way.Furthermore,we are working onmaking the CIP core alsocertifiable according to IC standardsfor security and further mechanismsand further configurationsand tools will have to be integratedand those should also beintegrated,preintegratedSo,to summarize,So,CIP provides a long-termmaintained open-source base layerconsisting of kernel andessential packages right now,adding further features in the future,providing or aiming for asupport time of 10 years plus.We,the CIP core project,the CIP workgroup,the core workgroup within this projectdefines this package set basedon the input of our membersand of the workgroups,all of these elementsin a central layer.We have two flavors for these layers.Debi approach for smallerYokto OA compatibleprojects,while the ESA CIP coreis targeting at largerand Debian binary compatibleprojects.Both are still early in thestages,but more features are tocome around software updateand security hardening,as I mentioned.With that,just one note,We have a booth up therein the showcase room.Come and visit us.Look for the LEGO figures.There's also a CIP minisummit on Thursday.Unfortunately,we already sold out.But you can always approach uson question at the boothand when we're around here.Thank you.So you're using Yoktoto build the Debian sources.Yokto also comes withit's own upstream provided sources.Why did you choose Debian sourcesover the Yokto sources?What are the prongs and cons of eachlike source distributor?Basically,in the Yokto projectsources,upstream sourcesusing the Yokto project isthe newer.That ispros to use theoriginal Yokto project metadata.But at the same time,the Yokto project metadatais not responsible for providingthe long-term support.So that is why the mostbiggest reason we arejust to choose the Debian sourcesor as our targetin the meta-Debian and Debian.Thank you.Could you back to the slide numberfour?Thank you.Thank you.Are there anything tosolve license-queering phrase?Are there any tools?Do you think?How do you think?It is currentlythere is no activityas the CAP.But actuallythe companywe are doing someclearance phase.For example, wemean the Toshiba cooperationusing the phosologyto clarify theincluded licensing target.Yeah.Same situation at the cement.We are currently not yetmore achievable or morebetter situationto share things betweenthe companies.But inside our companies, weare trying to make the singlestop for the license-queeringbecause currently the lawyersnormally trust themselvesand not the others.To the clearing results areconfined, at least for thebinding results to thecompanies.But inside thecompanies, of course, there issignificant sharing possiblethis way.And there's alsoone of the motivations tothe clearing resultseternally, at least, in ashareable reasonable way.A few mics, yeah.Sorry, Jan.Hi.A question from my side.How do you handle theYokto branch compatibility?Because Yokto moves fastand which issues do you have?Thanks.Are you going todepy approach in thecompanies?Currently, the warrior 2.7maybe is the targetversion of the Yoktoproject.And itbasically depends onthis version.So, we are currentlynot responsible forsupporting other versionsin the debut repositories.So, that meanswe provide one branch namewarrioralso in themetadabian repositories.And we need to usewarrior branch of themetadabian with thepokeywarrior branch.At least, you know,answered for your question.This second part waswhich issues do you have?Because at some pointyou will have to upgradebit-bake compatibilityand other things thatwhat do you mean?At least, bit-bake compatibilityit is not compatiblebetween the recipesin the open embedded coreand the recipes in themetadabian.Because someof them depend on theimplementation ofpokey recipes.So, in that case,last case,we cannotget the errorsin a bit-bakepassing phase.Any other question?Otherwise, thank you again.