 Hello So I think it's start it's time to start to 25 Thanks, everyone. Thanks for joining today and Cam and I are going to talk about the Octo project and the RDK and What what we've been working on a couple of years ago and what is actually still a very active project? Okay, first a quick introduction. So my name is Nicolette Shen. I work for Linaro and Where I do a bunch of Linux and the Octo open-ended related stuff, and I'm also the committee manager for the Octo project So I'm Cam Raj. I work at Comcast. I basically I'm a RDK architect Primarily responsible for all open-source activities around RDK so a couple of years ago both Comcast and Linaro worked together on On I mean with the RDK software, which is the set-top box stack, which is actually used by Comcast and other MSO and we actually Worked and took what we use used to be the RDK before and we actually migrated everything to the Octo project Using all the practices and everything from the Octo project. So that's basically the the reason of the talk today Is to show what one very specific Ecosystem has been able to do and to build with the Octo project We are going to talk about I mean what is the Octo project in case some people don't know What's the RDK? some of the issues that we had before and Why we actually ended up choosing the Octo project to actually use as a baseline for the for the RDK And then what happened? So we started and we started to deploy and work with the Octo project and what are the benefits and actually also the challenges that We've had more from the Octo project side and from the from the RDK side So what is the Octo project? So that could be a very long discussion, but we are going to make a very short picture So basically that's that's a tool set. So that's a set of tools a set of processes That be allows you to build your own Linux So whether you want to do a small Linux a large Linux whether you want to do that for the embedded space for Cloud for anything containers You when you need to build your own Linux You have the choice to either use an existing Linux for many an existing vendor or you can actually use tools to make your own So that's the very high-level pitch about what it is It allows you to make your own Linux and customize the way you want so you can do anything with that after that So in terms of the project itself, so it's a it's a Linux foundation project. It's been a hostage since 2010 The Richard Purdy has been the main architect. I don't is here this weekend. Don't see him in the room But he's been the main architect and the commuter for the project since I mean since it started It uses the Octo project build system is basically open embedded So there is a very strong relationship between the two open source project They're open open embedded and the Octo project if you have any questions about that I mean you can always come and ask us It's basically support all kind of major architecture I mean our makes 86 Meep 65 I mean everything that exists today is pretty much supported by the project It's has become one of the main system which is used today to build Linux It started from the embedded space and what we see now It's actually use more and more outside of just the embedded space We make releases twice a year April and October We actually just made the Octo release 3.0 release a couple of days ago Why people should be using something? I mean the Octo project Basically the idea is that if you build your own Linux And if you hand up with your own tools and on set of scrapes or whatever I'm just called what it is You are basically spending all your time building your own build system so by using some standard tools and processes it comes with recipes it comes with Processes and you can have updated up-to-date recipes for pretty much every software that exists in Linux I mean with all the the component that you need to build your own user space have their own recipes are most likely already Existing and already supported by by the community you can very quickly Build your entire image and and and get started with with very standard Linux system We have support for package management for like advanced use cases where you actually do need to use packages or Make binary packages. That's something which is also possible and which is most of the time not possible to do by hand and Predict predictable so basically if you build today I mean there is a high chance that you will be able to build I mean next month the same thing that you build so that's actually all the Goodness and the things which are built in and and given for free to you by when you start using the project. It's It's it's used quite I mean so it there is there is a membership around the Octo projects So there are many companies like you sponsor and use the Octo projects So basically by relying on that project you use something which is standard and the standard in the industry It's flexible. So even though I mean you can you can actually build completely the the Linux system It's completely flexible in the sense that you can customize any package that you include You include the upstream packages and you choose how you compile Which piece of the system you need for your for use cases and you can actually tailor your image to exactly what you need So I'm gonna talk about R.D.K. So essentially what you see here is a R.D.K. video stack But essentially that's where we started R.D.K. Is essentially our home operating system That is actually started with video, but right now we'll talk about a bit more. There are more profiles we have But this is just a high-level picture that you will see how what are different components that are in there So you've got like certain OEM based components So this is a software stack, of course but you know these are like kind of various owners that contribute to R.D.K. stack and And then you see in the middle There is the R.D.K. stack essentially And as you can see there are many components in there And obviously you have application layer on top where you have all your UI and other applications The what you can see in there is that there is a a lot of open source components that are in there so Now given this What we had was to give you a little history and there was a R.D.K. build system that we had early on and Eventually we basically got on to Yachter project and Nico just listed few of the reasons I'll go over those in In short but when R.D.K. started it had its own build system like we all do many times That's how we start and it was called R.D.K. build system very innovative name And it started fine You know they were like bash scripts grew into Python and then Various of the things so people started adding you know their own build logics and It just went at 300 miles an hour no documentation After a while even you know the people who wrote it would have to refer to their memory and they were not finding the answers So it became very difficult to maintain right so Obviously, it's easy to snapshot the components, but then it's hard to maintain them later on so if you Snapshot Linux kernel, that's fine But then you want to upgrade then you have your own patches. You have other patches So you end up basically in a very weird situation where your velocity is impacted And obviously, you know, there are other tasks not only upgrades. You are scaling you are building new products So essentially build is the last thing you want to be doing right so or developing Soon or later, there was a realization that the project had that build is Not the differentiation that R.D.K. has So it's better to use something and share with the rest of the community So essentially I think what Niko is going to tell you is basically What 1.0x looked like in terms of high-level architecture from R.D.K.'s point of view and then how we transformed it into Yachto eventually So it's it's quite easy to understand that if you make your own build system Which is very difficult to use and so on that's a problem Not upgrading your system very often is a problem But there is a biggest issue that we had before is that because everything was so complicated What ended up happening is that we we ended up with this very weird pyramid where Most of the software that was actually in the product in the end came from the baseball vendor, which is Hope I'm almost the opposite of what you would expect and you expect from the vendor I mean that the vendor and the BSP vendor is basically whoever provides the SOC where this stuff is going to run The only thing you expect from the vendor is more or less like a Linux kernel Which is one very small piece of the problem But what we've seen is that the build system was so complex to use that the SDK that came from the Every vendor started to grow and grow and grow. So every vendor that was working with the SDK They started to add the Linux kernel and then the busy box and then they started to a g-streamer and they started to add They started to customize GCC because they could and then they started to basically add Qt5 and everything and everything So basically in the end The Rdk project and the MSO I mean the companies like Comcast even though they actually own the platform if you actually look at What's running on their own platform? They actually didn't own much of it. So that was a very big problem So the the base port as the whole the whole power to actually control what went into the product And that was because mostly because I mean of the build scalability issues and and the intrinsic issue because of the build systems so What happened? We started this project together and we we sat down and we basically wanted to unify all the build systems And and all the values vendors and use cases from the Rdk into like a single build We definitely wanted to address the ownership and the responsibility so the MSO and people on the layers above should be responsible for the for most of the software and And and yes, I mean of course, I mean your project was I'm chosen as the platform to build the system The and then we wanted to also address the other kind of issues like being able to much easier upgrade the system and and more importantly We wanted to make it very easy for like a new base port vendor Like if you if you want to pull this setup box software to a new SOC vendor That should be very trivial or if we on the other end if we wanted to add a new user of the Rdk like a new MSO System it should also be simple for them. So that's that's the kind of that was the goals when we started the project So when we we ended up with this thing which looks more much more better Basically everything is actually coming from the Yachter project from the open embedded side So most of the software you need to at your own Linux already exists and it's already supported by the project And then you start stacking stuff And layers of software on top. So you have all the generic APIs that are needed or frameworks That are needed to run the Rdk and then the customization from the Specific MSO I mean like for example Comcast and then eventually you basically had the the small pieces that you expect like how do I get My GPU to work. How do I get to boots and these kind of things for the baseballs? So This is actually an interesting diagram It shows at the end of this project, which was more or less like a year Even maybe in eight months. Maybe we were able to basically port the existing Rdk to pretty different Vendors like one was an arm mips and an Intel platforms We had a single build and that's a single images. Everybody was actually using the same image. Everybody was using the same Tools and configurations and you just change and just give the machine name and you would be able to generate a new image When you look at the content of the image That's actually a graph that was made from that if you look at every file of the image And if you look where these files are coming from basically that tells you that Three-quarter of the system comes from the recipes metadata which are actually coming from open embedded It comes for free so which looks much better than if you remember the previous pyramid from before So you definitely own the complete system and the base port now is a very small It's actually one of the small part of the system, which is exactly how you would expect things to be So what that means is that now 75% of your system comes for free Which means if you want to upgrade or if you want to make any change You can apply that change and it will be deployed to all your All your variations all your flavors of the ADK for all your vendors On top of that what we also realize is that if we have one issue that the That the development so that there is the platform itself But then there are all the applications that actually runs on a set of box They were developed on the hardware themselves what we realize if we instead of building for a specific hardware We would just build for QMU x86 We would be able to just run the same image that actually runs on the real hardware on the emulator So basically people who don't care about the low-level stuff and only cared about providing a Applications for the MSO they could actually develop on the on the PC just I mean without even worrying about the board So that was a very nice addition that was not planned at the beginning that just came for free by actually choosing to Do what we decided to do okay, so I think you know As I mentioned earlier We wanted to differentiate where it mattered and we understood that build system is critical. It's It's actually a backbone and you need to have a strong one and that's where you know your project helped We could go to different as to see vendors and you know They basically had a consistent system to deliver into And obviously once they did it for one and they could easily reuse it for other projects and eventually it also helped our kind of like Lowering the bar of entry for a new or SSEs So that was an added advantage because you know you were not like R.D.K. Wasn't vertically integrated and that provided a lot of You know benefit to Because R.D.K. Is run actually it's a open project so therefore, you know Different users have different needs. They have different SSEs and it's very important that a New SSE can easily be ported R.D.K. Into their ecosystems and and Then we could define standard health That we could then ask various Portors to work on so essentially as you can see as the triangle kind of you know inverted that gave a lot of stability at the bottom and That gave a consistent set of health that we'd work with right so And obviously we also had a lot of community knowledge about building systems which also helped in Growing the R.D.K. Community especially on the platform side And we could offer rich set of tools and packages for new app writers so They have a lot of dependencies right on Applications they develop and many cases you would get a bundle right and on embedded systems You you don't want to ship three different versions of same library most of the time without any solid reason So this helps you to kind of you know, shed some weight in that area as well then because your platform is common So so it ended up you know in creating this standard template for us that basically also helped us to scale Vertically into our stack and also scale horizontally that I'll cover a little bit so Because now we could focus on where it really mattered as I mentioned So you would see that you know our teams came up with a few interesting components essentially Vestor us which is a valent compositor very small has like embedded focus can do nested compositions and And is very suitable for like a set of box operating system similarly, we have a UI engine that you can do native application rendering There's an SDK and that we came up with because you know, it's based on open embedded and the SDK features that we have We could basically have WPE which is basically a Webkit for you know on top of Wayland and it's kind of optimized for embedded systems and Open CDM so you know as you can see what it's a video specific Tasks, but what it ended up was filling in the gaps where we really needed to concentrate And then we went horizontal which means that there were different started with video But then there were more projects like broadband routers, you know cameras smart cameras Wi-Fi access points Wi-Fi mesh We could scale horizontally because Given the scalability that we have flexibility we have we could create, you know different profiles Seal use same tools so we could create images which are really really small and Then we could go across have very complex tags as I showed with the video for example so this provided a very solid baseline to Go into those directions that where we are today after a few years So One of the pain points were build times when we had our own build system our build times were linear, you know, they were Humongous because we didn't have any build acceleration technologies is underneath and we didn't do shares like common builds and those were like straight away benefits we got from Yachto and Incremental bills were huge Shared state another feature of Yachto that we use today is a very good first-time experience for somebody checking out a new tree and trying to do bills and There's a lot of documentation. I know that there's a bit of Probably Unknown in that people probably don't know it has a lot of specific use cases If you go through the mega manual everything is in there So we started there and our developers really loved it from going from zero documentation to you know ton of documentation and As a result actually they came up with FAQs that were specific to our decay and that's very valuable for us They have certain playbooks that they have created That are specifically based on the documentation that the October project provides if you're doing something similar internally that you know I recommend that for your own use case to you know train your own developers that helps a lot and Obviously we use the licensing tools for our compliance and we have testing infrastructure that we Indirectly depend upon all the testing that is happening on emulators. For example in the Yachto community and then we also rely on the security patches that are being you know done by the community and Previously they were all onus was on us. So in this a lot of benefits in that sections that we have been having There are challenges of course learning curve main region for that was adopting a different software and The cultural change where instead of you know having an internal bill system now you have an open source bill system So it wasn't more about okay. It's hard, but it's different And you have to get to know the processes and how to work with it Developer workflow when we deployed Yachto project it had a lot stronger release focus That problem was fixed but then developers ended up with you know a Suboptimal workflow or at least they wanted improved one and Then we work with the community actually on the dev to Project that was added actually to Yachto project in past three to five years and That solves a lot of the issues that were deported by the developers in their workflow improvements The project upgrade is always an issue and we basically find that there could be some improvements in there Where you know we could rely on a more collaborative release that can be maintained for a lot longer maybe a few years so there are kind of ideas around those but you know That's actually a challenge. Once you deploy it how often you can upgrade and how much effort it should be So we have a focus on that where we want to make that nimble basically have easy way to upgrade And then obviously, you know the build time improvement was there, but then of course, you know more is always better and There has been features that has been coming into Yachto that we basically Will will work on deploying even into Rdk hash equivalence is one of the latest features Where it can do a much better job identifying something not to be rebuilt so So I think there's a detailed talks if you're attending Yachto summit you can talk to experts find more about it But it's a it's a very cool feature. It's kind of a game-changer in the in the build area I think essentially where you can now Deploy your pre-built artifacts Smartly and not end up rebuilding things accidentally. So that kind of gives a much better You know view for end users too when they don't have false rebuilds happening so Yeah, so I think Having said that That's pretty much. I want you to share so if you have any questions or comments Or you would like to ask specific questions because this has been happening over years I might have forgotten many things that you know, we went through I would be happy to answer and project kind of view The the main reason why we've wanted to I mean have this talk here was to basically explain I mean how we actually did solve like a real life problem. I mean, it's it's been a bigger success story both on on both sides actually on the on the Yachto project side I mean is it and this is like a huge user base for the for the octoproject But that's also a good success story for you as you've seen I mean by focusing less on things that you don't care You can do more. I mean, that's that's the basic thing that we always say but we now have like some some really good evidence of that Yeah, so any And and what has been done once I mean it can actually be done in different Ecosystem as well. Yes Thanks. Yeah, what was the particular challenge in adopting open source culture? Yeah, that's for you it's a so essentially I think Look change is always hard right and One of the challenge you always have is you're familiar with something but Sometimes you know one of the things you you have to realize your development teams is and They will always say we are doing well. We are running fast, right? But you have to tell them it's a hamster wheel that you're running up, right? So that's the challenging part. How do you make them realize that that? The things that you've been doing are suboptimal, right? And so in many cases we did that where we basically created, you know parallel workflows Right and demonstrated that what we do today and what we can do in future And then contrast to them So a lot of kind of these kind of documents do exist where we do compare and contrast certain things that were strong points of Culture so you have to approach it from database It has to be backed by data Right though preaching doesn't work All right, so many times people know it is right thing to do But if you have to make it effective you have to back it up with data So we always created use cases where we then went to the people with data and they realized that this is a better way of doing things So that was I think one of the learning that you know And once you present the data, it's a very compelling case to adopt And speaking about data this diagram where I was showing this 75% and we actually this is a slide We showed I mean the management say I mean this is what we've done I mean this is data. This is not just what we would like to do This is what has been done and this is how much you can actually say if I mean This is a leader we've presented that to the management at that time Yeah, so I think you have to pick the battles and you will know those internally in your own companies and organizations What the battles are right in many cases people might have Acceleration tools already deployed right and then what there might be different Challenges that you're running into but the template is always same collect data Right and then compare and contrast and then then talk about the benefits a Lot of trainings that helps to if we are like new commerce right and many times people are eager to learn but Trainings is actually one of the big things that help them to onboard quickly So think about trainings Quite a lot right and specific trainings to your use cases Many times, you know, you might not want to do I mean everything is not relevant to you So just call out which areas will be the onboarding areas that you want to do So sometimes that helps quite a lot with your teams and from the project perspective We we are very well aware that I mean this is there is a very I mean difficult learning curve I mean this is nobody denies that so what we try to do is we this is also why we organize I mean developer days at each of these conference. You might have attended some of this We also have provided and actually support a really good documentation as well We are always hoping to I mean how we could do more and better things and there are lots of companies I mean out there that actually can help you and train your teams I mean in many different languages everywhere. So we try to make sure that within the ecosystem We actually enable these companies. So that's what we'll try to do More questions. How did your developers react to adopting such a change in the build like chain of your product? Because most of the times what I've seen is that developers don't really care about the integration in the build process They only want to code their Component and then somebody else handle the entire chain of deployment integration that that all Have you included some of these activities in the garden or are you handling completely independent of them? Yeah So I think it's a very good question. So I think if you see the triangle it's always It's a funnel, right? So you got like system developers kernel developers handful of them and then you know you got like middleware and then you got like a large set of application people and So we did talk about emulator for example, right? So you have to kind of create those Those use cases where they feel happy about because you know that you are changing underneath things that will make them unhappy so so once you have like an emulator like platform where They can develop their app and test their app 80% That's a big boost for them because they realize that doing hundred percent on This app on it puny device is very painful and think of debugging You know webkit even on Raspberry Pi 3, you know the stack traces don't show up for 10 minutes, right? Right. So that's the practical problem But when you can provide additional tooling where you say hey, you know the process is gonna remain same so ease out their workflow and They are they're basically More susceptible to accept the changes that you're bringing in You also have to approach it from the process side where you know you have the DevOps team so we basically in enlarge Developer teams process has to be there So you have to basically look at take a very hard look at the process you have and Then design the tools to fit in into your process many times those are not pretty and You know we might have a tool that is Pretty neat, but you might need to add the glue to it To basically rough the edges so you can ride into the developer Workflows and then once you are in there then you can make like optimizations improvements, but I think Again, that's use case by use case I think the Changes are very common, but what I saw was if you were to say hey, you know, I'm changing your workflow They were simply say no, right, but if you say here is a better way of doing it Would you like to adopt it? Then they adopt to that and then you can change the workflow for release and all those things So we found that's more effective, but there is a consistent resistance that you will always have So if your developers are not happy, it won't fly So essentially, I think of course you have to measure a lot You know sometimes we forget about that, but we put in a lot of measurements. For example Our auto builders measure build times and they plot them over time, right? So, you know, if you use Jenkins or whatever it might not have that feature built in you might have to build that on top Right. So what it gives you is it gives you insight into What it is doing, right? Is it bringing any benefits, right? And how it is going right as developers adopted Is it getting worse it is getting better? so so I think more data-based approach and then understanding the inflection points where you can insert those workflows, so I Find that very useful and people are very open to adopt It might sound very counterintuitive, but you had data where I mean it was showing that Using the Rdk after the October project they actually save build time So I mean if you see if I mean most of the complaints that we hear about the project It is built takes a long time and download and everything But then if you actually apply that to a really I mean if you only make one image Yes, maybe it's wrong, but then if you actually use that on your daily for like Plenty of images and plenty of machines and everything and you actually start making them count of how much you save I mean you did you do end up saving a huge amount of actually build time Yeah, and I think there's another thing where you try to basically I've seen you try to push it through You have to identify your users pretty well And in many cases an SDK an application SDK would make perfect sense, right? And if you push the whole build system into them that might not be the you know optimal solution So you have to work through those depending upon the use cases So the beautiful thing about open embedded the October project tooling is that there is a a set of tools You have available to you now how you use it to improve the productivity and you know effectiveness of your developers Would be slightly different so one time one size won't fit all right So if you have like 10 developers Maybe just using the full-build system is fine, but you have 500 developers, right? You might want to give them a consistent Platform to work on and you might want to have a more federated model where you know you have application development You have a platform development happening separately and then Devise a CI system basically which marries both of them So but you can think about those things because now you have a solid platform. That's my methods Okay, so I guess we are on time One thing just last word. We are here. I mean the yachter project is here at the conference We are the showcase and we have a booth so there will there are lots of people from the project here So if you have any questions wants to talk about anything You can just come and see us and then we also have on Thursday and Friday a two-day conference about the yachter project so It may be some of you will come there This is the first time we do that we do a two-day conference We call that the yachter project summit and the hope is to basically put users of the project and Developers and do the same home and just talking together and just showing us what people are doing with the yachter project So I encourage you to actually learn about that and just obviously come and talk to us Thank you. Thank you