 So, my name is Michael Brown. I am the lead technologist for our next generation chassis controller at Dell. Still getting used to saying Dell EMC. So, you'll hear me still but just say Dell once more. And what I'm here to talk to you today about is updating your Yachto environment and setting up strategies or setting up one method that we have used in my group to keep ourselves up to date and make it a little easier. So, as I said, questions are important. Feel free to raise your hand anytime. Ask a question. My job is to keep the thing going and end it on time so I will keep it. I will cut things off if I am starting to go a little bit long. So, who am I? I've been at Dell, now Dell EMC for 17 years. And I am the, basically you could say I'm the Yachto architect of the Yachto environment at Dell. So, kind of a funny story how I got into that. I was on the Linux team for a very long time and we had a couple of other folks on the embedded side that did a lot of evangelization around open embedded and Yachto. And anybody that ever came to me, I did a lot of containers, VMs, bootable ISOs, that kind of thing. Anybody that came to me and mentioned the word embedded or anything non-X86, I said, go talk to Sean Hudson. By the way, he's here. He's presenting. He has a couple of BF sessions. He's a great guy. I recommend checking him out. But he worked at Dell and then he decided to leave. Three weeks later, I get a call from the director of the embedded team asking, can I come over and lead an effort to port away from our homegrown system and use some kind of standard base, which I had been advocating for several years. Kind of pointing at Sean to do that. But that's how I became the Yachto guy. So currently working all over on next generation stuff. I can't get into too much details about stuff that's not shipping, as you guys probably know about. But I can talk about the development process. So the problem, as we talked about a little bit before we started, this is about keeping your development environment up to date. It's not talking about device updates. There's several other sessions in this talk that talk about actual updating your device. And I recommend going to them. I'll probably be a few of them. So but as a prerequisite, before you update a device, you kind of have to build that updated component and you have to test it out and get it all working before you can actually put it on a device. So that's what we're going to talk about today. Keeping your development update updated is very hard. As we kind of saw at the show of hands before we started, the majority, I would say, of people in this room are using not current Yachto. A lot of reasons for that. One of the things is kind of inertia, fear of the unknown, and breakages. So when something breaks, if you break a development environment, you're putting teams of dozens, possibly hundreds of people, kind of out of a job for a period of time until you can get that stuff fixed. So as you can imagine, if it happens once, happens twice, people get gun shy about updating their environment. And it happens pretty rarely. So on the other extreme, when you get a really out of date environment, we're shipping IoT devices that we talk about in this conference, we're shipping embedded, in my case, embedded server management. We're shipping stuff that's really old. And if you get a security vulnerability in one component, sometimes there's interlocking dependencies. If I want to update open SSL, well, I need a newer version of GCC. It goes on and on, and you get into this situation really quickly, where you physically can't update to fix a security vulnerability. I did update a system ported into Yachto two years ago. We were on a 2.4 kernel. So it can get kind of hairy trying to update stuff that's that old. So I'm going to talk about that on the next slide. So I'm giving this talk to specifically talk about Yachto, but the concepts I'm going to talk about here can actually be applied on most systems. I have one slide, it's the big idea slide, that you can take that and kind of cookie cutter it. So Dell, we put what I'm going to talk about in this slide. We put this in place two years ago with our next generation. So we've been in development for our next generation of server products for a couple of years now. And at the beginning of that process, we put this system in place. And we've used it to update through four major Yachto releases in our current head master development environment. So we've gotten this pretty much down and have a benefit slide at the end. I need to save some material for that. So we've used this to keep ourselves up to date. I have to admit right now that we are certain right now sitting on 2.0 plus a little delta because the guy that was heading up this project, he was a new college hire. We put him in charge of this. And because of the fact that he updated the environment, he got familiar with everything. And he became one of our go-to guys and became really valuable and got a raise. And so do this stuff for career advancement. And he ended up getting pulled off of that to do other things because of the knowledge you gain directly from doing this. So this might seem like a drudgery type of thing. But from a career advancement perspective, it's a really quick way for a new guy to get familiar with your entire system. So the Dell EMC firmware team, specifically the people using Yachto, all of our embedded server management, so our iDRAX from our 12th generation onwards are all running Yachtos. The chassis management controller and all of our chassis, the M1000E, the Vertex, all the other chassis variants are all running on a Yachto base. And all of our in development systems are all for our next generation to be released servers. And chassis are all running on Yachto. Of note, we've actually come up with a nice little way to have one code base that we build all of these different products on starting with our next generation stuff. It actually get more bang for our buck because we can update our Yachto base once and update several products. So the question was how many developers you're talking about? Liga won't let me tell you, sorry. It's a large development team. So our Dell Yachto environment, we basically have Pokey plus MetoE, which hopefully everybody here kind of knows that. We have roughly 300 individual components that we use, and those are all individual standalone components, each with their own auto tools, with their own bit bake recipes, et cetera. And basically I wrote almost all of the recipes, all the auto tools and everything, so any questions about that stuff I can pretty much answer. So we have about 300 Git repositories to build all our products. Managing that is a little bit challenging. I don't know why that's blue, interesting. Anyway, it's not supposed to be highlighted. Export error. So we use the Android repo tool. When we started this back in 2012 is when we started the porting effort, and really we had Git sub modules, which at that time I couldn't wrap my head around, and we had the repo tools, the only real two tools we had to manage hundreds of repositories. So we chose repo tool, and heavily based our environment around the repo tool to a lot of success. Today there's other tools that can do equivalent things. So if you're starting out today, you can start with the repo tool. There's a couple other ones I heard about. It's names escape me at the moment. Stress from presenting. So our environment is a full source build. So every component is all built from source. So every developer checks out and builds everything from source. So we leverage things like the YachtOS state cache to speed that kind of thing up. From the way our tree works, we have, check out at the top, we have the 14G, you create a directory to check your stuff out into, and we have a directory structure, where we have Poke, MetaOE, we have our own custom layer MetaDRAC, where we have all of the Dell BitBake recipes. We have a policy in place, and this is a little thing that I highly recommend, we do not allow check-ins to our copy of the Poke repository. So we always use straight standard out of the box upstream Poke. And then we do all our stuff as BB appends or writing our own classes. And then so all of our custom BB pans or all of our custom bit bake recipes are all in the MetaDRAC directory. External source, so I haven't really seen many people do it this way, but we check out all of our source tree, all of our Git trees under individual and external source. This talk isn't about that, so I won't go into too much detail on it, but we leverage a custom written external source BB class that I wrote before the upstream Yachto got one that lets us do our builds of our components inside of our Git repositories. And then our builds are separated out since we build several products from one source tree, you'll change your directory into one of the build subdirectors for the product that you're trying to build for and do a build. So this is kind of like the way the development environment is set up and you'll kind of need to understand that to see the next part. Question. So the question is copies of public repositories, do we copy? So internal to Dell, firewalls are a thing, so anything that we build, it's like Poke, Meta, we have our own internal mirror, and I treat those as if they are the public. And then we have a few repositories that we just mirror and so there's like maybe a half a dozen to a dozen, so like the kernel, Poke, Meta, OE, and a few other things like that. Most of the other things that are external Git repositories use the standard Poke Git mirroring tar ball where you'll download a tar ball of their source and plug it in. So I've cut out a lot of that stuff just for time. So branching, we have a naming convention we use for our branches, REL for release slash 14G slash master, it's our current end development. And then we don't have any branches off that yet but our other products that are already released would have a REL 14G 1.0, 1.1 and master branches. So we have a pretty hierarchical naming convention so we have can group all of our tags, group all of our branches and you can easily see what's related to what. From a reusing, for example, with Poke, we don't use master at all in any of our repositories because it conflicts with people's, if we're pulling in an upstream repository and using master, it's kind of hard. So I like to take their branch and pull it into our naming convention. So that's that stuff. So, the big idea. We have a separate build called Poke Next and you're like, okay, we came here for this. But it sounds really basic but it's really the core of keeping yourself up to date. So, the core requirements and then the next slide, I'll show you the kind of directory structure. We have a parallel build structure. So you can build, I pointed to the external source directory. So, nobody in this room is shipping Yachto. People in this room are shipping your app. You have your code, you don't care about Yachto, you care about your app. So this is a parallel build where you build your code against either the upstream Poke that you're running on or Poke Next. And because we set this up so that you can do either or, you can get your code to run on both of them at the same time. This also means that there's no pressure that you have to have that, I have to get this working within a week or two weeks or you can update the Poke Next and if there's a bug with it, it's not stopping anybody from doing their work. They still have their normal Poke. So, small units of work, once you get your code working against Poke Next, you do dedicate one of your college hires to do updates of Poke Next. Once every three weeks to six weeks is what we found kind of works. These are small incremental updates. You're basically just doing pulling in the baseline from upstream and usually nothing will break. So if you do every three weeks, you do a get fetch and then push into your Poke Next, the new version, it'll just build. In our case, if upstream updates a version of a component, we might have to update a tar ball because we can't access the external world. So the every three weeks involves pulling down a new tar ball. In our case, this can work several different ways. The way we've done it and what I'm gonna show you in the slides is we have two separate source control repositories for Poke and Poke Next and we control the flow of code between them. So the benefits of this is that you can work on the hardstuffs, so a Yachto update breaks or is incompatible, maybe there's a GCC update. You have to go fix a bunch of warnings or compilers in your code. You can kind of do that in a way that works with the current Yachto release while making it forward compatible with a new Yachto release on your schedule and when it's done, then you can switch over. So we're talking about parallel build structure. We actually have this .next directory which will be hidden from the normal developers doing an LS so it doesn't confuse them. But we have another build directory under there and so you can change directory to build slash CMC under .next and the BB layers are all set up to pull from these Yachto repositories but the common code base. So we can do, so developers can do a build against the next version of Yachto or they can do of the currently working version of Yachto. So you have one guy that's just every few weeks, he does this update. He sends emails out to everybody who has new compiler warnings or new things that they need to fix in their code base in the .next and then the development teams can schedule this. So Shilpa Hins, who's using Agile? Okay, so you have your sprint planning and your sprints so the person doing the Yachto next updates can then write stories or write defects against the code for Yachto next and then you can plan this out just like any other development task. It's not, the updates can tend to be unpredictable. You don't know what kind of works gonna come your way. This kind of evens that out. So the get repositories. So basically, because we're using the repo tool and I like to keep all of the repositories, we check out on one branch. The way we do it is we have a separate backend get repository for Pokey and Pokey Next. So we can continuously update this version with the upstream version and then once you have this next version to your liking which I'll talk about some branching timelines, once you have this to your liking you basically can just flip it into so using the repo tool, you can just change repository manifest to then point to this repository or you can push code between the two repositories. It's kind of your choice. And repo tool, just the, in some people I've talked to, the repo tool seems really hard and difficult to use. It's actually fairly simple. You have an XML file. Here you can see we have a default revision. This is the branch that we're on. And then we have the Pokey directory and we're pointing it at that idrack slash pokey dot get repository and we have our dot next repository. So this is just a small snippet of our actual manifest for illustration purposes. Oh, questions? So the question is do we move Pokey next to Pokey between releases or at release points which is the next slide. I'll go over that in a second. You had a question as well? Yeah, so the way it works is internally we have a mirror and that mirror, when you update that mirror if you change anything on it it completely blows it away and re-mirrors, right? So that upstream mirror has just what the Yachto project get that Yachto project has. All the Morty and Korgoth and all those funny names that I can never remember. In fact, I had to update the slide last night to be current and had to actually go with the Yachto project and look at the Wiki to get the names. We have our own copy and what you can do with Git is I can make a branch and I can make my repository just have one branch and have it be just one branch from the source. So the Morty branch in all of its history in the upstream becomes my REL 14G master and I have nothing else from that repository. And I can update that using a Git pull from my mirror or from upstream. And from my developers perspective they don't see any difference. Answer your question? Okay, let's go to the, so this is the kind of timelines we're talking about. So this is kind of like entry level, let's keep majorly up to date every six months type thing. We're gonna go into a little bit more fine grain after we got this concept down. But what we have here is at the very bottom we have our upstream Pokey. So there's hundreds of commits per day to upstream Pokey. We pull them in on a regular cadence, three weeks, six weeks into our .next Pokey repository. So Yachter releases are six months long. And so for usually around four months or so of that kind of ignore it mostly, do a few updates, make sure it works okay. But coming on that four, five, six month mark when we're coming up on the release we try to pick up the cadence on this so that we are kind of up to date with the upstream. Couple of cool, nice reasons for that. If we have a problem with something upstream is doing we'll catch it and be able to submit a bug report and get it fixed before it gets set in stone. So this is one of the big reasons. And every six months of work this fits really well into an agile planning process. You can put stories in and you can kind of plan this out. And from 2.1 to 2.2 you can't predict how many breakages you're gonna get. But you can kind of plan out the kind of work you'll do per sprint and kind of guesstimate on the number of breakages and the more you do it the smaller the breakages are gonna be. So the more fine grain you get the less work you'll do and the better you'll be able to plan it. So then when 2.1 comes out we update it so a little delta, so you see the little jog in the line going up. We've updated our next repository and then I cut out one slide for time but that 2.1 up into the main repository there's a lot of slack in there and we can plan that basically around our release time frame. So we can do it immediately or if there's something important milestone where we need to do a release we can say okay we'll wait a month or so. So this is what gives us our flexibility on when we do our updates. Cause that next pokey directory is not gonna go away it's sitting there and it's buildable and we have Jenkins jobs that build it so we know that it's gonna stay working and once you get it working it stays working. So that's a kind of a principle it's not. If you got Jenkins jobs and you, I lied when I said it when it works it stays working because developers will break it but you can manage like one or two defects. If you have a Jenkins job that does a build once a day then you can file a couple defects and get that stuff fixed in a timely manner. So the question is are Jenkins jobs running automated tests and I think my personal opinion is that should be the goal of everybody is like you Jenkins you do a commit it triggers a Jenkins job puts it on hardware or an emulation and those jobs. Automated build verification is what we call them ABBT. So yeah in our world we do have a build verification test suite that runs and for the engine product I'm working on we have a full QEMU instance set up so you know right in Jenkins it fires up QEMU and starts running tests and it's really great. We actually got a lot large amount of our code based up and running before we ever got hardware and kind of amazed cool tricks to amaze your management team. So this is your branching and merging. So the nice thing about this this is great for long live development projects. So for example I said we started our next generation two years ago and two years ago I think 1.7, 1.8 was the current and without some kind of strategy for keeping that up to date we would today be in our final development stages of still using 1.7 with all the bugs and it's no longer under long-term support et cetera and looking at lots of re-validation effort to try to update it at the end of a development cycle and who wants to do that. So smaller units of work means that you can start parceling this kind of work out to your junior developers. This is a great way as I kind of gave away at the beginning the guys that you put on doing this if they're any good at all, this is a really good way for them to jump start their knowledge of how everything interacts and interconnects and they quickly become very valuable. So but you can start it out by chunking little small things, you can write up you know hey run this git command, run this git command run this build command, see what breaks. So it's something that junior guys can do and it gets them up to speed. You get your control over when you get updates in the production, so old way that I've seen, this is not a compare and contrast, I'm not trying to slam other methods of doing this but one other way I've seen them doing this is people have their pokey repository and they just do a git update on it and in there on somebody developers checkout and that developer hacks on it on his checkout until he gets everything working and then commits it all. Well the problem with that, lots of problems with that, A, you don't know how long it's gonna take him to get it working with the next version, B, you don't know how extensively he tested all that to make sure that everything is working the way it's supposed to before he checks it in. And you know if it sits there his code goes stale and then he has to rebase all your code on top of changes that have went in so it just becomes this kind of Red Queen's race to keep up to date on that updates checkout before you can check it in. So this lets you have control, everything's checked into version control and it's all parallel buildable and then you can work with your project management team to schedule hey two weeks from now on Friday we're moving pokey next in the pokey. From a political standpoint I mean when we put this in place it was a lot of resistance. We have old time embedded guys that just never wanted to update because they've been burned in the past and don't wanna do it. So the very first time we did it we had sign offs from everybody, everybody had to go into the dot next directory do a build sign off that their team is okay with it and we had a bunch of politics around that. After doing that four times now it's okay Friday's the date we're doing it. Speak now or forever hold your peace and it's people are more comfortable with the idea now. So it's a lot of it's a kind of a steep learning curve but once people get used to this idea it becomes a lot easier. Jenkins, we talked about Jenkins. I highly recommend whatever build system you have this automated build system is invaluable. We use Jenkins extensively. Okay so wow we're up to date with Pokey now but we now ship the product and now the product's in the customer's hands and we just found out there's an open SSL vulnerability. Now what do we do? Okay so we don't wanna do a Yachto 2.2 to the master in development version. Let's apply the same exact concept but now do it with a minor and I call it so we call it the dot next and the dot minor. So the dot minor is for your shipping products and you can do a 2.2.2.1 to 2.2.2 updates in using the same exact methodology and then so you can get those updates in do a quick build and test on them. Yachto products is in so far from what I've seen pretty good about making not breaking stuff in those minor releases. So it's a quick build verification test run it through all your automated builds and then ship it. And then on top of that if you have some people in here are making products that you ship for six months and then it's never to be seen again and that minor stuff is perfectly okay for that but my team we ship stuff we have to support for five years. So the long-term support for any Yachto release is shorter than five years. So we have to plan for a Yachto update in that lifetime of that system at some point. So we can combine these two for doing major Yachto updates on shipping products. So use dot minor to do the small updates and then you can kind of figure it out yourself. Do you want to do every Yachto or do you want to skip a couple versions? It's keeping in mind the cost benefits of those but now that you have those two concepts you can combine them and figure out the best mix for yourself. Well there's three, there's so on any branch so we've now done a release branch. So we have, and going back to my couple slides, rel 14G 1.0 master now. So in that 1.0 master now there will be a pokey and a pokey minor if it's a short thing or a pokey, pokey minor and pokey next that is specific for that branch that we can rotate through. And the rel 14G master has its own pokey copies. Do I have any special way of dealing with the pen files? So my main one that I came up with really early on is never, ever, ever check in the pokey or meta OE and we do everything by via BB pen. And luckily 99% of the stuff we can do we put in our own meta drag directory and use BB pens for it and it works just fine. So since we put, I put that policy of not checking in the pokey in place at the beginning of this in 2012 after having a big blow up because somebody checked in without my knowledge we did update a pokey that lost their changes, reverted a bunch of stuff, big mess. So now that we have this system in place it's conceivable that you could check in to pokey and have a, and rebase those changes on top of newer versions. It can get a little complex though and unless you have a good expert, I don't recommend it. But the main thing is in our meta drag we have our BB pen files. All the open source BB pens are in one spot. So we have a meta drag recipes OSS and then a directory for each one. So it's in one spot where we can find them. So if you're, so that's the other problem about Yachto is you have all these layers finding out who modified what where it can be a little bit challenging. So on our side we do our part to keep it all centralized. So if you have a BB pen for a Yachto or meta OE it's all under one directory or run directory structure not everything in one directory. So branching and the overview for branching for minor it's the same thing as for the next except it's more frequent. So in this case I just cut the slide out where we combine the whole thing but the pokey upstream is at top and here we do an as needed merge down in the dot minor pokey and then here you can do merges into your main shipping pokey directory several times during a release on an as needed basis. So obviously if there's no changes going into your minor pokey then there's no need for you to do a bunch of work there. So this is more of an as needed basis for your shipping products. And basically if you probably can't read it sorry the flow chart is pull changes from upstream pokey integrate into minor and build are there build issues fix them if there's our do your build verification test is it does it past if not fix them and then once you've got all that stuff you push it to your main line pokey. This whole process it takes like a day. So if you have the aforementioned really good college hire they can do this kind of thing in a day for one update a minor update minor updates are usually no problem. So that's the end of my slides have 15 minutes left and I have a quick very basic demo nothing elaborate that I wanted to walk through you guys walk through with you guys. Does anybody have any questions on this before I start to do a demo? Yeah so the question was you have to bite the bullet to get from your 1.3 Yachto up to Morty before you can do this and the answer that is yes you can put this branching strategy in place so you have your existing code base all working with your existing one dot old version of Yachto and then you can pull it pokey next and attempt to get it working. I myself did a our 12 12 G and 13 G where servers were running Yachto 1.3 and then when we started 14 G development it was 1.6 and actually did a 1.5 prototype before that so I went from 1.3 to 1.6 they things were deprecated in BitBake and you had to go do a bunch of stuff it was a big P.I.T.A. So yes you do have to figure out that first step once you've done that first one then it becomes much easier yes so like as you're just repeating what you said bits and pieces are easier than that big hit so the first part is just getting your organization your manager is your leads to sign up on this concept that we're gonna keep ourself up to date and put whatever rigor morale or processes in place that first to make sure everybody's comfortable we had a whole process with sign offs I took a couple of those slides from the initial deck we put together two years ago to get this sold and we had a whole process flow and sign offs from teams and stuff like that be accommodating however accommodating except you you're already up to date but everybody else be as accommodating as you can to get that initial update but once you've got that initial update subsequent updates become easier and easier to do so we had another question yeah I had another proposal for a talk of how we'd structure all of that but that wasn't accepted this one was but the basic, oh so the question was do we do anything cash any of those downloads et cetera so we use bb underscore node network or bb network equals zero I forget what the exact variable is but you set this bit big variable to say don't talk to the network at all under any circumstances and that's set for all of our builds and we have a separate git repository called sources that we place all of the tar balls that we've downloaded that are needed for that build in that git repository so there's a archive format for all the upstream git repositories that you can do it takes a couple of times to do it to forget how to tar it up properly and name it but once you've done that it's straightforward you everything goes into that repository it's a couple gigabytes of repository so we don't keep well we keep version history but we rotate that repository so we have for every release we have a source repository and get a new one for every release so that because otherwise the repository history just explodes but that's basically how we do it and then we have one or two developers who have proxies set up to go and update that as necessary yeah so two answers to that so at an overall level yes there's a bb underscore no underscore network equals one you set that in your local.conf and it completely disables and then there's also another one called yeah you can also say bb premiere only I think is the other variable where it will look for you can have it set up a local mirror to get stuff from you did say something that triggered me that's just one thing I forget exactly how you worded it but how you set up your recipe matters you need to have a if you have any recipes that point to an external get repository you have to have a source rev and the actual hash of the rev in there if you have it as a branch it's going to try to check that every single time it builds so in your recipe for anything that's pointing at a get repository as the source make sure you have source rev and all of the pokey in meadow we all have that uh... it's just if you have your own recipes uh... that point to random stuff make sure you have a source rev or it it won't work it'll break yes so the quest so the question is what is the time period it takes for this to happen my examples one day for the minors uh... I just have to emphasize the smaller updates you do the faster it is if you the longer you wait between them it gets a little bit inconsistent because you know if there was a gcc upgrade that went into upstream in that time period you might be looking at a lot more work however the the system itself works if you have the branches set up uh... if you run out of developer time or in our case while our developer got pulled off to do other things uh... it'll sit there and it won't be waiting for you to come back and do it right uh... so this is gonna scale with the size of your team i have a pretty large team and we can have uh... one person dedicate you know basically a couple of days uh... every three weeks to do this on a smaller team i can see how that it might be a little bit harder uh... but you know at that point he becomes more of a team effort hey it's your turn to do it this time in the rotated around again this is a really great way to become proficient at yachto fast uh... so rotating this responsibility around a small team would probably be a great way of doing that uh... our build host okay so what is our build environment run on so uh... i'm the one that set up all of our initial builds all our initial Jenkins jobs and all that stuff but we actually have a dedicated team that that i turned all that over eventually they're gonna remember figure out that i still have admin access and take that away but uh... so we have uh... we are a computer company and we have the biggest biggest piece of iron that you can imagine on that we have like a a machine of like a hundred forty four cores and uh... some a couple dozen build workers in jenkins uh... and they also do some this is an aside but vm's don't work uh... we had a whole development environment based on developers having vm's that they did their work in it complete and utter crap do not do not use vm's for this use real hardware uh... qualified assignment uh... yeah we own okay so it's slow it was no matter how much resources you gave a vm for a developer to do their build uh... it was ten uh... it took one-tenth the time to do a build on a real hardware then it did on our best vm so i'm a developer just running bit-bake a no-op bit-bake command it took ten times as long and so on my real hardware i could do a build it uh... so we actually have some shared infrastructure now for real hardware developers have logins to a couple of beast machines and we do the builds that take twenty minutes on that machine and on our or vm it was taken like three hours to do it from scratch no cash build sorry i'm so okay i'll qualify cement some people can have uh... success of the m's we had some really smart people dedicated to doing vm's and they could never get it to be as as good as hardware and i saw a lot of head shaking here so seems to be a sense of it so if you want to know how to make vm's run really well i'll go talk to that guy uh... so okay sorry might have run out of time for my demo i have five minutes to do this demo and it's about five minute demo okay so uh... oh my gosh that's not going to work uh... fix the font size i guess okay what's the increased font size khaki command plus thank you not great but usually does my next box sorry uh... c okay resets and work that i okay so just to give an idea uh... setting up these mirrors just a quick and sorry i didn't ask can everybody see this relatively okay we have a mirror internal but for conference purposes because wife i sucks i have my own little uh... copy of pokey uh... so i'm i'm basically gonna set up a tree where i have a uh... a pokey and a pokey next i'm gonna call them pokey court crogoth and pokey morty so have a upstream copy here and we have a a very positive for you here and another very positive and then i'm creating a uh... check out of the pokey and then i'm gonna push uh... what this is the i think yes the question i'm pushing the crogoth branch in the upstream pokey into ralph fourteen g master of my very positive three so my bear my very positive three now has just that one branch and then do the same thing for with morty i call it pokey pokey next the group back in repositories i just named them after the release so pokey crogoth pokey morty uh... so we're gonna just switch the pointer of which repository we point at because it's because between the branches and pokey you can't pull between them so that's that uh... so i have a repository a repo i'll show you the manifest in a second but have a manifest that just checks out those two pokey repositories and i'll show you what that looks like after finishes i'm checking out so this is my manifest uh... here we have revision uh... we're checking out ralph fourteen g master and pokey crogoth and pokey morty so this get checked out into the pokey directory in the dot next pokey respectively and i have one minute so you take me at work a couple i can see the into this repository and you can see here i have dot next directory that'll have pokey and pokey in it i could do get get logged in both those directors and show you that they're uh... the different branches but i'm not gonna do that for time miss which the manifest to now let's say we got everything that done the way we want it to and now in the manifest on the switch pokey over to morty and then i'm gonna check that in and so now if i go back into my repo and i do a repo sync you'll see that switching it's switching my pokey crogoth over to pokey morty so you'll have to use your imagination for the rest of this but it's all fairly straightforward and said works for us uh... i think that's the end of my time uh... thank you guys