 Well, ladies and gentlemen welcome to Bosch splitter see this is the part where I had a line in here There was gonna be applause and raving Yeah confetti. No not confetti. That's that's not very green Now I'm not dr. Nick. I don't have my own theme music when I walk in so Actually, we should switch sides because I'm Chris Weibel work at the engineering team lead over at Stark and Wayne Biology Nagarajan director of software engineering at GE digital working on predicts What is Bosch splitter? I'm sure you guys are waiting to know that so Everybody is familiar with Bosch or at least some time somewhat familiar with Bosch at GE we have Many Cloud Foundry deployments in production. We have quite a handful of them and as our journey with Cloud Foundry grows What we found out that managing a single large deployment of Cloud Foundry is no joke Especially when we throw in the practices and policies that we have at GE and which is very typical with any large company that you have Where you have rigorous change control and change windows Doing a change a single change through Bosch starts taking hours days and probably sometimes a week and Given that particular problem what we we were facing with was if we started a stem cell change on a Friday evening Because I'm someone decided that changes are good on Friday evenings But I don't know who these people are but a change start that starts on Friday evening typically takes it all the way to a Monday morning and Anybody who's done production rollouts know that When you start a change like that, it's a it's a it's a no starter for one And given how Bosch works It is so hard for us to stop that change to do if there's a production outage in between if you want to fix something else in between It is such a hard thing to do which was our Reason behind going down this path of Bosch splitter the use case essentially presents itself as where given our requirements that we have we wanted to Write a tool around Bosch that given how we build our manifest given how we build our own templates and Given how a repo is managed to get and concourse without doing too much of plumbing We wanted to break down a single manifest into a more manageable Deployment so that every deployment can be Started on Independently of the other one and we could start and stop each deployment at a meaningful or meaningful pace This gives us the flexibility of doing Some sort of an AB deployment or canaries and we can have stem cell changes where you can leave a particular Runners in one stem cell while others are in our oldest stem cell as you proper get a change through meaningful working hours as well And I just want to reiterate something the biology said Some of us actually wind up doing the production deploys And being able to break those up into eight-hour windows so that you can just Without having to hit control C on Bosch and stop your deploy at a nice happy place That makes everybody happy so then you can just keep going on Monday morning Makes your change control board much happier that you're able to control this little beastie For your stem cell upgrades or AB testing or whatever it is that you want to do so and this actually came out of one of one time when we were actually doing a change it was a Stem cell rollout. We were going through a deployment of close to 600 VMs There was a production outage where we actually had to go scale one of the Cell runner groups. You could not do anything other than Control C the deployment reward do some major surgery to reward the changes and then just for adding a bunch of VMs in one Runner group. You got to go do that. That was one of the genesis of why we went down this particular so This is a late add to the slide deck here that doesn't even have a slide Biology and I both have children so we wind up with random things that are inserted into our book bags. Thank you This is everybody else cloud foundry deployment This is our cloud foundry deployment. It's much bigger. It requires a lot more to keep it afloat That's where tools like Bosch blitter and and some of the other things that we won't be showing you today But feel free feel free to stop by and ask us questions on how to maintain big duck versus little duck And I think our children for for giving us these So how does it work? Bosch blitter Couple bits and pieces that we need to understand before we can really dig into this One for Bosch blitter our goal was to make sure that we could continue to use the templating That everybody is used to whether it's the templating inside of CF release or its genesis or whatever your templating tool happens to be We don't want you to change any of that What it is is a post process that will take your big Deployment manifest and carve it up into smaller deployment manifest that are easier to manage for whatever your reason is So we have a make file that's going to help us do that Where I'm going to specify what the jobs are that I want to split out of the main big deployment manifest And I'm going to wind up with something that's called core and something that's called split So before you see core on all the rest of the slides What core is is your original deployment manifest with the one job in it or multiple jobs in it that you would like Split out into its own separate deployment manifest, but I still want to use all the templates that I already have What the splits are is that one job or more jobs that you wanted to split out from your main deployment I wind up with two deployment manifest and then I deploy them just like I always had with any other deployment manifest that we have out There so we're not rewriting Bosch We're not rewriting any of your templates So our traditional Way of generating a CF deployment manifest Sure, everybody's done this before right? Okay? You're going to start out with a series of templates and CF release or CF deployment or CF whatever comes up with this week You're going to make your big giant deployment manifest You're going to deploy that deployment manifest in our example here. We have a very simple one We have three jobs x y and z three x instances three y instances and job C is this huge one with six instances let's pretend that Job Z is our a cell group or Group of runners and I would like to maintain those separate First stem cell upgrades or for a B testing or whatever it is that I want to do This is what the Lifecycle looks like using Bosch splitter again. We're going to use the same templates. We are always have We're going to generate our deployment manifest But at the end of that we're just going to run a little split Which will split it out into two deployment manifests the first one like I mentioned before would be Core.yaml and what that represents back from our example when once we deploy it is jobs x and jobs y the three instances still And then we were going to split out job Z So I'm going to have a separate deployment manifest for that exam that can't talk again sourced from the original deployment manifest I'm sorry the templates that we already had We'll deploy that and I get my six instances of job Z If you didn't follow that flow chart, let me try to explain this in a slightly different fashion What we have here on the left is the original deployment manifest that we had If I was not using Bosch splitter that would have been the deployment manifest I would have used so there was the three jobs in there with all the properties and networks and releases and all sorts of fun Stuff that's in a deployment manifest The next to represent. Hey, I split out job Z So the core has everything except for job Z in it There's a strike through there that just shows you these are the pieces that actually aren't in this deployment manifest The one on the far right represents the split job, which is just job Z in it Important thing to see here is is that the properties the network Sorry the properties the networks the resource pools, etc. Are all the same amongst all three of them We're not trying to reinvent how Bosch is going to deploy any of this So demo time So Before I actually start showing the individual pieces the demo Want to let you know that what we're using here is the templating inside of CF release as a version 260 That I really didn't do much else to it other than just created it using the existing templates that were there so this repo here takes that Bosch light manifest and I've got a very simple make file that I created This make file. There's two important bits of information in it. The first one is is Where is the yaml file that I created with my templates? So in this case, I've got a subfolder called manifest and inside of that because I lack creativity on naming anything We have a file called manifest yaml. That's my deployment manifest for CF The next line in this This is the list of jobs that I would like split out into their own deployment manifest So anything that isn't in this list is going to wind up in core yaml Anything that is in this list is going to get its own deployment manifest file generated for it that only has run or z1 And it or only has run or z2 in it from a from a jobs perspective So Now for the magic because that's what we're paid to do and that's why you are all here So we're gonna run make split My clear did not work Magic Thank you. All right. Let's see what that actually did So Adam because everybody has to have an editor and this is the one I'm using You'll notice that there are three additional files in here. There's our core file and our three split files With Adam Since the manifest yaml file is not green. It was not changed in any way So the main point of this is the original manifest yaml file that we created with our templating We're not making any changes to it whatsoever So if I needed to revert if I need to roll back something has gone wrong epically with Bosch splitter I've still got my original deployment manifest that I can use for Rolling back. So we're going to take a look at core.yaml here real quick What you'll find is while there are references to run or z1 in here because I've got resource pools and network definitions in here There is no job for runner z1 anymore in here If I look at one of the split jobs specifically for runner z1 The only thing that's in here is one job for runner z1. There are no other jobs in here. That's it So when we go through and we deploy these files Bosch light, please work Bosch light, please work Bosch light, please work got dark Core.yaml Deploy Yeah, nothing special here other than hitting yes We're going to go through and we're going to do going We're going to deploy the core file which is going to have all the cloud foundry components Minus runner z1 and runner z2. I cheated. I've already deployed this so We also show the same thing. I'll show you the split real quick Even though we I will cheat and tell you that it is a no op as well That's an operation would have been good If I do a Bosch deployments We have three deployments in here Bosch sees this as three distinct deployments CF warden, which is the name of my deployment manifest because again, I didn't change the names of anything inside of the The CF releases script That deployment is always called CF warden The runner z1 and runner z2 version of this were automatically named for me Inside of the split script if you want to come up with a different naming convention You're welcome to do so. This is just the one that's provided you by default So if I look at Bosch VMs, which this looked really big really awesome on a different screen I've got all of my cloud foundry deployment or all my cloud foundry VMs Plus one for runners e1. I've got my runner in here and runners e2. I've got two runners over here Let's show you how I have nothing up my sleeves And to prove that this really works We have an app already deployed. This is CFM. I don't know who wrote this dr. Nick if it was you It is an awesome tool that I use just about daily to debug things But to show you that you actually have a working cloud foundry deployment split across multiple deployments Inside of Bosch tada app works even though the runners aren't inside of the core cloud foundry deployment Pauses for applause Thank you Now I need to mess with this and switch back to that Demo time is over so the split script is Located on a cloud foundry community repo Before there's a mad dash to write this down It's hoping there was a mad dash to write this down The slide deck that you've seen here is actually uploaded to the CF summit site for this particular talk so you can pull it down later on At this point I'm going to open it up to any questions anybody might have Sir So what he's asking the slide to was our use case. Why were we doing this again? And I'm just going to give this to mr. biology So the biggest use case why we are doing this is typically our deployment is pretty huge We have like a 500 to 600 VM deployment and when you roll a stem cell across that if you say max in flight is for That takes three days The problem with that is if you have a if you have to pause it to do something else in between That's that's impossible. You got to control see it You got to go undo everything that you've done so far and then you got to do some major surgery to get back to Bosch to do whatever you want So what we've done in our deployments is to weave split our entire cloud foundry into multiple splits So you you have a we have a split of all of our core components into one manifest and we have runner per Availability zone into different For example today, we often when we're talking about splitting gnats out of into own split So that rather touch gnats if you really don't need to touch gnats That's the biggest use case that we have for this particular tool and the other thing is You don't really need to run a change for that long if you if you are for example We have maintenance windows where they say start your change and stop it during normal business hours So there aren't a base your weekends. This is what we do for that So what I pulled up on the screen here is actually our concourse deployment that we have That we've actually got the individual Diego cells that are split out into their own Concourse jobs here. So there's if I were to do a Bosch deployments against this I would actually see five deployments that are associated with this particular Cloud Foundry what this allows us to do is if I've got a change window that That looks much better on my screen. Sorry If I've got a change window that only allows me to do deployments for eight hours for a stem cell upgrade The way that we have these scaled out is I know I have Enough cells that are in each one of these groups that it'll be about eight hours for me to upgrade each one of these so if I need More capacity out there. I'll actually create a new cell group just so that I can leverage the split tool that we have here So that I can do a Monday through Friday deploy from 9 a.m. to 5 p.m Instead of starting at Monday morning and then watching paint dry for 72 hours straight while I'm doing a stem cell upgrade We should win awards for watching paint dry so The only other part I had on this if there aren't any more questions There is a question and there is a question It's fun to watch Brian run Thank you so in case if there's a use case to do a unsplit for example, can you spruce it? Can you spruce the tools split manifest? Left asses so you don't need to unspruce the you when you ran when when he ran the split The original manifest left asses so what you get out of it is three split files, but you're split your file is still there I thought you had like the core and You're like two different. Yeah, so there is a core Flip over to this So inside of here I'm gonna leave the original manifest at yaml file alone, and I'm actually not using that as part of my deployment process So it's it's there as an artifact that I can reference back when I'm actually deploying is core.yaml in the two split files So the other thing is as funny you mentioned spruce spruce is actually the magic that glues all of this together So this is as much of a talk as hey, here's a cool little tool to do this as much as Biology had a problem. I've got spruce. I've got templates. I've got concourse. How do I solve that problem and Spruce is the thing that really makes all of this work without it. I'm dead in the water to be honest so Awesome sir, so For the most part if you're using cloud config not an issue. It's just gonna carve it up Under the the old Bosch one style deployments that was where I did cheat just a little bit Because the static IP is what I've actually done is I've created different network blocks that would be specifically associated with say cell The cells are a bad example because they're not statically IP but say we statically IP the cells I have a separate network block for cells e1 and cells e2 that I define and I carve up ahead of time So yes, there is a little bit cheating that we do if you use an old Bosch one Style manifest but and if you download this get hub repo, you'll see that's actually what I did is I've got the separate Network block for runners e1 and runners e2 Just because Bosch lights goofy with the networking Sir yes, sure So we're using this primarily we're using this in production right now for for cells and for for runners I haven't tried to take more complex things that are going to use the Bosch links like that and try to do it It's a great question. I wish I had a better answer for you to be honest thing between them. Yeah Hi long Brian's coming up behind you So we can just split just master deployment name and add more features that it's just something that they'd never used before right Yeah, so having on that for so if you know at the end of the day It That you're going to reference a deployment that's going to be split out You actually know what the name of that deployment is going to be at the end because you're controlling the algorithm on what your deployment is named So yes, that is how you you could account for that But it's not that complex of a tool truthfully. It's about five lines of bash long, but it's it's it's again because Spruce does all the heavy lifting for us that I basically just have to tell spruce to go do this thing for me Anyone else? Fire No, because I'm going to let Bosch do that for me What I am going to do though is if I want to control and say I can do 100 runners a day I am actually going to come up with multiple jobs They were not going to be called z1 or z2 or z3 anymore, but I am going to call it runner one I'm going to give it a hundred runner two is going to get a hundred I'll let Bosch figure out where it's actually going to stick those individual But the main point of this was was for the change control Of we have yeah, yeah, I Would like to visit my children my children do like me still so at the end I would like to cleanly break this at the end of the day and if I've got to do artificial things like Group a hundred runners at a time and a job called runner one. That's what I'm going to do so I can go home at night Anyone else? It's been fun. I didn't anticipate this many questions As far as what's next One probably thing that I can think of is when you go look at a Bosch deployments you see you used to see one now you see 20 We should probably think about how to make a logical representation of how each one is linked into the other one Yeah, same thing with the pipeline so how do you give a logical view of a job that's being split into one or more How do you link everything together to give you a logical representation of what's going on last little bit here? So actually when we did put this in a series of concourse pipelines One of the first things that we had to do after we were finished with that was the all view We couldn't do that anymore because unless you had a 4k 4k screen on something about that big You couldn't read any of the blocks anymore So there is when you when you had the individual concourse jobs. I can juggle here And so this is just one environment. I didn't get the top of it so we can see here that Core.yaml is the one on the left even doesn't even though it doesn't say core. That's a good point And the individual cell splits are on the right-hand side. I've got four groups of these now Plus my core I've now got five blocks that I got to go through in order to do a stem cell upgrade for a whole environment This is great. You get 20 environments that went from having 20 blocks total to now a hundred blocks Plus all the other stuff in between a little bit. You got to do for concourse But yeah, that's something we don't have an answer to yet, but we're looking for input on it Anyone else? Dr. Nick anything? Thank you very much everyone