 Today we're going to spend half an hour or so talking about Bosch package rendering and collocated errands, which are two fairly recent Bosch features that will allow you to build simpler releases and achieve faster upgrades, which is quite exciting. This is not working. Cool. All right, quick intros. My name is Maria. I am a software engineer for Pivotal. I work off the London office and I'm currently on the team that offers contributions to open source Kubernetes. And I'm Maya Rose Grants. I'm on the PCF Redis Pivotal team and we managed something like 15 Bosch releases at the moment. Okay, so a quick look at what we're going to talk about in the next 30 minutes or so. I'm going to be honest, the title gives away quite a bit, but we will start with a very high-level view of Bosch. And the intention of that is not to cover everything that Bosch does in a very comprehensive way, but rather give us a little bit of context and build a little bit of a foundation to help us frame the second part of the talk. We will focus then on package rendering, which is a feature that Bosch added over the last year or so. And it allows additional flexibility and abstraction when it comes to specifying dependencies within the context of Bosch releases. And after that, we will talk about errant collocation that offers additional flexibility on where one of tasks that run on Bosch deployed instances also refer to as errant's run. Okay. So Bosch is talked about a lot at dinner parties and the such, and you might have heard of it as a tool chain, a package manager, a framework, a deployment manager, an orchestrator, a release engineering tool, among many other names. But it's so powerful because it's really all of the above and it offers concepts and tools in all of these areas. And that's why time and again, Bosch has been the go-to tool for operators to take software from source code that a development team, authors, and maintains through a series of steps represented very accurately by three dots there into a long-running and evolving system. Exactly how does it do that? What are those three dots? The first step is by offering a framework for software packaging. What comes under that? Packaging includes, first of all, instructions on how to prepare the software, a description on a description of dependencies of what other libraries or binaries or what other software our software depends on. And there are a few Bosch concepts that come into play here that are quite interesting. The first one is that of a Bosch release. And that is a flexible structure that both source code but also pre-compiled binaries can fit in and can be expressed by. Within releases themselves, code is organized into packages that contain the specification but also any supporting scripts for the software to be prepared on the instance that it will run on. And that means that it's copied in the right places, that it's properly compiled, and so on. All right, so after software packaging, second step is orchestration. And that again is a huge umbrella of things that happen but really in short, orchestration is the business of organizing packaged software into a running topology and arranging it into components, coordinating those components, taking on the deployment of these components and more specifically things like provisioning, things like interacting with the underlying infrastructure layer. And finally, addressing configuration management. So ensuring that the software that ends up on instances that Bosch has deployed is correctly configured. Specifically, how does it do it? You see towards the bottom of the list, we have some interesting Bosch concepts that come into play here. The main one is that of Bosch deployments which create a bridge from references to packaged software and a deployment on the other side. And it creates that bridge through Bosch manifests. When running on the instances themselves, the next step, software is represented as Bosch jobs. And then there is a mapping between Bosch jobs and a set of processes on the instance itself. Typically, one job is one process but it's not necessarily so. Even at that level, there's additional flexibility. So jobs can be long-running and in this case they're referred to as service jobs or simply jobs. Or operators can decide that a task has a very specific task, a very specific functionality to complete and is short-lived and in that case, it's designated as an errand. Finally, once a Bosch deployment has been orchestrated and has been created, Bosch is in the business of keeping it alive, keeping it healthy, managing its evolution and offers a lot of tooling in that area. What that means in a bit more detail are things like upgrades, for example, and that refers to keeping the underlying operating system up-to-date with things like Bosch stem cells which serve as an abstraction between the software that runs on the VM and the supporting operating system underneath so that the two can move independently. Things like updates come into play here as well and those refer to the versions of the software that run on the VM themselves and Bosch manages that change as well. Topology changes, any additions of instances, removals of instances are under the responsibility of Bosch too. And finally, any health monitoring and lifecycle management of instances with things like Bosch agent that is responsible for keeping a specific instance alive and healthy, things like monit on the instances to ensure health of the very specific processes and concepts like canaries and deployments where Bosch will choose to essentially try out a change on a subset of a deployment before extending it out to the whole thing. So this was a very high-level view of all the stages in the software lifecycle that Bosch offers tooling and gets involved in. Later on we're going to focus in two, so we're going to talk about vendor packages that come into play in the software packaging stage and then collocated errands that are most active during orchestration and the responsibilities of Bosch over a long-running system. And with that, Maya will take you over package vendering. All right, so we'll start with this diagram. And this diagram maps very, very closely to that process that Maya just walked you through. So you have your source code or maybe you're just wrapping a binary and your source code would also have its language dependencies or any kind of like test binaries and you have your package that kind of wraps it and that gets provided to a job which handles the orchestration of it. It will do the start, the stop and handle kind of the lifecycle of it and that goes to a Bosch release which was actually deployed on VMs. The other thing that's kind of worth maybe calling out is that it's fully atomic, it has all the bits it needs to run. Potentially you don't even need the internet connection at this point. If you look at the package itself, when you make a package with, you know, Bosch-Iron package, you end up with these two files. You have a packaging file which will copy the actual bits into the correct portion on the Bosch VM and then you have a spec file which will specify any compile time dependencies and the files that actually make up the package. So kind of the problem that package vending is trying to solve is that when you're making many releases, you kind of end up seeing the same dependencies over and over and over again. Things like programming language. And the problem that can arise is you can have all these inconsistencies. Like if you have two releases, you might call both your language pencies go, but in one, it's go18 and one, it's go111 and when you try to co-locate them, you're going to get naming conflicts. And hopefully you have right access and you can rename one of them. Another problem is that if you are specifying your own configuration, you might have to be specifying goPath or goRoot and you might be doing that differently in different releases. So if you're making updates, you hopefully will do it right in all of them. And it's not really transparent what versions you're using. All right, so Bosch vending packages are trying to address that. So they kind of have rolled out five initial languages or four initial languages in Nginx. They're in this Git repo, Git org, and then CFC.ly actually lives in there as well, but it's not a vended package and we'll kind of go into why later on. One thing to also note is that there is now pipeline. So any new versions to go or any of these languages will automatically be there pretty much the day that they're released. New majors tend to take a day or two to go in, but miners, I mean, sorry, patches make it in pretty instantly. So how to actually use this tool, you copy down whatever Bosch package you want, go into your release and run Bosch vendor package. What it actually will do is it will make a spec.walk file rather than the spec and packaging file that was there before. And that pretty much just references a fingerprint of what Bosch vendor package version you're using. What that kind of looks like in more practical terms is you kind of have this link. So rather than the error going straight from, you know, your Bosch release to kind of the third-party blob or wherever that lives in the cloud, it goes via this invenderable goling. Many of these languages also provide these two files called compile and run time bump, and they specify the configuration. Most of the time they're pretty identical between the two of them because you kind of need go root and go path or Java home or Python path in kind of any use case you have for these languages. So the compile end will typically call it from kind of like your other packages when you're compiling and your run time might be called from your jobs when you're executing code. In terms of the benefits of using these rather than the kind of more roll your own approach, they kind of have cross OS support by default. So goling rolls with Darwin, Linux and Windows. It standardizes package interactions. So now you just source this compile end or run time bump file and you don't need to set this yourself. And it simplifies package patching. So if, you know, a new goling one 10, four, one 11, one, I think just came out a week ago, you can just pull down the, re-pull down the Bosch vendor package, iterate through your releases and then run Bosch vendor package and it will automatically update. That said, you do need to cut new release versions because packages are fundamental units of your Bosch release. You have to recut. In terms of requirements, they're not, it's not the newest version of Bosch, but you do have to be within the most recent year. And if you're using the Bosch vendor packages that are provided by Bosch, you have to have an internet connection because the blobs themselves aren't a public S3. One thing to make a touch on the second is when you actually like do a Bosch might create my release with a vendor package, you end up with an identical to release to what you would have had in the old world. So it's fully atomic tassel to code. You don't need an internet connection anymore. This is kind of the part that's really cool. If you're making your own Bosch release, you actually have all the tools you need to kind of make your own vendor package as well. There's really nothing special about this file. You make your own release. You create a package of whatever it is you want to make a vendorable. If you have configuration, you can make your own compiler and the runtime them. But if it's just a binary, you don't even need that. Upload it to a blob store and you can password protect that if you want. And if you're willing to give that password to your end users, cut off on a release and then you can just use it. So if you're interested, I have a Ginko package where I have instructions that I used to actually create one. In terms of why not to use them, the existing manual package creation flow makes a lot more sense if it's your source code, right? If it's not reusable, then don't make it reusable. Also, if you need fine control for patch versions. So the way the existing language ones go, it floats on a miner, but not the patch. So if you actually want specifically go 110.0 and you don't want to float on it, then this tool hasn't been built yet for that. That's why I say it couldn't be. You can easily have go 110.0, 110.1 float on the major end float on the miner, but it's only float on the miner at the moment. Another thing is maybe consider if co-located jobs are the right tool. If it's just a runtime dependency and cutting new releases is very painful, then maybe that makes more sense. So for CFC-LI's use case, they started as a vulnerable package and decided that actually doesn't make sense, because most people are using it in their Bosch start scripts. So they need to talk to the cloud controller before they actually execute their code or whatever that is. In which case, if you want to make a new update to CFC-LI, maybe you don't want to cut a new release. Maybe all you want to do is do Bosch upload release CFC-LI and then redeploy your Bosch deployment. So from here, I'm going to hand it back to Maria. We're going to talk about a specific strain of co-located jobs, and that's co-located errands. Cool. Thanks. OK, so a quick step back to talk about errands super quickly. They're nothing but a special category of a Bosch job really, in that they are started to perform a task, then just get themselves turned on once that task is completed, as opposed to service jobs or just jobs usually, which are long-running and are expected to have uptime and be healthy. OK, so so far, pre-co-located errands world, this is what a deployment with an errand would look like. Imagine that this is a very simple deployment that just has a long-running job called app, and it also has an errand. At deployment time, Bosch would create the VM for the long-running instance and would run the process there. Now, when the moment comes for the errand job to run, the operator would invoke it, typically by running something like Bosch run errand tests, and then Bosch would go ahead and create a brand new instance, which means VM for most of the cloud providers. It would run the script there, and it would then tear down that VM once the task was completed. Now, there are advantages and disadvantages with that approach. The main benefit is that it implements a design of single responsibility. In that software, with this design, will live on its separate VM unless it has a good reason to coexist with other software. And that gives us isolation and, by extension, some level of security. On the flip side, it's very common for errands to take typically a few seconds to complete what it is that they were created to do. But on the contrast, creating a VM takes up to five minutes, depending on what provider you're using, and is also subject to minimum costs at the infrastructure layer. And that means that it costs money and time disproportionate to the amount of time that the errand itself needs to complete its work. And then additionally to that, when we're looking at running errands periodically, or when we're looking at running errands for huge deployments, such as, imagine, like a multi-service cloud foundry foundation that gets upgraded with a collaboration of many releases, this can end up adding hours to the process. And that is exactly the problem, or one of the problems, that errand colocation comes in to solve. And the main idea is that the errand job is placed on the same instance group as a long-running job. And that way, Bosch does not need to create a new VM to run the errand script from there. But instead, when invoked, it simply executes the script that it has previously collocated in an existing long-running VM. There are a couple of obvious instant benefits to this approach, and they're mostly related to time and cost savings. So first of all, there is no need to spin up a separate VM. It means that deployments go way faster than before. Secondly, the VM footprint is smaller, and that means radically lower costs. But it's also an elegant solution for a scenario where an errands responsibility is to reach out to a number of instances, or all instances in an instance group, for example, and run the exact same thing on all of them. So previously, this would look something like that. You would have an errand VM that would be sort of like central, and it would need to be configured with all of the destinations that it needs to reach out to and perform checks on them. Some examples that releases are doing are periodic smoke tests or specialized health checks on top of the Bosch-provided ones. With collocated errands, that looks a little bit more like that. So authors and operators now can move this functionality onto the VMs themselves, and instances are now empowered to perform self-checks, as opposed to having a central job that needs to be configured with what is a list of places to check, and so on and so forth. And it also could mean that you have to do less work on configuring the network and the accessibility between VMs. Another benefit from this one is that previously the errand job would need to somehow be updated through the manifest to know of any changes in the topology of the instances that it needs to reach out to, which is also something, which is also a piece of work that you don't need to do anymore. There's always a flip side, and there are scenarios where collocated errands might not make sense. Perhaps the biggest point to think about is isolation, because as far as Bosch is concerned, the errand is not a special job, it's just another process on that same instance. So at the very minimum it's definitely worth taking a step back and considering the risk of collocating an errand process with a long running and perhaps business critical piece of software. And that includes things like what permissions you would like to set on the data that lives on the disk of that VM and so on and so forth. The second point talks about resource allocation. Again, the errand job is not a special process, and in that there are no specific limits placed on it in terms of how much of the resources it can use. So if we're not careful, we can end up with errands eating up all of the resources on a VM, not leaving space for the other processes to run. Additionally, when Bosch is asked to collocate errands, it's going to place a copy of the errand on every instance of the instance group that we mentioned in the manifest that the errand is collocated on, if that makes sense. This could be a requirement, specifically on the use case that we saw just before with self checks. But if not, one question to ask is, is it appropriate if the errand runs multiple times? Is the functionality idempotent and is it going to be fine if it runs multiple times rather than one? And if not, it's worth considering the more granular UX that was introduced with collocated errands and that we're about to cover in a moment. Okay, a couple of requirements to keep in mind if you do decide that collocated errands are a good solution for your use case. You will need a version of Bosch's director of 263 and above and that's to support collocation at the director level. Linux Temsals need a version of 3, 4, 4, 5 and up and that is so that you can have a Bosch agent that understands collocated errands jobs at the instance level. Finally, if you want to take advantage of the more granular UX that came in support of collocated errands, you would need a Bosch CLI of 2031 or up. All of these were released in August 2017 so you might already be on them. Okay, in terms of how we would go about implementing them, what needs to change, the good news is that not very much. Release authors really don't see much of a change. They still create a job in a release that still has a bin slash run script just like old errands did. A little bit of a change on the manifest author side of things and that's the human operator or the system that composes the manifest for a Bosch deployment. The change that has to happen there, the change that has to happen there is that the errand job needs to be moved from its own instance group as it's own on the left to a pre-existing instance group or a separate preferably pre-existing instance group as it's shown on the right. It could be interesting to mention a couple of approaches that we've seen in releases out in the wild. The first one is a kind of a hybrid flavor where some errands will be collocated with other jobs and some will choose to live in their own VM. There are a couple of reasons why this might be appropriate. Again, it could be just a mix of use cases. Some of the jobs, some of the errands were from tasks that are applicable to more than one instance. Again, health checks is a very easy example that comes to mind, but others might be applicable to the whole deployment. For example, registration of a service broker. The second pattern that we've seen is using collocated errands in combination with some process isolation component. An example is BPM, Bosch Process Manager. And the interesting functionality that it adds on top is that it namespaces processes, including errands, and that allows for an additional layer of isolation and could reassure operators that have a particular focus on processes being isolated. The last use case, which is quite interesting, is the more granular control that comes with operating collocated errands and specifically on errand invocation. Let's have a little bit of a more detailed look into that. Invocation for collocated errands works just like before, mostly. And what I mean by that is, yes, in the simplest case, you could run the exact same command. You could say, Bosch run errands tests, but now the difference is that Bosch will trigger the errand job on all of the instances of the instance group where it's been collocated. In contrast to previously where it would trigger the one instance group. You might be interested to run only copies of the errand on a specific instance group, and you can do that by passing in the Duster's instance flag and mentioning the instance group name. You can also choose to run the copy of the errand on one specific instance of one specific instance group, and you can specify that with instance group slash first. If you care about which specific instance you want the errand to run on, you can simply pass the UID of that instance when calling Duster's instance. And you can extend that previous pattern by passing a list of instances in an instance group or multiple instance groups, and that gives you complete flexibility in where the errands will run. Okay, just a very quick recap of the things that we went through. We looked at a couple of features added to Bosch over the last year or so. The first one is package rendering that allows service author, sorry, release authors and operators to create more modular packages, and they're quite simple to make in that they are really under the cover Bosch releases. And secondly, error collocation is an idea and the Bosch implementation that allows for faster deployments and a smaller VM footprint. It's there to offer more runtime granularity and more choice to operators about how many times and where errand jobs are going to run. But it does come with trade-offs, the more important one being isolation for speeds. With that, I think we want to thank you very much for coming along and enjoy the rest of the conference.