 My name is Dr. Nick, it is time for us to chit-chat about some things. So, Bush releases. Actually, who's ever made their own Bush release? That's awesome. Who has ever thought, wow, I wonder if this could have been faster, because this took more than some mentally allocated amount of time I thought this should take. Cool. And obviously, how long we think something takes is affected by something else we've seen. If you've ever, from scratch, tried to figure out how to make Debian packages, I don't know how long that takes. By the way, if you ever do need to make Debian packages, there's a thing called FPM, and you should use that instead. But if you've used Docker, and you've had the benefit of just copying, pasting other people's Docker crap, and just into yours, it kind of works. And it somewhat sets an expectation of how fast everything else should be, and making Bush releases can be slower than that. So, let's talk about how we can at least make it faster than your previous experience. I do reference URLs to things, because, you know, material I'd like to talk about does exist on the internet. Therefore, you might want to know, do I need to write down all the URLs? No, you don't. The slides are on the internet. You can get them later on. All right. To start with. So, when I wrote this book, I called it a book, even though it's just on the internet, but you can, like, get fake book covers, and then you take it to a website, and they put it on fake pictures, so it looks like it's a real book. And this I call her the devops girl, and she's loving my book, and I think she's fantastic. And when I wrote it, the way in which I wrote this book was, you know, it's very hard to think about how do you teach people Bosch. People talk about it's like there's a learning curve for Bosch. So, the core idea on this book is just to skip the curve and just start on the sweet downhill ride of enjoying it. So, the book starts with everything's already working, and you've been put in this new job of living with this thing. And so, we sort of talk about, you know, let's have a look what's running. Okay. There's ZooKeeper. There's the processes. There's Monit. There's Bosch. There's where the VMs came from, and you could have go out that way. And then, even though it says it's the ultimate guide to Bosch, it does seem to be missing, like the entire section on how do I make my own Bosch releases. And, yeah, sure, that's missing. I don't care. And so, this talk is somewhat in a fill-in for that missing entire section of how, you know, how I would write a Bosch release. Not how you would do them. Because, I mean, I wrote my own tool six years ago called Bosch Gen. It helps me. That's not how the core team or anyone else than the kind of foundry group seems to write theirs. So, you know, my approach to it is different from theirs, but you're all here to hear my approach. So, screw them. So, 10 ways. Obviously, as you can imagine, I made up 10 when I was coming up with the talk title. It was accepted. And I thought, holy crap. I wonder if there are actually 10 ways. 7, 9, 10, 15. Any of these numbers are the ones I could have picked. Turns out there are 10 really good with things. So, Docker containers. And then we're going to look at what is the basic empty Bosch release. Because, let's be honest, nothing's faster than a release that does nothing. Not practical, but fast. We'll look at the Bosch Gen tool I referred to. The thing called create.yaml or how to do version create. It's a faster way to iterate. An approach to building Bosch releases by just worrying about packages first and not worrying about jobs. And you might think, well, how do I compile a package? I have to do a deploy. We'll get to it. To hold your horses. Geez. Well, I call language packs, but other people call Bosch packages. Which, as was brought up by anyone at the keynote today, the winner of the hackathon prize is doing a central thing based around Bosch packages, language packs. The fact that you can actually just use Davian packages. And if you've already got them, nothing faster than using something you've already got. And I'm going to argue perfectly valid to use them. And we'll go in what that looks like. Moving on to writing the jobs. If you haven't already started using BPM, I highly encourage it both because it's the future and because it makes it faster. And looking at how you're actually going to get your Bosch itself and how to do faster iteration if you're not using a variation of Bosch light and how to get that. And then perhaps less about being faster for you, but being faster for everyone else but you, which is sharing pre-compiled releases. All right. Let's get into it. So nothing's faster about Bosch than not actually writing your own release. Like, an empty release is slower than not writing one at all. And many people are already publishing Docker images. And if you've already got one, and perhaps, and you can argue, okay, why not Kubernetes? Like, okay, why not Kubernetes? If you've got Kubernetes, use that. I don't need to argue. But if you want to just bring up VMs and you want to manage processes, but you've already got Docker images, you can just use like a Docker Bosch release and run things. And you're winning. Like, I mean, what are we doing? We're running processes. We're exposing ports. We're storing things to disk. And we're talking other machines. That's it. That's our profession. And there's security in there somewhere, I think. Backups and restores, you know, that sort of stuff. So don't take it all so seriously. And if you want to get something up and running, just throw those Docker images as running processes and that you use Bosch for the VM lifecycle and the disk lifecycle. That's it. You know, what does nothing else do, like Bosch does, manage the VM and disk lifecycle? Cool. Let's do that. You can argue that the future of everything we do might just be Bosch runs Kubernetes, which runs everything else. That might be the future as we look back. So anyway, this is an option. So there is, and again, if I made a little fake, it's not a fake Bosch release. There's a little Bosch release that I built as I went along making these slides. You can look at this. One of the YAML files, there's two, literally, it just runs a job with the Docker Bosch release on it. There's a second job, a second sort of job in that VM which says what Docker container to run. So two jobs, one runs Bosch, one runs Dockerdame and the other one tells it what to download in the store. And it works. So the gist of it, obviously it's always difficult to show Bosch YAML, so there's bits missing here for the sake of making a bigger font. We want bigger fonts. We're all getting old. I like big fonts, but there are bits missing. The gist of it is this. So we've got two jobs running on the VM, the Dockerdame and all the configuration around what Docker is looking like and then what containers to run. And there whilst you could use the upstream Docker, you could write your own Docker image, Docker Bosch release if you want. By the way, that's certainly an approach. One I've used. All right, step two, what is an empty Bosch release? I mean, this really is more of a function of what makes Bosch create release not fail. It used to require more things like it used to have to have an empty jobs folder and empty packages folder. Now it doesn't complain about those things. But really, this is the gist of it. It's a final.yaml with a name in it and an empty blobs and then if you run create release, it doesn't fail. You could upload it. You could include it in your browser. Don't do anything. You literally can't really include it because you can't add to any instance groups because there's no jobs. But you can have jobs. Just to take this iteration, it's not the slides. You can have jobs that don't use packages. You can have packages that aren't referenced by jobs. They won't be included in anything. They won't get compiled because there's nothing linking to them. But this idea of an empty job is an interesting idea that we'll get to when we can talk about creating packages first. The idea of an empty monop file that Jeff references packages. All right, so an empty release. Kind of silly, but it gives us a starting point. The tool I created in 2012 when I first saw Bosch was Bosch Gen. Bosch CLI's always had a generator and I've always hated it. So that's my relationship with Bosch. I like my generator a lot better. So one of the things it does in addition to creating a scaffold is it also tries to set up your S3 bucket and make it public so you're ready to share. That's not on the slide, but it does that. What it does generate works. So the moment you generate that project, you can deploy it. It's kind of like a pseudo-empty project. There's a job, no packages. The job does nothing. But the manifest includes that job and will deploy it. So the moment you've got it, you go into the project. You can go straight into creating the release. You see that it'll have a dummy job in there with no packages. And then step four we're now onto is deploying. But at this point, yeah, sorry, this is missing. You could go on to upload and then Bosch deploy that manifest. So where it says about midway it says manifest.cf.summit.yaml. That is the working manifest that it generates for you. Now a core idea about the middle of the beginning of last year when sort of the Bosch 2 bits and pieces were starting to come together and we got the new Bosch deployment project. We got the new Bosch CLI. Dimitri had this idea of what deployment projects should look like. Either you have a separate project like CF deployment, which collects a lot of releases, or you have a manifest folder for that one release. But the idea is that every project should have one base manifest that works. Ideally with no variables, like no required variables, internally generated variables for certificates, but the manifest should just work. The CF deployment project is an incredible tribute to that because it has one variable that's required. It generates something like 80 certificates and internal passwords and has one which is the system domain. So if CF deployment can do this, you can. What is a good production deployment of a thing that works? All the operator files are just curating it, making it smaller, making it less secure, or making it more secure perhaps. I'd like to see that the base manifest is the best possible manifest and then the operator files are iterations of that. Right. A best of fault, right. Once we get to the end where I talk about pre-compiled, in that manifest you would put the exact references to upstream Bosch releases that are pre-compiled. So that someone comes along, all they need is that manifest and they just go Bosch deploy that and it works. Unfortunately, you can't include references to stem cells, so you still need to manually upload a stem cell. Hopefully you'll all be using Xenial stem cells for your projects, but other than that, your Bosch will automatically download all the releases. All right. This process of create, upload, deploy. So you've all done this where you do the create, you do the upload, you do the deploy. Turns out you no longer need to. So when you first do the Bosch gen, the base manifest that you get, that's kind of just for you. Like when you first tell other people about your project, it's not the same manifest because it has this mechanism and you're going to change that to reference final releases. So by the time you share, I'm going to say that a different way, by the time you first start sharing your Bosch release with other people, that base manifest, that base manifest, you will have modified to have all final releases in it. No latest nonsense. All right. I'll say that a different way. That manifest should not have version colon latest in it. That is bullshit. All right. That is all about you making you happy. That manifest should work for everyone else that's not you. Okay. Full of final releases or even better pre-compiled final releases. All right. But right now you've got a manifest that's got this cool concept in it. And the cool concept is that all you do is deploy it. This is you as a developer building your fast Bosch release. You can automatically create, upload, and deploy the dev release. You can just run that over and over again. It will always create a new dev release, always upload it, and always deploy it. And the mechanism is this thing called the version create. And you can actually have multiple, like you can actually have your manifest be creating multiple releases. So you can have, when you do your deploy, because I want to have a dev release of that Bosch release, that Bosch release, and that Bosch release, which is why the URL is kind of, oh, I don't know that dot's correct. I think it's supposed to be file colon slash slash dot. I think it's supposed to be URL. You'll notice I've already got BPM in there for completeness. But this is the gist of the idea that that's included. Later on, once you start putting final releases in there, because you care about other people, not just yourself, then what's also included is this create operator. So you'll start, after you start shipping your own final releases, you'll start to do this locally. And then you'll start to deploy base manifest, and then this create dot yaml. And that's the correct URL there, that file colon slash slash. All right, number five. So we've got a mechanism. We've got a workflow now. Bosch deploy, you can do that over and over again. What do we do first? What I like to do first is to just get packages to work. Not worry about the configuration of anything. Just let's just text that the compilation works. So that is kind of weird and awkward. And I'm not going to go into the techniques of, you know, when it fails, what to do about it. We'll go through this idea. And what I'm doing here, and this is kind of a flow you can do, is just start by making a package. Now, that package doesn't get deployed unless a job includes it. All right, so that's the only other step we need to do is add it to our dummy job. Bosch includes a dummy job called CF summit in this one. So all we do is update the package to reference it. We're going to do the deploy, and it will start the compilation of that package. That package does nothing, because we haven't told us to do anything, but we've got our workflow. It's all hooked up, and now we can start putting blobs and how to compile that blob. And you can just keep SSH'ing into this machine or stay SSH'ed into it. And just do cd. Because it's a sim link, this folder path here is not literal. It's just a sim link. If you do a new deploy, you'll need to do cd. Just to update that sim link. And then just check what's in there. Did it compile correctly? Can I run things? However it is that you go about making things work. Just get it working, and then you can go on the process of creating jobs. This is what I do. All right, language packs or Bosch packages. This was brought out throughout last throughout this year, and there's one for Java, Go, Ruby, Python. Some of these are like the Python one only has Python 2.7, doesn't have 3.anything. The Ruby one is 2.7.6. I've never heard of a sem-ver number with attitude, but there are there for you to help maintain. The Go one is very updated. Perhaps the best one is the Java one because finding versions of Java to include is kind of tedious. You've got someone else managing the updates of latest versions of OpenJDK. Hopefully he's right behind you. You'll make this happen. You've got people on this. You have no one that works for you, do you, Marko? You've got to cover it. Sweet. Good. Can you update the Python one? Ruby is missing 2.5. Great. But the premise is there. I'm sure you can help out. If one of these language packs is the one you want to use, I'm sure you could beg nicely and be given access to the S3 blob and help maintain the versions for everyone. But what they include isn't just the blobs, but some nice wrapper scripts to set up the environment variables so you can use them both inside your packages because if you're using Ruby, for example, you need to use Ruby both in your app package to do the bundle and whatever, and then you'll need to use it later on in your job to actually run your app on top of Ruby. If you use Go, then you'll just need Go in your compilation package, but won't need it later on. Java is like Ruby, you'll need it. Well, Java, you might already pre-compiled. But you know what you need for your dependencies, but these language packs have a nice, a nice consistent experience for both the compile.env and the runtime.env. A little blog post I did when it came out because I thought it was cool. Well, it's not just cool, I use it. It's not just about the pre-existing ones. You can do this yourself. Like any package from any existing Bosch release, you can use this concept to vendor back in any other project. And any time there's a new version, you can just re-vendor it and update it. So it's a way of sharing, which leads, I guess, to the prize that was discussed about, I guess, so what's the idea? It's going to be a shared, a communal place of packages. So that means it's now your fault that the Python one doesn't have 3.6. Is that what the award was? A new person to blame? Oh, we're just saying now. It's not your fault. Now that it's your fault, it's not your fault. You suck. Sorry, I do have a slide, so we took this picture. So yeah, sorry about that thing. If you didn't understand what that was about, it's about what I call them language packs because Bosch packages just seem a bit overloaded as a concept. All right, seven. I'm not suggesting, let's be clear, I know this slide to people and let's say, why would I want to make Debian packages? I didn't tell you to do that. But we certainly have had, wanted to Boschify things. So we've been Boschifying Azure Service Fabric, which is, I guess, came out of long, it's how basically Azure runs. Service Fabric, Cosmos DB, and all the bits that you ever use Azure. Actually running on this thing called Service Fabric. And when Kubernetes came out, the Service Fabric people got quite cranky about why didn't you find an open source it, so they did. But now everyone loves Kubernetes. But nonetheless, Service Fabric is super interesting. And so we're working in Microsoft to Boschify it, and it seems silly for us to rewrite their entire build tool when they were already publishing Debian packages. And a good thing about Debian packages is it references all the dependencies. The only thing we don't know is what dependencies to include. Because that's actually one of the benefits of using Debian packages. If you've ever tried to Boschify something that has like a pretty gnarly set of dependencies, and you're thinking, wow, I'm going to have to Boschify all that? Don't. That's another good reason for using the Debian packages. Because the way this works, what have I got here? So, okay, sorry, there are two steps. I'll come back to the one I was about to refer to. So step number one is you can just try this out. Which is, there's a Bosch release called OSConf. You might be using it implicitly for your jump boxes. If you use the jumpbox.yaml in your Bosch deployment, that's this, right? OSConf for creating users. The other thing it's got in it is a pre-start script hook. Which basically means you can run anything. And this is how I do jump boxes now. Like, literally, my jump box is OSConf for creating users, and then just a pretty gnarly script for installing Debian things. Which means I can go into my jump box anytime I like, and do app get things. It's a jump box. Let's not take it too seriously. I know there are other approaches now around using Docker images. I think that's pretty cool too. But you can get this idea. Like, at least install the package, and then perhaps you can move on to try to write the job whilst you build up the courage and their mental capacity to actually Boschify the package. Or not, right? But you can at least see how far you get with that package installed. So that's idea number one. Idea number two is missing on this button. Here we go. Idea number two is to still use Bosch packages to bundle up, right? Everything. But what this does is it actually uses the Bosch compilation step. So you've got the stem cell you want to use, Zennial, whatever, Zennial 97. And run that app get inside that, collect all the Debian packages that you're going to need later on. And that's your blob. That's your compiled package right there. Now, we'll get to the corollary of this, which is whether or not you want your end users to do this or not. It also means that if it's just inside your environment, you can do this with a purely private Debian environment. Again, you're going to think, well, man, my users can't do this, but I'd like to argue point number 10 is your users shouldn't be compiling packages. Like, what is wrong with you that you make other people sit there watching Java compile or whatever. That's not your fault. It's just a part of Bosch. But I'd like to argue that if we start with point 10, which is we're going to be sharing pre-compiled packages, that gives us a lot of flexibility to do anything we like during package compilation, because we are the only people that are going to do it. It could be purely private. It could be based on Debian packages. And that's what the notion is here is the Redis server package is actually running app get commands and we're expecting a lot of Debian packages and saying, I'm just going to keep these for later. And then the job called Redis server package install, it's a sort of a no-op job. All it has is a pre-start script which unpacks them all in order. So when we collect them all, we don't just collect them, we also figure out what order they need to be installed back in. And it sort of handles all that for however big your dependency tree is and how they're almost circular. And some of them are kind of curly, right? So it tries to figure that out. There's a little helper command in there that gets pulled in. And if you play with it, you'll see what's inside that packaging script. It's kind of long. And all you have to do is to say what top-level Debian packages. And if you want more examples of this, just ping me and I'll show you some of the private ones. Or look at the service fabric one because it's kind of pulling things in for it pulls in Docker from the Docker Debian system. It pulls in service fabric from Microsoft's Debian system. So it has a good set of examples of getting it from your own bespoke. I'm not arguing you do this to pull shit in from conicals. Right? This is more maybe it's a good idea, I don't know. But certainly if you've got your own internal Debian. All right. BPM. Am I winning? Am I losing on time? We're close. Oh, six minutes. Yay. BPM. There was a talk given yesterday on BPM. There was a talk given at this conference a year ago. BPM. If you don't know why you want to use BPM, that means you are so desensitized to all those shell scripts who have been writing for five years that, you know, you need help. When they first gave this talk a year ago at this extension track, they introduced BPM and said remember all those shell scripts, it's like yeah, I had forgotten them. You just sort of ignore them. Generators generate them and then you just find the bit where you put your stuff and you sort of mentally block out the rest of the pain. It's kind of like walking through Manhattan. It's like you get to where you're going and you don't really notice you did it with a million other people all at the same time. You just block them out. Before BPM, we just blocked it all out. But now with BPM it is a lot easier just to do the thing you need to do and therefore a lot faster and gives you a better contract. You realize, oh yes, I do need to write to disk. Oh yes, I do need to do this and you expose an explicit contract. So yeah, there's that. You should look at that. All right, faster getting of your own Bosch. If the only Bosch you've gotten the one you want to do development on is your production AWS vSphere Bosch, you might notice that it's slow, slow to bring up VMs, etc. Faster to use Bosch Lite. Bosch Lite doesn't have to be just on your laptop. I'm using Buc because Buc wraps up a lot of nice things but even if you just go back to the Bosch deployment repo, you'll see there's a couple of Bosch Lite operator files in there. Add those in there and basically you say I want to deploy to Amazon but instead of putting the Amazon CPI on it whenever you deploy Bosch there's two CPIs involved. There's a CPI locally to bring up the VM and then you put in a CPI that will be used for all your deployments. Bosch Lite fundamentally is CPI to bring it up on virtual box and then we put in the Warden CPI or the Garden Run CPI to do the subsequent ones. So I take that step back, let's bring up one on Amazon and then put Warden inside that. And that's the ci2.starkandwain.com, a lot of the Bosch Loosen there basically just reference that's running on one Buc but then it references the second Buc which is a Bosch Lite running on Amazon. So we have that. And then Buc is nice also because it has little helpers for setting up access to CredHub and Bosch and whatever. So if you don't like Buc great do it whatever way but the idea of using Bosch Lite for your deployments makes everything a lot faster. And finally shared releases, so I want to explain this in two parts. One is this and then the next is the pipelines that we use which you can borrow. But the gist of it is this export release command. So once you've created it and compiled it you can export it. The documentation reference there includes a little example of a dummy release a dummy deployment which doesn't deploy anything but it does it so that you can then export it. And when you look at this you think that's incredible, this is great. I should CI this. And that's what we did. So there's this project which is what we use in Starkandwain for all our Bosch releases. It's why it's pretty low for us to maintain Bosch releases because we maintain a consistent pipeline for all of them. And one of the things it's now doing is once we cut new releases on the right hand side that feeds back into the compile release job which then compiles it and then the use compile releases job updates that base manifest to inject. So ship it we'll update the base manifest to put in whatever the new version is and what is that get URL and then three minutes later it'll be updated again with the final release URL. So all of the Bosch releases I've maintained now I assume people are using pre-compiled releases not. Now it's unfortunate and we have two minutes to bemoan this. Where am I going to go? So this is a finished but I do want to take my two minutes to talk about something whilst Marco's here. Where are we at? And everyone else. So you might have seen this. Hands up if you've ever used the Git plus HBS knowingly and know what it does. It's not enough people that's a terrible number of people. So this is actually really valuable but mutually exclusive with pre-compiled releases. So any package software any package software you download and use you have no idea what's in it. It's trust really throwing it out there and trusting that if I download the Docker binary from Docker or Inc I trust that it's the same as what's in the source code. But I haven't actually seen them build it and we have nothing in our profession that sort of does this audit trail of I promise obviously if we did it would be using blockchain of some sort but whatever there's no audit trail that says I promise that what's running a production is entirely traceable back to source code and I promise that that's it. But we definitely don't have it in Bosch ad blob boof we have no reference to where that blob even came from. You don't go ad blob in a URL and we keep store of that URL and everyone else can say ah that's where it came from. Nope gone no one even like you just trust the blob. Like yes you can look at the Bosch release and go I can see how we compile it but you don't know where the blob came from. So ignoring that problem which is not what I'm talking about what we do offer in Bosch is a Git repo. A Git repo that tracks every time we added a final release. Much more trustworthy to be sharing a Git repo which has that full audit history of changes and let local people rebuild their Bosch releases from that Git history and that's what this URL does. So when you deploys the first time what happens behind the scenes is it looks at URL downloads the Git repo the cool thing about the Git repo is it means you can use a private Git repo because you're going to use your local Git credentials and so it brings that Git repo down behind the scenes it figures out what the Bosch director doesn't already have downloads those pre-compiled or those pre-built blobs and the jobs and everything uploads them and it's perhaps a much more safer pleasant way of doing it. The downside is it doesn't support pre-compiled Bosch releases so you pick one or the other. Dimitri did have the idea he was aware of the exclusive nature of these two things one idea was that perhaps final releases could include I got nine seconds I'm fucking using them one idea might be that final releases could include pre-compiled assets but anyway so there's ten ways let's go back to use my zero seconds to look at them we had these five and those five so thank you very much I wanted to make sure we have time for questions that's part of the pressure here so five years I've never had questions yet in which case you could just keep blubbing away on that's right questions Marco I would be happy to mention that actually deploying Debian packages is kind of breaking a philosophy in Bosch in building Bosch releases which is being hermetic builds means you ship with all what you need to create the final results I'm aware of that your Bosch gen dash dash APT is here to bring back this hermetic build to ship all the one of the arguments against the Debian packages is oh but what about sentos stem cells hands up who's ever deployed anything to the sentos stem cell alright let's move on yeah and you should all be moving to Zenil anyway and they've now started to cut final releases that kind of have a three month lifespan 97 and whatever the next one is looking at Marco he's giving me nothing see ya well sorry there'll be another one every three months but the 97 will be maintained for some period of time because it feeds into a year I guess is a good number because you've got it for pivotals okay yeah the downside of the pre-compiled thing is you are coupling to a major version of a broader stem cell so a bunch of Zenil 97 dot anything so for everybody the gentleman speaking here with the nice beard is Marco from SAP who's the new PMC PMC lead for Bosch which we have not yet figured out what that means well that means you can twist his arm if you want to Marco do you do this alright I'll come back later there's also Danny, Danny does important things he should so I have to admit that I didn't write Bosch releases in quite a while now but I think basic things didn't change during the last year or so so what I was always missing was convenience things that you for example get in tools like Chef or Ansible and you often have to come up with stuff like item potency on your own in shell scripts and stuff like that so I always thought why don't you integrate something like Ansible for example sorry my argument I did a talk five years ago about Bosch vs Chef I was in the community back then it's not versus you know it's not versus I've given you no time to have that conversation why can't you write is it not more convenient to write that it's incredibly convenient it's tragically item potent we came up with the idea to provide Bosch as a service to service providers on our platform and to be honest after two years it's almost like ok we skipped that because no one comes up with a production ready Bosch release and it's just not fizzle I'm sorry you're hiring Stan no I'm happy to help I don't mean to be smart about it you know what I mean to be smart about it that's exactly what I was doing let's get out to a closer show I mean I understand the problem it's hard to get people to focus on learning new things which is why the attraction to Kubernetes is because people seem to like it and it's hard to make people like things it has to be a comment would it be possible to add in Bosch and the default CI pipeline it would be great to generate yeah that's a good idea the pipelines are pretty cool and they are in a different repo so yeah alright let's give them a big hand again