 Ladies and gentlemen, Dr. Nick Williams! Oh, that's excellent. With the automatic fade out. A little bit meta there. That's Walk on Music called Walking on Sunshine. I liked it. So, the topic that you've all come to see is vaguely similar to what I'm going to talk about. The title said Bosch Workspaces, how we deploy everything with Bosch. That is the topic. But as you tell the story, we have new ideas on this topic, and it will be remiss of us not to share them with great enthusiasm just because the title said something else. So, I'm more interested in you having what's interesting than what we've done in the past. Let's quickly go through for those of you who don't know me, us, Stark and Wayne. That is a very large logo. Ridiculous, large quite frankly. We're a consulting company, so we only get to do things if we can convince customers that it's worth solving for the most part. And that's a lot of fun. You look around, this conference grew, we grew. I think this time last year there were six of us. It's now 15, and that's pretty awesome. And many of them, very, very new. So, it's been interesting watching our people learn this stuff as quickly as possible. And I'd like to think that that onboarding process is not just for us, but for everyone getting a lot easier. If you've got help. What else have we got? This and many other things. We have Twitter account, please follow us because we have no idea how to find you all. We make all these blog posts, and we really don't know how to get them to you. So, this is, you know, since none of you use RSS anymore, I don't know what we're supposed to do a mailing list. We have a fax, we could do that. That would be pretty, we'll just send it out to the engineering groups, fax systems. GitHub, how does that fix anything? Who honestly looks at the GitHub homepage for the activity feed and gets useful information out of that? That's not a solution to any problems. So, we did grow, and as such, had to bring them all in. So, we bought the jet, and that's pretty cool. That cost five bucks to get that done. It's substantially cheaper than buying an actual jet. But one day, I will get the jet, and then the laughs will be, well, temporarily on you, and then every month on me. If you want to find out, yeah, we have a stock in GitHub account, but we don't really put anything in it. We find this a lot of fun, putting everything we do in the Cloud Foundry community, organization, and helping people put their things in there. This org is kind of a discovery solution. How do you find other interesting projects? There's now like 167 repositories and 91 different people who have ownership of this. So, if you're new to Cloud Foundry, lots of interesting little side projects and core ideas that might help you along. Some of the ones that I guess I and other people have stuck with when I've worked on is these ones. Bosch Gen, I looked up the other day. This is like the first thing I ever did when I discovered Bosch three years ago, was I love making generators, so I wrote one, and I've been looking after three years. And I think many people, I look at other people's Bosch releases, and I can sort of see the tell-tale signatures they may have used this tool, mostly from all the craft that it generates that you didn't delete. But, Bosch Bootstrap, we have a bunch of interesting Terraform plans to bootstrap on Amazon and OpenStack, and hopefully soon we can do an Azure one. That'd be cool, because Microsoft is all cool now, right? Because they wrote like a new text editor that runs on your Mac, since you haven't owned a Windows machine in 10 years. And Bosch Workspace, which is somewhat the top of today. And lots of other interesting things, but these are a few. And a gazillion blog posts. It's got to the point, there's like 15 of us writing blog posts, and there's a lot of blog posts, and I'm almost going to apologize that we write too many. But there's a lot of interesting stuff, not just advanced things, like perhaps the topic of today, but a lot of, as our new people, learn things. We basically just, if you have an idea, share it. At least there's one person in the world that reads it, and that's Future You. You invariably end up reading your own blog posts, because you can't remember anything. But it is pretty awesome. So if you read these blog posts, we do not know you do that. Yeah, you get good analytics, but who trusts that? It's probably all robots. I would appreciate feedback, because it feels a little bit like writing until void sometimes. It is a lot of love and affection in sharing these ideas, and I'm happy to hear about it if you use any of them. And there'd be URL, because it's the internet. And of any topic, so if we had this problem, like you get to do one talk, theoretically, well, I have 60-something interesting topics to talk about. So let's go through them all sequentially, chronologically, tomorrow. We can use this time slot for whatever. It should be a lot of fun. And so if you have any questions of this or anything else, we'll do that. Okie dokie. So, the problem. And this is the problem that I've talked about this idea for three years since I discovered Bosch, because everyone else, when they talk about deploying something, they get very excited about deploying it at all, like docker people. Look, it comes up in 30 milliseconds, and you've never created a production problem so quickly. It's an all-new world of production disasters. It worked on my machine. Here's a one gig Java docker image. Have fun with that, production people. So, you know, if a production system lasts for five years, they're 60 months, I find the next 59 months as fascinating, if not a lot more fascinating, in part because Bosch can do it and no one else's infrastructure or the concentration tool can. It doesn't justify at all how goddamn hard it is to learn Bosch, but once you do, the next 59 months are a lot better. So, the challenge then becomes awesome. So, we have Bosch. Hooray! Let's say we tried to deploy Cloud Foundry. Well, in the last four months alone, there have been 12 separate releases of the entire thing. Some of them are small releases, some of them substantial. 500 commits just to the CF release, so some of them are bumping submodules, templates, the job specs, et cetera. But in the jobs templates alone, the definitions of the jobs, 99 different changes this year. So, you know, and it's not like they don't try to help. One of the reasons I became fascinated by Cloud Foundry, so I was previously at Engine Yard, and so somewhat a traitorous action to leave one pass to go to another one, but this was open, and not just open, but even from when I first discovered it three years ago, they gave us Bosch with somewhat of a Rubik's Cube type challenge, although I bet you can't make it work, but there was this vague idea that they did care that you should be able to run it yourself. And so, you know, a year and a half ago I think I spoke at this conference, and I joked being the only person in the audience that got Cloud Foundry running, and I know now that everyone can do it, and if you can't come and ask and we'll get you, you do not go home this conference without knowing how to run Cloud Foundry and Bosch. We can solve that problem together in a sort of Lord Darth Sidious type way. You join the dark side, we'll see if we can't get Cloud Foundry running. So, but each version that comes out, they do a great job of documenting the release, links to every pivotal tracker thing. You cannot be remiss of information, and obviously the source is open. You can learn all of it. I mean, you know, it had to be a full-time job keeping track of it. But a couple of ideas just in case you're looking for some Gitfoo. Gitlog is one of the things I do to get a feel for what changed between versions, so you can reference a couple of shards or tags, and then double-dash and then some sort of glob of the files that you care about or subfolders. That's kind of fun. Do a Gitf of the same sort of thing, so let's look specifically what changed in a spec. Now, this is interesting. So, A here, we've sort of fundamentally changed some aspects of the UAA worth looking into. Should you be looking at doing this upgrade? And I bring this particular one from 195 to 207. It's relatively, 195 was the last one from last year to 207, the current one. Well, we're working a customer where they've been running version 194 for about five months. And then they decided to just upgrade to 205. And it didn't work. I can't believe it. Who would have thought it just can't randomly jump from one version to another without doing any investigation whatsoever to what might have changed. And these are some of the things that changed. All of a sudden, there's a whole new way of the CF tool looking up, doing the translation between user ID and username or email. We need to set that up. Not a major one, but watching out for changes in defaults. That's a good one. Because I bet you didn't specify it in the first place. So it went from a value you didn't know what it was to another value you didn't know what it was. So who knows what that might be. And whilst you're struggling to figure this all out, literally none of your users care about what you're going through. They just want it to work. It's a platform. It's supposed to just run forever. And here we are struggling with premise that a new version comes out every week or two. I guess at this point, if I was a vendor, I'd say, and now you can use our product that just does all this for you. In this conversation, we're going to look at Bausch and deploying things where it is our problem. So this is what the users care about, developers, I guess, are a user. They care about they've got a test app. They're happy about that. This is as complex as their life gets. Deploying the app twice. Not all of them are that complex. For one, we don't have the ability to copy a droplet. We have copy source. I don't know if you know that, but you can copy source from one app to another, the build pack will run again and hopefully you get the same droplet. But this is about as complex as many customers will get. An app, test it, and then promote it into production. But we know it's more complex than that. Very soon you're going to think, I've got to stop just upgrading production Cloud Foundry and hoping it works. You have the bright idea of deploying a second Cloud Foundry, which we'll call a test Cloud Foundry. And you might want to put some test workload on it, test apps to check that it's all working around the cats and things like that. That's pretty awesome. Now, if you're running OpenStack, you've probably learnt the most valuable lesson about OpenStack, which is it doesn't work. And so you might want a second one of those. I mean, I'm sure they'll fix this now, but when we started years ago with OpenStack, literally there was no upgrade model to OpenStack. The upgrade path literally was we're starting another data center in another part of the country because you might as well make it exciting and hope your customers find it. There still would be the old versions of OpenStack running because there's no upgrade path. So anyway, as funny as it is to laugh at OpenStack, I mean, it's their fault, but nonetheless we will make the jokes, not them. Very serious people not capable of laughing themselves. That would require agreement on something. I don't think they're capable of that. So you can see now we have three cloud founders running on different clouds, different bosses, different targets, you know, let alone multi-cloud, let alone multi-AZ, et cetera, multi-region. And so we need a solution to this and many other problems. Hopefully that at least gives you an example of the problem. So fortunately, Alex from Pivot wrote a tool called Spiff, which is, I guess how could you put it, is the best in-class solution to this problem. That's politely put it in Alex. That's pretty good. You know, at this point, I wish I had, I almost like, fuck it, I'm going to find it, because he is in the audience and this is hilarious. I just think this is the funniest thing I'd ever seen. Go straight to the issues. I assume it's still an open issue. Alex, you know which one? Oh, no internet. Okay, I'll fix that in a second. Let's fix my internet problem first. This better be worth it. There's a guy with slides that thinks how much time I've got left and what he doesn't have one is saying you're wasting time. Issues, how do I get, how do I get, there's that. All right, all right, all right, this is so worth it. Alex, do you remember which one it is? It's like, here we go, that's the one. This is gold. This is classic open source. I've personally stopped using Spiff on my own projects. Sorry. To be fair, a little bit of bye beware as we proceed through the rest of the talk. Well, that's not, you know, that gets distracted by that. It still works and it still gets bumped along and as I said, it's still best thing you tell me what else exists and we go with it. And the Bosch project this year is going to do some fascinating stuff with cloud configuration and links and manifest themselves may get really simple and we may not need this tool and awesome and nothing I talk about at the end of this talk will be affected. The next thing I'm going to talk about will be substantially affected but in the long term, you know, everyone knows this is a problem. But this is where we're at. Okay, so we want to Spiff merge, we want templates and we want to be able to have something that we think will work, you know, for the three and feel confident that as we move through we, you know, we're going to get reproducible behavior. All right, so Spiff, we saw it. It's got, it's got ASCII art, so you know it's good. How I quickly defined, you know, watches, blah, blah, blah. Did you bother making ASCII art? Care about your project, for God's sake. And so here's the idea and you've probably hopefully seen these. This allowed, this sort of came about and I thank God finally Pivotal who were deploying, they had templates that were deploying off to Amazon but until Bosch Lite came out they didn't really have a motivation to start making templates for anything else. And much, much later they start doing vSphere and then much, much later OpenStack. And so Bosch Workspace was this tool that came about because there were some of these a year ago, but it's not the ones we needed. And our deployments were different from Pivotal's and we wanted different clusters and so we needed a solution to this. But nonetheless, very awesome that Cloud Foundry comes with a bunch of example templates on how to deploy it. And all you do is you take the ones you care about and you merge them and you get a manifest. That will just work for some value of work. Now, a couple things to keep in mind is do not just go ahead of the project and assume those templates work. That will be an unhappy day for you. You want templates that go with the version of the software. And I'm of the belief that, yes, it's kind of good that the templates are in the repo. I will be forever unhappy that deploying anything with Bosch ultimately means git cloning something. So if spiff templates are where we are, personally I would like them to be shipped off and packaged somewhere else as part of that. And I guess that's kind of where we're at with the concourse stuff to some degree. But anyway, at the moment you're pretty much going to have to git clone, build these sort of things. But you do need to keep in mind, which of those do you care about? What order do they get merged in? And I guess more importantly, you're going to very quickly want them to be different. And so now you've diverged. And then do you remember how and do you remember what? And so Bosch Workspace was an attempt to continue to live with the continuing spiff templates that came out because they were tested to some, you know, Pivotal, the core team were testing them for some subset of testing. Obviously not for everything we were doing, but they were a pretty good starting point. But we wanted our own changes. We wanted different cluster sizes. And so Bosch Workspace was this idea of how can we make this happen. And so what we had is another YAML file because we're all good with that. And nothing bad ever happened with YAML. It's just when it gets to 3,000 lines long, you start asking the what the fuck question. You're on your fourth coffee before you get to the end. This has got to stop. There's no storylines, no arc, there's no character development. It's longer than a novel. So this is a smaller manifest and to some degree and more readable. You can get the gist from the list of templates now that what this is going to be doing. It's going to be using Postgres instead of not Postgres. I mean RDS, I guess, or however else you want to do Postgres. You can bring your own Postgres, which is probably a good idea because your Postgres might be highly available, whereas ours isn't. You might use HAProxy instead of ELBs or an external load balancer, et cetera, et cetera. NFS because you hate yourself. Apparently you're using Google where it only gives you results from the 90s and you've learned about NFS for the first time. That's an awesome idea. And other ideas that are a pure goal. Let's turn off SSL because we don't have security problems around this place. And it allows you and your team to understand at a glance what you've got. Now, in order to do that, you do need to, one thing, we've sort of evolved in having lots of templates in order to get this sort of readable, cognitively, I've lost the word, but readable, that's a good word. Understanding of what's coming in. The ones here are the ones from upstream. And Bosch Workspace just makes all this happen. It has the git URL for where this BIF templates come from and you can add releases. It makes it a lot easier to mesh these things together. And the thing is, if I really wanted to use it, I'd talk more about it. And I'm talking about it because it's mentioned in the title. It would be rude not to at least give it a glancing blow. But it's served us. It allows us to solve these problems. Specify the stem cells, the top level information, the inputs. What do you care about? Now, there's another set of inputs, which Alex, did you come up with Meta as the idea of the top level? Meta or not, instead of inputs? Gold! Gold! Gold! That's cool. That's some definition of cool. This is the things that are going to go into your templates. Hopefully you can get it down to be small and readable as opposed to long and excessive. But it may change from company to company as to what goes into that because of your deployment. Part of what Bosch Workspace gave us was the liberation from what the upstream templates were. We can start to curate our own templates. What else have we got? So there's your inputs. If you're interested in deploying Cloud Foundry, there is a workspace. The idea of Bosch Workspace is an extension to the Bosch CLI. You don't really set it up yourself. You tend to find these things lying around. So Cloud Foundry Community CLI of Bosch Workspace is a Bosch Workspace. Extra templates to do extra things, like the examples they gave. And this is what we used for the Terraform work. You start the Terraform thing. It brings up a bastion or a jumpbox. It brings up a NAT. It brings up all the subnets and the VPC in the other order. And then on the bastion, it does this stuff. Which is pretty cool. Now, one of the things that's super cool is we started writing job templates for a much smaller cluster. And it bemuses me to no end every time we find someone which is many people who think that Cloud Foundry cannot be deployed. There's this idea that Cloud Foundry needs 20 VMs. Sorry if I could take a moment to dispel that. It's just software! There's nothing about those bits that preclude them from being on the same machine. There wasn't like a blood feud. Whatever imaginary reason you can say, well that needs to be on it. That's Ruby and that's Java and they hate each other. Most things, really it should all be and I think the Bosch News Project spends a bunch of time trying to make sure Cloud Foundry can all work on one VM. But 6 is a good number, especially compared to 20. And I get to use this bit. That's awesome. This is new, so watch me use it. And you can double it. You can scale it. It's HA, so you can double it and get high availability. You can scale it out, the DEAs, etc. It's nice if only as a source of examples of templates. But I sort of fell out of love with Bosch Workspace if you didn't figure out where this was going. I didn't have a solution to the problem, but it got complex. It's a little icky. And as I started asking other people what they were doing about this problem, everyone was just using shell scripts. So at some point you've got to realize you've probably gone down a dark path of complexity to solve what? What problem were we solving? We wrapped this command or running spiff merge in a truckload of Ruby that was a plugin that did metaprogramming on Bosch which is already complex and constantly under threat of actually being changed at some point. And so this became an idea. Let's just go back to this. Continue with the idea of writing our own manifest templates that described it, how we want to describe it. And about this time I was... I got to see Alex who let us down so severely with spiff redeeming himself. Sorry. No, no, no. I don't mean that at all. Best of his class. And with a thing called concourse. And so this led me to... to get to re-explore this problem of the last month. And we'll come back to this in a minute. So this is somewhat complex looking thing but it's pretty awesome, I think, in that what this... This is a concourse pipeline. And I think this is another fourth time I've brought this up in various meetups in the last 24 hours so I'm pretty excited. What this one does as we'll go through it is it deploys something. In this case it could be Cloud Foundry, Redis, whatever it is you're deploying. Three times. The goal is to protect production. Partly from Bosch. Because I don't know if you've ever used Bosch. But if it doesn't go well, sometimes delete deployment is the only solution out of your particular predicament. And if we can keep that to a minimum over in this department, we might get to keep our jobs. So this pipeline is designed to protect us from Bosch and us from screwing up production. And so the idea here is that various different inputs come in. We're going to give them a crack. We're going to try them out. And then we are going to have a candidate that we will pass through this pipeline. Ultimately, I automatically am manually deploying. And this became very exciting as I started to learn how concourse worked. And ultimately as of yesterday got a variation of this and some other sort of interesting variations working. So I did really want to share them. So back to our shell script. So one thing that's different as I rethought what I was trying to do with Spiff templates was I no longer wanted a big stub of lots of things. Like the name of the deployment to me is not a stub thing. It doesn't change. It's just a piece of configuration. And nearly everything in what we've done with stubs, if you've ever looked at the Cloud Foundry things which ran the generate manifest or make manifest spiff and looked at what it did, it's this big stub and it kind of goes through. And that is a terrible sign. I'm going to ignore that. And to me, the only thing that's going to the stub are what you're testing in the pipeline. So the release Canada versions and that sort of thing. And this is where I've come to and I'm happy to share it because any feedback would be the first feedback because this is all relatively new to many people. So we have all the today and tomorrow to discuss this. And so here's a simple look at what we've got. We've got, oops, pressed the wrong magic button. Whatever. So those three there are the inputs. I know there's looks like there's many inputs. But the ones we're testing, a new version of Redis, a new version of the stem cell, sorry, a Docker, a new version of LogStash, a new version of the stem cell, they're the things I care about checking. And ultimately, if it succeeds, I'm going to take those templates and I'm going to say this is the candidate that I want to test ultimately in production. And there's my pipeline. It gets passed through this pipeline as I test it. Now, Concourse, my understanding, mostly being used by people moving artifacts through shipping software. And while Concourse does have a Bosch deployment resource, it also has a cloud for a CF one. Chris can say to me whether he thinks it's good or not. Is it good? Would you use it? You would use it? Does it fix the bug where it deploys apps for the first time? No, not maybe. Deploy Cloud Foundry. You can use this for both. I don't know if that was the intended purpose, but this is very exciting to me as using it for production systems more so than artifacts. And I want to give a little, my one remaining minute that he tells me, yeah, I've got 10 other minutes, they're fine. You're doing a great job and I do not want to take away from the magic of the service you're providing. Providing me any service. Terrible service. Context insensitive timekeeping. How about you show me the five minutes? Bang, when there's five minutes to go. Not that I know when that is, but that's why I've got you. So let me know. All right. This is an example of watching Concourse. It takes a minute to watch. Again, it's doing a Bosch deploy because that's what I wanted to do. It could, it's Concourse, it could deploy anything, but we care about Bosch. Let's look at doing a Bosch deployment. It had to do the package compilation for the first time because this Bosch had never done it before. But what we're going to show is, is that I actually export them and ship them into all the other Bosches. So no other Bosch ever does them. Why? Because it's one thing that less that could possibly go wrong in production. And I'm trying to protect production with every possible talent and skill and tool that I can. So here we are exporting those compiled packages and then because we have two other environments, they're going to ship them off and install them. And it takes longer than that. But he's rushing me. So I'm going to get through the animated GIF really quickly. We then deploy pre-prod and that would do a deployment from production, whatever those templates were, rebuild it, then upgrade it to whatever came from the candidate. If that works, now we've got something we can ship to production. And if you're interested to know how that ends up, mysteriously, come to the booth. You know, we can talk about this. It's really interesting. And we can go through it because, you know, he's not giving me enough time to share. So this is the subset of templates that are going through the pipeline. This is that candidate. If you want to know how to make templates, I wrote a tool called Make Me Spiffy. And it takes values from a temp, from a manifest and puts in a spiff template and just makes it more spiffy-like. That's kind of fun. I'm not very good at naming. Alex makes awesome names. I'm very envious. Pro tip, because of concourse, he uses Docker so that now you know it's cool. You were unsure a minute ago, but I've said it uses Docker and you're right, we've answered some big important questions. Super important part about using Docker, I think, is to put the Docker files in with a project that uses them. I have other points, but this is, you know, at least I get to say Docker, so now I validate. It's on every line. Nailed it. There's some of those tools, the things I mentioned. They're all on GitHub somewhere. And I decided that the tutorial I've been written is I'm going to make a book for it, because I'm really excited about what we're going to be able to do with concourse and certainly how it's going to roll out for our customers and for everyone else. Logo and Twitter thing again. And if you want to come tomorrow at 10.40, we can talk about this and any other sort of associated topic. This isn't going to be as few. Really, how many people are going to turn up? That's three. Me and three cardboard cobbings of me. So there will be space for you. We have a couple of things going on at our booth. We've got cool t-shirts. Made cooler if you wear them. And if you come back at 3.30 tomorrow, there's prizes of cash and a TV or you can, you know, spend time with me. Who comes up with these quizzes? That's right. And another one, if you don't want to wear our stuff, go through our blog and answer some questions. So thank you very, very, very much for coming.