 mind up just go and grab someone you know physical violence hello work for us I say we I say we kick this bad boy off welcome to to the conference I hope you've enjoyed the warm-up for the last day and a half it's been good if you could fill me in I missed it writing slides I just to the lady doing the the hand things if you could do them in Australian I'm sure more kangaroo gestures or something I don't so I just want to introduce me introduce this topic and and and the t-shirt that I'm wearing so over five years ago I put a blog post up because there was a there was this thing it was silly it was called Twitter and and they one of them complained a lot about Ruby and rails and blah blah blah and and so I wrote a blog post and I released a little bit of code which basically said look I don't want to make a big deal of it but I've solved all of your scaling problems it was silly you'd never use it but as you said the bottom I said I wonder if I'll give me a free Twitter account I didn't have I didn't know what it was for and or at least a t-shirt and actually so at the next rail Ruby Koffler and the Alex Payne actually gave me this t-shirt and completely forgot about it except there is a Twitter bot that on your five-year anniversary apparently tells you congratulations you've had a Twitter account for five years how's it going for you so I that you know I thought I should get this do you know if you go to Twitter now you cannot get this t-shirt I mean it's mine so fuck off so more about me my background it turns out that I have been thinking about distributed systems for a while whilst at the same time not caring at all about them very confusing state of mind to be in but my actually remembered my topic was something along the lines of how do you change distributed applications at runtime in a sort of enterprisey scenario with lots of different participants and and it's awkward and and how do you let that application do it all itself and I highly recommend not reading that 200 pages of quality text mostly because it was written when my one of my PhD supervisors gave me this top tip on how do you write a PhD thesis and she said the more boring the better you can imagine why they're not selling with O'Reilly or pragmatic programmers but making it read was about my only sort of daringness at the end so that was a long time ago I for a couple years did rails consulting but I did that app bit I made the thing I didn't make it pretty and I didn't make it run I just made it someone else did the other bits of doing ops but then I became more interested in this space of production apps and so I came to this country whichever one it is and when in four days time it could be different so who knows remember vote whatever doesn't matter what you do you're gonna keep killing foreigners with drones so you know I'd like to say it was safer to be here away from the drones but then everyone's got a gun actually if you notice if you go to the front doors not now right at the bottom is the little sign little like no smoking sign that has no guns and no knives I'm not sure how short people are that have guns I would notice that but so when I so engineered if you don't know we one of the you know we do the hosting of the Ruby apps that you may be having one or two of specifically mostly around businesses people that have actual you know big apps they're doing big things so well it's really fun and interesting to hang out with those people I've learned more than I mean honestly I've learned stuff and everything I learned I didn't want to learn I didn't come to engineering out to learn DevOps and system administration and cloud stuff that's I just like writing apps so in the last few years I've accidentally learned things and unfortunately due to your choice of session you're gonna now learn some of those things and so I will attempt to at least make it interesting so when I how I came to be thinking about this particular topic which we'll get into was that last year I tried to convince a large number of people that you should use J Ruby and Rubinius and I tried to focus specifically on one aspect which was threading because last year there was a lot of who are about how much fun it would be to do JavaScript and event programming fortunately we've all got over that it was all a bit silly and now we've got back to writing apps that are both neither invented nor concurrent so but I kind of make it really simple I said you know you still do want event and programming and you still want threads and fortunately web apps really easy put the procedural stuff wrap it up an implicit thread around the request and then put it vented at the front and use engine X and that was it you know what I thought it was a really compelling argument no one changed the behavior based on that and so I start to think about this idea that that the choosing J Ruby choosing Rubinius was perhaps more of an operations choice you're going to choose it for business reasons like using less resources and perhaps getting better debugging out of a production system and things like that and perhaps developers didn't make those choices or didn't want to or they knew they wanted to but it didn't for some mystical reason and that became really interesting to me which is now kind of where this this talk starts off so I guess unfortunately there is another slide about me I've done lots of stuff there is a website that actually tells you approximately how many projects you've I don't know if you're a dog urinated on how many you've participated sorry that's the word and a lot and but what have you come interested in is the idea of resilient production systems but also ones that I could just ignore because I've gotten really good ignoring static systems I have a GitHub repo full of them easy to ignore you just turn off the notifications easy as you like but production systems don't seem to be as ignorable and so here's my sort of fun challenge for you to spot the difference between 300 code bases and 300 apps all right now I don't know if you spot it so this is my metaphor is sort of 300 books on a shelf versus 300 people in the company the books are a lot more fun all right people suck and I mean look around you're all ugly and that's not true you know all likely and but you know so static things are a lot easier to live with and think about and and so it's kind of in the nature of our profession as engineers we're a Ruby conference another DevOps conference we'd like to write code and we like to write code we like to test code and it's just a really sad unfortunate sub part of our profession that for most of us it unfortunately goes into production and we don't talk about it that's just just did you know it's just annoying yes it's running and yes there's a dot-com and look at my code and how pretty it is that's the important part and look how fast my tests are look at the no we don't talk about the production stuff very much I mean it's a sad indictment on Ruby that not sad yes sad I mean Engine Yard Heroku and those sorts of companies came out into existence created concepts like pads no one else needed to solve these problems but we did that's how bad it was to try and run Ruby apps so this is this idea that that as a group we have so optimized around our own developer happiness that I'd like you to start thinking that as your production code bases get bigger don't get more traffic you are going to need to give up some of that you're going to start to think more about production happiness because if you don't you won't be a happy developer actually I can't prove that it just true that's the quality argument you're going to get here the next 30 minutes trust me all right it sucks so you need to do both all right let's go through this just in case you've never had a production app that's what one looks like traffic comes from the left I don't know which way you're looking and but it's not not just easy constant traffic sometimes it comes in impulses sometimes it comes in sort of large stressful batches and keeping your app running in those situations is not something you've probably ever done before and we're quite torturous to live with other things you're going to do with production systems is change them deploy things which you may have seen in the title of the talk and I will come back to I'm gonna spend a lot of time just talking about tools in general round thinking around this space and then specifically focus on the title of the talk which is what I think is the frontier of the next sort of thing we need to be thinking about in deployment because obviously there's a lot of arrows involved and we need to be careful of them and then we do that's called the magic move feature of keynote it's very cool I'm gonna do it again so inside your app it's a bunch of bits I know this is tricky because you all think your app your Ruby app is like God's gift that you know if only you could just put that on the homepage people would buy more stuff sadly it's just another box surrounded by other boxes which do stuff and your app talks to those boxes and if all things go well perhaps your users do stuff but all the boxes need to keep working and need to talk to each other and if you start to scale and if you watch some of these other talks people saying you know one web app box is not enough we need lots more web app boxes then it'll look like this and you can only imagine this is getting simpler but this is popular at the moment the service are integrity architecture because Steve Yeggy told us to because he worked at Amazon that's kind of his logic um anyway no no I don't want to diss the SOA I just want to say that when you do this something's gonna happen not only yet you the code you wrote and the things you control talk to other things that you have less control of I guess my best example is if you watch the Travis I know disrespect of Travis but this I've noticed this they and and Engine Yards own Twitter accounts of Travis's Twitter account our Twitter account often has to make comment about how our customers can't do something because for example GitHub might be down or Ruby gems might be down so this is this idea that the external dependencies a part of our production system even if we don't control them and but nonetheless so even if you've never sort of had that picture in your head let me just give you the picture that's actually in your head the dashboard of your brain dashboard of your brain pretty much says oh shit something's wrong or what it thinking is they're just the normal errors no one needs to build that dashboard alright locked in but I just want to sort of say do some basic math basic math I'm gonna use the water if anyone else would like some just up here not enough cups to go around but I think on the whole most of you are probably not going to come up even though you were invited alright hopefully you've gone through the sums I don't show if you call them sums but that's what they are the arithmetic that if if you've got a sequence of parts could be an app to a database an app to a caching to database whatever an app to another app to a database if there's some sort of success rate which there is if you're lucky it's a hundred percent but if you've got a hundred percent success rate that's probably because you've had one request and it was successful and you're feeling very confident about yourself but assuming there's some you know number then if they're dependent on each other then as you go through and I've used the same number 95 95 95 the basic math is you multiply it out and you get a smaller number that's the problem so when you go back and look that SOA diagram as you add bits and things talking to each other the math means that it's more likely something's going to be wrong at any point in time both because of the traffic that's coming in normal traffic impulse traffic stress traffic or when you change something so when you're doing deployment you're changing something and there's a chance something will go wrong and the question is what else is going to be affected what bad experiences your customers are going to have and so yeah so this diagram is not taken from any real production system just kind of looked pretty couldn't figure out the math for that one so this is kind of this is kind of the high level idea I'd like you to think about and take away is is the things I'm talking about may not be relevant to you yet because you may not have a lot of traffic here at because if say you've got a 95% or 85% success rate which is pretty low so you should fix that now and you're only getting 100 requests a day then you're only gonna have you got five errors 15 errors and that might be okay as it gets to a thousand requests 10,000 requests whatever the next one is in the sequence you're gonna be less than impressed with what happens we often talk about technical debt but when all you're doing is doing support because you've just got errors coming at you make something up then you know that's meant to lie I start to think of this as sort of operational debt your system is causing you so much headache and pain that regards of how pretty your ruby code is especially when it's got syntax highlighting your life is going to be correct not very fun and so as you get more traffic the requirement for you is that you have to constantly be improving the success rate it's definitely not making it worse which may or may not be what we think about and that's kind of this high level idea so the unfortunate part is point two which I tried to hide in the middle and then pointed it out let's go back I'm gonna hide it more better this alright alright you can't see it now no it doesn't work so in order to get to this we are gonna have to think as your company becomes more successful as your product your thing gets more probably and success is probably traffic you're gonna have to think more and more about how to make that system happy which mathematically well let's call it you know reducing our error rate or increasing our success rate and my hypothesis is that's gonna make you less happy so let's just go over that and let's get on with the rest of the talk but moving on point three a real and really hoping is that the people who are primarily interested in production happiness which as a group aren't here it's their responsibility our responsibility to give you tools that help you do the right thing by the production system and that's that's this from the next frontier of deployment how fast you can ship an app into production it's not very interesting if all you're doing is is keeping the error rates the same if the traffic keeps going up you have to be thinking about how do I keep reducing that traffic and we may have to slow things down a little bit so I just want to break this out into just in case you've never wondered why you're got a job how it is that someone else can afford to give you money to do your profession I just thought to go through it for a bit so suggest there's arrows let's go through the arrows first there's terrible animation I feel like fixing it now so the arrows so we make code just in case you haven't seen this before you make code you put it in a production and if you're lucky you get to be in business never this I don't want to over summarize it but and unfortunately though the priority of your users stakeholders and all that sort of stuff is that you were in business doing a service or a product that's useful and valuable to people they are able to find it they are able to determine this is the thing they want to use at the time that they're having the problem that's that's important otherwise your pretty syntax highlighted Ruby codes are relevant so in order to do that this is the magical step number two you need to have a production system whether you wrote the code or not is an important that your production system could provide the value that to the people that want the value at the time and place that they think they want the value that's important now for most of us we wrote the code yet it's third on the list of important things so awkward nonetheless so if we do number one too well we get to have jobs to get to do number three so you might think well we should all care about all things well the fellow mr. Smith he had this idea that you know amongst other very good ideas yeah three four five wasn't 500 years of four this is old book 300 years ago of division of labor that if we specialize we can be better at our thing and it turns out if our profession wasn't hard enough probably best we do specialize in being developers not doing development and maintaining production systems and trying to make money so I'm not trying to I would like you all to care about production whilst writing code this slide is me admitting that it's tough all right so what I'd rather solve the problem is by and I asked from over summarizing what I'd rather try and solve the problem is like pushing tools towards developers and educating you on choices that will make the production lives better at perhaps a small cost to your developer life so okay this is our most businesses might split themselves out back into those three categories sales people ops people and engineers but here in this is kind of the I guess I already had this slide in my head so I should share it first we all have different fundamental priorities we are paid and we get joy out of adding features curating the app making it look nice doing all the sorts of things and certainly most of our stakeholders think that's what we should be doing but implicit and priority number two is that we have a system that has is up certainly from our users perspective right that's that's kind of what they expect at the time they want to use it it should be there and the business priority is to make users happy which hopefully makes money so hands it up if you think one of these columns is correct actually start again read the chart then we'll play the game and go very well there when I tested this in my head everyone's a lot more responsive all right these these two lists are kind of what everyone thinks about you know it cares about and yeah summarized down I mean when I wrote it down it was a lot longer but I figured a few points and bigger font better than my handwritten scroll and unfortunately I didn't really go well together and we all implicitly know this so this is this is what I'm about to now talk about is what can we do what can we give developers from a deployment perspective comes that's kind of what I care about where it's easy for you to do the right thing than not to choosing J. Ruby turns out to be a little bit hard some reason I don't know you should just use it grow up and but I understand I'll talk about the distinction why am I right what people do one versus the other but this is what I want to talk about I'm going to do a demo of a piece of software that I think is really interesting when you get to scale I'd certainly like to see engineer I'd have a product that's similar to it that's what we're working on and so there's the this is idea that the tools that you've chosen I use tools to represent everything libraries languages tools and they were all built to solve a problem but they come with consequences and if you get too much in love with the good part you perhaps ignore or dismiss sort of in a DHHS fashion the consequences that wasn't a joke he does that all the time it's really good at it and you go yeah yeah that's right we should all own a car that's drives 300 kilometers an hour traffic a little faster he's never said that but I thought it and anyway that's just stupid so yes as I just said every tool has some purpose why it was written and we're going to go through some examples and they come with a thought model that they're sort of imposing on you or hoping you adopt us in order to make the tool make sense and but in in adopting the tool it makes something else harder for example I thought no no I'll come to that example it's in a second so here's an example I like people track things fantastic looks very pretty it's you know this idea of well look you know you've got that that that stakeholder that that owner that says yes we should do x come back three days later let's do why you say that letters of the alphabet that's stupid and see you know and so the solution to some degree is to well let's write them all down let's put them in an ordered list and let's just see where x and y turn up and whether they're going to fit into the budget this is I think you know agile in general I think it's fantastic getting your system into production and finding what errors turn up and fix of them is just the right thing but what's interesting of this tool is there's not really a way of talking about constant things that should do constant things it shouldn't do like crash and another thing of list that I've apparently not put on the slide I mean who doing fricking checking my slides can't just have a list that has one item in a comma I got ad lib this I've done a stupid thing I can't remember the rest of list let me get my notebook out no anyway you know it shouldn't crash it shouldn't destroy small planets it shouldn't it shouldn't make you know customers cry and hate you and talk about your negatively there's a lot of things but it's just as a tool it's hard to sort of mention that and allocate time to doing these things so why do we not do some of the things we're going to talk about and use the tools because perhaps some of the tools don't infer that we should spend time that way so you may need to take the tool subvert it slightly and put in every week spend time as chores let's do some logging looking you know let's look at some logs see what we find let's go back through all our exceptions or four million of them in you know airbreak let's pick out a good one fix it chef chef was built for a purpose it was built by a group of people who had got sick and tired of provisioning you know static service and how quickly and redoing them those servers were there ultimately cloud came along and it became useful then but it couldn't provision anything but it turns out for all the greatness of chef that it's still it's you know you still need to know how to configure my sequel and that's shit so what does chef do it just hid that for a bit so and so what am I saying I'm saying that chef is great but it comes with you know there's it hides things there's still things you still need to do what's another one Ruby that's physically MRI because I do want to make a comment j Ruby and may never quite get off that bandwagon so Ruby was a fantastic language right that's what we're all here and I'm not going to play that game where I put us people your hand up because you're terrible at it um got lost hand putting up privileges stop it right bloody poms um and so all my point before is I mean Ruby was written and with a with a thought model of we should be happy as we go about a profession as we write our code and then someone came up with syntax highlighting and world was complete but it's not really was never really designed or have that thought process around long-running production systems I mean it can't and I no disrespect at all I mean it was that's you know and so but there's also other things that are part of the the community that we have the ecosystem the thought processes we all bring to code and we all agree on code should be pretty and I'll tell you what's ugly handling errors so let's not do that let's let those bad boys go straight up to the top that get very excited about that um and I mean you know I think madame and any mention it I mean I really only thought about this when I saw Blake misery do a talk on go and so compared there's sort of things from go that you could bring to Ruby and one of the ideas and go with the language is dealing with errors immediately and when you start to think about that is exactly what you should do as you accumulate when you start to realize you know exactly why this error is here when you start throwing it up the stack it doesn't know why the exception what the relevance is what it should do about that how to handle it it's completely subverted the context and said you know this base code now you now need to know about what that was doing and that's not very nice encapsulation so dealing with errors locally helps manage error rates and unfortunately it's gonna make you code ugly so this I guess is my prime example of the idea that as you care about production as you want to keep improving your success rates and reducing error rates you're gonna need to do some things that perhaps are not very ruby ish like handle errors so every day put another if statement in I'm becoming a not a fan of exceptions in general unfortunately all the libraries are use exceptions as a form of declaring that something went bad go I kind of like that idea of another let me know it's objective see you can all right I mean it's so yes I should finish that thought and then go back to just the idea of exceptions don't like so yes so the idea as he just said is to put your rescue blocks explicitly in that context there's just still something about exceptions that I'd rather they came through the standard medium of a method which was either that read write parameters or the response so the idea of returning a real value and perhaps an error value if necessary I just think that is it makes how do I put it I actually have a way Blake mentioned it was he said the word exceptional or the word exception makes it sound like exceptions are exceptional they never happen they happen all the time all right we do distributed programming you know the boxes talking to each other the moment you add that second box congratulations you got distributed programming like a database and it's not always there but that's not exceptional you know it's not going to be there sometimes you know it sometimes it will be out and you know that because you put your app on Amazon US East so it's just not all have Amazon's fault no it's your fault listen for that error which will happen again even if you move somewhere else and and make your app behave in an appropriate fashion don't bitch I mean yes you want to fix it and get back up and running but I mean you can't stop physical hardware having problems but your code can handle it all right so that's really early rabbit on about that one for a long time so from the Ruby on Rails website it says web development that doesn't hurt no mention about production so I think I've covered this topic for a while but no actually I haven't let's do it again so I heard a good tip for dealing with DB's as your DB is your sequel database scales you're going to think about shouting one of the things that makes shouting hard is when you do long joins between tables so don't be you know be conscious of of the join as you start to get you know more production orientators start to scope down and think about your app in such a way that you don't have joins across seven tables because you're not gonna be a shard that but rails with its beautiful active record syntax makes it really easy and fun you want to join everything it's like you want to get back in a circle hey I'm back to the user model again I've been a prize not very good for production so platform as a service you know like deployment really easy let you describe all your universe things that I like a lot but not really a lot of assistance in the context of apps talking to each other this idea of perhaps perhaps a platform as a service should make this easier to deal with failure of you know systems that talk to each other so I you know just because you should still use engineered no one else fixes this problem here so just use engineered you just keep sign language to the one person I at least want that person being a customer by the end all right so let's get back to production so there's that was sort of my summary of tools in general thought process just to sort of get you to think about this idea I do want to talk about deployment specifically I just want to scale it down to this one topic so I mentioned before I don't know how to prove or suggest or infer this if your production system is happy you'll be happier so we'll skip it and just go with it so here's a couple of things that you might like to choose which are gonna affect your development life a little bit but at the benefit of having a better production life J Ruby being on the JVM has a whole bunch of really cool tools inspecting getting snapshots of what was going on at the time that's blocked up or whatever obviously the threading but I don't want to talk about ready that's not gonna make you buy anything I found that out last year don't do threading don't I dare you I dare you not to don't yeah I win it's like telling a dog that's already lying down lie stay I'm an excellent owner so but the cost of using JV one of them is that you know all that sort of just day-to-day usage of Ruby that you're scripting usage it gets slower and that's a little bit annoying and so this is sort of that main example their idea that something that's good for production you as developers or let me put this another way all the people around you because you've changed your mind already you know choosing perhaps an inferior production choice perhaps because because we'd rather have the one that's good for you as a developer you want the zippy fast one when you're on your eight task but it may not be the good one for in production so the other reason I mean this is an example if you've never seen a screenshot of visual VM a whole bunch of interesting the introspection of what's going on so as you put that chore on your weekly list of things how we're going to make our system better throwing up visual VM and watching for interesting data is one thing a bunch of other cool tools so cues using cues rather than so blocking htb htb API's are awesome I mean you can use curl and play with it I love it but blocking not good especially with your non concurrent apps that you're all right so so perhaps investigate an asynchronous queuing what else logging there's an excellent especially if you have multiple systems which you already have because you have an app talking with database pulling all those logs together initially will just be ugly and you go well what was the point of that well then go back to the app and start to curate interesting useful logs that tell you a story so perhaps you can search for user and watch what that user was doing throughout the system people so out of all the sort of app service companies that come out recently I'm sorry I started again Splunk is worth a truckload of money Splunk is anyone heard of Splunk cool I did that judge I did actually ask that question she terrible this put your hand up the wrong time and put it up the right time spiteful that's what you are and so I was at Java one and did a quick talk on log stash and ask that same question and no one had heard of Splunk so I saw I just told a joke to them and I'm gonna tell that same joke but I have to set it up and you answered incorrectly with the joke was the reason that you couldn't you don't know about us because you can't afford it Splunk is really really expensive and but log stash and Kibana open source and and really quite interesting for you to look at but there's also paper trail loggily and and use them start putting logs in there start to get a feel for the flow of your system you know you don't really have perhaps in your mind a good mental picture of what how your app behaves and production really and logging is one and watching all the events is one way to do that the trick is obviously you have to spend time looking at them but the bush is something I like I've liked and bush solves this interesting problem of deploying entire systems and knowing what's happening as opposed to you know perhaps the chef mentality of well a node came up and it became something hopefully good but has this more declarative you I'd like to sort of tell Terry and master of the universe I will tell all the VMs what they are and they shall be happy for it and I've started really appreciate the value of the mental the mental thought that's gone into Bosch and for the lack of there being a commercial tool that I my company is produced I will keep talking about this until we have one and I'll have to stop so I'm actually going to do a demo of Bosch because I think it's the thing I would like you to think about as you get bigger as a company this idea of being absolutely declarative not having a reducing the number of external dependencies of your production system things that could go wrong because as we talked about we need to continually reduce those so Bosch may not be for you today we're running a couple of dinos on heroku if you've got a couple of VMs with engine yard you're good right but as you get bigger and you want to you know you've got more and more requests you know this is one of the things perhaps to think about so let's let's have a look so what the example is get lab HQ which is I chose because it's kind of a rails app it's got a few moving parts in fact here's my my my architectural diagram it's a rails app right within also it's got get get a light which is does that sort of the get stuff and so we're going to deploy this as a sort of a complex system and it's gonna be awesome look even found an icon that makes you want to click it but stay seated everyone don't fall for the trap of the large touchscreen monitor all right so this this I'm not ever going to pretend to type for 10 minutes look no hands so what we're going to do is we're going to deploy this so I have already written a description of how you deploy get lab HQ and it's going to Bosch release now we can look at it for anyone that's interested if we have time we can look at it but it has the source code to get lab HQ the source to get a light sub module all the other dependencies you'll see and now being downloaded now they're not being downloaded from some magical other place I've already downloaded them once from the magical other place and there are my S3 account because we're trying to reduce external things that could go wrong and and once I pull them all down and we create a Bosch release and put it into my Bosch that's it that's the last time we go out to the rest of the world for the rest of that apps life unless we want more dependencies like you know a new version of Ruby or something I haven't touched this for a while new version of Postgres anyway let's move on stop looking at version numbers I feel like I want to type something but so fake I should pretend oh fuck anyway you can make screencast that go wrong really fast and don't have the inconvenient and awkward pauses but then you don't learn anything so we'll have the inconvenient awkward pauses so what it's doing now it's downloaded all the assets that I need because we're not going to use app get or you know those packages because they're external that you could build this system using that concept of you just have your own you know app get repo and so the Bosch does it all itself so now what we're going to do is I can't remember we're gonna is that me sweet I'll be back Jesus Christ okay so what we've done the context here is we're actually on not my machine but just a VM lot faster if you do it in the cloud and so we pull all these things down from S3 with turn them into a sort of a big table and now what we're doing is we're uploading it to Bosch Bosch is not us like a command line just a pure command line it's a running service bit like a pads might be and so what we're doing is we're uploading it now Bosch has the big tube of t-shirts what stop it distracting me because I'm packing those things now this time what we have is so it now knows about a release called GitLab HQ bit and I've it's there 715 I mean 7 sorry 7 to 7.1 dev so now what we need to be able to do is deploy it so this of these two main concepts of release and deployment manifest deployment manifest is that is awkward alright I keep thinking that's something that's awkward when you're hearing it from me so a deployment manifest is a big yaml file which is great because it's text and you can read it awkward because you know I really would rather have that much like love yaml sometimes I wish there was a schema I could validate it against and you know potion shouldn't say that out loud but I want XML back so because I really talked through that what we're doing now jobs the different moving parts or sort of the most but you can think of them as one VM's worth of work you can merge them into the same VM but that's kind of the context and so you can sort of see them as those boxes I had get light redness GitLab and rescue which isn't you know sorry Mike I can't believe I moved to sidekick I'm in soldier on your behalf and and then the last part which we skipped over is all the sort of the parameters the arguments they're gonna go into all the templates so they're all in one place like a data bag if you're doing chef and I'm not really a puppet person so if you want to sort of tell me about puppet relative to this that'd be great so now we've just told the command line tool this is the point manifest of what I care about and the way Bosch does deployment is it sort of says well what have you already got what do you want and I'll go get that for you the first time you've got nothing so it looks like it does everything well he does right so first thing it's going to do is compile stuff this I think this is fantastic it comes built in with its own packaging binary packaging thing against whatever the base operating system that you have got so in this case it's some version of Ubuntu but it's your version of Ubuntu so there's no chance of packages having been built on a slightly different environment being applied to yours they're gonna be built in exactly the same environment so as they're compiling here this is my Amazon account you can see there's four VMs available for compilation it finished and we moved on that was terrible so what we're doing now is while it's running because one of the package was a pearl and whoever built pearl had a lot of spare time it takes like 30 something minutes to build on an M1 medium so this is a bit of Bosch's tooling you can see what you know processes are running you can go on from another machine you can go and watch that process just as if you were doing it when you did it yourself so teams can watch a deployment that's kind of interesting nearly every action is a rescue job so to speak and you can watch it and also all the log information for that is kept and you can go and get it so there's lots of different ways to look at tasks let's just take a moment to appreciate that I didn't make you watch Pearl being compiled for 30 minutes are you getting why I did a screencast yeah stuff takes forever one of the things we have to give up so this sort of just another way of looking at that data so this is raw data that you could perhaps build a nicer interface on top of should you wish to certainly one of the things I've been working on for mode amusement all right so now we've finished compiling packages now we're gonna boot up some VMs why we're booting VMs because that's where stuff runs and that's what we put in the deployment manifest we go back to it you'd get to see that there we had five VMs one for each of the different jobs Bosch jobs and so now we're booting them up God this takes forever even when you cut it all out and make it faster I'm so impatient and there's really no funny jokes to tell about VMs booting your patience like you get the cannon there's a squad of three and a decoy squad go so it takes a little while but eventually you know GitLab is running and so I think this is really interesting I was able to deploy yeah I mean it's the manifest is a bit icky but at least you know it's an interface you could build a tool that made it easy to work with the YAML manifest this is the VMs running you can see on the right four of them have elastic IPs assigned to them that does Bosch now does have an internal DNS so I didn't really need all those elastic IPs anymore but I haven't figured out how to make that work yet and so now what we're going to do is we're going to change it the deployment in this case there's a deploy in Boston language and certainly why I think about it is it could be changing scale attributes bigger VMs more VMs it could be changing configuration or it could be changing the release so we may have cut out a new release of some software which might be a new version of Postgres might be a new version of your web app new version of one of the things that's part of your system Bosch doesn't think of your Ruby code as all that special just just to get the gist of it your Ruby code is just a thing that runs so here we're going to add an extra rescue thing we've got we've you know somehow the rescue workers weren't very productive and so the solution for rescue is to just add more resources or use sidekick and I'm sorry Mike for not putting a sidekick slide in there please in your mind put in a sidekick slide moved a sidekick it's good so you can see a sort of a prompted this is the Delta do you want me to do this yeah good and off it goes adding a new VM turning into rescue and Bob's your uncle it does take a lot longer than this now we get the extra VM look perhaps this doesn't impress you but if you have to manage your own VMs this is awesome the other tool it has is built in SSH so it will go off to that VM create a random username account for the purpose of that one session you give it a password for this session for sudo so you can change it each time if you want it doesn't do auditing which is a nice sort of you know what did that person do whilst on the VM better be nice feature doesn't have that yet so you can see that random username and what we're going to do is just look at the process list just to see why you know we still haven't quite go enough rescue workers oh look there's only one of them so whichever genius wrote this and me didn't put in enough rescue workers so now let's look at some some of a little bit of how this works it's all on my github repo if you want to play with it have a look around so this instead of chef it's sort of it's shell script shells actually kind of good when you you know get over it so here I'm running rake and you can see I've only got one of them oh and look there's a fix me to add more workers so anyway I'm a genius and that's that's that right I find this the ability to describe everything exactly is just removes all the possible things that go wrong it makes it makes understanding the system simpler and the less things I have to think about the less the chance I'm going to make a mistake somewhere there may be other things that are similar and again you know it's hard to play with everything and if you have a tool or a set of tool chain that fits into a similar model I'd love to talk to you about it because I think this stuff is fascinating I mentioned this before but your Ruby app really is is is just another process it's not special when it comes to ops it's special in that it borks all the time or blots or does all sorts of other aberrant behavior but I'll be on that so the way this sort of works is you know it's just processes and we're running on VMs a boss release is fundamentally two things set of job descriptions which as you saw is sort of it's shell scripts because they're kind of easy to write and templates so chef as an example you programmatically say whether you want a template or not your program actors say where you want a package or not your program actually say what you know what to do next chef with boss you know none of that you just in a gamble file say what packages you want and I'll be there you render all the templates and why not and then you show scripts to generate everything else through start stop on monitor packages as we sort of talked about you go and get all the source code for yourself you do your own installation scripts they're all easy to do and once you've done them you never do them again so that compilation step never happens again so we deserve these nice things and you can read this load later I do want as much as I like Bosch there if it's not you know perfect by any means and why what's the highlight there you know getting started and you're gonna go run your own Bosch so if you're you know don't have a big budget for running Amazon VMs I don't have a v sphere account or whatever you know it sounds expensive so it's not for everyone can't run a bare metal at the moment because it wants to manage VMs and attach discs and all that sort of cool stuff and it's sort of single region so if you want multi region you have to have separate washes but it is an idea if you've described everything absolutely and have no external dependencies theoretically you might be able to answer this question with a different answer when that fancy enterprise person says what this looks fantastic can I run it in my data center with this kind of tool the idea here perhaps you could say yes even if it's completely locked off from you I think that's that's an interesting idea of the future of deployment people have talked to people they say no I never want to sell to enterprise I you know it's hard enough running this one thing well what if it wasn't hard to run your one thing what if deployment and management production systems made it as easy to run a thousand copies as it was to run one so the summary of my talk really is this math is as you add more traffic you need to constantly be improving your error rates or your success rates and that you are going to need to change some of your behaviors as you go along and you I don't want to say you're gonna have to grow up except can't think of another way of putting it so what I think the job of ops people in general or the sort of people who care about production systems is to be constantly giving tools to us developers the engineers that help us do the right thing that Bosch deploy anyone could do that once the packaging is done I mean I could figure it out and I'm not very bright so thank you very much now the last thing is we are hiring because we still have money left and I mean this is whilst we're not doing Bosch we're doing something very similar as it's really exciting stuff if this is stuff tickles your fancy and you'd like to create the new frontier of tooling for production system please make sure you come and work for us we'd love to have you we've got the party on tonight here are some snapshots of people you should look for and talk to us about how much alcohol is awesome and whether we'd like you to come and work for us vice versa so thank you very much everyone