 Video equipment rental costs paid for by peep code screencasts Okay. All right. Is this that can everybody hear me any technical problems before I start? All right Well, I want to start out my presentation. I like to jump right in Just get right into the thick of things This is one of the Tools that I'm going to talk about This is a part of a god.rb watch configuration file and I'm going to return to that in a little while So I work I'm the co-founder of other inbox, which is a startup based here in Austin And pretty much exactly one year ago. We were Running a prototype of our app was rails app running on a tiger Server that we just happened to have on a run in the office. It was up on a breadboard That actually worked pretty well for a long time We've got a couple of dozen early early beta testers using it giving us feedback But obviously in the long run that that isn't any that isn't what we wanted to be running our whole app on so today We were running in the cloud and this was my this is how I'm visualizing This is meant to represent the excitement of cloud computing and how distributed and wonderful it is and We're the super hip developers and we could be on a hay bale working on our app, and it's totally fine We're not tied to some server room So, you know, but more seriously We're really able to take advantage of running our app on a cloud platform We can really think about our infrastructure completely differently Our servers are disposable. It's easy for us to experiment You know when we do major changes to our database We can launch two or three different copies of the database and run the migration at the same time So if it fails on one if it's something that's long-running, we're not totally totally stuck and There's a lot of different ways we can distribute work between our servers But how do we go from that one server that just we just happen to have that we could get up and running quickly To being to working in the cloud Well, it took me, you know a few hours to get the servers built if you're whatever cloud platform you choose There's usually a lot of pre-built images that have most of the tools you need So really isn't that difficult for us we use EC2 There's a ton of images out there that contain most of what you need and the rest of the time You can customize it to your own needs So it probably took me a few hours to build the servers It only took me like maybe 20 30 minutes to move the app over, you know I have a little bit of cognitive dissonance because I'm About nine months having passed having done it So I'm probably being a little optimistic, but I don't remember this part taking much time at all Moving from one whole ecosystem running on this one machine to a whole ecosystem that is running on any number of machines, you know the That scales itself and is very Highly distributed that only only took 30 minutes to get that kind of up and running the rest of the time I was working on building new features You know taking advantage of being in the cloud you be having this flexible environment Being able to use disposable servers being able to spin things up when there was more work to do and shut them Down when there isn't So that's what I'm going to talk about Why how why was it so easy for us to transition to this cloud environment? What are some of the things available to you to help you take take advantage of it? I think it does tells pretty well with the last talk You know Where I'm going to show you how maybe a little bit lower level of detail Not not using jet not using third-party or your own gems, but using some third-party tools to help you manage your systems So any kind of talk like this There's a you have to we need to define our terms because cloud computing is now like web 2.0 It kind of can mean whatever you want depending on what you're trying to sell people So let's just for the sake of argument for this talk agree. This is what I personally Mike Zabelski mean by cloud computing All the way to all we're talking about is Servers running your code that you consume those servers as a service It means you rent them by the hour. You don't own any physical hardware. You don't have a You're not responsible for providing power to them It's just something that you can count on depending on whatever level of service agreement you have with the the provider But it also means that your servers are loosely coupled. You have code running at different places It's not you know You're not necessarily going to know who you're talking to ahead of time You don't have a whole lot of information about each side of a connection It's distributed. So like for us We have essential the essential parts of our system are not all running in the same place within Amazon's Computing cloud so but we don't really care exactly where the stuff is running We just want to make sure there's enough Segmentation so that they have a problem in one place Nothing fails in other inbox the service keeps running on and also it's a single point of access So wherever in the world our data is stored or the code is running It doesn't matter to us because we have this all we had to do is SSH into an IP address And it's there all we have to do is set up our DNS and the users can find the server So cloud computing it's grid like if you're more comfortable thinking of it that way Just being this grid of servers that all talk to each other It's utility like because you buy this stuff by the hour the cheapest instance But we use is like ten cents an hour I you know, I don't know the pricing models that Heroku and engine are engineered to use But it's the same kind of thing where you're you're not like stuck with this Device that you just have to own and maintain you're buying it by the hour and you know the main thing is that it's the clock computing it means you're cheap redundant scalable and flexible But maybe a little too flexible so some of the things that You might not Have had to deal with when you just were dealing with this one server that you could if you had to hard reset Or that you could actually you know connect a monitor and keyboard to actually get to you're gonna confront some of these issues I'm putting up here. I'm not gonna This beyond the scope of my talk to talk about sort of like lessons learned probably because I actually think I'm still learning the lessons of How to build an app and deploy it in a cloud? But what I'm trying to say here is that It's great being will turn all these servers on and off and have them auto-scale and communicate with each other but that adds a lot of complexity and Ruby can help you with that complexity I want to share with you how we're using Ruby to help us manage the complexity of of being in a cloud So to start with here's all I was thinking about it We really use Ruby as the code that glues other Ruby code together But I feel that that really underestimates Like it even that doesn't really capture it because you might as well you could just use pearl for that You know It's that implies it will basically it's just a bunch of scripts that kind of hang together And it just makes it so we don't have to type in as many keystrokes, but that's not what I'm saying at all more than that We really have a ruby ecosystem Where it's ruby code that's animating ruby code the connective tissue of other inbox is ruby You know, we're not but we're not like dogmatic about it If there's some other better thing that comes along we'll definitely use it. I'm just telling you For me working on this for one year. I was only developer for maybe six or seven months I was able to get a whole lot done and not spend a lot of time on sysadmin stuff Just by by taking this approach. So I do want to throw that in there like I hope nobody's an IRC right now saying oh my god This guy's a freak. He totally disrespects all their languages. I don't mean that at all I'm just trying to show you some of the benefits that that we've derived. So what am I talking about? What kind of what does this mean a ruby ecosystem? I mean Ruby code that is scheduling the execution of other ruby code. This is an example from a gem I'll talk about later called Rufus scheduler Ruby code that is monitoring other ruby code. This is God again, which I'll talk about Ruby code that kills other ruby code. This is a task from Capistrano Actually, if you want to get technical, this is a Capistrano task. That's calling God RGB So it's like two layers of ruby connective tissue ruby code that launches your servers. So We right now have a homegrown script that we use to bring up new servers Which I actually wouldn't recommend in the future that since I wrote this there's been a lot better solutions and you know I'm going to show you though this full thing of code and it's just kind of one of those scripts It just got longer and longer So I'm not super proud of this but I'm putting it out there, you know I was able to write it really fast and that's why it didn't take me very long to get us up and running on a new platform Ruby code that Configures your servers. So when part of the launch process that we use Is we use the DD client to talk to DIN DNS so that when we launch one of these new servers It's able to get the proper domain name. Whatever domain name we want associated with that machine We can go get it because that's not something you get with Amazon although You can get some static IP addresses, but you're always going to need some kind of dynamic DNS if you want to do this at scale at all so that's this what I'm saying it's ruby that provides a Blanket of freedom and security that we as ruby developers can rise and sleep under All right. I wasn't sure if anybody was going to get that. I have a greater responsibility than you could possibly fathom Okay All right. So why? But why would you do this? If I was in the IRC channel right now, I'd probably be saying like well, doesn't this guy know about Monit or you know Launch D's the bomb Whatever You know or some of this probably I'm sure there's like a hundred things that are even better than what I'm going to show you Why would you want to make everything in your system all ruby all the time? Well, I think for any one of these of cases you can definitely make the case There's probably some better thing out there that's been around longer that you know the original Unix developers wrote and it's awesome So like I'm not saying don't don't do that. Although I'm not really super fan of launch D So the reason why you would go this way is going back to I said earlier about complexity all those different Systems They all configured a different way you have to use you to think about them differently. They have a different syntax And I'll show you some examples of that. Yeah, and if you're doing cloud computing, right? You should it's not just like the exact same thing that you were getting from your traditional conventional ISP It means you are Taking advantage of the loose coupling and you have one machine what you know some code running one place That's giving tasks putting them on a queue for some other Server to pull off and start working on and you got your storage is all distributed So that's that's part of what has is a complexity. I alluded to earlier The fact that the servers are disposable means you got to start keeping track of them You you know that it's easy when you have these five boxes You know where they are you can get your hands on them You don't have that anymore or your ISP could get their hands on them Now you could just start starting these up So this is a snippet from the we use the right scales free management tool to kind of to help us with this part of it Which is a rails app by the way? So like look at the names of some of those their purpose launched They only exist to do a test for somebody named Ian You know one of them doesn't even have a name yet. Who knows what that is Let's just say one example of the kinds of complexity you can find with all this newfound power So Ruby can help you reduce that complexity. That's what I'm here to talk about So let's talk about some of the reasons. How does that work? Well, first of all We all know this I don't think you'd be at this conference if you didn't love Ruby and its expressiveness So you can just say things more clearly more powerfully in Ruby than you can and in some of those other systems so What's what's more readable to you if you were a non programmer and you had to wait in and try to figure out what was going on No non Ruby person or just yeah, not say non developer non sys admin. Would you rather read this or that? You know I can never remember it's like I seriously can't I can't tell you Does that mean it's going to do every five minutes or at five o'clock? I don't know I don't want to know I don't have to do that anymore. What was that? Five a.m. Every day. Well see look at that. I might have written that wrong, but there's no there's no doubt what that means Okay scheduled every five minutes. There's a whole bunch of ways that you can tell Rufus to what time you want things to happen I'll show you more examples All right, so what else what else is so great about Ruby for these kinds of tasks? Well This is not something that I had anticipated, but it gives you platform independence. So as long as you have this All a lot of these tools you're depending on probably will work, you know I mean something like God rb has native C extensions you this it's obviously this oversimplifies it But it means that you know, you're not tied to any one particular Implementation of all those tools. I showed you before now this pretty much give or take is going to run everywhere It's going to have a better chance of running everywhere than all those individual systems that are all written a little bit differently another example of platform independence You don't have to write shell commands There's stuff in the standard library that will save you from having to use back ticks on some of the code I'm going to show you I didn't do that. I kind of so busy trying to figure out how to launch things. I just Defaulted to my knowledge of shell scripting, but if I had to rewrite it I would I would use all file utilities. So then it doesn't matter at all what platform I can feel really confident that if Ruby compiles there, it's probably going to work Who does anybody recognize the syntax in the middle here? This is from a really cool gem called rush Ruby shell and it It basically makes it takes a different approach to the standard library. It's not very object oriented You might have noticed rush gives you it's like an object oriented wrapper It does all the same things that file and file utils do So obviously the latter one is more readable But they're both more readable than having to try to find a shell script that might work in one place And might might not work another one thing that rush does really well As it gives you more powerful globbing and file matching things like that Makes it just makes more sense Okay, another reason why it's this is such a great idea and why Ruby has really helped other inbox Is that this that means you have testable code? You can write test to figure out if your Connective tissue if this glue that's holding your whole app together actually works That was something that they alluded to in the in the last presentation So this really isn't testable is it? How do you know that it runs at 5 a.m. You did I I'm sure a lot of people here How many people have ever fat finger to cron job before? Yeah, I mean look at that. It's just you know, it's it's great when you know I'm sure like I've used gone over that's fine, but like you know, I don't does that supposed to be a star I don't remember. I don't use it anymore instead. I use that and So you can write a test that proves you're you're scheduling this the right way By the way, I also I definitely I feel we've been very much saved by this approach where we put as little as possible in the glue code Like every rate task we write every roof is scheduled job with a few exceptions They all just call a method on a class because then you can unit test that class much more easily it's separated from this scheduling code and I'm finding that if I ever writing something like this I almost always want to be able to execute it from the console at some arbitrary time of my choosing or just to exercise it So if you end up you should start scheduling long lists of tasks in there It just gets kind of messy exception handlings were difficult logging problems, etc. All right So I feel like I could almost give a whole talk on this but Less context switching what do I mean by that? Have you ever been working really hard on the JavaScript part of your app the stuff that runs inside of a browser or? You know you have a piece. That's Java and you get your search engine is written in Java Or what what have you and then you have to go back to Ruby or you've been doing all Ruby And then you got to go make a change in the early part of your app and it's like you just You just feel stupid and all fumbling you're putting semicons in the wrong place And your your variables are camel case when it really doesn't look right. They should all be underscored whatever same thing applies here if I find that It's like especially when I'm doing things with the God RB which is it actually I don't want to get ahead of my slides here but I just find that I Can stay in Ruby the longer I can stay in one context one language environment the better So if I'm writing a Ruby application that is being monitored killed destroyed launched What have you in Ruby? It's so much easier for me to just jump over to that configuration file Change a few things to get right back into my code Also The if you treat this some of this glue like it's really part of your application It's a lot easier to get it under version control So everything all the stuff that's critical for our app all our config files All all out of these examples. I'm showing you it's all in subversion so we can go back in time All the benefits that come with version control we have Something else that would you know, we're just starting to see is that There's a little bit more of a line on alliance between the people who the network operations folks and the system administrators because They're working on all speaking the same language I mean it does require that you find the people have some facility with Ruby that can make your hiring more difficult But you know, I would say Ruby is a pretty easy to learn language. It's pretty great And it's you know the whole point of what I'm showing you is you can write this stuff in a way That's easy to understand and read so it's not that big of a barrier and what you end up with is developers who really understand the environment and People who are in charge of the environment who really understand the code and people can do a little bit of both I think there's a lot of benefit so not keeping those people in silos like the example He just gave about how he had to give his code to this other System administrator and then walk away that that doesn't see that that wouldn't be good for us as a startup trying to be nimble And just I also would like to say that you can be very you just already know that in this language You can be very productive and you could spend, you know more time on Features and things like that and you can because you're of all these other benefits that accrue to you You could just get more done. You don't have to wrestle with the configuration file all day okay So how about some let's let's get some some red meat here some useful tools Oh, my There we go. All right. Sorry about that. So there's some of the things I want to talk about not in a super lot of detail But I just wanted to Wet your palate to get you excited about some of the possibilities out there. All right, so this is my favorite one That's why I put it first. How many people have heard of God rb before? Yeah, all right, so this is really great for monitoring processes on your on your server and Taking action so it's it has a reporting capability It could send you an email something goes wrong But the real value of it is that if it sees Your your mom girls die or if it sees one of your your daemons go down It knows how to restart it and even better you can you can be very fine-grained about it Only try to restore this thing five times and then stop so you don't get a flapping condition. So here's what one looks like This is the like the canonical example is using God to monitor an application server like mongrel This is a watch condition and you could see a lot of Familiar Ruby help here like I'm using an iterator over a bunch of arrays So I only have to repeat this configuration block one time for four different mongrel instances And I've got this start command there stop command restart And how often how long do I want to wait? What's my interval before I start monitoring something? So I know it takes about 10 seconds to get everything totally up and running I'm not going to start checking to see if it's running for 10 seconds. So The syntax can be a little bit hard to understand at first The bottom one here saying all right if this process that you're monitoring is not running I want you to start it If and at the top here if it starts to use more than this many megs of memory something's wrong I want you to restart it must be some crazy image magic thing going on there You know if your CPU usage goes up. I want you to restart it and here's that Flapping one that I just talked about the life cycle So if you start to have problems more than three out of the last five times you check We're just gonna stop monitoring this we're gonna leave it turned off And we're gonna send out the notification to the warnings alias So something else that I use a lot. I don't find a lot of people talking about but I just think is the coolest Jam ever is Rufus scheduler. So this is an example of how Rufus scheduler works It's from the documentation. I like this because it shows you all the different ways you can express Time it understands all the different formats that that you might want to use You can even you can pass call into it if you're most comfortable with that But this is This is not the method that I recommend Because it's harder to test what you're actually doing in your scheduler instead. Like I said just Tell tell Rufus schedule what time you want something to happen or how often you want it to happen And then have just have it call one of your when your task and maybe I wrap a little bit of logging around it all right So there's a couple of different gems like if you happen to be running on on the Amazon platform There's a couple different gems you can use. This is the one that I think works the best It's put out by right scale. I have no affiliation with them. I just think it's a cool jam And it lets you manage this stuff that your queues your servers your storage has all the different API's you need All in one namespace and works pretty well, and it means that most of that stuff I just showed you really boils down to this all right find the queue that has this name I want you to put a message onto that queue There's also a get that which takes me just off the queue, you know And often the pattern is that you are putting a file name into into your queue and the file name is something that's in Your bulk storage in s3, so it's not much more code than that You can you could probably you could write an app that is Sending tasks back and forth to ask us in just a few minutes if you know you're doing if you have a gem like this All right a lot of people most people know about Capistrano There's also this alternative called Vlad the Deployer. How many people have heard of Vlad the Deployer? I Actually kind of like it better It's a simpler and it uses more native things in your environment because a lot of things people wrestle with with Capistrano can boils down to problems with SSH connections and your keys being wrong and also Vlad uses the Rake so it doesn't have its own different taskings if you already know right you can do Vlad and it has a really cool logo That's cool. So I think I'm gonna you may have like a Rate of lad task file or deployment file looks as simple as that You might not need more than that that last command is what actually will deploy your app And you can make it more complicated than that, but that's usually all you need So that's an example of Vlad all right But also you don't necessarily most of the time or often you're not gonna need something else someone else wrote you just get You are gonna need to write some of that glue code some of that like roll up your sleeves and like and let's get this done so Actually want to show you a little bit of that so we can get out of slide land for a second Okay, I had this size for different resolutions that will work Oh Think preview there. Sorry Losing control here. Okay, so here's an example of a plain old script Obviously, you know if I had to write this again today I probably I would do it totally differently One of the problems that I found is that I put too much into this script I'll explain that in a second. So this is launch.rb Launch.rb is as a wrapper for the command line tools that Amazon gives you to launch your instances And just means that I don't instead of us having to remember all the time What are our image ideas? You know to get the instance up and running What image goes with what server? We just have this one wrapper right here that does everything You know complains if you don't tell it the right parameters and things like that So the whole point of launch.rb though is where we wrap this Command line tool and we also make a small zip file that is available to the instance at the time that it launches And that's called we go at the payload the payload has everything needed to Configure this server from scratch. So we have this disk image That's very large and hard to change and that just represents the raw state of one of our servers But we can make that server become whatever we want So like a couple of the tasks that what takes care of is assigning the host name So that as soon as your server comes alive, it knows what you want to call it It knows what domain you want it in because now you can launch the staging environment Which looks exactly like your production environment, but has a different domain name You can just have all this taken care of Plus all other configuration files that you want to have present right at launch time So we zip it up. We figure out what arguments we want to pass to the command line Then we launch it. So that's the command what we're doing right here down here ec2 run instances Type what group you want it in and this is the whole point of this exercise is this payload file name down here That's what contains the stuff to provision the server. So the way it actually works is when I'm sure this is you get something similar in other cloud platforms, but in Amazon we have these Linux boxes and The they have RC local files that work So it's the last script that gets run as this as the server is being Booted the last thing that happens is RC local runs RC local looks for that payload file Unzips it and it looks for a file called auto run rb And it just does whatever's in auto run rb So the thing that I would change is there's tons of stuff in here That I don't think really belongs in here auto run should just be like the bare minimum needed so that you can communicate with the server So putting an SSH key on there so that your team can get to it You know giving it a name stuff like that like I'm making whatever changes you need since you built that solid disk image That's the basis of the server. So Unfortunately, that's this what we have I haven't had time to fix it But this gives you an idea of how such a system might work What I would change is that instead of doing this in auto run rb I would make a Vlad recipe or a cabastrano recipe that would do this after launch time So there would be like get there be this launch script It's very simple very lightweight and then there would be a cabastrano task called provision That puts up all the different configuration files and make sure all the packages are present that you need And then I would have like a separate deploy task that would work just like any other app All right, so I'm looking at code. All right There's a couple of other gems that I know I haven't used but I hear good things about that I wouldn't want you to be aware of So pool party is in real active development right now And they don't have any other examples that I could see except this one, but this I think this is very intriguing I would be very interested to have this kind of power to Bring up my servers and provision them This sounds like a lot better way to go than trying to write my own thing If I can if I can use something else and continue to work at an even higher and higher level to manage my servers So that I can spend more time getting features out the door. I think that would be great I've already talked a little bit about rush not only something I've played with but I think it would Really change the way that I deal with the operating system I feel like a lot of the problems that we've wrestled with as our team has gotten bigger and more people have worked that This is not everyone's familiar with what files should go where and you know, we This is the part where we could use Ruby's expressiveness a lot more And I think a tool like like rush could do that There's also a gem called pal which has a similar Parts of it are very similar to rush, but this just looks nice to read. This is more intention revealing to me This is more. This is just more fun than writing a bunch of back ticks And I think you would this will get you you up and running faster, which is the whole point of doing this Okay, and let's not forget. There's there's workhorses of the Ruby world that I Would you know, I wish I had taken more advantage of earlier in this process There's there's rake which has all kinds of things you need past dependencies All you know, you're most of us have used rake that it's can be very helpful for managing complexity Think about using a rake task before you just write a random one-off script I think maybe that launch thing I use that probably should be a rake test also And the file and file details give you a good abstraction layer on top of the operating system because of the importance of Which helps with the importance of being platform independent like I talked about like we're thinking about maybe moving through a different operating system Using a different disk image to to base her system on and to me It's like I don't even care because I know as long as all this stuff compiles and we could test it You know we get Ruby interpreter and our gems work pretty well. It's going to be fine Okay, there's a couple other gems that are specific for Managing cloud stuff and you geared towards EC2. There's the rubber gem. There's elastic rails. There's Ruby works I don't know a lot about them, but they're definitely Things that can help you manage a cloud and you know, they're free unlike some of the other Services that you have to pay for so I would give these a serious look if you're considering going this way Here's some things that I would like to see that I haven't come across that We don't use a Ruby for or we just don't have right now Being able to configure a firewall like I wish there was something like adhesion for firewalls And maybe you could even use adhesion for that. I wish there was a Ruby health monitoring system and log analysis I mean there there is I know there are I'm aware that there are things that will analyze logs But something that's a little bit more like that you could monitor my entire clouds logs in Ruby and Backup that's something else that we have a shell script for it. It's just like I just want to get all under ruby okay I've talked already about some of my lessons learned what I would do differently I would I would use rush or make heavier use of the standard library I would write more unit tests for that glue now that I've seen how easy it is to write It's like it's not that much more work to write some tests for it for something That's so important to the functioning of the app and I would do as little as possible at that launch time because When there's a problem when you have to go look at that log file the console log It's really hard to debug why didn't your auto run script fire? I feel like I spent too much time on that one. I should have been trying to make that as thin as possible But it's just me not understanding the what was how often we want to be launching these things and changing the purpose of the server Midway without having to relaunch it etc Does anybody have any questions? We have a little bit of time left We don't we have we are we have a firewall on the on the hosts It's a that's stateful, but also for at least for us and Amazon gives you a packet filtering firewall as well So it's really easy to get something That kind of blocks the low hanging fruit like okay I don't want to you can block whole ports out But if you want to protect yourself from a sin flood those kinds of things on the ports that you do allow And you need you will need to install something yourself, and I imagine that's the same for all the all the providers Yeah, so there are there are companies that put a management layer on top of of You see to or there's just companies like engine yard where you could just buy the whole thing includes a management layer I think I think we're getting the point. We have to do that. We have I would say If you get beyond six or seven servers, it just starts to be a little bit Be really nice to have some of that we feel like we are starting to outgrow some of the like the hand-rolled ruby scripts that I've been using and Like the event also some of these companies have pre-built components that they you know They can help you configure like a load balancer. So yeah, I'm definitely excited to try that out Do you mean like making sure the storage is like retaining what's stored on the server? You can kill them you could yeah, it's easy to terminate them. There's I showed you the EC2 launch There's also a terminate command. I'm sure the other providers work similarly So the the the question was how do you manage DNS when you you don't necessarily know what the IP address of the server is going to be those two two-fold answer one at least for We use for Amazon they give you five static addresses now so You could do a lot with just those five static addresses and you can easily reassign them to different instances And then we also use a dynamic DNS company that let so that you know We had the servers can tell the the DNS provider, you know, okay. I'm now Alpha one or I'm now mail or whatever or whatever we want. So we use there's a that actually is not Ruby That's DD client is the name of it, but that's just that's like the industry standard apparently so no reason to reinvent that wheel shoot We've we're very happy with the cost of it. I was not involved in like the the decision to compare this against other hosting solutions we mainly picked it because we wanted to be flexible and Have disposable servers and be able to Launch a staging environment and play around and then shut it off and not affecting the production database So really wasn't wasn't mainly a cost thing, but it it seems doesn't seem overly expensive to us I don't think it's as cheap as some people think it is. I mean it kind of can add up because So if you're running the servers 24 hours a day and you need beyond the most minimal basic kind of server it can get expensive But I don't think I think it's competitive, but I don't but I'm not I wasn't involved in that much detail Yeah Yes Yes, so They just added it used to be that you had ephemeral stores Whatever you wrote to a disk device you would lose when your server died Amazon just added that and we have started moving some of our servers over to it It's pretty cool So we've been up We've been up on it for More than the nine months or so and we've had one just recently They emailed us to say hey the server that your Instance running on is not looking good. You might want to move migrated over and so we haven't any problems with it But you definitely put one of the things I didn't get into in the talk, but that you have to think about is You don't have that much confidence in any one server See if they build a lot of redundancy in so like my sequel master slave relationships What are clustering? I mean there's a lot of there's a lot of different ways you can do it But that's one of the things you are gonna pay for is like, you know what each any one instance They're cheap, but they're also not as reliable as you might like No, and we have I Mean we're other much it has to do with email So that's that's not a that not much I could say so it's very it has to be super reliable So we have never had one of our No, no, but we had the mail servers running for like a hundred fifty days and one of them Had a problem, but because that's so important. We have two of them Our DNS is set up so that the other mail the other server picked up the mail and absolutely no interruptions service whatsoever It's in a different availability zone So even if some disaster befalls one zone, we'll still get your mail. You know, we'll ever be the wiser Yes, we were but we built our system was built in a way that if any of this stuff isn't available any of this cloud magic We just build up the whatever work like if you're making a site that There's image processing for someone you should build it in a way that assumes that the queue is not always gonna be available That like you might have to store a batch of work on the local machine You know like I guess in that case it would be people upload Gigantic image file to you and then you turn around and push it into Some kind of lock storage cloud like s3 and then you so another server can pull it off Well, you should assume that some percentage of the time you're not gonna be able to do that right away And so our stuff is written to be as dumb as possible So I really I just have a loop that like looks for any messages that have not been pushed up into the cloud And if it sees one it just puts it up there So that means there's some kind of IO error or yeah, there's an outage That loop will just keep running keep trying to do the job until it succeeds And that's also the way that they Taking things off the queue works the same way So if you take something off of an SQS queue, you can say if I don't report back to you in 10 minutes You should assume I didn't do this work And so then if there's a problem on that end of it if your server takes a job off the queue and then it fails 10 minutes later, it'll be available for another server. So I just feel like That's just one of the things I've had to really change my thinking about it so not to not count on the reliability of the servers, but also to That's a little bit freeing in a way Like you can make things that are very loosely coupled it kind of forces you to do that And then it hasn't been a problem Believe me, I wouldn't be up there talking so enthusiastically about it if like it's definitely made my life easier Well, this is one of that's one of the things that I can't get into too much detail on that part of our architecture but there are a lot of like What I would point you to is the the blog that right scale puts out has really awesome explanations of how to set all that up and how you can have your slave can be Almost exactly up to date with your master I mean, you know if you're gonna make a e-commerce site There's you know banking transactions, you know, this might not might not want to put your database in this environment unless you really know what you're doing But you know we We don't put everything in the database would be my answer to it We keep all we keep a copy of all the critical data in text format other places That's just because because of the kinds of problems that were solved. So I'm not trying to be a jerk. It's just that's all I can say about it I mean people have found a way to make it work and that's that's the resource I would point you to because it's it's fascinating to me to read about how People have made are making databases work and in clouds Work Okay, yeah scout I didn't know that I was 100% sure that I was getting one of those things I was gonna put up I was really hoping somebody would tell me ah didn't you know this awesome thing already exists. So thank you I'm seriously. I will check that out. That was on my Google queue. All right We will we will check it out. So I'm gonna I think I want to wrap it up So if you have any questions send me an email I love talking to people and making new connections and you know, maybe we can go out to lunch or something like that And I do you know work Can't say much about what we do But we're coming soon and I will be remiss not to mention that we are an Austin company It's a really great environment to work in everybody's super cool. It's as cool than me or cooler It's very flexible, but also highly professional So if you want to you know work really hard on something that is really game-changing But not have to go to a lot of boring meetings like come talk to us All right, so thank you very much video equipment rental costs paid for by peep code screencasts