 Hopefully we can make it a bit less dread-inducing for you. This is Don't Panic, disaster planning for the rest of us. My name is Ronan Dowling. I'm currently the agency tools lead at Pantheon. I was the developer and maintainer, I'm the maintainer of Backup and Migrate, a backup module for Drupal. I had also led me to create Node Squirrel, which is an offsite backup service for Drupal, which was purchased by Pantheon, which is why I am now at Pantheon. So I've been thinking about and worrying about Drupal recovery for eight years, so I've done a lot of thought about it. Hopefully I can help with some advice or just perspective on it today. Who are you? The intended audience for this session is Drupal site owners and builders who are working on small to medium sites. That is not to say that if you are doing enough for these categories, this won't be interesting, it might be, but I am not going to be aiming at a very high level of technical knowledge. This is not for DevOps specialists or sys admins. If you know how to do all of these things yourself and build them all and script them all and get them all working, you've probably already done it. And if you're working on large enterprise sites, most of this advice is probably a bit too basic. So, but if you have running either for yourself or for clients, small to medium size Drupal sites, and you know you need to do more to be prepared for downtime, this is going to be a very accessible way to add some protection, a very low cost, easy way to do it. Before we talk about disaster recovery, we should probably define disaster. So over this specific case, I am defining it as any event which prevents your website's content from reaching your end user. Drupal is a content management system. That's what it's there for. The content is what we need to do. The reason we store the content is usually to get it to the end user. If anything interrupts that, it's a disaster. Now that is not a disaster on the grand scheme of things with all the things happening in the world today. To call that a disaster for a small website is a bit strong. So this is really more of a major bummer. So by the end of this session, we're going to have written a major bummer recovery plan. But I'm not just being dramatic. Disaster recovery is a term of art and a disaster recovery plan is a thing. So that's why I use the term. But whenever you see disaster, whenever I mention it, just think major bummer recovery plan. If we go to Wikipedia, they would be happy to tell us the disaster recovery plan is a documented process or a set of procedures to recover and protect business IT infrastructure in the event of a disaster. Which is a lot of words to add almost nothing descriptive beyond what's in the title. But it will tell us that most of them come down to three basic features. They have preventative measures, detective measures and corrective measures. Past, present, future. That's kind of useful. So I'm going to keep those in mind. Those will come up later. We're going to break things down by those three steps. There is not a ton of disaster recovery advice on the web for websites, specifically for websites. But that which is there usually breaks it down to three simple steps that you can do to create your plan. First step is identify all of the threats to your site. Anything that might happen that might interrupt operations. Anything that might constitute a disaster. Figure them all out, write them all down, prioritize them by likelihood and by impact. Plan the recovery of each threat. Write down a plan for when each of those happens and what your duty respond. And practice. Run through dry runs of your recovery plan so that when you have to do them in action you won't be doing them for the first time and you know they'll work. And this is actually, it's very good advice. It's solid industry standard advice and if you are doing disaster recovery planning for FEMA, you should do this. You should know what happens when the water rises a certain level in certain cities. You should know specifically what threats exist to the things you're protecting and how to deal with them specifically. But it's a bit much for your cupcake blog. It's a lot of work for a small website and which means if that's the level, if that's what you have to do to be ready for a disaster you're just not gonna do anything. You're just going to pray and hope that nothing ever happens. Which is where most of us are with our smaller clients or personal websites. And one of the reasons is step one, listing the risks. I had an attempt, I put together a list. We can go through them, it's interesting. It's good to know what's out there. Natural disasters, that's always the big one. That's the one that's used as an example in every disaster recovery presentation and blog post and scare tactic, that's the big bugaboo. Fact of the matter is if you're running a small website you're usually so far removed from the physical hardware that there's nothing you can do about a storm or a hurricane. You just don't, your web host needs to prepare for a hurricane on the coast but you're probably not going to run in and pull the server and bring it up above water. For small websites, this one isn't really a threat we deal with. Hackers, that's more realistic. Every site has a risk of being hacked. Whether that's an intrusion, password stolen, site defaced or denial of service where they don't need any access, they just need to point too many people at your box and now your legitimate end users can't get at your content. As a small site owner you're less at risk than a high profile website but the risk is absolutely there. User errors, this is actually the big one. If you are, if anybody here is in a web agency I can guess that probably 80 to 90 plus percent of the disaster recovery you have done is in response to something that was done by one of your clients. Somebody who has legitimate access to the site and was trying to do their job and deleted something they shouldn't have. This is the big one for us but it's not something that's usually listed in a disaster recovery plan. Imperfect services, smaller sites, usually not hosted on high availability, expensive, redundant hardware. They're usually hosted on cheap stuff and those go down and there are balances to be made and downtime is one of the things you deal with. That's a real risk for a small website and success. That's a selected denial of service but in a good way. They can be pretty rare but that's also the worst time to go down is when you go viral and everybody wants to see your website. It's good to be ready for those. That's a disaster, that's a denial of service. And dot, dot, dot, that's what makes this so hard. We could sit here all day and we could come up with a hundred more things that could break our website and we still wouldn't be done. And there's no way we could then write an individual plan for each of those even if we could categorize them, prioritize them and calculate their impact and likelihood. So who cares? Who cares why the website went down? Who cares? At the end of the day, whether a hacker broke into your website and deleted your home directory or a squirrel chewed through a cable in Chicago, your website's gone, your users can't get at the content and your recovery might be the same either way. It may not matter why it's down or where it's gone as long as you know how to get it back. Maybe we don't worry about the why so much. So I created a less intimidating approach. The major bummer recovery plan, which is identify all of the pieces that can fail. Not the why they're gonna fail but just all of those pieces that are necessary for your content to get from your author's brains to your user's brains. Figure out how to replace them, not to recover them, not to restore them, not to repair them but how to replace them. The only scenario that we are actually going to plan for is the one where your server or your database or whatever piece or more has disappeared off the face of the planet with no sign and no warning. Because if you can recover from that, if you can recover from the alien abduction scenario, you can recover from any scenario. There is nothing that can defeat you if you know how to replace every part and you know how long it's going to take you. And practice, that is still good advice. So we kept that one. So let's do it. Let's figure out all the things that can fail that can make our content not available to our end users thereby causing a disaster. Here they are, there's only six of them. Six things, that's all you have to worry about. If you're on the more medium end of the scale of site that we talked about, you may have a few more. You may have load balancer, you may have reddish cash or solar. I didn't put some of those periphery pieces in because I know you have these. You have all six of these. You may or may not have the other ones. If you do, feel free to add a couple more but most of the advice I give you for these will apply. So I'm not going to address those. But these are the six. If you were able to replace every one of these on short notice and you know how long it's going to take and you know who to contact to do it, you're bulletproof. You will have downtime but you'll get back up again. So here they are. I drew a terrible network diagram to describe the relationship. Not to scale, not realistic but it just sort of shows you the six, lets you count them. The purple cloud is not our problem. So it's purple. Or zoom in a little bit because that one was hard to read. The domain, it's the first thing you do when you want to build a website you need a domain. So you go to migratebusiness.ie. You go to buy it. You go to your registrar. You pay them anywhere from nine to $100 or euros and they say, yep, you own that now. They go engage in the DNS magic. We've nothing to do with that. We can't help that. And they tell the world that this authoritative domain name server is in charge of that domain and you are in charge of that name server. So those are two pieces that could fail. Those are two pieces you're responsible for. You need to make sure you know how to get them back. Registrar, I don't really know how to replace a registrar. If GoDaddy goes away, there's probably a process for reclaiming the domains that you own. It's probably involved, it's probably complicated. It's probably not something you want to practice. Even though I'm telling you you need to be able to replace all these parts and practice this, that one's a bit unrealistic. But it's the beginning of the process. You at least need to make sure that you are in full control of your registrar. You can access it at all times and then you just go back to hoping it doesn't go out of business and take you down with it. Name server, much more in your control. That is needed on basically every request. So that's where we are going to pay some attention. And then your users come along, these two nice people. They use more DNS magic. I dropped the cloud this time, but they eventually find out from your name server what the IP is for your web server. The contact web server, the web server uses a Drupal that you've uploaded. The files your content authors have created and the database to create a response, return it back to them. That's the other pieces. Those are the pieces that can fail. And if any one of those goes away, so does your website. So there they are. Green and blue, you're responsible for. Purple is somebody else's problem. The blue is blue because it's iffy. And we'll get to that a bit later. So back to our measures. If we, for following Wikipedia, we know that we need preventative measures, detective measures and corrective measures. We need to see it coming or we need to prevent it. We need to see it coming and we need to get back when it happens, whatever the disaster or bummer it might be. Because of the twisted, the way I'm twisted the question here, we're mostly going to worry about corrective measures. This is to figure out how to replace them part of the plan. We're not going to ignore preventative or detective, but no matter how many preventative measures you put in place, you will not be 100% free of downtime. So spend some time, do what you can, but don't sweat it. Detective measures will let you start your recovery faster, but they're not going to recover anything any more thoroughly or any better than you would if you didn't have them. Again, it's cheap, it's easy to do them, but really what we're concentrating on is replacing. So we'll deal with the other two, preventative measures. There are some basic steps you can do ahead of time to lower the chance of downtime. Again, it's never going to be zero. So we're not going to worry too much about it. These are all pretty basic. I bet you're all doing them. So I'm not going to spend too much time here. Security best practices. Do your best to prevent hackers getting into the first place. You don't have to worry about this a bit less. Use good vendors, smaller sites. Again, come with lower cost hosts, lower cost vendors, and those come with more downtime. That's what you pay when you've got enterprise redundancy, when you've got, when you're paying $200,000 a year for hosting, that's going to stay up a lot more than when you're paying $60 a year from Bluehost or whoever. But while that's great advice, and I stick to that advice, if you really are talking about a small website with low budget, the fact that you, by the end of this, you will know what it will take to replace every one of your vendors in an emergency, what it'll cost, and how long it'll take, you might be able to cheap out. Because you might be willing to accept a half a day of downtime for your non-critical site in exchange for hundreds or thousands of dollars worth of savings. And you'll be able to actually quantify it and say that half a day will cost me X. I feel like it might happen once a year, might not. You take that risk. So while I would always say, you know, spend a little upfront to avoid the headache, if you're going to have a high risk setup, a non-redundant setup, it's good to know what the cost, the actual cost will be if something goes wrong. And then, yeah, cheap out, who cares. Building redundancy, redundancy is expensive, so I'm not gonna dwell on that, but wherever you can, wherever you can find cheap ways of having a backup, a hot spare, an easy replacement, look for it. You'll always be thinking about, it's more to maintain, so it's not, even if it's free, it's not free, but something to think of, something to keep your eyes open for, and it'll be more clear once you've practiced replacing something. The really hard stuff, you know, you get a jump start by having some redundancy and train your users. Again, number one cause of failure for most of us is the users, you're training them anyway, maybe do, you know, a specific unit in your training and you hand over to your admins about, like, you know, read the dialogue that says, are you sure? Things like that, that's preventative. There are some tools we can use, cheap and free, which is great, that's what we like for cheap disaster recovery. CDN, these used to be the domain of very expensive high-end websites. They are no longer, they are now free or very, very cheap. We normally think of these as a performance enhancement or, you know, distribution for world. They also do offer us some disaster recovery advantages. Cloudflare and Fastly are the two I've picked on. There's a few more, these are the big ones. Cloudflare, at least, they have different services, but they do offer some protection from hackers. They will try and detect malicious access attempts and just block them, so that's really nice. That's preventative. They will prevent DDOS by either detecting malicious traffic from locations you're not usually receiving traffic and just blocking them all together, or unintentional DDOS, you get that slash dot or that reddit hug and they can help keep your site up during those spikes, which is great. So that's preventing the downtime. They will also, the little added bonus here is because they sit at the very front of everything else, they're in front of those other pieces, except your registrar. They'll give you a little bit more breathing room if something does go wrong. They will serve old content, day old content when your site's not responding. So you've got time, you can actually have something fail and you've got time to bring it back up. Maybe there won't be any downtime at all if you get it done quickly enough, if you've practiced this, if you're okay with a little bit of older content staying up. So that's another little advantage. It's kind of more of a recovery thing than it, but it's great for disaster recovery. So free is the lowest level of Cloudflare. It's almost a no-brainer if their terms and services don't disagree with your site or your client. You can use a third-party host to DNS. The thing about the name servers, it's one of your pieces. It's usually tied, married to a different one of your pieces. You either do it with your registrar or you do it with your web host. That's very, very common. You can't actually outsource this. There are ways to do it. None of these are actually free, but they're all quite inexpensive. They will give you some protection from DDOS. They will do some of that same traffic, about that same traffic analysis. They're gonna have better uptime if you're finding that your name servers keeps going out and people can't get to your website. These guys don't. These guys are very, very, very reliable for the cost. And then there's actual redundancy. When you registered that domain, it gave you two name servers or it told you to put in two name servers and s1.migratesite.com and s2.migratesite.com. But in a lot of cases, if you are using a VPS from a sort of budget host, not only are both of those domains, those name servers pointing at the same network, they're possibly pointing at the same box. And that box might be your web server. So even though you're supposed to pick a primary and a secondary name server for redundancy, you're not actually getting redundancy if you're cheaping out in this case. This will let you have actual redundancy and inexpensive redundancy. And that's what we're looking for. It is one more point of failure. And that's one more thing to be monitoring. It's one more thing to be checking. It's one more thing that can go down. So I don't necessarily say that this is a no-brainer. You might wanna have a reason for this. I will say if you're choosing between your web host or your registrar for hosting your name server, I would go with registrar over web host, unless you were really, really, really, really cheaped out on a registrar and found somebody really iffy. The chances are the major guys have pretty good uptime. They are hosting a lot of domains. If they go down, your site is probably at least to your worries. Also, if your web server goes down, it can't be replaced if it's also tied in with the network where your name server is. Replacing the web server requires switching your name server to point to a new web server. So it's harder. Again, I'm punting on the registrar a bit, but as long as you're going to trust them, trust them with both. On the other hand, if you took my advice and installed Cloudflare, that's now your DNS. So this whole slide is moot. They're detective measures. We can figure out what's happening before we start getting angry calls from clients. So subscribe to Security Advisories. Know what's coming down the line, what risks you might be at. If you stay abreast of that, you're going to have a better job fighting it. Set up Twitter alerts when your site goes down. If you've got any kind of traffic at all, you're going to hear about it from your loyal and now angry followers. If you're doing this for clients, set up alerts for their names too, just so that you know, so that when you get that call, you're not super surprised. Audit site users and content is a hard practice to be in because there's not really a lot of great, simple, easy tools. This is actually hard work. I'm trying to create a system here that involves no hard work and certainly no ongoing hard work because if you were to come away, if I was to give you a plan that required you to do something every week, it would get done done. So, but maybe if you've got an admin who's in every day, maybe you make a habit of checking the user list and the node list for irregularities because that can be an indication of something getting corrupted. Somebody who's in there that they shouldn't be, it's good to keep in heads up. It's a bit more manual, but it's usually not a lot of effort and set up automatic monitoring, which is basically the crux of our detective tools. Cheap and free tools that will let us figure out when we have downtime as quickly as possible. Uptime monitoring, this is the one. This is the big daddy of web monitoring tools, probably already using them because this is not new news. If in case you aren't, these things go and visit your website periodically, usually every few minutes, try and load your homepage. If the site's down, you get an email. Simple as that. It is that simple. It's so simple that there are literally thousands of these services. They start at free. I don't think Pingdom has a free plan anymore, but it is very cheap. It's extremely worth the money if you care at all about downtime because you'll find out sooner and that's all there is to it. You can step it up a little bit. Application monitoring. These services will check the health of the server in a deeper way than simply is it up or is it down. It'll do resource usage, memory, time of response, all of these things. And that will let you detect problems before they happen a little bit. If you're starting to see things starting to spike and things are starting to get slowed, it's not down, it's not working, you can sort of dig in and figure out before there's a disaster. They do have to be installed on your server. These all come with very, very expensive contracts, which is why five years ago I probably would never have included this in a session like this, because I'm looking for cheap, easy to set up, no brainer solutions. However, these are now included because talk to your host, you may be getting them for free. Acquia gives you the light version of New Relic for free. Pantheon gives you the pro version. I've been hearing around the floor lots of other people giving away free versions of these things or maybe it's like stripped down version or it's a full version or it's limited time, but you may have it and not know it. You're not gonna get it with the super, super cheap host, but it's worth asking and it's worth knowing it's there. And then the meat of it, the corrective measures. What do we do when it happens? We've got our six pieces, one or more of them disappeared. It was abducted, it's gone, we can't find it. We can't rely on it to help us get the new one up. So what do we need to do? How do we do that? Well, obviously, the reason I'm here, backup. Backup is redundancy for your data. All of your work, this is informational technology. So all of your work results in data. Some of it can't be reproduced. Some of it would take a long time to reproduce. And if your data doesn't exist in three places, it doesn't exist in all, at all. That's the old adage. So that means back it up, copy that backup. And then you know when you have to recover that at least the data is there, seriously. So what do we back up? Well, that's the question. It breaks down, again, quite simply, to four different things. There's four pieces you will need to back up if you want your site to come back up. Your server configuration, your code, your files, your database. They're separate because they have different levels of difficulty of backup, different levels of frequency of change and different levels of difficulty of restore. So you probably don't want to treat them all the same. You don't want to have one backup to rule them all. You need to treat these individual parts as separate backupable data. If you want to have the easiest way, the easiest and most success, so we'll go through those. Server configuration, HPI and I, NGINX, DNS, zone files, anything that's unique to your website. If it's standard out of the box in Fedora, you might not need to back it up because you can get that back pretty easily. But if it's a MIME file change in your patchy configuration, you need that for your website. That's data, it needs to be backed up somewhere. The good news is it almost never changes. For most sites, especially smaller sites, we do this once, it starts running. We don't touch it again until something breaks and that's usually because of some security update at some point. It's not too hard to recover with that backup. We don't spend dozens and dozens and dozens and dozens of hours configuring servers for small sites. So if we have to do it again from scratch, we usually can, it's stressful. We don't all love doing it. If we're using cheaper hosts, it can be a bit tough to get those changes in. But if you don't have a backup, you're not gonna die. It's just going to suck more. It is difficult to back up because each of those changes usually goes to a different file, possibly on a different piece of hardware, possibly that you have limited access to if you're not running your own hardware. You have to ask your host what their plan is for backup. How do we get this configuration back? You need to know, but you also wanna make sure you're keeping your own records of any custom configuration, whether you're going through a managed host, a shared host, or if you have a systems operations person you work with at a school or at a client or whatever. Every time you ask them to, hey, can you bump the memory on PHP? Can you install this Apache module that's on every other server but somehow wasn't on this one? Every time you have to do that, you need to keep a record of that, keep it together, because you may be asking for all of those things again, and you may be asking for them in an emergency. So you don't wanna have to go remember what was the thing we have to do to make that one module work on this one site. Just keep a record, keep it safe, and then you know how to get back from scratch. And the other thing you can do is adopt DevOps. This is what the track is. You're taking all of that server configuration that we only state to do, all those buttons that we had to check, all those things that were impossible to back up, and you're turning them into code which we know how to deal with. So that might be a bit of a big ask for small websites, but it's a great future because it means we can treat our server configuration like code. And code is easier. It changes rarely. Not much more frequently than server configuration, but for small websites, you're probably not pushing changes every day. You may not be pushing them every month. You may not be pushing them every six months. It changes pretty rarely. It's sometimes possible to recover your code without a backup. It all lives on, most of it lives on Drupal.org. You get it off from GitHub. You might have bought a theme if it really is a very small website. Chances are you have some custom code and some custom beaming, even on the very smallest of websites. So don't rely on not having a backup. It's a huge pain to gather everything and remember what it was and all that. So treat it as something to be backed up, but it's not, for some people it might not be the end of the world. If they're using it pretty out of the box, it always should be in version control. That might be enough for you if you've got version control and you've got hosted version control and you're comfortable that GitHub has more availability and uptime than you could possibly ever need. Maybe you don't have to worry about having a second copy of it. I still would if you can, but this is not sort of end of the world stuff if you're using decent providers. And try and automate your deployments. Again, if you can go to fully integrated DevOps, that would be great, but even if you're just hand managing code, pushing it by FTP, make sure that that's automated because that will mean there are fewer places for you to be the user error. And it also means that in that emergency, when you've got to recover, it's quicker. You've got one button to push, you switch your deployment over, it goes to the new server or it goes to the same server that just got recovered. Makes it a little bit quicker. If you don't have a sophisticated DevOps team setting you up with continuous integration, there's services like deploy bot that literally read your git, push it over to an FTP server, make a log. They're inexpensive, they're well worth it. Uploaded files, these are the files your users create. Videos, images, documents, upload them into Drupal. They don't change that frequently. Usually for smaller sites, people aren't churning out dozens of videos a day. These are pretty low changes. They're pretty difficult to recover without backup. This is, I mean, I should take the issue away here. They're difficult to recover, but they're not always impossible. Remember that there's always two copies of a file when you upload it to Drupal, it doesn't disappear. So it could be a research project to get back files if you're three weeks out of date, but you may only have three or four files uploaded in those three weeks and they may be available. So you need to backup, but you don't necessarily need to be obsessive about it and that's good because they're quite difficult to backup. They usually are, even a small site will have hundreds of megabytes of files that can go up to gigabytes, that could go up to terabytes. That's not easy to backup and restoring is slow. It might be easier to go find the one video than it is to restore gigabytes and gigabytes and gigabytes. Some tools to help you do that, backup and migrate works on files, but it's going to cap out at gigabytes. Our sync will be a little bit more work, but a little bit more efficient and you can tie them all together with custom scripts. The size of your backup directory is going to dictate what exactly you need to do to back it up and how frequently, if you can use something really simple and easy like backup and migrate and do, but if not, then try and have something in place, remembering that it's difficult to recover that backup and then use your database. This is all of the hard work you or your client has done to the website. This is the critical one. It changes very frequently, usually daily, sometimes hourly, sometimes less. It's impossible to recover that backup. If you're a blogger, you're probably writing your blog posts directly into the web and that's the only place that text has ever existed. This is not, this is impossible. If that's gone, you have to rewrite your entire website. So back it up. That's the priority. It's easy to backup. It's a small amount of data. There are lots of tools. Backup and migrate, I wrote it, it's great, use it. It will work for almost every site because your database is so small. You could be backing up hourly and on most places you won't feel the hit of the extra work that's being done. You won't feel the cost of the extra files that are being stored because it's a few megabytes of data. So that should be the most frequent. It should be the easiest, it should be the cheapest. But as well as the pieces we can back up, we can also, we can sort of turn the backup on its side and look at the different layers, the levels I'm calling them, of things that can be backed up. We have server level backup, we have application level backup and we have content level backup. Server level backup means backing up your whole server. It's usually provided by hosts. This is the kind of like C panel backup tool or most hosts have something, a lot of them promise they do, they don't always have it, they don't always have it working, they don't always have it as frequently as they say they're going to. But it will back up the entire server configuration, database, code, files, everything. It should be, it's quite slow to recover. You may need to file a ticket, it may take a week and the first three they try won't work. They'll have to find one that's a month old. So it's dependent on your hosts, cheap hosts are worse, better hosts are good. But it is pretty good for total system failure. If that server is completely wiped out, it's no good just having your files and your database. You kind of want everything to be restored all at once. So, look into it, make sure your host has it or you have it set up, make sure it gets tested once before you need it because it won't be, just because they say they have it, doesn't mean they do. Application level backup, this is a layer above. This is just your database and your files, you control this, so there's no reason not to do it. You're not relying on anybody else. You can recover a database in a couple seconds, which makes it a really easy operation. You don't need to open a ticket, but it only really works. It's best for user error, homepage got deleted, you're gonna back up, you're gonna restore your database and partial failure. If it's not completely dead, if the aliens didn't literally take the server, then you may be able to use an application level backup and that's backup and migrate or PHP might admin or whatever. And then content level backup, we don't usually think of as backup because it isn't, but it helps for disaster recovery and for our context or bummer recovery at least. It's version control, it's revisioning in Drupal. It's in core, turn it on, it's very easy, that's couldn't get any cheaper than that. And it means if someone messes up a page, you can be the hero by reverting it. It's not good for things that aren't entities, which is quite a bit of data and it's not good for deleting because the revisions get deleted too. But have it only because it costs almost nothing to turn it on and just to know that it's there and then be a superhero. It's also important where you back up. This one seems kind of obvious. You can back up onsite, offsite. Onsite backup can be very good. This means backing up to the same server or at least the same network that your site is hosted on. It's quick, it's quick so you can do it every hour and not worry too much about pushing a lot of data over the wire. It's quick, recovery's quick. Quickest around, it literally can take seconds. But it's not good for system failure. If your server goes away and all your backups are on your server, all your backups are gone too. So that's why you need offsite backup as well. This means backing up to a different server. It's slower. It will slow down your website if you do it too much and you're using something like backup and migrate which uses web resources to do the backup. You're gonna have a problem with gigabytes and gigabytes going over the wire every so often. It is more effort to set up because you have to have something, some place to send it to. You have to have that other thing that's offsite and that is independent and that takes set up. But it's there when total and utter disaster comes, when that piece is missing, the backup is not tied to it. Some of the solutions, NodeSquirrel, I built it. It's free, so for smaller sites, I highly recommend that. S3, not free but extremely cheap. So if NodeSquirrel doesn't work for you for whatever reason, it's not that hard to set up S3. It needs a little bit more maintenance because you gotta make sure that bill keeps getting paid. That's another big source of disasters for small sites is the credit card runs out and you forget to renew your domain. Or you can FTP to another host. If you found a cheap host, you can keep running as a hot spare. That's a bit iffy because two cheap hosts is not necessarily better than one cheap host. But if this is the only thing you take away from this backup section of this thing, then I hope that you take it away because offsite backups from your host are not offsite. Even if they tell you they're using S3 and they've got it redundant in six data centers and it's protected from anything that might affect them, if you need to access it through them, if you're reliant on them to get at the backups, then it's not offsite for you. And case in point, NodeSquirrel now owned by Pantheon. If you're not on Pantheon, use NodeSquirrel because it's free and it's awesome and it's offsite. But if you are on Pantheon, don't consider that offsite backup. Even though they are independent systems, we control them both. I mean, I trust us, you shouldn't trust us, right? That's not having independent offsite backup. So I hate to tell you not to use my product, but don't use them both and think you've got two because you don't. There have definitely been pretty high profile incidents of servers going down and backups going with them. So don't remember this part. And then restoring, obviously, what good is backing up if you can't restore? I'm not gonna go too much into detail here because restoring your site depends on your backup solutions, depends on which one of those things you picked. They're different. They're all different, slightly different restores. Depends on how down your site is. You really can do partial recoveries. You don't have to assume that everything's gone just because someone deleted a Node doesn't mean you should blow up your server and start the whole process of scratch even though I said you can and you're bulletproof. Layer your levels. Server level is going to restore everything but it might be a week old or it might be a month old. Your application level is only going to work after your server gets recovered but it could save you by having data that's an hour old. So make sure you know to go from the bottom up, keep applying backups all the way up the stack until everything looks great. Practice, I know you're not actually going to but feel guilty for not practicing and then time your practice. Again, if you know how long it's going to take you to do that recovery then you've got a better idea of whether it's worth it, whether it's worth spending more for this service or that service or whatever and you can tell your clients, hey, don't make any promises unless you're willing to put a contract in but you can have a good sense of what the recovery's going to look like. And there's a few more little pieces and then we should have time for some questions. There's a few more little sort of wrap up pieces that could help. I've identified six parts of your website in a lot of setups that's four or five different third party services, who you are paying, who have given you a login credentials. I've also recommended about four more so now you're going to come away with seven or eight accounts that all need to be accessible to you while you're panicking. So think about that before you start panicking, which means keep all your logins together. For a particular site, not just for your whole company, not for, you know, not just your key chain manager, keep all the logins for each particular site together. Web host, registrar, CDN, all the stuff I recommended. Make sure it's all in the same place. Make store copy online for when you're on holiday and something bad happens and offline for when something bad happens to the internet. Keep them all together and listed by site. Tech support contacts, all those same services. They, the good ones have good tech support but don't assume that their ticketing system is going to be up when they go down. Don't assume that you're going to be able to look up their phone number when their website goes down. Store their email, store their phone number, store their Twitter, store their status page because it's usually hosted somewhere else. It usually stays up when they go down. That's a good idea for them. Make sure you know where those are without having to go to the homepage of the thing that might be dead right now. Email, password, reset. Each of these accounts, I don't know if any of these have any exception here, so I'm going to assume every single one of these accounts has a user or an email based account and they use password reset via email. That's the standard right now. So you really should know how to do a recovery. If somebody hacks your stuff, you need to get those passwords back if you just forgot them or if they got changed. You're going to find that out at the worst possible time. Have them all go to the same email. They don't have to go rooting around and figuring out who owns the domain and who owns this, who owns Fastly and who owns the server. Have them all go to the same place. Takes a little time to coordinate that but it's well worth it. Don't use a real user's email address. People leave companies. People go on holidays. People get annoyed and people lose their own passwords. And so don't use a real person. Use some address that's designed for this purpose. Don't use an email that uses your website's domain. Your name server is one of the pieces. It's the second piece. If it goes away, you need to be able to replace it and if you can't replace it without getting an email sent to something on that domain, you're in big trouble. So use a different domain. There's nothing to do with the website. There's security issues with forwarding but you need to make sure that anybody who may need to recover, if it's just you, it's just you but if you've got a team, if you've got clients, you need to make sure that anybody who needs to recover can recover and there's no single human point of failure. And since we've just created this absolutely huge security risk by creating one email account that can now give access to every part of your operation to anybody who gets access to it, consider using two factor authentication and everything else you can do to make sure this email is secure because this really is. That's the downside of this convenience. It will be great in an emergency but it means if anybody figures out how to get in there, you've just created the biggest emergency you ever saw. So those four points tell me that you should really use a Gmail account. That's free, some of the best availability of any host and they have free features like two factor off and forwarding and they're not on your domain. Don't use your Google for works account. Use a Gmail account even if it feels cheesy. No, you don't have to tell anybody. And then test resetting your passwords. This is the practice piece. Test it, maybe you see how long it takes to reset a password. If it takes six hours, and you'll be a little less panicked when it takes six hours in an emergency, you'll just know that that's the process. If there's suddenly a security question that sometimes you do the reset and it's asking you a security question as well. I don't know. Make sure you know that ahead of time. And the best part about testing resetting your passwords is now you just rotated your passwords, which you knew you needed to do, but you haven't done it in years. But you just did because you tested it. And there we go. I told you you'd have a written plan. It doesn't look much like a plan. It doesn't have scenarios. It doesn't have responses. But what it does have and it's written down, it's not just you know these things exist. You are writing this down and you're putting it somewhere digitally and physically. It's going to have a list of all of the people, all of the parts, all of the services that you employ, either free or paid to make your website run. It's going to have the login credentials. It's going to have support contacts. It's going to have them all together. It's going to have a list of people internally in your company, in your client's company, your brother, whoever's going to help you at this, who's responsible so that there isn't a lot of confusion when things go down. And if you're working for clients, they need to know who to contact. You're probably their contact support. You're the one who will call all these other places if you're a vendor like that. And you're going to have the location type and frequency of every backup and any gotchas, anything that you may need to know to do a restore on those that wouldn't be obvious. You don't need to get into too much detail, but if you have this written down, it's going to help you a lot in an emergency and you're going to feel a lot better knowing that that's there. And you can write it down and you can leave it there and now pretty much everything I told you to do does not need to be done every week or every month or every year. Your small site is better protected than it was before and your day-to-day level of effort has gone nowhere. So that's my session. Thank you, we will have time. Yeah, we got plenty of time for questions. I do need to remind you to please evaluate this and all of these sessions. And before I go, we got to make sure you guys join us for sprints to their tomorrow. Start at nine, go pretty much all day. So thank you. And if anybody does have questions, the microphone. You had a slide up that said if your data isn't backed up in three places, it's not backed up. And then you said also off-site, on-site don't consider it with the host. So ideally, what are those three places? Well, if it doesn't exist in three places, it doesn't exist at all. So the original data, the data that you care about that's on your Drupal website, that's one. Then you should probably have, for as much as possible, it's not practical for everything. For as much as possible, you should have an on-site backup, something that's on your server, quick to restore. When your client calls, it's not an emergency, they just screwed something up, you just do a restore, and they feel bad about themselves. And then you should have something off-site. And that can be less frequent because it's going to be less critical, it's gonna be used less often. The chances of a disastrous thing happening to your server are a lot lower than something pedestrian-like deletions. So you can consider those time-based. So those would be the three I would do for at least your Drupal database, your files, and your code. Your server config, it's if-year. I don't know where the third place would be, but you should at least have that in two places. Anybody else? All right, well, thank you very much.