 This is less of a talk than it is an apology I Right last week in AWS and I host the screaming in the cloud podcast and I inadvertently have spent the last three years Misleading you all I've been a staunch advocate for cloud I now know that this was wrong and I wish to apologize unreservedly. It doesn't matter which vendor Google Amazon Microsoft Oracle. It's all the same. It is expensive and you deal with lock-in It's not acceptable. So instead. I'm going to handle it differently What you may not be aware of is I have a side project called Twitter for pets It's the leading social network for pets. It's like regular Twitter only 80 times less racist They're good dogs and We are not going to do it on cloud. We're going to build our own data center We hop in the Twitter for pets corporate jet with the pets and we fly to a different city We don't put our data center in the same city because of dr concerns and real estate in San Francisco is expensive We're based there because that's where the good developers live Our first data center we tore as a super nice security guard He lets us in and doesn't need to see our ID. That's an instant fail because we have auditors We manage both medical and financial data for the pets. It's a hard fail the second data center We look at our one-line diagram It's an art. It's an electrical graph of everything in the system. They show four transformers We walk through two pads two transformers. It's aspirational. It's a joke. It's a one-liner diagram We visit the third data center things seem mostly fine. We shake hands. We sign a three-year contract Everything gets done. We can start moving in in only four short months. We start buying servers from Dell It turns out that when we wind up placing orders We have to schedule it carefully if it goes too soon. We pay extra storage charges if they don't know it's coming They wind up leaving them out in the rain because of this because Dell screws it up We get both they leave it in the rain and charge us for it. We smile we sigh life goes on We unpack some boxes. There's no fiber. We don't need a g-bick Cisco screwed it up They sent us the fiber version instead of the copper version. We open a support case It only takes four short hours We then wind up building out our data center cage. We check it end-to-end it contains no children This is apparently a comp a hard thing to do This does render us ineligible for some government contracts, but that's okay unlike many tech companies We will not name Twitter for pets has standards It turns out we were wrong. There is in fact fiber there. Okay. We're gonna need that g-bick That's a bit of a problem. Fortunately that product case is still pending. We can swap it out It's annoying life goes on These are rack nuts. This is why everyone bleeds while building data centers. They tear your hands to hell Your eyes do not deceive you. There are four different sizes. No, they are not compatible Yes, we buy four different kinds because whatever we get we know we're going to be wrong We rack a bunch of servers 11 or perfect the 12th does not we begin diagnosing because we are adults and Professionals it only takes us four short hours to figure out the problem. The problem is it turns out as a failed cable This happens from time to time. We always have a cable tester for that exact reason We also bring in the world's best firewall. We cut the cable when we throw it away Why do we cut the cable before putting it in a trash? Terrific question. I'd like to introduce you to someone. His name is Dewey Dewey has a job that job is called remote hands He works for the data center and charges 75 bucks an hour to do simple tasks for 125 There's a smart hands option which really says insulting things about Dewey, but he's a trooper He lets it slide two o'clock in the morning in the hotel. We get a phone call. That's right Nagios It's the original call of duty. We rush back to the data center. What happened? Well, it's pretty obvious once we get there we go back and take a look at the networking side of things Remember that fiber? Yeah, it's loose for a reason. It turns out that one of the remote hands people decided to tighten it for us So it wouldn't get caught on anything tighten fiber. It breaks. It breaks. It doesn't pass light TCP now terminates on the floor. It's a problem We believe strongly in the in the culture of a blameless post mortem We conduct a root cause analysis and blamelessly determined the root cause is Dewey We talk about hiring potentially smarter hands budget makes that a bit of a non-starter We finally get back on the plane and who I fly back to begin work on our application And that plane is the only part of that story did that did not happen at a data center build out for one company This is not worst case. This is typical case, but we own our own destiny We are able to build out our own stuff without lock-in and now we can begin writing our code as soon as we finish Installing Kubernetes, which is a short thing to do when you don't get what you want You get character. My name is Corey Quinn. I'm a character