 Hey everybody! So I'm Mark Embriaco and I hope you like my clickbait title. They asked me, they asked me for a title for my keynote and I tried to come up with the most sort of obnoxious and annoying thing I could think of to say at a Rails Conf. And it's funny but it's also, well here let me introduce myself, I gotta say thank you to Pivotal first. So I work at Pivotal, I've got this stupid title, Field CTO, DevOps, and Transformation. What that really means though is, and I'll be talking about kind of the things I've learned over the past decade or so since I got involved with Rails. And what I do is I go and talk to big companies and tell them how exactly the same things we were doing at 37 Signals and Basecamp and Heroku and others can work for them. So I get this fancy title so they'll actually listen to me. And just to manage expectations about this talk, it's roughly half self-indulgent nostalgia for me, another half-ish of thought-leadering and then maybe accidentally an insight or two. We'll see if you can tell me afterwards if I was insightful at all. So scalability, let's talk about scalability. Since my talk was theoretically about scalability, let's define that for a second, right? And I think this is actually, it's interesting because this is a term that often is not well understood, right? The difference between performance and scalability. Performance is sort of how fast you can go, right? How fast can you go? Can you meet a response time for a specific number of users? And scalability is if I'm adding more users and I've got more throughput requirement, can I just add more stuff to make it go faster? Can I add another server? Can I add another Heroku Dino? Can I add another, you know, Docker container? Can I make the thing go handle more users by adding another resource? So this is interesting because it applies to a lot of things. And we could talk a lot about kind of the details here. We could talk about universal scalability law. This is actually really interesting, but we're not going to talk about it right now. We will talk about the equal bang for the buck model here, which is linear scalability. So linear scalability is sort of what we're talking about usually. When people said Rails doesn't scale, they were talking about this. They were saying that you couldn't just add another server and get double the throughput. Linear scalability is sort of the utopian model, right? If one server gets saturated, I add another one. And things are, you know, I can handle twice as much throughput now. Plot twist, Rails actually scales, it turns out. So I lied, but I hope it'll stick around anyway. But the thing that we're going to explore in the rest of this time, I think, is the fact that linear scalability of technology is actually kind of boring, right? It's an architectural concern. Anything can scale. It's all about how you design your system. And Rails doesn't preclude that in any way. It never has. Even when Twitter said Rails doesn't scale, they were wrong. We used to fight this all the time. But the reality is linear scalability of technology is actually boring. I'm way more interested in what are the resources, right? So if on the left we've got linear scalability of technology and I'm adding resources, I'm adding servers and I continue to scale linearly, are there things that I can add that'll let me scale sort of hyper-linearly, right? Can I scale disproportionately by adding resources? And I think the answer is yes. And we're going to talk about that. And here's the reason why I think the sort of linear scalability in technical systems is not super interesting. It's because scalability is a characteristic of the system as a whole. When we look at the systems that we run, there's a technical system, there's that Rails app that we built. But that's only part of the story. In fact, it's probably the least interesting part of the story. There's a whole additional social system that's attached to that technical system, right? This is your users, this is your system administrators, hello. This is your developers. This is all of the social parts that sort of live around and with your technical systems. Now, when you put these things together, you get this thing called a socio-technical system. And this is what is the most important thing to scale. I think for me personally, in fact, I strongly believe that scaling on the social side is far more important and interesting than the technical side. So, story time. I'm going to kind of spend the rest of my time giving you a hopefully interesting, long winded historical presentation of my life, I guess. Especially my sort of work life and kind of the things I've learned along the way. I'll try to be remotely interesting. So, 92. 92 is the date that I picked as a start because that's the year I graduated from high school and went to college. And it's interesting because it's sort of when I really got involved in systems. My first job in college was in the fall of my freshman year in the CS department as a system administrator. So, I've been doing this for, you know, David said 20 years. He was actually lying. He's not quite old enough for it to be 20 years. I say 25 and I'm underselling it, but we'll split the difference. And my story like other historically great stories starts with a camel. In 1992, in fact, I was all about camels. I was about this camel, in fact. Pearl was sort of the first language that it wasn't the first language I learned, but it was the first language I fell in love with. There were a lot of things I loved about Pearl. I loved a lot of the same things I see in Ruby today are what I loved about Pearl. Pearl gave me this power to be expressive. It let me do things in a way that felt natural. And it was sort of the natural language of the web at the time, right? If you're doing anything on the web, well, if you're doing anything on the web in 1992, congratulations. Because we didn't have inline images yet until 93. I looked this up earlier. I thought I was right. Mosaic 0.3 came out in sort of the middle of 93. And that's what we got inline images. And I love Pearl because there are a lot of things I loved about Pearl. It was this ethos around Pearl, right? Larry Wall is incredible. And he had this Tim Totey thing. There's more than one way to do it. And that was the ethos of Pearl. And there was this Pound Pearl channel on Fnet, which was sort of the manifestation of community at the time. It was a relatively toxic and neck beardy community, but it was a community. It was not at all as friendly as what we have today, which is interesting, but I still loved it because I felt back in 1992, 1993, you sort of felt like a magician, right? When you're working on the computer, there's very limited documentation. You can't just go get books everywhere. Hell, just getting on the internet was an accomplishment. So here I am bending the computer to my will. I'm the CS major, and I'm writing this horrible code that nobody else can read, and it's full of regular expressions, and I'm loving every second of it. And I got involved in the Pearl community for a long time. And Mod Pearl was my first foray here. I dropped out of college in 1994 and started a company. And it was a web company. So we spent more time, this was 1994, we spent more time telling people what the web is than we did actually earning money. But so it goes. And I got involved in this Mod Pearl community, and this was the first community online that really felt like a community to me. I ran the mailing list for this, and I was heavily involved in this community for a long time. I went to the Pearl conference, I spoke, I went to the first two actually, or the first three. And it was a ton of fun, and it was a great time in my life. But this startup, you know, eventually when you spend too much time explaining what the web is and not making money, life comes at you fast and you have a couple kids. And suddenly you have to figure out how to balance these things that are fun and interesting to you with, you know, feeding your children, right? Having a good life. So I left the startup that I founded, I went to work for a bank. Don't recommend that. In fact, I've turned on another offer at the bank since then. So I don't recommend it, but I learned a lot. And the thing I learned here was that you have to care about what you do, right? Just making that paycheck is not enough. So I didn't live, I didn't last very long there. And I had some other jobs along the way. I worked in America online for a couple of years, which sounds crazy, but it was actually awesome. Because I was running the largest websites on the planet. And we were carpet bombing the world in CDs and making cool sculptures out of them. So that was cool, right? That was fun. We had unlimited budget and like more internet that we could, we could use. Right? I had, I had 100 megabit connection to my desk in 1994 or 1998, right? It was unbelievable. And then I left AOL. And I left AOL because, because my family was more important to me than living in Ernie talked a lot about this a lot just before me and, and Ernie kind of stole my thunder a little bit, which is great, because I'm hearing recurring themes and I like it. AOL had this culture that you had to be in the office and, and the office was in Northern Virginia and for anybody that lives in Northern Virginia, I'm sorry. It's not much fun, right? It wasn't for me. It wasn't for me. I know people don't love it there, but I don't know how. So, so we lived there for a couple years. And we just decided that we wanted to be with our family again and we wanted to move back. And life came even faster, right? So, you know, Facebook moved fast and break things. Well, I had four kids now. I had to move fast with stable infrastructure. So I had to have a real job. And this sort of led to what I call the Java years. So this is the real application that I built. And honestly, we laugh about the Java years, right? But I'm incredibly grateful for them. Because this was a time, think about sort of 2000, 2001, the dot com crash. I spent all that time in the dot com downturn employed at a startup writing Java. So I got the right, and, and I worked remotely. Now, I was building software to do estimating repairs on, you know, heavy trucks. But hey, you take what you can get. So I was grateful for those years. But there's a downside. So the job years are great. Let me move fast with stable infrastructure. Actually, let me move slow with stable infrastructure. But, but it was valuable. Except for the fact that I grew to not love coding anymore. And this was a huge bummer, right? Because this was a big part of my life. I knew the entire time I was growing up that I was going to work with computers, right? When I was a kid, you didn't really know what that meant. But I knew that I wanted to write code. And I knew that I wanted to work with computers in some way. And here I was. I was 2,000 five. So I was 30 years old, 31 years old. And I just didn't love coding anymore. And it sucked. So my whole life was around coding, you know, my whole professional life. And I didn't want to do it anymore. So I started looking for outlets. And I went to this, I went to this conference called no fluff just stuff. And this was sort of a kind of an elitist, right? It was a weekend conference. And their pitch was, you know, you only go to things you really care about getting better. But it was for like sort of the dark corners and the nerdy bits of Java. When I first started going to it was sort of the right way to think about it. And I went to this thing for a couple of years. And it helped. But one time I went to this talk by Dave Thomas. Dave was a regular on the no fluff just stuff circuit. He was talking about Ruby. And I went and sat down because I had, you know, I'd been to this thing several times. I'd seen most of the other tracks already. So I went to Dave's talk. And Dave started talking about Ruby. And I was sitting in the front, I remember it like a gesture. I was sitting in the front row and watching. And I just, my jaw hit the floor, right? Because I was looking at Ruby and I was seeing Pearl. And it brought back all the same feelings that I had when I first fell in love with coding with Pearl. But it brought objects and these other things that I had grown to actually think were valuable along the way as well. And I'd been, you know, you'd seen lots of conversation about rails at the time. This was 2005. So we were at the very beginning. David was being, you know, antagonistic towards everyone in the universe about rails, which was great. And we were all skeptical, which was great. And I went to this thing, this talk from Dave and it just blew my mind. So I went back to my hotel room that night and I wrote my first Rails app. And this is what I loved about Ruby when I first saw it. I'm not even kidding, right? When I watched Dave talk about Ruby and he started doing this crazy opening of class and adding methods and shit. I was like, what is happening? This looks like Pearl. This is amazing. I can do all kinds of horrible crap that no one can ever understand. Yes, these are my people. And I got a little obsessed. And I started, I started reading the 37 singles blog. And I started learning more about rails. And I somehow, I still don't know how, convinced my boss to let me, we had a new project that we had to start working on. And I guess I was sort of the lead developer at the time. And I convinced him to let me build it in rails, which I had known for like two weeks. And it was amazing. And it went shockingly well. In fact, it went so well that I convinced him, after we did that for about a month, to let me rebuild the thing that we'd been building for four years in rails. And we did it in four months. And it was way better. That company was called Decisive. And I specifically want to shout them out because the faith that the CTO there, Satish Joshi, had in me, to let me do that changed my life because that's what led to me working at 37 singles. And it sort of set the stage for the rest of my career. They're still using rails. In fact, someone from our community, I don't know if it's public, I won't say his name, but someone from this community that many of us know just started there as their VP of Engineering yesterday, literally. So this is awesome. They're still in rails. I felt a little bad because, you know, we've finished transitioning this big app to rails. But along the way, you know, things sort of escalated kind of quickly. We were finishing this thing up and we had deployed this new version of the app like, you know, two or three weeks before 37 singles posted this. And they escalated really fast because I was really excitable. And anybody who knows me knows I tend to be, especially when I write, very long-winded. It's a I have to struggle to write things briefly. Anybody who's ever read one of my post mortems knows this. So 37 singles is looking at higher assistant men and I read this job posting and I should have posted it here and I was like, oh my god, this job posting was written for me. I was on a vacation with my wife. We were in New Jersey watching my steps, my sister-in-law finish postcard basic training. So we were there for a graduation and I saw this post and I looked at my wife and I was like, I got, I got to go for this. So I pulled my laptop out. I wasn't looking for a job. In fact, I was happy for the first time in a couple years because I was doing the rail stuff that was super fun. But then I saw this and I just knew it was what I wanted to do. So I wrote this, I wrote this email that David basically said, hey, well you should hire me because I know all these things you asked for plus I know rails. And I don't have a resume because I wasn't looking for a job but here's 1300 words about why you might like me. Oops, sorry David. I read it. I read this, my email to him again when I was working on those and I cringed but you know I guess that's the way it goes. 1300 words on the fly from a hotel room sitting in bed on my laptop. It was insane. I'm shocked that that that I you know made it to this. So let's look at the dates again because it's interesting. So I sent this email August 17th and September a month later I was at 37 signals and it changed my life and that's not an exaggeration. Almost everything since then in my career I sort of owe it to my time there and to David and Jason and the team at 37 signals. And it's not just technical things it's not it's not that Ruby on Rails was the the remarkable part of all this. It was the the people and it was the people in the Rails community and it was the ethos that David had and that everyone in the Rails community had this desire to enable everyone right. David this morning his talk this morning really touched me because it's exactly what I care about. The this idea of broadening the base bringing people in making people feel empowered to develop things. The the lessons that the 37 signal shared stick with me. But the reality is we had no idea what we were doing right 2006 and we had the first part right we had this whole you could build a blog in 15 minutes and you could maybe get it running on some hosting provider in three or four days. So you know we still had some ground to cover and we didn't really have much idea what we were doing. You know I was a the way that I described myself technically as I'm a mediocre developer and I'm really good at troubleshooting. So I think I'm probably I don't know I I'm very good at troubleshooting and solving problems and breaking things too so I guess those things go together. So we were figuring things out on the fly and it was fun but it took a village it took a bunch of people working together. The engine yard thing choked me up a little bit because I saw Ezra's name and Ezra one of the founders of the engine yard was you know he and I were I guess the two people who were sharing the most about how to deploy and manage Rails apps of any size at the time and I learned a ton from Ezra and I like to think that I helped spread knowledge as well but the reality is it took all of us working together to compress do this this conceptual compression right we needed to simplify what it meant to write a Rails app we needed to make this accessible to everybody now we didn't really know how to automate that stuff yet we didn't really know we didn't have Heroku yet we'll get to that in a minute but we knew we needed to make it simpler and we needed to enable people to be able to run this code right so we spent a lot of time writing blog posts and we spent a lot of time sharing and I wrote postmortems and blog posts about HAProxy and experiments and Erlang and tons of other topics and I spoke at conferences although hilariously you know I joined 37 singles 12 years ago this was my first Rails conf that I've spoken at so this is really exciting for me so sharing is is the the thing that enabled the community to move forward and we spent a lot of time dealing with the sort of flip side right it's okay great we've got this Rails app built but why is Twitter fail-wailing all the time Rails can't scale Rails is too hard to deploy that'll never work in the real world you know we got this at 30s of the singles all the time because we were very unapologetic about giving advice and we didn't use weasel words to surround that and say this depends because we assume adults recognize that it depends and you have to deal with this and the way that we deal with this in the Ruby world is this is I think this has been lost a little bit along the way and I wanted to bring this up specifically because this ethos we had in Ruby at the time was much as nice to we are nice and this picture is is my son from the life comes at you fast when he was I guess 11 or 12 years old at RubyConf in Orlando and he had just learned Ruby and I said hey let's go take a picture with Mott's and we walked over and Mott's was was gracious enough to take a photo with him and this this sort of ethos of being nice and of sharing and of community I think is what makes Rails special so you know I talked about technology scaling not being super interesting but the reality is if you want to look at some of these resources that if you add you get a disproportionate return community is one of those right if if you share something in the community you're not helping one person you're not getting a linear 1x lift on that that curve you're getting a multiplicative effect so by contributing and participating in the community like all of you are here by the way thank you when you share with the community when you you make connections with one another when you help one another you are making that scale become this hyper linear growth that we're looking for so community scales so I spent four years at 37 singles and it was amazing um and I ultimately decided for a bunch of reasons that it was time for me to move on and do something else and the thing that I had discovered about myself that I was I was passionate about and what's really driven my career since then was this idea of enabling people um in particular enabling developers I was excited when David did his softwares eating the world pacman thing because I trot that line out all the time um andreason was right it's a it's kind of a meme at this point almost this whole software is eating the world thing but it's really true right if you want to have an impact on the world you don't have the best return on your investment if you want to have disproportionate impact software is the place to go whether it's positive or negative impact so to me this idea that I can enable other people and I can help shift that scale better than linear is what's driven me so in 2010 I joined heroku yikes I don't know what I just did hang on oh well we lost the slide I think sorry we'll figure it out I just skip so I joined 30 something I mean I joined heroku um and these two slides that you see here are are talks that I've been giving to pivotal audiences lately in fact this was from a keynote that I gave it devops enterprise summit to I don't know a couple of thousand people that worked at fortune 500 and global 2000 companies who are doing the doing the devops right um and what's interesting to me is that all the lessons I've learned here these were lessons I've learned at places that I worked since 2010 so I've been really fortunate I worked at heroku heroku sort of hit me right in the fields with this enablement story right heroku solve that it takes three or four days to run your rails that problem incredibly beautifully and I spent a couple years at heroku and I learned a bunch of things and then I went to living social um yep I went to living social because Chad Fowler is an evil genius um and uh and he somehow convinced me to go to a daily deal site I still don't know how he did it um I think it was all the people there there's a bunch of amazing people there so it was you know I wanted to go hang out with my friends and get paid for it and that's actually kind of cool um then I had the chance to go work at github and again this story of enablement and community um sort of rings through and and then digital ocean and the thing that's interesting and and what I like about this talk that I gave at that event was that these lessons are things that are very familiar to you if you read Getting Real and if you've sort of read any of the blog posts in our community and what I love about this is that in 2006 and 7 and 8 and 9 we were being told all the time that won't work in the real world that won't work at my company and now we've got people paying a lot of money for us to tell them the same thing we've been telling them for a decade and they believe us now so it turns out that these lessons that we've learned in the Rails community and in the various communities we've been a part of and and have learned the hard way and we've been told over and over again aren't relevant to the real world they're all very relevant and then I left digital ocean I'll tell a quick story here this one apologies if I get emotional here Operable is a startup that I founded in 2000 and I guess 2014 maybe whatever three years ago whatever that was 15 I guess so 15 and 16 I did it and the reason I did it was because of someone in our community and this is the part where I make it emotional so I had been working at digital ocean and digital ocean was in New York and I spent you know my first six months of digital ocean I spent 30 days in a hotel in New York so I was spending a lot of time in New York and I had a friend who had recently moved to New York James Gollick and James and I had sort of known each other in the community for a long time James was very active in our community for those who have been around for a while I'm sure you know who he is and James was very active in sort of the the system side and the scaling side and the operability side of Rails much like I was and we became closer friends but the thing that the thing that happened that sort of led to me finally taking a chance on my startup was over the holidays in 2014 James passed away unexpectedly he was in a car accident and in a taxi and he and his fiance unexpectedly died and it was interesting they respite I had a response to this personally that was disproportionate to how close we were as friends I felt right like we hung out when I was visiting New York but I we weren't super close friends but what really struck me was sort of the fragility of life and the thing that that really struck me was that James was his sort of ethos was give a damn about what you're doing right and care and contribute the way that you think you can deliver the most value to the world so I spent a lot of time thinking about this over the the holiday break and the first day back from the holidays I quit my job because I was convinced that the place where I could have more impact was by delivering something that I cared more about so I found it a startup beginning of 2015 and life came at me really fast and I raised a bunch of money and tried to build some things that I believed in and we built a product and long story short this when I give this talk at at other events I that last bullet I didn't close the doors is fills in later but the the long story short part is we failed and that's okay right I say this and people get mad at me sometimes when I say we failed but I'm very reflective about successes and failures and I'm very I try to be honest with myself about them and learn from them so when I say we failed it doesn't mean that I'm a horrible person or that I didn't get anything valuable for that time it just means that we didn't do what we set out to do but I wouldn't trade that time for anything because I learned a ton and interestingly some of the things I learned were some of the things that I already knew so you know those who don't learn history are doing repeated their next conference presentation and knowing is only half the battle as we all know the other half is red lasers and blue lasers so you know it's like these things that I learned this whole make tiny decisions and and sort of fighting hero culture I learned these things as well as a 37 signals and and but suddenly I'm doing my own startup and it's like yeah well you know that's like your opinion man like I can I can be a hero if I want I can you know I can work a hundred hours in a week you're not my real dad you know I can I can have a nervous breakdown and and these things I learned at at Digital Ocean so these lessons you know sort of around MVPs and feedback loops and I'm like well you know this thing we're building is really complicated though I can't just start with a piece I gotta build the whole thing it turns out not so much so these things just because you you sort of have learned the lessons before doesn't mean that you're going to remember them when you need them but for me the best way to I don't have a my next slide well how beautiful yes I love it when a plan comes together the best way to remember these things is to teach them to other people which is why I've been repeating these lessons to people every chance I get because the more I tell other people the things that I've screwed up the less likely I am to do it again and the last thing I want is for someone else to have the experience that I had where I didn't listen to myself about hero culture because I grinded as hard as I could on the startup for two years and I can tell you the exact moment I hit rock bottom there's a conference called monitor almost one of my favorite events every year and we were sponsoring and we'd been working on a demo and I worked 110 hours the week leading up to that my flight was Sunday morning at like you know 7 a.m. to the west coast and it was 2 a.m. the Saturday night before that and I was still awake because I was working and I finally got to the point where I knew that I wasn't going to be done there was no universe in which the demo that I wanted to give I could give I had to do something else and I was I finally sort of threw in the towel walked downstairs to my bedroom and I'm standing in my room in the dark knowing that I need to go to bed and I just stood there and I started crying because I was just done right I was so exhausted and emotionally drained from this experience of this abuse that I'd put myself through that I just didn't have anything else in me right I was numb for days afterwards it was it was the low point in that experience and the worst part of it was it wasn't so much that I put in the effort and failed like I said failures it happens right I don't like it but I want to learn from it if it's going to happen at least I want to learn from it the thing that really upsets me though is that I already knew that lesson and I did it again anyway so sharing these lessons with one another not only helps other people it helps you too so teaching scales just like participating in the community scales if you can teach one person and they teach someone else or if I can teach a room full of people these lessons maybe I can have a real impact so you know we started with this how do we go from linear to hyperlinear what's that what's that piece that what's that question mark what's the resource that we can add that helps us scale in a fundamentally better way and to me and I've heard this throughout the day from other folks and I'm super happy to hear it to me you are that question mark everyone in this room is that question mark everyone in this room is the resource that helps us scale whether it's the community whether it's the technology our socio-technical system as a whole you are what makes it scale you're what makes it recover when it fails you're what makes us learn so we don't have to repeat the same mistakes you are the thing that helps us as a community as a world scale so people are what scale systems not rails not technology people that's it thanks