 Okay, I have a lot of slides to get through so I'm just gonna start if you have questions Please hold them off to the end. There's two slides that ask the query for questions. Do you hear that? It's just a query for questions. That's cool. Okay So I'm here to talk about beehive and it's the first part of the talk is going to go by kind of fast And you'll probably go what am I actually listening to that's okay because the second part of the talk Hopefully I'll reveal that Just have to get through the first stuff. So just to get started beehive scalable application deployment by Ari Lerner Who is Ari Lerner? That would be me. Oh boy Okay, so I'm a lead software engineer at AT&T Interactive in Glendale, California I'm soon to be an innovation coach at AT&T on Wednesday. I moved to the San Francisco Bay Area Or SF Bay Area and I will be Having a new jab so Okay, so I've been working with Ruby since 2004. Oh, man, it's getting lower and lower I've been working with rails since 2005. I haven't done rails in about a year though. I work primarily right now. I Work primarily right now in distributed computing and systems and deployment at AT&T Interactive I have a totally rad Mini Cooper and sadly it's not this one Because that's a really rad Mini Cooper and I also have a totally rad dog Don't tell me when I have that because that's child porn. She's only three Okay, I do a lot of open source. I primarily only work in open source You can see all my open source stuff at github slash a user a user is my name everywhere Except for here. I'm Ari Lerner these are just an examples of some of my Open source work you like I said you can you can check it out there So that's about me and more about what I'm actually going to talk about today Which is beehive which the subtitle is scalable application deployment as The program suggested so what? Is it the top Man, okay Raise your hand if you can't deal with the resolution problems. All right, so What is it beehive is an open source solution to complex application deployment stacks? It's fault tolerant. It's application agnostic We'll get into that later. It's natively scalable event driven which is important another point that we'll touch on in a little while It's multi-tenant. Oh, I already said scalable. It's has intelligent routing. It's a deployment system I'm going to clear that up. It's a deployment system. It's not a ruby framework. It's not a ruby library It's a deployment system. I met a ruby to conference. So I should say that it's also totally MIT license and Free and like I said before it's open source It's tested. We're using it in internal production at 18 t interactive. It's extensible I'll show you I'll show you how in a little while distributed written in ruby bash see an earling primarily an earling and there are three 18 t interactive Developers on it as well as open source Collaborators on the web notably one guy in Argentina So that was a whole bunch of buzzwords. Am I right? So what actually is it? Beehive is a tool that enables developers to deploy and manage and manage their own applications in a scalable Multi-tenant environment without having to deal with deployment Okay, so basically it's just an active intelligent routing mesh That's a lot easier to understand, huh? So why do we do this? Why even add change? Well because deployment today is really rough I think it centers around the server requires lots of resources people resources Computer resources we most of the time you end up deploying one application to one computer So ninety nine point nine percent of the time that computer is doing nothing unless unless it's an AM Hadoop mesh in which case eighty eight percent of the time. It's not doing anything And also the other problem with deployment is today is that there's you have to deal with like scheduled releases We're not going to deploy until Thursday at 2 a.m. Then you have to stay up till Thursday at 2 a.m Of course, that means you have to stay up until Thursday or Friday at like 12 or 1 p.m Just because you always have to deal with problems that happen Etc. So really I'm a developer. I'm not a deployment person Well, that's not entirely true anymore for me, but most of the time or what I want to do with is only development I don't want to do with deployment so Deployment or the deployment today does not focus on the application nor does it focus on the developer nor on this really sad-looking lady So I suggest we have an application revolution. Let is let's focus on the application but let the Let the server managers manage the servers and let the developers develop so Because we're writing this what do we want it to do We want to make we want to make it easy to deploy rapidly So when you add a change and it's all all your continuous integration server stuff passes you want to be able to Deploy it right there. Why wait? So and enables beehive enables rapid deployment. We want it to be able to do that It's also built to scale So your application becomes like Twitter and boom people in Ethiopia are using it to escape their freedom issues That are going actually Ethiopia has pretty good freedom issues. They're pretty free press. What's a better one? I ran. Yeah, so like I ran or people in Iran are using it to say how poor they live Right now in their environment in their political environment. So all of a sudden your application needs to scale So let's build this deployment solution to scale. It also handles failures pretty gracefully Probably not graceful that I use the Twitter fail well, but it's okay anyway So another thing don't want to use any new tools I don't want to have to teach my developers anything not that I have developers But it'd be cool if I did but I don't as a developer want to learn anything new and add to my current framework So for us at AT&T interactive we like to use get push You can also use pure HTTP with beehive because there's a pure HTTP server in there So most of the examples I'll go over with today. We'll be using pure HTTP which curl in our case However, there is a ruby gem in development. It's actually checked in in the source code So if you want to add to it, and you don't know Erlang or C or bash You can add to it there Or if you like Python I don't know why you'd be at a Ruby conference if you like Python But you could just write one in Python or you could write a HTTP client in Python Oh, that's another thing we use HTTP. It's been around for a long time why I changed it Okay, so how do we work with beehive? Well, first you have to define an app The way you do that is oh god The way you do that is you use a curl request or soon the beehive gem to create an application and that basically takes your fancy triangle like application that you can't see here and puts it into the yellow curvy Rectangle over there, which I represent as beehive and then you're set. So how do you deploy an application? Well Use either a git post commit hook, which is what we use at 18-10 reactive or you just do a curl request and You hit beehive with that Like this curl request. Can you guys all read that? Okay, am I talking like drone talking? Should I spice it up? Is this good? Okay, let me know Okay, so what does beehive do then? Well beehive will if it has not already checked it out It will check out and bundle the application. I'll go over what that means in like 10 seconds And I'll also check for errors Every stage by the way it checks for errors So if it can't bundle if it can't clone it for some reason your public key isn't there It won't be able to clone it. It'll fail I'll let you know if you can't bundle it for some odd reason Like you can't find a dependency then it will let you know It will try to start the application if that fails it will let you know It's pretty communicative as a framework or as a tool itself And I'll try to start the application and then it will take Remember I said it was an intelligent router So all the requests that come into the router are held for a specific amount of time a time out When this when I call them bees when the bees start up then that router will say okay Now I can start routing requests to that application or that be if you will and so let's again This you can't really see it here, but it's a square app that represents a bee you're putting it into the round-ish rectangle that represents beehive over here Okay, and then you're ready. Oh, that looks good Then you're ready to go Okay, so what did that do that took our deployment time from scheduled releases on Thursday at 2 a.m Where we have to actually stay up till like Friday at 1 p.m. To finish that release to seconds and if There is a problem on the server it will let you know Oh, I should move I'll take a step back real quick if your application is already running and you're deploying a new feature And then for some reason you forgot to check it into continuous integration and your continuous integration server fails And then your application goes up, but there's a problem It can't start your application that it won't start rerouting new requests to the new application So your old application will stay up. So it's 99.9 times 10 nines of uptime because there is that slight chance that the application actually No, there's no chance. I lied totally completely died There's a hundred percent uptime when you update an application because if it can't start it won't start Okay, so take to finish this slide it takes your deployment time from minutes hours months to seconds So I guess alluded to before what happens if an HTTP request comes in why you're deploying an app that isn't up the router The the distributed router in BI will hold that request until it can start an application as I said before my Representation for apps are bees. So I'll probably just use that for now on Bees represent a running application. That's why it's called beehive because there's lots of bees in the beehive and the router is kind of like the Queen Okay So you're this the HTTP requests that comes in say I'm Bob And I'm trying to hit my application at get beehive dot beehive.com or whatever I call it and My application can or I'm sorry me connects to the application to the router which turns into a proxy Then that proxy will be shoved off of the router and handled in its own process early It's really nice at process handling process management. So the router totally forgets it So the only time your router is actually engaged is when there's an active request coming in that has not been connected to a back-end bee Okay, so if thousands of requests come in in a second like let's say I ran all of a sudden their government is over I probably shouldn't use Iran. It's like a political hotbed. I'll say it anyway Say like the current government and Iran is overthrown and there's protests in the street and everyone's trying to We'll say your application is called twit because it's not tweet Twitter It's just twit and everyone does like post the twit to your server And then all of a sudden you have like a million requests coming in at the same time Beehive because it knows how many requests are coming into every single application. It can spawn new applications Seamlessly you don't as a developer even have to deal with it because it's not dealt. It's not on the application It's on the router. It's outside your application. That's I think that's really cool I don't have to deal with it. So knows about the proxy real quick Because I'm definitely going slower than I need to be The the router acts as a proxy like I said if anything happened or I'm sorry like I said before like Bob is here and twit is here and they connect and then a new process is spawned off one neat thing about that process that spawned off Is it it's also spawned with a supervisor in earlying a supervisor? Just its entire job is to watch this process So if this process dies something happens on the server side say the router that you're working on It's not distributed or it is distributed and one of the routers that you're working on goes down That supervisor will restart will restart this the proxy that's connected to the two clients seamlessly so if I'm connected to server a and There's a server B and server a goes down server B already knows about my process So it will start restart the server and I as Bob will never know that twit actually happened Nothing happened on twit to me except for it might experience some slowness if twit's not running on server B or Behind server B possibly It's it's all that's all done seamlessly and automatically by the way You don't ever have to deal with that or set any settings As I said the router holds requests until a given timeout if it can't reach the server If that timeout is reached then the user sees an error because we don't want them waiting forever So what exactly is a be like I said before it's It's like a I didn't say it before B is a single file containing everything the application needs to run So this includes environments this includes like for instance in rack apps and Ruby apps gems Includes all the dependencies that you need to run and it doesn't really look like that because it's just code But I think that's kind of a fun app fun abstraction So really what is it right now currently baked in it's a tar archive So just take just imagine your app and you tar your app and then you compress it take things out that you don't need That's all abuse So what does that do why do we want it into a single application rather than dealing with Check cloning out and just doing our sinks and stuff like that. Well, it's super easy It turns out it's super easy to pass one file around rather than pass a bunch of files around it's it's not rocket science, but I Giggled when I said that at least okay. Anyway So this also allows for continuous deployment and if you live in LA like I do you see this all the time So it's pretty fancy and fun, but if you live in Austin, you don't see this all the time. So oh you do I'm sorry. Okay, I'll rephrase we see this all the time And it allows developers to see it even more to realize it even more So another thing this allows us to do at AT&T interactive. We have tons and tons of applications So it allows us to take that take servers to server resources bunch them together and over provision on those servers so we have we have Application one through like a thousand one through a thousand are not running all at the same time because 500 through nine hundred and ninety nine is not running of course what application one thousand is still running for some reason and So allows us to naturally reap those resources and use them for other applications that are in use One thing I didn't note on the slides is be have will terminate applications that don't see a request in a given amount of time or that are over using the Resources that you that would like kill the server So it will naturally reap reap those and save all your other applications from pure from instant death. Okay What is this one? Oh, yeah, okay, that's the representation of reaping resources So, oh my god, well the server's gone on fire because a gigantic fireball hit it Okay So What does be have do be have know normally what you do if you're on a server and your your application lights on fire You'd use something like Monit and you'd restart it, but be have takes a different approach and says Okay, well, we don't have to deal with it because no requests are coming in or if there are requests coming in Restart the application, but we'll just let it die like who cares if it dies That does represent a problem if you write your app and your app is poorly written and it dies all the time You may not want the user to see that but be have will handle that seamlessly So that's the representation of letting that application go away. Oh, Canada's internet died a The fuzzy one is Canada So that's okay as long as because be hive is distributed by default As long as you have another servers located somewhere else outside of Canada be have your be have will still continue to function so what member again, I had that like list of reasons that we shouldn't do something like this well It turns it turns Development with be hive without be hive into development with be hive So be development with be hive is developer driven uses shared virtualized resources reduces application management overhead It allows you to release release pretty much whenever you want. It's agile and blah blah blah There's growing community already around it You can go to any one of these these places the IRC room is Pretty active. I'm also very low-quacious. So I hope you enjoy that And there's a there's the be have web page that's currently up right now. Actually, I told you lad This is an old screenshot, but okay. Anyway, so now remember I said there's two slides for questions any questions so far You can think of that, you know mind talking No Okay, awesome So I'm just like said before this is copy and this Presentation is supposed to be representative of copy and paste. So I will show you exactly how you can use it So we'll do a server quick start Of course, you have to get the code when you want to do it So you can't just start it if the code somewhere else. So go get the code if you want to start in development mode which enables Enables you to actually do stuff and change stuff on the fly when you're developing without having to do Without having to do any like fancy release develop or release deployment I mean you can do this one or if you want to deploy it for real you can do that From within the code base So what are we doing when we when you type all that funky make stuff, right? We're Ruby. We don't do make Okay, so it starts the following what it does is it will start the distributed router the distributed database in Erlang It's known as amnesia it starts the distributed event manager, which is really key to be high that's how beehive knows about that's how Beehive like all the Distributed routers know about all the different connections is because it starts the event the event machine. I'm sorry the event manager I didn't really go over that. Maybe I'll go over in a minute. We'll see And then also starts the HTTP based restful server where we're hitting all our curl stuff and storage management stuff But that's boring Okay, this is in Erlang if you've ever used Erlang This is a neat little tool. It shows you everything that's running. So this is everything that's running when you type make rail Lot more stuffs happens if you're actually to hit this and keep this it's called at mon If you're to keep this running, you'll see a whole bunch of new PIDs prop popping up and stuff. It's really neat Anyway, so we'll say you're a client. You you're not actively developing on beehive or you don't want to develop on Beehive again, you got to get the source Go build the gem and install it or Soon you just be able to gem install beehive So basic usage Okay, so there's basic authentication right now. It's token based authentication So you have to be in this you have to be in the system. You have to and you have to be able to add a user So if you're running your beehive, you have to be able to write on administrative interface on top of that Add the user and they're just fine so it enables you to have like a sign up on a web page for instance and If you wanted to charge people for it like developer hours like your own developers or other people Then you'd have to make sure that they're signed up on your web page. So beehive doesn't deal with any of that So if you require a credit card to use your beehive, then that's outside of the scope But if you don't require it doesn't matter you can just add one as a as an administrative user But it has basic ACLs anyway, so well, let's say we want to add a user the different levels You'll notice like right here There's levels level one is administrative it goes down from there or up from there down up to to as a base is a If you don't specify that level it's a to so it's a regular user So you can't do crazy things with other people's applications. That's safety Well, that was creepy wasn't it? Sorry, that's for safety Okay, so this is how you'd add an application. I know it looks really gross to us Ruby users Like I said the beehive the actually the beehive create Works in the Ruby gem. So if you did want to just use the Ruby gem You don't have to do this crazy application a crazy curl call But this also works if you're more comfortable with that So let's add an application that will fail you can see that's one that will fail because a User doesn't exist get on github doesn't exist. That's why it's called doesn't exist We can and if you try to start doesn't exist Be have will say oh my god. It failed on the bundle step on the clone step So then you can update it here. It's just basic rest stuff. You can check what applications you have up there You have to pass a token in if you're checking your users applications You don't have to pass a token in if you're checking all the applications, which I realized is a security issue Oh, actually, no, I totally lied. This is outdated. We changed that I'm giving this today, aren't I? So let's update. How would you update a running application where you'd redeploy it? That's why there's deploy at the end of that So let's say I go and add a yellow tool bar on the top of my stuff and I want to see that reflected in beehive You hit that one curl request. It'll do everything. It'll do all the cloning the checking out Try to start the app and then move the requests over to the new app One thing is is if you have users connected to your old app, they will not see the new app when they run Because that application will not die until those sockets go away Which is nifty, right? Okay I don't know why you'd want to delete an app because I think beehive is pretty much just a write an app and throw it up on beehive But if you did want to delete it out and clean up some stuff or let's say your users credit card failed To process or your developer credit card or whatever You could just go through and delete the app although that happens when users are deleted anyway So you could just delete that okay, so I'm some advanced fund. There's just some neat advanced stuff Remember I said before it was extensible you guys remember that? I don't know. It was a long time ago It's extensible in the sense that it's application agnostic and if you can write a deployment Frame if you can write deployment in bash you can add it to beehive So currently right now we support natively rack and rails apps that also do include bundler isolate and gem files But let's say you wanted to write a Python app and you want to deploy it on there I don't know anything about Python. So you'll have to are they called apps in Python? Okay Say you wanted to deploy a Python app you could write it if you could write it in bash So app templates is what I'm getting to So let's just pretend for a second that rails is doesn't I'm sorry be have doesn't support. What is this a rails app? Yeah, okay, so let's pretend for a second that be have doesn't support a rack app We can write one like this put it on your system, and then There's the system by the way beehive home is whatever you want it to be if it's nothing Then I'll just use the core Etsy home or Etsy path and Then update the running system. What's one really neat thing about beehive is if you update the running system It doesn't affect the uptime of the of the system That's one reason for I chose Erlang to write this in is because Erlang was written It's one of two languages written for business use and it's written for business use in the telecom industry that needs 99.99 times 9x Uptime so if you update if you're gonna update beehive you never have to see it go down You never have to restart or anything So what is an app template? There are six different Six different pieces of an application life cycle first in from within beehive. There's the bundle step like I said before I compress it into a single file Set up all the dependencies and then compress. Oh, I said that backwards There's the mount step where you actually take that be and you put it on to a server and start running it And then there's the start command so like when beehive needs to go Oh my god I need to start that because the other one lit on fire then The start command will be executed, and then stop is pretty much, you know stopping an app unmounting is cleaning up that code and Clean up I am actually happens automatically so you don't have to deal with that Okay, yeah, so this would be like the end of this would be like if you were to add up The this is a bundle a representation of a bundle step in beehive Yeah, so if you were to add this and then do that curl and update the system then your system would be going and You wouldn't have to deal with any like then then you could deploy Python apps or in this case rack apps Okay Let's say you want to get the stats of your server This is like all like how many people are running like what's the usage like what's the bandwidth usage? You can do this for different users Let's say you want to get the note the nodes on the system So you have like a hundred nodes and you want to get a specific node you can get this specific node stats For ranges it even supports ranges. I know crazy, right? Okay, so real quick some more advanced fun just to show you how powerful beehive actually is you can add custom event handlers Currently right now it only supports Erlang based event handlers on the roadmap You'll see I'm planning on supporting other languages, but if you wanted to do that. This would be how you do it Oh One one thing these are the exact same They do the exact same thing so either you can add an Etsy file Which I like a little bit more or you can just do export so you're very so your home So your beehive can run in different environments in different homes You can do this one as well, and then do a post reload the system It'll reload all the system that propagates throughout the whole network And then you're done, and then then then you can do that that okay So let's say you wanted to start one in Canada Because you're afraid that the US internet might go down and you want to start one in the US because you're afraid that the Canada internet might go down. This is how you do it These are short names you can do this with long names so you can do it across data centers You can do it across the world, and it'll look the exact same Oh, and one neat thing is after you do add a second server beehive will notice that you added a second server and start moving It's the bees around So that your bees in the US are not too heavy over the ones that are in Canada I don't know why I like Canada so much as an example Anyway, so let's say we want to change how the bees are chosen currently right now It's a random Decision on the bees because that's the one that's baked in by default, but let's say you want to do one that's least loaded So like a round robin type system So like yeah, I'm sorry So your router sitting up here, and then you have a whole bunch of bees down here How do you choose which bee to connect the client to right now? It's random. So are actually I Man these slides are so outdated. I changed it to least loaded. So let's say you want to do random for whatever reason you could take You Start beehive with this with this command it'll start with adding or choosing the bees on a random location if you do this Change how the host name is looked up. This is important because beehive uses the in your HTTP request It uses the host header. Let's say you don't want that. Let's say you want to change You want app or you want users to be balanced based on the application or the application they're using to Wrap to the router. So like the user agent for example say it like matches something like Mozilla And you want that to see a different application you could do that. Oh my god. That's so cool All right, so um If you if you need more help There's been in beehive that will help there's actually six or seven of them or something like that Some of them should go away probably but these two are the ones will never go away start beehive you can see that and queen is the Queen bee of the beehive Okay, so what's on the roadmap beehive Ruby gem we have an active developer on it Who's working on right now want to automatic automatically scale apps currently right now? The scaling is done very dumb the scaling right now is how many requests are coming in how many requests you expect if that exceeds it scale out a new app Scaled down is not done scale down is like based on time and load, but scaling up is so that's coming Automatically scale beehive. So remember I said it was all event driven So because it is all event driven we can fire off an event and say oh my god I need a new server to run beehive on then it's say you're running on ec2 and you're watching for that event You can say oh I need a new server So start up a new ec2 instance start beehive over there and boom beehive is already is automatically scaled out That's something that I'm actively working on right now actually event handling in other languages so we can write So we can write events like we can watch it for events in other languages So like Ruby because that's a lot easier than relying And get a better built-in beehive dashboard. Oh, I totally forgot there's a dashboard Look, but it needs to be better, but it's a dashboard It does right and command line We could call it be yeah, you want to write it? Okay? We actually have somebody who's who's like road mapping that out So hopefully that will be open source soon. That will be a rack app I believe so if you know rack and you're here I would assume that you know at least some Ruby and Ruby or rack is allows you to write easy Ruby applications If you don't know if you do know, I'm welcome to Ruby So that will be open source so you can add to it when you want or when it goes open source Oh, yeah working on a built-in git server. So rather than having to use something like github beehive will have its own git server That just uses git so we're not re rewriting the wheel wait re-implementing the wheel Inventing the wheel another Thing that we're looking at is ISO bundles rather than tar so that you can it's a read-only file system So you don't have to deal with any like temp files or whatever and like so you don't want your logs to go on the on the system If you use an ISO you have to enforce that it will go somewhere else. So be have will fit well What it will fail to start the application if you can't write to the you get it Okay, and better documentation although there is a lot of documentation lots of it is out of date as you saw Okay, more stuff on the roadmap We'll see If you want to get involved, there's the there's the Path to the source there's the Google group that is very inactive and there is the web page Now remember I promise there would be a second one a second slide that asks for questions anyone have any questions now Awesome, that's like two thousand times better than the last time. I'm gonna start over here because I saw you first defended Yes Yes, yes, and I said it was extensible If you can write that in bash, which I can I'm an early developer also I will eventually get that in there that does processing though what be have really is the way I look at it Is there's a distributed router on top of a bunch of processes that can be if they can be connected to by a socket? So you don't have to do you don't have to have a web app that runs on that runs on Like is exposed the outside world as long as the way that you want to connect to your app is through a Right now HTTP X MPP is coming ish because the developer in Argentina wants to write it. I believe Or actually there's development Colorado that wants to write that or is it Argentina? I don't remember anyway The point is is that you the different protocols you we can write handlers for different protocols So as long as you want to connect to it using external socket then you can use be hive And you don't have to even expose it to the outside world I Answer the question sometimes I go all over the place. All right you wait. What's your name so I can actually say Austin, Austin? Really? Well, are you from Austin? Okay? That's awesome Austin No, it's distributed so That's why I said before I didn't mention I didn't heavily emphasize how event driven this is the event the events are the Distributed part of be hive so once an event gets fired off everyone in the system knows about that event most specifically all the routers know about that event because the routers are like the pyramid of be hive so If that router goes down all the other the other supervisors can start it on a different node somewhere else and You're fine. You don't have to deal with I mean you would want to start another be hive probably up in Australia or something if both Canada and the US go down just in case I Mean your server in Canada and your server in the US Yeah, I'm also in Texas so gotta watch it, right Any other questions Okay, cool. Thanks. There's my Twitter if you're interested in that If you're interested in email, there's my email and if you're interested in my name, there's my name And if you're interested awesome