 Excellent. Okay, well, it's about time and it looks like it's a pretty packed house. So, uh, wow, hopefully I don't disappoint all of you. Um, going after maths is pretty intimidating, but, uh, I guess I'll launch right into it since it's about time to start. Uh, so I'm here to talk about abstracting features into custom reverse proxies, aka making better lemonade from chaos. So, what am I actually talking about? I don't know if that title makes sense to anybody else, but what I'm here to talk about are reverse proxies to start with. So, just to make sure everybody's sort of on the same page with knowledge and, you know, familiarity with everything, I first just want to lay out some basics of, you know, what is a reverse proxy in case you're not familiar with what a reverse proxy is. Basically, a reverse proxy is a server that sits in front of your other servers, inside your internal network that sort of, um, does stuff. It does something in between your main server and the end users on the internet. So, what kind of stuff does that reverse proxy do? A lot of you in the Ruby community might already be familiar with reverse proxies. You might have already used one. A lot of you might use Unicorn to deploy your apps. And in that case, you oftentimes put InginX in front of Unicorn, and that acts as a reverse proxy. So in that case, InginX's role is to sort of serve static files and to deal with slow clients. So there, that's the role of the reverse proxy. But what I'm here to talk about is sort of taking that reverse proxy layer and doing other fun things inside of it. You can sort of implement your own layer there in Ruby. You can implement custom features. You can use Event Machine to do this, to do highly scalable reverse proxies. And that's a lot of fun. So that sort of is the outline of reverse proxies, but you might still be confused as to what this is actually all about. So first, I'm sort of going to go into the why of why we've sort of built these reverse proxies and why they might be suitable for certain use cases, and then also get into a little bit of how you would actually implement this kind of thing. So the easiest way to get started is to start with a story. And that story is sort of how we're using customer reverse proxies. And I think it's perhaps a story that some of you might be able to relate to in some way. So several years ago, we started building a lot of web services. And we wanted to expose all of our web services to the world, but another part of this story is silos. And I'm not talking agricultural silos with delicious coffee or sugar. I'm talking about organizational silos. So I work at NREL, the National Renewable Energy Lab. It's just down the road in Golden, so I'm local. Yeah, renewables. So NREL is a big company. It's, I think, I don't know what the latest count is. It's grown a lot. It might be around 2,000 employees, something like that. Not all of us are developers. There's a lot of very smart scientists there doing very cool research on renewable energy stuff. But we do have a lot of sort of separate development groups within our organization. So our group that I work in has historically done vehicle stuff, while other groups maybe do stuff related to buildings. And another group does stuff related to solar. So we sort of have these different departments that all sort of work on different things, but we were all sort of wanting to build web services at the same time and sort of present those to the public in a standard way. And another way to look at this is because we sort of have this spread of groups and different development groups throughout the lab, you know, our group happens to do Ruby stuff. We have some other groups that do Python stuff. We have others that do Java, PHP, et cetera. So there's a big diversity there. So even if you're not working in a big organization, you might be able to relate to this from the perspective of, you might have legacy apps that you deal with. You might have, you know, a lot of, there might be different languages in your stack. So there can be a lot of aspects to your stack, even in a smaller company. But from our perspective, we were dealing with sort of different departments and sort of we wanted to make all of our web services sort of come together. And so, and we wanted to present those to users as sort of a single offering. And really users don't care about this kind of thing. I mean, they don't care that our group happens to do transportation stuff and another group happens to do solar stuff. They're just interested in, hey, what does NREL have to offer as far as web services go? And how can I most easily find them? So what we really wanted out of this platform that we were building to expose our web services is we wanted one entry point for users to find all of our APIs. So we sort of wanted to bring together all these APIs that were being developed in separate groups in different ways and bring them together. And we wanted it to make it easier for the, easy for those users to access the APIs. We basically wanted to have the user sign up for one account and then they can access all of our APIs. So they don't have to get an account from our transportation group to access the Ruby services and the solar group to access the Python services and so on. And then from our perspective, on the development side of, you know, we have all these different groups building web services, we wanted sort of API key based management to access all of our APIs across the board. We sort of decided that's okay for what we're doing. We were doing pretty public stuff. We wanted rate limiting. That was something that a lot of our groups hadn't really explored or implemented within their individual APIs. But we wanted that, that was a need across the lab. And we also wanted analytics. Again, another thing that, you know, other groups hadn't really tackled and it was something that we really thought we could, you know, standardize and there would be benefits to doing that. So what we didn't want out of this were requiring changes to all of these different groups. You know, it's a big uphill battle to go into, you know, I mean, the one option is sort of standardizing for all of our APIs and saying it must be written in Ruby, which I wouldn't mind, but other people would. And so, but we didn't want to have that mentality of saying you have to do it this way if you want to be part of our cool club of APIs. We just wanted, it's a big battle to make that change within different groups. And for historical reasons, you know, there are the separate groups. We're trying to do better to work closer together, but I'm sure you guys can empathize, hopefully, with sort of those big organizational structures or just, again, even if you're not in a big organization, sort of dealing with legacy apps and sort of this diversity even within your code base. And it's also time consuming to have to go back and touch all of these other apps. So that's where customer-versed proxies come in, to the rescue. And so what does this look like? So if we look at our diagram again, it's at that same stack, at that level of the reverse proxy, we were able to implement custom features like authentication, rate limiting, analytics. Those are the things we decided we needed across all of our APIs and then they can be shared among all the back-end servers. So it sort of just slips in there without having to make any changes on any of the existing APIs that were already built. And it implements that common functionality. The existing APIs don't have to change. They can still exist wherever they wanted to exist. And the proxy is just agnostic to what sort of back-end technology is happening. But it sort of provides that unification, that single entry point to all of our APIs that makes it a lot easier to apply a standard authentication scheme across all of our APIs, do rate limiting across our APIs. And analytics. So that sort of is the basics of what we've done. So how did this really help us? Did this really make better lemonade from chaos? So from the user perspective, I think it did. It helped our users. We were able to build one website where users can go to and find all of our web services. And they only have to sign up for one API key and they can get access to all of our APIs. And so they're sort of shielded from sort of the internal... They don't need to know what department does what and we have different web presences throughout the internet on various .gov domains. So they don't have to know where to find stuff. There's just a single website for renewable energy APIs that they can go to and start to find this stuff. And it's easy for them to just dive in and start using any of them, no matter what all is happening on the back end. And for our developers, the real advantage is that for old APIs, they had to do absolutely nothing. So we had an existing suite of APIs already out there, but they were all sort of all over the place. But they did absolutely nothing, but just sort of by existing, we were able to put them behind this reverse proxy and then we were immediately able to start layering, authentication, rate limiting, and analytics on top of it. And the same goes for new APIs. When somebody is building a new API, those are just things they don't have to worry about. They just sort of assume that all of that is taken care of when they're building a new API. And it's just outside the scope of it, they just say, if I'm being accessed, I'm assuming the user is fully authenticated, they haven't exceeded rate limits, and so on. So that's, I think it's been also advantageous for us, just from a development perspective, of speeding up and not having to reimplement those same details and so forth. So yeah, so it's led to reduced implementation code because we don't sort of have to, individual APIs don't have to implement the same sort of logic over and over again. And there are definitely ways you can abstract this, and there's definitely ways you can reuse code in a clean way, but it just reduces the need to do any of that. There's really no code involved to implement these features at the individual API level. There is code obviously at our custom reverse proxy level, but that's a little easier to maintain. And the nice also thing is that standardization is enforced across the board, because we could abstract this into some sort of library and things could reuse it, but we don't run the risk of somebody messing up authentication within their individual API. By putting this sort of at a higher level, it sort of enforces that that's gonna happen. So the other advantages is that because of how this operates, at the layer that it operates, any new features we add to this reverse proxy layer benefit everybody. So you know, whether or not they're Python services, Ruby services, PHP services, Java services, this architecture benefits everyone when we decide to implement new functionality. Obviously not all functionality is suitable for this kind of thing, but it could be a powerful mechanism for certain types of functionality that can be layered like this. And another thing is that just reverse proxies in general are a nice scaling mechanism. A lot of times they're just used, you use reverse proxies as load balancers. So sort of having this in place and sort of getting everybody on board with this architecture of having reverse proxies up front allows us a lot more flexibility on the back end to sort of scale things independently. So that's sort of the basics of what we did. On to how we actually built these things. So yeah, how would you actually take Ruby and do some custom magic stuff at that layer and do it fast and efficiently? So currently we're using EM proxy. It's a nice event machine proxy library and I certainly can't take credit for any of this stuff I'm about to show code wise. We're just users of it, but it's open source, it's nice, it's easy, and it's out there. So EM proxy is Ruby and event machine and if you're not familiar with event machine it's just sort of an event-based system for writing Ruby code in an event way, evented way. So if you've heard all the stuff about Node.js it's similar in architecture to that but Ruby. So it has some nice advantages of being blazing fast. It's also flexible and it's low level. That low level aspect has its pros and its cons but I'll get into that in a bit. So a very basic example, this is sort of what an EM proxy looks like. So it's pretty basic, but you sort of set up a server so when you run this script you basically start up a server, in this case it would be running on your current server, listening to all the IPs running on port 80. And then in here we say we're going to proxy to the same server 127.001 on port 81. And really that's all you need to do to do sort of a transparent proxy. But the real power of this is these callbacks that you can do. So you have on data, on response, on finish and I think there's even one other one. But basically it gives you a lot of flexibility as far as intercepting chunks of data as they stream through the proxy and doing stuff with that. So you can do something as the data for the request comes in and then you can also do something to the response as it goes out. And then you can also do cleanup stuff on finish. So as a quick example of what this might look like this is an on data callback where I'm modifying the user agent. So the user agent comes in on the request and basically I'm just doing a search and replace replacing any existing user agent with the user agent of EM proxy. And you'll note that I'm doing a search and replace but again you're dealing with sort of chunks of data you're dealing with sort of the raw HTTP at the raw HTTP level here. So you have to be a little careful but you'll see here that I'm actually searching for user agent and then two new line characters that is how the HTTP headers work. So there's some things you have to be aware of it's not quite all simple but the other thing to note is that you're dealing with chunks of data in this case so you're sort of you're getting a stream of data as it comes through in chunks so you can't always assume that you have like say the full request in here so you can't just do a search and replace and assume you have all the data you sort of have to deal with there's other things and I'll get into a bit of that as far as buffering things if you need to have the full request. So that's all well and good but you could do that kind of thing with any sort of reverse proxy or most of them to do sort of a change the user agent that's pretty basic change a header do something like that at the reverse proxy layer here's something that you know now because this is written in Ruby you can start to tap into all these Ruby libraries so that's the real advantage of sort of this approach and implementing custom stuff so here's an example of I'm setting up I'm connecting to Redis and every time I get a chunk of data I'm incrementing the IP address I have a counter for that IP address in Redis so again this is a little taste of what you can start to do because it is Ruby you can just sort of start to write things and access your Ruby libraries and it can be a lot of fun and as I mentioned you're sort of dealing at a low level here you know you basically have raw HTTP strings so if you want a higher level interface into HTTP it's sort of is up to you to do that yourself and there's libraries to do that and here's a very quick example of sort of as you get data on data so chunks of data are coming in you pass that to this HTTP parser library and then once it's determined that all the headers have read in or there's other callbacks on the HTTP library you can then once the headers are completely read in then I am accessing that user agent header as a Ruby hash so this is sort of if you want that higher level interface into the HTTP request you sort of have to do that yourself but there are libraries to do that kind of stuff so you might be asking why would I do this this sounds sort of like a pain to be dealing with things at this low level and having to deal with stuff like parsing HTTP yourself a lot of us might do web development a lot of us might be used to nice high level frameworks like Rails and Sinatra and things like that and so there's a few reasons why you would go down this path a big reason is transparency at this reverse proxy level at this proxy level implementing it this way you are dealing with the raw HTTP request but that gives you a lot more flexibility to sort of pass that request on to the back end in a completely transparent manner so it's not apparent that that there's something in there in between doing something because if you would try to do this with something like Rack or Rails or Sinatra by that point by the time your application has been hit the web server has already taken in the HTTP request so it becomes difficult if you then want to recreate that HTTP request to send it to a back end because by that time you don't really have access to that raw low level stuff so it can be hard to sort of you sort of have to manually try to reconstruct the request and there's a lot of edge cases with HTTP stuff that it makes that hard to deal with so the other reason why you would do this is purely just speed and efficiency you know higher level frameworks are great we use them a lot but at this kind of level we really want things to be very fast and very efficient and scalable and event machine is very fast and evented systems are very suitable for proxies that's why you see a lot of proxies being built in Node.js this is similar in concept they scale nicely and efficiently and there's a few reasons for that I'll throw up some terribly unscientific benchmarks just to give you a sense I just ran these on my computer there's probably lots of things going on here but the basic thing is that basically I benchmarked making a direct request to a back end server and then making it through EM proxy event machine proxy and then making it through a proxy I just found I didn't know much about it called rack reverse proxy that basically does take a higher level approach to proxy and you have a lot of nice access to sort of the parse HTTP request so in this case I mean it's already fast it's one millisecond EM proxy adds 0.5 milliseconds rack reverse proxy adds almost three milliseconds so I mean three milliseconds maybe not the end of the world who really cares it starts to get a little the picture becomes a little more complicated once you start to get into bigger requests and this is partially determined by your needs and so here's an example of a larger request where there's more data involved and here EM proxy adds 150 millisecond and rack reverse proxy adds 800 milliseconds so why this is the case is that rack sort of deals with a complete request and complete response so it sort of takes in the full request then it tries to recreate that and send it to the back-end server gets the response from the back-end server and then sends it along to the original client so it sort of has to buffer all of those in memory so you know if if you can imagine that you're uploading a one gigabyte file through some web service and you need to download five gigabytes I mean that's a lot of data but there are cases where this happens with rack reverse proxy that sort of is bottlenecked at that and it would read that into full memory at each step and again there are ways around this but it sort of is the difference is EM proxy deals with things at a chunk level so you're sort of dealing with just chunks of data at a time and you can just stream them on to the back-end as fast as you're receiving them so as soon as you have enough information you need to have to make your decisions that you want to do inside your custom reverse proxy you can just start streaming that data very quickly and then stream the data back so there isn't that required buffering and so that sort of gets to another aspect of this which is just flexibility of EM proxy and in line with that it's low level and it's up to you to implement more but it's up to you to decide if you do want to implement something like buffering or if you want to stream everything live and another thing you can use EM proxy for is non-http things so you could use it for sort of any tcp level things so you could do it with web sockets mail servers database protocols you can sort of put it in the middle of anywhere and any of those sort of types of connections and potentially do custom things so what else could you sort of do with these custom reverse proxies I've talked a bit about what we did as far as tackling some of our issues with APIs and implementing authentication rate limiting analytics and you know I think those are all really good candidates for something that could be implemented at that level you know because you do have to be careful this you know you can't really implement your it's not suitable to implement your whole application at this level but it sort of is those high level features that you know you might have potentially diverse backends or even if you don't have diverse backends it could be an interesting different way to structure your application so what else could you do with this I don't really have any concrete examples but I'm just throwing out ideas here you could do error handling so you know you could monitor all requests coming back for a non-success HTTP code and then you could do something with that you could log that you could send emails and you know a lot of us do that within our Rails apps but you know if you're maybe dealing with we have legacy Pearl applications I'm ashamed to admit but we have to deal with that and you know we don't sometimes have those mechanisms to deal with sort of that error handling in the same way in those applications but you could sort of slip in something that deals with all your errors across all of your applications no matter how they're written and again they're pros and cons you might not have access to all the debug details at that level but you could certainly you know be notified about requests that failed you could even do this to sort of manipulate web pages you know say you wanted to insert your Google Analytics snippet on all of your pages you could actually do that with a custom reverse proxy you could sort of take the response as it's going back sneak something in there you know you could even do crazier stuff with like taking all your JavaScript files compressing them on the fly and then returning it as one JavaScript file to the client and we do that in Rails but again if you have diverse backends and maybe don't have those capabilities you could do that at a higher level you could you know maybe do your template in your web page template put in a header and a footer I don't know why you do it at this level but you could you could also do things like say you had a bunch of JSON APIs already and you know maybe you can't touch them for a variety of reasons but you wanted to add JSONP you could sort of add that at this level and then all of because JSONP is just sort of a matter of wrapping an existing JSON in a callback you could implement that at this level by altering the response as it goes back and it would apply for all of your existing services you could do things like check for security things see if incoming requests look malicious in any way for all of your servers do stuff like that and again you can sort of do more than HTTP with this you can intercept and sort of manipulate all sorts of TCP things so you could do email database calls all sorts of fun stuff and there's a lot of great examples in EM Proxies GitHub repo there's an examples folder just filled with sort of interesting ideas of what you could sort of do so again I don't want to take credit for any of this stuff this is sort of us just repurposing a lot of this stuff so there are a few things to be aware of when you're building these as I've sort of hinted one of them is buffering so and again I've already sort of talked about this but imagine you have you know one gigabyte upload and a five gigabyte download if your proxy layer buffers that request that becomes sort of a sort of a bottleneck but it also becomes a place where as the request gets uploaded it all has to halt until it's fully uploaded and then it gets sent on to your backend server and in the case of a one gigabyte upload or a five gigabyte download that buffering can add significant delays and so sometimes buffering is desirable though and sometimes you can't achieve stuff without buffering and you know for example Unicorn actually wants buffering to deal with slow clients so it can be advantageous but other times it's not and in our case we're we're dealing with a diverse set of APIs that we don't really know the use case of all those APIs we opted not to buffer because we just don't know what all those backends are doing or if they want to stream data and we didn't want to prevent that streaming from happening at our proxy level because we just wanted to be as transparent as possible some other things to be aware of at this reverse proxy layer is that if you are going to modify the response going back to the client you can do that but it can be a little tricky you in the header going back to the client there's usually a content length header and you have to adjust that accordingly if you're going to do any manipulation so again this isn't the easiest way to perhaps alter your website but for certain use cases I think it can be powerful and in line with that another thing to be careful of is gzip responses going back to the client so if you're a back-end server decides to gzip something up send it back to the client if you want to alter the response body in that case it gets tricky again because you sort of you do have to buffer in that case because you sort of have to get the full gzip because you can't just gzip the different un-gzip the different chunks you have to get the full response body buffer it un-gzip it do whatever you were wanting to manipulate and then read gzip it and send it back to the client so those are just some things to be aware of that we've just sort of learned as we've gone through this process so now I'm going to talk a little bit about some other stuff so this is all a little good but I don't know if you're interested in bigger stuff so this digital strategy for the federal government came out earlier this year May 23rd 2012 apparently and so I'm involved in the API portion of sort of some of this digital strategy and a big part of this digital strategy for the entire federal government is just a web services bonanza I mean it's just like everybody should be doing web services web services are great all agencies should be you know delivering data to people in the form of web services and I I would tend to agree with that I mean I think you know as app developers it's always nice to find data out there that's just in an open web format in an API that's easy to use and get access to and you know the government has a lot of data but it's not always in the most easily accessible format you know I've seen stuff where it's like it's a printed Excel spreadsheet that's been scanned in and then put into a word document and then they just give that to you and it's like I'm providing data not really so a big part of this push for this federal strategy is to encourage agencies to develop a lot more APIs and web services so expect that sort of within the coming year to see a lot more government data out there so that's exciting and of itself but the portion that I'm involved in is sort of the API thing and sort of the main objectives there is you know there is this big push for web services but they sort of want to tackle two things on the one hand they want to make it easier for users like a lot of you to find and consume federal APIs there are some out there already but a lot of times they're not easy to find you know you might be interested in some data but it's just hard to find through the bureaucratic websites that are government websites and the other aspect of this is that if they are pushing agencies to develop more APIs they they need to make it easier for agencies to develop and deploy more APIs so you know if there's this big push for APIs it just needs to become easier a lot of agencies you know a lot of them are ahead of the game and they're already building APIs they're doing great work but some of them are sort of they need help with this kind of thing so in a lot of ways this is the exact same problem we had within our organization at NREL just on a much bigger scale you know there are silos of organizations there's different agencies that are all doing things independently but a lot of the stuff that needs to be addressed are similar issues so it's sort of mirrors our same problem so what we're looking at right now is is it sort of the same solution we're currently evaluating using basically our software stack that we developed at NREL to sort of proxy to all the agencies within a federal government or possibly other solutions that sort of do similar things but we've been talking to a lot of different agencies and the consensus was they want something like this they don't want to have to deal with authentication on their own you know they want and from the federal level and from a user level they want to make it easy for users to just get one API key and to be able to generally access all the federal API so you don't need to have a bajillion accounts for all the different agencies so so yeah I mean it's sort of the same issue sort of I perhaps the same solution and so yeah we're currently involved in getting something like this up and running perhaps within the next six months for federal agencies to start taking advantage of so that's all exciting there's lots of federal there's lots of web service action going on in the federal government over the next year so stay tuned if you're at all interested in all that so I'm starting to wind down here but I have some more slides to go through so what has all this been about really so to sort of summarize you know what I really want to encourage here is sort of a different way of perhaps thinking about some of your architecture again I reverse proxies can't solve all problems they're not suitable for all problems but I think they can be an interesting different approach that I don't see utilized as much to solve certain problems because they're fun for the whole family you know anything you do within the reverse proxy layer affects all of your back in applications so as you start to add new features to your reverse proxy layer it can be advantageous for everyone so it's an interesting way to sort of abstract things completely outside of the application level into a higher level that sits in front of all of your applications and you know what I want to encourage you here is you might be able to do more with reverse proxies than you realize you might think of reverse proxy just is you know a software like IngenX or HA proxy that sort of just does proxying but there's not a lot of logic that can happen in there or rather like implementation details that can happen in there but since you can write these in Ruby you can start to leverage all these libraries you can connect to databases you can do all sorts of crazy fun stuff at that level so yes again just sort of think about it is a different way to perhaps architect some of your applications so I'll go through some just random resources that might be useful one of them is API umbrella this is our full API management system we've built at NREL and we've just recently finally got approval to open source it so it's all out on Github we're super excited to finally open source the project and includes our custom event machine based proxy so you know even if this is even if you're not interested in APIs you might check it out as you know what you can do with what you can do with that reverse proxy level and how you would implement some of those details so it's github slash NREL slash API umbrella and it's a new open source project for us so we're sadly behind the times as far as getting all documented and everything but definitely reach out to me if you have questions about any of that as far as just Ruby and event machine just low level proxies there's EM proxy which I talked about and those are the examples I've shown it's just sort of a simple bare bones but very capable reverse proxy there's proxy machine which is actually what our current production system is based on it's a github project it's simpler the one and and it can be easier to get started perhaps but the one disadvantage is it can only act on incoming requests it doesn't track the response going outwards so it's sort of just does incoming stuff and then does routing it doesn't keep track of the response coming out so you can't keep track of some of the stuff you can in EM proxy so we're currently in the process of switching over to EM proxy and then there's Goliath which is another thing that's based on all this event machine magic and it's more of a sort of higher level framework it still is pretty low level but it then uses EM synchrony which uses some fiber stuff so you it actually hides all the event stuff from you so that's an interesting project to also check out as far as just general reverse proxies you might be interested in we use there's a lot of them out there that certainly is an exhaustive these are just sort of the ones we use in variety of capacities there's HA proxy HA proxy is amazingly fast and scalable and it can do all sorts of fun load balancing stuff and it acts as a great general proxy varnish cash is an interesting one it's a reverse proxy but it also is a caching layer and so that sort of gets back to sort of we're actually going to be adding this to sort of our stack we're going to slip this in as another reverse proxy in our stack and then the advantage there is that some of our older apis that maybe don't have as good of caching capabilities as some of those stuff built into rails does can start to use the varnish caching server as a caching layer and varnish is nice just for caching all sorts of stuff and then there's nginx which is really more of a web server but it also has some pretty nice proxying capabilities it isn't as exhaustive as something like HA proxy but if you're already using it it can do quite a bit of stuff and so if you happen to be interested in renewable energy apis this is our site that we sort of built that sort of this is what this was all about is sort of making one website for users to find all of our apis even though they happen to be on all sorts of different servers within our organization and so you can check that out at developer.nrel.gov and there's lots more apis coming soon some of my colleagues sitting down here in the front are working hard way at building new apis so there's a lot of cool stuff in the pipeline and so those are some contact details for myself I don't really use Twitter I'm sort of a curmudgeon but feel free to contact me maybe I'll start using it and finally this is a completely off topic and a shameless plug but if you've been wondering about this ridiculous thing on my upper lip we just finished our mustache competition at work this week and we have a mustache competition every year for charity and so we haven't met last year's goal but if you're interested it's a local denver-based charity to help support kids and education resources so if you're interested bit.ly slash ruby stash you can donate but anyway I think there's actually some time for some questions because I was cooking yeah that's a great topic thank you for sharing I have a question how do you deal with inter-services communication so if what about the does it go through the proxy as well yeah so for everybody else the question was how do we deal with sort of inner process communication you know so if one web service needs to call another service does that travel through the reverse proxy as well and the answer is in our current system yes it isn't strictly necessary on the back end we definitely could communicate directly server to server if we wanted to save the overhead but I will say this reverse proxy is very fast I mean it in benchmarks I think it adds like four milliseconds to deal with the rate limiting and analytics and user authentication and that utilizes Redis and MongoDB to do all that so it's fast so for the time being we haven't really seen any problems with routing those sort of inner service communications through the proxy and that sort of just gains us some advantages as far as analytics mainly just because we're interested ourselves to know internally how we're using our APIs yeah sorry just the same way so how do you deal with off the idea of the concept of sessions or how does it work so the question was how to deal with authentication then yeah so we still use the same our authentication is admittedly very simple it's all API key based so it's just a big long API key and then on the back end we have capabilities to remove the rate limits from certain API keys so basically we set up those back end things as unlimited API keys so we have API keys that we send back and forth and that's what we use for authentication and that sort of identifies each app so again for analytics yep hello how do you deal with the fact that your the pattern the text you're looking for may be split between blocks do you have a facility to offer adjacent blocks or something like that yeah that's a question so for the first part of that question that you might have missed was how do we deal with sort of sense things are coming into this proxy as chunks and you know if we're doing something like a regular expression how do we make sure that it doesn't span chunks and we don't have that data so the answer there is first of all I wouldn't really recommend doing regular expression searches sort of for that reason that was sort of a very simple example but that is why you would use a higher level library like that HTTP parser library that then is capable of knowing when all of your headers are read in and then you can take some sort of action on all the headers or once all the body is read in but you're right I mean in that case you do sort of need to buffer and that's why if you're doing anything with the response you there are some ways you can get around it for very specific cases but for the most part if you're dealing with the response you have to buffer and then if you're incoming requests sort of the same deal if you want to do anything with the body and yeah yep you can take a look at our github repo it's probably not as well tested as it should be so yes there are I mean I would say I would say slightly yes I mean we I would say the testing right now tries to break it down on a it doesn't really get involved in the event machine sort of level it sort of is just testing the more basic blocks within there so yeah we're probably lacking on tests for sort of the whole thing running as a whole with all the events going crazy so yeah testing is more of a challenge with the vent machine stuff Apache does do it yeah and Apache can be used I would say it's probably not as scalable as a lot of these other solutions it depends on how you're running Apache whether or not you're sort of the it's been a while since I've used Apache but the worker model versus the other model whatever that is by default Apache will sort of spin up new processes for every single request whereas I think all of these other ones HA proxy and engine exit at least are event-based model so they're a lot lighter weight so Apache will work the other ones possibly just a little more scalable but again if you're already using Apache go for it I mean I don't want to tell you not to use it yeah yeah so we oh sorry yeah the question was of the previous question I don't know if I repeated that was about using Apache if you couldn't pick that up this question was about how we deploy our proxy and if we deal with any sort of missed requests or any of that so to deal with deploying we all of our deployment stuff runs through Capistrano but within that how we're really deploying that and dealing with that issue of we do basically do a rolling restart all of these we basically run I think three of these processes or four of these processes it's sort of is the same models engine X where you run as many as you know cores as you have on your CPU so we have you know several of these processes of running and then we do do a rolling restart we do that through a Python project called Supervisor it's a pretty nice library just it's not really Python related at all it just happens to be written in Python but it's a nice process management library that you can deal with sort of treating things as process groups I know a lot of people might use like Monit and I think there's some other blue pill God I think there's a few other those I've we've been I think probably more happy with Supervisor and you can do so we basically do we did a custom implementation of a rolling restart with Supervisor so we stop one and start another I think there might be some edge case stuff for if somebody's in the middle of request I'm not sure if we're completely gracefully shutting down right now but I think actually part of proxy machine in EMProxy does should handle graceful shutdowns so theoretically that should happen without any loss requests any other questions oh what perfect timing well thank you guys very much I hope this was okay