 Good morning everyone and hope everyone's having a great time at Drupal con on the last day and you guys have been filling Your heads with awesome Drupal knowledge eating great people as well as enjoying some of the best that Portland has to offer So we're gonna get started. This is a panel case study weathering storms how Robin Hood uses Drupal to fight poverty and I'm gonna be introducing the project team as well as myself in a bit, but I'm We're gonna get started We also have a website Robinhood.org and a web page slash Drupal con and which has some social media links for Robin Hood foundation as well as We've got a hashtag P2 Robin Hood if people have questions or want to connect with each other during the presentation We can use that and we've got a Robin Hood. I might see and at phase two on the Drupal con site We have a follower counter at the bottom of the page which has the numbers of people that are supporting Robinhood to fight poverty as well as tracking Twitter followers So if anyone falls at Robin Hood NYC during the presentation, we can see that number go up So we're at a 1 6 2 200,000 230,000 164 right now so Moving on to introductions. My name is Molly Burns. I'm a project manager at phase two technology And I joined the Robin Hood team at phase two right after phase one of the website launched And I've had the opportunity to work with Heather Johnson who's the senior application manager at Robin Hood foundation Who's the primary contact that we have working with all the stakeholders at Robin Hood? working on the web and Brian McMurray It's a software architect at phase two. He's worked on a lot of the front-end architecture for Robin Hood org Steven Merrill Is the director of engineering at phase two technology and he's also one of our server hosting gurus So he'll be talking a little bit about web performance In later on in the session and last but not least we have John Robert Wilson Who's working on the software architect of phase two and he's been doing a lot of the back-end architecture for Robin Hood So I'm with that Heather's gonna jump in and talk a little bit So first of all, we're really excited to have everybody here today. Thank you for coming to our session We want to tell you a story about the Robin Hood Foundation and How phase two and our partnership with phase two helped us solve some pretty Significant challenges that we had with our website and with our organization First of all, though, I want to give you a little bit of insight into the Robin Hood Foundation itself Robin Hood isn't really that well known Mostly by by design Robin Hood tries to put all of its energy into publicizing The efforts of its grantees and stays kind of behind the scenes But in general what we are is foundation based in New York City. His primary mission is fighting poverty and There are a few things that make Robin Hood, especially unique relative to other foundations of its kind One of them is that out of almost 30,000 poverty fighting focused organizations in New York Robin Hood actually finds and funds about 200 give or take of those So right off the bat, we're looking to find the very best grantees and by very best We mean the most effective as measured by hardcore concrete metrics so Robin Hood takes a very specific kind of approach to Measuring the effectiveness of its grantees and puts the burden on the grantee to show that they are actually Delivering on the mission that they set out to perform Another thing that makes us really unique relative to other foundations is that we give 100% of the money donated to the foundation back to our grantees And this is something that is possible through the generosity of our board which pays all of our operating costs Robin Hood has no endowment. So if you give five dollars to Robin Hood today Your money will immediately go into the field within a very short period of time We don't hold on to the money and distribute it years later And we'll talk a lot more about this later But as an example of that with Hurricane Sandy relief which Robin Hood took a leading role in We collected around 70 million dollars in the service of Sandy relief all of which has now been distributed to our grantees We do expect a lot of these grantees But we don't put them in a position to perform all by themselves. We ask Our grantees to show performance. We ask them to show success, but we also help them So we have what we call a management assistance program Where grantees can learn to do things like make the equivalent of a business plan If they need help by way of legal services even technology services We have people on staff who help them with that stuff We help staff their own boards And we help them to basically turn around and show us the very things that we're hoping to be shown by them The the measures of those successes So just a couple of quick things about Robin Hood despite are not really being all that well known Particularly outside of New York City. We are New York City's largest private poverty fighting organization We're also the largest private funder of emergency food in New York City We are the largest private funder of job training programs in New York City And as well as as all of those services We have provided a first-class education to over 45,000 children in in the city Robin Hood is the largest private funder of homeless shelters in New York City It also provides the largest amount of assistance in the private sector to programs that help newborns Despite having that kind of breath Robin Hood about two years ago say mid 2011 had a number of significant challenges with its website For our organization the website is critical. It doesn't get a whole lot of traffic Well, at least not on an ordinary day-to-day basis But it is an extremely important marketing tool to target very high-end donors and as such it needs to be slick It needs to work. It needs to be effective in communicating our message Unfortunately, it did none of these things two years ago And today we want to talk with you about three very specific Challenges that our website had to meet and that were effectively met as we partnered with face to technologies The first of those is just the challenge of unleashing our editors in 2011 we were on a ten-year-old dot-net based site With almost no ability to edit anything without the assistance of a developer that had to change Robin Hood hires the best in terms of its digital staffers and It wasn't cool to allow them to have their hands tied in the way that they were in mid 2011 in addition Robin Hood is frequently engaged in very very large-scale events many of which require Unique branding and a completely different approach to messaging our website Basically couldn't be used to serve any of those purposes any time we did a major event We had to create a brand new microsite from scratch, which was extremely not cost-effective and and very time-consuming and Finally while I did mention that we don't get a lot of traffic on a day-to-day basis There are some extremely significant exceptions to that Robin Hood is about the spikiest organization I've ever heard of in terms of traffic and so while it's true that day-to-day our infrastructure needs were relatively You know uninteresting On occasion we really really needed to step it up and we wanted to do that in a cost effective But also very effective way and so that was our third challenge something we approached face-to-with in the fall of 2011 so with that I'll turn it back over to Molly who will tell you a little bit about face-to-and then we'll hear from our team as to how we met these challenges These two is a general strategy and development agency we're basically solving a lot of problems for our clients and working really hard to understand the business as the organizational challenges and really making sure that we can deliver solutions that are really innovative and Provide a lot of high value and so With that we're going to talk about some of the challenges that Robin Hood had came to face-to-with and we worked through a discovery process initially Which is sort of when we talk through and do a lot of interviews and discuss all the issues before we actually go ahead and build something and Out of this process came Robin Hood.org So challenge number one for us was about unleashing our editors as I mentioned We were in an extremely limited position when it came to making changes to our website specifically our site header in 2011 was all flash and We had no no concept of anything like a content type if we needed to do anything new with our data Or we needed to display it in a particularly new way We were either out of luck or we had to employ the services of a developer our content team had their hands practically tied When it came to retheming changing the look and feel of our site. It was a similar story We really were none not in a position to do that either at all Depending on the timeline requirements of our project or we had to go outside of the scope of our team To hire development firm to create a brand new site for some specific branding effort So we were definitely in a bad place Performance wise we had a single web server with a single database sitting on physical hardware at rack space It was adequate most of the time But once again completely inadequate for certain special needs that you're going to get to hear more about momentarily So our first challenge we want to tell you about today is this business of unleashing our editors and giving our digital team Back the control that they deserved of our site. So Brian's going to tell you a little bit about how we did that So I'd like to take you back a few years if you remember about two years ago Drupal 7 was just getting out of the gates and You've probably seen this graphic before this is from a few years back This is a picture of how a Drupal developer might think about a page when you're when you're Considering building something out. It's a mixture of you know views slideshows or views listings blocks and node content and Menus and panels and all these sorts of things and each of these then and also still now really has Many different interfaces to to work with There's not one consistent editor experience You need to know a whole lot of the Drupal jargon and methodology to really work with this method And when we were talking with Robin Hood, you know, they just didn't Know what to do with this. They weren't familiar with Drupal. They hadn't been working with it for years Like some of us have when they thought about Content, this is what they thought Everything is content and they want to be able to change and edit and adapt everything very easily with a consistent experience in addition As we started to see the designs that they are working through They started to look like this these highly designed pages that don't really fit this the common metaphors of sidebars and content areas and You know, perhaps you might be wondering well, couldn't you just use panels? Well panels two years ago was a little hard to use and didn't quite work with revisions and workflow And these were things that we were concerned about We really wanted to build a system that would let Robin Hood's editors be very free to create Anything that they could dream up with their design agency or just on their own with their own creativity And in addition a number of these things that they were starting to think about were interactive content So areas of these graphics weren't just static images. They were actually going to be Interactive pieces that you could enter, you know You could click on and touch on your mobile devices and and play with and in the past You've probably thought about interactive content and you think okay We need to build You know something special in JavaScript or we need to build something special in flash and in order to get Data into it we need to build a whole separate Area inside of our Drupal site to manage that content and or maybe it's just going to be an XML file and in order to update the The data that goes into that flash piece, you know, we need to like update an XML file on the server somewhere That's not what we wanted to do and that's not how they wanted to work with it That they found that a really frustrating experience on their previous site So we created this thing that Jair is going to tell you about a little more called template field which helped us to to work through this create super flexible content areas and Even enable us to create interactive content areas that all have a similar and consistent editor experience The way we put this site together we called a constant, you know, we reuse the term pain Pulling it from the panel's world But we called these pains these sort of content chunks that were going to be Reusable and customizable on a on a per page basis and then we assembled them together and you know another concept We called pain stacks, which was basically just a bunch of content chunks Brought together and and Displayed as one consistent page Cool, thanks Brian. So I'm gonna talk about beans for a second. So out of the box There is a different workflow process between blocks and nodes and what we were just talking about People that actually use the sites that we build for them considered these things as content They're the same thing in their head But out of the box there are two different workflow processes for creating and placing blocks and creating content So there's this module called bean that was written by Neil Hastings and maintained maintained by a lot of other smart people right now There's the link for you that have your laptops are right now and what it does is introduces Things these things called bean entities these block entities aren't nodes and It bridges this this gap to this disconnect between these two pieces of Drupal media That should be that end users considered content and it Makes the editorial process behind managing blocks a lot more similar to managing nodes So bean adds an editorial UI similar to the node content creation to your Drupal site So they're entities so they're fieldable and typable just like nodes are fieldable and typable like content types So there's one more disconnect though a lot of site editors And a lot of site builders generally place blocks using context, but context isn't the most site end user friendly tool So to bridge this last piece of disconnect we use a module called embeddables and embeddables adds a Button to your WYSIWYG editor that allows you to embed Beans or other blocks you've defined in the system directly into your WYSIWYG body So an editor can go in create a beam and when they're editing their panes their nodes They can place the blocks directly into the content without and it's one workflow. It's all content to them And it's all one process So Brian mentioned the module earlier called template field and We used template field to give the site editors a choice on what layouts They wanted to use per pane in each pane in the pain stack And so we I've got up here on the screen as just some examples of some templates that are available to our site editors You'll notice that the top four are layout specific. So we have you know 5050 columns 6633 columns three column layout, but you can also see at the bottom We also give the editors the ability to have interactive pieces So template field doesn't necessarily have to be layout specific. It can be media specific or interactive specific It's always up to you to decide what those templates are and what the markup is For when they choose a template and they use it. So just to show a little example of what it looks like to the end user this is just What your standard two-column layout it creates a field set so you can see there's a top middle left middle right and bottom and Just provides a simple whizzy with text area and the editors can use all the predefined styles and the site scheme and the site color scheme to provide to make nice rich Simple layouts, but then in the end that they're all laid out with the correct template that the editor uses and to talk a little bit more about These text editors and and how we use these whizzy we get editors to empower editors I'll let Brian talk a little bit more about that So you're probably thinking whizzy wigs What are they doing with whizzy wigs Karen McGrane was talking all about how terrible whizzy would can be And you're probably thinking just like I was thinking a few years ago When we use a whizzy wig all we get is a whole bunch of junk We really didn't want to do that But we really wanted to give our editors a lot of power and to control everything from font choices and a Color to you know links and adding images and adding these interactive widgets these beans so really we we tried to Change that and we worked a lot with the whizzy wig API to create a whizzy wig editor that kept Editors on brand on message with all the tools that they needed to quickly get to you know brand style guidelines But also give them the freedom to create within that system So again, here's a picture of one of these panes or these template field areas that we're Using on the site these content chunks There's a lot going on here And this is only one example of many different types of designs that need to be functional within the system The way this gets put together as J.R. Was showing before quickly is We've got a number of different whizzy wig spaces within our template panes And in each area Continent editors can just go in and start working We use a little bit of extra JavaScript to change the background color of our whizzy wig editor when they pick a background color for their Template area so that they can actually see this and something pretty close to what's going to actually show up on the front end in addition, we use the whizzy wig API to Go in and change the default options of things like the color picker so that their initial choices are Immediately only the brand approved color set So they don't have to mess with you know hex colors and trying to look those up and find them again If they need to go off-brand there is still the ability to do the more colors option to you know get into a color picker and actually You know pick something totally unique and we'll talk about cases where they surprised us with this in a little bit In addition, we provided a whole bunch of default styles into the whizzy wig That allow them to just select their text and pick a style that puts them right into the correct styling for a Header or an area with a special icon to call out text or a call to action to sort of link and All of this together gave our editors a whole lot of power to you know create and be creative within the content system that we set up and Still stay very close to the brand guidelines without You know forcing them to use hundreds of fields to set colors and You know write little text and and lots and lots of extra theming They've got a whole lot of flexibility here and almost never do we you know We as the developers of the website have to get involved to to fix something for them So having spent all this time Setting up this fantastic back-end admin interface for our editors You can imagine that when we got finished with this project. They were pretty happy They were in fact ecstatic They felt for the first time like they could get in and really get their hands dirty and take control of the site in a way that they hadn't been able to do in a decade and so they were really happy, but I'm sure that everybody has worked in context like this before the happiness the complacency it never lasts and We had a digital director who loved this But it also kind of created a monster in the sense that he wanted just that much more and So we were thinking about exactly what more meant when at the end of October of last year Hurricane Sandy hit and suddenly necessity was the mother of invention So I'm sure everybody in the room knows the story of hurricane Sandy It was one of the worst storms to ever impact the East Coast and it had particularly huge effect on New York, New Jersey and Connecticut Robin Hood as an organization generally limits its Charitable efforts to New York City proper and in the outer boroughs But for the first time in its history Robin Hood expanded its relief effort to incorporate counties in New Jersey and Connecticut that were impacted by Sandy Robin Hood raised in the course of Less than a month after Sandy hit almost 15 million dollars and that wasn't the end of it That was just in the first month But in connection with that effort the website played a critical role and one of the things That we found out maybe two or three weeks into the relief effort at Robin Hood Was that somebody had an even bigger idea than that 15 million dollars? Specifically the Weinstein group and Chase came up with the idea to create what would end up being the biggest music event in history the 12 12 12 relief concert and They wanted to make the Robin Hood foundation the soul been a factor of funds raised through that effort We quickly learned as the web team behind this effort that the website would be the vehicle For live streaming the event collecting donations for the event We had three and a half weeks to prepare for this and we were told to expect as much as 350 million unique users in five hours To make matters worse. We were told that the site needed to go through three distinct design iterations We would start out hosting it on Robin Hood org, but it had to have a completely unique look and feel relative to the rest of the site Within about a week. We would completely revamp that and give it a second design look and feel and then we would finally Launch with something that was unique relative to those first two iterations Once again this needed to all be done in three and a half weeks by our small team and the help of our excellent partners at phase two in The end and I'm kind of giving away the punchline here But we did raise over 50 million dollars in connection with this event for Sandy relief But before we get to that I'll let Brian talk and Molly talk a little bit about how we got there and what some of the challenges were in the meantime To introduce how phase two came on the scene We sat in a senior staff meeting one afternoon when we were talking about the three design iteration issue and We had already done a little bit of work on the existing platform to create a beautiful 12 12 12 page But our senior staff said you know Robin Hood's the benefactor of this, but we are the beneficiary rather of this But we are not looking to publicize our brand through it at all. In fact, we think that would be completely inappropriate So what we need you to do is get rid of anything on this site that says anything Robin Hood But still please host it on the Robin Hood site because we love these wonderful tools that you've built for us over the last year Can you do that? So I texted Molly from the meeting and I said Molly can we do that and Within 20 minutes we got the answer So yes, we could do it but This is actually a question that comes up a lot and I've seen it in my previous work in the music industry where people want a branded separate experience and with the move from kind of a Static FTP based web to more of a dynamic CMS We don't really necessarily have the option to just upload a new site into a folder and take advantage of all of the workflows that we've built in that Brian and Jared already talked about so You know it talked with Brian and said well, what can we do and turns out the way that we had already architected the site Allowed us to put a pretty cool solution together Yeah, so when when Molly and Heather first asked me can we remove all of the Robin Hood branding and make essentially a micro site within the Drupal site without writing a new theme or Writing some static HTML and dropping in a directory and they still wanted to be able to tweak this content They were constantly tweaking the content throughout this whole three and a half week period It was a little shocking, you know, like typically when you think about those like sorts of requests and you and you're told it It needs to be live in basically less than a week for the first version You're stuck with a few decisions, you know You can write that static HTML site put it all in code and take away all of your content editor's ability to tweak and work with It unless they can handle, you know working with git and your deployment process or something Or you can write a whole new site you know Start a new WordPress or Drupal site right off the bat and see what happens But then you still have to like build that and you have to theme it and that's that's not a quick effort necessarily and When you've got lots of design iterations coming you don't necessarily want to redo that multiple times and the other you know other choices to try and write a new Drupal theme and get it to work with the site in a Very quick way, but if you haven't built a site that's going to be switching themes really quickly that can be You know quite a headache to try and put together in a short period of time what we found is within a few hours of the Robinhood mobilizing to start working with Sandy relief they had already started creating these painstack pages in the site that were starting to look different than the rest of The site in some some big ways And so we kind of thought well What if we just give them a checkbox on their painstacks that turns off all of the rest of the Robin Hood regions that add You know the the branded header and the branded footer and then they would just have essentially a microsite in a page They could do anything they wanted with their content pains They could add new images They could add all all the content that they want they already had the tools to change colors and fonts They could do Everything they needed to within the system we had already built So that's what happened the the entire solution was a single checkbox and you know four or five lines of code and That got us through this this huge thing. They were able to constantly tweak their content and Constantly tweak the design and updated on in their own without us having to be involved To to help them through that effort. They were able to use the tools that already existed To to pull off something really amazing So again, you know, they were able to just create different pains and put them together into a painstack that really built out these totally different microsite experiences and and You know, we'll hear a little more from Steven in a few minutes, but when the request came, you know What about putting this on a different domain instead of it being at Robinhood org slash Something we're at a sub domain. Can it be at an entirely different domain? We are super pleasantly surprised to find that the architecture that we had already put into building this Server server infrastructure was going to help us out in a really big way, too So that takes us to challenge number three You probably heard me say two things earlier on during course of this presentation so far that seemed completely in conflict with One another one is that Robinhood usually has very little traffic to speak of and the other is 450 million Uniques in five hours So we had to do something that would enable us to handle incredible spikes of traffic And this was to be to be honest. This is a pretty unique situation I don't know that we would expect to see again a situation in which we were given those kinds of numbers But in general Robinhood does have extremely spiky traffic about two weeks ago We were featured on 60 minutes So if some of you are trying to think about where you might have heard of Robinhood that might be it and that certainly spiked us considerably at about a 19,000% spike in unique users during that broadcast We had the Black Eyed Peas play concert in Central Park for us two years ago That obviously had a big effect Lady Gaga played our annual benefit and she was at the time the most followed person on Twitter and tweeted us Which was a blessing and a curse So we really needed to be able to handle these kinds of variations in traffic and So Stephen Merrill whom I'm sure many of you know and love Helped build the infrastructure behind Robinhood org and also behind what became our 12 12 12 site He's gonna talk a little bit more about the magic behind that So as Heather mentioned there were a couple of hosting challenges with Robinhood.org obviously their existing website was a Set of physical hardware with their legacy CMS that lived in a data center at Rackspace And as Heather mentioned there were quite a few event driven traffic spikes And so we wanted to be able to without needing to involve phase 2 without needing to involve the Robinhood team Make sure that we could scale to meet any sort of demand There's also always security Considerations when putting on high profile sites like this you have to make sure that they're secure We won't talk too much about that I just some other things for building out the initial version of the Drupal site like any large organization Robin use Robinhood uses a number of different systems to manage its data So for example one of the big systems that it uses to manage information about its grantees is the open source Sugar CRM product So we had to be able to have sugar be the canonical source of data for Grantee information and just have an automated process to import it into the website And finally the the most important one here is the last one that when we built the initial version of the site There were sort of two parts to it Drupal for the Content side of the site using the tools that we had talked about But that the donation portion of the site still lived on their existing net infrastructure So in other words in order to make it much more painless to launch We decided that we would start out with using Drupal for the majority of the pages But still have the ability to use their tested and true donation system on net so we had to have some way to Seamlessly to the user be able to deliver certain pages out of Drupal and certain pages out of the net infrastructure And of course we've since actually switched it so that the donations are also powered by Drupal So in in coming to this Robinhood had a decent amount of familiarity with the lamp stack and with other hosting providers because they had Hosted some of their things like sugar CRM or some of their micro sites on the lamp stack Using custom PHP or other applications But we wanted to make sure that we could also in this theme of scaling up to meet demand be able to really easily add machines Have their configuration in known state So we use two different tools to accomplish this We use puppet for the configuration management part of this in other words for all three of their environments Which include a single machine dev a three machine stage and a six machine production infrastructure Everything is provisioned by puppet and what that means is that if it would come to the case where Robinhood needs to spin up another Webhead they could add it based on the name pattern It would come up with the appropriate set of services of they named it web 3 it would come up with the lamp stack And with it, you know ready to be tab Drupal deployed to it Since puppet was also new to most folks Robinhood's security folks We're already familiar with it as where we we also Utilized vagrant and what vagrant is is a really fantastic piece of open-source software that allows you to basically Create your own local dev environment using a virtual machine So vagrant itself runs on Mac OS 10 on Linux or Windows And then it can spin up pretty much any unixi or Linux type of VM for you So with all these configurations of all the things that I'll tell you about in a bit any developer on their first day On the project can go in and clone to repositories type vagrant up which will download a VM image and which will apply all this Configuration you do have to go and you know maybe make yourself some coffee toast a bagel But when you come back you've got the full stack up and ready to go And so this means that you can easily test with the entire stack And it means you can also test changes before they even hit any hosting environment So if you decide you want to change around the configuration of your cache server You can test that locally before you even put that into a standard workflow So I also mentioned that Robinhood had some systems hosted at rack space And you know there are certainly quite a few options out there for hosting a Drupal site You know Robinhood had actually used other cloud providers for some of their micro sites But we decided to use rack space for a couple of different reasons The first one is that Robinhood already had their legacy physical machines At rack space and one of the products that rack space offers is called a rack connect and what rack connect does is it lets you combine At least a web application firewall and maybe also a dedicated load balancer to provide a secure way to let your Virtual machines and your physical machines talk to each other and talk to the world So once you have rack connect up and going you can make a comprehensive set of firewall rules just through a control panel So you don't have to do anything with puppet anything on your Linux machines You actually just get a control panel where you can say I need these servers to be able to talk to the old web head In order to go and get some data or I need these new Linux servers to be able to talk to each other in these ports So that's another thing where you know You don't need to have someone who can go and be a wizard with IP tables to have a secure environment and Another nice thing about rack connect is that it lets you do match things based on name So you can again in line with the puppet thing you can go in and say Any of my web servers should be able to talk to any of my database servers on port 3306 And then it's all automated. There's no need to worry about having to do that yourself Another thing as we mentioned, you know, we wanted to make sure that in general This would be a low-cost hosting solution But that it could quickly scale up to any spikes and in any sort of high-traffic environment you're going to need load balancing Now rack rack space in conjunction with rack connect will give you a very very powerful f5 load balancer But that'll cost you a pretty penny. They also have a cloud load balancers product Which you can just basically say here's port 80 and here's port 443 Give me a load balancer and they count it based on the number of connections per hour So if you're under a hundred concurrent connections an hour for a month, it'll cost you about ten bucks It's really really low-cost But then you scale up and you start having five thousand six thousand ten thousand people on your site at the same time They they scale up and meet the demand But again, you're only charged on a per hour basis based on how many connections you have So it's really cost-effective The rack space cloud itself has recently launched a new version called their next-gen cloud servers Which has some really really great attributes one of those is that you can get SSD backed block storage So if you need a database server to be really really fast You can attach an SSD for $70 a month and your throughput goes through the roof They also expose the open stack API So all this means is that you can use gems like the open stack compute gem to automatically spin up new instances So if you discover that something's up you can do that We use that for the 12 12 12 hardware and and finally again since Robin Hood was traditionally used to hosting on Windows Rack space offers, you know One of their hallmarks is their fanatical support and they do offer a managed server level And so what this means is that you pay a little bit more per Server per month and you pay a hundred dollars a month on your account But certainly, you know with phase two and Robin Hood We've gotten way more than that hundred dollars worth for we did have a different hosting environment I'll briefly talk about for the 12 12 12 concert because of the increased traffic needs and You know for paying a hundred bucks and maybe another 50 bucks in cost sentences They help to spin up new machines. They vetted our approach for things They sat during the concert on a call in our chat room So it's certainly an incredible value that you get and they can really help you Manage your servers if you don't want to keep a team of sysadmins on staff So as Heather mentioned Most of Robin Hood's traffic looks like this. This is traffic per day on a generally average month So you might get around 1200 visits per day But we knew that we would need to be able to handle large spikes of traffic and the good thing about these spikes is that they're largely Anonymous right? It's mostly folks who say the black eyed peas just tweeted or Robin Hood was on 60 minutes I want to go take a look and see what's happening So the tool that we I would normally reach for and that a lot of folks in the drip will reach for is varnish varnish is a fantastic reverse proxy cash and Basically in a nutshell if you don't know varnish what varnish will do is Requests will come in and if they're not for Drupal users that are signed in it can cash a request and send it back So in other words you get mentioned I always say slash dot but I don't think that's a thing anymore if you're on daring fireball or if you're on red It you know you start getting these requests that come back for the home page Varnish will keep a copy of the home page and then it'll cash it for say five minutes so the next X thousand people that come in five minutes will hit varnish and Drupal doesn't even notice that anything's happening The good thing is that during the huge traffic spikes on the 12th till 12th concert You couldn't have told that there was an event going on if you just looked at the CPU usage on our web servers Now varnish itself wasn't enough as I mentioned one of the big things that had to be done for this system Was that it had to still accept donations and it had to still work with the existing dot net infrastructure? So Varnish is a great reverse proxy cash It's very flexible, but one of the things it doesn't do is it does not speak SSL So it can't actually make an SSL handshake with a client So one of the important things that we had to be able to do was have some sort of way that if a user came in and typed HTTPS colon slash slash www.Robinhood.org They'd have to be able to connect with a valid SSL session Something would have to decrypt the session and say oh this is going over to dot net or it's not and then based on that it would Either send traffic to Robin Hood's physical servers in the same data center or back to the new cloud-based environment that was set up for the Robin Hood site So for non SSL traffic This is kind of a truncated version of what it looked like But this is more or less how the site went the blue lines are for HTTP traffic and the red lines are for HTTPS traffic So if you requested just plain www.Robinhood.org over HTTP not over SSL a request would come into the cloud load balancers They would balance back to however many proxies were there which was just two of them And the proxy machines are the ones that run engine X and varnish. That's it And we'll talk in a little bit about why we wanted to keep them separate for the first phase of the project Then when it reached the proxy server, it would look at a number of paths And we were pretty lucky because the Robin Hood CMS was in net It was pretty easy to say okay if you get a dot ASPX page throw it back to the net system If you get a dot ASHX dot AXD page and most of the rest of the static resources for the Robin Hood site also had paths that we could Key off of like slash media slash CSS or slash JS So with the exceptions of a few special pages We could really pretty much just say if these things happen send them over the dot net infrastructure And then serve them back out and again because we were using rat connect and because these things were in the same data center There was very little latency. It added maybe an extra 50 milliseconds overall to the page So that was another benefit of being able to use rack space for this and of course for the SSL traffic The same thing happened a request comes in for a CPS colon slash slash Robinhood org the cloud load balancers When we launched Robin Hood cloud load balancers couldn't speak SSL It actually turns out that they now can terminate SSL So you can actually even put your certificate there and let them handle it So they could have actually balanced directly back But it's the same idea in this case they're doing straight pass through it sends it back to one of the proxy servers The proxy server does the handshake with the browser You get the secure connection information and then in the proxy layer it'll say okay is this for slash media send it back And again, it's always the the proxies themselves were also connecting over hd2bs to the existing net site So the connection was always SSL completely through So as I mentioned before we had these proxy servers up front They were running engine X for SSL termination rewriting and then we had varnish and the reason that we kept them separate was that We had to do it to enable the Drupal and net split another nice thing is that varnish and engine X actually use very little CPU Even when doing SSL termination We found out that as we were saturating the network connections of a box We'd only use about 14% CPU So as long as you're using SSL on an SSL session cache, which is one of the things you can do with engine X You can really use very small instances and then you just have to worry about how much network throughput they have And the last thing is that generally when you tune Drupal you have to tune it to the number of Apache processes You can run so you can let Apache deliver a page and then let engine X actually deliver that out to a client So instead of tying up your 8 or 10 or 20 Drupal processes, you can let engine X do that hard work So one of the wrinkle that we had, you know, pretty much everything else was really easy We said if it's an ASP X page if it's slash media send it through The legacy CMS also used slash admin and of course Drupal also uses slash admin But luckily the solution for this was not even something that we had to do in the infrastructure layer Just like with everything else in Drupal. There's a module for that So there's a module called rename admin paths and we got to use that to put a prefix before the Robin hood admin screens And it meant that we just didn't even have to do anything to allow that to work So then Heather did mention that the 12 to 12 concert came up and there were a couple different phases of Working with the 12 to 12 concert the first two versions of that design that we showed earlier were hosted from the existing Robin Hood Drupal infrastructure and the third was a completely separate set of infrastructure Redundant across two different data centers across the country So for the first phase when the 12 to 12 concert org site was being served off of Robin Hood's existing Drupal server infrastructure This was a case of some of the spikes So we saw as you know a hundred times increase in daily traffic and the couple of peaks that we have out here were around when the new designs came out and also around when pre-sales were being announced for the concert tickets And another thing that that Brian had mentioned earlier was that in addition to you know Everything happened very very fast when we were putting together the 12 to 12 concert website infrastructure And one of the things that happened was we had this page up on the Robin Hood site It had all the information we were using the height You know the show site Chrome checkbox to turn that all off and then right at the end of the meeting one of the most Important executives stakeholders said well, when is this going to be on 12 so self concert org and No one had mentioned this before and so we were thinking and but luckily we had engine exit the front So we were just able to do a very simple rewrite where we went in and said okay If a request comes in to 12 12 12 concert org Engine X will do a proxy patch request to Robin Hood org slash 12 12 12 So in other words we could have a piece of content Hosted in Drupal Using varnishes caching rules and actually using the tools to be able to clear out the cache when editors Edited it and so you know end users were none the wiser. They went to 12 of the concert org They saw the perfect branded site experience It was incredibly fast because it was served by varnish And if editors updated it the changes would be live in a couple milliseconds because varnish clears out very quickly So that's how he did that So for the anatomy of a spike this graph from Google Analytics is the overall traffic per week For the year of 2012 so you can maybe see out in the May time frame There's a little blip there and under a normal graph that would look huge But yeah, this is really what Robin Hood's spiky traffic looks like, you know 600,000 users in a week whereas normally there were maybe 6,000 and then finally I won't go into it too much But we did use a separate web infrastructure for 12 so 12 concert org The good part about it was that we could reuse all of our existing puppet setup to set up these new servers We used Akamai to do global load balancing for them and another development partner was brought in to do the site itself Just because it was such a compressed time frame, but we helped them architect the The code to make sure that it was incredibly cashable by Akamai and yet at the same time things could update within 30 seconds Or a minute so during the concert when we did have millions and millions and millions of uniques They were getting live updates, you know, you'd be watching the concert in a minute later There'd be a different call to action up at the top There'd be a different new set of featured tumbler or Twitter posts and yet the load on the web servers was about 0.001 in the words that they really didn't know that anything was going on So we had plenty of capacity to spare in that case So just to wrap up We have successfully over the last two years met three really significant challenges for Robin Hood the first being The way in which our editors hands were literally tied I mean we we needed to be able to give them a lot more control and not only were we able to give them an incredible amount of control without impacting code at all On our typical day-to-day website with robinhood.org But ultimately we were able to enable them to have Micro-sites hosted on the same platform With unique URLs thanks to engine X and all of this on an infrastructure that costs us less than $2,000 a month and can handle huge spikes of traffic relative to our daily So thank you to all of you for hearing our story and thank you to phase two and our team for making all of this possible Thanks. If you have any questions, please come up here. There's a mic up front and that will make sure that The questions making into the recording and we're we're happy to answer any questions you have First of all, Sven you make me feel really dumb So But so I had a couple questions Number one you guys mentioned you move the donations to Drupal. How are you guys handling that now? Specific module I guess maybe I'm asking So the new the new site or the the phase two rollout of the Robin Hood infrastructure We rebuilt the donation platform using Drupal commerce So the entire site is Drupal 7 and we use Drupal commerce with a PayPal back end The other question was with the beans and you know, these really custom implementations Have you guys discussed any type of responsive or mobile implementation and what are like the challenges that you're struggling with there? That's a great question What we found with this site is that it actually works super well already out of the box on iPad The way the content is already structured with With in the in the way it's designed it fits super great on to Mobile devices and it actually is very readable still on on iPhones and androids typically When we created the interactive pieces that was a big concern, you know when we started this project We were told, you know the most important stakeholders the executives of the board use Internet Explorer 7 and iPad and Creating interactive experiences that are good on both of those is not particularly easy What we found is that there's a library called Raphael J s Which is a drawing library that handles in modern browsers. It draws in SVG and then Internet Explorer it it it draws in VML, which is sort of the bastardized version of SVG that Internet Explorer has the older versions and What we found there is that we could use that to create interactive pieces that were touch-enabled that worked really well on iPad But we're also seamlessly interactive on desktop experiences First I just want to say it's really informative and learned a lot So I have a couple of questions about your individual setup Did you guys talk about why you didn't use the proxy server to handle the static assets? Is that typically not something that's done with varnish seemed like was hopping off to something else? Yeah, so actually I should have also mentioned we are using Akamai to handle the delivery of all the static assets We're not using it to deliver the site content itself to do that You have to pay extra and get a special configuration to say if you've got a session cookie pass through so we didn't mention it But yeah, the you know, we're not actually doing delivery of CSS JavaScript web font files Those are actually handled by Akamai But for our use case that could even probably be done by something like cloud front You know that just anything to offload it from our own web origin And that was my second question was trying to get a picture about how Akamai fit into everything And you guys were saying you had a minute to killer cash And you know we use Akamai as well and we use it for you know full-page caching not just static resources and That minute was quite it's it's they don't guarantee that kind of thing I think there was like eight minutes or something like that So yes, so we're talking about here. So for 12 to 12 concert that are we did actually use Akamai to do full-page delivery and Sorry, what's your name Jonathan so what Jonathan's talking about is that generally to purge something out of Akamai We'll take at least seven minutes Let's say and it could take up to maybe 15 minutes to hit all the edge nodes Akamai has got you know I think like a hundred fifty thousand servers around the world and that's what makes it you know They have a server close to everybody. That's what makes them fast. So yeah, if you're if you're doing Long TTLs and doing purging that works But instead what we did for the 12 to 12 concert was we literally set like a 30 second TTL on everything and had the origin update the file and so You know and basically the layout was there were a number of like kind of page for the there were a number of different pages That the users would visit and then we had a number of different JSON feeds, you know Like tumblr.json Twitter.json instagram.json like alert.json for the banner at the top that was the call to action So instead we just set those Akamai TTLs to be I think either 30 seconds or a minute And then Akamai would just check back from its origin nodes and we did still have varnish at the front of the origin here So varnish can very easily serve a 304 So just if you don't know You'll what you do if you want to work in a scenario that's very compatible with H2B caching You send a last modified header out to your clients to say this file was last modified say two minutes ago And then Akamai will check back when its TTL runs out every 30 seconds with an if modified since request In other words just asking your web server, which would be varnish in this case Has the content been modified since a minute ago if not varnish will send a 304 not modified header And it doesn't have to send any content. So that's how we we did that You know varnish was serving a lot of 304s to Akamai, but because it's so efficient. There was actually very little load of the system Thank you. Yep Great presentation guys. Thank you. I have three questions One of them is really to the bean module. Do you know if there's anything you can do with provisioning? so beans are entities and Being entities. They are revisionable. So they just you have to specify I think you have to specify a revision table, but they're entities. So they're revisionable Yeah, and workflow so if you have a workflow module out there that allows you to apply State state it's a workflow states and transitions to your entities that can be applied to beans as well Okay, thank you second question With what you did with the wizzy wig if someone were to do like a paste from word like with those fonts be Fonts and colors would maybe use specifically or do you have to like Set them up in another way where it it you know the branding stays right? This is this is something that you know We run into all the time you have editors that set up things in word and you paste it in and word creates all this crazy Markup, and they actually mentioned this during the keynote yesterday I think the way that we kind of get around this we just make sure that word isn't the canonical Place where the content is kept. So they there is no there is that there still is that Bug where you know word will put in all that bad markup in your wizard if we're just copying pasting directly And so there's definitely at the top of the project a perception problem like you know This is where the content lives, and you're going to be editing it using these nicely Approved themed tools to create the content and style it given the brand accepted Color schemes and so on Thank you, and my third question is you might have mentioned this but What was the reasoning behind not doing SSL termination at the LB level? It was just that when we started out cloud load balancers didn't have that capability So when we first launched you they couldn't do it now They can which is yet I mean both like Amazon's ELB's and Raxus's cloud load balancers can do it And so we might consider doing that at some point as well I mean we have it up and working with engine X and we're not having any capacity problems at the moment But yeah, it is now an option just that it didn't happen to be when we started Okay, and is there a pro to that like less latency Not necessarily I mean you're gonna incur the same costs. It's still gonna have to hop through as many layers. I think you know Given that We might be able to remove engine X from the equation if we were to get rid of engine X doing SSL termination Now in the donation now that we have everything being run through Drupal We don't have to do that kind of partial rewrite over to the existing data center So yes It might be possible in the future to kind of get rid of one of the hops that it has to travel in the data center Before it goes back to users browser. So that would probably be the advantage is that yeah We we could potentially remove engine X from the equation But it's not something that we're you know, we're not particularly pressed for For CPU or or on the speed of our delivery right now, right awesome. Thank you How you done? So I have a question regarding templating So I think there was a no that you demonstrated where you can select like the bubbles and the different layouts So how did you approach that what you use for that? So the templates I'll use the template field and the template API modules and the we created a number of different templates in code and The way each template works is you can define a number of custom inputs and HTML file that has uses the mustache rendering library to Print out those inputs that you do custom inputs that you define and you can also bundle with that CSS and JavaScript and Any Drupal libraries that are available to the system so that when you choose a template and input content It will all get rendered through the HTML Template file that you provide added the JavaScript and CSS get added to that page and it renders out The custom thing. Okay. So all the fields will be created through that template Module, right? So they're not or would that be like a cck? They're not Drupal fields the fields that are part of template field or the template API aren't It's not an entity and it doesn't use the field API there But you do add a single template field to a node that lets you pick from the Templates and the plugs right in right because we were looking at doing something similar. So, okay. Yeah, thank you Hey, I'm wondering if you guys had any requirements for tracking source codes or campaign codes refers from Google Analytics my membership and Marketing folks would be salivating to get you know that data for 120,000 visitors and in one shot and are always asking for cooking and that sort of thing so we can pass it on to Convio and Convio can track those codes, which means all that traffic is not going through varnish So I was wondering if you had to deal with that and how you would have so Well, so admittedly Well, yeah, like everyone who comes to the Robin Hood site gets a Google Analytics cookie, right? And so I'm not a hundred percent sure about that But as long as you keep your your cookie setting in purely in JavaScript Yeah, I mean every visitor to the Robin Hood site has a number of cookies including the Google Analytics ones the standard practice with the with using varnish with Drupal is to actually in VCL receive you go in and you cut down the cookies that Get sent to the back end because generally It depends on what you want to do But generally, you know, you don't care exactly what Google Google Analytics cookies you have Because the JavaScript will run on a page load and do those so generally what you'll do And you can see this like if you take a look at the press flow or the Lullabot VCLs in VCL Receive you can actually say like kind of take all the cookies that were sent in which could include a lot of different things from Quancer from Google Analytics from chart beat and actually strip them down and only send them to the back end if it's a Drupal session Cookie and then maybe also like, you know for a site like this You might see also allow any cookie that starts with Robin Hood underscore and that way you can still preserve Cacheability while having all these cookies that work with JavaScript things like chart beat crazy a Google Analytics Well, all those are sort of external so they get passed through fine, but if And I don't know how sugar serum works But it like if we want to set a domain cookie for you know Source code that's coming in and have convio be able to read that then we need to do that with PHP And that I guess it sounds like maybe you're saying there are some varnish settings that I could look into that would Let me still cache that page even though It would depend I mean if it would have to go back and talk to convio then it wouldn't necessarily although Again, what you might do with that would be you could actually make it so there'd be like a JavaScript include that You know that could do a request to convio which again would have all the cookies and then it would go back and be there So it really depends on if you are actually bootstrapping Drupal to talk to convio Or if there would be some other way and like also with engine X in the mix You could considerably also do like an SSI which is like an ESI Include although I mean that that's getting pretty deep to just do sort of like source code tracking So most of the time if you can do something like doing it on the front end You know if you can do it through JavaScript, then it very much increases the cache ability of your pages Okay, and add one more question. I looked into the embeddables module while you guys were talking it seems it's still an alpha It's not really very widely used and I was wondering what benefit or using token filter Was that an option to to give people your content editors kind of a library of blocks blocks that are generated by views? And is that something that you looked into or you're familiar with that or did you decide that that wouldn't work for what you're trying to do? Oh We did look into some of the other options besides embeddables What we like about embeddables is that it adds a little bit of stuff into the whizzy wig that you can then use the styling options Built into the whizzy wig to do things like center the blocks or or other sorts of styling improvements that need to be treated to those specific bean entities or whatever that are going to be included so we found that that worked a lot better than asking them to You know no tokens and look up the tokens and then figure out how to style them either by writing some HTML themselves or You know through some sort of special whizzy wig interface Thanks So my question kind of piggybacks onto that Is the content for those field things or those reusable content can you find them again? And how do you manage those reusable things like things for the navigation all the micro sites? I'm sure that's a reused piece of content. Does that get really you know There's a huge page of reusable stuff people have to go through now or how do you guys manage that? so in in this site Content reusability was was pretty important when we were starting out So that's why we broke things into two concepts the panes Which is a content type which has the template fields basically It's just a template field and a little other metadata fields on that content type and then pain stacks Which is another content type with node references for pulling in the pains. So they're They're reusable in that any specific content chunk can be used on any number of different pain stack pages And then pain stacks are really the main building block for the site. Those are sort of main level landing pages Okay, awesome. Thank you Hi guys, thanks really informative Going back to the bean revisioning question so much of the content of the site it sounds like was in beans But I mentioned that you needed workflow capability How did you sort of resolve the revisioning needs with the reusable components and so on? so we We're not using in there's no official sort of like editorial workflow behind beans that they use right now They generally go in and just create it just like they would a node and then they can reuse it in any of their wizarding areas If I were going to use a workflow module around it I mean knock on what I would probably use maybe state flow or a workbench just depending on what your specific needs were But since their entities those modules allow, you know Well in a workflow to be applied to them just like any like a noted to D would are the editors currently Are the pardon me are the nodes themselves currently subject to workflow? Not right now. Okay, so there is not really a currently a workflow of any kind No, no generally just like a published and unpublished sort of thing. Okay, great. Thanks I'll add to that just real quick that we did build a full workflow system And then they found that their content editing team is really two people and they're fairly responsible Adults and they felt that they could do the entire thing themselves without needing the the software to enforce a workflow practice without You know causing lots of trouble Great. Thank you. Thanks. Thanks everyone For coming and we'll be down around the phase two booth if you guys want to talk more Thank you