 Okay, thanks everyone. My name's Kieran Lowell. I'm the technical director for Enterprise Sales at Aquia. I want to tell you what I think is one of the most exciting stories that I've been involved in about migrating Drupal sites. Worked with a customer that had a lot of Drupal and needed to move to the cloud and I worked with them over a period of time to help them do that. I think they had a combination of business challenges that they had to address as well as a series of technical challenges that they had to address and they had to think about scaling and code migration and tools that they needed for their developers. And so one of the challenges they had was what would it take to move all of their artists to one platform? Imagine if you will, this photo, we are the world being done for Haiti where all these artists got together, Bieber, Celine Dion, Usher, a bunch of folks in there, coming together and getting on one platform for one thing to deliver their artistic expression and help raise money to help people together. What would it take to build a platform for all of them to be able to have their websites and get delivered out of the cloud? Well first, there'd be a bunch of costs involved, thinking about what it would take to cost that out. And the next question is, could you do it? Could you have one platform that could scale that big? Could really get up there. Well before you did that, I think you'd want to take a few minutes and pull back and say, if you wanted to do this on Drupal, what are other large organizations doing to address this first and see what's going on in the marketplace? So what we're seeing today is a lot of people are migrating away from physical servers to cloud services because they don't want to manage hardware anymore. They don't want to manage capital costs. They want to be able to take Drupal, which is for many people a web content management system or a web content management framework, and they want to turn it into a product that their customers can use, that the people who work for their organizations can have it in a finished form just to do their job. Another thing we see is we see incredibly small Drupal sites and incredibly large Drupal sites and those demand different service levels, right? Different organizations need different service levels to be able to deliver those sites. And then they're searching for Drupal talent. How do they get this done? What are the parts of managing an entire Drupal stack that they want to do, that they want to be a center of excellence and which parts do they not want to have to be a center of excellence? I'll tell you a little bit of a story about an organization that, and it says there on the slide, but that was going through a transition. One of the things they had to do was restructure their business as many organizations do and they were looking at how to make things more manageable cost-wise. And so one of the challenges they had as they got towards the end of 2010 was can we manage these costs more effectively? And many organizations today really want to be able to align the costs of doing business with the operating parts of their business, not their upfront capital parts of their business. And so they realigned their costs to be closer to operating costs as opposed to pure capital costs. It's a great picture of Metallica, great band from near my hometown, San Francisco. And they have this great quote called sleep with one eye open. And I think when you're trying to help with a business, and I think for those of you who've had to stay up in the middle of the night and help keep a Drupal site up, you know that having service levels and being around 24 by seven, managing things in the middle of the night is something that you have to worry about. And so like Metallica sleeping with one eye open, you have to create a business that's got service levels and has staff that can help. And if you get it wrong, then it gets really disruptive. And I think we all know what the experience is, having Drupal developers working on a site, trying to get something done on schedule, and then all of a sudden they're pulled off and they go into a firefight for 18 hours because there's been a spike in traffic or something has gone wrong, right? And so being able to have a team that's on standby to be able to go fight those fires and deal with those issues is really critical. And so in order to do that and have that team available for that firefighting and that kind of support, you need to build out some infrastructure. So you gotta make an investment in your service level infrastructure. So you gotta first think about what kind of proactive monitoring? How do I stop problems and identify problems before they occur? How do I resize and have elastic computing resources? How do I have more storage, more bandwidth, more computing resources, more memory? How do I do runtime monitoring so that I know when my system's what's going on at that time? How do I have a 24 by seven operations team? Any guesses on how many people it takes to run a 24 by seven operations team? Yeah. 10. 10, yeah. Some people say 10, some people say seven. Two people for each shift and then one backup because somebody goes on vacation. Pretty heavy investment for running a bunch of websites. And then you also have to be able to diagnose problems across many systems that are involved in Drupal sites. You've gotta look at the logging infrastructure and log your memcache and your databases and your web servers so that you can go in and look at these more complex debugging situations and look at everything simultaneously. So it's a big thing. And then you gotta go out in the marketplace and you gotta hire expert Drupal systems ops people which are really hard to come by. There's not a lot of them. And so it gets really competitive. So a pretty hefty investment in meeting service levels for Drupal. This is a picture of Lollapalooza and they've just opened the fan gates, letting the fans rush the stage, migrating if you will, from the queue up to the stage to go see their favorite artists. When we worked with this customer, they had to move a lot of data, a lot of fan sites in a similar way from their hardware to the cloud. In order to do that, there were a lot of things that were involved. They had to worry about things like, how do I move the code? Where are the paths that the code are pointing to? How do I move the database over? How do I break up the data that's in the database? Do I bring it over all? Do I bring the caches? Do I bring the non-stateful data over? How do I get assets? If I've got media artists and lots of content teams, maybe potentially hundreds of content teams that are putting up all kinds of digital assets, how do I get that all over to the infrastructure and have the site still pointing at where those assets are so that when the sites come back up, they're not missing pictures and those kinds of things. And these were all sorts of the challenges that they had to be able to do. So we built some tools to help them do that migration. And a lot of it was spent debugging hard-coded paths in their environment that made it less flexible for them to be able to move from one virtual environment to another virtual environment or from their hardware environment to a cloud environment. And the reasons you want to be able to do this is in the media business and in the music business, you have artists that maybe release an album every two years, maybe every three years, and there's a huge amount of anticipation when that happens. You get floods. And whenever you see an artist come out, you often hear about, they got the number one album of the week or the month, and getting that buzz and getting the press to cover that, you don't get a lot of shots when you launch that album. It might be three years for a career artist to put out an album. And that day that it goes live, you've got to sell a lot of albums. You've got to sell a lot of tickets. You've got to sell a lot of merchandise. You've got to be ready for those spikes. And sometimes you got to move from one cloud provider to another cloud provider. You got to move from a private cloud to a public cloud. You got to be able to resize really quickly. And so to do that, you need those tools that I talked about, those code tools, those asset management tools, and those data tools. Another thing that a lot of people are afraid of in the cloud and we hear this feedback all the time is that they say, well, we need special hardware. We need really big systems because our site's huge and we have massive database that has a lot of data in it. But what we found is that using a common systems principle of componentizing systems, you can actually break up the amount of data you're storing so that you're not required to have as hefty hardware with really intense IO operations by breaking the systems up. So for example, in Drupal, you can put everything in MySQL but you can also take some of your dynamic data and put it in Memcache and serve the most user data, session cookies, things like that that you need to get access to from Memcache and have a network Memcache pool. You can take the data that you're logging all of your Drupal activities and instead of logging that into the Drupal database, you can log it out to syslog which is much more resource effective than logging in the database. You can use a tool like varnish which is a reverse proxy and so you can set the lifetimes that have custom varnish configuration files that say exactly how long that page can last and then deliver it from virtual memory on Linux rather than generate it with a Drupal bootstrap and pulling it from the database and so you can get huge scalability and we've seen things where in the Acquia Cloud where we've had over 8,000 pages delivered a minute to be able to hit those kind of varnish benchmarks and then using a CDN helps a lot because you can take, you can put a lot of the assets there, it avoids doing bootstraps or it avoids pages being delivered from the Apache web servers. And last but not least, Drupal's native search is not optimized for sites with a lot of data and so pulling search out of Drupal Core and putting it into something like Solar, you can break it. And so you can see, you can break it down and then get more advanced search, better index content, be able to deliver faceted search. So instead of saying I need really special, gigantic hardware that can only be delivered in a data center where I own the machines, you can move to the cloud if you're willing to break your data up and deliver it from five or six different services. Another thing that people talk about is tools, people be able to have better code management or better tools for developers because Drupal developers are expensive and having to bottleneck on an operations team that's got a long queue of tasks to do is difficult. People want self-service tools. So one of the things we did was we worked with them to take their code that was stored largely in a one major product or a few products in an SVN repository and take it from being in Drupal to PressFlow, a distribution of Drupal in Drupal 6 that had in particular some great performance tweaks that were in Drupal 7 and in particular, those were allowing, varnished the reverse proxy to be used. And rather than having one SVN repository for all the products, you could give every team their own SVN repository so every team could sync up with the main project branch but then they could deploy code in many phases in many branches that they wanted to and that really empowered the team to be able to do development a lot faster and have a lot more independence. So I wanna just take a second and show you what some of those tools are like and show you why it was so powerful. So for some organizations, let me start to get this right. For some organizations, when they have a problem on a website or they have a problem on their staging server, they call their developer and they ask their developer to push it to the next branch because really it's the developers in the organization that have command line skills, not the content people, not the project managers, not the designers. So you can imagine a situation where your project manager basically says, hey, did you fix that bug? And the developer says, yes, I fixed that bug. And they say, well, what branch is it on? You say, hey, it's on the master branch. You say, that's great. I'm gonna grab the code on the master branch and I'm gonna just go ahead and deploy it into my staging environment. Totally bypassing the need for somebody to be an expert code command line jockey who understands all the systems in different development environments. And then they go and they look at that bug and they say, you know what? It's not working here and we're still seeing it in a production. That's a real problem for us. We really need you to debug what's going on in production. And so they can go in here and they can say, hey, let's copy the content from the production system. And so we'll grab the databases, the file system and the code and automatically move it from the production system back into the dev environment and it's going and doing that. And you can see it's copying files and importing database and giving an update, right? So now project managers, content editors, designers are able to use this UI and have a lot more flexibility to deploy code to different environments and make the changes they need. And if they want to go do QA, well, there's a nice little link right there and they can go and click on that button and the site launches and they can go and immediately do QA. And this design has come from something called continuous integration and the 10 principles of continuous integration or the nine principles of software deployment making it easy for people to be able to do this stuff without again having to be command line experts. We've got other tools in here that help people to do it and I won't show them all, but if you're interested, you can swing by the Acquia booth and get a look at some of those tools. So you have some really incredibly famous artists and hypothetically, we've seen in the past, artists can get a huge amount of traffic on a given day. Maybe they're doing ticket sales, maybe they get on Twitter and they encourage their fans to come and talk about a popular event or to come talk about something in the gossip magazines or something like that and have a discussion. And so different artists can get a huge amount of traffic. And so here's an example of an artist that can go on TV, can have a show, can get on social media and you have to be prepared to be able to scale. And so what we've talked about so far is we talked about the business rules and the business needs which were moving your capital costs to operating costs, moving to be able to support different service levels. We talked a little bit about the migration of code and content and I showed different tools that the teams needed to be able to do those kinds of migrations and manage their code and then you have to deal with scalability. And scalability, Drupal's pretty scalable out of the box and you've added a lot of systems you can really scale in a really great way. But sometimes you need an expert. And at AQUIA we have some really great experts in particular a guy named Eric Webb but among many on our team. And Eric travels around the country to meet with customers. And so back in late 2010 and early 2011 I got a phone call from this customer and they were actually, my boss got a phone call from the customer and they said, I want to come out and talk to you about what can we do to basically do some optimization? Can we run Drupal a little bit better? We're so busy delivering artist sites all the time we don't get a chance to go back and really reflect and do all the optimization that we want. Can you come help us? So I met with them at their office in Rockefeller Plaza in New York and then left that meeting, had a great meeting. Was able to encourage them that Drupal could work better and that there was a way to be able to get better service levels and operating costs that they wanted. And so I flew back and flew to Los Angeles and met with their team in LA and went and spent a day on site and really sat down and got the team to give me a bit of a brain dump on what their systems were. I took a lot of notes and was able to write a report and give it to my colleague. And so my colleague, Eric was able then to come, we were scheduling him and he was able to come and spend a couple of weeks on site and he went through and he did a lot of analysis to be able to take a look and see what kind of systems tuning and what kind of performance. And so he wrote a report and gave it to the team about all the different things that they could do to improve. And as part of that assessment also was a consideration of could we move off the existing hardware infrastructure to a cloud solution? Was that viable? Would they be able to get the kinds of things and the performance improvements that they wanted? And so they were. And so after we did that initial round of performance tuning recommendations, we had to take into consideration something else. Does anybody know what this is? There you go, spinal tap, right? In the music business, the big artists, their speakers go to 11, right? And so when you're talking to somebody in the music industry, they know that they might have the biggest hottest band and the biggest hottest launch at any time. And so their sites have to go to 11, right? And so we partnered with a company called SOSTA that basically would go in and do load testing to 11. And we did dozens and dozens of load tests to try to get the Acquia platform up for them and to be able to support their artists. And somewhere close to a couple hundred artists, dozens and dozens of artists and then be able to have sites where a certain amount of the traffic, significant amounts of the traffic were authenticated, some were uncached and some were anonymous. And so we would go through that and we would tune and tune the platform constantly finding things. And that was really useful to be getting ready because with Drupal, a lot of the sites are custom. The modules are constantly getting updated and the traffic patterns can be different. And you never really know for sure until you do a really solid load test whether you can scale to hit those kinds of numbers. So to help with that, we have an operations team. And here's a couple of guys who's one of the leads on our operations team in our office in Burlington. And so you can see that he's got, I think, four different screens here where he's watching the different systems and being able to monitor that. And we had a guy on our team, an operations team, a guy named Chris Rudder, who's a really amazing SysOps guy who just poured his heart and his soul and just was a real hero through this project where he just worked on it, you know, with everything he's got, tuned our systems at an amazing level and constantly found problems, made recommendations, tuned it and hit some pretty tough deadlines because, you know, in a hit-driven business, you know, things are often on demand, you gotta hit these tough deadlines. There's a lot of scheduling that's going on and you gotta be ready to go. And so I wanna geek out for a little bit and I wanna talk about some of the issues that we ran into. Now some of you will wanna take notes on this and you wanna write down these specific optimizations and sometimes maybe they're generic for you. Other times they're only appropriate at really high levels of scale that we were starting to see. So we tuned the PHP and we made a series of recommendations previously but one of the things we found out and one of the things you may find is that if you have a lot of modules that you wanna compress the code and you wanna pre-compile it using an opcode cache. And so the opcode cache that we used is APC. And in APC, what we found was that we were setting a default amount for everybody on our platform of the same amount of memory. And it turns out that wasn't enough memory to do the proper opcode caching, particularly for some of the larger sites. So what we did is we made it so that each individual customer and each individual site could go in and tune their own opcode cache and allocate a certain amount of memory. And so for this customer, we had a really large amount of memory made available to APC. We also found that they were very focused on IO and how important it was to have a really fast network system to be able to read IO. But it turns out that when PHP reads code off the file system, it can read on every page pull and every time it reads the code or you can set a cache about how fast and how frequently it reads. And just by tuning that cache, you can effectively replace really expensive hardware that's been optimized to deliver code really fast just by tuning the cache correctly. And so we were able to find that that really helped with PHP scaling. At the database level, we looked at the write buffer. And so one of the things that's happening is when you've got highly available databases, you're doing replication and every transaction that's happening in a database like NODB as part of my SQL, you're effectively writing out to the write buffer. And so you can make a decision. Do I want to write on every transaction to disk or do I want to take a risk and put some of those in memory and hope that my server doesn't go down? And I think for this customer, appropriately so, we wanted to write to disk on every one. But tuning that write buffer helped. And then we also looked at the replication bin log and in my SQL, you've got a binary log that's effectively writing every transaction and then it's being replicated across to the failover server. And so looking at the speed and the tuning of that and trying to reduce the amount of data that's being written to that bin log can really help in the performance. One of the other things that we did is we turned off the access log and pointed it at syslog. So every time somebody did something in Drupal instead of writing it to the database, we pointed it to the syslog. And then the other thing that we needed to do is this business was really critical. They had a, you know, you can't have an artist that hits that one magic moment in a couple of years and then just because the data center goes down the websites off, that's just not possible. It's not acceptable in those businesses. And so the kinds of thing you have to do is set up global replication. And so in this case, we did a series of things where we had different databases and we put one database in one data center at Amazon and another database in another data center at Amazon. So if somebody drove a truck into the power pole outside of that data center or there was a failure in that data center, the database, one of the database active databases would be up. But even that wasn't enough. Really had to go the next level. And so what we did was introduce Akamai Global Traffic Redirector. So you effectively had high availability across multiple data centers in the same region. But then globally, we were either on the East Coast or the West Coast or over in Ireland and we're able to fail over because we were going through the Akamai Global Traffic Redirector to a whole other cluster set up in a whole other data center in the event that it went down. And those are the kinds of services that a business that has to be up unconditionally needs. So that kind of disaster recovery. And we introduced a technology called Tungsten which was a drop-in replacement for MySQL's native replication that enabled us to do that. For caching, a lot of people ask about how to cache. And sometimes when you have these sudden bursts of traffic a lot of times that people aren't authenticated they're just getting static pages. And so there's a couple of things you can do. With Akamai, you could go and store all of your pages in Drupal and deliver all your static pages in Drupal. Sometimes you want a different profile and so what you do is you store your assets in Akamai, your media assets, your JavaScript, your CSS, all your images, all your videos. But then the XHTML that Drupal's generating, you store and varnish. And so in this case, we basically found that right mix of XHTML served from large varnish servers and then Akamai for all the assets. Now getting each of the content teams as they're going and configuring their Drupal site to know how to use the Akamai out of the box is a pretty daunting challenge. So one of the things we did is we scripted it so that anytime we rolled out a site on their platform caching and integration with Akamai automatically worked. And so they automatically got that benefit of scalability out of the box. And so that was a huge help in helping them to scale that every site could scale. And that was consistent with what people needed in the business. So this is Nirvana, one of my favorite bands. And the thing I love about Nirvana is that Nirvana was alternative rock. They came in at a time where it was really about how tight your spandex was, how frizzy your hair was, how much swaying on the microphone you could do. And Nirvana came in with this totally different alternative approach to rock and roll and shook up the music scene. And I think what was special is that they're very similar to open source in that way. They're kind of this grungy, alternative, different way of rock and roll that's not as polished and stuff like that. But that wasn't enough. What they also had to do was do some of the things that made the business work. And I think what we've seen here is that Nirvana and their alternative music was like open source but they also needed to get their business right. And in order to meet those demands, they needed a technical solution. I've been in the Drupal community for eight years and I think for a long time ago, we would sit back and we thought open source is gonna take over the world. And we thought if we wrote great code and put it out there, that everybody would just see the wisdom of open source and they would move automatically. And maybe people in alternative music thought that they were recreating music and everybody would just do alternative music. But it wasn't enough. And so now with things like cloud solutions we're actually able to put open source in the cloud, in an open cloud and be able to deliver both the innovation of the code but the business levels and the service levels. And that's what I love about this picture is it's Nirvana alternative rock band wearing suits and getting it right. And they fundamentally changed the market of music where that was a whole new genre and the market shifted over to them. And I think that's where we're at the crux right now is we're partway between moving open source to the mainstream where the market has basically said we recognize and acknowledge open source in the way that the market acknowledged the role of open source. So I wanna talk a little bit about what we've seen here. You start off with Drupal and you're doing internal hosting. And then you say, hey, can you take care of it? Give me the service levels, get my operating costs under control, deliver it and so you move to this platform as a service. But that's not the whole vision, right? That's only part of the vision. And you have revolutionary customers who are visionary who really understand how far they can go with open source, can they get all the features they want? Can they get the rate of innovation? Can they participate in an active community? But then have every feature that they want altogether. And so we've got and we've been really lucky to have a great partner who's been able to build out a product with us called Enterprise Gardens. And Enterprise Gardens is all the wonders of Drupal in the cloud with the business and the service levels but a fixed set of features that their content editors, their designers, their teams can basically work, integrate, build and launch hundreds and hundreds of sites and deliver that vision of unifying all artists on a single platform. And I think this is something that, when you look at the marketplace going forward, people are looking for great SaaS solutions that work but they want the freedom to be able to press the eject button and take it outside and go innovate somewhere else if they need to. And I think this is the kinds of things that you should be looking at. So we've got internal hosting, AquiaCloud, Drupal Gardens. And what I'd encourage you to do is to come see some of the amazing stuff that's gonna happen tomorrow at 10.45 just to cross over here on the day stage. We're gonna have some of the people who are involved in the site, in the building of the platform and launching a lot of sites be able to do that. And they're gonna showcase some cool stuff. There's bands like Kitten and they're basically building in mobile integration and you saw Drew's keynote and they're coming together and being able to deliver that. And so you'll see some of those demos and be able to check it out and see that and ask them questions about the platform they've built and the limitations and what kinds of site they've done and they're gonna continue to do development. And the best thing I think and what the most amazing thing about this company is not only do they love and get the innovation of open source and the cloud but they chose to share it with everybody so that if you have a garden site so many of those features that they're developing are gonna be shared back because they funded it because they understand that making that code available to people is really the right thing to do and to really be collaborative in a way that we've seen great work come out of when artists collaborate. If you wanna see some more details, there's a great diagram and you can see sort of desktop and mobile apps and social authentication with JanRain, be able to plug in and then have a series of templates that are built out and we can have templated sites that go out in certain ways and have all kinds of business reporting and analytics and you can plug in APIs if you need external systems because these sites are never really just Drupal. They're Drupal's just one part of that. They've got dozens of business systems that need to integrate, pull data in, put data out and be able to plug into all that kind of stuff and so that's really key. And then what's key is that they're on a fundamental platform that they can scale up and they can roll out lots of sites. So come and learn more about that tomorrow. So I wanna thank everybody for attending today. I really think this is some of the most exciting work moving to the cloud, getting the right business model around Drupal has been really incredible. It's been an exciting project to see and I hope you're able to check out Enterprise Gardens and come see some of the visionary thinking behind where they're going in the next phase. Hopefully in DrupalCon Munich, we'll be able to have a full presentation or at the next DrupalCon site, you'll be able to see a full presentation of exactly what they wanna do but you'll get a great preview tomorrow. If you like user experience and you like some of the things that we've done around Gardens and trying to make it better at usable people, make it usable and Drupal usable for people so that people who just want that point and click experience for building sites, we're doing a bunch of user experience testing. Darmash and Lisa Rex from the AQUI team are working on Drupal usability testing as well as some of the engineering of the products and so they're doing that. So you can contact them here and they're gonna do over a hundred usability tests when they're here. So thank you very much. Really appreciate it. Thank you, Steve. I first started doing the research story around the story when Steve Jobs passed away and I think he's kind of one of those visionary revolutionary leaders who made it technology easier for all of us to use and I think there's a lot of inspiration there from all of us about what we can do and where we can go if we really put our minds and our innovation and our passion towards it. So I'll take some questions now and be happy to hear from anybody who's worked on the project as well. Hey Kieran, thanks a little, is this on? Thank you, it's a cool presentation. I just wanna give a shout out to some of the people that worked on this project. Yeah. Tim Holt from my team is one of the unsung heroes at Warner Music. This was arguably one of the most difficult projects in all of the company in the year 2011 and he killed it and then I completely echoed the sentiment about Chris Rudder from Aquia, an amazing guy. Shout out to Chris if you're here. And anyway, thank you guys for all your work on this. Thanks Tim, you wanna raise your hand? Awesome, taking Drupal to new levels. Cool. So I was wondering, for a surprise hit situation like you see kind of internet spikes these days, is there a possibility of like a, and just a basically an insurance service or something like that, if you were to run into like an on demand, if you will, for a hit. Well, I think there's things that you can do. I don't think there's a magic silver bullet. I think that you can set up a service, you go to a right scale or write some own ruby scripts that would just start deploying servers automatically. I can pretty much guarantee you that if you did that, you would figure out really, really quickly that the reason you had a gigantic hosting bill is because you have a bug in the Drupal application layer. Somebody pushed out a query. We've seen instances where we had a customer recently was the first person to post a picture of Snooki without her makeup on, right? Boom, got linked from the front page of Yahoo and boom, that site goes down. So is the answer to add more hardware or is the answer to get somebody who really understands views queries to go in, make sure that query is cached, rewrite that query on the fly and be able to do it. So I think when you build infrastructure, like I showed in the scaling, where you say we've got cloud elastic resources in the database, we segment the services, we fine tune everything so we can get as much scalability, we use Akamai, we use big Varner servers, we have failover data centers. You're doing a really good job of preparing for that, but it's really the people who are there on call 24 by seven, making decisions. Am I under a DOS attack, right? Am I, did we just push out code? Are we seeing patterns of traffic come in a way because Ashton Kutcher or some artist, you know, posted something on Twitter and all of a sudden we're hitting some part of the site that's never seen a surge of traffic like that because they directly link to it. So I think, you know, it's a combination of magic, cool technology, but at the end of the day it's really people. And the business, you know, you can't underestimate the importance of the business having a real gut check of saying, we want to be this big, we need to be able to deal with this and then preparing and financing it. One cool thing that we're doing with AQUI on this is they actually have the authority to add up to five times the current servers that we have on Amazon without even asking us. So in our relationship, if there's a spike they can just do it and tell us later. And the cool thing about Amazon is it's the on demand instances. So it can be up for 24 hours and then back down again. So they're auto scaling for us as part of our agreement. Cool, not sure how much the mic picked up but just to repeat what was said was that in this relationship we've got the ability to allocate as much as five times extra hardware on demand as part of the contract so that we can do it and then get back to them and Amazon has, you know, the capacity to be able to do that. Go ahead. So for something like the AQUIA continuous integration thing that you were showing up there earlier, I actually have two questions, this is the first one. Sure. Could we mix and match something where we are using like an AWS type of setup? Can we mix and match and use that continuous integration on top of that? Yeah. And second, this is kind of unrelated. What would you say gives the most impact to high traffic to Drupal site out of everything? Memcache, reverse proxy, whatever else? So I think part of the answer is it depends as always. So let me address the first one. And if you haven't got a trial of AQUIA library, you can go in here and you can type in continuous integration and we're starting to see some customers do this. And basically what they're doing is they're using our dev and staging environments in the cloud but then tying into their production environments in the back end because for some business reason or whatever they wanna be able to do that. They wanna use these tools. And so I've written up a story here about how to do it in some R and D that we were doing with a customer. And basically it comes down to three things. First, you're moving your Drupal site. It's the code, it's the database, it's the files. And you isolate that in the cloud environment and you make sure that you can migrate between those environments. We've added something to the cloud called cloud hooks. And so at the end of any of those UI actions that you see, you can have a cloud hook that copies from our production environment, launches a Perl script, launches a Bash script, launches a, sorry not a Perl PHP or a Bash script, maybe a Ruby script and then sends it to that remote production environment. That's one thing. The second thing is that there's always environment variables that are different between Drupal sites. In your dev, you want HTTP auth. In your staging, you want HTTP auth. In your production, you don't. So you have a file that's located in the same place in the file system and your settings.php file, you basically point to that file which has environment specific variables that have things like in dev and staging, don't only use a read solar index but in production use the real live and write to that index. So those kinds of environments. Then last but not least, you've got to be able to get your data from your production environment all the way over here. So now when you go and you say, I want to grab code from the production environment in AquaCloud, it launches a script, talks to your production server in a staging environment, says give me a copy of the database, scrub all the data out of it, pull it in SSH and ship it back in and be able to do that. So three ways to do that about to integrate it. And if you're interested in doing that, we're piloting with some customers, be happy to work with you to have a professional services engagement to do that or you could give it a shot on your own and tell us how it works out. The second question was what's better, reverse proxy or memcache? I mean, sir, I think for anonymous page delivery, there's nothing that's, nothing is as impactful as varnish, as a reverse proxy. Being able to just basically say, I mean, I think there's two, I think there's a bunch of things that varnish can do for you. One, it can basically say, hey, we can deliver 8,000 pages a second. Anonymous pages will never even hit the Drupal, will never even hit the web server. That's immensely useful. The second thing is, is in varnish three and some parts of varnish two, you can do things like have something called grace and saint mode, which basically will handle errors correctly if you're getting errors from the backend. So if there's a timeout on the database, it'll continue to deliver a page or you can say if you can't reach the backend at all under the following rules, continue to deliver scale content until the site comes back up. So I think from a high availability and a scalability varnish is the wonder kid. But things like, you know, a lot of people on our support team really like recommending memcache and always to use it for certain things because it's just more effective than just going all the way to the database and getting it. So, you know, certain tables and we could talk about which ones it's appropriate to just always, almost always put them in memcache. Does that answer your question? Cool. Any other questions that I can answer? Let me, could I talk a little bit more about tungsten? I can try. So tungsten is a, it's like a distribution of the MySQL replication and it's got a couple of features. Chris, do you remember exactly what the features are that tungsten has that make it, allow us to do global replication?