 Hi, how's everyone doing today? All right. Let me start up by welcoming you here to the talk. We're going to dive into a case study on a user's real-world path to OpenStack with Big Fish Games. And we'll do introductions to the second here. We're going to do a little bit different format. We're not going to go through a pile of slides, because I assume you've seen enough slides in the last two days, and we'll see for the next two days. And we just want to go into a Q&A session with us, Blue Box, the provider, and Big Fish Games, the customer. So, first of all, I'm Pernan Alvarez, the Vice President of Product and Operations for Blue Box. Blue Box has a company with 10 years of operating experience in managed hosting, three global data centers, about 600 customers worldwide. Our primary focus today is private hosted OpenStack, which is a fully managed offering where we take care of all the troubles. Our data center, our network, our hardware procurement. We let you focus on working with OpenStack, not on OpenStack, which is what we're here to talk about today with Big Fish. And to my left here is Andreas Stoller. Andreas, do you want to give yourself an introduction? Sure. Hi, everybody. I'm Andreas Stoller. I'm a senior architect at Big Fish Games. Big Fish Games has been around for about 12 years. It's become a leading provider of casual games, which probably none of you guys play. But anyway, lots of people do. And that's why I'm here. I've been there for about a little over eight years. During that time, the company's grown from about 70 to about 700 employees, you know, 40 million to well over $250 million in revenue. You know, a typical game model is you download a game and if you like it, you can buy it. That was a very simple thing. And over that time, we've grown up to three data centers. We have about 1,500 servers. And to support the downloadable game business, we're now pushing over 40 gigabits of traffic. We do make some games in our studio, but we typically contract with about over 500 partners who make games and sell their games exclusively through our marketplace and e-commerce site. So a pretty large and well-established IT organization. Yes. Lots of practices and procedures in place. So you recently had a change in business model where you added a new style of game free-to-play. And there's some challenges with that. Do you want to dive into what the free-to-play model is and some of the challenges in delivering that? Right. So both mobile and free-to-play have presented two new challenges. The free-to-play games seems to be the way of the future. In that situation, instead of downloading an executable and having it lock out after an hour and then you have to buy it, you download the game, you can keep playing, you can grind away forever, but you can also buy a little magic dust and diamonds and chips. So if you like the casino game, you should really play that casino game actually, in the game as you're going. And that model proved very successful and we're just a couple of years into that. The challenge of that is that we're working with well over 500 developers and for them it was a fire and forget model, kind of like UDP games. They gave us the game and we put it on our servers and you can download it and they were done. After that they were completely done. Well now we've got all these developers which to support in-game purchases, we need a back-end infrastructure to support these games and each game needs its own back-end infrastructure because each game is a unique entity. So that's a new challenge for us and all these mobile developers, partners, free-to-play game partners are mostly game developers and some of them have very little server-side experience writing server-side code. So we have to help them along a little bit. We can't just send them off. So your previous infrastructure was more of a traditional IT architecture and access controls and the devs were, the third-party developers, the 500 third-party developers were not satisfied with this. It was deemed as unfriendly. Yeah. I mean, so traditionally we'd grown up with in-house developers. We had our nice development environments, QA, stage environments. They were all in-house and since they were all employees that was very easy to do. Well now we're looking at signing up these partners that need development environments and such but they aren't actually employees. So we need to give them VPN accounts that made the InfoSec guys and the company pull their hair out. We had to sign contracts and non-disclosure agreements to get the VPN going and after about 28, 29 days we'd get their VPN accounts squared away and all their access done. 30th day, the account would shut off due to InfoSec policy to go back and re-enable it. It was painful, so it took us a month to get one, to onboard one developer on one game. It was really terrible. So the next logical step is you start diving into public cloud. You go out and use one of the providers out there and you had some experience with that that was mixed? Mixed. I mean, it was certainly the most scalable. We wanted Amazon AWS like every else did years ago. It was poking around with the cloud and that worked fine and we were able to spin up as much instances as we needed and could give our partners access very easily because it wasn't in our infrastructure. But I mean, the performance in Amazon is terrible and if you're running 24i7, it gets very expensive, especially because of the performance. We had to basically over build everything. I mean, it was crazy. IO performance in the databases was so horrible we would mash together four elastic block storage units into software array to try to squeak some kind of IO performance out of it. But, you know, at the end of the day, six months later, we have two games run and they're each costing us about $70,000 a month to run in Amazon AWS. And that was even with automatic scaling going up and down. You had to keep the databases there. Yeah, so the business was not too pleased with that because those getting from not making $70,000 a month. We decided to bring that back in-house. We spent about $70,000, actually a little less on gear, put it in one rack in our data center and those games are running twice as fast as they ever did in Amazon. And that made the business a little bit sketchy on going back to the cloud again. Yeah, but it rolled the clock forward. 12 months, 13 months. You went back and looked at cloud options again and there were some business drivers with free-to-play. Right. Well, the business driver for free-to-for going to the cloud again was scalability. So after our sort of unfortunate cloud experience, we tried to go down another product line, which was streaming games. And that was even more impossible in the cloud because we needed GPU-enabled units in Amazon, which is really expensive. And so we did it all in-house. We ended up signing contracts for Data Center Space, bought hundreds and hundreds of servers. And, well, that business line never really took off. So at the end of the day, we were holding a bag with all the racks and racks of Data Center Space, hundreds of servers that we don't- and video graphics cards in them that are completely useless for anything else. For games that may have been a smash hit or it turned out to be nice. Yeah, and then streaming product, we were not there to support, you know, Gaikai on live didn't really make it either. So it's not just because we were bad. It was tough enough to crack. So hybrid starts to make sense at that point, right? Where you can have a mix of on-demand as well as a mix of legacy IT infrastructure. Right, we were really interested in being able to utilize some of our own infrastructure that we had and expertise. Obviously, we have large three data centers, big data centers. Tech engineers, DBAs. We have very nice, fancy F5 load balancers that work great. We have 40 gigabytes of upstream bandwidth. We wanted to take advantage of that, but we also wanted some flexibility in scaling. So we started looking around it, some private club providers, ones that we could connect, make a back end connection to our data center. Right. We ended up at Blue Box. So Big Fish Games ended up contracting with us last year to deploy a hybrid cloud where there's a mixed environment between our two data centers cross-connected. Do you want to tell us a little bit about what lives in each organization's facility, how it's connected, what pieces did you need to maintain to be successful? Yeah, so what we really knew is that all virtualized databases are not such a great idea and bare metal is kind of the way to go at the current time. So we wanted to run our own databases and our own infrastructure. We wanted to use our own bandwidth and our own load balancers because we can take advantage of the APIs to automatically spin up things there. But we wanted to move the application servers because the other challenge with free-to-play games is right now we're on-boarding, we're launching a game a week and most of these will not hit, but one of them's going to and when it does, we're going to need to build 100 application servers like in 15 minutes to support it. So it comes Blue Box and Hybrid Cloud and we have all our application servers over in their cloud. We got two 10 gig cross-connects that happen to be in the same data center as our data center is in Seattle. And also they have a facility in Aspern so we can expand out there as well. So we're using the cross-connects, we use the application servers over in Blue Box's cloud. They are now configured as nodes in our F5. They talk to the databases in our data center and the games connect to our load balancers and use our upstream bandwidth. So you're dynamically spinning up capacity inside of Blue Box's cloud using your configuration management to configure load balancers in your facility via that cross-connect and you're transiting all the egress. That's right. Yeah, and even better we were able to leverage all our existing CF engine configurations on the virtual images that we spin up in Blue Box's cloud. So it was really easy to make that happen. Excellent. What other design considerations went into this IT infrastructure, contracts, things that made you think that hybrid was the right way to go? Yeah, like I said, we had a... I mean, we've been doing this for a while and we have a lot of data center space and a lot of expertise in-house. We did poke around in OpenStack a little bit but OpenStack is pretty new and about this time last year when we started to look into it and all those sites that say it's been up OpenStack in 15 minutes, wow, it took a little longer than that actually in most cases and we found we could probably do it but we were way too busy and I couldn't dedicate like half of my team to building this thing up and making it enterprise like for four or six months traveling along it would take them at least that long. We need to get to market now because the game industry is really competitive and if a game is a month late I mean somebody else is going to take it over with the same game and then you get no attraction. So it was time to market really. We need to get to market fast and leveraging OpenStack and a excellent provider such as BlueBox really helped us along that path. Nice plug, thank you. I appreciate that. So time to market. So time to market. We're paying them, actually. Technically, yes. Time to market was one of the main considerations. You looked at on-prem, you obviously had some experience with public so Hosted made sense for you guys to get on board right now. Yes, indeed. So now I'm going to shameless plug why BlueBox? Well, BlueBox had a number of things that we found attractive. Like I said, we were pretty convinced we needed to do some kind of hybrid cloud to be able to leverage our own data center space and expertise but have some scalability out there. We knew we wanted some kind of backing out-of-band connection to whatever provider's network we would be partnering with. So we started looking at vendors and we also early identified some of these cloud frameworks. A couple years ago, CloudStack seemed interesting and OpenStack seemed interesting. They were in the same space but in the last year, OpenStack really took over that. They got the industry support and the commits to the repository is really skyrocketed. So we're also convinced that we wanted to go to OpenStack where we think OpenStack is now, or I do, OpenStack is now where Linux was in 1997 or where VMware was in 2001. At those times, people thought you were kind of crazy to use this. Who are these hippies running this open source operating system? You guys are crazy. You need to use Microsoft. Microsoft, that's the only thing. Well, look at us now. Same thing with VMware. That's crazy, but that's the industry standard these days. I'm pretty convinced that OpenStack is going to become the industry standard in the years to come. So given that, we wanted to use OpenStack. We looked at several providers. Only a couple of them were supporting OpenStack and among those, meeting with BlueBox guys, we found their data center was in the same location as ours and that's very easy and low latency. Latency between our application servers and the database servers is really critical to the games because sometimes there's a lot of back and forth before it goes out to the client app. So that can really amplify. And then after meeting with the guys, we felt that they were an excellent partner that we could learn about this OpenStack thing and grow with it as it became more mature. So now we've been in deployment for about nine months. What are the results to the business? What are the real business results? Yeah, well, so like I said, using our traditional model of spending up VMs and VPN accounts, it took us a month to get a new partner onboarded. Now it takes us really a couple of hours. And we are so much faster than business, they can't sign contracts fast enough for us. We're way ahead of them, which is a good place to be in IT. You're rarely there. So can't get comfortable now, but yeah, that's... It's a great business results, but what about user experience? User experience. 500 developers out across the planet. They previously had a lot of friction. Right. Yeah, the other benefits were that now all their environments are publicly available on the Internet. So when they're testing these games on their mobile environments, they can get to them. Our partners don't require VPN access because they go through the OpenStack portal to access their instances, and they can deploy the code themselves and do all the testing on all theirs. They have two QA environments, the stage environment and production environment, all with full CDN support, and they're all identical, except that there's more instances and the base is bigger. So now we went from taking a month to deploy to a couple of hours. That's really the driver, and the developers are happy. Certainly not all unicorns and rainbows. There's been challenges. Yeah. OpenStack is still maturing, I would say, and there's a couple of Opsies along the way. We had some servers go down on us. I won't go into the details of that, but for a couple of hours, it makes the business a little nervous because this is new technology, so that's probably our biggest challenge going forward is convincing the business that this is really a stable technology and it's going to be the platform in the future. So any team culture challenges with bringing on this new technology? There are some because it's so new, and we still have partners that are not employees that can get console access on instances that technically have access to some of our shared services. That makes the traditional IT guys pretty nervous and the InfoSec guys pretty nervous. But I think we've put enough guardrails around it to keep in a good shape. What about the general transition? On a daily basis, from really 100% traditional IT? Well, I mean, the kicker there to business is that instead of paying Amazon $70,000 a month to support one game where we now have about 30 games running on our Opsac platform and let's just say it's costing less than that to support about 30 games. What about other workloads planned for this cloud? Everything is self-service and I think there needs to be a little bit of growth with our internal developers to be able to manage self-service themselves. But that's really the dev-opt model going forward. They have to understand better where their code is going and what it's doing when it gets there. So really it actually is driving, it's helping drive that model. So our next steps are to be able to just hand off a quota to each one of our developers and say, here you guys run with it don't call us unless the instances start. Otherwise it's your code and it's your problem. Right. And I think my sales teams in the audience I have to ask the obvious question. What are your expansion plans with this fine product? Well, our expansion plans are we're launching a game a week that may accelerate and we're hoping that several of those games become hits and that we are forced into expanding. So that summarizes my questions. I'd like to open it up to the audience for Q&A either to Big Fish or to Blue Box. Anything we can tell you about the environment? Just grab the mic. Any questions? Nope. Or you can yell. That's a question for me. I take it. Repeat the question. The question was how are we deploying code to the instances and configuring them? So we had a we have been using CF Engine to configure instances so as soon as we bring up a node it becomes it classifies itself by the name of the host so it automatically installs all the packages it needs once you spin out that VM. That's done in less than 10 minutes. Then our release team utilizes Jenkins mostly to both hit the APIs on our Fives to configure the nodes and Vips and deploy code and actually what we end up doing is we hand that once they configure the Jenkins jobs we hand that off to the game producers and then we're totally out of it. So it's all we let these guys that have no idea really what's going on the game producers really have no idea of the technology that's going on. They hit the button and if it doesn't work they can hit the back button and we never have to get involved. What level of orchestration and are we using heat I think is the question? Yeah this is we're not using heat in this deployment this is primarily Nova, Neutron, Horizon, Keystone. So it's a pretty standard installation primarily focused on compute. They had we have a mixed environment with a lot of existing IT infrastructure on the storage side so they're leveraging a lot of that that's there. Yes sir. We can both answer that once you go ahead first. Yes. What experience have we had with PCI audits and other compliant organizations? Right since most of this is in the mobile world it's not our problem because we're tied by Apple. You can't do anything except send your money through Apple and hope that they give us some of it. It's similar with Android so we don't deal with that at all. There are PC applications as well because we're pretty equal opportunity. Those ones we do have an e-commerce system but that one is well protected and sequestered behind a set of firewalls and then it just by API call sends the payload this customer bought these many chips into the servers so that can show up in the game but the interaction between the game servers and the client game itself actually doesn't have, there's no e-com that way the e-com goes this way and then back around. So we've been in pretty good shape with PCI actually. And for Blue Box we host a lot of HIPAA PCI based companies, compliant companies we do regular PCI audits with several of those companies and you're operating history of compliance and security so it's very common for us almost on a monthly basis we have somebody doing a tour and an audit. Any other questions? Yes sir. So the question is what release and how big so this is based on Havana and this particular instance is still fairly small it's under 20 nodes and we have lots of those for lots of different customers our primary focus is building lots of private clouds for lots of customers rather than building one large cloud for public consumption. Yeah, we expand with Blue Box by the hypervisor so when we need more nodes if our hypervisors are full we get a whole one new box and that shows up. It's great compared to Khan because the oversubscription is completely in our control so we can oversubscribe all we like which is great, I think we have some future plans maybe to put QA in stage environments on a certain set of hypervisors and oversubscribe the hell out of those and then keep our production ones a little bit more one to one. And we're able to get one plus. We're able to give that level of control to the customer so they can manage the subscription rate or use host aggregates to focus workloads towards a particular part of the fleet so it's actually much more flexible than a lot of the standard. Yeah, so we have the, so to big fish people in my admins we have full access to the API and CLI tools what we expose to our partners is just the interface and they usually just go into the portal and get a console to check out why their code is broken but they have a much limited access which is perfect it's exactly what we need. Yes, can you step the mic I can't hear that for him, sorry. How do you provide high availability within OpenStack? Ah, yes, so we have a distributed control plane our product is based on a three node architecture initially so every private cloud gets three nodes and that's where we do HA for the control plane we slice a little bit of the capacity off of those first three nodes for Nova and for Neutron and the HA databases we have the third node there as an arbiter for the Percona database and we're using Ucarp on Neutron for HA failover and HA rabbit for all the queues so those first three nodes just we shave a little bit off and then every node that adds on to the cloud has full service capacity. Correct. What were the challenges with I think there was any challenges in the integration between the split environments of Blue Box and Big Fish where your databases were in your data center? Yeah, I mean so it was a little challenging on well it was easy on one hand because that's what the DBAs wanted to do because they like their bare metal and their server and everything the biggest challenge I think was the networking component and the access controls it's all minus QL Percona minus QL and they like to keep those access controls really tight and when we first deployed there was a little bit of weirdness through the Neutron networking layer at first it was all matted through one IP and then we solved that problem and I think we're going to actually solve that problem a little bit better coming up here in the future and the other thing we were concerned about was latency and since we're I think four floors between each other in a high-rise data center it's pretty good. Yeah, it's land speed between the two. Yes sir. No, it's actually part of our normal product so if they need to express host aggregates we actually do that configuration for them or if they want to change the subscription weight we can do that for them. They don't actually modify the underlying OpenStack. So we build and maintain the entire OpenStack infrastructure and give them the API UI and CLI and then do all the management. So the following question is what was the relationship there? Obviously we're doing advanced things with OpenStack how does the customer know to communicate with us and how does BlueBox know what the customer needs so part of our engagement is that we work very closely with each of the customers. We know that this is a big cultural as well as technological challenge for customers to move from traditional IT into an OpenStack based public or private cloud. So our value add is we really sit down with the customer and understand what their business case is, understand what their workload is and work with them through that. We keep everything pretty well productized but we can change a little bit in some parameters like the type of hardware we'll move to SSDs for some types of workloads or say to drives for other types of workloads or different CPUs but it's all really based on the relationship and understanding the business. Any more questions? Yes sir. That's right. It's an instance of OpenStack for each customer this really contains the failure domain so each customer has a contained control plane and service plane where they can spin up of course containers or food instances or heat orchestration as we've just deployed or so we keep everything in that scope our special sauce is the back end management is the bare metal provisioning is inventory management is the tied in support systems so that's where we do we're doing 100% community OpenStack but it's dedicated per customer and that way if there is if something an upgrade or a failure it's not big fish isn't affected by some other customers It's really isolated enough so I mean even in OpenStack instances one game pretty much can't affect the other game because they have their own nodes I mean they would have to do something really horribly bad to affect the full OpenStack infrastructure And back to the gentleman over there who asked about PCI this is why a lot of PCI hip customers like this deployment model it alleviates the shared infrastructure concern and makes those audits much much easier it's much easier to control security in that model Any other questions? The question is do we have any challenges with over the network environment? So the question is do we have any challenges with the network environment after we've deployed this I think that's actually the primary challenge I think Neutron is probably one of the most difficult components in OpenStack to effectively manage at scale but a lot of there's gotchas in there and I think that's our primary challenge has been in deploying and managing and maintaining Neutron I think the compute is very stable Swift is very stable even some of the newer components are coming out they're stable, they're easier to manage but Neutron is still where we spend a lot of our energy Yeah I can expand on that I can expand on that a little bit when we first did our spun up or proof of concept we wanted to actually do business with them the latency was pretty bad enough that it would have made some of our games fail OVS We knew from our previous experience with Zen that OpenVswitch was a problem and so we asked these guys is that part of one of the components they're like why yes it is and so we forced them to upgrade OpenVswitch to what 11 I can't remember the versions but 11 and that I wouldn't say eliminated latency problems but it made it acceptable Any other questions? Who's ready for some lunch? Alright I want to say thanks for coming and listening to us talk about OpenStack and how we got there if you have any additional questions we'll be standing up here for a few minutes we appreciate your time and look forward to seeing you around the conference Thank you