 OpenStack as a service, and if you want to start by telling us why OpenStack at InterNAP. Sure, so definitely. Sorry. Can you hear me? Great. Okay, so at InterNAP, we were actually one of the first. It's on. Maybe. There we go. So at InterNAP, we were actually one of the first, if not the first. Next we have Sound. I'll just hold it like this. Alright, so at InterNAP, we were actually the first. Out. Alright. Okay, let's try this one more time. So at InterNAP was actually one of the first, if not the first company to employ OpenStack back in 2011. And at the time it was really immature, so it was kind of an adventure for us. But the thing that we were always attracted to by InterNAP is just the idea of a fast-moving open-source project, which allows us to concentrate our development resources and all the cool stuff that we're trying to do above and around the plumbing orchestration platform. We really have no interest in developing the core orchestration platform. So just like Linux, just like Apache, it's all about writing coattails. I mean, InterNAP is a publicly traded company. We're 700 people, but we're still a speck in the face of companies like AWS from a resource standpoint. So from our perspective, OpenStack is really attractive because it allows us to build on top of, extend upon, kind of use to provide our vision of infrastructure to our customers and essentially time to market. That's really at a nutshell. Can I get the clock turned back on, please? So given that you're using essentially a standard orchestration platform, how do you differentiate what you do from what other cloud IIS providers do? Sure. So we definitely offer a public cloud. There's a few things that we do, given who our customers are and sort of our approach to infrastructure in the market. So a few things. We have a lot of co-location customers. These customers are generally struggling with trying to make their infrastructure more agile. They're kind of enamored by all the promise of the cloud holes, but they're restricted because they want control. They want to spend the capex. They're basically server huggers, right? So they want to keep their servers under their own lock and key. So one of the things we're using OpenStack for is essentially allowing those customers to call it private cloud, call it like an enhanced managed colo, but essentially bring agility to a more traditional service offering. Other things that we're doing, you know, one of our most popular products is a bare metal server, which is basically a, you know, no hypervisor, no virtualization, but just a raw server with an OS. And we basically use OpenStack and have extended OpenStack to essentially be able to provide bare metal service on demand, you know, just as another flavor. So our customers can basically, you know, provision both virtualized and non-virtualized resources. So those are some of the things that we're doing. We're also doing things, you know, related to networking. We allow people to extend their layer to networks. It's kind of about bridging the old school and the new fangle, right? Because it's really, this transition is really interesting. We actually think the most interesting opportunity isn't on the extreme cloud-only side of things, but it's actually helping, you know, these two worlds that are colliding. So what's the typical use case you see amongst your customers for OpenStack? It really, you know, as a service provider, it really, you know, runs the gamut. You know, anything from, I mean, key verticals for us would be things like ad tech, gaming. You know, we've even got some educational customers. Really, you know, any customer who was running infrastructure at scale, they're interested in OpenStack in one way or another. Even if they're not interested in using OpenStack specifically, they're interested in the advantages that it can bring. You know, agility, automation, consistency, you know, that's key. Here's an example. You know, we have co-location customers that have deployed a lot of VMware. And I think Ken was talking about something similar to this, but it's really common for these guys to have such a long provisioning time, even though they're fully virtualized, to the point where the network department and the application owners are in two different worlds, right? And the application owner doesn't really care. They just want their server. And from their perspective, it's a commodity service. They're able to swipe their credit card on Amazon. So why is it so damn hard for them to get capacity internally? And oftentimes, you know, they'll put in a request for the server. The server will be all automated, but yet it goes to the internal network people, and they sit on it for days because, you know, assigning IP addresses and VLANs are so complicated, I guess, right? So basically they're interested in, you know, more agility, more automation, and the things that OpenStack enable, the value that OpenStack provides are in those areas. They're not necessarily looking for OpenStack in and of itself. You mentioned developers swiping their credit cards on Amazon. What pulls them back into an OpenStack ecosystem? I think we have a term for it that our salespeople use. I'm not sure if I'm a fan of it or not, but we call them the Amazon graduates, right? It's very easy to start on Amazon, but costs can spiral out of control. And that's not a... It's not meant to be disparaging against Amazon. I think the same thing can be said for just about any public cloud provider. Our sales included probably, right? So a lot of it is cost, whereby they realize that if they have a baseline workload that's always on 24-7, you know, no matter what study or benchmark public cloud providers publish, it's not cheaper than doing it in-house. So that's probably one of the big drivers. So, of course, you're saying they're not doing it in-house, but they're still doing it with you. Yes. So you're running the OpenStack-based cloud. Correct. Why do you think it's substantially cheaper to get a cloud from you rather than getting it from someone like an Amazon? A few things. One is bare metal. You know, I think the dedicated server industry has different pricing dynamics as the cloud industry. There's no virtualization tax. There's no noisy neighbor issue. There's no co-tenancy at all. So that's one. And then two is just the ability to leverage OpenStack on infrastructure that the customer owns. So essentially helping them, you know, build a more automated, agile co-location infrastructure or similar to a private cloud. And how many of your customers have tried to do things on OpenStack to build their own OpenStack-based private cloud before coming to you? I think a fair bit have. You know, I mean, it's so theoretically accessible that there's a very low barrier to playing around with it. But I think people assume it's a product or assume that it's more productized than it is. So their first attempt to kind of stand the whole thing up and, you know, get a proof of concept going, especially if they're doing it themselves and they don't have really sharp systems people and network people and DevOps people, it doesn't tend to go well. They get in over their heads. You know, so we've definitely had customers before coming to us who, you know, probably gave it a shot and have basically failed. But I think that's going to change over time, you know, as the whole stack matures, as the installers get better, as the distributions get better. You know, but for now it's, I think it's a pretty high bar for someone, especially if they're not a technology company to implement it themselves. Tell us a little bit about how difficult has it been for you to implement OpenStack as a service provider at scale? It's been fairly challenging. I mean, we've definitely had, you know, some missteps. You know, like I said, we got started in late 2011, you know, with the Diablo release or even the Cactus release. So, you know, for everyone who's following, that's like, you know, ancient in terms of OpenStack land. I don't think even the most, you know, perhaps only the most optimistic OpenStack developers would have called Cactus or Diablo production ready. But, you know, we kind of got going with it. We also just acquired a company called iWeb at the end of last year, and they're completely punching above their weight. They're already in production with a public OpenStack cloud in Canada. So now our development teams are combined with iWeb. So we're kind of, if anything, doubling down on our commitment to OpenStack. We've also got our own beta as internet for a next generation cloud that we launched last year. So a couple of disparate efforts are hopefully going to culminate in a major new release for us in the first half of this year. And so how much are you relying upon OpenStack, essentially, for the capabilities versus how much are you building yourself? So we're building a lot of things around OpenStack, but we're definitely relying on OpenStack for all the core capabilities that it, you know, provides. I mean, we're fully consuming things like Nova, Glantz, Zinder. You know, we're using other vendors that are, you know, in the OpenStack community like SolidFire, and we're also adding a lot of, you know, call it surrounding, orbiting ecosystem-like tools, you know, like our own billing platform, monitoring, you know, integrating with OS, R-O-S-S, B-S-S, you know, adding some of the bare metal and hybridization capabilities that I mentioned. But we're definitely consuming it, you know, in full, and we're extending it sort of to, you know, put our vision of infrastructure, you know, and expose that to our customers. But, you know, 100% of our forward infrastructure platform is OpenStack-powered. And for you and your customers, who is important in your ecosystem that's related to OpenStack? So we have the belief that if you're operating a public cloud, you really need to have the internal capability to kind of pull the code, you know, make changes, have, you know, internal development capabilities to, you know, do that. So we're not really using too many vendors, you know, from a distribution to a consulting standpoint. But we are certainly using vendors who are playing in the OpenStack community, you know, and that's definitely an advantage to us if they have integrations or if they have some sort of commitment to the project. You know, so vendors like SolidFire or Orista or, you know, some of these other vendors are definitely interesting to us. But, you know, while we are using specific, you know, commercial vendors and partners, let's not forget that one of the most attractive things about OpenStack is the fact that it doesn't lock us into anything, right? I mean, every single component, whether it's storage or network or, you know, the underlying servers or, you know, just about anything can be swapped out. So the fact that it, you know, basically avoids lock-in for us as a service provider, aside from the fact that it avoids lock-in for our customers is hugely appealing. And in terms of, you know, who your customers are using, what are some of them? Some of them are, for sure. And what we've really, you know, kind of been pleased about over the last couple of years is the ecosystem of tools that actually integrate with OpenStack has really blossomed, you know, over the last couple of years in particular. I think when we first launched our first version of the cloud back in 2011, just about every tool that claimed to integrate with OpenStack actually integrated with the EC2 compatibility API, you know, from a different perspective. But at this point, our customers, while they don't necessarily have a tool that is OpenStack only, their ecosystem, whether it's right scale or scale extreme or, you know, even fog or chef or puppet or whatever, these tools are all fully supporting OpenStack now. So it's like a checkbox for them, you know, rather than some home-brewed system. How big is your typical customer in terms of the size of their OpenStack deployment? Definitely people at scale, meaning they have hundreds and hundreds of physical servers. You know, we certainly have customers that only have a single VM, but that's definitely not our sweet spot. Our sweet spot is people who are, you know, the so-called redshift users, you know, generally underserved by Moore's law. You know, they're sitting there banging against the table that Intel only, you know, increased the performance by 20% this year. So yeah, I mean, it's people who are scaling out rather than scaling up. No worries. You want to move? Yeah. So tell us what eBay is doing with OpenStack. So I can start a bit. So in 2008, we started developing our own private cloud and the main driver was Agility. At the time, there was no project like OpenStack, so we had to do our own. And when OpenStack started to mature, we looked at all our options and we started adopting an open source cloud to replace our in-house custom cloud management system because it gave us many advantages compared to a proprietary in-house solution. So we are using it in three main categories of projects. We have developer projects, developer cloud if you want. We have QA and analytics. And then we have also production in some area. So what's your role in eBay at all? So my role, I'm chief architect for the global cloud services that is developing cloud across eBay Inc. Hello. I'm the architect for storage and PayPal, but our teams are now merged together. So we're developing the PayPal side of the question was eBay is doing OpenStack deployment and PayPal, before I started this year, wanted to go faster. So they made their own OpenStack deployment and we had the advantage of not having as many legacy things to deal with. So we could start Greenfield and do whatever we wanted. So now that we've succeeded in that, we're combining back together and trying to reimplement everything together. So my focus is on storage and mostly center and object storage. So when you're looking at these use cases, what made you decide to use OpenStack? What else did you consider using? So I can talk a bit about that. Around like two or three years ago we tried to find partners to develop our cloud. We talked to many software vendors or vectorization vendors and we also looked at cloud.com before they were acquired by Citrix. So the main reason why we selected OpenStack was first the architecture which was very distributed. The language is also a plus because we can find developers much more easily than Java even though we are a Java, we are even Java developers at eBay. The community I think at large is more attracted by languages like Python than Java and also the size of the community as a result of all those benefits. So for us the main decision point between the two was who is going to win at the end and that was the indicator was the size of the community and the ecosystem that was started to build around OpenStack. Same thing for you? I was brought in actually as an OpenStack developer specifically because they had already chosen OpenStack. For me that's the focus. Tell us about the initial experience of deploying OpenStack. What was that like? What were the challenges that you saw? What went well? What didn't? Pretty much everything first it's nice and friendly and then you get to a certain size and you find a lot of issues. So our experience has been successful finding solutions to those issues and then trying to work with the community to find out if our solutions are the best or going to be supported in the future. So in that sense we try and stay close to the community trunk and not fragment off our own fork because we're trying to leverage that and I think that's a big piece of not going with a vendor solution completely is having something that's supported by the community and not supported everything ourselves. So you're not using a vendor distribution at all? No. What were the challenges that you encountered more specifically? Well I think as you mentioned earlier a lot of the vendors say they inter-operate and they don't really. Things of running out of resources and systems that were scaled to maybe go to a hundred instances we go to a thousand or ten thousand devices and things don't work so well. For example in the architecture I don't know if we want to get so technical but in the queuing system all of the hypervisors are sending updates of their statuses and how much availability they have. When you get to thousand ten thousand hypervisors all going through the same queue things get overloaded and things can't happen in the time they need to and we end up with strange errors because the code isn't designed to catch something that you end up with a lot of limbo sort of thing. So that's sort of just when things get big enough to burst at the seams that's where we're running into trouble. You're running a solution at scale. How big is your solution today? I think we can answer in partly. Our bigger open stack deployment is the 500 hypervisors on virtual networks using Quantum. So that's one deployment we have in other data centers multiple of deployments that are running different use case and we are looking at consolidating those deployments. The size is multiple thousands hypervisors more or less distributed across multiple data centers. So how much operational effort does it take to keep this open stack cloud up and running happily? So it depends if you are just talking about the operations of the open stack components. At the last summit one of my colleagues did a presentation that open stack is not cloud. Open stack is just the automation system or the framework that allows you to operate a cloud. But there's many other things that are going around open stack like monitoring like all your processes around capacity management fulfillment, all of that. And for us this is something that we used to do before because we were running our cloud, our original cloud. So we have a system for example for onboarding full racks of compute servers automatically and converting that into open stack compute. So for us this was not a gap it allows us to very quickly add capacity for example to open stack cloud. But I can imagine that someone without those capabilities would have much more effort to spend on this type of activities. So the answer is it depends it depends on your existing capabilities. So adding some open stack to already mature processes and operations is not so much of a stretch but if you are starting from scratch I think that there's a lot of things to look at and it can be overwhelming. What have you found to be the most effective way to engage with the community? Which community? The open stack community. We actually have employees who are on the various teams. We attend the summits we're involved a lot. You actually work directly on some of the communities. Right so we I'm part of the user community. We have tried to foster new projects participating into existing projects following very closely either directly or through some of the vendors that we are using so we have multiple ways to contribute back to the community. But the main recommendation is that someone that deploys a cloud should make sure that if there's any customization or anything that they quite rapidly push back to the community because otherwise they kind of develop this technical depth that accumulates and every upgrade is going to be more and more complex because of all those customizations. So right now we don't have like a metric to know how many changes we keep in house versus what we push to the community but we have no more than 10 changes that we did in OpenStack for our own use cases. So that's the range that you want to look at if you go much higher than that upgrades are going to be nightmare. And how do your users interact with OpenStack? Are you using the Horizon Portal? Have you built things yourself? Developers using a lot of the API? Sure that's kind of interesting because at first people think of it as Horizon, the GUI and then you may know about Asgard the Netflix's version PayPal supported that to control OpenStack. It's an interesting thing. But the command line tools, the CLIs are also if you're really going to do anything you can't click 100 times to launch these things. It's just crazy. So you quickly end up going to that but we've gone beyond that to where we're integrating in with the APIs directly we have a tooling system that was used to run with our old virtual provider that would make what we call stages, which is PayPal in a box and the developer pushes a button because they need a new dev environment and they push the button in like a day or so later they get their development environment with the OpenStack we've moved that down to like 10 minutes or so and so that's a big win for everybody and they can make as many stages as they like and they just use the same tool that they used before because the API calls go to Horizon and Nova and such instead of to whatever it was So your developers don't have to mess with that? They don't know. We have two type of users though we have users that are looking at efficiency, they don't want to deal with infrastructure and they are going to use platform as a service tools that we have and those tools are integrated with the OpenStack APIs but there are some developers that are trying to innovate and want to be more agile by trying to figure out new ways of doing things and they will develop their own like pass layer almost on top of the existing OpenStack API or they'll use the UI that is provided either Horizon or another custom one and we have like there's a legacy kind of developer we'll be used to using bare metal and storage services in an old style versus the newer companies that are being acquired that use all Amazon services so they're used to using S3 using EC2 and Elastic Block Store I think over the next couple of years that will be the standard all the new stuff will be all the kids coming from school use Amazon and they'll know that will be what people are asking for right now we're more pushing to the architects that are really bright guys that develop the applications the company is using they don't know about Amazon it's just some sort of thing and now the company is asking them to use it so we're trying to show them why it's faster better they can do more experience of people on board because cloud is just a bunch of buzzwords to them so So, Sarendra from Park, would you come up? You all moved there? Yeah, let's take out It's like a Carson shirt So could you tell us a little bit about what your organization is doing with OpenStack and what your role in that is? So I work for a company called Mock I think most of you are better Can you hear me? Yeah, there you go Can you hear me? I work for Park Same problem Simon is a bunch of researchers at Park who are on the big data side so we have a requirement to run the parallel algorithms at a very large set of data so we look at different solutions like Amazon Web Services we looked at joint we also looked at VMware and our researchers are not programmers guys I don't have any engineers to do this automation layer we need something, we roll it in we should be able to start working without any programming effort so we went to Nibbla guys which provides the OpenStack distribution that's where we started back in early 2012 the first time with the SX distribution of it so our use case is primarily for big data applications getting a lot of sensor data a lot of algorithms at a very large scale hundreds and thousands of models automatically score them, pick the right model and then give that to the actual business analyst to really find you on it so we do a lot of automation frameworks through the rendering visualization so we need lightweight containers which we can execute without really taking a lot of deployment time and also zero administration zero DevOps, zero programming and no system admin I only have six people in our support team to support 250 PhDs so that's where OpenStack really helped us a lot to automate some other layers through some sort of Chef scripts or Puppet scripts and give that tool to the researchers to really focus on their primary research and so what's your role? I'm a CTO for cloud and big data research at PARC and I manage a bunch of PhDs around high performance analytics and also high performance computing and big data and also looking at a lot of sensor data that comes in through a lot of Xerox projects so you've built in a bunch of additional tooling on top of OpenStack? Yeah we've built the whole framework to enable the researchers to with one push button I should be able to deploy we call it disposable Hadoop cluster I can launch a 10 terabyte Hadoop cluster or Hayu or take Spark cluster or any of the big data tools we can really deploy it in less than two to three minutes today and then run our experiments and tear it down when we are done so that's not possible doing either in Amazon or any other cloud services So you're still using the Nebula solution? Which one? You're still using the Nebula solution? Yes so we have three controllers total we have 15 oz in production so we started the project it's a long journey actually nothing worked in the beginning but at the end of the 12 months everything turned out to be very nice we started in 2012 April then we went production in 2012 in November so a pretty short journey then we upgraded that to be the Palsum early 2013 so now pretty much on Palsum right now we're looking to migrate to the next version so Nebula has been an amazing job for us to put some people on site to help us really find some of those things fixing some of those core capabilities for example, CNIMA assignment a pretty simple thing changing the passwords security groups some basic terminology problem people came from Amazon using Amazon Web Services how do I really use this with a new lingo and new tools and new terminology aligning all these things is really a big challenge for us so you're still on the Palsum release which means that you're almost three releases behind now as we go from Havana into the Ice House timeframe in April are you finding the pace of OpenStack in the next six months to be challenging in your organization or what's causing that? yeah it's really a big pain in the neck to upgrade them when we upgrade from Essex to Palsum we lost all our images we had to redo everything because there's a format change and the glance doesn't work the way it's supposed to work so those transitions are really painful so with the Palsum we stay on Palsum because that can do 90% of our work we're not really chasing out all the new features coming in in OpenStack we're allowed to go with all the virtualization network virtualization techniques but for now I think it's good for us to do all the big data research it does my work so why should I go with the versions? so you mentioned that the initial implementation was very challenging what made it challenging? a lot of things first of all you know rolling that in is the security groups especially and some of the APIs are the hard wire parameters for example I can only create 10 security groups a lot of our automation scripts create hundreds and thousands of security groups to really partition those parallel running processes so we really hit those hard limits and I cannot change those variables I need to call some of the vendors to can you change those variables? oh no you can't change there's a hard wire in the system so we don't want any of those hard wires and I want those things to be controlled by the higher level application layer services think of that as a TCP IP right so how much of code is hardwired in the TCP IP layer provide the higher level and application level does all the magic so that's what we want the OpenStack to be you give me the control to program it the way I want it to be programmed but then give me all the flexibility to turn all apps so what are you most want out of OpenStack or probably perhaps a question would be what would be compelling enough to cause you to want to go through the upgrade cycle pain can you repeat the question again? you mentioned that there wasn't anything really in the subsequent releases that you really felt you needed what could OpenStack do that would be compelling enough to want you to for you to go through an upgrade cycle zero loss upgrade just roll it in the new version it works seamlessly without redoing anything for me to rebuild for example we also work with SAP HANA so our the OpenStack the base deployment is very complex one I have 50 node clustered all together out of which I have 27 node Hadoop cluster which runs 2.5 petabytes of data analytics platform and I have 4 terabytes of SAP HANA sit next to that and then I have computational framework which is basically OpenStack based if I make any changes to any of these things I can't really roll these images hundreds and thousands of them and upgrade them within a short period of time I need a zero downtime zero maintenance risk free if you can do it I can take it if I can't do it I can live with existing version it does my job all right so what is the ongoing operational experience with OpenStack been so once you've gotten through the sort of initial hurdles in the upgrade what's the day to day like it's pretty good now earlier there's a maximum half life of my cluster was 7 days after 7 days everything disappears then all my images gone everything lost the story start all over again now actually we're not even looking into that no one knows there's an Ebola running there it is so seamless people go and ask if it's available to them and just punch a bunch of commands and they're operational they do their job and just go and what's going to be necessary to get you to adopt OpenStack more broadly in your organization for more than those analytics workloads yeah we're looking at the next big step is you know we're working with a couple of hardware vendors to create the innovation center open innovation lab by basically democratizing the data centric algorithmic research by bringing even all the Xerox researchers into that you know we have around 700 plus PhDs across the globe how do we really bring them to use all these they don't want to spend time really dishing out the VM who wants to really disher the VM I wanted to really a service provisioning I don't need to spend my time typical our data you know data research applications you know it's very complex hierarchical applications and I need to spin out maybe 200 HPCC application agents to process the data in parallel and that has to scale out automatically without really me adding a note to that so that will save a lot of time to our researchers so our next step is to take this expanded to the much higher level so that even our researchers in other labs and the second thing is we're also trying to bring the customers into that it's a secure cloud we have two regions in the cloud one is the public cloud other is the secure cloud so what we do all the edge computing is done in the public cloud so it's a private public cloud surprisingly it's on our own VPN so and then the internal cloud that's where all the data fusion takes place so that's very secure and we built around the ring of security controls we do all the data processing we have customers to connect their sensors connect their the devices into this network and use the park researchers we have 250 PHDs in Palo Alto and as backdoor research can do a lot of interesting algorithmic research can really take it to the market to commercialize or productize or democratize those innovations that's our next goal for OpenStack and to do that are you going to be continuing to write your own tools for the past vendors recently I invited scholar guys to come and try and connect us with the Google computing engine also Amazon Web Services to kind of a bustable computing mode and also cumulogic guys brought in their past to try it out with the Nebula so basically I'm opening up for people to come and try your applications see if that is something help our researchers our customers our vendors as well as no because a lot of security requirements we do a lot of applied research for our customers so a lot of data governance requirements if you read the Amazon 50 page terms of use document it will never pass through Xerox legal so I don't want to spend two months reviewing the document instead focus on solving the problem so bring the data into the park lab then our researchers can focus on finding solutions with the data governance pain and everything is contained in the park lab so that's easy for us to manage so going back to the group as a whole if each of you were to give one piece of advice to other companies who are looking and adopting an open stack based solution what would it be can I start start set our expectations correctly don't promise boil the version kind of thing pick the use case well defined use case with the R.O.I. that's what we did at park we're doing a massive project for L.A. basically doing the dynamic parking assignment very complex project so they were actually setting up their missions taking two to three months to set up a mission configure it and show the demo to the customer so we went to them your lead time is two months can I reduce that to two hours that's a compelling value proposition so at that point in the mall we are giving reducing the two months to two hours or three hours whatever that is then they get aha that's really exciting let's go to next project next project so you can set those expectations and keep progressing because there's so many unknowns in the open stack you don't know what you are getting into you don't want to expose yourself unnecessarily with all the risk factors by promising too much there's a big lesson we learned in park so I think I have to the first one build a private cloud versus using a public cloud I think that if it's like you said before just an extension of a virtualization platform with self-service maybe the value is not worth the pain if you want but if you you understand the value proposition that a private cloud is going to bring to your organization I think that it's better and you understand a bit more what features we need is to hire the right the right persons because traditionally in enterprise system administrators are not used to the type of components that open stack is using it requires a lot more application on the code that they are used to do and a different mindset and we found that we had to almost like train people to this new world and hire the right mindset up front I'd continue that and say that in the past you've been able to hire people that are specialized in network or compute or storage and they all have to understand a lot more about how it all fits together because it's all running in the same machine but further open stack isn't a package you just buy and turn on it's more like a set of tools for craftsmen who know what they're doing to use you have to have software I think it's really important that companies deploying open stack figure out the process and the cultural implications of what they're doing before and kind of get that scoped out really well and understand what changes they're going to need to make internally to support that kind of installation and then from a technical side we've seen some people attempt to stand up open stack and use a very reactive manual systems administration approach they're not treating their infrastructure as code they're not using configuration management they're treating it like one-offs just like they're doing with the rest of their infrastructure and if they're doing that they're just in for a world of hurt because things are just going to get inconsistent and fall fall over very very quickly so questions from the audience and networking as it stands right now do you just use flat networking because of the scale or do you have to use additional software? so on our case for networking we have two models we are using VMware NSX so most of our networks are virtual networks and we have production networks that are bridge network so our networking model is maybe where we spent most of the time on open stack we were one of the early adopters of quantum now renamed Neutron so I would say that this is maybe one of the component of open stack that is the most challenging because out of the box it's difficult to have a production deployment without additional components okay we've got one in the back my name is Udin private call thank you for the insights we've heard about many obstacles and challenges that relates to network in other aspects skills, expertise I wonder I haven't heard anything about security especially knowing that in the open stack world compromising a node could compromise the whole stack what can you share with us about security considerations so we can take that do you want to talk about the paypal side you know paypal of course security is very very important everything is vetted by our security teams and they're in agreement everything's kosher we do tenant networks we do lots of firewalls and things to segregate our data and our traffic and things like that we haven't really run into specifics no so I think that open stack by itself is not solving any security or adding more security edX that you would have otherwise I think it's maybe what is missing in open stack is a blueprint on how to deploy open stack in various compliance environment and kind of a reference implementation if you want but otherwise we didn't find specific issues besides HTTPS support missing in some components and so on but otherwise it more goes to what I was saying before it's not a package you just install you have to know what you're doing it's very easy you can install it totally wrong and be very secure there's nothing within open stack that'll make that inherently more secure or less secure at the end of the day your virtualization layer is not secure or you have problems with individual servers or hypervisors those problems are still going to exist under open stack and open stack problems are still you know very much there and are left as an exercise to the reader sort of speaking okay we've got some additional questions over here also for those of you that are online please go ahead and enter your questions into twitter OE forum is the hashtag and we'll make sure we get them answered for you I've got a question open stack storage performance have you ever run into any storage bottlenecks and what do you have different performing storage elements in your infrastructure I'm not yeah okay generally the problem that some of the architects say developer architects would have a question about his performance because it's virtualized so there's extra layers of hops one of the specific examples would be say Cinder with its block storage it maps it through the hypervisor if you're using an iSCSI target map through the hypervisor to your VM wouldn't it be faster to connect directly with your iSCSI we've done some measurements and we don't see any significant one, two percent perhaps because the VM the hypervisor is not doing that much but I really need to see more measurements and I haven't seen any white papers or anything maybe other people know more in general the benefits of having the automation of the storage deployments and the abilities of flexibility you get outweigh any sort of performance impact for the day intensive applications we did notice significant difference when we run the Hadoop applications on the open stack with the virtualization on the storage side and we saw actually almost 40 percent degradation in the performance for example we did run the Terrasart algorithm to see how it runs on the bare metal versus the open stack we saw at least a good percentage of degradation so we saw guys to see if we can able to mount those volumes directly as the data node so which can really accelerate I don't need to really do the pass through to the virtualization also other alternative we are exploring is the single root Ivo virtualization to see if we can directly attach through the pass through into the device so when you have the mix of these workloads actually we have research team focusing on the diagnostics what they do is they go and measure your workload characteristics and see how can I really optimize placing of these jobs so that the compute intensive job will not go into the hypervisor where it's contending for the network and storage resources we got pretty good the hypervisor so that I can distribute the jobs with the dynamically through the algorithmic way versus randomly distributing it we did a lot of optimizations at that layer the earlier version of OpenStack all jobs first goes and fill in everything first and then goes to the next hypervisor that's pretty random up it doesn't really work very well for the complex job scenarios we want that to be kind of a system then you have the random mix the first experiment we did second thing we took the algorithmic optimizations I can place the jobs based on the work characterization I think storage is really a major pain point on the public cloud particularly network attached storage and it's not just bad performance it's inconsistent performance and performance that varies over time of day or what day it is from an internet standpoint there are additional vendor landscape for people who sell sands there are very few people thinking about how to enforce QoS on a per lawn or per tenant basis solid fire is one of the few companies that are doing this most enterprise sands don't address that at all the other thing we're doing is we offer bare metal so essentially the SSDs or the disks on that storage and consistency of storage performance is a huge problem on just about every public cloud I think you also have to take into account the architecture we use bonded 10 gigabit networks off the hyper visors to pairs of 10 gigs which is everything's 10 gig if you're doing this with one gig networking you don't want network attached storage it's just going to be a bottleneck that would make a total difference if you've got a traditional you know all of your 10 gig networking that would totally change your plans on how you're going to allocate your space but that said 10 gig networking is getting much much cheaper copper SFP plus Cable 8 is really cheap it's definitely the way to go we're going 40 gig networking bonded soon so network storage is I think the I'm biased I'm in storage but network storage is the future of storage a couple more questions that we want to get answered so being here two hours so far I still have no clue what is the state of OpenStack could you guys offer us you know one single slogan like let us know what is OpenStack trying to accomplish you know like Linux we all know Linux is open source Unix Chrome what is OpenStack you know I'm not sure where the people to answer that but my quick answer is it's an open source virtualization API specification what's it trying to accomplish so for us I can answer what OpenStack is doing OpenStack is the cloud management system that we are using for implementing either private or public clouds for example PayPal, eBay and other companies that I've worked at we've had deployments of the data center we shut down the data center and do an audit of the machines and find some of the machines have never been used they were allocated very carefully in projects but five years later no one ever even logged into them no one ever pushed code to it but they got lost so we end up with 10, 20% of the machines never used if we could use that allocation for other projects before we found out that they were not used we'd save millions of dollars a quarter so this OpenStack source code is the way that the company is implementing that virtualization instead of going to a vendor and being locked into that virtualization I think you can also I think you can also look at OpenStack as the open source vehicle that has the potential to knock many billions of dollars off the market caps of companies like VMware and Amazon I mean that's not a I don't mean to be provocative about it but I mean that's really I think the expectation is it's a democratizing open source thing right I mean framework it gives you options it keeps the vendors honest right and also it offers the great programmability right that's one of the things you can you program the hardware Microsoft did it through the HAL to make the windows work with all the drivers but after that there's no innovation available openly for to program the network switches programming the routing philosophies are you know able to partition the hardware into slices there's OpenStack I see it at the TCP for the cloud and then you can build nice applications on top of it the same lingua frank that you know what Amazon is doing or other cloud service providers are doing it provides a uniform language and syntax and semantics to talk to the hardware I'm wondering do you guys use Chargeback do you do Chargeback on your OpenStack environment and if yes what do you use and are you happy with it so I know where the question is coming from so I'm sure the others will have different depending if you are a service provider or if you are an enterprise but what we do is we cannot do Chargeback the way a service provider would do because of the way our finance system is built so what we do is almost like prepaid plan for cell phones organizations are transferring budget to our organization at the beginning of the year so for example like the MyEbay group we have that much CAPEX for the year for new projects and so on so they would transfer that money to us at the beginning of the year and then we would provide capacity instead of providing like previously instances of server instances we would provide just the capacity on demand so we pull this those resource into one large common infrastructure and we then offer it on demand so we we don't do directly the typical Chargeback but we do a showback to show how much they consumed out of their allocation so we are not doing cellometer right now because we used to have so is 90% on our legacy cloud software and we have our own mechanisms to do capacity management showback and these type of activities but we will definitely look at for the capacity that is an open stack using cellometer for tracking the fine grain usage which our other software is not doing for this cloud you are a very popular audience channel so we are going to squeeze one more in thank you my question is how do you back up the huge data in eBay sorry how do you back up the data in the cloud so what we do is we bring compute to the data so we have very large clusters that are holding most of our data warehouse and we a bit like kind of a placement trick we provision capacity closer to where the data exist on the networks where the data is located basically we ask the NSA for a copy of whatever we lose we also use a cluster to mount those datasets closer to the compute node so that you don't need to do any kind of moving the data around okay big round of applause for this panel very many thanks okay thank you all just to get everything