 Well, welcome everyone. Thanks for showing up, beer to follow, but we hope this is going to be an interesting lead-in to the booth crawl. I'm Ashish Mukherjee. I manage partnerships for Mirakura. I joined my first day at work, was actually the first day of the Vancouver Summit. So I know what it is to be new to OpenStack and to have to learn a lot from the firehose. And we're hoping to give you some entertainment and some good concepts of what to do if you're just starting out. Mirakura is an open platform in the spirit of openness. We are open to hardware and software and services vendors from Acosta spectrum. And we have several here. Before I introduce them, let me say we have a bunch of questions that we came up with in advance, but we would love it if you would ask the really tough ones that get some disagreement going. And whatever's on your mind. It's not about hearing our voices. It's about answering your questions so that you can really learn and thrive from OpenStack and not repeat past mistakes from history. And here we have our final fan list. Thank you, Rob. Wasn't a subtle way of doing that. Sorry. Why don't we lead off with some introductions? OK, hi. I'm Rob Esker. Working on OpenStack for, I guess, five and a half years or maybe a little bit more, whatever the math works out to within a couple of months of inception of the project. I work at NetApp, so I spend all my time in product management strategy, but have done work on the architecture and deployment end in the early days, a little bit of development as well. I guess in terms of the community, I'm a co-founder of Manila Project with Ben Schwarzlander and also serve on the OpenStack Foundation Board of Directors for the last couple of years. My name is Mark Williams. I'm the CTO of Redapt. Redapt is a systems integrator. We help companies typically in the web and tech space fill their data centers full of servers and integrated racks and networking and also help them deploy cloud technologies on top of that. I've been there for about three years. Prior to that, I was a customer of Redapt. And working at Zynga, where I had the good fortune to take Zynga during its good years of exploding into Amazon, consuming a lot of public cloud and ultimately building a private cloud based on CloudStack. And I don't know if everybody's following Zynga anymore. No, but certainly your Facebook feeds aren't, but now they're completely back in Amazon. So that's a recent development there. My name's Frank Rigo. I work for SUSE. I manage technology partners around several of our ecosystems, including OpenStack. Been around the OpenStack infrastructure since we released our first product based on Essex. We have a, if you don't know, SUSE is a Linux distribution company. We have a distribution of OpenStack that we call SUSE OpenStack Cloud. And I blame Mark for a lot of my time wasting if you're related to Zynga. I'm just saying, yeah. Hi, my name is Takahiro Kojima. I work for Fujitsu, you know, Fujitsu. And yeah, actually, Tokyo based IT company, ICT company. And I am a product manager of Fujitsu OpenStack product. So I have some experience on delivery, OpenStack system delivery to our enterprise customers and public cloud K5. So I'd like to give you some stories about struggle struggling with OpenStack in touch our customer and K5. Thank you. Thank you all for coming. I think, can you hear me? I think Takahiro gets the prize for having traveled the furthest from Japan. So, and he mentions struggles. And you know, I have these questions and some of them sound sort of negative, but it's not about that. We're all here because we think OpenStack is great or we think OpenStack at least might be great. The idea once again is that we've all made mistakes. We've all seen mistakes. You don't have to make the same mistakes. We wanna arm you to make your own mistakes, which are gonna be more interesting. So we're gonna go through these questions. You ask questions. We love OpenStack. These are tough questions, but that's it's a reflection on our collective struggle. So I see about a hundred people in the audience right now, a few more coming in. Can we get by show of hands, how many of you are operators rather than vendors? Okay, a small maybe a quarter operators. Then how many who admitted it and how many are actually running OpenStack today as a POC or in production? Okay, so there's a lot of you who might run OpenStack in the future, I guess. So there's a lot for the panel to let's say prove. I wanna start off with one of my favorite questions, one that wasn't contributed, which was like what's a great story of something that falls in the category of a disaster. It really was bad and then somehow the ship turned and maybe there's a lesson in there. Who wants to, Frank, if you wanna start? Sure. I mean, you hate to use the word favorite disaster in OpenStack in the same sentence, especially with people that are new, but from the viewpoints of an OpenStack distribution vendor, we talked to a lot of companies that initially started with getting the bits from upstream. So it's an open source product. They can download the bits and start that way. And so there's a lot of struggles that happen in that sort of a, when companies start that way, not that it's impossible, but it's just very complex. So I think that's probably pretty common that we see that. I think another very common thing is when the businesses don't get the teams that are integral to the project involved. So the OpenStack project sits in the cloud team, but they haven't integrated the storage team or the network team or the application development team as well as they should. We had a project ongoing at a fairly large company that was a proof of concept and was going fairly well. And several weeks down the road, we get a call from the company and they say, yeah, things were working great in the POC. Everything's working, we're very happy, except when we moved into production, something happened to Cinder and it just, the performance slowed way down and we're not sure what happened. And we said, well, that's interesting. Is there anything changing between your POC and your production rollout? They said, no, nothing. Well, the storage back in changed, but other than that, nothing. So it was like, well, maybe that had something to do with it and it turns out that they were different, just different things about the POC storage vendor and the production storage vendor. So the lesson there was, yeah, it's important that you don't do this as an isolated project from the cloud team and expect it to work right when you roll it on into production, when you have to hook in all these other peripheral components. Yeah, most of the disasters I see are two parts. First is, right when you begin talking to the potential prospect for OpenStack, you ask them, well, what workloads are you trying to deploy in this private cloud? Or you'll say, well, what are you trying to achieve with this? And oftentimes I get, well, my boss told me to do this and there's really nothing more than that. They're trying to have technical credibility by saying we run OpenStack and there's really no business purpose or workload that it's aimed at. And when it's kind of a big red flag for us, we'll stop and say, look, you have to really know this to make an informed decision about what architectural decisions are going to make here. The second one is further down the path, once you've collaborated with a customer, they know directionally where they wanna go. Many customers tend to want to pick all of the fringe features, not necessarily those big tent projects that aren't core, but even within the core projects, which Cinder and Glantz and Neutron all those, within there, they'll wanna make decisions that drag you into the weakest features in there that really stretch the edge just because they believe they need to integrate with incumbent technology for some reason. And that's where I've heard ISVs who have OpenStack distribution say, that's like playing with a block box of razor blades. Don't do that. And it's very hard to steer that initial customer who in their first deployment to looking at keeping things simple. Keeping things simple gets you to that next step of you learn what the behavior of our workloads are on something that's relatively consistent standard. And then now you've learned more about OpenStack. You can take that next step and begin to dabble with some of those newer features. So I don't know if I have a favorite, but maybe I can briefly touch upon two so that I can fit it into the space of one. So I guess this is in process. So it looks good. I think it's a success, but not fully complete, but a large sort of online gaming company not SINGA, who I'm not sure actually is particularly well-known, is actually an OpenStack employer and had been since the earliest of days. They were bit by a couple of things. One, having been amongst the earliest adopters and directly from the upstream. So on one hand, they derived benefit very early on. On the other hand, not much was particularly mature at that time. And in particular, seemingly pretty obvious problem of needing to go from one major release of OpenStack to another was not particularly well-solved for. I'm not suggesting that it's a seamless and wonderful experience today, but it's dramatically better now and has successfully improved from one release to the next. So they were bit with having been early adopters, having to deal with some of the early pain associated with not quite mature, but furthermore not being able to get off of it. And as of course, successive releases were made available, the problem got worse because it was OpenStack from two years prior was substantially different and it was essentially a replatform exercise. It got to the point where actually some of the talent that deployed it originally ended up leaving. And so you had a certain amount of institutional expertise that made this thing tick in the first place and then suddenly you were handing this sort of what was at that stage an experimental project and handing it off to mere mortals. So what the heck do I do with this thing? And that was certainly a failure or it was not a failure in the sense that it fell over but they couldn't get off of it. The success portion component of that I guess is that and they didn't start down this path initially once this new staff became or once the new reconstituted admin and operations team was put in place. They started looking what we'll derive from upstream directly. I think what made it a success is they went with a distribution provider who has thought about and is delivering capabilities that contemplate what it would look like to go from one release to the next and abstract a lot of the complexity and make it deterministic. And wow, even actually provide a support model for it. So it made it consumable. Like I said, that's still in process but that's maybe two separate things. One, beware the sharp edge of being on the cutting edge and two, I think there's something to be said for going with one of the distribution providers. If it makes sense to go with upstream directly and roll your own then you probably not in this, well I'm not suggesting you're in this session, I'm saying you probably have the expertise in house type deal with those problems. Wow, just within a few answers, the three of you have touched on essentially every question I had, which is great. It shows we have the right questions and those are the hot topics. So I mean, carrying on, I wanted to, oh, we have a question from the audience. Well, why not? Go for it. So a lot of times I wanna hear about OpenStack being deployed, it's a company-wide initiative. But what if you're an app developer and let's say for whatever reason you need to sprinkle resources in 50 different locations around the world, say about 100 VMs per location. Give me a gut check of what kind of investment am I gonna need to do and will it need company-wide initiative in order to get it done? I mean, is it possible for the app level to roll out something like that or would I need to hire 200 OpenStack engineers or pay somebody $5 million to do it for me? Well, there's several different approaches you can take but certainly without the governance and partnership with IT, if you're the app developer, you're probably not gonna get the funding through purchasing and that related part of your business. But to the extent that you can go into that partnership, you, and if there's no expertise of OpenStack in-house, you're probably wanting to look at something that's like a managed, dedicated, private hosting approach. Rackspace or similar have places where they can do that, Bluebox is another one. And at that point you then are shopping for, well, where does that service provider have the locations? If you really need 100 different locations, if that's what you, I can't remember if that's what you said. Cloud gaming. Cloud gaming? Okay, yeah, you're gonna need lots of little points of presence. So that'd be the starter way to go to start quickly and start determining if OpenStack as an IaaS and another platform capabilities provides the automation that you really need. Beyond that, then once perhaps you have some, critical mass of a team that can own and operate and maintain an OpenStack, you're gonna want to very quickly co-architect with either a like a SUSE or other commercial provider of an OpenStack that can help you define the physical architecture and making that consistent in every one. So again, you don't wanna create 50 different snowflakes, versions or hardware compositions in that. You want it as normalized as possible. And you want that guidance from that commercial provider to make the right decisions to stay within the boundaries and not get into the bleeding edge features. Then at that point, you can work with like a systems integrator to help you build those consistently and install it consistently from the physical layer, every version of every component in the server matters, getting all of that. And then successively move towards more independence as you do that, helpful. So big investment. It's a, yeah, I can't give you a dollar figure, but it's probably several hundred thousand dollars depending on- And if your CEO views IT as a necessary evil and you'd rather just pay Amazon, it's gonna be a tough sell, right? Well, does it become a competitive differentiator for you to have your own technology? For Zynga, it absolutely was. At the time that Zynga started moving out of Amazon, like there were still growth pains in Amazon. There were lots of significant outages. Wasn't the performance that we wanted. All of those things we were able to make better in Zynga's cloud called Z-Cloud. And now I'm getting into the next, potentially the next question, but I'll save that. But anyway, yeah, it's a substantial investment. And then you're still competing with incumbent technology that your IT has as well as Amazon and other public cloud providers. And that's a hard sell. Thank you. So staying on the theme of starting with something and then preparing to grow it. I wanted to ask Tucker here from his experience with K5, what is your experience and your recommendation of starting with a POC, which many in the OpenStack world are doing today? And then how do you, while doing that POC, think forward to production? What to do? What are the mistakes people make? Okay, so I'd like to explain about my favorite disaster. Actually, my opinion is very similar to the flank. So the POC is very important for open source. You need to demonstrate. You can deliver the value to the customer, actually, you think that open source, open stack. So after that, yeah, as Frank said, yeah, you will encounter many problems when you are moving to the production. One problem we encountered in K5 testing is that compute service or compute service, that's a 250 compute in that time. At that time, all went down. How that happened? So actually, we configured a parameter in heat to increase the performance of heat, the worker's stress, increase the number of stress. Then we encountered, we found a problem in the HAProxy, you know, HAProxy, and that HAProxy, we found a error over maximum connections. So we configured the kernel of the HAProxy VM and increased the parameter for max connection. Then, all computes down. Why? So it's max connections now are maybe are 10, 10 so thousand, I think. So in that configuration, all computes try to access to the database by your conductor. So that means that all computes try to connect to mq, message queue. So many, many messages that are queued in that server. So, you know, our OpenStack service cannot work without mq, so all OpenStack API fails. So we are powered up. We scale up the message queue and wait for several days and et cetera. So what we learned from that is that in the large-scale system, it's very difficult to expect the result of a single configuration change. Even experts cannot detect everything. So you need to test in the scale of your production before upright to the production. So that's my recommendation. Mark? I just want to add, so I totally agree that establishing at proof of concept stage that indeed there is proof of concept is critical. But one of the things, and it's actually hard to expect your first question of how things can go awry or maybe things that didn't start out so particularly well are great. I've seen a lot of POCs try to lump in every possible shiny object, you know, like emerging technology into one. And apologies if this comes across this common sense, but I guess the lesson I've taken away is try to start with OpenStack alone and then layer in perhaps your neutron plugin of choice or perhaps your cinderback end of choice, but not necessarily at the same time and maybe in successive phases. You know, there's a number of emerging technologies that are synonymous in some ways with OpenStack, but not actually part of OpenStack. And while they may be valuable and you may want to incorporate them, it might make sense to evaluate them on a slightly different timeline. Fair enough, although I do think there's some neutron plugins that just make your installation so much easier. I wanted to turn to a big picture, but pretty important question as we're discovering which is you've built this great technology, but what then, do you build a cloud and hope that the developers will come? Is it a pull or push model and how do you evangelize this essentially technology or platform within the company and how does the success of OpenStack depend on adoption? I mean, it does depend on the success of adoption, so how do you drive that in line with the actual install? I'll be brief. So I've seen it work and I've seen it fail. I've seen perfectly functional OpenStack clouds not be used, not be consumed. Likewise, I've seen places that were perfect, OpenStack was perfect for and it failed technically. The point is the places where it seems to stick the most are where there's a tight coupling to the application development community or those or perhaps actually on a way other end of the spectrum, maybe like a mandate that says, hey, and this goes to the whole OpenStack as a snowflake, it's hexagonal and frozen, but boy they sure do look a lot different from one to the next and you've probably heard that, it's really not original. The point is is some come to OpenStack from a perspective of I want to actually, I want to replace or maybe augment or provide a foil to like the incumbent enterprise virtualization vendor and I won't mention who that necessarily is, but the point is is that's a very different set of motivations from those who are trying to build like the cloud, fully cloud native, like next generation buzzword compliant cloud. The application developer coupling is probably more important on the latter spectrum, on the first side of it, it works best where there's that corporate mandate to say you will move your existing application base into this particular model and the place where I guess in particular where it works is if you actually can solve for the accounting problem who actually ends up actually paying for this thing. If there's actually a mandate that states, there's not just showback, but actually charge back. Now I'm not suggesting that's an easy thing and OpenStack doesn't solve for it, but where that's been done well, it seems to succeed overall. So I had this experience in 2011 when after consuming about 20,000 instances at any given point in Amazon, I built a 4,000 node cloud stack cloud and I couldn't get a first customer to move into it. And exactly what Robert said, the motivations of developers, I mean, their priorities, especially in gaming, is to continue to maintain the satisfaction of the users of our games. They don't care about cost until it becomes a mandate, right? So it really took collaborating with senior C level executives to make sure that cost mattered and we did have accounting. So we had to make sure that the cost per unit consumed by month in Amazon versus ZCloud was different. And actually it was a third cheaper to do in ZCloud. But even with that, it took extraordinary effort to collaborate with each partner on the gaming side to actually move them. Because again, they had to spend like nine weeks with us to migrate that application because it's got active users on it and that's very hard. But successfully starting with that first anchor tenant and building confidence, getting them to be your evangelists is the key part to success long-term. Yeah, I mean, I don't have a ton, I agree with both of you. I mean, we've had experiences where the technology is so new and so shiny that people just gravitate toward it. And so we end up doing some of these POCs where there isn't really an idea of what they wanna do from an IT perspective or from a line of business perspective. And more often than not, those don't go very far because you do need to get all the other groups involved eventually to make this thing a success. And so where we've seen the most success is where actually the line of business is driving something or the application development team is driving a particular goal and those generally succeed. So I have several more questions I'd be delighted to ask but I prefer to get them from the audience. Thank you. I'm Alex. I would not mention names or colors or anything but just I work for one of the largest telcos in Latin America and I'm located in Brazil. And we moved to the point beyond the proof of concept. As telcos, we were basic, we started doing cloud and after a few attempts of doing lots of crazy stuffs, we decided, okay, OpenStack is the way to go. And we were supported by the big names in the game. And I have a story to share with you guys is that even within the proof of concept, there are certain things that you cannot prevent or you cannot foresee. And I wanna ask you guys for some thoughts about that. What do you guys advise us to do just to give you a quick example? The whole OpenStack thing might be a buzzword and as Frank said, there are some, it's Frank, right? I'm sorry, okay, there are certain things that even within the core products there are major enough to be considered production ready. There are some details that made the whole difference and currently what we're doing is we do have a large storage solution based on OpenStack and we did the proof of concept and later on we figured out that we could not measure external transferring from internet because our customers, we do private and public cloud and for object storage, the way to go is you charge for volume and transferring and the solution that we deploy it, again, I won't mention colors or names, was based on SAF and SAF is not integrated very well with SWIFT at this point. I mean, you can do lots of things but measurement, internet transferring is you cannot differentiate between what it's external and internal and we are now at the point that people at the, you know, the high cleric, I mean, the top executives that invested in the product, I mean, in the technology, they believe it and they cannot differentiate what is SAF, what is OpenStack, what is SWIFT, what is Cisco UCS and whatever, they cannot do it and the only thing that they see it's the OpenStack buzzword and they say, that's not ready, you guys told us. You see, you know, I just mentioned Cisco. Cisco is, it's pushing OpenStack all over the place so now we, I'm a product manager and I wanna deliver a product that is cost effective and I cannot do it because the top execs are not, they didn't bought it. I mean, oh, I can't believe I spend, you know, as I said, we went beyond the proof of concept that we do have many nodes running and we are not charging our customers because of a very glitch and the expectations are super low now, meaning they are thinking about getting back to, I don't know, EMC and doing the way they always did in the past and just because of very small thing and how do you guys see the bandwagon including the larger telcos, I'm sorry, the larger vendors in the arena. When it comes to these small examples, I have a few of them but just, let's just stay with this. I think you actually have a question kind of built around this which kind of goes to the point as OpenStack been oversold in some way. And, you know, there's a hype curve. You've seen the hype curve from Gartner, right? I mean, this industry is full of hype, right? It was big data a couple of years ago and containers and OpenStack. I mean, the technology is awesome but it's kind of dangerous. You really have to know, going into it, what it's built for and what it can do and what it can't do and sometimes that's difficult, right? Especially if you're talking to somebody who's supposed to be an expert and they're telling you one thing but in reality it's something else. I don't know actually what the solution is. These guys maybe have a better idea. I'm not gonna claim to have the solution but I did wanna riff a little bit off of what you said. So the hype cycle of innovation for those who are not familiar, Gartner's, you know, attribution to Gartner, you know, there's a peak of expectations filled with a trough of disillusionment and the slow sort of ramp towards productive use, right? It's important to understand with OpenStack that you really need to think of each different project as existing in different points on that hype cycle. So something that's very nascent, you know, a newer project perhaps on the fringe, not necessarily, you know, been around for several releases is gonna be quite a bit further behind. Something like Nova, you know, not perfect but generally a known quantity at this state. And so I guess it comes with a little bit of a maturity in having traveled down the path of working with open source projects that you need to develop some organizational confidence about the ability to adopt Open in the first place. The extreme end of it, it seems like you maybe you're encountering some of it. There's a pretty, I guess I won't mention the name, but a couple of articles that have come out where. The opposite of green, that's the only I can tell you. Yeah, no, I mean, and I'm not sure, yeah. So the point is, there are folks who are in the business of providing the proprietary alternative. And it is often expedient to impugn the open. And it's not that hard to find the warts. And so unless you've like invested in building the institutional confidence in something that is solid, you might end up burning yourself by getting too close to the leading edge with the open thing. You might have burned open in the organization for a while. So I don't have the answer, but a few thoughts. No, that's it, that's it. This is where proprietary closed commercial solutions actually have a lot of value. I was a customer of cloud.com. And we basically owned them as an engineering organization to deliver the scale that we needed. And it was huge, like we got to 42,000 nodes. You still can't do that with OpenStack and it's been six, seven years. That's particularly frustrating to me, but it kind of goes back to as somebody consuming an open product that even might have commercial support for it, you don't have as much leverage on that because it's ultimately tied to the community and the release cadence cycle of OpenStack itself. And it's extraordinarily hard. So then it becomes your fiduciary responsibility to focus on, okay, we can only do this. We can't boil the ocean to completely transform our business. We have to do more iterative consumption of this technology, improve that each point works as we go and expand as we go. That's hard to do. Okay, so we definitely cannot end on a note if proprietary solutions are better. So let's take one more question from the audience. Hi, my name is Raj and this is my first OpenStack. So bear with me if I sound a little ignorant. So my question is regarding choosing that first project for a POC and a lot rides on it because you're kind of building the credibility of OpenStack as a technology that can be used within my organization, right? So if I look at all the use cases and at least the ones that are out there, front and center, what I'm seeing is that most of the use cases have something where the application that's developed is very custom. It needs to scale to something like 2000, 3000 nodes and things like that. So is OpenStack a viable choice for your traditional three tier? I'm running an app on Microsoft SQL Server or Oracle. Is OpenStack a viable option to move that kind of workloads into OpenStack? Or it has to be something very, requires custom development, requires scale in order for it to be successful. And we have just a few minutes left between us. So I'll talk fast. So it absolutely depends on how you compose the cloud. So take something like Nova and Cinder and Neutron, part of the power of it is extensibility. There's a lot of different plugins and different options. So I guess I contend that the capabilities of the cloud you end up with are determined by that which you compose it with. So if you intend to, for example, deliver like high SLA storage with fully open on commodity gear capability, it's probably not the right fit. But it happens that there are back ends in the storage sense that can provide terse quality of service capabilities and other differentiated storage capabilities that align to those classic POSIX built applications for which it would be appropriate. I'm not going to suggest it's completely smooth sailing. There are some things that aren't quite there that aren't fully provided for. But I can definitely tell you, just as NetApp, we have in the three digits of production customers. And among them are some of the larger enterprises, largest enterprises, at least a few of whom are actually replacing enterprise virtualization with this, inclusive of all of the applications that they're delivering. It's not without pain, but they see the ROI down the path. I'll just ask, what business outcome are you trying to achieve? Are you trying to accelerate the ability to affect change and introduce change into this three tier application? Or are you trying to save cost? Certainly the former is more conducive to what OpenStack will provide for you. If you're really just moving it as is and there's really not much more change gonna happen to it, I think you're kind of wasting your time in that particular vector. And I would maybe look for something that's completely net new. And it was covered pretty well in one of the keynotes this morning. In the bimodal, the newer innovation side is more akin to what OpenStack and even Amazon, for that matter, are aligned to. Mostly the mode two is where it happens. Oh, two, yeah. So we, unfortunately, are just about out of time because I have 30 more questions to ask. But thank you to our panelists all for joining. And I believe there's beer, but for those who would like to stick around and ask some more detailed questions, we're here. Thanks all for coming.