 Good afternoon everybody. My name is Pete Chadwick. I'm a member of the Product Working Group as we participate in terms of the community to help understand what customer requirements there may be and reflect those back into the technical committees to drive that. And one of the areas we've been focused on is what does OpenStack need to do if anything to be more prepared for dealing with enterprise workloads. And so the purpose of this panel today is really to have free people who've been in the trenches working on enterprise activities, you know, share their thoughts, but we'd also like to encourage any any questions and comments from the from the audience. So real quickly let me introduce the panel. Again my name is Pete Chadwick. I'm a product manager at SUSE responsible for our OpenStack distribution, SUSE OpenStack Cloud. Starting to the left we have K. Tokenaga. He's the manager of OpenStack and Linux development at Fujitsu. He's been participating in open source environments in 2002 and his team is really focused on contributions to Linux and making things more high availability. Moving to the left Andrew Batty is a principal architect and developer at SAP. 18 years of experience delivering enterprise technology and he's focused now on defining, he's working with a small group of technologists to define the global infrastructure to run SAP's cloud products. And last but not least is Prashanth Rao who managed his cloud computing engineering team at Wells Fargo and they're responsible for the enterprise-wide deployment of enterprise cloud workloads. So with that what I'd like to do, I talked about the goals to start off the discussion by asking some questions to the panelists and as I said as we go through this if there's, I mean we're blinded up here by these lights so we can hardly see anything but if anybody has a question they would like to answer of anybody on the panel please feel free to do that. So just to start off real quickly why don't each of you talk a little bit about what your experiences have been with OpenStack especially in the area of enterprise computing? Yeah sure. So as Pete said, so I work at Wells Fargo and basically a couple years ago we started on an initiative to deploy an enterprise-grade private cloud within the company and Wells Fargo is obviously in the financial services large bank in fact the largest bank in the world by some measures and so regulated industry. So it's not as if we can have our app dev groups within the company basically use Amazon or Google or Microsoft Azure or things like that. So because that data needs to stay within the company there is a need to basically offer that same set of services for cloud computing to the app dev community within the bank but offer it in you know secured environment. So we started out a few years ago about three years ago with Havana and then we moved forward with Icehouse and we're currently in the process of upgrading to Liberty which is going to be a significant jump but we initially started with a single data center deployment and then we branched out to about seven different data centers with different clouds for different types of workloads and we're kind of continuing that expansion and yeah I'll leave it at there for now and we can kind of get into more details later on. Okay hi I'm my experience with OpenStack really started surprisingly about 15 years ago when we I was sat in a basement in our headquarters building what is now or would be recognized now as a sort of cinder nova clone and if I flash forward 12 years I'm still doing that and that was three years ago and by that time I think we had something like 20-25 systems doing exactly the same thing and somebody said why are we doing this and why aren't using OpenStack and nobody could give us a compelling reason not to and that started our I guess our real journey with OpenStack and we start with a very small to us pilot it's running about 1500 VMs now I think a little more and it worked very well we learned quite a lot I think we both good and bad and we're now proceeding on a sort of scale-out mission we're looking at Kubernetes containers 13 data centers around the world I think and yeah we're looking forward to it I guess. So my background is Linux development so we've been working on to make Linux enterprise grade for even mission-critical customers and the Fujitsu provides both public and private cloud service that are based on the OpenStack and so last February we've announced we will migrate all internal systems to the public cloud platform so the internal systems consist of more than 600 of systems across 13,000 servers globally that encompass a mixture of legacy and cutting-edge servers and to share our experience in the enterprise I'll just do I'd like to just explain one example of the problems we encountered in the past so in order to obtain more performance on the deployment we increased the number of threats workers in heat and that caused a max connection errors in HAA proxy so we configured the Linux kernel and HAA proxy to allow more connections and then that caused the total disaster so it's several tens of thousands of messages were issued from hundreds of compute nodes to the conductor that got stacked in the messaging queue servers and so causing the whole OpenStack system to be out of service so we increased the processing capability of RabbitMQ servers and they waited for like several days and then the problem was resolved so the lessons we learned from the experience is even a single small change can cause such a big disaster and it's really hard to investigate the problem root cause of the problem because OpenStack consists of many components and there are intricately interrelated and some of the components don't even provide enough data for troubleshooting so I think that's one of the things we need to improve in the future. That leads into the second question which is from each of your perspectives what does the community development team need to be addressing to improve the suitability of OpenStack for enterprise environments and who wants to start. Yeah I mean I'll start out with a couple of things so one of the things that we encountered over the last couple years was Wells Fargo has a lot of requirements for deploying servers. One of the key requirements is essentially that every single server needs to be tracked as an asset and so that means even a virtual machine and whether that was pre-cloud or within the cloud that asset as a server needs to be tracked in our enterprise asset management system and given that cloud instances right you can spin up spin down you may you know in a matter of minutes that requirement is very challenging so one of the things that my engineering team had to do is essentially build out interfaces that were event trigger based right based on you know messages off of Nova or RabbitMQ or Oslo to essentially fire off connections to say yes this asset is a real asset here's its UUID and IP address and other information about it pass it off to our asset management system well by the time that interface and because some of these systems within the bank for example are batch based by the time it gets processed in the asset management system it's it's no longer even alive so it kind of defeats the purpose for one and but two I guess with regard one of the things we had to do was we have to track a lot of metadata or not metadata in the sense of Nova but attributes about those assets and the tenants etc and what we found failing or lacking was essentially the extensibility what are that be extensions to hook into you know the Oslo messaging very easily because it's a little complicated or even just tracking additional metadata attributes about tenants and instances it made it challenging not to have kind of that extensible capability and that was including for asset management as well as for charge back I guess in a number of other those areas we found that okay you get a lot of the core capabilities that are great for you know let's say a public cloud and spinning up infrastructure but all of that other stuff to keep track of what you're doing in the cloud you know from a regulated point of view that was you know that continues to be a challenge for us another one that I think is and I think one of the well I think one of the things I learned in the last couple of days for example there's a new project called Mistral or at least new to me and that seems to have a lot of promise because that does seem to provide some of that workflow capability that I think will you know potentially can't address some of those needs so I think the community is getting there and then some of the product capabilities are evolving to support some of those needs and those hooks into the cloud infrastructure but I think you know that's what we've seen over the last couple of years yeah so as I said earlier one of our focuses is it's containerizing open stack and moving the whole control plane into containers and I think the big thing for us at the moment is to just be able to move the data plane off those containers and out of software and into hardware devices we're getting a lot of help I think from the sort of vendor extensions in the vendor community in doing that and it's a requirement but it's largely fulfilled I think it's very surprising I think how ephemeral we can start to treat the open stack control plane we can just spin the whole thing down and the whole customer workload is is still in place and I think that that vendor community is really powerful for us and I think there's a couple of other I guess issues that we see and things that really I'm not sure they hurt us but they cause us a lot of disruption and one of the things is around sort of intercomponent interoperability so something like Keystone v3 which has been around for I don't know six releases five releases and still there are things that that don't work with it and it causes us you know we have to patch we have to find the problems we have to change our approach to things and it's we have a similar problem where we're scaling our network with multi-segment networks and it was introduced I think two releases ago and it's still largely unknown and unsupported in a lot of in a lot of stuff and I think that there are two examples but there are others but I think that's quite quite a big issue in terms of the amount of time it takes us to adopt stuff I think the other thing is around it's not something we've really encountered and maybe some of the other guys can comment on this but I think we're hearing a lot around upgrade and the way and the release cycle API stability and making sure that they really are sort of contracts that are supported so I think it's all about you know consistency and making sure that things work around the entire sort of components that's quite important to us okay so okay I'd like to explain the what the enterprise means to us first so for us enterprise basically are the organization that that needs to run their systems without downtime so for example the bankings government and the stock exchange or something like that and so once such a systems goes down they will have a significant impact to the society so let me talk about three requirements from enterprise here because those are the ones I'm focusing on right now and so the first one is predictable and consistent performance enterprise systems needs to get predictable and consistent performance from compute networking and storage services and I think bare metals provisioning service should address that issue as because users can have physical server for themselves that means you can get away from the noisy neighbors that you may encounter on VMs and the second one is high availability so in case that hardware failure or power failure in the data center enterprise systems running there have to continue running on somewhere else and I think multi site open stack should address that and the third one is serviceability I think there is two things for that the one thing is an ability to maintain open stack system without requiring downtime so the cloud platform has to be able to apply a fix or even upgrade while the system is running now another thing is fast and reliable trouble shooting so in a case user encounter a problem we the root cause of the problem needs to be investigated immediately so that the user can apply an appropriate fix and then they can continue their business without further concerns so those are the three things Okay I think you raised an interesting point and I think for Sean raised it as well which is there seems to be a lot of debate in the community and I'd ask for a show of hands but again I can't see is whether or not that should be allowed in open stack there's a lot of people that say open stack is a cloud platform it should only be cattle we shouldn't worry about that but it seems that there is a requirement that you want to move what we would consider traditional enterprise workloads into the into open stack comments on that is that I think I'm not going to answer the question directly but it's it's it's quite related it goes back to another requirement and it's it's interesting you say that because we have a real issue when we try to put open stack into containers that we have to start running open stack as a pet it's like it's really you mean and it's from my perspective applications should be able to just start up and it doesn't matter whether the database isn't there they they they should just work and I think I think my answer would be yes and no because there's always going to be a legacy sort of application that you can't change but I think I think it's as a requirement for open stack it would be great if we could just start up nova conductor without rabbit mq and it not hang that that's that you mean which is which is a real sort of pet condition that we have to sort of deal with and I think um yeah I think it doesn't answer the question but I think I think those those pets should be as eradicated if we can and I think open stack could start by um by doing some of that itself I think yeah I mean I uh I think it's a little bit far-fetched and perhaps even naive to think that uh we're going to get rid of pets right I mean and and I think you know at least from my vantage point within you know a company like Wells Fargo in order to justify that that spend and and the product itself there's got to be this you know there's got to be some savings associated with it right or some you know increased ability and capabilities and you know the way we've done it of course is said that we've continued to use exist our existing hypervisor you know which is very pet friendly and therefore really open stack becomes a control plane that we could migrate both pet friendly apps as well as you know applications that are rewritten and developed for the cloud because given the size of my company there's thousands upon thousands of applications and they're not going to be rewritten overnight now there may be only about 10 percent of those maybe uh like the big big rock applications that really are will move the needle and the rest are kind of your small you know dot net applications or tomcat based web apps and things like that or like really small ones like those yeah those are there makes no sense to rewrite those into a cloud native format so I think with that said I mean I think they're in many companies in in enterprises there's just too many applications that if the barrier to entry is that your cloud requires you to have a only cloud native apps that's a impediment to getting adoption so really you need to have that you know that offering that's going to allow for you know really these legacy applications to onboard safely and you know to meet their needs right if they're not going to be they're not going to fail safe at the application layer I mean they're going to rely upon that underlying hardware and that infrastructure to fail so that's at least my vantage point in my view okay sorry can I just follow up on on that and I think I think we've maybe got some more questions around the topic but I I also agree I think despite what I said around um around the the sort of pets and containers I think I think realistically it is unreasonable to say you're not gonna um you're not gonna have applications that need some care and I think I think you know we maybe talk about it later but there is definitely a sort of even not in the cloud but integrating with cloud applications is something that's going to happen I think for the foreseeable future from opposite okay as nander just as a follow-up to that um you know a lot of the application workloads that enterprise uses come from ISVs and one of the challenges we've seen in the product working group is trying to get ISVs to get more active and saying yes we're going to bring our applications we're going to you know put them on an open stack whether they have to be rewritten or not you know from an SAP perspective to the extent that you can what do you think is needed to get ISVs to be more active in supporting open stack well I think I mean we sort of have two hats on that we are to some degree an ISV as well as a as well as a platform and I think um I think the it really is um unrealistic to expect that every application in it from every vendor is gonna is gonna be able to be put into a cloud and we we're all you know we see um some of our customers pushing back to say you know we don't want to run your software in the cloud we want to run it in-premise and and they've got very valid reasons for doing so and um I think the um you know there's a move to say you know the periphery applications can start to push out there the non-mission critical or the less mission critical applications can can start to go out there but I think uh I think you know the the bigger issue I think for ISVs is more the move from from a software vendor that's shipping CDs or binary downloads to service providers I think that's a really big challenge in terms of moving their applications from from something they just ship and support remotely to actually owning the service and actually keeping the uptime and and doing all that and I think anything that can make it easier for people to do that would really help and I think from our perspective that's also true so the easier it is for us to deliver a fully scalable cloud that's that's got 99.xxxx percent uptime that's going to make it less risky for us to sort of push out those more mission critical applications out there I think but the risk of starting a debate I think Prashantz just said there's certain things that you do that you're never going to do on a public cloud so you know how do you how do you talk to your ISV you know partners to say here's what here's where we're going and and what what should you know this is this is our new target how but I think that one of the solutions that we're seeing for that is is this sort of on-premise on-demand solution where you can actually link the two through services like the VPN as a service where you you're not forcing people into a particular scenario you're looking very much at hybrid application I think it's I think that's a very valid scenario for a lot of our customers so anything to know not really I mean just because I mean hybrid cloud and regulated industries is probably many years away so I don't think we're we're quite there yet I mean yeah I mean most of our application delivery for you know from an ISV point of view is still going to be that traditional method of you know either baking it into the image or as part of a post provisioning process deploying that software and are there things that the ISVs could do to make that easier in an open stack environment yeah that's interesting I guess you know with um yeah I guess we're we haven't seen it yet but you know pre-canned pre-canned images from ISVs delivered you know as images that we could then important to glance you know and just like fire up let's say a you know an app server and things like that I mean a lot of that stuff you can just build you know um you know you can build on your own our biggest challenges is around patching okay so we've got a lot of challenges around um get you know just there's different support groups in an organization like like the bank it's not just the core cloud group which maintains those assets once those assets or those servers respond up then it's a whole host of other operational support groups that kind of take control so it's in this raises kind of a side question or a topic I was going to bring up later on but I'll just bring up now is just around the organizational skill set that's required to support a cloud um and what we found and is that like I obviously have network leads and and compute leads and storage leads but overwhelmingly and like one of the lessons learned for the past three years that I've seen is that um that skill set is is hard to come by in many enterprises and so I think that's actually one of the challenges to the community is to like how do you get um how do you kind of foster more knowledge around like this hybrid new set of skills that's required I mean there's I know there's a lot of initiatives taking place but it's very very challenging to present you know a product offering like OpenStack and then expect everyone just to go off and learn it and and so that you know everyone's probably aware of it who's run running a cloud or even try to hire for it it's not a easy skill set to come by so I think and that kind of touches upon um you know just the operational aspects of okay you have a cloud and you've stood it up now how do you maintain it how do you essentially build that expertise and you know I think there are things that the community could do to help address some of that I'm not sure what all of that is aside from mentoring and learning but there's there's probably things at a product level that could be done to make things easier where you don't have to be an expert in every single one of these technologies so okay so let me let me pause for a second and just there are anybody in the audience that wants to ask a question from the panelists we have microphones here and here please Scott asked you with Dell services like the discussion thanks guys I'd like to hear more about um when I walk in and have sort of my thumb on the pulse of a lot of Fortune 100 companies it's talking about what you're discussing operationalizing um open stack but I think it's sort of the common denominator is the question is how do you abstract complexities of the technology in such that uh users can consume or use a cloud and how and be uh how does that work in enterprises it works in enterprises by matching three things right making it highly available easy to use easy to recover from um and um easy to consume I guess and I just wanted to see what your thoughts and comments on that work um yeah sure I'll go first um so yeah I mean we rely uh you know at least within the bank I mean as I mentioned we we use a hypervisor that is you know provides all those capabilities right that high availability that we're alive um um you know and and uh I guess ease of integration so that that addresses some of that one of the questions you asked was about like the ease of use and in what's interesting um as contrast to like public cloud let's say with amazon or google or so on right whereas typically app devs themselves like you know logging in and basically provisioning the infrastructure they want well in in the bank we for example the app dev groups don't do that there's actually separate team there's a whole build and release team and and essentially that group is essentially the the broker or mediator between the app dev groups and the infrastructure teams or between the cloud teams namely us so that build and release team for example they're already kind of they've got that skill set and they're the ones responsible for translating app dev requirements into uh infrastructure requirements so in the previous talk that was just in this room they were talking about the the tenant and basically the needs of the tenant and it got me thinking that you know the tenant by and large in the enterprise isn't necessarily your app dev community they're the consumers of that infrastructure they're the users of that infrastructure but they're not the provisioners of that infrastructure and especially now when when we look at um like we're just starting to introduce neutron and neutron obviously networking that's a really scary thing from a self-service point of view so really don't i mean an app dev folks by and large don't know much about networking like they really don't care about it but you have this other kind of persona which is like the build and release engineer a group they're the ones who are going to utilize and kind of make that translation so i'm not necessarily sure if you should have if the cloud how usable it needs to be from an app dev point of view it needs to be usable to certain experts along those lines right yeah i think i can echo quite a lot of a lot of those those thoughts i think one thing we do which we we do want to make a sort of self-service nice experience for for everyone who's using at least the internal part of our of our cloud we sort of split between our customer payload and our sort of internal needs and from from our developers and we we've accepted i think that we're going to have to build a essentially a horizon replacement to handle a lot of that sort of onboarding management of projects the the network scenarios there's quite a lot of of of stuff which people don't want to care about they just want a vm in a particular network zone and i think horizon doesn't do a great job doing that for an end user it's great for admins and for visualizing what's going on below the or above the the apis but the that onboarding process is something that we we yeah we we couldn't give horizon i think to our end users and we're going to develop or we are developing a sort of an alternative which includes a lot of those processes i think well so they've already covered you know pretty much the topics so i'd like to talk about very specific one so many things are automated in open stack but the deployment complexity is the one of the great blockers for enterprise so that's the reason why there are you know many vendors who developed their own deployment tools and i think it's ideal to have one standard deployment tool for open stack so that the you know enterprise users don't have to learn you know many procedures depending on the type of the deployment tools okay yeah i have a question for Andrew so for your container orchestration and and deployment are you using upstream project projects like cola or magnum or are you using your your own we're using at the moment we're using the color container build so we're we're not using the the orchestration elements of color we're using the the the color to build the the the base containers and we found i think we we've discussed a little bit with the the color ptl around the rationale behind some of the the constructs in color and i think what what we've done is we're running on kubernetes and that handles a lot of the the the things that we're missing i think in the initial or when color started and we found that you know despite what i was saying about the the pets and the and the and the issues that we see there i think we found that there's there's some really nice capabilities in the container world and particularly in kubernetes that allows us to to basically fire everything out there using liveness probes and checks that things are you know doing what they should be doing that we we can reach a sort of a state of eventual stability it takes maybe two three minutes maybe five for the whole thing to to to settle down but it's um yeah it's it's it's working quite well so the the orchestration part of color is is a little heavy weight for us i think in the in the approach we've taken at least so so so that being said um as this container ecosystem matures other than multi tenancy do you see any need in future to run it on open stack or just running bare metal containerized well we yeah we we we run kubernetes on bare metal and then we run open stack on top of it and i think you know we we're gonna we're seeing already a big demand in our organization not necessarily from from um from our customers for for containers and a container as a service type thing is gonna is gonna come i hope but i think right now where you know where we need to be able to provision virtual machines bare metal and open stack is going to do that on top of uh kubernetes so good hi this is sanjay um can you share your experience and your wish list for operationalizing security in these environments uh to be honest it's not something it's not something within my sort of um sphere of um of expertise we have a we have a large security team who who when we're when we're um when we've provisioned a platform come along and say um you know this doesn't work this doesn't work this doesn't work and i think we've found that certainly based on our our previous um sort of iterations that open stack really helps us to do that and there's there's a lot less um of that sort of requirement just adopting the base um the base open stack stuff i think there's there's a lot of detail that they they go into which unfortunately i i couldn't really comment on but i think we we're seeing a massive improvement i think in our ability to provide a a secure base platform already um over the some of the stuff that we were developing ourselves yeah i mean i guess one of the one of my top items has been just the um the r-back model with an open stack and and right now using the policy dot json file in each in each project and not having a way to manage that aside from just editing that file like like i mean i think the r-back uh issue i'm just surprised it's taken this long like that would be like my number one um wish list item okay yes absolutely we'll we'll see what we can do um but you know that's uh that's just one the other one was um uh i know recently like for example in the large company like you know the bank you know there's uh uh active directory is you know kind of that's how we manage users obviously um so for recently we were essentially using active directory for both authorization uh association and authentication and i think there's some changes there that are taking place so i think there's some improvements happening uh and i think you know there's going to be the uh a move towards i think using um mysql for um uh i think association and authorization and then uh i'll adapt for active directory for authentication so i think there are some things that are happening there that are probably beneficial um but i think it's still evolving i mean i attended some talk yesterday on Keystone um and i think there's it's evolving but i think like the policy.json was the one that's like you know i know it's been if there's a lot of work going on but that's like the one one thing that i'd love to see you know continue to evolve the other one is around key management i haven't followed barbacon too closely but i know that that's uh another project that provides a lot of promise and that would be something that you know is interesting as well for all the services that are used by open stack. So as we're as we're moving our pets to cattle do you find that the total number of osis under management within the firms is doubling tripling quadrupling now that it's easier for end users to spin up their own vms or do you find that you're pretty solid on number of total osis and it's really just a shift as time goes on from corporate it managed over to sort of self managed i don't know if it's if it's increasing but it's um i think and i can really only speak for our previous platform which was which was um more about the other open stack platforms that we're putting in place now much more about our customers the the previous platform was about enabling our developers and i think we found that we put the system out there and people would just come and it grew and it grew and it grew and we went from nothing to 20 000 plus the m's growing at i've no three or four hundred a day for within within sort of two or three years and i think uh yeah it we put things in place to to um to try and throttle that or to encourage sort of expiration we were a a nice presentation earlier about some some uh stuff coming into open stack to do that but i think if you put it that people will come i think is our experience at least in our internal um internal environment so i think we're about um one thing i wanted to do was was k if you could just talk a little bit about the product working group because one of the things that the product working group as i mentioned was trying to get to get feedback so you know okay if you want to talk a little bit more about that and sure what we do so product working group uh what started about two years ago uh coming out of the uh paris summit and product working group is to increase the voice of the markets operators and the users into a development process of open stack and so so we so basically the basic flow is we create users user story that include problem descriptions and use cases and the usage scenarios i guess and so we also use the gerrit process uh to to manage user stories and also get comments to have other people to contribute them so if we agree that a user story is complete enough we translate it into a cross project spec blueprint rfv or spec appropriately and so um so that's the first time to expose our stories to the community so we can get more attention from the community or developers and so that's pretty much the workflow of the product working you know one of the key points is anyone's welcome to contribute um on commenting on user stories at all is done in the open and if people want to submit them we'd be more than happy to take a look at that so i think with that we're out of time um so i thank everyone for joining us i especially want to thank for shanth andrew and um and k for spending some time here with us so thank you and hope you have a great rest of the summit