 Hello. Can you guys hear me? Go. Thank you, everyone, for making it all the way back to the far reaches of the conference center for this talk, especially right after lunch. I also, those of us who are going to be presenting today about Confronting Complexity, Mark Baker from Ubuntu. I'm Kenny Johnson from Rackspace, and here are Nori Sheena from Fujitsu. If you've read the abstract, what we wanted to go through today is both a recognition of where is OpenStack complex? Let's try to put our finger on the problem and say, here's where we think operators experience complexity with OpenStack, and then also talk through some of the initiatives that the community and others are undertaking to work through or improve on that complexity. So hopefully, we end on a kind of rosy note, not just whining about how complex everything is. So before we start on all that, we just wanted to say upfront, there's a bunch of complexity in OpenStack, especially we kind of think of from two different personas, the first being end users who are using the APIs who are writing apps on top of the infrastructure as a service platform, and then operators, so people within organizations who are deploying and operating OpenStack. We really tried to focus on the operator persona for this presentation, so that's why you might not see some of the end user complexity, whether that be API inconsistency or lack of tested SDKs for certain languages, et cetera. So that will be what we're just saying from the upfront. We're really trying to focus on operators. So how is OpenStack complex? What are some of the things people say about the complexity in OpenStack? You hear things like, it just doesn't work. It's really hard to figure out how to deploy it. What's the architecture I should use? There's a lot of complexity in it. It's all got to be tackled by somebody, and typically, that somebody is me, the operator. All of this complexity really hinders the benefits I get from it, so this is supposed to be an innovation platform for me, but that complexity really hurts my ability to extract some value out of that innovation platform. For those of us who've worked on OpenStack for a long time, those are hard things to hear. It certainly hurts when you think that all these great new features that you're adding into the service really amounts to far less because of the complexity just to stand up the service at the start. That's really not all. There's tons more comments around the same. There's a high-barried entry, and it's a tough learning curve, meaning if I try to deploy a proof of concept of OpenStack, it's going to take my team who's unfamiliar with OpenStack months and months just to become familiar enough to deploy something that might be useful. It's hard to operate. We hear a lot of people saying, I would love to deploy OpenStack, but you know what, I like my weekends. If I stand that thing up, I'm going to have to be managing this thing all day, all night, 24-7. Who's going to do that? I don't have a lot of confidence in day two operations. It's a sprawling mess. There's too many projects. I don't know how to get involved in each specific project. Each project has its own community. There are too many core components, so I have to bundle together all of these components. I hear people telling me that ServiceX is really great for this feature set that I really want, but I have no trust in whether or not that's reliable. So these are a bunch of verbatim comments that I've just showed you. Also, lastly, there's this sense that OpenStack is not a polished product like you might get from, I think this quote actually ended from third-party other alternative packaged product. But OpenStack is at its heart a core set of projects that you can make with what you want, but there's no kind of spit shine on it that you might expect from a commercial product. So where did all those come from? They actually came from our user survey report, which anyone can go and download and view. I encourage everyone to take a look. We go through a NPS analysis of where people think they're, why they rate OpenStack poorly. And if you look at the verbatim analysis, there's a lot of highlights about complexity. The interesting thing that I just want to highlight for all of those in the room, think about who took this survey. So these are people who probably successfully navigated the waters of complexity with OpenStack, or did so as operators in enough fashion that they actually got involved with the community. So they're spending their free time, probably their weekends, that they're also busy operating, involved in the community. They're not only involved in the community, they are involved enough to even know that there was a user survey, and then even further, they're involved enough to know and take the user survey. So these are a special breed of people who've taken the user survey, and even they have these kinds of comments. So think about all the people who didn't make it through that journey. Think about the people who were evaluating OpenStack, maybe from afar, and heard that it was too complex, and so didn't take that jump. The people who made it slightly down the POC, but decided they wanted to go to their kids' basketball or soccer games instead of dealing with OpenStack and gave up on it. Just think about what a small slice we got from this language and all the other people who were missing. That's why we really think of this complexity as the largest barrier to adoption for OpenStack. It's something that we all as a community need to go about tackling, and it's really critical. And so those were just some verbatim comments. Mark is going to walk you through a couple of the very specific points of complexity within OpenStack. Thank you, Kenny. So this was kind of a slightly sarcastic. Let's double click or drill down into some of what that complexity is. This was certainly a meme when we were putting the presentation together. So before I do that, who's already running OpenStack today? Who's not running OpenStack? Is the reason you're not running it because you found it too complex, or you just haven't gotten there yet? You just haven't got there yet, right? You'll probably have it installed by Friday, right? So who's heard of the Homer? Yeah, it's always good to get a Simpsons reference in the presentation, I think. The first real aspect of complexity with OpenStack is that of choice, right? There are very many choices to be made, choices of project, choices of different things. And in just the same way that Homer decided that he wanted to build his own car, and he checked all of the boxes. I want all of these different things. I want every single extra possible, right? Because he had that freedom of choice. He ended up with something that, I don't know, I think is kind of cool, but it's not something that you would want to drive every day, right? You're probably going to be much better off with the Toyota there on your left in terms of it's going to be much more reliable, much more serviceable, a better everyday drive than the Homer. And that's kind of the problem, I think, with OpenStack. There were a myriad of different choices right from the very beginning. So how do you get going? Well, first up is projects. Anyone know how many projects there are? 54, that's a great guess. That's somebody that's been on the Project Navigator, right? Did you write the Project Navigator? So absolutely spot on. 54 different projects. So this is just a selection of them. And you know how the foundation breaks it down into what's considered core, the six in the middle, and then what's the big 10, right? The projects that sit outside that. And so if there's essentially 48 projects that sit in the big 10, how do you, as an end user, choose which ones you want, right? If you're interested in container, there's at least three containers, at least three different projects, you may want to try, install, configure, integrate, right? So there's a lot of choice to begin with. And choice can be a confusing thing, right? It's a complex thing. But there are other complexities. So, and that starts right from the very beginning, deployment, right? How do you deploy your operating system? Bare metal provisioning. People use cobbler, pixie scripts, foreman. Lots of different ways of doing that. And each of those has their own communities and advocates around that will say, this is the best way of doing it. Likewise, OpenStack, right? There are many different ways to install OpenStack. You know, using Puppet, Ansible, Chef, Salt, tool of your choice, right? Very many, and you go and talk to any of those communities and they will say that theirs is a great way of doing it, right? That choice, again, is confusing. Likewise, network configuration, you know, every different OpenStack cloud has a different network set up. Storage, lots of choices there. Monitoring. People use Nagios, people use Abix, or the tool of their choice. But likewise, there are different projects emerged that sit in the Big Ten around monitoring. How do you navigate that to choose the one you want? And then like backups. Now, starts, tools, projects, again, different ways to do it. Is this patient familiar? Go on to the OpenStack website. This is the click on getting started, right? And there's at least six, seven different choices there. Install DevStack, go run it on a public cloud, go to the marketplace and try any of the sort of 13 or 14 different vendors that sit on there that will offer a distribution, each of which is the leading OpenStack distribution, right? So making your choice is hard. If you compare that, right, has anyone tried this? Can you read what it is? Azure Stack, right? If you go on to getting started on Azure Stack, it has like three buttons, right? Enter your name, click download, install it on Windows Server, right? I'm not saying that Azure Stack's a great product. I've actually never tried it because I don't have Windows. But it's saying, for an end user, which is gonna be the easiest path for me to take, right? And then there's management, right? So we look at operational management. Has anyone upgraded a cloud? Nope, people that are doing that are still actually doing it and they couldn't come to the session. But likewise, monitoring, troubleshooting, backups, patching, the operational stuff, right? It's hard to do. And then lots of different opinions on how you should do it, lots of different tools you can do. So again, all of this choice is confusing. So how do we address it? Well, at this point, I'm gonna hand over to Peeranori who's gonna look at some of the ways we think you can address this complexity. So how do we address complexity? So first, we'd like to talk about community activities. These are community activities for solving the complexities of OpenStack, better docs, community initiatives, and user feedback. So first, I'd like to introduce documentation. OpenStack community provides such documentation and community continues enhancing this documentation to provide good navigation to operators. So there are over 1000 commits in OpenStack manual project in Mitaka. This is the third largest number among all projects. The effort to provide better documentation will be continued in Newton cycle. Secondly, I'm going to talk about community initiatives. I'd like to explain trouble shooting case as an example. Trouble shooting is one complex work in OpenStack. So in this case, it is hard to analyze log files between services to solve this problem. Community has developed a request ID tracing feature. In this feature, when a user calls an API, our request ID is generated. This ID is recorded in log files of services which processes the request. Finally, the request ID is returned to the user in an API that is generated. In an API response, if a problem happens, the user can notify this ID to an operator. Then the operator can trace how the request was processed in log files in all service, all related services by using this ID. Another issue of trouble shooting is regarding error message of an API. Recently, API working group in the community published a guideline for errors for developers. Recently, the developers of OpenStack community has started to follow the guideline to provide error messages in consistency among projects. The detailed error message would make trouble shooting easier. Thanks, Hironori. And so Hironori just gave an example of one of the community initiatives that are ongoing. And so I'm just gonna go back to the previous slide. There are tons of other initiatives. Think about when we talked about the complexities around upgrades. There's a whole rolling upgrades tag that many of the projects are working towards completing, which makes upgrades less disruptive. There's initiatives to build better fleet management and inventory management to help you manage your day two operations of running an OpenStack cloud. There's the deployment projects that we talked about earlier. From the docs perspective, they're working on revamping the install guide. So it's more inclusive of other projects. We mentioned the project navigator earlier. So there are lots of places in which the community is reacting to this known complexity and working, kind of tackling it, all kind of swarming to it and tackling it so that we improve this experience. The heartening thing that I wanna leave those of you who are thinking about OpenStack with is that we have a very robust community and a bunch of ways of getting this feedback to the right places. And so those come in the form of user groups that we just mentioned, but there's also some other mechanisms. For instance, the API working group, which is meant to tackle some of the complexity we find in API error codes and other portions of the API. We've got kind of user focused or operator focused working groups around enterprise, telco. There's a scientific working group that just started the real function of these and kind of the muscle that we're building as part of a community is the ability to bring like-minded people who might share the same experience, have them talk about best practices, have them give direct feedback that is a little bit more genericized and consumable by the development community to then say, okay, we recognize this is a problem. We kind of understand your specific use case and we're gonna go tackle it. There's other working groups like the product work group, on which we're all three a member of, which kind of is a place to consolidate those user stories to say, okay, we've heard from the enterprise work group that rolling upgrades is a problem or upgrades are a problem. And so we might write user stories to help define that so that the development team can then go implement them. And the development teams themselves are starting to build the muscle to tackle cross-project initiatives. It's been going on for a number of cycles, but so there is a lot of work within the community to enable it to kind of self-heal around these complexity problems. And I think it's important to realize that yes, it's complex today, but as a community, we've really rallied around this problem and are working to fix it in future iterations. Another work group I'll just quickly highlight is the user experience work group, which is a newly formed group. I think they're still in their first year, but they're really tackling the, let's think about OpenStack from a persona perspective. So let's take that operator persona, think about the tasks that they need to do and really build a set of experiences across projects that improve that experience of an operator. I'm a really great resource for the development teams and the product work group and others, kind of just rounding out our community of user feedback options so that we can make sure that we're honing in on the problems that are most important to our users. So with that, we talked about the community portion, there's a whole another portion around curation, which is how do you take the, what is out there in the community, the unlimited amount of choice, and even if the community provides great data and get to kind of more specified prescriptive deployments and tools for OpenStack, Mark's gonna cover that for us real quick. Thank you. So curation is essentially leaving that choice making, not leaving it up to other people, but getting other people to make some of those choices for you, right? So taking some input, and most often you think about museum curators that are choosing which artifacts and things to display, and this is very similar. We've kind of debated a little as to what to call the section, but it's not on curation because it kind of sounded smart and it was kind of quite a good way of describing it. So this is people that are making choices on your behalf. And we divided again this into two sections. So on the left hand side there, community curation, easy for me to say. Community curation. So there are a number of projects and working groups or initiatives if you like around curating or making selections about how you go about deploying OpenStack and managing it. So some of those OpenStack. Ansible is being a good example. Puppet, Chef, the COLA, triple O, and Fuel is being installed as managers for OpenStack that don't necessarily let you install all 48 non-core projects, right? That do have an opinion over the projects that they will support, do have an opinion about the way that they should be deployed and about the configuration, right? Now, the good thing is that you can go in and you can disagree with some of those and make some changes, right? So it can be quite flexible, but it gives you a very good starting point. People have installed OpenStack. Did you use any of those methods on the left? Okay, clearly need to do. Oh, thank you, sir, at the back, good. So, somebody did, yes, exactly. But then, if you look on the other side, it's kind of corporate curation, right? And so these are distributions by and large, right? So companies or communities that are making building and OpenStack distribution with a set of projects, integrating those projects together and often wrapping them around with some tools which may or may not be part of OpenStack, right? And that provides for a lot of people. And I think this is the way that many people increasingly are installing and managing their OpenStack environments. But whether it's Adio, Ubuntu, Rackspace, Helion, Moran, Sousa, and very many more, right? There are many more that sit in that space. These are curated implementations of OpenStack. So, and to say that, I think that's generally the way that the market is going. So, again, taken from the last survey that came out, data came out a couple of weeks ago, unmodified packages from the operating system and unmodified packages from a distribution essentially is by far the most popular way of installing OpenStack, right? Compare and contrast that. If you only go back a couple of years ago, so many people were building packages themselves or modifying packages, right? But there's an old adage of your, obviously I've worked for a Linux distribution and have done for a long time, it's just because you can doesn't mean you should, right? So, freedom in OpenStack is something that we love, it's something that we celebrate, but it's something that you need to be really careful about. Just because you can modify, just because you can change things, doesn't mean that you should change them. And actually, there are very good reasons that these distributions, whether it's Ubuntu, Rackspace or whatever, have made the choices that they have done, right? So that curation, something that you place them trust in, is something you have to kind of respect to. Of course, they don't always get it right all the time. And that kind of comes with the warning, right? So, it's prescriptive. Curation can be prescriptive. It means that it does have an opinion on how things like the networking setup, the architecture, whether it could be a converged architecture, for example, or separate architecture with storage and data. Storage and compute rather. What kind of storage is chosen, right? Do you use theft? Do you use sender? Do you use Swift, et cetera? Different versions, different versions of projects that are taken, and then the hardware requirements, right? Some distributions you can run on a single laptop. Others have a minimum of whatever it is, six machines. So, when you're looking at curation of, or curated OpenStack implementations, whilst that may be a nice path to be able to go up, on ramp onto OpenStack pretty easily, you do need to be, ensure that the choices that have been made on your behalf, are ones that are gonna deliver the value that you need for your business, right? So, the final piece of curation is in terms of consumption. Does anyone use a fully-managed service here? No. Again, an opportunity for someone there. But, you know, one of the reasons that public cloud is so popular, is that it's very, very easy to go and consume that service, right? Credit card sign up, offer go, access to my VMs very, very quickly, right? It's consumed as a service. And this is, I think, also becoming more popular in OpenStack, so a number of different companies, Bluebox and others, that will offer a fully-managed service for your environment in Rackspace, I'm sure, do it too. So, and that is just consumed as a utility, right? You don't know what choices have been made around the storage or necessarily around the network or what hardware is being used. And actually, you don't care. In the same way, you don't really care what hardware is underneath Amazon, right? Well, how that's configured, because you're just consuming as a service. And the thing with virtual private cloud, too, right? So, for many people, the fastest on-ramp to OpenStack and addressing that complexity is to just let somebody else deal with it. So, what do you think? This is opportunity for you and the audience. What do you think? Does anyone have any ideas of how we, either through curation or through the community, can help address some of the complexity? Are we on the right path? So, that's kind of like a distribution comparison, do you think? Right. And so, just to repeat the question, it was that, having done some research to compare different distributions and see what the differences between them are, you felt like you were kind of trailblazing there, right? So, has anyone done comparisons of distributions here? Okay. So, there are, I certainly know of at least one sort of analyst type comparison that says X is good for NFV and Y is good for whatever it is, and stuff. But then, no, you're right. There's not a lot of that. And the marketplace today, the OpenStack marketplace is very generic, right? It lets the vendors that publish the distribution write what they want about the distribution, which is why all of the distributions are number one and the best, right? So, and it, but it's difficult to do that. And I think that's a very good thing to be, it's not something that we as vendors, I work for a vendor, so it'd be very hard for me to be able to do that objectively. But I think there's definitely an opportunity for whether it's analysts or community to do some kind of comparison to help people to do that. Yeah, so I'll try and summarize, and please correct me if I don't get this right, is that to build an OpenStack environment, even just to test it out and check it out, can take as many resources as to build a big one. So that could be off-putting, or why not just build a big one, right? Yeah, yeah, a different opinion, right? So, yes, it's an obstacle that needs to be overcome. Yeah, so I mean, I agree, and that's often, who runs DevStack, right? DevStack is kind of running on a laptop pretty easy, it's a developer thing, but it's not a cloud, it's a development environment, allows you to be able to develop. But an awful lot of people will try that because I can run it on my laptop and I don't need to go and negotiate with the networking and storage teams to be able to do it. And of course they look at it and think, this is not what I want to do, right? So, yes, there are distributions that you can run on a single machine, but again, that can be complex to set up. So yes, I think it's a very valid point, and maybe that's something that we want to try and do. I remember back in, I can't remember, a few years ago, it was early on in OpenStack Live, you could put an OpenStack distribution on a USB and run it from that, right? Maybe that's an approach we want to look at again. Good comment. Yes, sir. Sure, so again, I'll try and summarize, is that perhaps we need better documentation about reference architecture for different types of implementation. Do you think that's by vertical or by industry segment or use case for all of those things? All of those things. Kenny, I know that there has been work on reference architectures that's happened in the working groups perhaps. So there is a document on docs.openstock.org that's the architecture reference guide, which will show you some architectures. But I think another point you're bringing up is of all the different deployment tooling projects that we have out there, they all have a different prescription for reference architecture, and that isn't displayed in a way that you can see on a table, like which one of these should I choose? There's no comparison across those deployment projects. So from one perspective, if you are approaching it as a, just kind of native, how should I architect my OpenStack deployment? I think there is some good documentation and we can certainly work as a community to improve it. But I would point you to the architecture guide for that, but I do think I have an idea of improving the choice of deployment tooling that you might use, or whether you use Fuel, or TripleO, or OpenStack Ansible, or Kola. Their architectures are not as opaque as they probably should be in kind of one place when you make a choice. Yeah, totally agree. I think we might be out of time. Was that? We do have time, but is there any other questions? Other questions? Yeah, so I don't know the exact URL, but it essentially shows all the different projects in the BigTent and gives different metrics on them. So one of the things that we brought about with the BigTent is something called Project Metadata Tagging. So essentially there are tags that different projects apply for, whether that might be that it has some security vulnerability testing, that it's considered mature, that it's considered being worked on by a large number or a variety of different companies, and there are other sort of kind of capability-specific tags, but those tags are aggregated and presented in a kind of visual way via the Project Navigator. I think it also has, I don't know if it's a voted on degree of maturity, so it's kind of how long the project's been around, but it's a way for you to, at a glance, get an idea of, hey, this is a car messaging service. How reliable should I consider it? Should I invest in putting this as a part of my deployment? Again, I don't have the exact URL, but if you go to OpenStack.org into software, I know it's somewhere in that section. Oh yeah, sorry, the question was, how do the project's teams prioritize the work that they're gonna work on in terms of adding new features? OpenStack is a duocracy, right? We're a set of developers who do what they want to do, and we have a governance board for ensuring that when governance model for when code is added that it doesn't break things, and that it's generally accepted by the rest of the community, but someone just saying, hey, this is a really big problem. I think we should fix it. It does not make anything get fixed until somebody actually does it, and all of these, every single developer, the vast majority of developers who work on OpenStack are working for some corporate entity, so there's some mechanisms as a product work group. Again, the reason why Mark and I and Hinori are standing on stage together is because we met through this product work group and we try to prioritize to say, we're hearing from our customers and Mark's hearing from his customers, this thing is a problem, we should try to tackle it universally, and the developers obviously at the individual project level do the same, but there is no stack-ranked list of features. Individual projects in the design summit sessions that they're having here across the building, across the way in the Hilton, are going through that process this week saying, what are the things we should work on? Who's gonna do what is essentially the statement, not what are the things we should work on? It's who's willing to put in the time to develop on a specific feature. So in that sense, it's not a typical software development model. If there are things that you wanna see in OpenStack, then we encourage you to come and participate in the product group, and you can write a user story and write a spec, and it doesn't mean it'll get done. The only way to ensure it gets done is to do it yourself, but you can put it forward, and at least see if it's a common requirement. I think we now are out of time. So if there's any other questions, please grab us in the halls. Otherwise, thank you very much for listening.