 Okay, right, so we're going to get started. I guess everybody can hear me okay, the microphone seems to be working. So hello, it's nice to be here in Boston. My first time here personally, it's a beautiful city. So my name is Sean Handley, I work for a company over in the UK called Data Centered. We run an open stack based public cloud. I personally have been working with OpenStack for about three years, and I mostly come from a web applications background. So my way of talking to OpenStack is all API driven. Hi everyone, my name is Tobias and I work for a company called CD Network in Sweden. We are also a public cloud based upon OpenStack. I've been working with OpenStack for about three years now, and as Sean here, I've also mostly been working with the APIs, and my main focus has been developing our cloud management system that we are using for managing our cloud. So we are here today to talk about how it is to run a public cloud using OpenStack. So as we said, both we at CD Network and Data Center are using OpenStack as the base for our public cloud business. And today we will cover what life is like behind the scenes, just doing that, running a public cloud with OpenStack. And just to have it said, this is not a sales pitch at all, yes, you know that. We just want to make OpenStack better for us as a public cloud provider. Yeah, indeed. I mean, ultimately, it's kind of a scratch your niche sort of thing. So we want to focus on how to simplify life for ourselves and for other public cloud providers, because OpenStack is a wonderful toolkit, but we found that there are some gaps in it that make our lives busier than we'd like to be. We're obviously very lazy people, you know, we don't like to be too busy if we can help it. So what we want to do is start conversations, get ideas from the community about other features that may be missing from OpenStack as it stands and how we can fix that, ultimately, as a community together. So it's a bit late in the day. I'm sure everybody's starting to feel a little bit tired, so we're going to do some stretching exercises. So by show of hands, hands up if you use OpenStack on a weekly basis or a daily basis. So almost everybody, cool. Okay, hands up if you operate an OpenStack cloud. Okay, mostly the same people. Hands up still if it's public cloud. Okay, great. This is just the kind of room that we wanted. Any upstream developers here? Okay, good. Excellent. These are the people we need. Cool. Okay, so the outline for this presentation is that we will talk to you about very important stuff for about 30 minutes and then we finish off by doing some questions. And yeah, last year, me and Sean met at Summit in Barcelona. That was the first time we met. And it became pretty clear, pretty quickly, that we were doing about the same job on a daily basis with plugging the gaps as we express it, what we feel are missing in OpenStack for us as a public cloud provider. So that could be for performance reasons. It could be missing features or just things that don't work the way that we want them to work. That would be good for us. So this is how the world looks like that we created, right? Yeah, so we have two layers here. We have the sort of, well, bluish layer here. This is OpenStack as you will recognize it. And then we have this red layer of magical proprietary code on top of it that gives us the abilities that we need to run a public cloud business out of this. And without this magical layer, we'd actually have quite an impossible task of running OpenStack as a public cloud business because it just, when we first started getting into this about three years ago, it just wasn't all there. So that's how I started working with OpenStack. I got drawn into figuring out how we can get this up and running as quickly as possible. So rather than actually get involved with contributing code directly to OpenStack, I was like, hey, I know Ruby. I'm gonna write some Rails applications and do all of this stuff. So it's worth mentioning that when our users interact with our public clouds, they're not necessarily just doing it through the magical proprietary code stuff. They do actually have access to the APIs as well, and that's all exposed and modified. But for things like user signup and some aspects of billing and quota assignment, we just had to have this magical proprietary code layer. And this is where we live. This is where we spend most of our lives. So there's a problem with this, and that's that magical proprietary code sucks. We know because we've read it. It doesn't benefit from the love and care of being out in the open source community. There could be bugs in it for a long time before even we realize it. If we get hit by a bus, that's bad news, because suddenly a large percentage of people who know this code will disappear. So it's not really an ideal situation for us to be in. So before we start talking about how we wanna change things, we wanna just do a little bit of information on how we think running a public cloud might differ from running a private cloud when it comes to OpenStack. Yeah, it goes without saying that it's a difference between running a private cloud and a public cloud, and a different challenge. So yeah, what difference make it necessary for us to create this? Do you say crappy code? Yeah, that's one word. Propetarily, at least, yeah. So do you have the... Oh yeah, go on, you take the magic device. Do you have these in the U.S., self-service gas stations? Yeah, you have, good, good, good. So in theory, everything should just work automatically, right? That it's like the same in the public cloud. Users should just be able to come along, sign up, consume some resources, hopefully a lot of resources, and pay the bill, hopefully as well. And the reason for that is because it scales, right? Yeah, it's the only way it couldn't work. Oh, exactly. So to be able to keep up with the virtual demands that comes along with this, we carefully need to add physical capacity along the way. So due to that, we care a lot about quota, usage tracking, resource management, and those kind of things. And yeah, it's probably one of the most, yeah. It's things that we could care about the most. It's something that worries us, I guess, making sure we have enough quota. Exactly. So we also get these weird and unpredictable workloads. We don't know what people are gonna be using our cloud for because it's public, and if they're paying, then as long as they're doing nothing illegal, they can do what they like. So we don't know what's gonna be running on the VMs. We don't know when there are gonna be spikes in demand for us, or how big those spikes are gonna be necessarily. We do have relationships with some customers where we have some idea of what they might be doing, but generally every day is a surprise for us. And this is my least favorite part, really. We have to stop people signing up and abusing the service. I'm setting up spam servers, doing other nasty things. And it's tricky, really, figuring out what's actually going on the cloud and what people are using it for. We can do some things in the signup process to reduce the risk. There are organizations who you can send IP addresses and emails to, and they'll give you some kind of risk score. But that doesn't always work, and it's hard just keeping tabs on what people might be doing because they might start out legit, and then suddenly start trying to sneak in some spam along the way. So we need to stop this happening for good court management, and also just, because we build people once a month. We don't want them coming in and using the system for a month, and then suddenly they're not paying us anything because that card's broken. We get quite a lot of these every week, and it's really annoying. I still don't know how to fix this, but it's something that takes up too much of, certainly, my life having to buy YouTube ads. Yeah, pretty much, yeah. Yeah. So before we get into the things that we think that we are missing in OpenStack, it's worth mentioning some good stuff about it as well. Since we use it to, yeah, it's our bread. So, yeah, OpenStack is mostly a very good platform to run a public cloud with. It's very easy, actually, and that is for several reasons. It's a very large toolkit of very cool cloud features, as you said, very large toolkit, and yeah, it makes it possible to us to run a cloud that evolves together with the users and their needs, and don't fall behind and become obsolete in the end. So, transparency. This is a great thing if you can get to the code and the code is broken, then you can fix it. Half the time, if we realize something's broken, it turns out it's already fixed. It's just in a newer release than what we happen to be running. Personally, I think Garrett is a little bit off-putting for a lot of people, but ultimately people, people start using it, they get used to how you contribute through it, and once they realize that it can be done, okay, it's not GitHub, but it works, it's pretty much fine, and if people are technically competent, they can get in and get their hands dirty and start fixing things. What's the next thing we're gonna talk about? Oh, yeah, DevOps tools. So, this is the kind of thing that makes our DevOps team happy because there are so many deployment options and good automation tools for this. You can deploy OpenStack with Puppet, Ansible, SaltStack, Docker, Packer, all kinds of other things, more things than I care to really pay attention to. There are a million different ways to deploy code these days. I think we jumped away. No, no, no, it's all good. Yeah, OpenAPIs, that makes it possible for customers to manage the resources and from any applications that they want to, even if they have a proprietary code in their system. Magical proprietary. That's the tutorial. And there's no vendor locking. You can easily choose from any cloud out there with the same deployment code that you have to where you want to put your application. So, for that, there's a lot of good tools like Shade and Clouds.yaml, those kind of stuff. It's very good for that. So, we're gonna kind of get to the reason that we wanted to have this talk now really and that we wanna figure out, well, what is missing? What do we think isn't an OpenStack but should be and how can we get it there? So, for me personally, this is the really obvious one and this is how I started getting involved with OpenStack. So, has anybody ever seen the film, The Hobbit? Yeah, a few nodding heads. So, we recognize this door, this magical door in the side of the mountain. So, it's not something people consider too much when they're running private clouds but if you're running a public cloud, there is no door in OpenStack. People can't just walk in and start getting involved because they don't have any credentials. So, if they have no credentials, they can do nothing. So, I was first hired to effectively build a door. It's set on top of Keystone. It had its own Keystone admin account where it could create users and then people would arrive on a page, put in their credit card details that would go via Stripe. We'd tick it off and say, yeah, okay, these are valid payment details and then they would be able to get into the system and do wonderful things with OpenStack. So, I don't see why we couldn't do this in Horizon and Keystone. I don't know if anybody's tried in the past. I've asked and no one's really given me much of an answer but I don't see why we couldn't have some kind of sign-up flow mechanism where you could kind of pull out to things that you wanna tick in the journey through to saying yes or no when somebody signs up online. So, maybe you wanna integrate with Stripe in some way. You wanna integrate with MaxMind for fraud prevention, whatever other checks you wanna have along the way but basically I don't see why they shouldn't be a way for people to arrive in the middle of the night and sign up on OpenStack without somebody answering the phone or responding to an email when they asked to get in. So Tobias, do you wanna talk about some, is this the quota stuff? This is the domain level stuff. So, one thing that we think would be really good for us is to have better domain level support. Policies at domain level so that we can let users do all the admin stuff at the domain level. For example, project management, create products, delete products, domain-specific IP pools. It could be maybe authenticating to all the different, all the products that you have inside your domain at once. It would probably save a lot of load for Keystone. Which will reduce a lot of the proprietary code as well that we have today. A lot of the code is deleting products and that kind of stuff. Also, quota management at the domain level. It would be much easier for us if we could just hand out some amount of, yeah, quota to one domain, one user, who can split that between the products he decides to create. Today we do a lot of, yeah, quota management behind the scenes for users. And we don't want to do that. We don't want to do other fun stuff instead. Yes, a lot of interesting problems. Yeah. We have been to a few sessions together here and we also realized that some of this is coming in, I think it is in the next upcoming release of Queens in October. Yeah, Queens is introducing hierarchical quota support in Keystone. That's going to make life much simpler for us. Yeah, it will for sure. And we also have the orphaned resources. Today, if you delete a project, all the resources that are behind that project will still be allocated in the system. So this is something that we saw with medical proprietary code today. And yeah, a couple of scripts will be needed for that. And that's nothing we want to do either. And it's also something that has been discussed in Keystone as well. So hopefully we will see some results there as well for us. This picture of Oliver is because these are orphaned resources, by the way. So when you delete a project, suddenly all of the cloud resources that belong to that project still exist. But you've got to clean it up because it's taking up capacity on your hypervisors and so on. I want to complain about Cilometer. It does look like the telemetry project is going in a better direction. And I'm assured that it's much more scalable these days. But boy, has this been a pain in the ass for us for such a long time. What are we, Nick, around the metaca level release? For Cilometer? Yeah, where are we? Kilo, okay. So we've got this backed by Mongo, which was what we were advised to do several years ago. Pretty decently sized clusters. The ones that we're running data-centered, we've got eight CPUs, 32 gig of RAM, SSD. And the ones over at CityCloud, I think are... It's a little bit bigger. I think it's 24 cores and 128 gigs of RAM. So these are sizable machines. They have some good horsepower to them. And good disks. Yeah, so any guesses how long it would take to return if you query it for one day's worth of public cloud usage data? Any guess? No, nobody. Well, that's good because we don't know either. He's a little bit of pseudo ruby code of me just getting samples from Cilometer. And this is across everything. This is instances, volumes, images, neutron stuff, everything. Just querying it for a slice of two hours worth took 25 seconds. Single instance? Single instance usage? Usage for a single instance? No, for everything. For any event that's happened and been recorded by Cilometer for two hours, 25 seconds. For four hours, we're up to 44 seconds. And for eight hours, the gateway timed out. So we have to do some horrible things in just taking chunks of data and then aggregating data off that and then we can actually turn that into something we can put into a billing system. Big problem with this is we can't give it to customers because it's just too slow. And if too many people use it, then the whole thing grinds to a halt. It's just, it's never gonna work. Hopefully we can fix this. I mean, we're paying close attention to what's happening in the telemetry projects. And this is the kind of thing that our bosses really want us to get right because if we can't build people, we don't get paid and people get sad when it happens. The customer does really likes when we bill them right as well. Yeah, they do. They don't like mistakes. So yeah, hopefully this is all getting better. I'd like to see this exportable in some kind of common formats that you could feed into invoicing and accountancy programs. I'm not sure if there are any standards, particularly for that, because I haven't started looking into it in anger, but it would be nice to just feed that into, say, your financial force or whatever it is that the business has decided would be good for billing people with. To buy it? Yeah. As I said earlier, customers would like to consume any cloud anywhere. And they would, yeah, it should be easy to switch between clouds. And today what we see is that different clouds needs different parameters put them into the authentication part. So when you're authenticating to Keystone, some clouds, you will have to put in your product ID, your user ID, your domain ID, and for in some other clouds, also running OpenStack, they don't accept those kind of things. You need to specify project name instead, and or tenant ID and product ID to be able to authenticate correctly. Maybe all of these clouds are doing it wrong, I don't know, but it creates a hassle for the customers when they are trying to consume resources in different clouds. So what we would like to have said with that is that some kind of more generic structure for authenticating to Keystone would be awesome. Actually, it makes the world easier for our customers, at least. And yeah, security. With this slide, I don't want to say that OpenStack isn't secure, that's not about it. The security team is doing a great job with the vulnerability management team. And yeah, it's good. And so, but we are missing a couple of features here. And one of those are two-factor authentication, together with Keystone, that would be great to have, and also a decent VPN solution. Yeah, because of the one we have right now, VPN as a service is left to die. And interesting session tomorrow about that. And the one we have, the existing one has also been a hassle for customers to work with. And a lot of customers really like to have a VPN solution onboarding into the cloud, and also when they are using multiple clouds and communicating between them. A good audit trail would also be nice to have, logging whatever happens from API point of view with your, inside your tenant would be great. And also better separation of inside the logs so that you can easily separate out all the logs for one tenant. And yeah, maybe even expose those logs to the customer, if it's not, if it doesn't contain too much sensitive data. Okay. So how do we close the gap? How do we fix these problems? So the answer is fairly straightforward actually. In the last three, four months, about four months ago, the public cloud working group was established. We're still figuring out what our agendas are and what work we wanna get done and so on. But the formation of this is just bringing awareness to the community of things that OpenStack needs to have to better support public cloud providers. So... Yeah, we're looking for more involvement. We are, yeah, this is kind of a call to arms really for anybody else who's in a similar situation to us and doesn't wanna keep duplicating workload and living in the magical proprietary code layer forever more. We want you guys to get involved, tell us what you think is missing if we've missed something that's important to you. And hopefully we can have some discussions about people actually developing solutions to solving these problems really. Sorry, I think I just talked through your slide. Oh, it's okay. You did it well. It may not be obvious, because we're that good, obviously. We didn't practice this at all. Tobias lives in Sweden and I don't. So sorry if we're a little bit rough around the edges today. So yeah, we wanna take our proprietary code and start porting those features into existing OpenStack projects. I'm not suggesting we form a new project because I think there are way too many OpenStack projects anyway, but bringing things into the core projects like Keystone and Horizon and Nova and Neutron and stuff like this. So there is a little bit of a problem that we found so far in that most companies offering OpenStack public cloud are small to medium enterprises. They might not be able to sort of get the money aside for paying for a full-time upstream developer to build features that suit public cloud providers. So I don't know, there might be a few ways around this. Maybe people could give 20% time to one developer and they could spend one week a month doing this sort of thing, or maybe one day a week, depending on how they prefer to work. Upstream Friday. Upstream Friday, absolutely. Pizza and beer. That's the way I get most things working when it comes to changing things. So yeah, if you work in a company that's interested in doing this and talk to the right people and hopefully many hands can make light work of these changes. Cool, so we're 25 minutes in. We're a little bit faster than we planned, but we're up to question time. So if anybody has questions, ideally come to a microphone, but if you're too far away, just shout it out and I'll repeat it for the video camera. Yes. So for the first problem you just mentioned, I think it's about send up, right? So you mentioned there is no way for OpenStack as a public cloud for customer. So we have an internal product named StackTask so that you can, external customer can send up and after you send up, you have the project manager role and you can invite more project members into your project. And we are going to open source it in the next couple of weeks, I think. And we are using it in our production environment. Ah, sorry. We are a public cloud based in New Zealand. Yeah, we're using OpenStack. So, and as for the COTA policy stuff, I think it's happening, but it's very slow. So don't expect it to happen very soon. For example, in GANs, for now, I'm still a cloud member of GANs. So for GANs, GANs is still using hard-code COTA, not like NOAA, Cinder, and Neutron. So you can expect it's still very slow to get a global COTA policy. And as for VPN, as a service, for now we have one developer working on that. So it shouldn't be dying. Okay, good to hear. What was the name of the project you're open sourcing? For now, internal we call it StackTask, but it's going to be released with another name. We will send an email in the OpenStack dial, mail list, and it could be operator's mail list as well. Excellent, cool. And as for Celerometer, we are running into the same problem. It's very, very slow. But as for billing, if you want to just use the metrics from Celerometer for billing, I assume you will have another billing system to get the metric samples from Celerometer. For example, per hour or for two hours. So it should be fine, but if you want to use Celerometer for alarm, another option is just deploy another Celerometer, but only set a worry short TTL if you are using Mongo. Just set it as a worry TTL so that after you got the alarm, the metrics will be deleted by Mongo automatically in a very short time. And as for billing, we also have a project named distil. It has been open sourced yet, and it can help you worry easy to map to your backend ERP. I think if you are using OpenStack as a public cloud, I assume you have an ERP system to manage your customer and billing invoice, all the stuff. So for our project is named distil. In distil, you can create your ERP as a backend so that you can map in, you can export your invoice quotations and even credits to your customer because I assume you have already got some TKs asked. I want to see all the invoice history by API, and I want to see my current balance of my credits. How much am I paying this week? That sounds really useful. I think there's a fishbowl session right after this and it's downstairs, I think in room 103. 102, I think it is. If you can come down and talk to us and tell everybody about that and we can write that stuff down in an etherpad that sounds really useful. Yeah, yeah. More questions? One more. You guys were asking about the public cloud working group and what to do next. By the way, disclaimer, I work on the same company of that guy so I appreciate that we're taking too much time at the microphone here. The only thing I would like to say is today I went out for lunch with the guys that run the internet cloud and also VH and just realized how much overlap and how much time we're spending exactly on the same problems. Exactly. So a good start would be, let's just get the people that are running this public cloud together. I can write a little list and say, hey, if we are all fixing the multi-tenancy problem on Trove or the workflow for signing up, creating something that allows people to define a workflow there, maybe we can share this precious development resource that we have and say, okay, we're focusing the two people on this one, the rest of the people on this one and spread the load a little bit. Absolutely. I think that overlap is not very productive. So that would be a good start. The work group doesn't need four months to define what it will do. We have a little list of things that need action right now and a few developers that each one of us can contribute. So maybe that's a good place to start. Definitely. Cool, thank you for that. Thank you. More questions? Yeah, go for it. We're good for another 10 minutes, according to my watch. Okay. So I'm just kind of inspired by watching the VIO presentation, VMware Integrated OpenStack and wondering why, why they would do it. And the reason they seem to be doing it is just for simplicity. What they do is they go in there with a sort of take some of the chaos of OpenStack, if I may use that term, and they frame it off and suddenly make it a lot more approachable. And then I couldn't help but think of Linux itself and how the more technically difficult the problem was, the more Linux seemed to attend to it during its developmental years in the 90s and the knots. And the more user friendly side of it, the gooey side of it always seemed to stumble. And I wondered, I don't think they're any more technically difficult. There's probably something about the community, the economy of the system that sort of leads to this strange development where the really tough parts in the core, you know, the symmetric multiprocessing and so on are dealt with exceptionally well. And then outside on the edge, you're left with a gooey that, you know, half the time you can't comprehend. And I wonder if we're not seeing the same, you know, when you're looking at laying out a bit of a future for a public OpenStack, if we're not seeing the same kind of pattern development, if we are, what do you think is behind it? So you're, just to try and understand that in its depth, you're saying that OpenStack behind the scenes is very complicated, but the interface that we put in front of it is not good enough, so. Well, yeah, and I know I could get nailed to that, probably. But yeah, sure. It's fine, I mean, a lot of people don't like Horizon, for instance, if that's what you're referring to, but. For example, we created our own control panel, totally, because Horizon wasn't suitable for our customers, too complex for them. And yeah, we had to create our own. Yeah, but just the install and the update, I mean, if somebody is a relative newcomer to it all, and sort of over the last year is, you know, tried to expose and understand and so on, what I found was it just, it reminded me really of circa 1995 trying to install Linux when we had Linux Clubs. And it was, and I always marveled at how Linux was so brilliant in one way. I mean, Microsoft was on the run and the majority of web servers today, you know, Microsoft lost that market because the system just works better. And this technically is more difficult in many ways than just facilitating a front end that would allow more people to get into it more quickly and maybe move it forward more effectively. It'd be good to discuss what we think that would look like, how we could improve on what we already have. I mean, personally, I don't think Horizon's too bad, but I guess that comes with a lot of time and domain knowledge and I'd probably take it a little bit for granted. I'm sure it puts a lot of people off interacting with it. But yeah, it'd be good to define what a good user interface would look like. Absolutely. But still kind of tricky because OpenStack does a lot of stuff behind the scenes and you have to try and plug it all together. So, no, it'd be good to discuss it some more anyway. We'll talk about that in the fishbowl session. Any more questions, please? Good afternoon, I'm Antonio. I work in America Mobile. And we're in the process of building a public cloud. We have built others based on being where an open virtualization and so on. But what we face with those products is that we have to wait a lot to get new features and every time it's very expensive, right? And now, going into extending the cloud with the features missing with OpenStack, which was an option, CloudStack, still they're an option as well. We are entangled in the discussion of which distribution should we use? Because at least in Latin America, there are not so many developers participating in the community and the skills are expensive and are hard to find with. So, first question would be, so that was a bad one, sorry. So, is there any recommendation for a public cloud OpenStack distribution? And the second would be, do you have your own developer teams? Do you maintain it by your own or do you depending on one after the party? Do you not go first? We use the one that comes upstream. We have early use REO as well. I guess it depends on what you are looking for. If you're looking for getting features quickly, for example, you should probably look for using Vanilla. Basically, that's the biggest difference, I think. Time to market of new features. That's my experience of it, at least. Yeah, we run Ubuntu and we just get, we use the puppet, not libraries, puppet modules for deploying OpenStack. So we just take whatever release we wanna deploy and when we deploy it that way and then we don't get anything sort of vendor specific beyond that. And if it breaks, we have a team of, how many people Nick now? Eight, seven, we have seven DevOps and developer technical type people who can get their hands dirty when it comes to OpenStack. And yeah, some days we have to just really dive into the code and scratch our heads and we have been in positions in the past where it's hurt us because it's been broken. But we've always been able to fix it. I mean, that's the great thing about being able to get to the source code. And so we'll put a fix in place and then we'll try and get that upstream. So that's the way we prefer working because we love details and we love being able to control everything. And that's worked pretty well for us so far. More questions, two more minutes, right? Two more minutes, no? Okay, so we've got the link here for the public cloud working group. If anyone wants to get involved, we are on Free Node IRC and the OpenStack public cloud channel. These are our nicknames if you wanna send us messages or anything. Yeah, as we said, we're having a session shortly in one or two. Yep, downstairs. And on Thursday, there is another public cloud working group session. Can't remember where or when that is, but it is on the summit schedule. I think it's around lunch somewhere, cool. So if you wanna have a conversation with us, do come and talk to us afterwards and thank you very much for coming along and listening. Thank you.