 source project track, lunch was awesome, so I'm feeling good. I'm really excited to introduce this collaboration of HP and Rackspace that, I mean, this project's been going on as long as OpenStack, I feel like, and iterating and becoming a really first-class solution for database as a service, and it's an essential component of a cloud solution, so I'll let you guys continue the introduction, but thank you very much. All right, so I am Vipo Zabaya, I work for HP Cloud. I'm the technical lead for the Red Dwarf project at HP. And I'm Tim Simpson, I'm a developer at Rackspace, I'm working on cloud database. So first off, thanks everyone for coming out here. It's heartening to see so many people here, you know, to learn more about Red Dwarf. We've been working on this project for a long time. It means a lot to us. Getting it into the Open means a lot to us. And so we're glad that, you know, if you already know a little bit about Red Dwarf, we're going to talk about the history today, and we're going to talk about the current status of where it's at, and where our efforts are at to get it into the public. We're going to go over the architecture a bit, and we're also going to talk about how we're leveraging Nova and the rest of OpenStack to build this service. So if you're building some sort of orchestration layer, it might be interesting to you as well. All right, so back in 2011, Rackspace was starting on cloud database as a service. And the idea at the time was that we were going to be using OpenVZ and creating MySQL applications inside OpenVZ containers. You know, when we started it, we kind of thought, oh, this is a different sort of than cloud servers, we're not just going to put a database on it. This is sort of something fundamentally different. And we started building an application in Java that used frameworks like Service Mix and ZooKeeper. And one day, the lead of Red Dwarf, Mike Bassknight, was talking to some people who worked on Nova. And he realized that they had a method for sending RPC calls, and we needed a method for RPC calls. They were building a scheduler, and we were about to have to build a scheduler. And so he looked at this and he said, well, geez, it makes no sense to be reinventing the wheel when Nova's doing all this stuff that is very useful for our project as well. So that day, that's sort of when Red Dwarf was born. We decided we were going to fork Nova, and we were going to start building databases as a service on top of it. And I got to work that day and we just passed this big milestone. And so I hear, okay, everything's changed. It's all Python now. We're going to OpenStack. And so we sort of have a joke on the team, like whenever we get back from vacation, and someone says, you won't believe what happened. We'll say, what, did we rewrite everything in Perl? But that said, it's been really fun. It's been really fun learning Python and learning sort of how all the OpenStack operates. And it was ultimately for the best. So we went to the Diablo Summit and we announced what we were doing. And that's when we met HP. Okay. So HP, we had deployed OpenStack, essentially, and we're looking to build services on top of OpenStack at that point. So we wanted to build value-added services, things like database services, things like messaging services. And we wanted to make sure that it was using OpenStack. It was built on top of OpenStack and it provided other services that are core to OpenStack. One of the things that we're looking for is that we wanted something that aligned with OpenStack's visions as well as goals. And also performance. So databases are different from other types of applications. They have specific performance requirements. And one of those things is consistency. Consistency in IO, consistency in read throughput, write throughput. So Red Dwarf was one of those solutions that sort of gave us the flexibility to get to that consistency. And the final point is open source. So we wanted to build something that we could push back to the community, that we could have community participation in. And, you know, Red Dwarf hit the bill. Okay. So we announced Red Dwarf at the Diablo Summit. We joined forces with HP at the next summit. And HP runs like a black box. The way we roll out of Rackspace, we have control over Nova. But HP is a bit different. And when they were talking to us, they said, hey, we like these ideas, but we really think that it shouldn't be a fork. We think it should be something separate. It should talk to Nova like a black box. And the more we talk to them and we were looking over our own architecture, we started to realize that makes a lot of sense. This shouldn't be a fork anymore. And so we started work on Red Dwarf Lite to rewrite those parts of Red Dwarf so they can talk to OpenStack completely independently. And we did a small presentation at the last summit if anyone saw that about using Red Dwarf to sort of talk to Nova. And so since that time, Rackspace is actually, our cloud database service is in GA. And HP is on the cusp of public data. So both of these are actually being used by customers. So now let's see where we are today. So to back up a little bit, what exactly is like Red Dwarf? Red Dwarf is the treatment of the MySQL application like a first class citizen in OpenStack. It takes sort of MySQL and makes it the same way the first class citizen like with the cloud server or anything else. So there's a rich set of APIs for manipulating MySQL applications so you can create, delete them. And when you do this, you get access to it. You can give access to a user who can log in like they own that application themselves. And then there's some extensions. At Rackspace, we have extensions for creating and deleting databases and users. And in HP, they have some extra stuff for snapshots. But we're all, the core is the same. And we just use extensions for those small differences. Yeah, one of the things I wanted to point out is, you know, we both have different approaches. So HP, we provide root access to the end user. So we didn't really need these things where we, you know, do whatever you could do in MySQL, such as creating a user, creating a database, we decided, hey, let the user do that through root. Rackspace is a different approach. And at the same time, you know, even though we have different approaches, we're able to leverage RedDwarf to, you know, meet that common goal. This is really based, we're starting with MySQL. So the core of RedDwarf, you could potentially run other applications as well. Mike Bastant gave a talk at the last summit where he actually showed people standing up, PHP and some other technologies, using the same underlying methods. Now all that code is just sort of in the fork that he made experimentally. But in theory, this could be used to stand up all sorts of things. Yeah, exactly. So right now the focus is on MySQL, but it totally has potential there for that sort of thing. So here's the architecture of RedDwarf. And if you look at it, it sort of, well, it looks like a little version of the Nova architecture, right? And Nova, you have various daemons like Nova Compute, Nova Network, and they're talking to an infrastructure database and over a message bus to communicate to each other. Well, RedDwarf looks very similar, and that's not an accident. It looks that way because of course we started as a fork of Nova, but it's also because we want to be as close to Nova as we can. We want this to be a very comfortable environment for people who are used to open stack. So we have our own infrastructure database and message bus that we talked to. And one of the few differences is that the RedDwarf API and Task Manager communicate to the Nova API to get information on Compute instances because a MySQL instance in RedDwarf is really a Compute instance plus the status of the MySQL application inside it as determined by a guest. So when you issue a call to create a MySQL application instance, what you actually do is RedDwarf will provision a new Compute instance with an image on it that actually has a guest inside. When the instance wakes up, the guest starts receiving messages over the message queue and will basically check the status of MySQL. And when MySQL is confirmed to be active, that's when the user sees the status of active and they know it's okay to use their database. So at Rack Space, we have things just a little bit different from the reference implementation. So our version of Nova is a little as we've tweaked it a bit. We're using OpenVZ for the virtualization. So we actually, rather than provisioning VMs, we're provisioning OpenVZ containers. And instead of provisioning normal VMs, we added a driver for HP Sand Gear that we're using. And both of those are things that we're hoping to open source. In addition, the guest agent, we use an agent written in C++. We decided to make the reference agent based with Python because people didn't really want to mess with C++ and building it all. So that's another minor difference. Okay. So at HP, the picture looks very similar. A couple of changes here. So the backend service, our backend service is actually swapped out with a Java-based backend service. So we have this application server that can do things like an orchestration layer that has an orchestration layer built in. The other difference is that we have an agent that is Python-based. We're not using the C++ agent. We're using a Python agent. The main thing to note here is that Nova is a black box to us. We talked to Nova purely via the public endpoint. What that means is we don't actually share anything with Nova. We don't share the message bus. We don't share the database. We have separate instances of everything forward dwarf. And the other piece that comes into the picture here is Swift. We make heavy use of Swift. It's used to store our customer database backups. Essentially, we're also using features in Swift, such as ACLs, to make sure that, you know, snapshots are stored securely within Swift. Okay, so the status of this. Both of these services are in production right now. You can go to either of our company's websites and you can start using these today. So this, you know, technology, it's not really theoretical where we actually have launched on this. So the status of the code and the open source efforts. Right now, the API code is shared. If you provision a MySQL instance on either company service, you're going to be going through the same code paths. We use extensions where necessary, but even those, many of those are in the public right now. For, we also strive to have RPC compatibility. So when we send a message from the API daemon to the task manager, we want that to be compatible with the way HP is doing it, with their Java back-end. Similarly, for the guest, we want to make sure that those RPC calls are the same as well. And finally, there's open source versions of the guest and the back-end in case you want to pull this down and play with it today. Okay, so we're talking about, you know, API as API compatibility. So, you know, one of the things that HP and Rack Space are both committed to is the core API for RedWolf. So we're working, you know, we're collaborating with one another to come up with that core API. And we want to make sure that the API also follows OpenStack standards. So, you know, when you use it, it should look like Nova, it should feel like Nova, it should feel like other, you know, APIs with an OpenStack. So, you know, we're working hard to make sure that that happens. Both companies are also committed to the same CLI. And that CLI is called Python RedWolf Client. You can go and actually get that today. This CLI, you know, if you were to use that today, you could point it to Rack Space, it would work. If you point at HP, it would also work, you know, just the same. We're also doing some work with J Clouds to get some Java support for RedWolf. And we're trying to get a patch into J Clouds here pretty soon. That'll give us that support. So, leveraging OpenStack. What parts of this are we using for this application that we're building? Well, of course, we use Nova to provision virtual machines, which is pretty obvious. And we also use it for OpenVZ containers. We use OpenStack Common pretty heavily inside RedWolf. We use it for our REST API. There's a lot of good bits there that you can pull out if you're creating a REST API. And we use it also for the RPC communication. We use OpenStack Common RPC, which does all of that stuff for us. So, it's pretty cool because we get bug fixes for free by updating the latest OpenStack Common code. HP uses Swift for securely storing snapshots, which, you know, if you need to upload something to an object store, you know, Swift's there and it works really well. We use Keystone inside the API so we can have user permissions on the different objects. We just use the Keystone middleware. And we use Glantz so we can store the images that have the agent baked in. Yeah, so one of the other points here is that, you know, we have a management API. We have a public API. So Keystone comes into play as far as, you know, setting the right roles, setting the right, you know, permissions for the different APIs. So Nova is a building block. I think maybe a year ago, well, Nova's really made a lot of big strides in the last year. It's a lot more stable than it used to be and it's able to point where you can sort of bet the farm on it. You can build an application on top of it. When we first started, we forked Nova because we wanted to have, we figured that there are certain things we wouldn't be able to do if we didn't make it a fork. There was different code pass that we wanted to have to, you know, communicate with it. And in some ways, there is, of course, more you can do. And there are some things we miss. But these days, you can get plenty of information through the API. And I think, you know, having unforked it has made it much easier to work on the app. So yeah, another thing is that you don't have to use Python to work with Nova. I mean, HP's having a great experience using Java. And because it has a pluggable architecture, we're able to sort of use Nova even in ways that are very different, like using OpenVZ is, of course, the best example. That's something where we figured we'd have to sort of write this provisioning engine from scratch. And being able to plug the OpenVZ driver in has really helped us there. OK, so, you know, we use Nova a lot. We use OpenStack a lot. So we've learned a couple of things along the way. One of those is notifications. You know, there's a lot of stuff that actually happens when you boot an instance. There's a lot of events that go back and forth. And at this point, you know, there isn't a really easy way to get those notifications. And if there was something like that, it would help, you know, build applications on top of Nova that, you know, could react to failures, react to events much more quickly. HP is deployed in a cross-AZ environment. We have three AZs in each region. And we've also deployed Red Dwarf in all three AZs. So one of the issues we've found is, you know, across these AZs, things are not actually shared. So things like key security groups and everything you have to provision, you know, you have to duplicate all of your provisioning in both and every single AZ. So if there was some way to actually get some of those resources to be shared, it would help, you know, build more robust applications on top of Nova. Resource cleanup, you know, Nova is distributed. It's async. Things fail. You know, there are, you know, opportunities for implementing fault tolerance. So things like reconnecting to, you know, RabbitMQ, things like cleaning up things that are stuck in between, you know, ideally, there would be some better handling of that within Nova. Transactional state management, Nova is async, Nova is distributed, and it's hard to implement transactions across distributed systems. So, you know, booting an instance takes a lot of things that actually happens when you boot an instance. One of them is booting an instance, creating a volume and attaching that volume. If any of those things fail, that instance is useless as far as a database instance is concerned. So ideally, there would be some support for, you know, rolling back things that are, you know, partly completed. So either I get everything or I get nothing. And Nova or something shouldn't take care of that. And that same story goes for the long-running job support. So some system, whether it's a Nova, whether it's, you know, with an open stack somewhere, should allow us to, you know, submit a ticket and say, okay, you know what, I want you to do these three things and let me know when those three things are done. And if nothing, if something fails in between, I want you to roll everything back. Yeah, and that's something that could be put in an open stack comment or it could be another project. But it's definitely something in open stack. Hopefully we'll gestate. Another thing is we really want better error messages. So we have a management API in Red Dwarf as one of our extensions. And when something goes wrong, on the Red Dwarf layer, we can call the management API and it'll tell us, oh, the volume didn't provision. It would be nice to have sort of a similar functionality in Nova where if, like, for example, the volume failed because the driver had some issue. We could, from an API, actually get that information back. Right now there's nothing that's that nice in Nova. I know they're making strides, but it'd be really cool if we had that available to us through some sort of API so we could actually query the Nova API and return that through our management functions and admins didn't have to go searching through different log files to find it. The other thing is that if we could get better traceability when we go and do create calls where we maybe don't necessarily have the ID back, that would also help us just figure out what went wrong quicker. And finally, OpenStackCommon, if that could be packaged in some way, right now we're sort of copying in files and I think I know there's talks going on this summit to make that more efficient, but that would help us out a lot. So where are we headed? You know, we have a road map as far as the features that goes within database as a service and we want to support, you know, things like master slave. We want to support things like automatic failover, right? We want Red Dwarf to do some of these things for us. So what's lacking at this point is this orchestration layer, thing that can... What I want to be able to tell Red Dwarf is, hey, go create me a master, go create me a slave and configure each one to talk to one other. And if anything fails in between, I want you to roll everything back. So we're trying to build that sort of a layer within Red Dwarf, something that can do transactional-based, long-running processes. Scheduling. We also have features that we want to build such as, you know, automated backups, automated upgrades of your instances. We want to build in some sort of a scheduling support within Red Dwarf so we can do these sorts of automated tasks. Language winding. So we already talked about Python Red Dwarf client and we talked about J Clouds. Those two, one of them is available today. J Clouds will be available soon. But what about Ruby? What about the OpenStack Java SDK? So we have some work to do as far as getting Red Dwarf to be embedded into some of these other environments. CI. So, you know, back in, I think it was the last summit, was when StackForge was informally or formally launched. We want to, you know, put some effort in getting Red Dwarf into StackForge. We want to, you know, we want it to actually be part of the OpenStack CI process. And what that means is we probably would have to do things like write DevStack extensions and RackSpace has actually started some of this work with what they're calling RedStack. So the other things are like Tempest, you know, we want to make sure that our tests are run along with the rest of OpenStack. We want to test Red Dwarf with the tip of, you know, Nova with the tip of Glass with the tip of Swift. The golden image. So, you know, Red Dwarf and database services, they're a little bit different from the rest of OpenStack in that there's a guest agent, there's a MySQL. There's all these things that are actually baked into the image. So we want some process that can actually, you know, automatically create this image and test changes to the code base and, you know, run the integration tests that are required to verify that the agent and MySQL and everything is functioning properly. Okay, so how are we planning on getting there? First off, we'd really like to focus more on community building. We'd love it if we had more contributors, especially if people in this room want to pull down the code and start making pull requests. And then to that end, we're going to dedicate a resource to doing stuff in the public. So if anyone's familiar with Mike Bassknight, he's going to start actually working more on pushing stuff to the public soon. And, you know, typically OpenStack, it's a two-step process. You can put code in the open, but you also need to really work to push it back. And I think where we faltered a little bit is on that second part. And Mike's hopefully going to help us there. So we also need to reduce the learning curve. RedDwarf, as it stands, isn't that hard to use, but you need someone to teach you how. We have a little bit of documentation. We need to add a lot more. We need to add tutorials and make it easier for people starting from scratch to be able to pull down this code and play with it. And we also want to make sure that we're following sort of OpenStack best practices, just so we can be more aligned. So things that are happening on Stackforge. So when people pull stuff down, it's a very, it's a common experience. They're sort of used to all the different stuffs that it takes. And finally, I think everyone of all the RedDwarf would love to see it get into incubation. So we'd like to start pushing to have that happen. Okay, so on the team right now, we've contributed to NOVA a little bit as we've been doing RedDwarf, just so you can see that this experience has sort of given back to OpenStack a bit. And then of course, we have contributors to the RedDwarf project itself. Most people at Rackspace have contributed to the public, Vipple's contributed to the public branch as well. And so just so you see that there's work going on in the Open right now because of this project. So if you want to get started with this today and pull it down and sort of run your fingers through the code and figure out how it works, you can and you don't necessarily have to install the whole thing. You don't have to have a running installation of NOVA. What you do is you go to HTTPS, github.com, hubcap, reddwarflight.get, and you clone that into your machine, your laptop that you've taken with you to the conference. And then if you have TOX installed, which you're gonna need to figure out how to do that, but you're gonna wanna run in the bin directory, startserver.sh, and what that's gonna do is actually gonna start running RedDwarf locally on your machine and you'll be able to hit the API. And what it does is it fakes out the NOVA pieces. This was one of the big advantages of not forking NOVA is it sort of fakes NOVA, so it fakes NOVA and the guest response. And we use this extensively for testing, but it's also useful just so you can see how the API reacts. I know there's teams that we've given this to in Rackspace that need to write like a GUI over RedDwarf and they can actually run it locally and they don't have to set up all the stuff. And it's a good starting point if you just wanna see what the code's like and how it runs. And if you're feeling really ambitious, you can also clone our tests. And there is a script inside the test directory called run local that will prompt you for the path to RedDwarf and also for a path to Python RedDwarf Client and it will run all the tests locally. So if anyone's feeling bored tonight and doesn't wanna go to the parties, wants to set this up, it'd be more fun. So, all right. So if you want to get more information, we have our Launchpad page. We have the different repos. We're always hanging out in RedDwarf, a pound RedDwarf on RSC. There's the original blueprint for the public API and it's in the wiki too. So, at this point I'd like to open the floor up to questions. I mean, which Nova are we compatible with? Which Nova are we compatible with? Yeah, so. Probably not Bexar. We're running it against the Nova Client that's designed for Essex. Yeah, and we're right now, we're on Folsom. We'd love to. I mean, so as a developer, we'd love to, right? It's exciting. So, Mike, last night at the last summit gave a presentation where he had branched off RedDwarf Light and he actually was, he was setting up Apache as a service. And I forget, he set up two other things as a service where he made it an API where it basically did the same thing. It would just launch the guest and the guest would just install something else. So, anyway, I mean, it's very possible. We're just focusing on MySQL. We'd like to get that solidified right now. Why not Juju? Well, I mean, that's a good question. I think it's very lightweight. And so, when we, when you get the image, right, it already has MySQL on there. It uses like AppGit to update MySQL and it's using a guest that's very, very small. So, things like Juju or Puppet or Chef, we could have used those and I think that's a good solution for a general, maybe if you're trying to make like an orchestration layer that can just put absolutely anything you'd want to on a server that would work. We really want this to be about you get MySQL and as little other stuff as you possibly need. So, you are just paying for MySQL. Okay, so at Rackspace, we connect a volume and volumes are actually, we make them mandatory. So, there's a flag in Red Door Flight that you can flip on or off. And we actually, the way we're configured, you always get a volume. So, we're using, yeah. So, we actually do use an HP, we actually use a Volumes API. So, we create a new volume and then we attach that to the instance. And that's not something you have to do separately like in Nova, it's actually part of the server or the MySQL instance when you create it through the REST call, it's the volume size and it hooks it all up for you. Yeah, so when we create the instance, we also provision the volume and similarly when we delete the instance, we destroy the volume as well. So, yes, the log actually I believe stays on the volume. We have it configured, so, yes. So, HP actually has this implemented. So, we have a Snapshots API where you can create a backup, you can create an instance from that backup. The plan is to actually get that into the reference implementation of Red Dwarf. So, we're a little bit ahead in our internal implementation and we're trying to push these changes into the public implementation as well. Resizing, so, right now we do support resizing for the volume and for the RAM. We don't offer any tools per se, although I know it's possible to run certain performance tools and figure out the performance of an instance. Okay, so, you're saying that we use different virtualization strategies. I think it just falls down to how we decided to run the products and how we're deployed. So, I mean, when we started, when we started Database as a Service, the idea was that we were going to be running OpenVZ and we sort of knew the gear we were gonna put it on and I think, I mean, when you guys started, it was you had a big NOVA rollout. Yeah, we already had NOVA with, you know, with the KVM hypervisor at that point, so we wanted to place something on top of that. You mean in Horizon? That would be cool. We haven't worked on that yet, but it would be neat. We have different GUIs planned for both products. Right now we have something in Reach at Rackspace and sure there's something. Yeah, we have a GUI at HP in the works as well. But it would be very nice to have something on Horizon. But we have to get to incubation first or else, you know, the Horizon people would have really no reason to accept poor requests with Reddwarf stuff. You know, I really wouldn't wanna say anything right here. We have, I know on Rackspace, we had a Rackspace blog entry that had some numbers that showed, it was pretty fast. But yeah, basically, I mean, it depends. The fact Reddwarf is configurable enough that it depends on how you're rolling it out, right? It depends on what Nova is provisioning to. Oh, at Rackspace? Yeah, at this point we don't have any published numbers. We have some work being done to actually get those and I think the plan is to actually just roll out a blog post for that. So at Rackspace we've done some metrics and I can't remember the numbers off the top of my head, but it's pretty fast. Because OpenBZ on top of the gear that we're running, it beats putting it in like a cloud server. Any more questions? You know, SQL, like I said, the core of Reddwarf, we could probably offer other things as a service. You know, as a developer that's kind of exciting. In terms of right now, I mean, we're focused on MySQL and I think something more like, well, I mean, we're not working, I don't know what we're gonna work on next, so I don't want anything to be misconstrued, but something like a SQL database would sort of make more sense. But I mean, we could really run anything on top of the core Reddwarf code base. Right now we're simply focusing on MySQL. Absolutely. When Mike gave that presentation last time, I think we felt that maybe the reception was a little lukewarm, and so we're focusing on MySQL, just getting that available publicly. Because I think just explaining it, people sort of say, oh wow, you could probably launch other things, but we probably want to push that publicly before we start branching in and doing other services. I wouldn't say so. I think Puppet and Chef, they're a lot richer, right? In terms of what they're doing. I mean, this requires an agent that is sort of tailored for the app. Now, how hard that is, it depends on each app, right? For some apps, I think it's fairly, I wouldn't say it's simple, nothing is simple, but I know that our lead was able to set something up that would provision Apache. Now, how good of a service you could build that would provision Apache using this is up to the app you build. So another way to think about this is if you are a company and you're launching a product, you probably want to go with Red Dwarf because Red Dwarf, it's all based around MySQL. So it's not just sort of you have a server and then there's Puppet and there's all this other stuff and you're giving people sort of a collection of the server and then, hey, you can provision whatever you want. This is MySQL, it's tailored around MySQL. And I think what Red Dwarf does better in my own opinion, which is biased than other sort of platform as a service things I've seen, is that if you're trying to offer an app as a service and make it very fast, I think the approach from Red Dwarf, it works. It's going to feel very good. Like it's gonna feel unencumbered, that makes any sense. Not particularly, I don't think. Yeah, for the actual like Red Dwarf project, no. You wanna answer that? No, that was, the architects who came up with the idea, they looked at the performance and they decided OpenVZ was gonna be much faster. Thank you, you've been a great audience. Thank you very much, guys. That was a great tag team by HP and Rackspace and one of the better QA sessions we've had. There's only a 10 minute break until the next talk in this room and the next talk is on...