 All right. So welcome. Good morning. My name is Alan Metz. I head up the Atlanta engineering office for Rackspace. We're the office that does engineering for cloud backup, for jungle disk, and most recently we've been working a lot on Project Marconi for OpenStack. This is Kurt Griffiths. He's our principal architect. He's been spearheading the Marconi development. He had a lot to do with the design and implementation of it. So what is Project Marconi? Basically, Marconi is intended to be the API, the REST-based API for queuing and for notification. So it has a multi-tenant design, very simple and clean, RESTful API. One of the things that we did with this is we have a unified producer, consumer, and public subscribed mode. So that, you know, really you can use either or both of those modes simultaneously. We designed it to be performant, to scale horizontally. Some of the foundational technology that we've chosen is pretty promising. I'll talk a little bit about that. And the other thing is that we really want it to be, rather than try to be a kind of monolithic queuing application, we really want it to be more a set of primitives so that it can really support all sorts of web messaging patterns and paradigms and so forth. So really what we want to do is look at all the use cases out there for any sort of web-based messaging communication, you know, public subscribed fan-out kind of stuff. And we really want to get into all those and make sure that the API supports all those. The core developers on the project, Kurt from Rackspace, Jamie Painters here also from Rackspace. The other individual who is a core contributor is Flavio Percolo-Premoli. And he's in Italy. He works for Red Hat. And we have several other developers at Red Hat that are at Rackspace that have been working on this as well. This is essentially what it looks like. So we've gone with a pluggable driver-based modular architecture. So out of the box, we're supporting WSGI as a transport. And the underlying library on that is Falcon. We'll talk a little more about that. So, you know, this is where the real guts happen. This is the queues, messages, claims, statistics and everything. And then we have storage drivers. We've written two storage drivers so far. One is a SQLite driver for development, for quick and dirty development purposes. And then MongoDB, which is intended to be our first sort of real world storage driver. Notice that authentication is not part of the project. The intention there is basically a middleware approach to go off to Keystone. We also have a Python-based client library written in Python, of course. So going further down the road, we're thinking about, you know, what are the transport drivers there may be? So we're looking at maybe, you know, something based on zero MQ. Additional functionality around auditing and being able to track what went into the queue, what people took out of it, and so forth. And then of course other storage drivers as the need arises. Kurt, why don't you just talk a little bit about Falcon with the underline? Okay, so Falcon is a project that grew out of some performance testing that we were doing a few months ago around MicroWhiskey and various Whiskey frameworks. The interesting thing about the Marconi project is, unlike many of the other open stack APIs, it's a data API. It's not really a management API. And so we have very aggressive performance and latency goals around that. And most Whiskey frameworks out there today are simply don't perform that well under those kinds of workloads. And so Falcon is a, it's a micro framework, but it's highly optimized for this use case for building cloud APIs to push a lot of transactions and a lot of data through there. So just to give you an idea of some things we've been able to achieve with it, it is about five or six times faster than Flask and about two times faster than Bottle, which is probably the fastest framework that we've tested. So we are, we're excited about this and we're hoping to get some good benefits out of it from our county. So, you know, if you look at our use of Falcon here, and as many of you know, we recently acquired Object Rocket for MongoDB expertise in the product suite that they have. And between, you know, what Kurt has done with Falcon and what some of the Object Rocket guys are telling us with Mongo, you know, the foundational technologies underneath this should be capable of messages on the order of tens of thousands and messages per second. So we are, we have high hopes for this from a performance standpoint. And then, you know, basically we, well, I'll talk about where we are in the development in just a minute, but just a quick review for anyone who may not be familiar with these concepts is to kind of get us all on the same page. So producer-consumer, you know, the idea there is that you use this in sort of a job dispatch environment. The idea is that you're going to have one and only one worker who's going to claim a job and do that work, so you don't want, you know, multiple workers getting the same message and trying to act on it. So we have this notion of messages being claimed. And when the worker is done after they've claimed it, the worker is responsible for deleting that message off the queue so that no one else will get it. If for some reason something happens and the, you know, the worker dies while it's in the middle of doing its job or something, then there is a TTL feature that will return the message to a claims, or to an unclaimed state so that, so basically some other worker can take it. So yeah, this is ideal for dispatching jobs to multiple processors. The other mode would be public subscribe, which is really more of an observer-listener kind of the pattern. You know, all of the components on the system are watching the queue. They're not claiming messages. This is good for alerts, notifications, event dispatching, that sort of stuff. You know, the queues will eventually expire with a TTL feature that's on there as well. And, you know, one of the things that's kind of important there is that the subscribers need to know what they've seen before and what they haven't. So in this case they can send a marker back to the queue to say, all right, I've seen this. Don't send me this stuff again. I need to see this stuff. This is a quick look at the guts of the API. It's really simple. You know, we have queues. You can create list queues, set metadata about them, get statistics, messages. You post messages to a queue. You can claim them. You can retrieve them, obviously. And then there's things you can do on a claim basis. So really clean and simple. This is what an example API call looks like. The payload is JSON-based. You can essentially put anything you want in here. You know, something that's well-formed JSON. This particular message is posting two messages to a queue called fizzbit. We put the TTL as per message. So we have two different TTLs here. And even if you notice the payload is different stuff in the payload as well. So really whatever you need, it's, you know, we're not really picky about that. And you get a standard HTTP response with object IDs that kind of points you back to the messages you just inserted. So where are we? We literally finished coding the API last week, at least for the happy path. So we got this up and running. And, you know, we're basically have what we consider the main line work done for the first version. We have a five-second installer, so you can do pip install. I actually have a screen that'll show you that. There is a client library that exercises a lot of API, you know, and we have unit tests. We were having, we did have 100% code coverage on the unit tests. I think we fell behind in the last couple weeks trying to get into crunch mode for today. And pretty much have the happy path done for system tests as well. So what needs to be done is edge cases, error handling, ruggedizing, packaging, all the things to really make this production quality. Performance tuning, as I indicated, we're excited about the foundation we have. I think the code itself, you know, we're just having just finished coding it. We now need to go into a performance tuning mode to start getting those messages per second way up. Non-Python language bindings. If anyone out here wants to take on some Ruby or some JavaScript, whatever, there we go. That was a volunteer. We haven't really done benchmarking through benchmarking. We have a development sandbox, and I think we got that thing up for like 500 messages a second, but that was really just, okay, crap, it's done. Let's turn it on and see what it does. So we expect a lot more. So getting our CI CD environment up, finishing up the system tests, another place that we can use some help is documentation. One of the things we'd like to do is, I mentioned all these use cases and common patterns of web messaging. We would love to build a catalog of those and say, hey, if you're trying to do this kind of an application, here's how you would use Marconi, the primitives of Marconi to achieve that. So we would love to do that with, I guess that's really more operator or user documentation, but then we can also use help with documenting the API itself. One of the things that the question that's really common is what are the use cases? Like what, we have 0MQ or we have rabbits, why do we need this? And we can talk about that a little bit more later, but I think that that catalog would really help people clarify when you would use what. We're not trying to be everything to everyone, but we kind of see Marconi as a macro version of some of these more intra cluster messaging buses. And a large system would likely use a hybrid of several different solutions. I mentioned the five second installer. It really is this easy. Jamie did this last night and sent me the screenshot. So that's actually really happened. But basically it's a pip install from Git and then you run it. And you've got Marconi server that you can use to test out the client at your own applications and so forth. So we're going to do a demo today. I wanted to show this slide to sort of give you the picture of what the demo is going to look like so you know what you're looking at when you see it. What we've got set up is we're going to have four queues running. There's a request queue and a response queue. We have some client applications here. This is some producers that are doing basically just pumping messages into the queue, putting jobs in there. Well, I think there's six of them running on two different nodes. We have consumers on the other end that are doing some computationally intensive work. I think they're calculating Fibonacci sequences and factoring primes or something like that. So there's six of those as well on the other end. They're going to be, so these guys are doing insertions. These guys are doing the claiming. They're going to be claiming messages off this queue. When they're done, they're going to do an insertion into this response queue. Additional two queues are really to control these guys to tell them what to do. So I'll start the whole thing off by putting in a command in this queue that says, hey, change your production rate, start publishing messages, and that'll start these guys off and it's time to stop them. I'll set this to zero. And while that's happening, we've got four graphite graphs that you'll be able to see. One of them is going to show... This is just periodically hitting the stats API on the queue, saying how many messages are in there. So that'll show you how many tasks are in there. This one will be showing you how much work is completed. This one will be showing you how much or how fast stuff is going in. And then this is going to show you the size of the response queue. One of the things you may notice here is there's nothing pulling stuff off of the response queue. What we're going to do there is we're going to show you that we're just going to let stuff sit in there and decay so you'll actually see the TTL kick in and you'll start seeing that message kick in. The message is decay. Notice that these guys are subscribing to the queue as those, so that's your public subscribe model going on simultaneously with the producer-consumer. And honestly, there's nothing that couldn't... You can use the public subscribe on the same queue as well. We could have like a listener here but just listening to every message and posting it on Twitter or something. And speaking of which, we have a little IRC bot here. It's going to listen to that message. It's going to post it on IRC. So with that, let me minimize this. And actually, before I do that, I did want to show that... One thing I didn't mention is that creating queues is really lightweight. Queues aren't supposed to be big, heavy things. So if you have an application that... I know we talked to some folks that were doing some stuff from South by Southwest and they wanted to have a queue for every user. So they wanted tens of thousands of queues. And that's really what this is kind of intended to be, it's intended to be lightweight. Let me reset this before I forget. I'm supposed to do that before we started. Okay. So just to show how lightweight queues are, I just made a quick little script here. This says create in queues. It'll say let's create a thousand of them. And in two seconds or so, we will have a thousand queues. Okay, three seconds. Four seconds. Come on. There we go. Show you that they're really there. There's your thousand queues. So that's the idea of queues are supposed to be lightweight. They're really just as lightweight as messages behind the scenes. So let's do this other demo. So what I'm going to do is I'm going to tell those producers to start putting in messages. So you should see, there they go. So the messages are now going into this task queue. You now see that there's tasks here. The workers have started working. They're working them all and they're putting the results in the results queue. Oh, and the demo bot heard that message on this queue or on the control queue and said go producers go. That's going. And just do circle. Actually, haven't got to see. Let's kick it up a notch. So kick it up a notch. There they go. Faster. Up. Them going in. Oops, let me bring that. I have terrible resolution on this. Well, we should see. Okay, so the consumers have already finished. You see that leveled out. The tasks in queue just emptied. And we're now seeing the TTL kick in. So these are just going to decay back to zero. And that was our quick and dirty simple demo. So really, that's where we are. That's project Marconi in a nutshell. We'd love help. Love contributions. This is where you can find us. As we mentioned, you can download the code. You can play with it. Please write language bindings for your favorite language. And let's help help us make this thing real. So we got some time for questions. If you guys have any. This is the queuing. I mean, we actually, we wrote it. We, the architecture is a pluggable. So you can, if you wanted to plug in, say, rabbit as your storage backend, you could. But we're not, we're targeting Mongo because that sort of gives us HA and durability for free. And sharding and all that good stuff. So we're not necessarily looking to replace existing queuing services. This, so there's been a lot of talk this conference about, this is the year of the user. And we're focusing on things for people writing apps on top of OpenStack. And I think this is more where Marconi would fit. Where I'm creating some cool, you know, I'm a startup. We were talking to Circle Media. They do, they do these big, they do social marketing for really big events like South by Southwest. And they have a lot of, you know, things that, some unique requirements that aren't fit by something like a rabbit or a zero MQ. Things like I need to be able to support hundreds of thousands, even millions of topics and clients with under one tenant. And things like I need to be able to go through firewalls. And I need to work over highly unreliable networks. You know, basically the internet. And things like that. So it's more plain in that world. Now, if over time people want to bring that in and use it within like a smaller cluster, we're not going to stop anyone. But I think that's, you know, the, looking at these cloud applications built on top of OpenStack is more where Marconi is best suited to help out, I think. Batching systems. Can you maybe give an example of what you're thinking of? Yeah. I think, so we're trying to keep this simple, kind of take the Linux mindset where we want to have, you know, it's simple tools that do one thing really well and then you can connect it up to build broader solutions. So to answer your question, I mean there are several ways you can do it. You can take the, you can use it for just building up the basic queues and then you have, like there was a session I went to yesterday about proposing workflow service. Now, the workflow service could use queuing underneath to do some of the plumbing so it doesn't have to reinvent that wheel. So I think that's a better, kind of a better story. Now I will say that, again, the architecture is pluggable. You can swap out the transport and the storage and we've also been talking about allowing you to do, put in like hooks so you can like do interesting things as messages fly by. So we're trying to give a lot of options but obviously if you're putting hooks in there and customizing it, that's something that doesn't work in a public cloud scenario. Yes. Yeah, that's an interesting thing. We haven't explored that very much so, yeah, like we can connect after and you can start. What are you thinking on that? Thinking that way. Yeah. So right now it's all polling based. Again, so we're talking about like the sweet spot for this being hundreds of thousands of clients and just massive concurrency and that's actually more efficient to do on the hardware side when you're doing polling, it turns out which kind of may seem counterintuitive but it does add a small amount of latency. Often that doesn't matter a whole lot. Like at Rackspace, our cloud backup product, the control panel has like a progress bar and stuff so you can see what's going on with your backup. You can even go and remotely browse your server and check off what files you want to back up and all of that actually happens through sort of the precursor to this project that we have internally and it's polling based. So we've actually seen really good usability even with a polling based model. A couple of things that help with that is keeping just the turnaround time for getting each request very fast. So between 20 and 40 milliseconds. Also, you can use Keep Alive, you can use some other things. So we're starting with that. Now, we have gotten requests for adding things like WebSockets or a zero-round queue transport and things like that and we can definitely look at that because I think there are lots of different interesting scenarios that you can enable. They're always the trade-offs though as far as scalability and efficiency. Anything else? Yes. Are you talking about client libraries or language bindings? Yeah. So like I mentioned before, the transport is pluggable. We've talked about 0MQ, AMQP. So if you wanted to bridge to those other systems, you could. Is that what you're... Maybe I'm misunderstanding your question. Yes. How do you do it? So everything in the system today is a REST API that looks like this. API documentation, is that what you're talking about? I'm sorry, I'm having trouble. Yeah, yeah, yeah. So we are on the OpenStack Wiki, and we have this slash Marconi that has a pretty good overview and it also has the blueprint for the API. I mean, it needs to be cleaned up and everything to be developer friendly. We've been just using it kind of as a working draft at this point. So if you get on there and you start playing with it, like Alan mentioned, we definitely need a lot of help getting everything documented and all the different scenarios. So we can do it with Marconi or is it even a good fit for that? So... But it is like, you can just... I mean, it's really simple to just interact with it from curl. The API, we intentionally tried to keep it very clean and simple. Yeah, essentially, when I was doing the demo of the little commands I was running, there were the little 10-line Python programs that just assembled a message and sent it out to the REST API, basically. So that... When this becomes so awesome that they want to use it, that the other guys want to use it, then they can. Right. So we really built this because our customers at Rackspace were looking at our cloud solutions and saying, hey, we love your object storage. We love your cloud servers. We really want to be able to build more complex applications that can connect all these pieces so I can build my application across the cloud components. So that's where we started. That's the intent of this, is to basically be an offering for cloud users. Again, we want to make it as performant as possible and as scalable as possible. We haven't... We didn't design it to be a replacement for the RPC mechanism in Nova. Like I said, if it becomes so awesome it won't stop them. But right now, that's not the plan. So... I mean, before I answer that question, can you talk about some of your... Well, I guess HA is a topic that people... Because you can make rabbit HA, but it's a big pain and stuff like that. I'm not saying we're specifically targeting it, but it's the easiest target, I guess. So with HA, one of the benefits of just doing a good old REST, using the REST architectural style, as you get a lot of those semantics, not really for free, but there are well-known patterns for achieving HA with that. So it's less a function of Marconi itself, I think, and more a function of the transport coupled with your storage. So, if you... For example, if I'm using HTTP, you can put the load balancer there and you can go across things and all that. So that's HA, just communicating with it. And then as far as the storage goes, Mongo has actually got... TenGen has got Mongo to the point where it's got a really good HA story there as well. And there are obviously other things you could plug into the back end, depending on the performance, durability, HA characteristics you have. Now, that being said, we're not planning currently to fail over to an entirely different data center, for example, at this point, although if people are interested in that, we can talk about enabling that at some point. It's kind of a weird... It's not really infrastructure. It's not really pass. I think you hit on it when you said it's an enabler. It's really a catalyst for tying it together. So, yeah, I think that's fair to say. Like Alan mentioned earlier, as OpenStack Story grows up and people want to run bigger, more complex applications on top of it, there are just certain pieces that you have to have and they don't necessarily fit in a certain layer. I think we need to focus more on just what people are trying to do with the cloud and go build that and make it easier. So people aren't writing. The other reason this Q&A project came up was because at Rackspace and other companies, everybody's been basically writing their own Q&A system. So there are like several dozen Q&A systems out there and we said, hey, let's factor that out. Let's just make this a service so people can stop worrying about it and wasting that time and they can focus on making their apps really great. Not so much a service as much as just a primitive so they can kind of put the Lego pieces together and what was the word you used last night? Affordances? Right, I mean we're really aiming at giving people the Lego blocks to build up these great products. We're not going to code like push to email and SMS and everything right into Marconi but if you want to build that kind of stuff on top of it as a value add, it's great and go and do it. And now you don't have to worry about just the plumbing part of it. I can worry about making a great user experience for my customers and I know how to do mail really well. Mail gun, they've got that figured out. I don't want to run my own mail server but I can do queuing so let's plug these things together and then now I've got an overall solution that my customers can build upon really great. Yeah we need to like I think was also mentioned when the ecosystems, you need a thriving ecosystem and kind of a core I mean think about Linux where would Linux be if it was just like the kernel? That's not very useful. So you're going to have stuff like KVM it's part of the kernel but you probably need something else if you want to do hypervisors but the core of it is there and it enables all these other scenarios. I think that's kind of what we should I mean this is me talking but you know it's part of OpenStack I think a good strategy is let's make that core let's get those common bits there and let's encourage this ecosystem around it to make a really great portfolio. And really our intention was to, you know we want to make this as real as possible and get it out there, get people working on it and then let's start having those discussions we'll start working with the technical committee and with the OpenStack board and let's figure out where it fits and go from there. Yes. Okay, well before we answer that do you have like particular things in mind so I make sure I address those? Let me talk about the FIFO. You got an hour? Well let me think maybe to kind of make a long story short one thing that okay so philosophically we went for a unified messaging pattern so instead of having two separate services I can do PubSub, produce a consumer in the same API and we did that you know so that basically you can mix and match those paradigms however you want and invent some new messaging pattern that nobody's even thought of yet as part of it and also because we kind of felt that SNS was a little bit too monolithic we wanted to just focus just on PubSub not like worry about tying all these web hooks and all this stuff into it you can build all that on top of it which is great but let's make one thing and make it run really well so as far as comparing to SNS I think that's one major differentiator it's a lot more focused on just providing that messaging infrastructure and also this is one of the things we ran into a few years ago I was at a startup before Rackspace and we were looking at using SNS when it was in beta for some stuff and we pretty quickly ran into the max number of topics limit because we were talking about hundreds of thousands of agents trying to talk to this thing and how do you do that anyway so one of our goals here was to just make cues or topics very very lightweight so you can create hundreds of thousands of these things millions even we hope that will lead to some innovation there as far as comparing to SQS I think a lot of the semantics are very similar the one thing that is a little different with SQS is they don't really guarantee FIFO to a certain level of granularity and we're trying to solve that with Marconi so if I have a producer who puts in A1, A2, A3 and a consumer would get back A1, A2, A3 in that order so there are some interesting challenges around making that scale but we have some we have a little bit more work to do on that but I think we've got a pretty good solution in mind one more minute final question alright well thank you for coming