 Hello, my name is Soren, I'm one of the OpenStack developers. I'm lucky enough to have Rackspace pay me to be one of those full-time and I have been for probably eight, nine months by now. So I'm going to tell you a bit about OpenStack, basically answer the questions of why, who, what, when, how, and so on and so on. So to first answer the question of who, we see that there are basically two major entities involved in the beginning at least and they have their own separate timelines in the history of OpenStack. So that's NASA and that's Rackspace. Rackspace got into the infrastructure as a service business back in 2005. There were two major components to the Rackspace cloud, the infrastructure part anyway. There's the Rackspace cloud files and there's Rackspace cloud service. They also have something called cloud sites but that's a completely different story. I'll only be dealing with the two others. So first of all, there's Rackspace cloud files. It's an object store which means that you can store objects in it and you can retrieve set objects again at a later time. So these objects are essentially blobs of data. It doesn't work as a block store. You cannot somehow make an object on Rackspace cloud files turn up as a block device on your Linux system. You cannot run MKFS on top of it and so on and so on. Because you can only address it as full objects at a time. Have any of you used Rackspace cloud files? Have anyone used Amazon S3? This is basically the same thing. So you have infinite storage. Of course there's no such thing as infinite storage because the universe is finite and all that but it's infinite enough. When you start providing a service like this, you promise that you will make sure that there's plenty of capacity for whatever amount of data that your entire customer base is going to throw at it at any given time. One of the things that's worth noting about... Come on. There we go. So ensuring that there's sufficient space for you is not your problem. You do not have to reserve an amount of space. You do not have a subscription that gives you like 50 gigabytes of data and then when you start to fill that up then you have to upgrade to the next bracket or anything like that. You can just upload data and you always only pay for how much you're using. You do so in very small units. So you only pay for the amount of gigabytes you have per month. The building model is very linear. If you have 100 gigabytes stored for one day, that's going to cost you the same as having 1 gigabyte stored for 100 days. So it's really very linear in that way. And yeah, it's very similar to Amazon S3. It's a slightly different API but the concepts are clearly the same. Cloud files has a CDN integration built into it and that's sort of an add on S3 but by and large it's the same thing. So the other major component is cloud service. Cloud service is what most people refer to. It does what most people call cloud computing. It's about running virtual machines. Like any other cloud computing offering it is virtually, it's essentially a VPS service with an API. This doesn't sound very fantastic but if we rewind about 10 or 12 years back when VMware just came out with their first thing that would let you run virtual machines or thereabouts, it's been a while. I'm sure there were some enter pricing companies that went out and started selling virtual machines in exactly the same way that you used to sell physical machines. So someone called up, say, I have this application and I need to run, what do I do? Oh, well, you can buy a server but we have these new fancy virtual machines that you can buy instead. It's basically the same thing, only it's cheaper. And then they would sell it to you on the same terms as they would have sold you a physical server, which means there's a setup fee. There would probably be a two-year contract because that's the way it was in the bad old days. And you paid for probably each month or something like that. Sorry. So fast forward a couple of years and we have some companies like Linode that sort of move a step forward and they start providing virtual machines in a much more automatic manner. So you could go onto a website and you could find, you could buy a virtual machine. There was still a setup fee and you probably were bound for at least a month ahead. But you could still, you could choose a number of different operating systems on it. You could choose different sizes of virtual machines and so on and so on. So it's a massive step forward. But it wasn't until a couple of years later when I think Amazon actually did it first where they, the taking the HTML part out of it isn't really a big mental leap. So just exposing the API directly to the end users is a small step for man but a giant leap for mankind. Instead of, when you do that and you remove the setup fee and you change the billing model to only charge you per hour, you suddenly have a much more flexible way of dealing with your resources. You can stop over provisioning much anyway because when adding more resources to your pool of computing power is something that takes minutes and can be done automatically instead of having to call someone up and get a server set up and all this, really fundamentally changes the way that we can deal with our computing resources. And that's when it suddenly becomes amazing what you can do with cloud computing. So, right. And again, it's the same very linear billing model. If you have 100 virtual machines running for one day, that's going to cost you the same as running one virtual machine for 100 days. This is especially fantastic for research institutions that very often suddenly they stumble upon a billion terabytes of data and they needed to just be computed quickly because, well, they're just sitting around waiting for the results. And normally they would have to have like massive amounts of computers sitting around not doing anything for most of the year except for that one day where they really need to grind some numbers. But now they can do that in a much more flexible way. And that's what cloud service is all about. It's basically the same as Amazon EC2 conceptually at least. It's a different API and slightly different feature set, but the concept is the same. Right. So, in 2009, some years have passed since they started doing this and I don't think anyone really anticipated how much cloud computing was going to take off. And even if they had, they might not have been able to realize where the bottlenecks are going to be and what are the problems that we're going to experience when this grows to petabytes of data. So, in 2009, Rackspace decided to rewrite the whole thing, cloud files at least. They started that in late 2009 and that finished around March or April 2010 and put into production in May or something like that. Right. And in 2010, they also decided to rewrite cloud servers and at the same time decided that they're going to open source the whole thing. For much the same reason they hadn't really anticipated and hadn't understood, you know, it's very difficult to anticipate where your bottlenecks are going to be five years from now, especially when it grows at a rate like cloud computing has. Okay. So the other distinct timeline is that of NASA. So NASA, we all know NASA as the people who send stuff into space and, you know, they deal with outer space and moon and, you know, stuff like that. Other than that, that's not all they do. They also function as a much more general purpose sort of research institution for the American government. And at some point, someone in the American government realized that this cloud computing thing is starting to sound like something that you should have a plan for. So the plan might be, don't do it, but they needed a plan nonetheless. So they called NASA and said, we need a plan, look into cloud computing, figure out if we're going to use it in a government sort of setting, how do we do it? And do we do it at all? And, you know, what are the caveats and so on and so on? Being a government, there's likely a lot of very sensitive information involved. You don't want to be tied to a particular provider because when your government is really dependent on one particular provider, that creates some very unfortunate dynamics with legislation and stuff. It wouldn't be good. So the NASA guys decided, well, we're going to set up our own cloud and, you know, try to see, you know, what are the challenges, how can we use it and how can we make, how can we save money, how can we do more cool stuff, so on and so on. And the way it did that back then was using something called eucalyptus. Eucalyptus is a software project that started at the University of California in Santa Barbara where a research group basically sat down and, I mean, this was very early on before no one really had defined what is cloud computing, what's it all about? It was all very cloudy. But even back then, everyone agreed that what Amazon is doing, that's cloud computing. So they said, hey, let's implement Amazon EC2 again for ourselves. So they did that and the result was eucalyptus. And NASA started using eucalyptus to build what they call the nebula. Nebula is this cloud that you have out in space. So they have a way with finding good names for stuff. So this is one of those. Nebula, it comes in great big containers that are like a great big pool of resources that they can just build clouds out of. It's very component-sized. It's really quite cool. So this is one of them. Let's see what's next. Right. But unfortunately, as time progressed, they started having more and more problems with the eucalyptus. There were reliability problems, scalability problems, different sort of utility problems. So unbeknownst to anyone but them, apparently, they had just chucked away eucalyptus and started writing their own thing. And this was in early 2010 as well as around February or thereabouts. So they started writing their own thing. And that was pretty interesting. Because at the same time, rack space was kind of sitting around like they had decided they were going to rewrite it. They were figuring out what sort of technology they were going to use, what are the operating systems that was kind of given. Programming languages and other technology choices. And they were sort of converging on writing it in Python. It was a distributed asynchronous system. So message queue seemed like a really good idea. And then out of the blue, this thing from NASA turns up. Someone tweeted about it. I think it was on May 28th. And I saw it on Twitter. And I went to look at it and it said, this is a car computing thing. It does easy to, it looks like easy to, it's running at NASA and this and that. I thought they were using eucalyptus. What's going on? But as it turns out, they had got fed up with eucalyptus and replaced it. So I went to look at it and I was quite surprised to find it was written in Python. It was using message queues and it was really modular. And it had a lot of really useful abstractions. It was just generally really well written. So we were super happy because, well, this could save us a whole bunch of time. Because we were going to do the exact same thing. NASA has this, part of the rules of NASA say that everything they do has to be for the benefit of all mankind or something like that. So everything they do has to be open sourced eventually. So they wanted to do open sourced stuff. Clearly they published it on the internet. So did we. So it's perfect. So eventually someone at a rack space called up NASA and saying, hey, we've got this cool new object store that we just finished writing. We want to really open source it. You've got this other thing that we also want to use. So how about we just put the two together and call it a cloud computing platform and we can work together. And they were up for it. So that was pretty cool. So that was mid June, I think. And then stuff started moving really fast. I think in early July we sent out an invitation to a bunch of different companies that we thought might be interested in using cloud computing, either service provider or in-house or whatever sort of way they might be using it. So we invited them to an inaugural design summit in Austin, Texas. And we were completely surprised that I think there were like 200 people that turned up and they didn't really know what was all about. The link to NASA wasn't no one really knew about that yet at that time. So all they knew was a rack space is doing this open source cloud thing like that because details were really scarce. Nevertheless, 200 people showed up from about 50 different companies and even though we didn't have much concrete to show them, of course they could follow along the discussions, they could see that people knew what they were talking about and there was this promise of what we're going to do, what the intents were. And based on that, there were like 30 or 40 companies that agreed to have their names put on the website and go out and be part of the big announcement. So the big announcement was at midnight between Sunday and Monday as Oskon was about to start. And I remember sitting in the hotel at Oskon and Twitter just exploded with all this stuff about OpenStack and I was just this was pretty insane. But it didn't really, it took another 36 hours before I realized how big this was going to be. I remember clearly sitting in the audience at Oskon and someone tweeted about a job advert that had been put out from a company that wasn't there last week at the summit. So this was someone who had only heard about it when the announcement came out and they were looking for OpenStack developers. I was like, okay, people clearly wanted this to succeed, everyone did. So that was rather spectacular. So the mission that we set out to engage on is to produce the ubiquitous open source cloud computing platform that will meet the needs of public and private cloud providers regardless of size but being simple to implement and massively scalable. That's a lot of words. Basically what we want to do is be, hopefully, be the apache of the cloud. So apaches, if you don't have a good reason to do anything else, most people just go with apache. Same thing for cloud computing. I kind of hope that we're going to have so many people behind and it's going to be like really just the standard. I mean, this is what you're going to do unless you have a good reason not to. We had these four open, the four opens of OpenStack that we talked about at the summit and I think this is perhaps somewhat surprisingly, this is one of the things that made some of the companies really come on with it. These are companies that might not usually be associated with open source software but the fact that we were doing it so openly made them feel reassured that we weren't going to do anything that they wouldn't be seeing. So the fact that we were aggressively open source about it made them like it even more and that was a bit surprising but it makes sense when you think about it. So this open source, it's all open source, it's apache licensed, there is no enterprise addition that's going to be sold for money afterwards. It's all open source. Open design, all the design decisions are done in the open. We have a design summit twice a year. The first one was the thing in Austin. We had one in San Antonio and in April we're going to have one in Santa Clara in California. Hopefully it's going to be Europe the next one after that. So open design, everyone can propose features. Everyone is free to work on features, more than free to work on features. And give input on how we're going to do this and that and solve different problems. Open development, you can do open source in many ways. I suppose it counts as open source if you just every six months you publish your table. So you sit in house, you develop a bunch of stuff and once every six months you throw a table over the fence and then it's open source. That doesn't really count in our book. So everything is open, the source code repository is free to everyone to look at and fiddle with all the time. There's no secret source anywhere. And open community, we try very hard to be very open about everything. I mean, old habits die hard. I'm sure that there's a lot of people working on OpenStack and Rackspace sitting in a small cubicle. I'm sure there's some of them that still talk to each other, even though we're supposed to speak on ISE. Nevertheless, for the most part everything is on ISE, it's on the mailing lists. There's certainly no bad intentions if something happens to not make it out in the open. And we have weekly ISE meetings where everyone's free to turn up. So we're really trying to do this the right way. As open as possible, as free software as possible. And if there's anything you think we're doing that looks kind of counter to that, it's not intentional, please tell us about it. We want to fix it. We want to do it the right way. Okay. So let's get technical. Unless someone has questions about all this history sort of stuff. We can wait. We'll wait. Right. So there are these two major components to OpenStack. The first one is... One of them. The one we're going to be speaking about first is Swift. OpenStack Storage, which is code named Swift. Swift is remarkably simple. Not simple as in primitive or as in crude or anything like that. It's simple in the way that sometimes when you're looking at a reasonably complex problem, sometimes you will have an epiphany and a solution reveals itself to be really obvious, really simple and really correct at the same time. And to me, that's kind of what Swift is. So traditional wisdom and in fact the old Rackspace Cloud Platform as far as I know, traditional wisdom when you're dealing with large scale systems is to... If you have some shared state, if you have some state that needs to share between all these components, one of the common things to do is to have a split your reads and writes and then have a bunch of database slaves that you can ask for stuff and then if you need to update data, then you can talk to a master over here that has a failover thing and all this stuff. Or you can use a gossip-like thing or you can use different kind of lookup protocols. Swift doesn't do anything of that. So if you want to store an object... There we go. So if you want to store an object, so you make this API call, it's HTTP, RESTY and all that. Normally, you'd think that, well, it'll probably look up in a database. Do I already have an object like this? Oh, I do. Okay. So it's on these three servers. I will just overwrite those. If it's a new object, I will just come up with three new servers to put it on. Everything's replicated on three distinct servers. You'd think that's what it did, but it turns out it's not. So instead, Swift does it all by looking at the stuff that you provide it. So the API version is... That's a given. It's not really very interesting. So you give it these three pieces of information. That's the account name, container and object. So I might want to store something. So the account name could be soren. Container could be where as an object could be windows7.iso. And what happens is that Swift looks at that stitches these three elements together and calculates the MD5 sum of it. Or some kind of a checksum anyway. I believe it's MD5. And then it takes... And then the first four bytes uniquely defines which three servers on the back end are going to hold this particular object. So it doesn't have to look anything up anywhere. When you set up the whole cluster, you create something that's called the ring, which is the mapping between this stuff and what backend server stuff goes on. So it's part of the configuration. It's not part of a shared state that needs to be updated ever. Unless you, of course, scale out. But that's handled pretty well as well. So without looking anything up on a data store or anything like that, it doesn't broadcast for anything. It just knows straight off the bat. If you give it this information, if you have this object, it knows exactly where to find it afterwards. It knows where to put it when you're putting it. And it's the exact same thing for... Right, that maps to a particular set of object servers. And it's the exact same thing for posts, which you use to change the metadata for the object. And the same for get and delete. So when you get something, it knows which three servers it can find it on and just... It does its thing and sends it back to you. And if it needs to delete it, it goes into the servers, removes the files, and then it puts a tombstone file there instead. Because if... The way it deals with petitions and such is that it will... If a file is missing on one of the servers that it's supposed to be on, then it uses rsync to synchronize stuff around. And then... In order to make sure that stuff that gets deleted actually gets deleted and it doesn't think, this went missing, I better recreate it. So we put this tombstone in there to make it see that it's actually supposed to be gone. So some of you might be clever enough to think, well, if we were using hashing and all this, that makes it really difficult to list stuff, doesn't it? That's going to be really expensive. You have to look all over the place to see where you have your stuff. Well, no, this is where it gets really clever. So... When you want to find which objects you have in your... in your wares container, you do get... The API version is not that important. Account name, the Sorin container that's wares. And what happens is that it takes these two elements instead, applies the same logic, it takes the MD5 sum, and that maps uniquely to three container servers. Container servers are servers that does the same basic thing as the object servers that hold the actual data. But what they store instead is an SQLite database that has a list of the objects that this particular user owns, or is in this particular container, I mean. So it uses the same mechanism to actually distribute the state all over the place, and the same mechanism to very quickly be able to find it. And the exact same thing happens for... Whoops. If you want to look, find the list of containers, then it takes just your account name, applies an MD5 hashing, and uses the result of that to figure out exactly which servers hold the SQLite database that tells you which containers that this particular user have. It's fantastic. It scales absolutely brilliantly. There's no... There's really no limit to how many front-end servers you can have. They all... It's basically a... completely flat scaling... Anyways. I love Swift. It's fantastic. It's kind of tricky to set up on if you only... It's designed to... and it's only really run in production environments in a particular way, where you always have three replicas of everything everywhere, and you know that it's distributed across different zones, so that it geographically split apart and all that. So if you just want to run it as a test sort of thing, if you don't care about replicas, you just want a single copy, that's the same sort of API, that's still kind of tricky, but setting it up in production environment is very well documented and all that. So if you need anything like this, go for it. It should be very approachable. So what I spent most of my day working on, though, is Nova. Nova is also a celestial body, an astronomical object, and it also means new in Latin, and some people claim that it stands for no other viable alternative. I'm not sure if that's actually true, but I think it's a good joke, given that they placed eucalyptus and all that. Yeah, as I said, they have a way with names. Right. So Nova is ever so slightly more complex than Swift. We haven't yet had the same epiphany that can reduce the whole thing to 50 lines of code. I'm not holding my breath on that one, but you never know. So it's rather more complex. Basically, this is not very complete. I tried my own graphical artist skills. I'm not very good. So I spend a lot of time trying to find a good diagram on the internet that really explains and encompasses the whole thing about how it does stuff. But the problem is, as soon as anyone tries, another two days pass, and then we refactor something, and then it's slightly different, and there's so many different topologies that are completely valid and just different ways that things can be interconnected, that it's really hard to have a diagram that has any reasonable amount of detail while still being even remotely correct for more than a very short while, if at all. So this is delightfully simple. So there's very few things here that can go wrong. So only a couple of things are really wrong with this. We'll never notice. Right, so at the center, we have, as I mentioned before, we have a message queue because that's really the basis of all the communication inside of Nova. So we have API servers, which are basically the servers that you speak to. If you want to run a virtual machine, you speak to the API server. If you want to kill a virtual machine, that's the server you talk to. If you want to get the infrastructure to do anything at all for you, you speak to the API server. There's a scheduler. There are compute workers, and network controllers, and volume workers, and there's a very, very crude object store that we inherited from the old Nova from the NASA guys. It implements the S3 interface because they were still doing the Amazon APIs back then. I'm perfectly happy saying that it's very, very crude, even though I wrote most of the current implementation. It's really not to be trusted. We hope to replace that entirely with Swift or Glance, which is an image registry and cache and stuff. Let's just ignore that for now and focus on what we have here. So what happens is that if you send an API request to, for instance, run a virtual machine, that happens over HTTP. You send it to the API server. The API server looks at it and validates it, making sure that you are actually a valid user. You exist. The image that you want to run exists. The size and type of virtual machine you're requesting is valid and exists, and your quota has not been exceeded. As soon as it's happy with these conditions, it basically sends a message to the message skew to please run this, and then it tells you to, all right, we've got it covered. Just move along. We'll take it from here. Eventually, the scheduler, or one of the schedulers, that's one of the errors in this diagram. There can be a crap load of them. Pretty much everything can just scale horizontally. You can add as many as you please. So one of the schedulers will eventually pick up this request. Usually it's a few microseconds, but it's a queue. It could take a while, theoretically. So the scheduler looks at the request, and based on the knowledge it has of the various compute workers, which are the actual servers that run virtual machines, it decides which one should be the lucky recipient of this particular virtual machine. So as soon as it's made the decision, it sends a message through the message queue to the compute worker, telling it, hey, you need to run this machine. It's this particular image. You need to fetch, and it has to be of this size, and so on and so on. And then the scheduler is done with it. The compute worker asks the network controller for an IP, and depending on the network topology you're using, it might tell the network controller to set up some routing for it, because it might act as a gateway, depending on various things. The network controller also runs DHCP server that responds to DHCP requests from the virtual machines. And then after the virtual machine is open running, you can add extra storage to it, which is located on the volume workers that might have a great big sand connected to it, or you can use Sheepdog, you can use Cep, you can use iSCSI, you can use ATA or Ethernet. There's various ways you can do this. And then you basically connect that that block device to the virtual machine. So some of the things we do to make sure that this scales is... For instance, we run the firewalls directly on the compute worker so that we keep that very close to the virtual machines instead of having to rely on a great big firewall that can handle the entire data center. Because one of the design goals we have is we want to be able to support a million physical hosts. That's the number we just throw around saying. If someone suggests something that doesn't seem to really scale well, we always say, well, this is not going to work at a million hosts. It's really hard to relate to a million hosts or an infinite amount of hosts. We need to distribute stuff as much as possible. We're doing reasonably well on that. There are still a few centralized things. One of them is the data store, actually. The data store is anything that speaks SQL, usually. Anything that has an SQL Alchemy driver. Commonly it's a MySQL server somewhere that... We're going to fix that either in the next release after that where we're going to be distributing the data store much more so that the information is much closer to where it really belongs. Information about virtual machines will live on the compute workers so that they will be authoritative for the information about stuff that runs on them. The network controllers will be authoritative for network information and so on and so on. Then we'll use a caching mechanism to make sure that everything else doesn't have to ask every single network server, for instance, if they need to have a global view of networks and stuff. LDAP is not necessary. You can choose to use LDAP. If you do, well, that's another central thing. That's something that we're trying to solve. What else is there to say? A lot of these things are very modular. As I said, you can have a lot of different types of storage attached so you can use Sheepdog and stuff and all this. We support a bunch of different hypervisors. We support KVM and SEN and user-mode Linux through Libvert. We also support SEN server, Hyper-V. I think there's another one. Apart from the EC2 API, which we inherited from the NASA guys, we've added something called the OpenStack API. We have our own API, which is based off of the RecSpace Cloud Service API, because that's really the only way that we can ever innovate if we're stuck following whatever Amazon does, even though how bad it is and how good it is. We don't want to be tied into whatever they want to do. We want to be able to innovate as well. The EC2 API isn't open. They've never really said that it's okay to implement it for anyone else. But they haven't complained either. Just not being based on an open API and something that we don't control ourselves is just not a cool position to be in as a free software project. That's the API that we really want to take over the world. Eventually. Any questions? Is that a hand or is that just a... What are the auto-scaling things? How do you manage to monitor different APIs? The question is about auto-scaling, how you manage that. Are you thinking from a consumer of this or as a person running the cloud? Right. The thing is when you're running a cloud, you're in the very unfortunate position that you do not get to run on the cloud because you don't get the magic of the scaling that a cloud gives you because you're the one who is supposed to... You're the only one who actually still has to go and buy a new hardware and put it in a rack and plug it in and get it all hooked up to this whole automated infrastructure thing. It's really very annoying. I used to work on virtualization when I worked for Canonical. I sit there and I buy all this virtualization stuff and I can't get to use it myself because it has to run on physical hardware. It's really extremely annoying. That's the same thing here. It's really not very easy to automatically scale physical hardware. That's really the point. From a user's perspective, if you're running stuff in the cloud, there's a number of different ways that you can do auto-scaling. It really depends on the sort of information that you can get out of the infrastructure. Amazon, for instance, they have this thing called CloudWatch which gives you a whole mass of information about a lot of the things that you're running. But really, it doesn't have to be that complicated. There are a number of different projects that can, for instance, install an agent inside all the virtual machines that will keep an eye out for if your request times are growing beyond a certain threshold, they can start up a new web server, a new front-end server, and if you're... I mean, they can respond to different things because the whole point is that you can... adding more resources is just one API call away. It's so simple. So you just need something that can respond to that and then perhaps scale back down again when your load spike is over. Does that answer your question? Yes. Cool. Anyone else? Yes. Perhaps a strange question, but how does OpenStack compare to OpenCura? I'm not sure if you know that. The question is how does OpenStack compare to OpenQRM? I don't know. I don't know OpenQRM. Sorry. Yes. He also explored things. He said he uses the investment parameters to construct a hash, and then he uses the first and provides the models in the set of servers. So the moment you stay out of the server, how do you put it back on? You know, what was used before, how do you put it back on? Right. Okay, so the question is when you are using this whole thing to keep track of stuff and to figure out where the stuff go and where can you find stuff again when you need to go look for it. What happens when you scale out, you add more servers because then this mapping isn't going to match anymore? Well, the process of changing this ring, as they call it, which is this mapping, you use a specific tool to do that. It's not just a flat configuration file. It is basically compiled based on some input. So when you add another server to your storage pool, it automatically rebalances stuff so that it moves some stuff around at a specific rate. It knows that this is where stuff used to be, this is where it's going to be, and it knows the rate at which stuff actually moves. There are simply tools that take care of moving it and changing the mapping in a little bit at a time. Yes? Is there a way you're going to implement, or is there a way to tweak the replication load? Sorry, what? Is there a way to tweak the replication load? Let's say I have this thing for you, Joe, and I'm going to take the file back inside. You cannot right now at least do it on a per-object level. It's a per-deployment level. It's only really been tested thoroughly with a replication level of three. That's what Rackspace does. Everything is copied in three places. Someone else is offering the same, using cloud files to offer the same sort of service, and I imagine they're doing the same. We don't really know. That's the only thing that's really been properly tested. I really hope that someone will eventually make it much easier to run with a replication of level one. It's configurable, but it's only really been tested with three. Yes? The question is whether I can say a little bit more about the network topologies. That doesn't really help anyway. There are basically three, well, two-and-a-half different topologies that you can use. One of them, the way that NASA does it, is they have... you add a great big private subnet to your configuration, like, for instance, 10.0.0.0.0.8, and then you say that a project has maybe a slash 20 each. And then you use VLAN to basically keep them separated, and you use VPN to connect to them from the outside, or you can use floating IPs to actually assign an external IP to a particular internal IP. So, yeah, that model uses VLANs a lot, but it also... this keeps changing, so it's really hard for me to remember what it's like right now. I think that's the model where it uses the network controller as sort of a gateway, so everything has to pass through that to get passed to the right places. There's a more flat model that doesn't use VLANs or anything. It just basically attaches virtual machines to a bridge that is then connected to the physical network, and then we do all the filtering locally on the compute workers, making sure that it doesn't spoof ARP or Mac or IPs or anything like that. But that doesn't use VLAN. Yeah, that's basically the two major models that there are. Yes? Well, that's not a problem, because it only defines which server it ends on. It doesn't define... it doesn't actually overwrite any data. It's just... I mean, because we're only really using the first four bytes anyway, there are collisions all the time. It only defines where does it go. It doesn't define are you going to be able to overwrite stuff that is already on the same server. Yes? What are your sole passports? Let's say I have this data storage unit which I exit with 10,000 machines all at the same time. How do you replicate it? Or how do you solve it? We don't know if we replicate it. We don't know if we replicate it. We don't know if we replicate it. We don't know if we replicate it. You don't really... I guess... What you would usually do if you... I mean, if you actually want to get this data very often, what you can do is you can use the CDN integration so that you actually get it distributed all over the internet. They... Rackspace Cloud files is integrated with Akamai. So this blob of data would just end up all over the place and you can fetch it from there instead, which is much faster. Yes? How many hours will it take from deploying to the machine, branding machines and robots to having a working cloud? Well, that depends. I mean, if you... Sorry, yeah. How long does it take to go from... You have a bunch of machines until you have a working cloud? That was the question? Right. Well, I can say that for a single machine, I can do it in... I have a Hudson instance that does it on three different machines and it can completely blow away the whole thing and install it again in less than a minute. It gets more complicated as you grow out, as you scale out because you need to make sure that everything is in sync before you... you press go. But it should be... It should be rather simple. As soon as you have a MySQL server set up and a RapidMQ server somewhere and you point everything at the same stuff, it just... just works, sort of. Mostly, usually, sometimes. Yes? What goes wrong if you... Oh, dear. Uh... I'd really rather not go into that when they're not here to defend themselves. I wouldn't feel comfortable... No. Sorry. Oh, the question was, what's wrong with your eucalyptus? Right, just one last thing. I have a crap load of t-shirts. Please take them because I'm not going to take them back home. A bureau, not for us. All right. Hang on, hang on, hang on, hang on. So these are...