 Building Scalable OpenSimGrid Services with Simeon. Mick Bowman is a principal engineer in Intel Labs and leads the Virtual World Infrastructure Research Project. His team develops technologies that enable order of magnitude, scalability improvements in virtual environments, opening the door to new levels of immersiveness and interaction among players. Welcome, Mick Bowman. Thanks, Dan. Just to sound check everyone here, okay. So, several years ago, we had just recently hired John Harleman, and we're starting to look at some new ways for doing authentication and identity and how to simplify access to virtual worlds. And in particular, the question was really, what does it take to be able to seamlessly move back and forth from a site like Facebook or something in your Google to a 3D space? And one of the things that we wanted to avoid for the most part was having to relaunch viewers and re-login and all those other things. So we started this discussion about how to do identity, how do I add new kinds of authentication to the OpenSim process. And we realized very quickly that the service architecture that existed in OpenSim was going to be very, very painful to work with for doing that kind of extensibility and research, the development cycles and other things were going to be very difficult for it. And so John and Jim Radford from the Open Metaverse Foundation went off and started building a set of replacement services for the core components of OpenSim called Simian. And that was sort of the genesis of the ideas. So let me give you just a quick overview of what Simian is. First off, it's not a replacement for the simulators, it's a replacement for the robust services and that includes things like logins and access inventory and a number of other things we'll go through in a couple of minutes here. It's implemented as a collection of web services using standard web technologies. So this is sort of based on a number of observations. If you look at the way the simulator works, the simulators are very much about 3D spaces. But the set of core services on the back end are really just data. It's really just transactional updates to collections of information that are data. And we already know how to do that, right? The people who've been architecting and building web services have been doing it for years. So we just came at this from a, if we have to start working on this, let's work on something where we already have a very mature, very reliable, well-tested set of technologies on which we can build. And so we moved from the kind of, not really proprietary, but certainly kind of tightly bound and constrained and closed robust implementations to a set of very open, well-maintained, well-managed, well-architected web services as the core foundation for it. The third thing at Simion is a collection of apps and services for managing and optimizing. So we'll talk a little bit about the architecture in a minute. There's a set of kind of basic transactional services. And then there's a bunch of other things that are sitting on top of it. Two that we've added recently that I'll spend a little more time talking about later are kind of offload handlers for textures and inventory. And what that means is that we're moving all texture and inventory capability handling out of the simulator and onto Apache-based web services. And there's a very significant performance improvement for the user who's retrieving the inventory in those cases. And also, because the simulator no longer has to worry about handling those sort of kind of raw data transfers, it can focus more on the simulation so we get better performance on the simulator as well. And then the final thing is, because of the way we have, again, because of the way we've kind of tiered the architecture here, it's really easy for us to actually provide libraries in a number of different languages for building and writing clients. And so there's a set of core services which define an API. There's a library in PHP for accessing that API. And all of our sort of extensible services like the hypergrid service and the login service and the texture redirection service all take advantage of that PHP library. So there is one place where we have to manage and take care of the API. Similarly, we've got a set of pro libraries that we use in order to write clients that allow us to do everything from asset manipulation and garbage collection on assets to inventory management. And one of the things that we do is we can archive, offload all of your inventory using the direct access to the simulator that way. And finally, one thing I'll say on this is that there is a relatively immature, but as good as anything you'll get an existing OpenSim capability model, a security model for managing and securing the connection between the simulators, the users and all of the clients and the core services. So what you get by using Simian Overrobust and again what you pay for is you've got another software package that has to be managed and configured. The configuration isn't that difficult once it's in place and for the most part it's been extremely stable. It's been more or less the same configuration approach for the last two and a half or three years that we've been running on our grids for it. But it is a separate package that has to be installed. But what you get out of it is a set of fairly stock approaches for doing performance, performance management, stability management. We can do load balancing using any of the existing Apache front-end load balancers. You can do hot swap and failover and other things like that in a relatively straightforward way. The three-tier architecture that we have does allow you to do a lot with extensibility. What we'll talk about later in the talk today is how to do performance optimization and scalability. How do you tune your deployment for the kinds of scalability requirements that you have and how do you get what you want for your users that way? And like I said, the final thing is that it does provide this authenticated access. And while we are still evolving the security models and the capability models that we have for doing security, what we have right now is good enough to be very flexible for expressing things like and creating things like these offload services for allowing users to have direct authenticated access to their inventory if the grid owners choose to provide it. So there's an awful lot of flexibility there in that model. So let's just sort of review for a second what we mean when we talk about these kind of robust services, the core services. We're kind of all familiar with the simulator. The simulator principle role is to manage space. And it manages most of the connection to the viewer. It's out there. The simulator does that right now by relying on a set of core services. So some of the central services give it some information about users, information about presence and where the user is currently logged in, provides services for doing logins and hypergrids and managing the transactions that go back and forth. Simulators for the most part in the way they're architected today rely on an external asset server, although most of the connections to the asset server are required principally to manage inventory and access to inventory. There's also a set of services for managing maps and how you register regions and how you find regions. And then there's a bunch of social services like friends and groups and other things like that that are part of it. The robust architecture is a huge improvement over where we started from. Christa can give you the dates on when she and Melanie went through and did the entire connector architecture. But it's a huge improvement over where we were before with a bunch of different invocation methods for accessing those services. But even now, the connection between the simulator and robust is mostly forms encoder, query encoded, key value pairs that are layered over HTTP with XML results coming back. But to call it an API would be something of a stretch. The robust services that exist there are all C-sharp implementation that are compiled into kind of custom server code. So there are ways and we've been working on robust plugins in order to add functions to it. But for the most part, you're going to have to go write C-sharp. Your development model is going to require you to shut down the server, add things, bring them back up again. And there's really no enforcement of any consistencies in the APIs. And so you have things like the login service, which has a particular viewer-specific protocol sitting side by side with the asset service, which is one of these HTTP-layered applications. Simeon has the same high-level boxes, right? The simulator continues to access this core set of services that's on the back end, and that's managed back through the set of database services. And the invocation of the services in Simeon, that is the connection between the simulator and Simeon, is fairly consistent with one exception, which we've actually removed that exception. Assets were being transferred in a particular way, but that's now even being embedded inside the single API call. But for the most part, it's JSON-RPC that's layered over the top of HTTP. There's a single set of API functions that moves things around. So for example, extending this to do all invocations via SSL is a relatively straightforward modification to the code and one that Doug Maxwell and Moses has asked for in order to be able to secure the communication between the simulator and Simeon. One of the other advantages of that is that when we wanted to get authenticated access from the simulators to the Simeon services in order to provide us a greater flexibility in being able to authenticate foreign simulators and then revoke their privileges to connect to a grid in the future, we simply added a set of capabilities into the RPC layer, and it was more or less, once all the plumbing was taken care of, it was more or less a couple of lines of code in order to pull that out. So the advantage of having this consistent protocol and the tiering architecture that I talk about in a minute is it really makes kind of having a well-defined protocol much easier to implement. On the server side, all of the Simeon services are implemented as PHP scripts that are executed in Apache server. As a development model, that has a lot of advantages. It doesn't require us to go in and take down services and add them back up. We can modify live and running systems and we frequently do that in order to make things continue to work. And of course, you know, we have a couple of back-end plugins for the database, but for the most part I think Moses is running with a file system-based asset store using the extensions that OS Grid got, but for the most part the rest of us are using MySQL, we're actually running it. There is old code for MongoDB, but I'm fairly certain it doesn't work anymore. So let's blow up that box then and take a look at what's inside. The set of Simeon services. It's very much designed with the intent of, again, taking advantage of everything we've learned over the last, what, almost 20 years about how to do web services. And more than that, 30 years of how to do robust and reliable services. And using this kind of three-tier architecture where we have databases on the back end. And again, you know, with database technology it's relatively easy to take stock systems and do things like create replicated databases with hot failovers. We understand how to do that. That's just sort of, you know, kind of standard IT management of the data layer. So we really didn't want to worry too much about that. The second layer, the second tier is that core service API, which is where really the logic comes in. Where the database transactions are converted into column 3D web or 3D application platform transactions. So the APIs that are exported by the core services or things like add a node to inventory, store an asset, save an asset, create a session, update the information that's associated with that session. And again, I'll talk about how we do access controls and some of the other things that are on that. So the front end then, the third tier is really the kind of presentation tier. And that's the thing that does all of the high-level transactions. And so each one of those services, the simulator, the login, and the others that are exporting, a very specific API out to things like scripts, the viewer web browsers and other things like that. And so our inventory access, our inventory offload is actually implemented, exporting the API that the viewer expects for accessing inventory, but it turns around and converts those calls into the core service API calls. Login similarly, and the simulator does that as well. Okay, so the simian components, simian again, based on those three tiers is really kind of three sets of packages. The one is the kind of core grid services that we have access to, and I'll talk about all of these in a second. Then there's that kind of set of services and applications that constitutes the third tier application. So it's grid login, which provides both grid and hyper grid services. And we're able to, for the most part, hyper grid back and forth. We don't have the inventory connection, the inventory limitations that Krista has in the robust implementation yet moved over. And we'll probably not. We'll probably do it based on our capability model, which will give us a lot more flexibility in what actually gets exposed that way. The grid public services are things like the Texture, Inventory, and Map offload. These are viewer capabilities, second life viewer capability handlers that allow us to move functionality out of the simulator and off into web-based services. And so one of the things we found, for example, when we just the moving of textures off of the simulator and onto an Apache server allowed us a huge amount of performance improvement in accessing the textures. And it was relatively straightforward. There's a little bit of quirkiness to the SL viewer protocols and what they're expecting with the web request, but it's really just web requests. Inventory has been a little more challenging because there are a number of expectations in the transaction which are not always effectively documented. And so that's been a much more of an adventure. And maps again, we were talking earlier about some of these things that we've moved essentially all of the map access into the central service, which the viewers can be configured to access on login. Grid front end then is the web grid manager and frankly others have worked on that. Jim and John did most of the implementation of that and it really has not been touched. And it really should be treated as an example for what you can do and what works out of the box as opposed to the way that you need to for it. And finally, the web dev access was really an experiment in how to do something that's completely different. So if you're not familiar with web dev, what it allows you to do is to essentially mount a network file system. So you can take your inventory, mount it as a file system on your Windows box or your Mac box or anything that supports a web dev protocol and interact with it through a file system interface, which if you've ever gone through and tried to do inventory management and other things and clean stuff up and archive other folders using the SL viewer interfaces, you know how painful that can be. And some of the file system access where you can write scripts and other things on top of it makes life a whole lot easier. So the web dev code is experimental and again it hasn't really been maintained much, but it was a demonstration of what we can do with this three-tier architecture to allow us to get access to things. And then the final set of packages are these things, the common package is a set of libraries that export the Simeon-PHP interface and it's the set of libraries that all of our clients are through a tier applications are built on. And then in the contrib directory we have a bunch of the Perl libraries and the Perl clients that are doing that and the Perl clients allow you to do everything from user and scene administration to inventory administration as well. So let's explore a little bit more of each of those packages and then we'll move on and talk a little bit about the scalability parts of it. So when you look at the grid services, it's really three layers inside there. There's the data abstraction layer at the bottom, which provides a consistent interface to the backend database, whether it's MySQL or MongoDB or file systems. There's a set of handlers and I'm calling these out specifically because we'll come back to how these can be partitioned, but each class of handlers is independent of the other classes, which allows us to manage them separately. The scene handlers are all unique to a particular data architecture and those data architectures are independent from, for example, the capability handlers or the inventory handlers. The dispatch layer, there's a single layer which handles all of the incoming requests, unpacks those requests and does the capability validation. And again, because it's really easy, especially in dynamic systems to expose too much, we wanted to have a single dispatcher that did all of the validation on the requests. And so everything that comes in has the capability validated in a single location and then turns around and hands it off. That gives us a lot of confidence in our ability to write things that are secure systems and that are not exposing too much additional information on things. In addition, because there's the API level securing of invocations. So for example, simulators have a certain set of APIs that they can call. Users, if you're authenticated as a user, you can manage your inventory, but you can't update anybody else and you can't change scenes. So there's a certain level of API security that's provided through the dispatch layer and the capability system that we have. In addition, we have all of the sort of stock methods for securing Apache services. So, you know, HD access files that deny certain IP addresses just works. Firewalls and firewall configuration just works in order to be able to access it. And so that gives us at least two different layers of maintaining the integrity of the core services that are there. The grid public services, which are really the third tier services include things, and I've mentioned this before, like login, which is the grid and the hyper grid login. And our login service does allow a variety of forms of authentication to be provided to it. Password based, which is the traditional viewer based. And we also have ways of doing open ID and Facebook ID that are actually based on our capability system. So you authenticate, create a capability and then turn around and provide that capability to the viewer. Hands that into the login URI that's given to our login service and that does the validation of the capability. So that allows us to extend the kind of core authentication service, which was the original motivating factor for a lot of what we're doing was to explore that. And then we have the texture offload and the inventory offload that I've already mentioned a couple of times, which are really handlers for the basic capabilities. That the viewer expects a grid to export. And then there's things like the map server, which handles the grid maps at the multiple levels of visibility. So then let me just give you some of the details here about why and how the inventory offload works and how the security system in Simeon kind of integrates with it. So right now when you log in, the login service creates a session for you on the simulator, the simulator turns around, the viewer then connects to the simulator and attaches itself to that session. The simulator then hands back a set of capabilities, a set of URLs for where the viewer should go for different functions. So one of those that's currently handed back this simulator says, get your textures here. And so it's a URL that the viewer can post requests to that the simulator will handle and respond with textures. And that's how the HTTP textures work. Similarly with inventory, there is a URL that's provided to the viewer. And the viewer, when it's doing HTTP based inventory, posts requests to that URL and the simulator handles it. And in theory, the simulator is trusted. And because of the structure of the URL, there's a shared secret which is encoded in the URL that maintains the integrity of the connection between the viewer and the simulator. So you can't have someone coming in and posting requests to your URL. And accessing your inventory through the simulator. In theory, that's how it's supposed to work. It's not quite that secure, but it works. So when we do the inventory offload, the real question is how do we make sure that the separate service also has the same user identity that it can't be spoofed that way? So the protocol that we've implemented, we use a similar protocol for the textures as well. The viewer requests the inventory capability in the simulator. The way it typically does, the simulator would normally respond with a URL that points the viewer to the simulator. In this case, in the Simian case, the simulator invokes an operation back on the Simian core services to create a capability. It's a security token for that particular user and it registers that with the core services. That security token is then embedded in the URL that's handed back to the viewer. That URL redirects the viewer to the inventory offload server. The viewer provides that security token to the inventory offload server. The offload server goes back to the core services and validates that that token in fact corresponds to that client IP address. So we have the ability in that way to use this capability system that's in Simian to store these shared secrets and pass those secrets back and forth in order to get the inventory offload. So that's how we ensure that the user who owns the inventory is the only one who's able to access it through this separate offload service. And then the final part that I mentioned as far as the components are the grid service APIs. And I've already mentioned that we've got client and pro libraries for it. The PHP library is principally used for the web services that we have. The pro library is really, it's incredibly convenient for doing service management. So we've had discussions in the various lists about how to do IAR backups, for example, for all of the users on the grid. And what you would need in order to implement IAR sort of out of the box is pretty challenging right now. So you'd have to pull in essentially a simulator because that's where the APIs are actually implemented for all of the components for it and invoke the inventory archive function for the collection of users that you have in your system. So in order to do that kind of backup, you literally bring up a simulator, write a region model for the simulator and dump all the archives out if that's the style of backup that you want to do. That's kind of painful. So the Pearl APIs allow us to do some very similar kinds of things, but it's essentially all command line driven. And again, it has all the capability access and the other things that make life easy for us that way. So administrative functions and functions for managing the grid are relatively straightforward for us to implement in these command line tools. And by the way, if you feel, you know, if you really like web front ends for things, you know, the PHP scripts are there that allow you to do all those management functions through the web front end as well. So, all right. So I want to change from a little bit about kind of what Simeon is to how it gets deployed in the grid and a little bit about the kind of scalability aspects of it. So the first assumption is that, you know, you have a patchy installed. That's one of the basics and PHP and all the JSON libraries that are necessary to it. The typical deployment really involves kind of two pieces. And these can be, you know, I have them split out here into two separate kind of services. It's relatively easy to run them on a single server and install everything on a single server. What you really need for, and I say private in the sense that it's the set of services that your simulators and public services have to access, but your clients, the viewers do not have to access, is really just the set of grid services that are available to you. And then you install the public server, which is the grid front end, grid public and grid login. Those are the three packages that are usually installed in a public location. Again, because of the way a patchy works, it's relatively easy to install the four high level directories and then drop an hd access file into your grid directory, which limits access just to your simulators and to your existing HTTP server. And that's the typical installation that we're running with Simian is one server, the five basic directories that we have, the library directory and the four service directories, with the grid directory protected by hd access files limiting access strictly to the simulators. And that does give us multiple levels of access that we can have. Alright, so that's the simple one. And like I said, what we're interested in, and what motivated us originally was the identity, but a lot of why we have stuck with Simian is because it does give us, well, because it gives us a lot of flexibility of being able to manage our grid, but also because it gives us the ability to do very scalable deployments. So because the services in the service installations are all stateless, it's relatively straightforward to replicate those in multiple locations. And so if you want scalability in your core services, you simply start up multiple servers on the back end. If you want scalability in the public services, which is actually a thing that we're finding much more relevant here, then you can simply make copies of those of the grid public directories. And before we leave that, let me talk a little bit specifically about that. So we were talking earlier about Klaus's talk and some of the struggles that he was having. And in one of the staff channels, I think Justin mentioned the fact that he's coming in from Hong Kong, and that's part of the reason. And we've got latency issues and other things like that. And certainly for things like simulator updates. But if you looked at the rates that we were seeing for updates, they were relatively low in the order of 15 to 20 kilobits a second coming out of the keynote regions for that. But there's an awful lot of data that has to be transferred and that data comes in through for inventory and texture assets and other things like that. The advantage of being able to replicate those grid public directories is that in a sense you can create a content distribution network using those public services. So for example, the texture offload does caching of the actual texture data. So it'd be relatively straightforward to create a grid public directory installation in Asia. And that would mean that you're transferring textures, the long haul back to Irvine in order to grab those textures off the UCI server that's hosting the conference. You only have to do it one time. Everything else is doing a local pull on it. So you get all of the advantages, performance advantages of content distribution networks. And all you really have to do is just install an Apache server in one package on top of it. And the rest of it is handled by the statelessness of the interactions between the public services and the core services. So we rarely see need for replication of the core services. There are some times, but we rarely see need for the replication of the core services. What we see a lot of use in is replication of those public services in order to give us that geographical distribution in order to make the sort of high bandwidth interactions as short as possible. Not use the internet connections with the internet latencies issues that are there. So the real win in having these things as web services is that flexibility for the deployment is a big thing that allows us to do these local accesses to things. But we've said several times that we just get all of the work that everyone has done on making web access fast and robust and reliable and responsive. And we get it for free. The Apache server performance-wise versus using a simulator, the HTTP server in the simulator is a huge win for us. Dali asked me that again at the end and I'll respond to it. But we get a huge win out of just using Apache, which has been heavily optimized for accesses. And it does one thing. It serves web pages and it serves them extremely well. But not only that, we get all of these technologies on the top for doing replication and load balancing. So we know how to do round-robin DNS. It just works. We know how to do load balancing across multiple web servers. You can go out and grab a number of packages that will provide off-the-shelf load balancing and sprayer technology for the requests that are coming in. All of that stuff is sort of known technology. It's relatively easy to just drop it in and it works. So that's really the win here with the web service-based approach is we leverage all of this web technology that is extremely mature in doing the performance optimizations on it. So the other thing, and I mentioned this a little bit in the HyperGrid panel the other day that I wanted to bring back, was just the other approach to doing the kind of scalability which is to start partitioning the services off themselves. And that's really disarticulating the services. So we've got the basic service has all of these pieces in it. There's no particular reason why we have to stick to that. The services again, because they're independent of one another, can be partitioned out in whatever way that we want. And the really nice thing about that is that we can then put them back together in different ways. So for example, we can have user capability, inventory, identity, and asset store in one service. And then everything that's actually responsible for scene management in a separate service. And there's no particular reason why two or three independent scene services can't exist. And so you can have a single set of user services that are supporting multiple maps, which allows you to have multiple independent grids, each one of which can be managed by itself without having all of the management of the overhead of dealing with users and communities and all of the other pieces that go along with that. So there's a set of core services that we can pull out and compose then into the grid services that we want. So it allows us a lot of flexibility for building fairly central, large populations of users and assets and inventories and still have very small, independent, isolated grids that are leveraging that. So just for summary then. Simulator is a set of web services. It's a replacement for the open some robust services. It's a bit of a challenge to install and get configured correctly. Well, if you've dealt with the simulator configuration files, maybe it's not quite so hard, relatively speaking. But it does take a little bit of work to get it right. The architecture has several advantages. We use very mature web technologies in order to build it. And as a result, we can leverage all of the performance and reliability techniques that we get from that. The three tier architecture really makes it a lot easier to add new services to update running systems. The development process is much easier. The deployment process is much easier. And there are real APIs that are encapsulated in sets of libraries that applications can take advantage of. The one thing I will point out is the capability model right now is every bit as expressive as what exists between simulators and the robust servers. And in fact, it's already more than that. But there's a lot of work that we're doing in order to provide us with fine-grained, much more fine-grained control, which would allow us to have different kinds of relationships between applications and the core services, which would allow us a lot more expressiveness, for example, in what can be exposed in our hypergrid interaction. So rather than having one suitcase folder, we have the ability to say, I trust that grid over there. I'm going to expose more, but I don't trust those guys over there, so we're going to expose less. And the capabilities that we can encode in the service URLs that we provide to each of those grids gives us that expressiveness. And that's really the kind of work that we're doing right now on it. So just to conclude then, the project has been hosted by the Open Metaverse Foundation for a number of years, although the development directory that most of our work is currently being done on and has not been pushed back up to OMF is on GitHub and you're more than welcome to grab the code out of there. All right, that's all. Questions? Okay, Dalia, you're going back to the self-replicating key value store packages. If you look at what is MongoDB, I mean, that's more or less what it is, is a structured document store. Mongo does provide some ability to do replication. What we have found is if you're doing local database replication, that's what they're really designed for and work well at. If you're doing long range, so if you're doing geographical distribution, database replication doesn't work, excuse me, doesn't work quite as well. The content distribution approaches seem to work much, much better in those approaches. So that's where we're going with that. All right, any other questions? Okay, Dan tells me I'm out of time. I'll be around if you'd like to chat later. Please get a hold of me. Thanks. Thank you, Mick Bowman for a terrific presentation. As a reminder to our audience, you can see what's coming up on the conference schedule at conference.opensimulator.org. In this room, the next session will be viewers and open simulator panel with Justin Clark Casey. Thank you again to our speaker and the audience. We'll be back shortly with the next session. Thanks.