 All right. Hello. How's everybody doing today? My name is Blake Yeager. I am the director of the director of next generation markets at Rackspace Which is basically a fancy way of saying I work on things that are not products yet And specifically what I've been working on for about the last six months or so is 0vm, which is a new open-source technology that Rackspace has invested heavily in most of the core contributors of the technology to work for Rackspace today But we're hoping that you know over the course of this week in the next few months We're going to start to change that and get more committers involved from outside of Rackspace as well I have cam. Well gilyadolf here. Who's the creator of 0vm? I'm going to invite cam up on stage later on when we do a little bit of Q&A at the end of this session But before then what we're going to do is basically give an overview of 0vm We're going to talk a little bit about some of the use cases that are enabled by 0vm And then specifically we're going to do a little bit of a deep dive into the integration between 0vm and OpenStack Swift And how you can kind of turn Swift from just a basic object storage solution into a converged compute and storage platform And I'm going to do a live demo. So I'm going to cross my fingers and hope that goes well Please be forgiving if if it does not So what is 0vm? You know the most the simplest way to describe 0vm is it creates a secure isolated Execution environment in which you can run a single application Process or really any arbitrary code and then you can take 0vm and embed it into storage systems Even multi-tenant storage systems such as Swift and you can allow users to process manipulate search Do any type of arbitrary computing on their data inside of the storage system without having to pull your data out of the storage system And load it either download it locally or pull it into a compute cluster or Hadoop cluster or something like that Some of the technical details behind 0vm, so it's based on a project called Google native client or knackle So knackle is a technology that Google originally created to allow them to allow users to safely run Native code inside the browser. It's embedded in in Chrome today And what we did was we took that native client technology We extracted it from the chromium project and we made it suitable to run server side and specifically to embed in storage systems Like we were talking about it leverages 0mq for networking although we're in the middle of kind of rewinding rewriting The way some of that networking works It includes a full compiler tool chain So there's a GCC tool chain that allows you to cross compile anything you've written in C even assembly And you wind up with an executable that you can execute inside of a 0vm instance And then we have what we call the 0vm runtime or ZRT So 0vm by itself basically provides you a bare x86 execution environment And what ZRT does is it includes a port of glibc? It provides basically a subset of the POSIX API It provides an emulated in-memory file system So you have kind of something that starts to look like a system instead of just kind of a bare process And it also includes things like a port of the CPython interpreter So if you have Python code You can actually just pass that into a 0vm instance as a tar ball And you can execute it using the port of CPython that we already have included in ZRT So another way that we like to describe 0vm is it's really a computing platform that consists of a group of technologies You know one of those is the core 0vm technology that creates the the sandbox that trusted execution environment Another is the ZRT technology that we were just describing All built around a set of core principles. So I want to kind of talk through those principles And what they mean to the 0vm team and the 0vm project The first is this idea of being secure. I'm sorry small light and fast So the the 0vm executable is only about 75 kilobytes of code. So it's very small You can start a 0vm instance in as little as five milliseconds. So you can start it very quickly It's very lightweight and it's fast So once you there is an upfront validation process that occurs But you can you can actually go ahead and pre-validate code that's going to execute inside of the 0vm Execution environment and once it's been pre-validated. You're basically running at native performance. There's no overhead. That's kind of you know sitting between Your code that's running and the hardware that it's running on It's very secure. We talked about the security derives from the Google native client project So not only do we create this inner and outer sandbox around the execution environment that we're providing to users But we also go through and we validate all of the code that's going to be running inside that sandbox We use the stock validator from from knackle And that ensures that nothing that's executing inside that sandbox is going to either maliciously Or accidentally break out and do anything that it should not be doing We like to describe 0vm as hyper elastic and this is the idea that you know You can use an entire cluster of machines hundreds or thousands of machines for a few seconds at a time By creating 0vm instances on all of these machines networking them together And then spinning them up and then spinning them back down and disposing of them when you're done So instead of having to you know pay for an hour or even a minute of a virtual machine You can start to you know use seconds or even milliseconds of compute across large clusters It's embeddable. So we talked about the ability to embed 0vm inside of multi-tenant storage systems We're gonna go a lot more into the integration with open-stack Swift today But it's by no means limited to being integrated into open-stack Swift. It could be embedded in other storage systems as well We like to think about it as being functionally pure or we sometimes use the word deterministic to describe this and Basically what that means is for any given set of inputs and we control the inputs into a 0vm instance very carefully You will always be guaranteed the exact same output and this is important for a couple of reasons one It allows us to do things like inside of Swift when you've got three copies of an object stored on three different physical nodes You can actually process all three of those copies in parallel and you can be guaranteed You're gonna wind up with the same result in all three locations It also allows us to do things like pause execution of that environment move it to a different physical Location restart execution We can replay execution If we can if we have the same inputs we can kind of replay and wind up the same output So it makes a lot easier for debugging and things like that And then of course the last point up here. It's open source. We believe in creating a community around this. It's all available today Apache 2.0 license All the repositories are available on github all of the documentation is published on our website 0vm.org Or docs dot 0vm.org is the the documentation site specifically So one of the questions that I get asked a lot when I'm describing 0vm to people is how does this fit in the in the Realm of other virtualization technologies So I kind of threw together this slide where I want to kind of talk a little bit about the difference between traditional virtual machines Containers things like LXC or open VZ and then 0vm and where they're similar and where they're different so the Common thread between all of these different technologies is there ways to safely share Hardware between multiple users or multiple, you know processes in on a single physical host But that's kind of where the similarities end You know we talk about traditional virtual machines You have a dedicated kernel and an operating system and you can customize that in any way that you would like You do have the overhead that's created by kind of presenting an entire virtualized set of hardware to each of those individual Machines Startup is a little bit tends to be a little bit slower But you do have really good security because everybody is very well isolated from from the other users Comparing that to something like containers you usually have a shared kernel and operating system between multiple containers That lowers the overhead it allows for a much faster startup time But you wind up Sacrificing some of that security now there's been a lot of work in the last few years around, you know C groups and namespaces specifically with LXC to make it more secure, but you still wind up opening yourselves up to some some chances of you know Somebody inside a container being able to either DDoS the underlying system, you know through syscalls or something like that Or a bug in the kernel Exposing something to other users that it shouldn't be exposing And then you've got zero VM where you don't have a kernel or an operating system at all It's we're really not virtualizing a system at all. It's really more of a single process a single execution environment that you're providing Very low overhead very fast startup and because of that validation that we're doing we maintain that security that makes it safe to run this And to even allow users to execute arbitrary code inside of multi-tenant systems such as public cloud storage services So what are some of the use cases enabled by zero VM? So the first one we think we usually think about two different groups of use cases The first one is going to be this idea of data local computing So you have your data stored somewhere perhaps in a storage service a public cloud storage service And now instead of you having to pull your data to your application to process it You can actually take your application and push it to your data and execute your data in place without having to move it around The other group of use cases that we like to talk about are kind of simple scale-out architecture So zero VM does impose some constraints on the design and architecture of applications But if you're willing to operate within the constraints of zero VM What you wind up with is an application and data bundle that can be scaled Horizontally as easily as you can distribute your data So instead of having a multi-tier application where you've got a load balancer tier and a web tier that you know scales out Very well horizontally and then some sort of centralized database. That's very difficult to scale horizontally And a bunch of interconnects between these different layers You wind up with an application and data bundle that you can very easily scale and just distribute with a dumb proxy In front of it distribute the load across that data So embedding zero VM and open stack Swift so For the Swift developers in the room apologize about my overly simplified Swift architecture diagram here, but for those that are not Swift developers And are not super familiar with the architecture of Swift at its most basic layer Swift has a layer of proxies Which can very easily be scaled and then a layer of storage nodes and there is what's called the ring It's basically a distributed hash table that maps Individual objects on to specific locations on those storage nodes So when an object gets stored in Swift it gets first sent to the proxy the proxy looks up in the ring where it should store The three copies of this well Swift is configurable You can store n number of copies most installations store three copies Rackspace cloud files uses a Replication factor of three so it picks three storage nodes. It sends the data to those storage nodes and then returns a Response to the user that the data has been stored successfully So when we integrate zero VM with open stack Swift and I'm going to get into a more detailed architecture diagram in just a second We install what we call the zero cloud middleware And that installs in the proxy nodes and we also run that on the object storage nodes themselves And that zero cloud middleware is what's going to enable us to actually execute the zero VM Instances on those object storage nodes where the data resides So I've realized this is a pretty busy diagram I'm going to try to kind of talk through this in in a in a real-world example And then I want to kind of pull up a live environment that we have and kind of show some demos of this running so Basically, if you look at what's happening inside of zero VM, we've got the zero cloud middleware, which is running on the proxy The user sends a request into that to that middleware And it's that request to the zero VM middleware are flagged with a specific header in the request And that request can include a couple of things it doesn't have to but it can include the Application or the image that you actually want to run inside the zero VM in instance And it can it needs to include what we call a job description file and that job description file Tells the middleware exactly what it needs to execute and on what objects inside of Swift it needs to execute those on And you can do things in this job description file like you know say I use wild card So you can say I want to search Every object inside this container and the middleware will actually go out and pull the container listing from the container nodes It'll figure out okay there's these hundred different objects that are stored inside this container and These objects reside on these you know hundred different physical nodes and Then the middleware goes out and starts a zero VM instance on each of those physical nodes to search the object That happens to be located on that node and then the result can be aggregated back together You can do you can kind of connect all of these zero VM instances to a single reducer node for example And you can aggregate those results on that reducer node You can concatenate those results in the proxy And you can either store that as a new object inside of Swift or you can return that to the user as a response to the To the request that initiated this You can also have the application that you're going to execute inside the zero VM instance can also be stored already Inside of Swift as an object and it can be pre validated if it's already stored inside of Swift So if you look at that connector that can you know talks to the get handler and access as a file That would be if the middleware needs to go out and actually pull a copy of the Application that's going to be executing inside of Swift. It does that using the standard get handler that already exists And like we talked about you can build networks of these instances And so you'll kind of see the executor that's residing on the object storage nodes It handles connecting the individual zero VM session that it creates on that node To other zero VM sessions that might be running in this cluster allowing you to kind of create entire map reduce operations or really any Directed a cyclic graph that you like that you would like to define in the job manifest file All right, so now I am going to swap over to a live demo. So let me exit out of here real fast. All right, so This environment that this environment that we're demoing right here is what we're calling zebra It's located at zebra dot zero VM org. This is a playground environment That the team the zero VM team has stood up to allow the community to begin to experiment and get their hands on The zero VM technology without having to download it and it set up their own environment to really experiment with this We're going to be making you know over the next few weeks We're going to be opening this environment up to more and more members of the community So if you're interested, you know, please let us know and we'll get you on the list to get access to this environment It's about four racks of gear that's currently stood up in a data center just out of just outside of Washington, DC And it contains a standard Swift installation with the zero VM middleware installed on the proxy nodes and the object storage nodes like we talked about the file manager that you're seeing up here, this is a Basic JavaScript file manager that the zero VM team has written It provides all of your standard Swift functionality that you would expect, you know the ability to create containers Upload objects download objects things like that Sorry as well as some additional functionality that's enabled By zero VM such as the ability to execute applications or you know execute Job descriptions like we're talking about So the first demo that we're going to go through is going to be kind of a basic word count demo And so what we've done is we've written a basic Mapper function in Python, so we're going to use the standard Python interpreter That's built into the zero VM runtime to execute this mapper And this is going to go through and it's going to open up a book We've got books from the Gutenberg data set pre-loaded into a container I'll show you I'll show you those in a second So it's going to go up open a book and go through and run a word count on that and Then we have a basic reducer function That again we've written in Python and this is basically going to take all of the output from each of these word count These mappers that are going to be running and create a report for us that we're going to be able to view in in the response And what we've got here this JSON file this word count Shakespeare JSON This is what we call a job description file So I'm going to kind of talk through this a little bit because this is it actually how you tell the zero VM middleware Exactly what you want it to do and how you want it to do it. So the first thing you'll see is You're basically telling it for these mapper nodes For the word count mapper nodes You're going to be executing it using the Python that's built into to zero VM And what you're doing is you want to connect standard in Based on based on the object that is connected to that mapper node and What you do here is you tell it the path that you want to actually execute this on so We've got Swift the little dot in there is just an easy way of saying my account inside of Swift Instead of having to put a big long off Token and so in the Gutenberg container I want to search every object that matches that wildcard 1ws star Which happens to be all of the works of William Shakespeare that are that are stored in the in the Gutenberg data set And then each of these mappers you want to connect those so here's we here's where we're kind of describing The the way we're going to connect the network to the single word word count reducer that we're creating And then we've got a description of the word count reducer Here is the path to the reducer code that we want to actually execute inside of this And then we're basically not going to connect standard out to anything Which will mean that it will be returned as the output of this will be returned as part of the request when it's executed And just to show you where this is actually going to be running against Here's the container where we have all of the Gutenberg data stored and so that's where it's actually going to be running so when we go and We actually go through and execute this we're sending this job description to the zero cloud middleware that's running on that proxy nodes It's going out and doing the lookups figuring out where those objects reside It's going out and starting Instances on all of those nodes and then it's connecting those to the reducer and then the reducer is returning the report that you're seeing here So we went out and searched 76 different books so we went out and store and created 76 different zero VM instances All those were connected to that single reducer and this is the output With the top hundred words that that showed up in Shakespeare's texts Below that we've got a detailed execution report for every one of those instances It's not displayed very pretty here in the in the UI. That's something we're going to be working on But basically for each of those individual instances You can tell exactly what the wall clock time that that instance was running for how much actual a CPU time that instance took Any IO that that instance did and this is where we can start to get that granular level of functionality to enable to you know us to you know potentially charge users based on just you know the few milliseconds Or seconds of CPU time that that their particular job took even though it was run across a large cluster of nodes So the next demo that I want to show is the ability to actually Manipulate images and objects dynamically as they're being retrieved out of Swift so what we have is we have a Images container. We've got a couple images here. One of them is spiral, which is what I'll use for the demo and so this is the image that is stored inside of Swift right now so the image doesn't contain any type of watermark or anything. It's just a plane. It's a plane image And what we've done is we've set we've set up our Account so that every time we open an image that is of a certain particular content type in this case JPEG We are actually going to execute a zero VM application on that image And we're going to dynamically insert a watermark on that image as it's being retrieved out of Swift so if we go ahead and Open this spiral image It's actually going and it's watermarking that image on the fly and inserting The zero VM watermark which we set up our application to just use that watermark And the cool thing is you can actually then pass arguments into that zero VM application To do different things so if you want to move that watermark around you can pass arguments and say x equals 100 And let's say y is 100 and this will If you open that URL now Those arguments got passed in through the URL to the zero VM application and it dynamically adjusted the position of the watermark in the image You could imagine using this to store kind of a single version of an image in Swift and then serving Different resolutions or different formats of that image based on different, you know target devices or different use cases Instead of having to kind of create an entire Group of images based on every possible configuration or or target device that you would like to see So switching back to the presentation here And it worked all right that was good So what are some of the future possibilities with zero VM in Swift so at the most basic use case You know you can imagine having a object storage offering where instead of just having to store Static objects and and you know retrieving them out of the storage offering if you want to do any processing with them You can do basic things like you know search those objects kind of like we were doing with the with the word count example Or somehow process or manipulate those objects You can imagine having a container full of images and you want to go through and generate thumbnails on all those images Well instead of having to download those all locally generate the thumbnails You can just write a little zero VM application that generates thumbnails You can push it up and it'll execute on the object storage nodes where those objects exist where those images exist And you can create the output as a new container That's full of just the thumbnails of all those images so things like this that are just a better object storage Offering and eventually if you start thinking about more and more use cases You can do like that you start to think about creating this entire compute and storage big data platform You can start to do full map reduce operations We are looking at the possibility of porting You know tools from the Hadoop ecosystem say to be able to run natively on top of the combination of Zero VM plus Swift so instead of having to stand up your own dedicated HDFS cluster You can just borrow a public cloud cluster for a few seconds to run any type of big data analysis jobs that you want We're also looking at kind of you know the idea of targeting solutions at specific verticals You know the media and entertainment space has a lot of use cases around Rendering and transcoding media. We could see kind of a portfolio solutions that could be built around that The big data space. There's a lot, you know, especially scientific Python Is something that we think there's a lot of different use cases that could be very applicable for this type of solution And then the idea that you could start to using zero VM push dynamic content creation to the edge So instead of just CD ending Static content actually being able to start pushing that dynamic content out to the edge For applications where that type of performance makes a lot of sense All right, so we've got about 10 minutes left and what I wanted to do is invite Camwell up on stage with me And open it up to the audience if they had any questions about zero VM the technology the integration with Swift Up here, you'll see all of the contact information for the project our website the github Organization are mailing lists as well as our IRC channel on free node So and if you do have questions if you can please use the microphones for the recording that way those can be Heard by anybody that happens to be watching this later on John How's it going? Great. This is a fun demo. It's great I have one question about the way that the compute is done with the replicas of the data So for example, you've got your full-text search or your water marking And you've got three replicas of the data So are you doing three times the compute or and just taking the one that fails first or I mean ignoring failures or? Is that like an ha strategy or are you just doing it on one or how's that working? Yeah, so so today we have what I like to describe a very naive job scheduler So it basically picks randomly one of the copies one of the replicas And goes and exit, you know decides to go execute on against that replica and it ignores the other two replicas There's a lot of opportunity there. We could we could execute on all three in parallel and you know, just grab whichever result finishes first We we could also Do things like you know have if you know if you have a zero VM instance that fails So right now if you happen to get to Constantine wants to come up here and talk about this as well I think but you know you could have something where if if one node is much busier than another node for whatever reason you could Pick like the least busiest and replica or the node with the replica. That's the least busy We've got some of that in Swift today So they'd be kind of cool to see those those kind of contributions for doing that with just the read and write affinity And then adding that in to say hey, let's put in something else. That'd be kind of cool Sounds good and constantly you have some dad. Yeah Right now if you write things back to Swift, we just run we can run things three times She's just works right now. I think the default action is if you write something back to Swift, it will run three times Oh, really? Okay. Yeah, look at that. I Learned something All right, another question over here. I guess since we're on the watermark example was that were you guys using pill for the Python or imaging library for the the watermark stuff. Yeah, so the library that we were using I'm trying to think Ron So as a lead-in, I'm actually curious like what about other Python compiled dependencies like numpy or numpy depending on your pronunciation Yeah, so we have compiled Python dependencies for the scripts that we're running in zero. Yeah, exactly So so we have a set of standard libraries already ported running inside the zero VM runtime We'd like to expand that set Like numpy is not one of those right now So if you'd like to do anything with that it's going to take a little bit of work to kind of get that ported and Compiled to run inside a zero VM As a community what we'd like to do is start growing that base of kind of cross-compiled libraries and application frameworks To to make it easier for people that that have different use cases like that one question How to keep the data safe safe safe? to keep to prevent the malicious code to do bad things a Damage your data Well as the whole point about zero VM is a very high security Zero VM is based on a Chrome and ACL as like described it and It's impossible to get out of the framework. So all your data That the code has is only the data which Operator or account owner allowed you to access So from inside zero VM, you can see data is open at files and you can write only to open file handlers Okay, so you cannot access any system Resources Yeah, and and right now when you present an object to a zero VM Instance it's actually presented as read-only. So you're not you're not going to be able to you know accidentally or maliciously you know Stomp on your existing the original object the original data So everything output is either written as a temporary file and then returned as part of the request Or it's written as a new object inside of Swift Okay, another question over here a very similar question. Does the Does the virtual machine block opcodes that? are Are not allowed inside the specification or does it rely on the compiler to not include those? We rely on our own tool chain our own compiler So you cannot just run any binary you need to use our GCC tool chain or we also support LLVM tool chain to tool chains and Only code compiler resource tool chain would actually run of course in runtime. We do is a validation Okay, so there is validation. So I I could I can't write a bad binary and then And get it to fully compile The validator we use is a verbatim copied from chromium project. So Zero M as secure as the Chrome sandbox Cool over here you showed in your watermark demo Executing code on retrieval. Can you also execute code on ingest? So we've talked about a couple ways to kind of put hooks into the system to do that right now It's it's not fully there. We've got some ideas about how to implement that That's something that we actually wanted to talk with some of the guys in the in the Swift community about because I think there's already some Some work being done around there. So we want to make sure we tie in with that also, how flexible is choosing what data to To execute on is it just my file type or? It's fully flexible right now you can access container database also as a read-only file So you can for example select from container database in any type of select you like for example You won't select something by name or by reggags anything you SQLite supports you can do that and then act on the result of that like choose objects Okay, another question over here. Yeah, if you still have time I Wanted to talk a little bit more about the data pipeline on the map reduce job that you had done How is the mapper processes communicating their results to the reducers? Does that get stored in Swift as a temporary file or is it using zero via this zero MQ channel? By the way, this this is Constantine here He's one of the co-creators of zero VM and he is also the guru with the zero cloud middleware So that's why he's answering a lot of these questions about the details in the work is there We do we can do it both ways In the demo it just relied on zero MQ and transfer it over the network to the reducer nodes How much data was being processed in the Gutenberg example? I think it was a very little data like 270 megabytes a few hundred mags So the most like the most time it takes is just to set up the cluster because if you set up the cluster of 76 machine or we had the demo where we set up cluster of 750 machines It's takes a couple of seconds like 10 seconds just to set up the thing Yeah, so so the way the way that the cluster is set up the way that networking works inside the zebra cluster today If you create a cluster of 750 zero VM instances for example, you know The networking would be set up for each of those and they would each be connected However, it was described in the job description file We've got some enhancements to the networking stack which we are getting ready to push out probably next week And that basically relies on a networking broker that was that sits on each physical node And maintains those network connections between the different physical nodes within the cluster So it's going to significantly reduce the startup time And and you know, we did some testing in our staging environment that you know Cut some of that time the execution time in half simply by reducing the startup time Right. I'm curious about what the biggest challenges are on the cross Compiling on the porting say of numpy and other things. What do you think the biggest the hardest parts are? Well There are two challenges the first challenge With a numpy and more sci-pi is that it's a library that's more scale up Yeah, and on our computer environment, we better suited for a scale out systems That's what the the second one is there that sci-pi uses in few places a Fortran and some exotic stuff which are not yet supported. So The most of parts of numpy is already Recompiled just from the first try and working So when you when you bring in a new library and you just I assume you just try to cross compile and see what doesn't work Exactly, what is that first few levels of stuff that needs to be ported? Like what are the calls what isn't supported? The the the main obstacle right now for numpy is lib ffi because it's written completely in assembly and not compatible assembly To the knuckle got you so you need to rewrite it with knuckle assembly and it's can be done only manually You cannot like do anything about that Got you. Thank you after lib ffi. I think it will be just a breeze because everything else is okay So usually as the stuff which is written in a pure python pure see is Usually work out of the box immediately All right. I think we got time for one more question. Okay. I'm one more question from the performance perspective Do you have some? how to handle the performance overhead because the Storage node is I will intensive but you introduce the computing into the storage node. How do you keep the to Keep the computing power For for the IO Yeah Well, if first of all in in regular swift if you don't Run there any computing that turn into would be to get all the data out and by getting all the data out is Also, not a very light with operation just to get data out of the swift. So If you do some data intensive light with processing then it makes sense to push it into the swift Yeah, of course if your processing is very heavy weight like hours over hours of processing the same data Probably it's not the best case to do it inside the swift Okay, so that type of computing is is basically for a light weight data intensive processing And in that case the actual load on Swift server would be less than the alternative just taking all the data out processing it and then putting it back All right, so so I wanted to thank everybody We are going to be around all afternoon at our booth which is kind of right outside across the aisle there tomorrow morning We also have a three-hour session where we're going to kind of doing a deep dive into some of the technical areas of This integration so if you're interested in learning more visit us tomorrow morning. We're gonna be up in room 405 And thank you