 Hello? Check. Hey. It works. Thanks all seven of you for coming and four or five of you I work with so I appreciate that. My name is Christopher Ado. I am a senior director of technology operations for Morph Labs and I do talks on occasion and I also make beer. I don't know if you guys are familiar with Morph Labs at all but we deploy private cloud based on OpenSec and they are meant to be consumed as an appliance. So we're actually out there in production. We have some real sales out there and some real users on OpenSec which is kind of fun to be one of the few people selling into the enterprises as an appliance and actually doing it successfully. And we work with big and small enterprises and one of my parts of my job is to work with the team that takes care of all the environments so it's important to me that things work really well and OpenSec has actually been a big help to us in that especially coming from Eucalyptus. So yay for OpenSec. We started out in 2009 working actually on top of Amazon offering a platform as a service and we were one of their earliest really big users. I think at one point we had 4,000 customers using our service on top of Amazon and during that we learned a lot about cloud and learned about what people were looking for with respect to cloud especially and specifically people who wanted a little bit more help with infrastructure as a service and with abstracted systems. And we saw that there was a really big opportunity there to offer people the same kind of improvements in functionality and reliability that you could potentially get from a infrastructure as a service play like Amazon except getting that into your own environment on dedicated hardware and in a place where you felt like maybe there was better security or at least better control over who had access to what within that environment. So we saw an opportunity and thought maybe we should go for deploying private clouds on Eucalyptus. So at the time really Eucalyptus was the only option short of trying to go on our own and build up our own solution and it seemed like Eucalyptus would be a good foundation for us to build on top of. Who here is really familiar with Eucalyptus? Who has used it or deployed it? Have you guys deployed it and actually used it in production or okay. So if you're familiar with it you know it's not lightweight and to be fair we have not spent any real time using I think through up to 3.1 now you know. So it's gotten better certainly and they had a long dark period that they have kind of come out of now especially in terms of community involvement and being open and moving a little bit quicker with the code but we were we kind of committed ourselves to it and were built on top of it for a while ran into a lot of problems. Some of the problems we ran into were the lack of modularity it was it really and I think it still is this way which is basically kind of a big monolithic thing and it's not that easy certainly when we were working with it it was not easy to add add on to it unless you built into it and when we ran into problems occasionally they were really terrifying or impossible to reproduce and I personally don't like Java very much and I find that to be problems like that where you have an error that you can never reproduce but it happens randomly over here or over there that seems common to Java. I don't know it's just bad luck maybe the applications that I've worked with but that was our experience and it wasn't fun and the biggest problem we had was when we would find problems with eucalyptus we would find bugs we'd figure out what was wrong and we'd write up a fix for it and we'd implement the fix and we'd submit the fix upstream and it would sit and we'd talk to people and it would sit and we'd talk and it would sit and we were seeing up to a year between submission and acceptance of some really critical important bugs which led us to I mean that was absurd and that's why we ended up forking it which was also probably the least bad thing that we could do but we were kind of stuck and we needed to be able to move a little bit quicker and we were working with NTT at the time and we wanted to actually collaborate with other people and see if we could have some other people helping us fix these bugs off on our own and move a little bit faster on the code base but that unfortunately got us into a place where we were also starting to make changes to eucalyptus and drifting further and further away from the trunk and or from the original stuff that we'd forked from and after a certain while it was basically going to be impossible for us or nearly impossible for us to actually merge what we had been working on back into mainline in hopes of taking advantage of some of the new things and the fixes that the eucalyptus people were introducing. One of the other problems that we ran into was totally self-imposed and that was that as we had moved away from the platform as a service offering on top of Amazon we realized we had an opportunity to bring that same thing into the private cloud and we brought it forward but the reality is if you're trying to offer a lot of platform as a service stuff you know a lot of different single click deployments for blogs or databases or anything else that kind of is a full-time job that you could build an entire company around keeping that stuff up to date and taking care of it so trying to do that on top of also trying to build and maintain and deploy and support our own private cloud became very problematic and it was that really wasn't eucalyptus' fault it was more of our fault but problems we ran into that was one of the biggest ones and we also really were trying to trying to adjust the product to make it work a little bit better for us and it was really hard for us to kind of orchestrate the cloud and make the cloud work the way we wanted without actually getting into VMs so there was it was really hard for us to reliably inject things into the VMs that were being launched within the eucalyptus environment which led us to kind of get more into the environment than we should have been like for instance right now we have a really solid line outside the environment so we won't go into a customer VM we make sure the environment works and make sure it will run just about any VM you can put up there but we won't go into the VM and we never should we never should inject anything in there to meter it or anything else any metering should be happening outside but because we really we there were things we wanted to do with respect to measurements and metering and some things like that we actually had to do it in the VMs that were being launched up launched within eucalyptus which was bad and by the way especially since there's only a few people here feel free to interrupt me raise your hand if you have questions or anything along the way yes how many developers at Morph Labs so what's that 50 50 50 developers yeah so we work I don't I kind of in in an effort to not make this marketing II I didn't talk too much about Morph Labs but we're a we're based in Manhattan Beach our founder is from the Philippines we have a pretty big office in Manila and then we also have sister company that has 200 developers called exist that we draw from when we need more developers specifically for our company so right now we're about a 55 person company based in Manhattan Beach Singapore Manila and Tokyo so that was the size of the team and it was challenging so meanwhile that's on rack space hooked up and made a baby hey half the crowd know more than half the crowd now works with me thanks guys you guys are probably all very familiar with OpenStack considering you're at the OpenStack design summit so what at this point we went OpenStack launched in 2010 we were very very interested and we saw that there was really incredible potential and so we stayed involved from the from the beginning we were doing a lot of testing internally and watching it grow a lot but at the time certainly at the beginning and even at at the point of Diablo we didn't think it was ready enough it was very close but it was probably in terms of Diablo versus Eucalyptus it probably would have had as many headaches with one or the other and then earlier this year with well actually with Essex development we saw that Essex was really going to be the build of OpenStack or the version of OpenStack that was ready for primetime and we were at the design summits and saw how quickly the community was growing and how passionate the people involved with OpenStack were and we it was a very very big difference from what we were seeing in Eucalyptus and in the people involved in Eucalyptus and giving back to it there was no comparison so looking forward we knew OpenStack was going to be for us so we started talking about how we were going to move from Eucalyptus to OpenStack and we had a lot of serious considerations so like you were asking how many developers we have that's a finite resource and if you are really neck deep in trying to keep an existing product working and try and deal with customers who are finding bugs and getting upset about it it's very very difficult to step away from that and go you know change direction without risking losing your existing customers ruining your name and and all that so so we we had some pretty animated conversations within the company about how and when we're going to do this and whether or not it was the right time for us and the right choice and unanimously we all agreed it was the OpenStack was the right choice there was a little disagreement on whether or not it was time for us to to jump to it but eventually everyone agreed and the last real question was who's going to be stuck behind on Eucalyptus to take care of it and we essentially broke off a tiger team of some of the sharpest people who could really be counted on to fix Eucalyptus stuff and keep it active for however long it was going to take us to get on to OpenStack and we hoped we could move pretty quick we weren't sure if we could and we wanted to insulate ourselves from that risk as much as possible so in actuality it only took us about six weeks to tear out the Eucalyptus Foundation and move over to OpenStack as part of that we moved we were previously deployed on top of Centos using Zen as our hypervisor we went to Ubuntu and KVM we extended the Fog API to support Essex and in fact Hunter right here was his team and Hunter were the largely responsible for that and gave that code back to Fog which is one of the things that we in terms of OpenStack we try to give back to the community too and one of the things we wanted to do is bring more Ruby Ruby people into OpenStack and kind of expose more people to it because certainly at the beginning of this year if you talked to almost anyone about OpenStack it was universally Python and everyone only talked about Python and primarily the only developers interested in it were really Python devs and we were trying to show that a shop like ours which is primarily Ruby could use OpenStack and could really make make pretty good use of it so Fog API and the let's see we're we're already been using Puppet for years and in fact Puppet was the was what we're using for our platform as a service stuff so we use Puppet and that helped us move really quickly on deploying OpenStack in a really reliable way that we felt we could count on for production environments so the foundation came from Puppet Labs the core of it is actually is from Puppet Forge and we actually work really closely with Dan and Dan Bode at Puppet Labs and we work we're good friends with the Puppet guys so we we are collaborating with them especially on the some of the Folsom and Grizzly stuff and trying to bring back some of the changes that we had to make more mainstream so we the foundation was certainly came from Puppet but we also had some unique unique requirements for the way we deploy and and then you know there may be the difference I think one of the big differences is that we deploy lots and lots of discrete environments we're not trying to deploy one massive gigantic OpenStack cloud we're selling and deploying lots of very small OpenStack clouds which we also need to continue to maintain and take care of so it it changes some of the considerations that you you would expect you know we think about when you normally just think about deploying OpenStack wait what's wrong why not have a single cloud yes so so we're so the question is what drove us to deploy in a like multiple small environments right because we're selling a single tenant solution and we're it's meant to be deployed in your data center or in your you know you you own it you own the hardware so you know we're not we're not a data center exactly correct so so when some of the things that really helped us make this transition possible to also had to do with the equipment it was a good we are we had been working really closely with Dell and their solutions team for several years and we were able to get our hands on some of their prototype and and some of their hardware before it was actually available to the public in order to do our testing of OpenStack on what was going to be the latest and greatest from Dell which is what we wanted to deploy on so we're you know we're using a 2u box that can support 80 virtual CPU or v-cores basically three terabytes of an excented backed block storage and that's expandable so you add more 2u units to make your private cloud bigger really easily so yes so nixenta is an open Solaris based storage appliance and that sits off on one node or multiple nodes in a larger environment and we use the Nova nixenta volume driver to connect to it so in terms of OpenStack from within OpenStack with the API or our UI there's no indication that that block storage is anything special but on the back end you can count on it it's extremely fast and extremely reliable and primarily that's because we're using the nixenta for it we're based on Essex and we are moving to Folsom actually as we speak hopefully within a month or less we will get all our environments up on Folsom so yeah so it took us about six weeks to gut eucalyptus out of the equation and it was brought a lot of really nice stuff to the to the table but we saw the problem of an hour we know we had a product that was ready with OpenStack but we had a bunch of production environments that were deployed on eucalyptus and we were hoping for some way to migrate them because we had to and we also we were obviously our own production stuff for our website and our internal things like JIRA and so on we're on our eucalyptus environment that we need to get them off of so we spent some time trying different approaches and really hoping for magic and it turned out that there really was no magic we exported VMs and imported them in OpenStack but it turned out for most of these VMs we were spending more work just trying to fix the VM up and get it to talk you know like adding cloud in it and things like that we were spending more time screwing around with the VM that we'd exported and imported than we would have if we just brought up a fresh Ubuntu 12 VM and installed the the app on top of it and then worried about transferring the data so there really didn't turn out to be very much magic involved it was pretty laborious and you know that was the the reality check that as probably everyone in this room knows once you're deploying an application on in a cloud environment it gets really complicated you've all the different considerations around networking all the different considerations around storage and how your VMs talk to each other how many nicks your virtual nicks you're putting in there VLAN considerations and all that stuff and going back to what I said earlier about us trying our best to stay out of the VM we never want to touch a customer VM we wanted to make this as seamless as possible but the reality was that there wasn't going to be a seamless transition so we knew we're going to be doing a ton of hand-holding for for the process and and I apologize if anyone saw the synopsis of this and hope that you're going to come in here and I was going to tell you a magic command that would actually you know send something out of eucalyptus in a glance I don't believe it exists but if anyone actually let me ask has anyone here deployed anything on eucalyptus or cloud stack and then move to open stack I work with a company that that does that's what they do it's called River Meadow software and they they do migrations from all sorts of clouds to all sorts of other clouds they actually do physical migrations as well so physical to cloud and they've done open stack to sorry cloud stack to open stack they've done AWS to open stack they've done it Zen server to AWS all different combinations so I can you know what's the experience been like what the secret sauce is of course is that you have to concentrate on the VM and and then you have to do a little bit there's some magic secret sauce in doing the conversion but you know if you know what you're doing you you write some scripts around that so so it's basically it's a two-step process it's an extraction little secret sauce and then a dump back in so at this point we're using raw primarily exactly so you know what we we what we looked at it as it was a data center and that's kind of how we treated it so we basically added additional environments for the customers that we're moving to so we were we doubled up on the environments to an extent but with a drastically reduced footprint open stack is so much more efficient we were able to really deploy comparable you know comparable clouds on much smaller heart you know much smaller footprint really so the approach basically was to give these guys an additional environment and then work with them on the transition from eucalyptus stuff into their new shiny open stack environment that was kind of how we did it we did some some scripting and some we tried to make some of the transition stuff easier especially around storage because on the eucalyptus product we our persistent storage was based on NFS so we had every project or cloud that you know that you spun off on eucalyptus we get its own slice of NFS server for the persistent data mounts and we were able to make it a little bit make it kind of easy to expose that directly on to open stack so that you could grab that data and copy it over to a block storage device based on Xenta so we did a few things like that to make it easier but for the most part you know we would either pull the VM out or create a new one that was similarly configured and then help them get the data over and then cut the traffic over to the yeah you know how you swung it over how much data was involved was it a small amount or large amount under a terabyte for the biggest environment so how did you do the data transfer did you just just ran it over the wire or did you basically yes we we had 10 gig networking between all of the storage stuff and because they some of these were actually live applications we would do it with the customer during a fixed window basically pause everything and hurry do an initial snapshot and then pause everything and re-sync it and then swing the traffic over reliability I know that NFS I think before I mean basically it was much easier for us to deploy with Nick sent deploy block storage on Xenta with OpenStack and make it highly performant versus attached you know basically connecting to an NFS mount so what we do where you were using Xenta and the storage nodes are all SSD accelerated so the read cache is on SSD and it's connected over 10 gig networking so you're getting a much much faster connection that way at least work it was for our use cases it made a lot more sense to do that versus just trying to set up a big NFS server for each new open-side environment and it also allows people to you know with the API to programatically provision and attach and destroy that storage as needed so this has been about six months I think since we cut over completely and we really love OpenStack it's been much much much better to us we've we've had certainly problems like the PTL for sender I think is in his session the other day he was saying it's not your config it's a bug and that's that's the only kind of thing we run into that's a little frustrating sometimes we feel like we've got everything right but then something doesn't work and we get into deep and then we turns out that it's actually an open-stack bug and those situations are getting fewer and fewer which is great and a really good sign that OpenStack is more and more ready for for actual honest to God prime time but in general OpenStack has been way better to us we've been much happier and it's worked much better and so looking forward for more labs we're working on Folsom we're looking forward to incorporating some of the stuff in quantum and some of the really good stuff that's coming up with sender and then also just the all of the projects that are that are around OpenStack that are offering some really incredible functionality that you're not seeing anywhere else any of the other open cloud platforms you know none of the alternatives are anywhere near as vibrant as OpenStack there are just nowhere near as many kind of projects and options in the future and being you know actively worked on and getting really big companies Cisco and other rack space great big companies throwing everything they've got behind OpenStack so we're really excited about it and we're going to continue to increase our involvement with the community in the in the months and years to come and so as everyone else says we're hiring if you'd like to send me your resume I'd appreciate it and thanks so much for coming out and any questions at all anymore it it depends on the customer it depends on how savvy they are so we certainly have some pretty big enterprise customers that are coming to us because they've considered build versus buy and they realize that they're just admin staff maybe very technically competent but they have a lot of stuff to do and they don't have the time to step back and figure out how to manage how to deploy manage OpenStack so we've got a lot of people coming to us with that in mind basically recognizing that we we could we offer them the potential of you up and running on OpenStack in less than a month and in a fully supported deployment so and then we also have other people who don't know anything about it and all they know is their CIO said this is the year for cloud so we're going to go cloud this year go get a cloud and that's we get some of that too. Any other questions? Yes. Can you talk about ZFS? I mean it's it's not the only answer but that's the that's a big piece of it the other piece is Nixenta as a corporation knows what they're doing and they they when we deploy every every Nixenta every storage node comes with a supported license from Nixenta so from a business and strategy perspective we we partner with Nixenta we also work with Ubuntu so even though the customer doesn't get access to the you know root level on the on the server that's hosting their environment just the same it comes with enterprise support from Ubuntu so that we're using the Ubuntu packages and we have the we have Nick and the other guys at Canonical working with us and anytime we need extra extra that extra layer of expertise we can we can get that help from Canonical so in terms of storage Nixenta is probably in terms of block storage Nixenta might be the most mature and solid offering available right now maybe maybe not but for us it seemed like a really good choice and we'd also just in our own data center for our own stuff outside of the Eucalyptus product we've been using Nixenta for some time so we were already pretty comfortable with it thank you everybody so much for coming