 are we live yep yes all right so maybe I'll give you words here first of all we're the last stop before the party so hopefully you can hang up and stick with us the the talk was actually was was supposed to be driven by Barack who is for because of a family agency couldn't be here so after picked up after our product manager picked up the challenge and learn a new presentation in less than a day so we'll try to do our best to actually substitute Barack and and cover his talk we're kind of an intimate forum here so I wanted to ask a question first before we kick that off how many people here are actually consumer of open stock raise their hand majority of them how many are developing products for open stock all right and there is a reason why I asked that because I think the challenges that we're gonna talk in this talk I think is very relevant to those who are developing products on open stock which tends to you know test and put open stock in a point in which it actually stresses it much more obviously you need to deliver a product that works in various scenarios and not for a particular organization or user and and you'll hear a lot about our experience our experience meaning gigs faces and cloudify specifically what is the journey that a product company that wants to develop a product into open stock how do we actually do that what was the choice of infrastructure that we've gone through over the years and we're talking about three four years now of a journey what worked what didn't work and then we'll try to also kind of get to the bottom line of that by all means a lot of that experience is our experience because of the way we run the product and the way we need to develop it it doesn't necessarily mean that you know all the lessons and all the experience that we've gonna present here and talk about is the same experience that you would have gone through there is no intent here to also bash any product here or do any comparison that says this product is better than other products really talking about from a really really personal experience and try to share our experience and and give you the lessons that we've learned over the course of years nothing more than that I would like it to be an interactive session so if you have questions and if you especially those are also coming from a product company that wants to challenge some of the lessons please do that the the whole idea is that it will be interactive and if we could make it in interactive especially again in this type of form that will be I think more entertaining for the rest of the people here so with that I wanted to give the mic to Arthur thank you right so first of all this is the Nati Shalom our CTO of Gigaspaces at the company behind Cloudify and I'm the product manager for Cloudify by Gigaspaces so we're going to do a quick introduction on what we do with Cloudify just to set the ground and set the stage on on on the types of interactions we are doing with OpenStack and our experience with that and we're gonna like quickly cover like private versus a public cloud discussion on in which types of cloud should you use when you're developing your own product and the clouds that we've been experienced with and where we ended up and what's our path forward so and just to set the ground Cloudify is a cloud orchestration platform you know just to get the jits of what Cloudify does it takes an application topology in a blueprint format and then basically works with the various APIs to realize that application blueprint obviously that's super simplified explanation of that but essentially Cloudify works a lot with the OpenStack APIs to bring up the amps to bring up networks connect to connect to everything that's that's involved in that and basically stresses OpenStack APIs a lot. So Gigaspaces started working with OpenStack at the SX release and basically ever since OpenStack has been the main path for developing Cloudify. So yeah that's the quick intro and as I mentioned the one thing we've realized is that Cloudify breaks OpenStack quite often and quite a lot and as I'm not to mention the main point of this discussion is not you know bash one one OpenStack distro or the other vendor rather to to basically take take us through the path that we've been going through with Gigaspaces and and and basically share our experiences and where we ended up and where we would like to go forward I would hope to go forward and nothing mentioned earlier that's that's I picked up this talk kind of in the last minute and the main reason I try to avoid this tag because I'm ex redhatter as part of the product team for for OpenStack so I don't feel too comfortable doing this talk specifically but interestingly enough and actually the slides were done by Barack they're kind of like interestingly put so I should be fine. All right so we started off by using by going to the to the approach of working with a private on-premise cloud basically bring it up OpenStack internally in our own IT environment and actually that was not not not really a blocker to work in public clouds you know we work with public public clouds a lot. Cloudify works with AWS with Azure and then almost every public cloud out there but we focus mostly on the private cloud due to some security reasons first of all and in the public cloud and when we worked on the public clouds basically we experienced some some some breaks and into the environment and some weird things that were going on that once we've started investigating we've realized we had some DDoS attacks some breakings into our own environment and you know working behind the firewall definitely definitely has its benefits even that you know the workloads that we run there are not you know real production workloads rather are in the infrastructure but still it would you know it's much much nicer to have an environment that you full control and second reason we focused on on private cloud initially was that fixed cost so whenever we work in a public cloud we're charged basically by the hour and the way cloudify works is it orchestrates you know different types of topologies different types of applications and basically brings up a lot of a lot of VMs it needs them like for a minute or couple of minutes and then tears them down so the price model of charging based on the hour per VM or per resource that you work so that you consume from the cloud basically blows up the costs of the overall of the of the overall R&D budget that we were hoping to achieve so going to the private cloud in the in the environment that we're trying to work actually worked a bit better for us a thirdly bandwidth so when we when you basically debug issues and you have huge huge log files and huge core dumps from the environment you pretty much would like to have a pretty good bandwidth to pull data to investigate data and you want to have a better access to those machines when working with large large quantity of data that you need to understand what's going on there and obviously multi-tendency and have better better control of which operations are allowed which operations are not allowed etc so public clouds tend to be more strict about that where in private clouds you're basically can configure whatever is works better for you so the first thing we did was go to DevStack obviously you know that's the first go to a go to a solution for developers and it's very easy to set up and then you know basically showcases what cloud what what OpenStack can do but it's not production great so once you set up DevStack usually it's a specific environment very different from customers actually are doing and you know when you work on a product that works with APIs it's it's really important to work with the same architecture as the customers are doing so working with with DevStack actually was quite limiting for us and like we realized pretty quickly that that it's nice to play around with it's nice to figure out how to work with OpenStack but to really build solutions it's it's quite challenging so the next step was to go to the do-it-yourself approach and install OpenStack so how many of you actually installed OpenStack themselves right pretty good set of hands in here how many of you actually read the documentation the installation guide for that how many enjoying the length of that guide so sorry you thought you should belong and how many pages do you think that guy is it's so so we download actually PDF and like went over all the installation guy and it's 164 pages long back in the day at least it was so you have to like read 160 something pages just to get started so I cannot believe it at this point I'm still showing this like super complex diagram of OpenStack architecture and still relevant and but but yeah so you know started with Grizzly and the architecture is super complex we had to go over all the services reconfigure them do it everything manually and it did work after a lot a lot a lot of effort and then it broke pretty quickly so we didn't actually like investigate on why realized we needed some more expertise in the house and so we did a sexto course by by Thorian has which was really good so that actually gave some background on the how OpenStack works all the internal services what's needed to be done in order to for for set up an environment that actually runs for more than two minutes and we installed the basically OpenStack in-house using some puppet puppet puppet scripts to bring that up and everything just started fine and then networking just stopped working randomly I think that was around the fall some release so VMs once VM started continued working but sometimes you know we had many like weird issues when the VM would bring up would come up without networking other issues when when we had networking but no DHCP other times DHCP was available but suddenly the VMs lost connectivity so we had lots of like weird weird scenarios where networking just was really not working well so at that point we realized we probably needed to work with one of the distros so Ubuntu was was really no brainer at the time it was one of the main paths on the OpenStack main documentation and pretty well documented bit easier to use the packaging at least from the commutation it seemed and then they do it yourself approach and the experience there that back in the day and this is you know 2014 couple of years back from now but we couldn't get it working with Juju so tried installing it tried getting up in the air but put a lot of efforts to bring that environment up and essentially couldn't get working and gave up after a couple of weeks so that was a bit disappointing and sort of saying next we've tried Mirantis's fuel to bring up OpenStack and that was also back at the day and that also did not work quite quite well so we did a lot of you know work and effort and actually actually ask questions around that but again back in the day it was really challenging and then we couldn't bring it up and to note that the first installation worked fluently when it was basically running in few minutes or few hours actually to get started with and the first few days was actually very good but once we broke it it was almost impossible to recover from that point and that's I think where it stopped working again I think that a lot of that had to do with the networking piece with Neutron specifically because in our test what we do is we create networks and destroy them and create and destroy them and that wasn't a stable part of the API OpenStack API and I think that was kind of the reasons why it got to that crash mode. Yeah and like one point one thing which is super important to stress back at the day it was really really challenging to debug to debug Neutron so you install Neutron you install the agents you bring up everything that you should be bringing up you look at the logs and it's really not clear on what's going on why the network is fail why do you don't have networking you know essentially everything works all the services are up in the air but it's really back at least it was much harder to debug than it is today I believe or hope to believe. I think it's also I think indicative on the state of OpenStack at the time I think there was a lot of lack of maturity and how OpenStack has been developed and what you would see in that journey is also kind of indication of the maturity that happens to OpenStack itself in the sense that you know I believe that you know the fact that we were running into that meaning that no one was actually testing a use case of creating and destroying things all the time to get it to the point of stability and that created a lot of timing issues and a lot of things that the distro couldn't solve because obviously it's a core issues within the OpenStack itself and that manifested in the distro itself and the distro tried to make the experience simple but at the end of the day if you hit a problem in the core OpenStack the distro didn't really solve that. Yeah and actually this kind of brings me back you know back to my redhead days where I used to work a lot with the OpenSort with the upstream community developers and you know working with the various Neutron teams developing Neutron itself and all the subcomponents and you know the kind of issues that you know I've been working around back in the day were like really trivial sort of like you would have expected it to be in but simply was not in so services would go down without any login or debugging messages so you know one thing to important to stress that you know back at the day OpenStack was much less stable and it's not specifically probably not specifically around this issue rather a general OpenStack upstream issue that you know simply Neutron was not stable and mature enough. So redhead days and this is back to the back to the Cloudify gigaspaces effort so the R&D team brought up RDO redhead's distro and the redhead's community distro using PackStack so that was rather simple to bring up you know simple simple CLI wrapper around the puppet modules to configure and bring up rather the same setup of OpenStack a couple weeks to install after all the investigation research understanding which configuration files needed to be like how the confile should be set up in order to bring up a multi-node environment with dedicated Neutron node and essentially glory days that was the title that Barack put here definitely this specific setup was up and running for quite a long time actually several months and you know because of the maturity we got we got we felt around this environment and this deployment and we felt we felt really secure really really really confident about about this deployment and started bringing workloads into it so you know initially we started that we tried to bring up some some CI workloads then we relied on that CI environment actually to build Cloudify itself and we started moving some some not real production but some IT workloads we needed into OpenStack and you know started feeling really confident about the environment and used this environment for the day-to-day R&D operations so if a developer needed a VM to do some research to do some some development some component needed on-demand resources and we used this OpenStack environment for these kinds of things and there's a big but there on the slide so you can assume what happens next so yeah so Cloudify obviously started breaking after several months of working rather smoothly Cloudify again started breaking OpenStack with various again networking issues so there was no DHCP suddenly suddenly at certain load neutral server would fail no connectivity to the ends and you know this is a screenshot of cases we've opened with Red Hat asking you know various things on how things work or how should be reconfigured most cases we got fixes for other cases we needed to do some reconfiguration different kinds of issues that we didn't actually have solution for so it was quite quite challenging after that period of time one of the challenges in developing OpenStack is that it's open it's developing an OpenStack sorry in an open source community that is not necessarily running a cloud unlike Amazon for example or other clouds and and that's challenging because a lot of the issues that we're discussing here right now for example making it stable over time that's not something that as a developer you could run and test easily you actually have to run it in a production type of environment for a long time to actually experience a lot of those problems and and get them manifested and in the open source community and the way OpenStack is being developed it actually comes back to the developers after the fact and in a way that it's almost hard to reproduce and not suddenly even get to the community it gets to read it and somehow indirectly it gets to the community which kind of shows one of the problem of the process itself the process of developing something like a cloud infrastructure in an open source type of environment which is good for certain things but it's more challenging when it becomes to something like a cloud in itself I think that's kind of the the interesting output from that. Yeah and actually like one another very interesting comment that this brings to mind is you have to have OpenStack expertise because if you rely on external expertise it's very challenging to keep this environment up and running and the real challenge here when you are an R&D organization working on your own product obviously your R&D or in your like most of your expertise are around your specific product so it's really like you know challenging to put some resources that would really understand OpenStack and then bring these types of expertise into your own R&D environment in order for you know to keep maintaining the environment but also bringing these expertise into the product itself so expertise are definitely key aspect when considering going into OpenStack and using OpenStack and developing against OpenStack in general. Right so skip a couple months forward after this environment started becoming unstable and the aftermath again we had too many errors and too much effort into fixing the errors in that environment and again essentially we gave up on this specific environment yet again and started looking for a different approach that would probably better fit our expertise with OpenStack. So then we went back to Mirantis and tried Mirantis Express so Mirantis Express is a hosted private cloud and it seemed like a very good option because it actually did not require too much resources from our hand to maintain and bring to bring up maintain install configure and then fix this environment. Maybe not too many people knows what is Mirantis Express. Yeah so Mirantis Express is a hosted private cloud solution basically providing you the infrastructure needed to set up and everything is needed to set up OpenStack and that is a hosted environment so that's not on-premise but that's not but also does not have like the downsides of a public cloud so essentially you get a set of servers and they set up OpenStack on top of that and help you maintaining that environment. So our experience with that was pretty good it was a quite a quick installation so we were up and running in quite a few hours in that environment and started moving workloads into that environment and you know gave our R&D members permissions and access to that environment they started working running workloads there and everything seems rather fine for quite a long period but again you have their current theme of a but on the end of the slide. Cloudify started breaking this environment as well and also we had quite a lot of issues we open with Mirantis some of them are specific some of them are very you know the issues that existed upstream in OpenStack itself some of them are configuration things that we needed to accomplish in order to get things going. So Cloudify definitely breaks OpenStack again is a nice and nice logo there and I think that again it's we're hitting a lot of those issues of that are result of creating and destroying creating of destroying environments and in that specific case we brought that environment to a point in which it was unrecoverable and so the only way to go around that it was to wipe up the entire thing and to create a new accounts and whatever which kind of brought us to a point in which we realized that that's not the right path. So next was public cloud back to public cloud again we reached a conclusion you know where you have an OpenStack of your own you're probably gonna get gonna get less attention to it from the experts because you forgot to mention that we're probably the only user of the HP OpenStack. Oh yeah. So we're the one. Up until the end the soaring end of moving all the resources from there into into our own environment. Yeah so back to public cloud so we started moving we started testing data data center so data center is a OpenStack based public cloud offering they're based out of the UK so you know distance wise it was quite okay and bandwidth what was was quite well to our offices in Israel and we started doing a lot of tests against it started working giving more and more access to to our developers to that environment and their environment was running a quite well quite stable environment you know compared to compared to what we've experienced beforehand and so something to be said about experts so definitely you know that was quite a different experience from everything else that you experienced so far until the fully public cloud offering so you know letting the handle the experts handle it and when you have like guys that are solely focused on maintaining and keeping a specific environment up and running as a production environment definitely made a huge difference you know compared to things we've experienced beforehand where we actually had to maintain the environment or we were relying on a small team that are probably were focused on several environments so quite a difference there and there's a still but on the on the presentation there and claudify did not break this environment entirely but we did have some weird experiences mostly around keystone and the and the load we were putting on keystone apparently keystone was not expecting to to handle such loads as claudify created so and there were some configuration changes they did and the day center did in in in their public offering to better support keystone and some may kiss some a quarter changes needed to be done at for for claudify to work well with with data centered so so definitely claudify breaks open stack a lot and I think you know this also brings some value back to open stack itself where you know these issues that we've encountered you know hopefully and hopefully are you know getting fixed and definitely you know as we go along with time where we're seeing that some of the issues that we've been encountering actually we're encountering them much less and less which you know kind of tells us that these kinds of things are getting fixed and these kinds of issues you know whether they're coming directly from community or from a vendor are getting attention and are getting fixed so open stack definitely you know gets more and more stable as time progresses and it's actually the data centered solution you know works quite well how many people here are familiar with platform 9 okay only few so I'll say a few words about what's the concept behind platform 9 platform 9 basically have a very interesting concept that we like a lot which is that the management gear if you like of open stack is hosted but the actual resources are on your premises or on your own if you'd like a cloud it could be also a public cloud and you install an agent and that's it so it provides a very simple way to run open stack and it takes a lot of the bear but still keeps the benefit of having relatively high control as opposed to a public cloud for example and I think that's we're actually the first beta or one of the first beta to actually try platform 9 so a lot of experience was part of that and unfortunately really unfortunately we couldn't really get past the point of setting up platform 9 under our environment after we investigated that I think it was to do with the with the data center that we had and the devices that we had the type of devices that we had and I think when we'll get to the lessons one thing that I think needs to be clear is that it's very hard to get open stack to run on all kind of environments and all permutation and all configurations it's probably not gonna work and so when you're picking on a platform you better know what is the recommended environment what is the recommended hardware gears that you have to have to actually make it run that is tested that you know that will run and in this specific configuration it didn't work we don't necessarily know why it didn't work we just know that it didn't work then we kind of passed to the next level yeah so talking specifically to platform 9 you know initially it seems like best of both worlds so you have all the servers in your data center next to you and the manager the controllers are basically managed as a service by platform 9 which is you know perfect right they managed they maintained the whole environment itself and the workloads themselves are really close on prem and actually like we do have some level of knowledge on what went wrong there so and the point where we started playing around with with platform 9 was where where they entered as what they introduced neutron as a beta feature and we really needed neutron and some specific features from neutron and very specific villain configurations needed to be supported from our hand by neutron so so that was a really a blocker for us to test neutron test the platform 9 and you know we definitely do hope to to try that again and hope that it would work in the next iteration because it does seem like promising the best of everything we need as an R&D but also but also like providing the stability without paying the costs sort of say of maintaining a production open stack environment so we're gonna talk a bit about rock space yeah so the next the next level was to go private hosted as you can see we almost tried every option on earth of using open stack so private hosted give you the benefit of someone else you basically outsourcing the responsibility of managing open stack to someone else and all the issues that we experienced we're hoping that someone else will experience or at least will be able to complain to him and he will fix it and and that's kind of the path in which we're pursuing right now that seems to be the most promising until the next open stack summit and and and again again I think that from at least a skillset perspective we expected to run better now it's also hosted in a local operator so the latency issue is not going to end the security issue and then we had we're accessing it through VPC kind of connections so we have secured access to that environment and we have full control over the infrastructure itself so it's supposedly going to be much more stable from our experience so far it looks very promising and still very promising yeah so definitely hope that this would work out next so yeah so so I think that if I kind of try to touch on that and there is other kind of guys here that are doing products on top of open stacks I want to hear your lessons and what are the type of environment that you've ended up working with if you ended up working with is that first of all I think that we can see that there is a lot of options to run open stack there is we started with Dev stack you could run open stack with distro without distro in a specific infrastructure without a specific infrastructure in a semi-hosted like in the case of platform 9 there is option to run it as a service but still private service like in the case of open stack express and there's also the option to run it on a private hosted environment so first and I think that's kind of an indicative of also a problem that you know there was no clear blueprints that says this is how you run open stack and this is how it's going to work I think that only now in the past I think last time it was so there is this type of blueprint that says okay it's probably not going to work if you're going to run open stack and expect it to run on all options and everyone going to bring its own gears and it's going to work and so the vendors itself and the distro themselves are getting to the point in which they don't just give you a distro they're telling you how to run it and which type of infrastructure including appliances themselves so I think first lesson is that don't try to run open stack yourself second if you're picking up and that's your business and for definitely for a product company even though we need an intimate knowledge of open stack that's not where we want to waste our time and we do we need to do other things as well the second part is that between a distro and a private hosted it's really a questions of where you feel more comfortable the if you need the benefit of a distro is obviously is that the installation is simpler but you still need to manage it and a private hosted you're also seeing that hosting environment the problem with hosted version is that it's if you need to run multiple versions of open stack it's harder to deal with that because then it's all outsourced to someone else and if you want a release cycle and you're saying that I need to now run a new version of open stack that's not going to be that possible so you probably still need to have some sort of flexibility somewhere else there is another option that wasn't mentioned here that we're actually using right now when we announced for example an NFV lab this week which is ravello which had really allowed us to create that type of POC environment on demand so whenever we we need to get to a customer engagement one of the problem that we had is that we needed to create a private environment for a specific case for specific POCs and running it whether it's in public or a distro or hosted is not going to be possible definitely if you want to share it with another user so we ended up using that so the result that we have today is that we are actually running some workload definitely the scalability on a public cloud data centered and we're going to continue to do that we have basically the hosted private cloud that we are running in open stack in the rack space and that's for the continuous if you'd like development that's for the most fixed workload type of testing and we're also using if you'd like some portion of open stack as I mentioned on the ravello environment to actually run a POC environment that we're using so we're using multiple options or from from the list and there is no one open stack that we're using that covers all of our use cases right now and that's where we ended up and I think that's kind of the the point maybe to open the mic to people here in the room who is developing a product on open stack I think there were a few don't be shy now there was at least 50 percent of the room so yeah exactly yes okay so you're from emc okay so obviously you're running on your own gears what's that okay ourselves if those guys will touch who else is developing products on open stack so what kind of products you work on guys yes there's someone there's someone brave yeah I'm from Nokia and we're developing a very similar product or at least we have been developing it for some years and we speak up a little bit yeah we encountered whatever we encountered very same issues as you did so for example the floating IP not reaching the the VM and fortunately we do not have to run it in production it's it's for the bad guys and and for our CI systems it was enough that we created several development crowds and whenever it broke we just scraped it and changed the CI to use another one yep that's the best option yeah destroy and create it from scratch excellent thanks who else yes so I work with we work with 40 net we're a next-gen firewall provider and hardware and I'm just wondering you ran into a lot of problems with Neutron and the networking piece it seems like there was a consistent theme through all that yep you know a lot of our customers are wanting us to to go virtual with these high-speed platform UTM next-gen features but what's your opinion on is that really there yet is that a need for that or did you guys find turning your networking services over to hardware ultimately this was a better fit so the question is should the recommendation would be to go with the virtualized network or hardware well a lot of your you just mentioned Neutron kept yelling on you so what did you guys do did you ultimately just end up going to hardware for all your networking or did you guys figure a way around that so I think it depends on what kind of product are trying to develop you know in our example cloudify needs to orchestrate networking as well so Neutron is very important to us and very important to us to have full control over how Neutron works so this is why we didn't look into the approach of you know having a physical specific setup of networking that would be you know very very very hardwired but if your business is around you know focusing on on cinder volumes for example or on VM specifically not you know and not bound into the solution to networking probably like going to the hardware approach would be would be the better the better choice there but I think that to your question a lot of our customers the enterprise customers specifically and also the carriers I know do choose their own if you like hardware network gear that is not the default one that comes with OpenStack and that's that's for the reason that I think a lot of the people in the room here are pretty familiar with that Neutron at least in its current in coronation is not stable yes I work for Rackspace so are you guys using RPC like the our private cloud stuff yes it is the private cloud stuff but it's the private hosted so it's we're not using it directly we're using it right right we're doing the ops types yes exactly using it indirectly so on the back end for us it is by default going to be just plain Neutron with Linux bridge and if that's working for you and then at least right now yeah so you probably did yeah we recognize that OBS kind of breaks and Linux bridge though is pretty stable and doesn't like to reset on anything like the level three stuff getting restarted was an issue yeah that's yeah I think that's a fairly that's a very good comment you know every every choice has its own you know advantages and setbacks so you know obviously it has less control so you have less things less less less types of configurations you can support but definitely that's more stable so that's the trade-off but yeah and and I think it continuously runs over time that's in itself will continue to be an issue especially when you develop new features and new services so that means that basically the maturity cycle is going to be longer as we've experienced with OpenStack so I think that's remain the major problem I think at OpenStack is that it does get stable and we are getting into production and I think now with the fifth release we're in in 50 years we're actually getting to the point in which we could see a lot of production workload but the cycle is very very long and there is a lot of iterations and it takes a long time until things get matured I think that remains a problem that the community have to figure out and in my view the only way to do that is either standardize on a specific kind of gears that says this is how it's going to run and be very specific about it second so that you could even test it and know that what you're testing is what the customers are going to use and not pitch it as a general purpose thing that you know install it and it would work hopefully and the second part is that it needs to run over time and the feedback loop needs to be much shorter which means that there needs to be some sort of environment like obviously in the case of Amazon that those tests would even run for a long time and over time and manifest much faster into the development cycles of OpenStack itself and I think the fact that there is starting to be a new wave of public cloud of OpenStack I think is a good thing because in a way those are the guys that probably you know the rack space and data centered of the world would send those feedbacks and will contribute back or at least fix those type of long running issues that otherwise you wouldn't even see them so that's kind of the positive side of that anyone else wants to share his experience or his comment any additional questions maybe all right so we are not going to hold you anymore from the OpenStack official party unless we want to do it here all right thank you guys thank you guys thank you