 Welcome, thanks for coming today. We're here to talk about the challenges that we faced in contributing OpenStack support to the open source project, Spinnaker. My name is Emily Burns, and I'm a software engineer at Veritas. My name is Ashley Casim, a software engineer on the same team. Ashley and I work for the Continuous Integration and Continuous Delivery team at Veritas. We're responsible for providing cloud tooling to Veritas' cloud engineers, and we help them get their code from source into production and deployed in the cloud. So today, we'll talk about why Veritas is using the cloud. We'll talk about why Veritas cares about Spinnaker and why it's good for the OpenStack community. We'll do a demo of the functionality that we help contribute, and then we'll talk about our journey. At the end of the talk, we hope that you'll learn why we care about Spinnaker and the value that it can bring. And we will also hope you learn how the OpenStack community can be more friendly to tooling integration. So why is Veritas in the cloud? Veritas is traditionally an appliance company with its product NetBackup. NetBackup helps customers back up their data in their physical data centers and then restore it in case of a problem. But our customers are moving to the cloud, and so their data is moving with them. We want to protect their customers' data wherever it is, so Veritas is also moving to the cloud. Additionally, we want to help our customers get value from their data, so not just back it up, figure out what their data is and how they can effectively use it. So being in the cloud gives us the ability to host applications and then scale them up and down as needed, and also to provide software as a service to our customers. So our infrastructure is a hybrid cloud. Our development environment is on-premise data centers all over the world, and they're running OpenStack. We do all of our appliance development in these data centers, and we also have dev environments, and we build code. We are container workloads run on Kubernetes on top of OpenStack. Our production environment is the public cloud. So we have most of our production in AWS, but some workloads in Microsoft Azure and Google Cloud. Our container workloads we run in Kubernetes on top of these clouds. As a cloud tooling team, our goal is to provide a consistent interface for all of the tools across all these environments so that our teams can use them effectively. And we also have to balance adding new tools and new features in that our engineers aren't familiar with and keeping the environment stable. So imagine you are a software engineer at Veritas. Here's what your deployment pipeline looks like. You start off with your code in source control, and we use Atlassian Bitbucket. Once you commit, your code is built by Jenkins into the appropriate package. And we like Jenkins because it provides a lot of different integrations into different environments, and you can build a lot of different things. And those integrations with these environments make it really powerful. Next, your code is pushed into JFrog's Artifactory. And we like Artifactory for a similar reason. You can have a lot of different artifacts stored in there, and our developers access it from the same UI. These integrations, again, make it really powerful and useful. So once your code is in the repository and built in the appropriate package, it needs to be deployed into your development and production environments. Again, we have a hybrid cloud. So we evaluated Jenkins for this continuous deployment task because a lot of people like it. But we found that it was hard to create robust scripts that worked in all of these clouds. We had to do all of the error reporting and natively talk to the cloud ourselves. And that was frustrating and brittle. So we turned to the open source community, and we found Spinnaker. Spinnaker is a continuous delivery technology, open sourced by Netflix in November of 2015, and it is their replacement for the project Asgard. Have any of you guys heard of Spinnaker? Maybe at the keynote on Tuesday? Cool. So for those of you who don't know, Spinnaker is a continuous delivery, and it has two main functionalities. The first one is infrastructure management. So Spinnaker can connect to the cloud provider of your choice and auto discover what's in the infrastructure, just like Spinnaker was managing it. And so you can create new VMs, containers, load balancers, security groups, and you can destroy them and you can manage them. But that alone isn't enough reason to use Spinnaker because you can do that from each cloud provider console natively. Where Spinnaker gives you real power is with deployment management. So you can tell Spinnaker with pipelines exactly how you want your artifact to be deployed. You can tell Spinnaker that you'd like your artifact to be packaged and built into an image and then deployed into a canary cluster. From there, you'd like Spinnaker to run tests, and then if those tests pass, to deploy it into your production cluster, disable the previous server groups, wait for a while and make sure that that's actually a good version, and then destroy all of your other server groups. And you can build any pipeline logic that you'd like. This helps your team encapsulate the infrastructure as code principle so that all of your developers can use these best practices and you don't have to SSH into your infrastructure and configure it. And so it allows you to deploy really fast and with a lot of confidence. So we were impressed. And as you might imagine, Spinnaker fit into our deployment pipeline. So at the time we evaluated Spinnaker, it supported AWS and Google Cloud and Microsoft Azure and Kubernetes, but it didn't support OpenStack. And so this is where Ashley and I come in. We reached out to the Spinnaker community and we asked the users there if they would like support of OpenStack. And we got a big yes from people. That would be something of value. So we found engineers at Target who were also willing to contribute this functionality and had a similar problem. So we collaborated with them to contribute the first OpenStack support to Spinnaker. So now I'd like to thank them. They can't be here today but thank you to our collaborators from Target, Daniel, John, Steve, Kevin and Edwin. We appreciate all of your work. Additionally, thank you to our coworkers from Veritas who couldn't be here. Thank you, Jeffrey, Sashrut and Mukesh. So enough talking or slides. I'd like to give a demo of the work that we've done. Do you know how to start the video in presenter mode? Yeah. Okay, so as we attempt to start this, perfect. This is the application screen of Spinnaker. So this is where it drops you in and you can see that I have configured a hybrid cloud. We've got some OpenStack and some Kubernetes and all of these applications on the left-hand side have been either auto-discovered from the infrastructure or I've configured them. So you can see on our OpenStack console we have a bunch of heat stacks that have been created by my teammates and I and those are discovered within Spinnaker. But we're gonna dive into one specific application, our OpenStack Summit application. So when you click on this application, you're dropped into the clusters page which is all of your infrastructure. You can see here I have one cluster that is disabled and one that is enabled. On my enabled cluster, which is showing us that green chocolate, it's healthy. I have my super complicated demo application for us today. And so you can see Spinnaker gives you information about this infrastructure. So if we take a look at the application, we will see that we have, hello OpenStack from Veritas. But we have a problem here. This is the old OpenStack logo. The OpenStack logo recently changed. So we need to change our application and we need to do this in a rolling deploy because we have a lot of users going to this and we really don't want any downtime. So we're gonna make use of Spinnaker to do that. And normally I do a code change and then I'd bake the image and then I deploy it. But for the sake of time, I have the image already pre-baked and I'm just gonna show you the deployment piece. So if we go back to Spinnaker, we can see our Pipelines page. Here we have our deployed new webpage pipeline which we're gonna first kick off and then I'm gonna talk about the configuration of it. So we're gonna give it a go. You can have this triggered via Jenkins or an automatic trigger, but we're just running it manually for today. So you can see that this has already started and it's giving you first green and then it's going on. So this is our configuration. We don't have any first configuration steps. So the first thing we're doing is finding the image from the cluster. Spinnaker is finding the image that we wanna deploy. The next thing Spinnaker is gonna do is actually deploy that image into your cluster in the new version. So this is the configuration for deploy and you can see that we've selected the provider of OpenStack. We've selected our OpenStack account. We wanna be in the same demo cluster and the same region and same subnet because we're replacing this old version. The next thing we're gonna do is disable the previous instance because we wanna make sure that it's there up and running in case we have to change the logo back, but not taking traffic. So we're going to select our demo cluster and keep the one newest server group enabled, our newest version. So this is a simple pipeline, but you can do many more complicated things with Spinnaker. Simple for demo purposes today. So when we check out our Pipelines tab again, we can see that this is our pipeline view and so we have our first stage, which completed really quickly. It's finding the image. Spinnaker knows what it's gonna deploy. The next thing is actually deploying that image. So you can see Spinnaker gives you an overview. So we're waiting for that instance to appear healthy on our infrastructure and it also gives you a detailed view of the tasks. So Spinnaker does all of these things for each health provider so we can determine our source server group. We can determine if the cloud provider is healthy. We actually deploy the instance. We monitor that deploy, make sure it's going well. All of those things. And so that's what's really great about Spinnaker. It breaks it down to tasks and it helps you do that. So here's our horizon console. We can see that indeed my application was created, my heat stack, version one. And we can see that below that we have version zero, which is the one that was already up and taking traffic. This new one was created one minute ago and if we jump to our instances page, we can see that those same instances are there or that one instance that we've deployed with our heat stack. And it was created two minutes ago. So if we go back to our Spinnaker view, we can see that the pipeline is still completing. It takes a while for it to monitor deploy and make sure your applications are up and registering is healthy. And for the sake of time, I've sped this up a little bit. So you can see that wait for up instances. Spinnaker has waited for them to be up. And then if you go to our clusters page, we can see that indeed our instance is up and taking traffic and healthy. The next thing Spinnaker is doing is disabling the previous server group. Again, you have the same individual task view where you can see all of the things that Spinnaker is doing. It determines whether your cloud is healthy. It disables it, it waits for that disable. It does a lot of caching on the back end so you don't hit your API limits. And again, I've sped this up for the sake of the presentation. So we can see now it's almost done. Spinnaker's almost completed. And our pipeline will be finished. We'll get all green. And so we didn't have to do anything manually. Spinnaker handles all of this. And in the case of an error, Spinnaker would nicely handle that too and alert you. So you can see that our pipeline has succeeded. We have all green. We're at the state we want. If we check out the clusters tab, we can also see that. Our new version is up. Our previous version is disabled. And our awesome application has been changed. So now we're trendy. We're with the times. And everyone can enjoy our application. So this was a really simple example of a Spinnaker pipeline. But you can imagine how you can create exactly the flow that you want in your deployments. And you can empower all of the people on your team to encapsulate the infrastructure as code principle and deploy often and really safely. And so we think this is really powerful and really beneficial for the OpenStack community. So now I'll turn it over to Ashley to talk about our journey. All right, cool. Well, now that we all know what Spinnaker is and how it works and why we should all be using it, let's talk about our process that we went through in contributing the OpenStack CloudDriver back to the community. This process started about a year ago and lasted several months due to some hangups that we will discuss later. And it's now released in the community and ready for people to try out. So let's start out by talking about the things that went well. So the first success we saw was that we saw pretty good community engagement. Within four months of releasing the driver, we were contacted by roughly a dozen individuals or organizations in our Slack channel that were interested in trying out what we had built. This is good. It means that OpenStack is relevant to enterprises and individuals using the hybrid cloud and that OpenStack's important tools like Spinnaker is something that's important to the community. The second is OpenStack seems to be under a lot of active development. There's new features and new services being released constantly, which is great if you wanna be using something that's cutting edge. However, there are drawbacks to this which we'll get into later. And the last thing is that we found the community to be very responsive. They responded promptly to our questions. And in particular, we noticed that the third party library support was very, very good. We used a third party library called OpenStack4j and we had to add several things to the library while we were developing this cloud driver. And they were very quick to merge our pull requests and to keep releasing the new features so that we could move forward quickly with Spinnaker. Unfortunately, there were things that didn't go as well for us and these fell into three big categories, the first development environment setup, component versioning and compatibility, and then documentation. So the first thing that we ran into was just basically how to get started. How do we get a development environment up that we can develop an integrating application with OpenStack on top of where we had all the correct components and the right versions and all of the services that Spinnaker needed in order to work? We didn't have any OpenStack infrastructure at the time because Veritas was transitioning from a very old installation we had and trying to move to something newer and more modern. So we started looking at the all-in-one VM solutions that are out there. The first one that we found was DevStack. DevStack is what it says on its documentation page that this is what OpenStack is primarily developed against. This means that it checks out all of the services at Master, the progress towards the next OpenStack release. This was problematic for us because since we wanted to develop against a certain release target, we had to find out how to get all these services at the right version, at the right tag. So this took a little bit of doing but eventually we figured it out with the help of several blog posts. But still once it was working, it wasn't very reliable for us and there were stability issues. We saw the HA proxy process failing a lot and when load balancers of service operations failed, they didn't always clean up properly which meant we had to go into my SQL tables and manually clean up them, not the greatest. It also was pretty slow in terms of performance and we had a lot of the timeouts in our client calls. So we decided to keep looking since DevStack was rather difficult to live with and the next alternative that emerged was PaxSack which is a puppet based deployment for OpenStack. So with PaxSack, we were able to just download from Yom a certain release and install. However, there was a lot of further configuration that was required. We found that all of the configuration values were pretty well documented on the GitHub. However, it wasn't always well explained what all the values should be. And this was difficult for us because we were new to OpenStack and we didn't have the specialized knowledge of an OpenStack administrator to figure these things out. In addition, once we finally got it configured correctly, we did see some discrepancies between what we were able to configure our PaxSack to do and what we actually got when we got our real infrastructure. And the best example of this was load balancers as a service V2. We got up and working in the PaxSack and then in our real infrastructure, we were unable to get it installed and working. So the end result of this is it took us way longer than we expected, like two months or so, to come up with a stable and reliable development environment. And moving forward, what we'd suggest to better enable application developers like us is to make these all-in-one VM solutions just a bit more easy and approachable to use. For example, the documentation could be enriched to explain the configuration values or perhaps sample configurations could be provided for each supported release version of OpenStack. Another thing that could be helpful that we've seen with other development environments is having a preconfigured image available in a public cloud like an Amazon AMI so that those who don't have access to OpenStack infrastructure or just don't know where to start have a place to go. So as we alluded to previously, we had a lot of trouble figuring out what components to use and the compatibility and versioning of them. The best example of this was, again, the load balancer as a service, which, as you can see, was something we had a lot of difficulty with, so it'll come up a lot. When we first started our journey in developing the CloudDriver, Veritas was using the IceHouse release, a very old release of IceHouse, and we were looking to modernize to something that was more in the supported zone. So we started looking at what services we would need in order to support all of Spinnaker's capabilities and load balancers are important key components of the Spinnaker. So we started looking at Sendlin, which looked like a good fit for auto-scaling. However, it implicitly creates load balancers, which wouldn't work with Spinnaker because Spinnaker expected them to be explicitly created. So this narrowed down things a little bit, and we also, at this point in time, got into a partnership with Target, and Target was using Kilo. So that's what we decided to target for our release version as well. Apologies, Apple is, there we go. This meant that we would go with load balancers as a service V1. However, load balancers as a service V1 had a lot of performance issues and did not perform at scale, and this is something that a lot of people had a problem with. And so load balancers as a service was deprecated in favor of load balancers as a service V2. So we decided to try to move to load balancers as a service V2, which meant moving our release target to Liberty. So this unfortunately did not solve the performance issues we saw, and it seemed to us in Liberty that load balancers service V2 was not fully matured and V1 was deprecated, and we weren't really sure how to do load balancing in Liberty. So we moved to Mataka finally in the hopes that load balancers as a service V2 would be more mature in this next release. Unfortunately, again, this didn't seem to be the case as we had the same issues and ultimately weren't even to get able to get load balancers as a service V2 installed and working on our Mataka infrastructure. So right now what we're doing and what you saw in the demo is raising console to do load balancing instead of any of the OpenStack built-in services. So just like we as the continuous delivery team have to balance the rapid addition of new features and stability and reliability of the release services, we suggest that OpenStack community take a more balanced approach to this as well. For us in the enterprise environment and those developing integrating applications, stability is often more important than being the most cutting edge. We need to ensure that new services that are added are adequately matured before deprecating the old ones. The final measure challenge that we faced was dealing with the OpenStack documentation. During the course of our development effort in the Spinnaker OpenStack Cloud Driver, we were required to add several features to the OpenStack for today client library that we were using, such as Barbican, load balancers service V2, and some parts of Heat. This meant that we needed to look at the OpenStack API documentations as a source of truth to inform us how the models should be sent in return, what headers to use, and other things. However, we ran into a lot of problems here where there would be incorrect or out-of-date information in the docs that kind of let us astray. And again, load balancers service V2 was the prime example here. The documented endpoints were different in spelling than the actual observed endpoints, which caused false positive test failures for our client code. This has since been fixed, but not by us. We had contacted the OpenStack community in regards to this, and we were told to open a bug rather than being enabled to submit a fix ourselves. So what we did to keep moving our development effort forward is we actually went and installed other OpenStack clients and looked at their source code to find a source of truth. And we used a debug flag of the OpenStack CLI a lot and then dug through the metadata to figure out what headers we should use and how the models are sent in return and everything. So what we suggest from this learning experience is that the OpenStack community should prioritize documentation as an acceptance criteria for every commit that comes in. In addition, we found the contribution process for small documentation changes like this one to be overly complex and difficult for people who are not regular OpenStack committers or maybe have never committed to OpenStack to do. So perhaps opening up documentation changes on maybe like GitHub Wiki or some other documentation management technology might help alleviate this and then also relieve the burden on the OpenStack community to be the sole source of documentation. And then integrators like us who are using things and find small issues could just contribute small fixes. So the ultimate takeaway from this is that enterprise and integrating applications need to the integration capabilities and hybrid cloud in order to succeed. OpenStack is a very important part of the Veritas hybrid cloud environment. And it's really the only open source thing cloud out there. And it has a lot of benefits over costly paid solutions. However, OpenStack in our experience is currently lagging behind close source competitors in terms of ease of integration. We would love to help fix this and see this change in the future because we believe that OpenStack can really be an empowering part of the hybrid cloud. So how can you help? Well, if anything that we've said at all sounds interesting to you, feel free to ask questions, come and talk to us after the presentation or contact us on the emails that are on the slide. If you'd like to collaborate with us in either the Spinnaker or OpenStack open source projects, that'd be great. We'd love to have you. And if you're interested in Spinnaker, the Spinnaker Slack instance has a thriving developer and user community. So you can join that at the link on the page. Thank you. And at this time, I'd like to open the floor for any questions. Hello, first of all, thank you for your work. Thank you for giving it back to the OpenStack community. One of the big advantages of Spinnaker is that it gives you a holistic view of everything. You have both the build process and the view of your cloud. So this advantage is offered by Spinnaker by, as it's a moment, there is no alternative to this. Unfortunately, or fortunately, Kubernetes has ended the scene. So now it's better for me to have a view on the third level as well. So not that just my VM is up, but my Kubernetes cluster is also up. So are there plans to integrate some Kubernetes support in Spinnaker so that you have a view that I finished the deployment, my VMs are up, and my Kubernetes cluster is also up? Is something that is interesting to you? Yeah, so Spinnaker support is already, sorry, Kubernetes support is already in Spinnaker. And so what you're talking about is spinning up the VMs which have Kubernetes, right? And then Kubernetes is discovered back in Spinnaker. And we thought a little bit about this, and I'm sure that the Kubernetes channel in the Spinnaker Slack would be interested in that, but the way that we've thought about doing that is doing federated Kubernetes clusters, and so they're kind of discovered by the main cluster and then discovered by Spinnaker, but we haven't tested it out. So if you try it, let me know. Yeah, the way that the Kubernetes is currently treated is as another like cloud provider in Spinnaker, so. Thank you. Thank you. I wonder if you can talk towards any of the other cool CICD pipelines that you're starting to see used or just kind of the simple example is great, I really like that, but I'm just wondering what other ideas you've had around CICD pipelines. Yeah, I can talk to that. So the question is about other CICD pipelines, and one of the coolest examples is actually Spinnaker can upgrade itself, and so we've tested this out a little bit, and this is what Netflix does, so Spinnaker will deploy a new version of the microservices that are under it, and then wait for that to be up and healthy and then kill its old self, and so that's really cool. Some other things that we've implemented are like deploying and then handling like when the version is not good, and so rolling back and then doing that cleanup within the pipelines, you can do like conditional logic and you can detect like cluster size or whether things succeeded or failed, and you can also do manual approval, so you can like gate your production environment and things like that. Are you able to integrate it with any analytics? And the example is something like a canary deployment where you wanna just put it out to your top five or 10% of users or something, have you guys played around with that at all yet? So you can actually create custom deployment strategies. This is something that I know is like the canary thing is on a roadmap, but it just hasn't, we haven't implemented it yet. Yeah, but the Spinnaker community does and there's like lots of chatter about that, and so you can do like at a very like simple level, you can hook into Jenkins with Spinnaker natively and so you can have Jenkins do any analytics and then if, and put your logic in there and then have the Jenkins job tell Spinnaker whether it succeeded or failed and then based on that Spinnaker can move forward. So that's like a really easy way to get started and get integrated with your existing infrastructure and your existing strategies. And in terms of a lot resiliency, another interesting application of Spinnaker is that's how the new version of Chaos Monkey is now running through Spinnaker. Chaos Monkey for those of you who don't know is a tool also open source by Netflix which randomly kills off all your infrastructure and then sees if you can survive it. Any other questions that we can answer? All right, if there was something you didn't want to ask feel free to reach out to us. Feel free to test out Spinnaker. We'd love to have your help contributing or just testing it out. Yeah, we'll be up here for a couple of minutes afterwards. So thank you everyone for coming. Have a nice day.