 Good morning everybody welcome Thank you for joining us today. My name is Matt McEwen. My name is Mark Burnett We are core members of the airship team and today. We'd like to talk to you a little bit about deckhand First we're going to give you an overview of Airship and what deckhand is and how they fit together Then we're going to go through deckhands features one by one so that you can see them Sort of how the magic happens And then we're going to go through use cases both airship and non airship use cases for deckhand And then we're going to have some Q&A, but please feel free to interrupt us The lights are bright up here, but I'll try to watch for waving hands and pause Feel free to interrupt us So before we get into deckhand Let's talk a little bit about airship as a whole to so you can see how they fit together airship is a platform that lets you take a collection of yaml manifests that Define your site so a many of them are Literally kubernetes manifests Many of them are inspired by kubernetes manifests because kubernetes did a really good job With their declarative yaml based approach and that's kind of what we extended For the rest of site provisioning with airship, so we take all we take a git repository of these yaml manifests We feed them through a front door api One of our components for airship is the sort of the front door api that orchestrates all of the activities involved in standing up a site and then All of the different actions like provisioning bare metal like Standing up kubernetes a resilient kubernetes cluster like defining the networks and the network policies And then deploying all of the software that goes into a full open stack Distribution like open stack itself as well as logging and monitoring Etc. And it does all of this by starting with containers as the primary units of software delivery and Helm charts as the I should I should say probably the only method of software deployment We are we have put our our our money on we have bet on the the horse of containers and helm charts with Airship, this is where we are opinionated So was there a question Okay, so you can see One of the different Airship components in that little box up there. You have dry dock shipyard deckhand diving bell armada and promenade that all work together They're all you know different Projects that can be used individually or together And deckhand is the one that manage manages the yaml manifests inside of the cluster If everything in your site is defined by a yaml clearly yaml is our central to what you are doing So I'm going to give you a little bit of background on how on how this platform has developed so We've been working on it for close to two years now And the first component that really had any any code was armada And that's the component that you may have seen in other talks where It's used to coordinate the deployment of multiple helm charts. So the initial work on open stack helm We quickly realized we needed something a little more than what helm offered for dependency management and multiple chart management And so that's where our moda came from And then very quickly after that we started working on dry dock for bare metal provisioning Which is really just a declarative front end for for pluggable back ends where the main back end that we're using is mass And then we started working on promenade, which is the resilient kubernetes deploy deployment tool And so in summer of last year We were doing a proof of concept where we pulled all these things together and and made it go We realized while we were working on this POC that Actually, we were becoming overwhelmed by the amount of configuration that we had so yaml is great. It's structured But we had a lot of it And in particular, there's quite a bit of duplication, right? So you need some of the same things in your chart that defines neutron and nova component configuration As well as in the bare metal nodes as well as In promenade for networking and node names and so on so we realized that we were getting surrounded by config And so we needed to consolidate this We cared very much about keeping the components separate We didn't want to just have airship config dot yaml and it'd be one thing We wanted you to be able to run drydock by itself without promenade promenade without drydock, etc And so they needed really to be independent which meant we were going to accept that there was going to be duplication in the format of Configuration documents that these things consumed so what we decided to do is build a new component that would address a few of The concerns around this it would help us with deduplicating this information Which is critical with how much we were becoming overwhelmed and also The the belief was and I think this has worked out pretty well that we could isolate Encryption of sensitive data so certificate keys come through these yamls as well and deckhand Manages securing those and the components don't care about it. They received them in clear text over TLS So yeah, so deckhand is really central to the platform. These are the five components again And the config while still large is much more manageable Routed through deckhand, then I think it would have been otherwise So again deduplication was the number one thing that we were concerned about while sticking to the to the idea of Keeping it independent a secret management. I mentioned Passphrases certificate keys are the two prime examples and we really wanted to keep a history of the configuration in the site We from the beginning were planning to deploy a very large number of sites and So it's one thing to have a get repository of your yam of your configuration saying this is what's going into the site That's just your intent right and that's great But how do you know what is actually been there? So we really wanted to keep that history in one place on the site What's been here? and so that was a those were the sort of the main features that we wanted and we had a few opinions about it or Things to go about here. So the configuration has to live on site It has to be immutable and we didn't want to impose a lot of restrictions on What the configuration for the next component was going to be like? and so given all these We came up with a Kubernetes inspired document format, which is Sort of a push versus pull model. So rather than doing templating like a number of configuration systems do We have structured data block at the bottom, which we'll get to in a minute so you define a schema for your particular type of configuration document and so These are namespaced. You can see example foo v1 might be like Prominade cooblet v1, which be cooblet configuration owned by promenade And so maybe I'll just click these And then we have a metadata Section which is quite large in some cases and this this supports the push model where we don't We don't pull via templating which I'll show in some of the features later But documents all have a name and the metadata has a schema Which is just a sort of a core part of it, which we'll get to later as well But the metadata section in general contains quite a few things labels again inspired by Kubernetes for selecting documents Details about feature usage. So here we have layering definitions substitutions Sort of deduplication features, which we will talk about in details They're all defined in this metadata section and and deckhand defines what this looks like this metadata section is sort of owned by deckhand and structured and Then so storage policy clear text This would be how you indicate storage policy encrypted would mean hey You want to treat this like a secret and stored encrypted fashion? But the data section is where we're unopinionated. This can be anything. That's valid yaml Okay, and you can write a schema for it a JSON schema style thing for validation so all right, so the first Feature that deckhand supports that is relevant for this is on solving those problems is substitution and Substitution is Something that takes two different yaml documents and takes data from one and puts it into the other so up here on top You have some data In the source document and then on the bottom you have your destination document and the way it works is You know well up here you see the the intention right the data that's going to get Get merged is or substituted rather is that key one value one and then in your destination document you have a Stanza in the metadata section that defines where you would like that substitution to come from and where you would like it to go so the source defines Not just the document, but also the data that you want to pull from that's what you why you see that little link there and then It puts it into the destination in the resulting documents the rendered documents that you get when you Take the the piece parts and then put them together So on the right you see the rendered documents The One important thing is that the two documents here don't have to have the same schema They don't have to be the same type of documents and in fact that's the way it normally is Usually you have A document that sort of lets you define in one place a Particular piece of data and then many different Schemas or many different types of documents that you'd like to substitute that value into so to make it a little more concrete The way that in our reference manifests that we have in the airship treasure map project we have a We have a file that has all of our containers Defined in one place all of the the different Repositories for them and the different hashes for the versions and those sorts of things right and then those that one File gets substituted into all of the different all the different files that need to know about containers which in airship is Many of them And so that's that does two different things First first and foremost. It is deduplication right so many of those containers are used across many different helm charts and the ability to Define them in one place gives you the flexibility to To specify them in one place and then have that get replicated out to many places The second thing that that does is just from an Organizational standpoint it lets you combine similar data together so that When you go and update The versions of containers that you're integrating into your sites integrating into your CI build There's just one place you have to go to make all of those changes and it cuts down on the guesswork Yeah, and this is an only for version management We use substitution anytime you have something that you are likely to intentionally change from site to site So it's used for like the control plane host names and IP addresses and stuff which go into a whole host of documents Related to certificates and all sorts of stuff you can imagine so that's that's one of the main purposes for use versions are a fantastic Example a big one And it's the kind of thing it's like we want to intentionally vary this so we're gonna use substitution here and so that's and you know going with the other example You know endpoints for example right where you have a lot of things that might go into one of these You might have a host name and a port and a context right Let's say that you had a You had one string one endpoint string that was generally the same But in different contexts you might want to use a different host name deckhand also supports substitution of a substring of a field using regular expressions so you could swap out Just the host name part of a field in your your yaml And then get a little bit more deduplication out of it a little more simplicity And up here you see one example of a of a substitution right you you see a particular Field that is getting substituted in if you can have as many of those as you want And they would just be additional items in the Substitutions list in the destination document and it you're also not limited to just a single a single key value You can substitute in you know a tree which we also actually make quite a bit of use of for OpenStack Helm charts Which use an endpoints format that's quite large, but repeated All right the next key feature of deckhand is layering Layering lets you split out your your yaml into things that are Common and things that are site specific That at least that's sort of the the main use case for it. It can be used for other things as well it is essentially a YAML hierarchy or a YAML inheritance model and so like I mentioned if you have YAML documents that are mostly the same across all of your sites then this lets you capture all of that sameness in the the parent document and then the child documents can inherit that and Specify in a very small child document Just the things that are site specific so like the IP addresses and the host names that kind of thing so the way this works is You the child will reference out to its parent using a label-based selector And here you see us using a single label Arbitrary colon label that that just means it can be anything it can also be more than one label that you join on This is essentially a join kind of operation right so you specify that and then the You specify sort of how you want the layering to take place and then if you In the normal case where you merge entire documents the data from the parents and the data on the In the child are deep merged together Which is to say it's You can see the the entire tree structure of the parent is merged with the entire tree structure of the child and that is That is Actually, you're gonna say something well that is the main thing that we do there are a few cases Where rather than doing a site specific override, which certainly we do sometimes We'll override and delete certain settings because it's a test environment that doesn't have so you can see there It's method merge. That's the normal one. It means merge the data from the two documents But there's also delete and replace So those are the kinds of things you can do yep So merges merges the normal case But yeah, if you didn't have a delete then you there would be no way for a child to say I don't want any of this particular, you know branch of the yaml structure. You just throw it away And we have a question. Yay. Thank you for asking a question So the question was were we inspired by tree-based data handling tools that existed not directly the Yeah, only indirectly I would say I think the main conversation around how to do this was on some level about Should we do a pull model where you have some template and you're just pulling the data in or should we do something more structured like this? Where it's a pushing into it and I think the main driver for going with this Which you'll see later is that applications can specify data schemas as well here So rather than some I and I file that you're templating into which is hard to validate You can validate that the data fields are there as you expect I'm not aware of something that has all of that But I could just be lack of sight there I would say the main inspiration for this was the use case that we were solving for in that You know at AT&T, right? We have done we have done multi site deployments managed many many clouds for a long long time and have seen the good the bad and the ugly of it and so we really wanted to Ensure when we were starting from scratch with airship that we were able to to capture the majority of configuration in one place and the site specific information in another place in a very predictable way that if you set it upright is easy to continuously integrate and to test ahead of time and then to You know to take all the guesswork out of it as much as possible. So That is what that is sort of the problem That is the one of the big domains that airship tries to solve for and why you'd want to use this stuff is Is if you are using many many sites if you're using one site well airship is great for that too But layering isn't going to get you as much So to take advantage of layering You have to define a layering policy, which is another YAML document in your get repository of YAML documents But it's what's called a control document and all that means is it's some extra configuration for Deckhand it's not something that gets rendered out into the different components in airship It's it's special information just for deckhand to tell it how to do its job And so in this layering policy you see that we have three different layers Defined a global layer a type layer and a site layer This is Driven by this document so it's completely configurable and these three choices were what we chose for our initial reference architecture for for layering in our real-life use cases as well as in our our upstream reference Manifests and so the sorry and to be clear these were chosen Not based on on some fully flushed out Vision of what we thought the configs would look like it was well, we need to deploy a site How many layers do we want to use and so we're like let's use three this makes sense So we sort of went with that for a long time and we're looking very seriously at branching into many more layers Some more features around picking and choosing what bits you want. Yes How far down do you go? Do you say oh I've got four racks and Does each rack represent a layer or do you just stop at seeing a site? So great question to be completely clear here These layers are only used by deckhand to determine how to do that layering feature So that inheritance feature for the config if you want to talk about particular racks or what have you? That would just be part of your configuration So the names here are not actually important. They're just conceptual that the name is global or site It's just a conceptual thing. Yeah within within a site You other other airship capabilities make use of kubernetes labeling To you know sort of treat different servers differently if they're different types of servers and deploy different services to them the all of the different Definitions for all the different servers and racks etc. Would be part of the same site definition So yeah Global is what you think it is. It's the intention here is that it is where you put most of your configuration That is applicable across all your sites a type would be a It's kind of a flexible definition, but it can be sort of like a flavor of site, right? If you have if you have sites for one particular use case Like 5g and you have sites for another use case like CICD Environments that have Jenkins or Zool Running in them then those might be different types and then at the site layer you Define all your the small bit of site specific stuff and you can you can see from the table up here that this in practice Really does help This is a taken from one of our internal definitions that we use for all of our labs and our internal sites right now at AT&T and At the global level We have 24,000 lines of YAML definition at that level and for our primary use case site type we have three and a half thousand and then for a Particular site we have 1.8,000 so you can see that there is a order of magnitude difference between each of these different layers Which means that there is that much configuration that you don't have to work with your you know deployment engineers and Etc to do specifically on a site by site basis that can all be Shared and continuously integrated ahead of time right and the the data at the site layer is is All sort of boilerplate and IP addresses basically. There's nothing particular at a site. Yeah Yeah So the question was this looks awfully complicated for a small set of deployments again We are starting coming from the standpoint of we want thousands or ten thousands of deployments Okay, so we believe it can reasonably scale down to say three sites And I think we're not so far from that But I think that what we really need is better discovery tooling to help you write these site definitions We have treasure map which will will reference a few times as a working example of a complete site configuration for It's like an eight node site or something if I recall However, there's definitely I what I would consider the authoring of the configuration like before it goes into the site There's definitely lots of room for improvement there and yeah, and You know we released or we announced our release candidate For airship right and that we are working toward a 1.0 release And so what's the difference between a release candidate and a 1.0 release as a release candidate not production ready? Well, no, that's not the message because we're many organizations are already using this release candidate in production And it is very reliable The biggest thing that we want to achieve for our 1.0 release is to is to make that easier So you don't have to learn Too much YAML black magic to get started because if if anyone in the audience hates YAMLs I apologize you should close your eyes because just right Jason. That's fine And let me put it this way There is a lot of complexity involved in standing up any open-stack site and even more if you're talking about Provisioning an entire site and defining all the things that go into a real-world Infrastructure and so part of the design decision of airship is it simplifies a lot of that And it puts all the complexity in one place So really if the YAMLs are complex well a little bit But at least that's the only thing that you have to care about the only thing you have to care for Great, so Matt actually alluded to some of these already So Matt went over basically the the core deduplication features and that was what we rolled out with and we discovered it wasn't really enough So in particular Because we reference documents for in two different ways sort of either by label or by name essentially It turned out that sometimes well I really want to document with this name But I want it slightly changed from globals And maybe this is a result of not having ideal factoring of documents up front But it turned out to be useful or or you can imagine like wow this one particular site has this problem It's a hardware problem and I can work around it with a very particular config override in something That's in global or something so replacement lets you replace a document from a higher layer basically and With minor tweaks and so that's particularly useful for triage and so forth And we use it a lot less now than we did we've really restructured our documents, but it's still a useful feature multiple destination destination substitution Sometimes we were substituting a control plane IP into like five or six places in one document So this simplifies that reduces the amount of boilerplate you could always do it But now you can just list three or four destinations in one block of metadata And then even that was too repetitive. So back to the versions definition that Matt was referring to we have We host all of our images and charts really an Artifactory But it's not always the same Artifactory per site basically And so what we'd really like to do is go through this entire document and replace every occurrence of a particular string With our particular Artifactory host name. That's what recursive substitution lets you do If the name is not clear, that's the intent. So that's the primary use case for that just to make that fairly simple to do All right validation. So this is back to the choice to make this a Sort of a push model not a template model. Yes question Great question. So the question was a when does layering and substitution happen? The answer is at rendering time which is on demand So when something queries the deckhand API it performs a rendering operation and delivers the rendered Output of those yamls And that hand has just a regular get API. It's a restful API You basically say hey deckhand give me give me this document It checks to make sure you are who you say you are and then it dynamically like mark says pulls together all of the All of the the parenting does a substitution does a lot of work to get you one document Right and so substitution would include things like passphrases, right? So you store the passphrase document as encrypted But say our mod of fetches just all the charts that it's going to deploy it has the passphrase in line there It doesn't have to know that it's special can just do its regular work Yeah, so I mean I think you're basically I would phrase that is do we cash the results of rendering? Yes in memory not in database because again some of those things are sensitive. Okay, so anyway that was schema validation You write Jason schemas It's very simple. It's just that you can ask that can't to do this you submit the schemas like regular documents again They're controlled documents, which means you can't do substitution and layering basically Traceability, maybe we we were a little low on time, so I'm gonna go very briefly here So I mentioned before that declarative is great. We can store everything in git We have some authoring tooling or some sort of front-end tooling and peg legs That is before you get to the site and it gathers up all this configuration from various get repos It doesn't do this yet. That's why I drew a dashed line But you can easily imagine it querying something for certificate data or other sensitive data Some sort of vault API or whatever So there's room for that kind of growth And then it feeds those into shipyard where they go into deckhand finally to be stored and locked down in Barbacon for if there's secret data and then queried by all the components in the site So that's sort of the flow and deckhand tracks what was delivered And shipyard will actually annotate the releases the revisions so that you know what was deployed and so forth So that you have that traceability on site So this is my favorite My favorite deckhand feature Encryption and it is my favorite because it's that simple there. You have a storage policy In your deckhand metadata that just says on a document by document basis whether it is encrypted or in clear text behind the scenes deckhand uses Barbican for persistence of encrypted data and so like Mark said you'd use this for things like like certificates and keys and passphrases anything that's anything that's a secret that you would not want to persist to disk in clear text And we're very lucky that we did it this way so it doesn't matter whether you have structured data or not here because It actually just uses a consistent serialization and then encrypts it And so we have been asked by our security department today to encrypt other kinds of documents that we wouldn't have expected to be sensitive Or wouldn't normally be considered sensitive in some cases And so those are structured documents. So hey look we got that one for free finally, so And so The let's see the Sorry, this is not the latest We integrate with several other open-stack projects to realize many of these things We wanted to reinvent the wheel as little as possible. I mentioned that we use Barbican for secrets management. We also leverage Keystone For our identity service and all of airship uses Keystone for identity Keystone is one of the the charts that gets deployed in an airship under cloud We also use Oslo for policy definitions. So if you're familiar with defining a policy for Open-stack components defining policy for airship is going to be very familiar And then we use Open-stack Helm extensively Open-stack Helm is the Helm chart library for Open-stack Which is what makes it easy for us to deploy Keystone and Barbican and other components as well and we are You know Mark had mentioned that we leverage mass for Our as the plug-in for dry dock to drive our bare metal provisioning We want to add other components like ironic And the fact that there is already an ironic Helm chart out there gives us a great head start on that effort so use cases The you know we've been talking about airship in about deckhand in an airship context Which is the driver for creating it, but we made it very very reusable. It is a very general purpose YAML rendering and storage mechanism and so if you have heard any features today that Sound like something that your own application could make use of from a data store perspective Well, that was part of why we wanted to give this talk We are very interested in your use cases. Please reach out to us and you know join join the deckhand team or join our airship IRC chats to let us know of your interest Because we want to make this a very reusable component in the open-stack community it so As Matt was saying we're looking for more use cases here One of the fundamental things that we care about in the airship team is making all the components as reasonably independent as possible So yes when you do a full airship deployment every component queries deckhand, but guess what no component requires deckhand So that's good But generally speaking deckhand is sort of a core component You're likely to use it if you're using much other stuff from airship But even so that doesn't mean you have to use all the other components and so we're looking for more use cases for deckhand specifically The implementation is a little challenging in some places. I think there's a lot of room for Streamlining that so I'm hopeful there And the other thing is that it's actually like a little bit difficult We have a couple of projects who have had need of using a deckhand sort of client side engine Peg leg is the main example where we want to do validation up front offline before it goes into the site But that's actually a little bit awkward today And that's more of our project structuring issue that we need to work on But I think that's those are the sort of the main things going forward the deckhand So we have a few upcoming talks for the airship project in general Actually all tomorrow, I guess so please take a look and join us. We'll be happy to have you and answer questions You can reach out to us on the airship at IRC channel And check things out on airship it.org. Thank you. Yeah, hope to see you guys tomorrow. Thank you very much for coming. Oh question Okay, so the question is have you considered rather than using a source document for substitution that you could make an API call So that that makes sense However, I would say the way we would be likely to handle that would be to have some Sort of farther left tooling like peg leg clearly an API to find out that data And then send it into the site as a document So have have some tooling construct a standard YAML document that gets used as a substitution source Does that make sense? the reason for that is that we want it all to live on site and so if you're depending on some sort of You know corporate API that knows the IP addresses of your servers For example, you don't want to have to do that on site at the moment when you're reprovisioning a down-host Right that would be awkward. So so that's all front loaded And what if it changes? What if you deploy the site with a particular revision of your software and that data changes? How do you track that? Those are the kinds of things that we worry about and why there's no dynamic configuration pulled there If that makes sense Okay, I'm gonna try to repeat this as best I can so the context of the question was Let's say so peg leg, you know imports the deckhand engine to use primarily for validation And the question was as I understand it if it existed in deckhand as a feature then peg leg could use it as a feature, right? Okay True, but but it doesn't necessarily make sense inside deckhand in any context because peg legs not gonna Deliver rendered documents to the site. It's gonna deliver the complete bundle of raw documents to the site And that's deck hands job to render them on demand Because some of them need to be stored and encrypted and that's deck hands job To make sure that hint happens through barbacon and peg leg would if it rendered everything it would be spreading secrets all over all of it You'd have to encrypt the entire payload. No, thank you And and also if this sounds like a whole lot of different little components of airship Please go to the treasure map project airship dash treasure map Which in the open stack get namespace which has some diagrams to sort of show the different responsibilities of these airship components And how they how they work together. I think we have a minute for a question. Oh, yeah Is it possible with deckhand to divide your conflicts and secrets into different environments like deaf, broad and staging? Yes, you would only typically deliver the configuration for your particular site to the site So peg legs job actually is you have this repository it chooses it selects the configuration from your get repositories They belong to that site and ships them along So it's a really peg legs job to make sure that the correct set of configuration is delivered to the site And it's a deck hands job to organize that and store it properly Does that make sense? So how would you? reuse Secrets or let's say configs that are reused across environments Right, so I would consider your different environments like dev prod staging whatever to be just different sites They're probably all the same type with some slight overrides based on the differences So you spell it out. This is this is at least how we're doing it in our environments and So they're all sharing the same global and type configs their site configs are different But as you have seen that's smaller and it's like 95% IP addresses basically right and so That's that's how you would do it You'd grab the same global and type peg legs job would be to grab this global this type or well global is global I guess right, but and then the particular details for the site and ship that off Does that make sense? Yep. Oh cool And if you have other kinds of use cases that you want to talk through we'd be happy to talk through those Offline because you know, we've we've solved several different flavors of that same challenge And there are different approaches that you can you can take Thank you everybody. Thank you