 Hello, and welcome to Tackling Security Challenges with Airship. Today, we will be discussing the Airship project, significantly how we as a community break down security issues into several categories, and then work to keep the platform safe and secure for your use. Before we dive into the agenda, let me just briefly introduce myself. My name is Alexander Hughes. I'm a DevOps engineer with more than five years of experience, initiating complex projects that illuminate significant issues within an organization's infrastructure and operations. I have had the pleasure of working on the Airship project for almost two years now as a developer. I have also served the community as a member of the technical committee for more than a year, working to improve the user onboarding and development experience and making the project more secure. With introductions out of the way, let's go over the agenda for today's presentation. We'll begin with an introduction to what the Airship project is. Next, we'll discuss what common vulnerability exposures are and how to detect and resolve them. From there, we'll move on to discussing viruses and how to scan for them. After that, we'll discuss secrets management in the form of restricting and obscuring access. And once we've discussed those security challenges, I'll share the Airship security vulnerability management process. Finally, at the end of the presentation, I'll summarize what was discussed and have some time for questions. Let's kick things off with what Airship is. Airship is a collection of loosely coupled but interoperable open source tools that declaratively automate cloud provisioning and lifecycle management. Leveraging containers as the unit of software delivery. AT&T, SKT, and Intel jointly launched Airship as an open infrastructure pilot project at the OpenStack Summit in May of 2018. The Airship open source community released its first enterprise grade release, Airship 1.0, on April 29, 2019. I'm happy to say Airship is still evolving. We announced Airship 2.0 beta earlier this month. We'll be sharing more information on Airship 2.0 at the Airship 101 and PTG sessions, so make sure you check those out. Today, though, with the history lesson out of the way, we'll be focusing on those units of software delivery, containers. Each Airship component is developed in either Python or Golang and installs the executable code into a Docker image via a Docker file. These images are then published on quay.io for the community to see and to use. Now that we know what Airship is and a brief history lesson, let's talk about the next agenda item, defining and detecting common vulnerability exposures, or CBEs. A common vulnerability exposure is a vulnerability that was detected for a component or an application or a project, and the details of that vulnerability, including how severe it is, what the impacts are, how to take advantage of it, and if there's a fix available, are all submitted to a database. And that's where the next chapter of our talk begins. How do we use that database? How do we find those common vulnerability exposures? Earlier, I mentioned that all of our images are published on quay.io. Let's take a look at quay.io and how it can help us out. This is the organization page for Airship. This lists all the images that we have published to quay.io for your use. Let's take a look in particular at Peg League though, as this is the project that I'm most familiar with. The repository activity here just shows a breakdown of how much the Peg League project is being developed and how frequent we push updates. Let's take a look at the tags to see all the image tags that are available to us. Now this screen breaks down the report for us in a few different columns. On the left-hand side, we've got the tag. This is a SHA or a common name identifier along with the operating system that we're using for the base image. We've got the last modified column, which shows when the image was submitted to quay. We've got the security scan, and this is where my journey on Airship Security began and where I want to focus for the next few minutes. You'll notice that security scan has either a number of vulnerabilities that are being shown, or as is the case of this image, it shows as being unsupported. So why is it unsupported? Well, some images use operating systems that quay.io is unable to scan, in particular the open source images. But if we hover over one of the Ubuntu images, we can see a summary view of how many vulnerabilities there are in the image, along with the count of the most severe issues. So in this case, the eight mediums. Now if we click on this, we can get to the summary view of this image and all the details on the CVEs. So this page here gives us a list of all the CVEs that are applicable to this Peg League image. It also includes the severity of the issue, the package name and version that is afflicted and what layer it's introduced in. The most important of these though for you are going to be the package name and version, how severe the issue is, and that CVE identifier. That identifier gives us a link to look up the issue in the vulnerability database so that we can see what the issue is and how to patch it if there's a patch available. So let's look at if there are any fixable issues by clicking on this checkbox. We can see that all the packages that we're using right now are the current versions that none of them are able to fix the issue with an update. So these are things that we're going to have to wait on for a fix, which brings me to an important distinction. The packages that are shown here are dependencies. These are things that we depend on. These are not the things that we built. So that distinction is important. We have vulnerabilities that come from our dependencies or vulnerabilities that come from the things that we built. Let's take a look at how that distinction affects our project by looking at the different dependencies that are in Peg League. To do that, we'll navigate over to opendev.org so that we can look at the Peg League Docker file. This is the landing page for all the airship projects. Again, we'll go to Peg League. And in the images folder, you can see that there are several images that we support. We've been to Xenial and Ubuntu Bionic and an openSUSA 15 image for Peg League. Let's look at Bionic. Up at the top, you'll see our first dependency that this image is built using Ubuntu 1804, Ubuntu Bionic base image. And then all the app packages that need to be installed and Python packages need to be installed that support Peg League's use. It's not until the very bottom that we get away from our dependencies and we start seeing the things that we developed as being installed. And that occurs way down on line 65 onwards where we install our sister projects, Deccan, Promenade, and Shipyard, as well as the Peg League code itself directly into the image. So now that we've made the distinction on where our vulnerabilities come from, whether they're dependencies or things that we install, let's talk about how do we detect them. We could use the quay.io dashboard that I showed you earlier, or we could try to catch these vulnerabilities proactively before they're submitted for image repository. And we can do that with a tool called Claire. Before we begin with Claire, though, we're going to need to do a little bit of setup. So Claire works by comparing images that are already built. And because we're talking about Airship Security, let's do a compare and contrast between two Peg League images, one being Ubuntu Xenial and one being Ubuntu Bionic. And this will help us illustrate why our dependencies affect our vulnerabilities so much. To begin, we'll look at the Docker files for both Ubuntu Xenial and Ubuntu Bionic. And we'll look at what the differences are between the two images. Here you can see that the two images are almost identical, the difference being the base image used, either Ubuntu 16.04 or 18.04. And in the 18.04 case, we also installed GPG and GPG agent. Now that we have our two images set up, let's go ahead and build them. We'll build each image in turn beginning with the Ubuntu Xenial image and we'll tag it as Peg League Xenial to make it easy to reference later. And then we'll build the Bionic image and we'll tag it Peg League Bionic to make it easy to reference later as well. So now that we have our two images built, let's get ready to set up Claire. And we set up Claire first by getting the database, which contains all of the security vulnerability definitions. And then by grabbing the Claire scanning container and we'll link that to our database with all the vulnerabilities. And now that we have the two images downloaded and the containers running, we're going to need one last utility that helps us perform the scan. This is in the form of a script. And as we download it, we're going to make it executable so that we can use it easily. And now that that's downloaded, we're going to go ahead and begin by setting up the Docker gateway. When Claire does it scan, it needs to know what the Docker interface is. So we're going to do a Docker network inspect so that we can get that IP address. And then just to make sure that I captured it correctly, I'm going to echo it back. I expect to see an IP. Now that everything is set up, let's make sure that I've got my two images that I built earlier, the Peg Lake Bionic and the Peg Lake Xenial images. And then we'll begin by scanning the Xenial image first. Because it's older, I expect it to have more vulnerabilities. The way that Claire works is it works by scanning each image layer by layer to determine if there's any vulnerabilities, whether it comes from the base layer or the additional layers that you've set up as part of your Docker file. In this case, we have 199 vulnerabilities. Let's do the same thing for the Bionic image. And we expect again to see that the vulnerability count is lower, just by virtue that the Bionic image is newer. So it has had more time to patch vulnerabilities that were detected in earlier releases of Ubuntu. 44 vulnerabilities. So that proves our hypothesis correct. But let's see about the severity of each vulnerability. Let's look and see if there's any highs and there aren't, because we try very hard to keep our project secure. And in mediums, we should see that Xenial has more and Bionic has fewer. We'll repeat the count with lows. And we'll look for any negligible issues. These are vulnerabilities that are out in the wild, but they have a negligible impact. There you have it. So the Xenial image, as we thought earlier, has more vulnerabilities. And in general, those vulnerabilities are more severe. So there's more mediums than there are in Bionic and there's more lows than there are in Bionic. So now that we understand how to perform a clear check to determine what the CDs are in our image locally, and we could apply the same concept to CICD, to make sure that we reject any images published to our image registry if they have a high vulnerabilities or medium vulnerabilities, whatever you choose. Let's move on to the next subject of our talk. Let's talk about viruses and containers. Is it possible that we could have viruses in our containers? And how would we scan for them? Well, yeah, viruses are possible, but you'll be relieved to hear that there are no known viruses in the airship project. To scan for these, we can take two approaches. First, we can install a scanning tool into the container and inspect the container's file system using a virus scanning tool, similar to how you secure your machines at work and home. The problem with this technique is that it isn't very scalable. Launching containers, installing a virus scanning tool, and then scanning it is time consuming. So how do we secure airship containers against viruses? Enter ClamBake. ClamBake is an open source tool based on ClamAB. It works by saving target images to the file system as archives, extracting those archives into folders, and then scanning those folders. Each image is composed of multiple layers represented by various nested archives. The extraction and scanning process is recursive, scanning from the top to the base layer, and caching the results along the way. Caching here is a vital component because multiple images can have a common layer, so by caching the results of every layer that we scan, it avoids duplicating work and saves time overall on the scan if you use it to scan numerous images at a time. So far, we've talked about securing the airship container images that we developed, but what about the secrets created and used by airship to deploy your cloud? Well, that's the subject of our talk, securing and obscuring secrets. Let's start with securing secrets by limiting access to the secret. Limiting access begins with the Peglig project, which is responsible for generating passphrases, certificates, and wrapping external secrets that are part of your site. Peglig runs as a container based on Ubuntu. So we're talking about file permissions. Basic file permissions are broken up into two categories, permission groups. Each file and directory has three user-based permission groups, the user, the group, and others. Each of these groups apply only to the category that it is. So user group permissions apply only to the user of the file or directory, or in other words, the owner, and they don't impact the actions of other users, where group permissions apply only to the group that has been assigned to the file or directory, and they don't affect the actions of other groups, and then the others, the all users permissions that apply to all other users on the system. This is the permission group that you want to watch the most. For each group, we have types, permission types. Each file and directory has three of them. There is the read permission, which reflects a user's ability to read the contents of a file, the write permissions referring to a user's capability to write or modify a file or directory, and the execute permission affecting a user's capability to execute a file or view the contents of a directory. When Pegleg generates or wraps a secret, it does so with 640 permissions, meaning that both the owner and the group can read the file, only the owner can write to the file and all other users have no access to the file at all. Both the owner and the Pegleg group are root, essentially limiting access to the secret to the root user only. These file permissions are preserved when secrets are stored in source control to be retrieved later to deploy your site. What about obscuring secrets? What if somebody has access to root? How do we prevent them from seeing the secret values? Well, that brings us to encryption. Pegleg encrypts all documents marked as secrets using a symmetric encryption key provided as an environment variable. This symmetric key can be used to encrypt the document and also to decrypt the document. So let's take a look at how this works in practice using sample documents in Treasure Map. So let's look through the Treasure Map directory real quick. Here's some secrets here in the AirSleep site secrets directory, certificates, pass raises. But most importantly, if we look at one of these secrets, we'll see that it's a plain text value. And the same is true of all of the secrets here. So how do we restrict access to each of these secrets? Well, we change the storage policy on each of these from being clear text to encrypted. And then we can encrypt the file. That storage policy is what tells Pegleg whether the file should be encrypted or not. Once the file is encrypted, we essentially have obscured access to the secret. So let's change all of those clear text values to encrypted. And then we'll encrypt it and we'll demonstrate that we can, in fact, obscure access. So this shows that all of our secrets have now been changed with that clear text to encrypted difference. And so we have symmetric keys that are environment variables. I'm going to just set a test password and a test salt for this demo. We will encrypt all of the files in the air salute secrets directory. Now we have all that encrypted data written out. So let's take a look at those pass phrases again. Make sure that those pass phrases are no longer clear text values. So this pass phrase is in fact encrypted. We can no longer see what the value is. And if we repeat the same step on any other pass phrase, then we'll see that it is encrypted as well. Now what about decrypting it? How do I get back a clear text value? It's done in the reverse order. You just give Peg Lake the pass phrase and salt that was used to encrypt it. And then you give it the decrypt command with the path that you're interested in. And going back into the air salute secrets directory, we can list a few secrets and see that we got back our original clear text data. There you have it. That's encrypting secrets with Peg Lake. Now earlier we made a distinction between vulnerabilities and our dependencies and vulnerabilities and airship code. Let's talk about how we disclose those vulnerabilities and how you can report them to us so that we can fix them. Let's take a quick look at the airship documentation page. This is a docs.airship.org. In particular, we're interested in the airship security vulnerability management process. This gives information on reporting vulnerabilities, on how you can receive early disclosures to any vulnerabilities in our code. This is especially important to those of you that are operators of airship and how you can receive public disclosures. Before we move on to questions, let's do a quick recap of what we discussed today. Detecting and resolving CVEs, you can use tools like Clare to see them. And in general, it's a best practice to use the latest stable dependencies that you can. For viruses, install only trusted code into your application. And then scan it with tools like Clamav or Clambake. For secrets management, you want to limit the access to secrets by setting restrictive file permissions. And then you obscure those secrets with encryption. And the airship vulnerability management process. The working committee reviews vulnerability disclosures, seeks to resolve the vulnerabilities and discloses the vulnerability and available fixes promptly. So thank you for joining me to talk about airship security today. Here's some social links for keeping in touch with the airship community. I'll take your questions now at the remaining time that we have.