 and there's how are you and others. Hello everyone, before we get right into it, I see folks already using the signup sheet to mark their attendance. Please state if you have an update or no update for this meeting. Today we intend to cover the security assessment for Harbor. Michael Michael will be presenting and I will conclude with how we are capturing the reviewer summary, which is still in process and we expect to be completed pretty soon. But the bulk of the presentation will be Michael presenting Harbor from a security scope. Absolutely. Michael and you will be sharing from your screen. Yeah, let me know when you want me to get started. I can do that. I seem to have audio issues because I see your mouth moving, but I cannot hear you. Can everybody else hear me? Let's talk into the red button. Let's talk into the red button. I hear you. Yeah, I think Andres is in trouble. I think the problem is on my hand, just as he's speaking as well. I apologize, one minute please. Bear with me just one moment. While he's doing that if anybody is available to potentially scribe and wouldn't mind doing so, there's some open slots for that below. Okay, I've seemed to have started out my audio issues and I can now hear Justin. Are you able to hear me? Yes. Fabulous. All right, so I'll go ahead and share my screen and then we can get started. Like, do we have Corum or not yet? Go for it. We have a scribe. Thanks, Vinay. Go for it, Michael. All right, so I'll start this in presentation mode and I will reference back to the due diligence self-assessment document as well as necessary. We wanna make this interactive. So for the last three plus weeks, myself and Daniel from the Harbor team. Daniel is our architect, as well as a security review team of Andres, Justin, Chase, Vinay, Robert, Martin and a couple of others also have added comments have been going through the due diligence self-assessment that Harbor has created to kind of basically complete a story. Who are all the different actors that are in the system? What are the different pieces are involved and different assets are being protected? What happens when there's a vulnerability? And this review and the presentation today is gonna center on the main key areas that we have identified as part of the review of Harbor. Like Andres mentioned, most of the assessment is completed and right now it's down to basically putting things down and getting the findings together. So looking at this from an agenda standpoint, I'm gonna go through a brief Harbor overview. Feel free to interrupt and ask questions. This needs to be interactive. I'm assuming we're recording this like Andres, right, Andres? Or is it being recorded? Correct, it is being recorded. Okay, cool, thank you. Then we're gonna go through an overview of actors and the tenancy model that Harbor has. We're gonna go through an overview of the protected assets, the blast rate using recovery. And then we're gonna focus on the assessment findings and recommendation summary. Obviously we'll leave some time for Q&A as well. So let's start a little bit with Harbor. So Harbor is an open-source container image registry. That means that it's not a hosted service. An enterprise or a user will download Harbor and will install on their hardware within their own environment. And Harbor is gonna enable them to secure their images and ensure they're free from vulnerabilities. It will give you all-base access controls so you can enforce who can access, what images and how. It will scan images for such vulnerabilities and it will enable you to sign your images as trusted to establish provenance. Our mission as a registry is to be the most secure, performance-skillable and available cloud-native repository for Kubernetes. So we're aligning our vision and the capabilities that we support with the vision and the market momentum that Kubernetes has. So some of the key features of Harbor at the high level, security and compliance, what does that mean? It means that me as an operator of Harbor can go and create enforcement rules and policy that says, if an image is not scanned, don't allow any user to pull it. What if an image is scanned and it has a vulnerability of severity equals to high or higher, don't allow a user to be pulled. Or if an image has not been signed by notary, don't allow anybody to pull it. These are some types of policing. There's more. We have CVE exceptions and other policies that as a operator of Harbor, you can go and enforce to have that peace of mind that my developers can operate in a self-service way, meaning they can push and pull images, but there are certain guardrails of what can and cannot happen. With high performance, we can actually scale to thousands of containers and images under management who can scale to support 200 plus Kubernetes clusters and we can scale to terabytes of storage under management. We have customers that are in the 50 terabytes of storage in Harbor. We're interoperable. I'm gonna cover that in a little bit when I show you guys the architecture diagram. So I'll skip it for now. But the last part, as part of aligning our vision with Kubernetes, we're providing you with this consistent image management and consistency is important when you're dealing with things at scale. You don't want drift. You don't want a nightmare where people are using DIY registries and there's no consistent policy applied. Consistency in operations is very important for enterprises. Let's take a look at the Harbor architecture a little bit and I'm gonna start from the edges first. So on the left, and there's gonna be very evident when we talk about interoperability, on the left to have the authentication providers. This is where Harbor enables you to bring your own identity. So we allow a federation of identity to come from an external identity provider. And we support Active Directory in LDAP as well as OIDC. And we only support one identity provider to be attached to Harbor. So you can either use the building authentication that Harbor has, or you can use OIDC or LDAP. For customers that are interested in creating multiple identity providers, they can use DEX as kind of an intermediary and DEX can federate to multiple identity providers in front of it. So think of this DEX, it becomes almost like a proxy identity provider. On the right side, we support building Aqua TV as well as Claire as the scanning providers. This is really the engine that basically does a static analysis on an image. It cracks it open and looks for vulnerabilities that are tied to vulnerability database. So if a CVE is not in a vulnerability database, then the scanner cannot identify it and the scanner cannot detect it in an image. But if it is reported, then depending different scanners have different databases that they check, but eventually it will detect the vulnerability and produce a result. The Harbor will then take on and be able to show in our user interface and API. So for example, I have an image, let's say of NGINX. NGINX version 2.1 has a lot of vulnerabilities. 3V or Claire will detect it. It will report it in Harbor and then a policy could apply that says, well, NGINX has multiple high severity vulnerabilities. Don't allow anybody to push it. Then a developer can come in and push NGINX version five into Harbor. And this one has no vulnerabilities into it. It's clean after the scan and now a developer can pull it. So if a developer in their manifest in Kubernetes says, I want to use NGINX version three, for example, they won't be able to pull it and that image will not run in Kubernetes. But as soon as they update the manifest to be NGINX version five, it will work just fine and it will be pullable from Harbor. Now the scan provider is a pluggable interface. It is populated out there and we have five scan providers today. So we have Aqua, 3V, as well as CSP. So we have two from Aqua. We have two from Encore, both NGIN and Enterprise. Dusek, which is a security provider in China also has provided a scanning adapter. We have already developed the Claire scanning provider because that was the original building into Harbor. And then Sysdig is in the middle of finalizing their scanning provider for Harbor as well. And we've pretty much engaged through all the scanning companies out there to ask them to create an adapter. So I think Twistlog or Palo Alto Networks, I guess, Sneak and others as well. Now, continuing down the path, replicated registered providers. As part of Harbor being a good ecosystem citizen, we understand there's a lot of customers that deploy Harbor in a hub and spoke model where they can use Harbor within their data center to do scanning of all the images for vulnerability, to the signing, being able to enforce policy. But then the workloads might not run in that data center. So they might have some workloads at the edge or some workloads in let's say East Asia running on Huawei infrastructure or in Western Europe running on Google. So it's very important that we enable our customers to replicate either to or from Harbor with a set of pluggable registered providers so you can push content in and out of those providers. So Harbor ultimately becomes your policy enforcement engine and this is where we scan images, but you can push and pull images to other providers. And on the right, you can see some of the most popular ones with support, but there's more and more support coming in as the community is asking for additional providers. For example, we're adding a Quaid or DIO right now and we're adding a couple of others like Artifactory one as well. Looking at some of the major components of Harbor, we have a notary component that's responsible for signing images and enforcing that provenance. We'll have the Docker registry. This is what basically implements our Docker distribution for basically pulling and pushing images. We'll have the chart using capability for pushing and pulling home charts. And then we'll have different controllers and services that basically are the core of Harbor, right? So for example, code management, the tag retention, the replication components, the logging, the job service. All of those are different services that really are the core of Harbor's operations. And the backend, we have three storage and data access layers. We have a Redis database that is our key value store. This is where we hold session accounts and a lot of the data that has needs frequent access with a very small penalty for being able to access them. We have our local remote storage. This is where we store the actual artifacts. So if you push a container image, it will end up going into storage. And then our SQL database that's built on top of Postgres contains the entire configuration of Harbor. So most of the data in terms of who has access to what, how often do you want me to garbage collect? Does this project have a code of a gigabyte or a terabyte? All of that config is in the SQL database. At the front of Harbor, we have NGINX that basically controls our API routing. And this is where the front door certificate will go. So if you enabled SSL in Harbor, the front door certificate will terminate at the NGINX layer. Any questions so far? All right, moving on. I wanted to bring this because there was a lot of questions on pluggable scanners in the document. I decided to include this slide as well and I'll spend a couple of minutes on this. Essentially, this is the scanning, pluggable scanning framework. Harbor has access to all of the images as well as access to their configuration and the policy around how often do you want to scan. If, for example, you want to bring a pluggable scanner like Aqua or Anchor or 3V or any other adapter, they implement that interface, which is very simplistic, it's a few API calls like an API call that says go ahead and scan. And during the scan interface, we give them credentials to a robot account that was created and tied to a project. So it has a limited scope and it has a limited lifetime. That robot account has a lifetime office as long as a scan job happens. And then the scanner API is gonna use that robot account to pull images from Harbor, scan them, and then report back the results. So the API is scan, here's some permissions to be able to scan, here's a project and the images that are contained here that you need to scan and give me back the results when you're done. All right. So let's go ahead and start talking a little bit about the overview of the different actors in the system and the tenancy model. We're gonna make it clear that the end user authentication that you have using building authentication where all of the data is stored in the Postgres database that Harbor has or if you're using external federated identity using YDC or LDAP or Active Directory, this is just for auth. So there's nothing else. There's no way that someone can create an OIDC token that has not only authentication but it also has claims and entitlements that doesn't work in Harbor today. We don't have that capability. So we start with auth, it's a building authentication that's usually used for DevTest environments or for developers, desktops, for example. But as soon as you push Harbor in production, most likely and what I've seen pretty much 100% of the enterprises is that they use OIDC or LDAP to federate existing identities and then tie them to Harbor projects. So now you brought your identity into Harbor. What do you do with that? Well, Harbor has a fairly robust role-based access control and concepts of multi-tenancy. And the way that works is that we have two main concepts. One of it is the project-wide config that can only be enforced and changed by the Harbor system administrator. This is the overall admin that owns Harbor operations. And then we have the logical concept of projects. Think of projects as a way to bundle tenancy. So a project can contain users. These users can be part of different roles. We're gonna talk about the roles in a second. A project can contain images and a project can contain policy. So for example, you can create a project that's your production images and put all your production images in there and say, because these images are gonna go into production, don't allow an image with high vulnerabilities to ever be deployed. And don't allow an image that's not signed to be deployed. And then only allow these two or three functional accounts from Kubernetes to be able to pull images from this project and nobody else. Now you can have another project that's your DevTest playground for the finance team. And you can say, you can do whatever you want in this project. Go ahead and push images, pull, run it through your CICD systems, test them. But these cannot be pulled in a Kubernetes production cluster. And I don't care about vulnerabilities here. Like, you know, feel free to develop. But when it's time for you to run it in production, you need to move that image in the project that's for production and then the entire policy gets enforced. Now, Harbour has five different roles within the concept of a project. We have the limited guest. Think of this guest as someone that has a read-only operations. So he can pull images but there's very few permissions for that limited guest beyond that. Then we have the guest which is someone that can log into our UI. He can pull images. He can view other users in the project but doesn't have really any access to modify things. Then you have the developer who's the person that can actually push images as well to the project. Then you have the master and the project admin that have elevated permissions. They can create robot accounts. They can change policy for the project. They can change who the scanner should be on a per project basis. So a lot of the configuration that happens here happens at the master or project admin level. I want to stop here for a second because I want to show you guys a couple of things. So the first thing I want to show you is I'm going to go to our website here and I'm going to search for user permissions and I'm going to show you guys this page that basically outlines each one of these roles of the Harbour project and this is documented in the due diligence as well by the way and these are the different permissions of each of the users and you can see as you're going from left to right the number of permissions gets increased depending on the needs and the type of account. The second thing I want to show you is because sometimes it's easier to basically see some of these things in action. So I'm going to go ahead and log into my Harbour project here and I'm going to pick an account that I personally don't have access to. Well, I'm the system over all administers so I can see everything but I didn't know create this. So I want to show you guys a few different things that the project encapsulation provides in Harbour. The first one is a list of all the repositories and actually this is by accident. I talked about the NGINX earlier, I didn't know that I was going to find NGINX here but this project has an NGINX image in it. Look at this, this is what's come from vulnerabilities and I get a little chart that tells me I have 15 negligible, 19 low, 11 medium and zero higher critical. So I get to see a very quick summary and I can click into it and I get a full view of all the vulnerabilities that you have for this image if I wanted to write. So I can click here and I get to see the full vulnerability. I can even click on the different CVs and get additional info. Beyond images, I also get home charts. There's nothing here but if I had any, they'll show up in here as well. I get to see members. So I can see who the different members of my project are. So I can say I want to create a new user and the RBAC permissions that you mentioned earlier come into play. So I can create Andres here and make him, I guess for example, now it's gonna tell me I can't find Andres in the system cause he's not the user of Hamburg but should he have been, I can add him here. For YDC and LDAP, you can also add a group. So if you don't wanna manage the membership of the users independently in Harbor, you can add an YDC group or an LDAC group and then you can manage that group at your active directory level and you can add or remove users to it and those users can come in and they will get the necessary access in Harbor. We can have, you get to see logs. You know, what's happening here, who pulled image, what image at what time and I can do some introspective or forensics into my project. We have robot accounts and think of robot accounts as an account that you can create for CI CD that enables you to control assets within this project in Harbor. So for example, can create, sorry to be picking on your name Andres but I can create the address CI CD robot account here and I can say what permissions I want it to have whether for images or home charts, do you wanna allow him to push images or only pull? Do you want him to allow pushing home charts or only pull? And depending on the needs of that, you get to define what permissions that you wanna have, right? So let's say I click save right now, it's gonna basically create a JWT token for me and I get to copy this token. It's only viewable once, I can never retrieve it again. So now I can click on this account and I can either disable it or delete it but I don't have access to that credential anymore. And if I wanna rotate the credential, I can delete this account and recreate it using the name Andres CI CD and I will get a new JWT token. Well, the concept called tag retention. This is a type of policy, like how far back do you wanna keep images? So for example, you can have a compliance policy in your enterprise that says, I don't wanna keep images beyond nine months. So a retention will enforce deletion of those images. We'll have tag immutability. And what that does is you could say, I wanna make sure that when someone pushes the image engine next version 3.0, nobody can come and override it whether that's a bad actor or not. I want when I publish a release version of my image, mark them as immutable. Don't allow anybody to override it and Harbor will enforce that. We have web hooks. I want to be able to be notified when things happen in Harbor. So I can come over here to my documentation and I can search for web hooks and I can see the different web hooks that Harbor pushes. So I can create a policy that says, go ahead and notify me when a scanning is finished. So I can go and execute a third party or fourth party action. And second to last is a scanner. I want to tell you as Harbor, which scanner I wanna use for this project. So if you have multiple scanning adapters deployed, this case, I don't. But if you did, you get to pick which one you wanna use. So you wanna use Claire, you wanna use TV, you wanna use Dusek or Anchor, you get the choice. So for different projects, you can use a different scanning adapter, but you can only use one in the future will allow you to seriously configure multiple scanners, but not yet. And the last part. Couple of questions on the chat. When you talked about robot accounts, the question is, are robot accounts like service accounts? Yes. That way. Yeah, I think of them as an automation service account that is correct. Just a suggestion, Michael. Can we just call it service accounts from an industry perspective? What is your thought on that? Because to help everybody wrap their heads around the same concept. We actually strike all of that. In the Kubernetes community, in the cloud native community, they're known as robots. It's not just as others have called it as well. So if we call them like headless accounts or service accounts or functional accounts, which is what they are, they're not tight. They're functional accounts, right? They're not tight to use a persona. Then we will have created more confusion because we will not be aligning with what others are calling them. So if I come over here, I'll do a very quick search, right? Then I create a type robot account, Quay. You're gonna see that they also call them robot accounts. So it's a lot of people in the industry are calling them that. Got it, thank you. So unfortunately, we're stuck with them that way. I had another question as well. I mean, what is the scope for these policies? How fine-grained can it be? Can different teams have full jurisdiction over their own policies and applied, et cetera? That's correct. So a lot of the policies are tied to the project. So for example, I can come here and look at some of the policies here. All of these things that you saw is at the project level. Every project gets to define all of these on their own. Got it, that's perfect, thank you. So for example, I can come in and say deployment security, allow only verify images to be deployed or prevent vulnerable images from running, prevent images of vulnerability severity higher or above from being deployed. Automatically scan images on push. Create some wider lists. All of these policies are tied to the project. All right, so I answered Vinay's questions, I believe. Vinay, let me know if you have more. I think that's it. Let's go to Payam. Are you able to define what the number of vulnerabilities found based on the category rather than all? Not sure I understand the question with the number. So this one tells you the number, right, 82 total. Is that what you're asking for? Yeah, so if your deployment is based on, let's say critical or high, right, that is the watermark for fail or success, are you able to have that number of you based on the criteria rather than just showing the total number of vulnerabilities? So if I'm only interested in knowing what is blocking the build or what is blocking the image. Yeah, so when you try to pull an image that fails the vulnerability threshold, for example, it will tell you that, hey, I couldn't pull this image because it has high severity vulnerabilities in it. So it will tell you that. But in the UI, we don't intermingle the policy with the actual vulnerabilities found. Here, we just tell you what we found and give you the breakdown, but we don't tell you if your policy will prevent you from pulling this image or not. Okay, and if it's okay, I had a follow-up question. I can wait. No, go ahead, go ahead, ask it. So in regards to Harbor, so I'm actually using it right now for testing, and I was curious, so you can mirror Docker image repo registers, let's say. And let's say you detect a vulnerability in that image. Can you use webhooks or some other form from Harbor to indicate in Dockers that that image can't be used if you're not using Harbor solely as your primary image register? Yeah, absolutely. You're coming from Docker to Harbor because you wanted to check an image from vulnerability in Harbor because Docker doesn't have that ability. You can use our webhook to go back and trigger an action that says, go and mark that image unavailable in Docker. You have to use the Docker API, but you can do that. That's exactly why the webhooks are very valuable in that. Okay, perfect, thanks. Or an additional webhook will be go and create an admission controller in Kubernetes and say, don't allow this image to be deployed. Great, thanks. Okay, a question from Ash. Do you have a plugin for external authorization? We do not today. We're actually looking into what that potentially will look like in the future. That's why I wanted to call it out that if you put claims or entitlements in your OIDC token, we won't respect them today, but that's something that we're looking for in the future. And essentially one of the things that we might do today are users and permissions are tied to a project. So you come to a project and you say which users have access to it, and that's great. But one of the things that we can do is that we can extract the users out of the project so you can create logical groupings of users and then tie those users to projects so you can tie the same set of users with three projects, for example, without having to recreate that RBAC. So we're looking into that. Once we do that, an optional added feature of that will be enabling these claims and entitlements. From Yiling, can this be installed and run locally? Absolutely. That was one of the first things we talked about that Harbor is a package software. You get to install it wherever you want. It's not a service or a hosted solution. All right, moving on here. Automation and CICD kind of talked about that. We have a couple of things that are really aid in plugging Harbor with other systems within the environment. Robot accounts enable you to interact with CICD systems to push and pull images. Robot accounts can act as functional accounts to pull images into Kubernetes environment, for example, where the only access they have is read and nothing else. And then web hooks allow you to connect Harbor to other systems where that's for notification, for billing, for inventory management, or for taking actions. And then I talked about it earlier when I showed the replication providers who have full external connectivity to a multitude of replication providers so you can either push content or pull content into Harbor and create a bi-directional pipe. Let's talk a little bit about protected assets. If you haven't read the documents, some of these might be a little bit, I don't want to call them obscure, but it'd be hard to kind of put a relevance into it. But we have a variety of protected assets in Harbor. And I put two notes here, like can this asset be rotated? And if it can be rotated, is it hard to rotate or not? And you can see that indexing on some of this. So starting from the top, we have the Harbor Private Key. This is one of the key capabilities in Harbor. This key essentially, and let me actually show you guys one thing really quick. Oops, sorry, forward on the slide. If any of you want to kind of read along, we have the CNCFC security Harbor self-assessment that's basically pushed a link here as well in the chat. But essentially here in this document, if you look at the index here, there's a couple of areas that we're going to talk about. This is a blast radius and recovery. I'm going to mention that a little bit. And then we'll have the breakdown of access and tokens. And this is what we're talking about right now. So if you want to take a look at that, feel free to do so, okay? So coming back to this, and the Harbor Private Key is, think of this as the key that Harbor generates your installation. And this is the key that we use for a variety of Harbor internal operations. The next one is the Harbor encryption secret key. And this is what we use to encrypt pretty much every secret in Harbor. So think of, earlier we talked about being able to add a replication provider in Harbor. Well, when you add a replication provider, one of the things that you need to do is you have to give access credentials, a username and password. We encrypt those using the encryption secret key. The FQDN certificate is the Frander certificate for Harbor, like basically you enable SSL in Harbor and you try to access HDP, co Harbor deployment, for example, that's the certificate that you have. We have the notary signer certificate. And this is a certificate that's basically used by, it's generated by Harbor and used by notary for basically the notary operations, like this is where you enforce and maintain that images are signed in Harbor. Then Harbor itself, you know, this from the diagram three slides ago, has a number of core services here. So we have config management, quota, charge service, tag retention, content trust. All those services can communicate to each other using SSL in the release of Harbor that's coming out this week. So each of them, their seven in total can have their own certificate for SSL communication. Then you have the Docker client credentials. These are the credentials that you provide to be able to push and pull images through the Docker distribution. We have the replication credentials. These are credentials that are encrypted using the encryption secret key up at the top, right? So this is basically, for example, I added Docker Hub, I believe it was we needed that earlier. So I have Docker Hub and I replicate from Docker Hub into Harbor. These are the credentials of my account into Docker Hub. We have the scanner credentials and these are the credentials that we use and we provide to each scanner, like I mentioned earlier, to be able to pull images from Harbor, to be able to scan. And then then we have robot credentials which are the credentials that you provide to robot accounts and that's the JWT token that I mentioned earlier. Now, if you can see here, the bottom ones are much easier to replace, right? Let's say, for example, the robot credential needs to be rotated, I delete the robot account, I recreate it, I have a new credential. The scanner credential is rotated automatically on every scanner jobs. Every time we start a scan, we generate a new account. So easy to rotate. Replication credentials, well, something bad happened, go ahead, delete that account, recreate it or update the password, you get a new password. Well, you wanna rotate the Harbor Private Key, that's where things can become a lot harder, right? Because you can only rotate it by basically upgrading Harbor or doing a reinstall of Harbor. So it's a much more involved operation. The encryption secret key, well, it cannot be rotated. Why? Because once you rotate that key, everything that you encrypted in Harbor so far can no longer be decrypted. So is it rotatable? In essence, yes it is. That means you have to go and update every single password in Harbor again. So you have to go and re-add all your replication credentials, re-add all your CLI secrets, re-add anything that basically was encrypted in Harbor and you can do it. Doable, but very, very hard. So that's why I didn't even put that can be wrote to the signal. And then the Docker client credentials, they cannot be rotated because Harbor automatically rotates them. So they're a short-lived credential. Okay, let's move on to this slide. And this is what we call the blast radius. I believe it was just in campus that mentioned this term. I liked it. I started using it. I'm assuming it's common in your industry. It wasn't common in mine. But basically this is around when bad things happened, what's your exposure? And how can you recover from it? If you wanna get a lot of details about this, you actually need to read the doc. I'm not gonna give it service by condensing it here, but essentially the big thing that I wanna mention here is that there are certain areas of Harbor that when they get compromised, it's worse than others. And I've classified them into three categories. Risk is red, as in shut things down. Things are really bad. If this happened, you need to stop everything and figure out how to plug the hole, identify everything that was compromised, everything that was changed, and then correct it. You might potentially have to go restore certain things from bug up and then you can start all over again. But why is this important? You trust as a user, the Harbor is gonna give you the images that you're gonna pull and run in your Kubernetes cluster in production. Should Harbor be compromised and with an elevated risk, it is potentially possible that a malicious attacker can come in and modify and replace an image that you thought was safe with an image that's malicious. And now when you pull that image into Kubernetes, you're essentially running malicious code in your Kubernetes cluster. Not ideal, actually really, really bad. And then the second thing that I added here is this recovery, this is contained. And what I mean by contained is, is a recovery contained to a small portion of the product versus having to deal with a wide-ranging recovery. All of these are indexed by number and this is how they're also indexed in the blast radius section in the document as well. So if you look for number 17, you can find number 17 in the document. Now, I wanna go through some of the yellows, for example, here, right? Because it's easier first. Compromise robot account. Well, the recovery is contained because a robot account only has access to a single project. So if someone compromised a robot account, he didn't get access to your entire hardware installation. He got access to a single project. So what can you do? Well, if they compromise a robot account that only has read-only access, then just delete the account, recreate it and you're done, easy. If the robot account also has push access, then you have to actually run forensics and figure out, did that robot account push an image? And if yes, go delete the image they pushed. That way you clean up after them, right? But then you create the account and you're done. Compromise scanner. Well, a compromise scanner can do a lot of things. They can tell you that an image is free from vulnerabilities when it's not. A compromise scanner can tell you an image has vulnerabilities when it doesn't and create a dosage tag because that image can no longer be pulled from your cluster. Or the scanner can do a lot of other bad things, right? It can basically never return. So the scanning will never complete for an image, for example. The fact that the scanner is relied on to provide a report on vulnerabilities in your system if it's compromised is bad. But how can you recover? Easy, delete the scanner, redeploy the scanner in safe infrastructure with strong passwords and tie it to hardware again and you're safe. These are the yellows with the green recovery. Let's talk about something that's bad. Someone compromise your hardware administrative password. That's the password that has access to the entire hardware installation. APIs and the like, well, the recovery is really, really bad because that means you have to figure out every single batting that they did from being able to push and pull bad images, creating accounts for themselves, masquerading as other users, for example, or deleting robot accounts, changing policies. They can do a lot of bad things in hardware. They have full access to the entire environment. We talked earlier a little bit about compromised. Let's see. What is it? Oh, why can't I not find it here? Bye-bye, guys. All right, I can't find it. I guess maybe I missed adding it somewhere. But let's say they compromise the identity provider. What can they do? Well, that means that you're thinking that Andres is logging in, but it's not Andres. Andres dash, the malicious Andres that's logging in. So whatever access Andres had, this new user will have. How do you clean up? And how do you recover from this? You have to actually run forensics and figure out everything Andres dash did in the time span that you were infected and go back and clean that up. A very time-consuming process and potentially you could have a loss of data. Someone compromise your Postgres database. On its own is not bad because certain items are encrypted in there. But if they were also able to compromise your private key, your encryption key number 10 here, yeah, that's why I couldn't find it. It was blue, but if they also, if they compromise your Postgres database, as well as number 10, your encryption key, that means not only can they read everything in the database and change the configuration, but they can also get access and decrypt the passwords. So now they get the password to that Docker hub account that Vinay added in to be able to pull images. So now they can go and recover, not only in Harbor, but also in your Docker hub account. If you combine number 16 with 10 and three, that means they have a lot of access. They compromise the infrastructure node. So they have access to your storage. They have access to your database and they can decrypt everything in your database. Really bad things happening. And then I wanna talk a little bit about the Harbor front door certificate. Should they compromise the front door certificate to Harbor? That means everything you push and pull into Harbor can be clear text from them. It can do a modern in the middle attack. So when you are coming to Harbor and saying, go ahead and create a new robot account and Harbor gives you back a job token, they can read that token. So that means they can actually use it to do bad things with it. When you say, I wanna connect Harbor to Docker hub and Vinay is adding his username and password there that's transmitted in clear text. They'd be able to pick it up and do bad things with it. So SSL compromised in bad thing, guard your certificates, make sure that nobody can compromise that. I don't need to go through all of this, but I think that gives you a picture of where we are. Let's move on to the next slide. We have questions. Let's see here really quickly. The question is, where is Harbor with TLS 1.3? Yeah, so we don't have a date for TLS 1.3 yet unfortunately. So we're on TLS 1.2 right now. Is it something on the roadmap or something not on the roadmap? It's not on the three month roadmap. We have a few other things we're working on right now, but it's after that, we rebalance every three months. After that we'll rebalance and if it makes sense for us to support TLS 1.3, we will. Be aware that Harbor on its own cannot be the only one that can make that choice. We depend on components, for example, notary, clear, 3V, Docker registry to also be able to support 1.3. So all of these have to support 1.3 for us to support it. So follow up question, if that's all right. Yeah, I'm wrong. So in regards to strong Cypher support, so mainly for things like this 1.42 environment, I'm assuming that you guys have supported Cypher's available? Yeah, we have under, if you search for Cypher's in the document, there's a section called compensating mechanisms. And here, let me go here really quickly. So if you look for compensating mechanisms somewhere here, we have a list of the Cypher's support. So you can find that. My last question, where's Harbor on HTTP3 support? I actually have another project that is looking into HTTP3 support and seeing trying to kind of gauge how that community is. We haven't looked at it in Harbor, to be honest with you. Adol, that's just not something that came up yet. Generally, a lot of these things, we kind of use our user base and our customers to kind of guide us and where they are and what they want. Like that, the day someone asked for IPv6 support and Kubernetes itself doesn't support IPv6 yet, not on its own, it supports a dual stack and even that, that's alpha, right? So we can't go support IPv6 yet. We have to follow, because we depend on a Kubernetes deployment, we have to wait for some of the other infrastructure tech to support some of these things as well. All right, Andres, the mic is yours. Thank you, Michael. Just quick overview for those that may be new to the call in the process. The goal of the security assessment is to review the project design goals in respect to security and analyze the different aspects that could introduce risk, what aspects of security are to be handled upstream, what items are to be handled downstream or complementary software in order to harden the solution and what steps can the project take towards a more secure cloud-native ecosystem within the project itself, as well as increasing the security of other applications that may interact with the system. We are about three weeks in and to the assessment. We're close to wrapping up. There are discussions still happening between the reviewers to summarize our findings. Some of the salient observations are that there are a lot of different components, a lot of different technologies that Harvard utilizes that are moderately complex, take scanners, registries, identity providers, and each of those systems have their own security properties and the interaction between these projects between these different components needs to be further analyzed in order to strengthen the security. There could be improvements in some of those areas. Harvard does have some defaults, but we believe there's opportunity where the boundaries between those different projects, the different technologies, there's opportunity for improvement there. We feel detection of attacks and recovery from attacks should be studied in more detail. It warrants a formal security audit to uncover cases an attacker may perform attacks without detection or what forensic needed to recover from difficult attacks. Same goes for consideration of, you saw the table around the different RBAC roles, how course-grained can those be and leading towards the combination of what roles to use. More so enters how do you productionize Harvard securely, how do users are actually deploying it and using it in practice and what configurations of the system are set up in a reasonable way or which knobs do they want them to turn on or flip to make sure they're in a more secure harbor operation. Is RBAC giving users the least privilege they need? Are they stumbling on blocks? So there's just opportunity to guide users on what are the recommended practices to build on top of the properties that the system has for security as default versus which not. While I hold the floor, I will open it up to some of the other reviewers if there's anything they would like to add somewhere on the call, perhaps starting by Justin. I think you've covered a lot of the points that I sent. Yeah, I mean, I think it's a nicely put together project and anytime you combine complicated things together though, you really have to worry about how you combine them. And so I wanna re-emphasize that point and I also think a lot more diving into how it's used in practice and how people could try to recover from things in practice. I just wanna reinforce those points. I think those are important. And by the way, part of what we do in this report is we give guidance to the TOC about areas where they might wanna have a future firm, like Cure 53 or somebody else, do a more limited focused assessment. And so that's in part some of the guidance that's being given. It's not saying that we're secretly hiding a list of like we're secretly hiding a list of 50 vulnerabilities that we see that we're just not sharing with you. It's saying that if we were gonna start to do, to spend bucks on a real audit, we would probably spend money here first. Yeah, and by the way, Justin, that's 100% valid. Just to give you guys an update because for the folks that haven't read it, so Harbour has been through three pen testing so far. So we had the first one by VMware. So VMware wide had tested Harbour for about two and a half, three weeks. Then we had a team of 10 folks from Cure 53 that basically put the exclusion on Harbour and tested that and identified some vulnerabilities that we immediately fixed. And then we have one of our commercial accounts that is a financial institution that securities of the utmost importance that run a pen test on Harbour and was only able to identify one minor item. So we've had three pen tests happening already. Just kind of the outcome of this, Harbour is actually applying on our B2 graduation. So SIG security is the last assessment that's happening. We had SIG storage, SIG runtime and SIG up delivery. And generally what from what I understood is the what's this TLC is looking for beyond the assessment and providing some guidance and what you should do better, which is obviously incredibly valuable as a key part of the assessment is if you support RB towards graduation or not. So I think they are looking for a binary answer as well. I mean, we tend to give non-binary feedback. We would in extreme cases, like I won't mention there's a project that's been discussed a lot on here that's actually graduated project that has a bunch of security limitations. And I think that would err on the side. We would be as close to giving binary feedback with a project like that, as I would imagine we'll get. But for everything, even for the thing that we think are quite secure, our goal is to say, these are the things it does well. These are areas where there's mild concern. Overall, we don't feel there's a major concern. And I think that's largely what our feedback would be for something like Harbor, that it seems solid. And here are some areas where you could perhaps spend some money well, but it's not in dire need. Got it, got it. Yeah, absolutely. And actually, as an aside, I've reached out to Chris and Etik from CNCF maybe about three weeks ago and I asked him if we can do schedule another Q53 PEN test where I give them the six security assessment document that we created and it's fairly comprehensive and tell them, hey, quite identify angles that you can attack Harbor using this assessment and these results. So that would be a very valuable exercise. But what I don't wanna do is, I don't wanna delay Harbor's graduation if someone sees it. Justin, I think I hear feedback from you. Sorry about that. Thank you. So basically, what I don't wanna do is, I don't wanna go tell Chris we wanna do this and he says, oh yeah, maybe we should wait graduation until Q53 gets another look at it and then we have to wait another three months. That would be detrimental to our project. Yeah, I mean, we would not make a recommendation for you to wait on something like that. Sorry, I think you wanted to say something, Dan. Oh yeah, I was gonna slightly change the topic so go ahead and wrap that up. Yeah, cool, thank you. I was gonna say, you know, we would be very unlikely to make a recommendation like that. So, I mean, obviously we can't control what the TOC or others do. We just make recommendations. But I don't think that would happen. Michael and the assessment team, I know we've sort of identified complexity in this project and dependencies downstream. In our results, is there anything downstream where as the security thing, we would sort of take learnings from Harbor and advise any of the downstream products that they can make any specific improvements on the security interfaces or the interfaces that eventually impact security. Maybe I have a question, Dan, on that. And Michael, I think you answered it. You know, I was thinking about using, you know, the encryption keys are exposed. I forget which one specifically it was in the assessment, but using Kubernetes secrets. And my sense is that's not the strongest posture. And I had a concern around that and would love to take it offline and have a discussion around that. Yeah, we can take it offline. We're actually looking into vaulting. You can use a third-party vault to host your Kubernetes secrets if you want added security. And we can talk about how to enable that. Yeah, so that's just one point that I had. Thank you. Yeah, obviously Kubernetes secrets have limitations and we talked about in the assessment about some things that you can do to harden the Kubernetes environment, but creating using a vaulting solution is probably the best way. I don't have a vaulting solution that we know that works with Harbor, but it can work with you to identify one and use it. You may have to do a little bit of legwork there too. Sounds good. Thank you. Yeah, there's also a little bit with the interface with Notary that feels a little odd to me, but that's probably a discussion that's in no way meant to be like a blocking thing or something like that. It's more meant to be here's an area to look and you know, that we should probably have a discussion with like Justin Cormack and get that all together and then take a look to see if there's things we can do a little better. And I have some comments, like a more specific example in the notes that I sent that'll be passed along, I think so. Sounds good, thank you. By the way, I do have a hard story in about two minutes. So Andres, I guess the next step is you and the rest of the reviewers kind of complete that assessment and then basically checking into the repo. That is correct, that'll be the next step. Looking forward, if possible, if we can wrap it up this week, that will be a tremendous benefit to us. That is what we're shooting for. Thank you. Ideally by Friday, if not, it'd be Monday or Tuesday. Yep, and you know, I wanna thank everybody that's spent a significant amount of their personal time on Harbor, right? Whether that's reviewing dogs, reviewing the assessment and asking questions, reading the answers. Thank you all for all your time. It's been incredibly valuable to us as Harbor to kind of put those things down. We're always thinking about security, always thinking about other things, but kind of writing all that down in a single document and going through an extensive list was very valuable and it will definitely benefit us in the future as well. Likewise, it's been a good experience. All right, well, thank you very much Michael. That was a great presentation to get. Really appreciate it. And I think someone asked for the link to the deck. The due diligence document has a link to the deck and I think they're all in the notes as well. Cool, thank you all, appreciate it. Yes, and we'll probably check in the deck and the same directory for the assessment. We'll put all the content there so folks can access it. Sounds good. Thanks everyone, see you all next week. Thank you Michael, thank you everyone. Bye. Thanks Michael.