 Hi everyone. I'm Alex. I'm very happy to be here at Kubicon North America to talk to everyone about the Harbor Project. It's a shame that this is still a virtual conference and we couldn't come in and share this in person, but I hope everyone is doing really well. And thank you for tuning in. So today, my co-presenter and I will talk a little bit about the Harbor Project. This is a graduated project in CNCF. And if this is your first time attending a Harbor session, think of it as a registry, a place to store and manage your container images and other cloud-native artifacts. Possibly you're using Helm charts, you're using singularity files or something else. But for the purpose of this session, we're going to be focusing mostly on the latest and greatest of the Harbor Project. So focusing on the latest version 2.1 release as well as some of the things we're working on for the upcoming 2.2. We're not familiar with what an image registry does and have never been exposed to the basic set of capabilities that Harbor provides. Please check out our website and check out talks at previous Kubicoms where we've always had an introductory session that sort of starts with what is an image registry, what does it do, why do you need it? So that can be really helpful. So quick introduction. I'm Alex Xu. I'm a product manager in the cloud-native team at DMR leading the Harbor effort. So responsible for trying to understand the requirements around the container registry. For responsible driving the roadmap, collecting feedback from you guys and making sure that we're building something that people will actually use and will actually want to recommend to their friends. And today with me is my co-presenter, Daniel Jiang. Hey, Daniel. Hey guys, this is Daniel. Yeah, I will present with Alex Xu. Today I will show you a few co-demos regarding the latest feature we did earlier in working 2.1. I'm working for BMR as a staff engineer and the maintainer of Harbor Project. Glad to meet you. Thanks, Daniel. We've also listed two of our colleagues who helped us with this presentation. So shout out to Steven and Steven so as well. So we, you know, we started with the intro slide. What is Harbor? Well, Harbor is a trusted cloud-native registry that can store, sign, and scan content. So it's basically a place to host and manage your artifacts. And, you know, when we started this project back in 2014, building an on-prem registry, leveraging Docker distribution, we came across some issues, right, when we were using Docker Hub and other public registries. So over time, we've, you know, addressed those issues and also added more services and features related to lifecycle management, security features like scanning and signing images. And recently we've been focusing on image distribution through P2P and proxy cache. And now we're focusing on high availability, delivering a highly available Harbor cluster through our operator. So our mission is to be the best cloud-native registry for Kubernetes. And we started with support for Docker images. We expanded to Helm charts. And with Harbor 2.0, we can now support any OCI-compatible artifacts. So let's start by looking at the community, which is thriving, doing really well. So we have more than 13K GitHub stars. We have 200 plus committers, 50 plus contributing companies, more than 4,000 forks. We have 14 containers across five different companies, across three different continents in North America and Europe, in Asia. We've also seen a lot of diversity in our contributors as well, in terms of backgrounds and skill sets from developers to DevOps engineers to system admins and basically everyone in their team has needs around a registry to manage their artifacts. So we're extremely happy about this. And we want to thank you guys for all the work you've done. And thanks again for tuning in. And so I want to just quickly jump into this and start with some of the things that we've been working on for the 2.1 release. The first one I want to talk about is called proxy cache, which is the ability for Harbor to act as a pull through cache for another remote registry. And, you know, that's usually we call that the target or the remote registry or the upstream. This is basically situations where you have a registry that you're trying to hit, but you have either limited connectivity or no connectivity to that target. And this could be because of compliance or because of security issues or limited egress options. And so Docker Hub is a really great example of a client registry of a target registry they're trying to hit, right? Docker Hub is a registry where Docker clients from all over the world are trying to pull images from. But if you're pulling too fast, or you're pulling too frequently, then you can trigger the Docker Hub rate limiter, which can follow your connection or even get UIP banned. And so Harbor as a proxy cache is meant to address this very problem. In this case, it means that Harbor will serve as an intermediary to pull the images and cache them locally for you and then serve it to you to serve it to the Docker clients. So it's much faster and, you know, it minimizes going over the network unnecessarily and prevents you from getting IP banned. So the way to utilize this feature is basically, as you would when you're creating a Harbor project, if you're familiar with that workflow, there's an added option to enable it as a proxy project where you would have to then put in the, select the target registry endpoint. That's the endpoint that you're proxying for. And so when you want to Docker pull from that remote registry, you will Docker pull from the proxy project instead. And every time you do a poll, it'll kick off a request to Harbor, which will kick off a request to that upstream registry and do a comparison to see if the image in its cache is indeed the most up to date copy. If it is, it will just serve the cache copy and if it isn't, it will repoll, recache it and then serve it to the Docker clients. So you will have to modify your Docker pull command in your pod manifest to hit the proxy project instead. So the second project or the second feature I want to highlight is called non blocking GC and it's called non blocking because prior to 2.1 corruption collection will put the registry in read only mode. So you can still interact with your existing artifacts in Harbor, you can still, you know, use the API as you can still use the web console. You can still run, you know, certain policies. You can still pull the images, but you can't push images and you can't delete them, essentially for the entire duration of that GC execution. And so with the 2.1 non blocking garbage collection, users can push images, they can pull images, they can delete images from Harbor while GC is running. So, you know, it's completely runs completely silent in the background with no impact to user operations and you have to worry about, you know, data corruption or anything like that. Data corruption is, you know, it was a potential concern for us because of the way images are laid out and stored in distribution. They're essentially, you know, this is owing to the cross referenceable nature of the distribution with image layers are stored in the most efficient way possible with no redundancy. They're essentially deduced across all the projects in a single Harbor instance. So it's really great from a storage perspective, but you know it can be tricky from from whenever you're doing something destructive like running garbage collection or you're deleting images. And the way we were finally able to achieve this was, we're essentially snapshotting all the image layers that are marked to be deleted, and then sort of continuously tracking and resolving these against any incoming image blobs as part of a push request while GC is running. So it's quite an important feature from an operational standpoint. It's very highly requested by large enterprise users that have anywhere from hundreds of gigabytes to we've seen tens of terabytes of images stored on each Harbor instance. And, and because you know GC can be quite time consuming, depending on how often you run it. If it's placed in read only mode for an extended amount of time. And on top of that there's no way to tell how long that job will run for or how much disk space you will it will be the allocated from each execution, you know that can be problematic and it can be quite stressful. Right so imagine if you're in the software delivery business sort of software supply chain business where all the images are landing in your harbor instance and you're the one responsible for distributing these to your customers. And so that's why we've added the ability to perform a dry run which will give you an estimation of the amount of disk space. And so by making it non blocking, not only is it, you know, not incurring a penalty on your user operations not only is it running in a faster and more efficient manner. But you know, the time it takes to run GC is no longer as much of a factor it's no longer as debilitating to your business. If it goes over, you know, if it goes runs for for quite some time right. You're you're taking the pressure off the garbage collection aspect of maintaining a registry. So before I move on to 2.2 Daniel will do a demo of the proxy cash as well as the online garbage the non blocking garbage collection sorry. So let me stop sharing. Let me make sure that's still recording. Okay. Yeah, it is still recording. Yeah, so let me get started. Hi guys so the first time I'm going to show you is proxy cash I need to use the recorded demo due to the instability of my network. Let's move on. So in the proxy cash to implement it we reuse the code of replication. So the user experience also has some overlap for you to create a proxy cash project first you need to create a registry endpoint. The same way as you created for a target registry create a target registry for a vacation. So in this case I just created a Docker hub endpoint with my personal account. And I can create a new product. Note that in this dialogue, there is a new switch of proxy cash. I can switch it all and select this endpoint I just created. Yeah, after that, this will be the proxy cash project plus images from Docker hub. Know that this is an empty product no images. And if I issue a Docker pool command against this proxy cash project. Here I want to pull go hover slash hover core images from Docker hub via my proxy cash product. I just append this relative uri to the uri of my project of the hover instance like this. And if I start pulling, you can see the images is serving directly via this proxy cash from Docker hub. Okay now at the same time under the hood there in the background there is a go routine to write the content of this image to harvest storage. And because it's asynchronous you need to wait a couple of seconds when you refresh, and you can see that there is a repository hover core and the image. Note that once this image is stored in hover storage, it's the same as the images you pushed to hover. You can access all the information and you can do all the stuff to this image such as scanning or adding text removing text. Here I simulate outage of Docker hub by messing up my DNS so that my hover instance cannot access this Docker hub, and you can see the test connection the ping doesn't work now. If I remove this image locally and pull it again from my Docker client. A hover will do the best effort to sync with the target registry but in this case, it is not accessible so hover will serve whatever is available locally. So in this case, you can still get this image that is cached in hover. We can see there is really helpful for you to alleviate the problem caused by the red limiting in the Docker hub. It's also worth mentioning that this proxy cache project. It has all the capabilities except for pushing, has all the capabilities of a regular project in hover. For example, you can use the code policy to control how much images can be proxied. And you can also set up this tag retention policy to remove this style stale image cache. You can serve and store the new proxied images. That's the demo for the proxy cache. Next is a non-blocking GC, non-blocking garbage collection demo. So as Alex mentioned, when we implement this feature, instead of relying on the Docker distributions code, we use this data in Harbor's database as the single source of truth to help us decide what layers should be removed and how much space will be freed is also available. So when you hit the dry run, we just mark the layers that are candidates to be removed and calculate the size. And we can provide a hint for the admin to let him know how much space will be freed if he run the GC right now. No, this is a very rough estimation because this is a non-blocking GC. So images may be pushed when the GC is running. You're going to see now. If I trigger the GC now, you can see the job is scheduled and it's running. You can notice that there is no longer the warning bar saying that Harbor is in read-only mode. And at the same time, I can do a Docker push to my Harbor instance while it's being garbage collected. Yeah, it's still running and pushing this in progress. It's an instance on AWS, so it's a bit slow. Okay, you can see this garbage collection is also finished. If we see the lock, you can see what layers are removed and there is actual size of the storage that is freed by calculating what layers are removed. This is a big improvement for the admin so that they will not need to worry about their developer not being able to push images during GC. Next step, we may improve by considering to improve the performance of GC by doing the deletion in parallel. But this is not that an issue because it's non-blocking. So if the GC runs a little bit longer, we consider it's okay. That's all my part. Thanks. Thank you, Daniel. I'll continue with the 2.2. So let me make sure it's still recording, yes. Let me share my screen. Can you see my screen? Okay, so let's continue. So thanks, Daniel, for that demo. Switching gears a bit now. I want to talk about the upcoming 2.2 release and some of the things that are being planned and being worked on. The first one is metrics and observability and this is really just a fancy way of saying we're going to be supporting Prometheus. So the goal is to monitor services deployed actions performed as part of your target deployment and expose those metrics through an HTTP endpoint for Prometheus. And so if you have Prometheus already set up in your environment, if it's already part of your stack, you can then scrape whatever data you want off that endpoint. We're going to make it highly customizable in terms of the kind of data you can retrieve. And when you think about key telemetry, telemetry can be divided into several layers. At the very bottom, you have the platform, which is your node level information and you can monitor your resource consumption. So CPU, RAM, disk, basic event logging, health status of the deployment as a whole. And then you have the container orchestration layer, which in the case of Kubernetes will provide monitoring for all the pods deployed in your cluster. You can see in the diagram we have some of the main pods that make up the harbor cluster. There's harbor core, there's harbor registry, there's DB, there's job service, but there's a bunch of others. So the web portal has its own engine access, its own chart museum. If you're using trivia or clear image scanners, they have their own. And then finally, we have telemetry at the application layer, which is basically the relevant registry operations and the thing that the harbor API provides is a potential candidate here. And we can potentially expose aggregated statistics over some period. So looking at the number of image pulls or a number of images pushes, image deletions. Also looking at top images, repositories, projects being requested. Possibly looking at, you know, number of push errors or delete errors or authentication errors. And, you know, any other statistics that could be useful for detecting abnormalities, right, like users, number of users logged into a particular instance, number of failed attempts, unique IPs, number of image requests served in some period, requests in flight, number of replications. So this is still being actively worked on. I think the PR to the proposal in our community repo. If you have any questions or comments or want to contribute to this work, please, please make comments on the PR or just reach out to us. So we've also got a lot of requests to deliver system level robot counts, which is a robot account, scope to the harbor instance level, so it has access to multiple projects, as opposed to the current robot implementation, which is scope to a particular project, right, if you're familiar with that, how that works, you're creating the robot count from within the project configuration page right now. And, you know, that's that's also understandable projects go permissions become an issue when an image in one project is expected to extend an image in a different project right which is not the case if you're using images within harbor as part of your Docker build, and you're executing the doctor builds in multiple stages. Each from instruction would use a different base image and built iteratively. So that's exactly we're going to be delivering in the 2.2 as the system admin persona. You'll be able to create system level robot accounts in addition to project level robot accounts which are only available to project admins. And then, you know, you can choose the projects that that robot account that particular robot account has permissions to do during the creation of their robot account. You can also choose the particular set of actions actions that they can perform. So like pushing image or pulling image deleting images and much more. And then you'll be able to change the set of projects and the set of actions at any given time without having to recreate a robot account with a new token right which is the experience right now. So if that token is hard coded somewhere into your CI, or it's being used to construct a poll secret and used to be manually recycled right now so we're trying to kind of. We're trying to remove this kind of friction so that changing the project scope or changing the permissions of that robot will not impact your CI in any way. So again, this is being actively discussed and being worked on. I've linked the PR as well to this proposal in our committee repo. If you have any questions, comments, please, you know, share your feelings on the PR. If you have already implemented your own solution, we'd be really interested in hearing that as well. And finally, if you've been tracking our progress or attended any of our community meetings over the last six to 12 months really. So you know that we've been working on an operator deployment of Harper that will, you know, run it as a Kubernetes cluster, sort of similar to our help and deployment right now but it has certain advantages when it comes to the day to operations and it's also much more intuitive for performing some of the more heavy duty operations like upgrades redeploys backups and restores. And to give a little background, this really started with OVH cloud. I want to give a shout out to OVH for kickstarting this effort and doing a really great job. OVH cloud team built the first version of the Harbor operator. And you can find this as the core operator repo under the Go Harbor project on GitHub. But it doesn't include certain dependent services like databases, Redis cache or storage because, you know, some vendors already have those services running as managed services. And they're perhaps already deployed in a much more highly distributed or highly available manner. And that makes sense right that's that's definitely one use case that we have to we have to cover. But, you know, for the other users that are perhaps not as advanced, that's looking for an all in one deployment that comes with all the components that Harbor needs to run. We have built what's called the Harbor cluster operator, and that will basically deploy Harbor with all the dependent services like database, Redis cache and storage and deploy these in HA mode. So we don't have to deep dive every term every aspect of this diagram, but the key takeaway here is that you can see that in this diagram there's a Harbor Harbor cluster CR at the very top right so there's a there's a Harbor cluster controller. That's essentially managing all the services needed for the registry and the cluster operator custom resource encapsulates the core operator custom resource that I just talked about. As well as the custom resources for all the other individual components. And so the Harbor cluster is basically watching over everything and continuously reconciling to make sure that what is running is is what is actually specified in the CR spec. And so if there are any failures or hiccups, then it can automatically redeploy those services. So now for anyone who's wondering about high availability in the sense of multiple harbors. Harbor clusters across different data centers, perhaps in different availability zones even we do have a story in the works to address that as well, but that's sort of outside the scope of the Harbor cluster operator. I know that there's the people have been asking about it so I just want to address it and say you know this is something that we are aware of and it's something that we're working towards. And so for each of these components you can customize the spec for that CRD essentially right you can specify the number of replicas you can for for something like cash for the red is cash or for the database. You can choose to use an external cash or you can declare in class and run that in a J mode. So we have a really detailed proposal detailed meeting notes on the progress for the cluster operator. Under the Go Harbor project so please feel free to check that out and share your thoughts, you know we're still wanting more feedback on design because we will only have released the very first cluster operator version 1.0. By the time this recording is shared. So that's everything we want to share related to the 2.1 as well as the upcoming 2.2. Just a quick word on collaborating with us and how to get more involved in the community. Please follow our Twitter if you want to be alerted to any of the latest announcements on releases community events collaborations or anything Harbor related. We're still holding the bi-weekly committee meetings, one for us time zone and one for a pack. This is where, you know we have some of the feature demos, like the ones that Daniel just did. And we highlight some of the latest developments in the project and anyone can come to resume meeting and ask questions or share your experiences around Harbor. You can find the detailed meeting notes in the meeting details in the read me for the Harbor project on GitHub. And finally we do have a demo deployed that anyone can just go on and and give it a try get familiar with the UI, create a project, add users, push and pull images to and from that project. So demo.coharvard.io. So that's all I have for today. Thanks Daniel for sharing a demo. And then, you know, we'll take any questions for the remaining five to 10 minutes. So thanks everyone for staying with us so far.