 Hello, everyone. Welcome to COOP Conference and the Cloud Native Conference China 2021. Thank you for joining this session. I'm Lu Yue from Shanghai. Ludwig and myself from HPIT will talk about end price and harbor, a good partnership in helping each other. This is our agenda. I will start with an introduction about Ludwig and myself and what we are doing in HP. And then we will describe the differences on the end price strategy and open source strategy. Why it's important that we should build a partnership between them. I will also briefly share the existing adoption and the status of harbor service in HP. And then I will hand over to Ludwig to share more about how we contribute back to the open source community and what we have done to turn harbor into an end price great docker-grade strategy. Both Ludwig and myself are from HPIT. I'm an IT developer manager that has automation and cloud native practice. Ludwig, do you want to introduce yourself? Hi. Good morning. Good evening, everyone. My name is Ludwig Lim. I'm an HPIT cloud architect. I mostly focus on cloud cost optimization and moving applications to containers. Back to you, Yeh. Thank you, Ludwig. In HP, IT delivers engineering and software development solutions for R&D communities to create great products, short the product life cycle, lower the cost, and improve the quality. We're also catching up in budget chant, modernize IT practice with latest technology, cloud native tools, and frameworks. IT supports over 17,000 developers from R&D, labs, CTO, and IT organization across the world. Ludwig and myself start to attend the CUBE conference from 2018. And we know Steve and build a very good relationship with Hub team. There was a gap on the Docker-registered service in HP. So we decided to adopt Harbor into HPIT. It's a very interesting journey. And we feel some essential gaps because of the key strategic differences between end price and open source. For example, end price requires the service to be stable. In HP, the essay of our Harbor service is 99.5% availability. And the customers are very sensitive on the performance. Operating teams care about the maintenance effort, such as patching and upgrading. Every end-price service also has some metrics like time-to-response, time-to-resolve on issues. So how to gather right support on right time is also critical. For HP Harbor service, time-to-response is one working hour. And time-to-resolve is four working hours on critical issues. So you can imagine the changes. Each end-price environment substituted is always an important consideration, be it vulnerabilities, authentication, authorization, encryption on rest. And meanwhile, and if you look at open-source work, it usually demo-cheats the flexibility, innovation, and latest technology. And the community is also a very good place for learning and collaboration. But there are no guarantees as there is no support because there are no dedicated resources. Each open-source software has its own roadmap for new features, but they're not always put the end-price requirements with high priority. So we definitely need some good approach to build the synergy between end-price and open-source. And we are going to share you our learning from the journey deploying the harbor service into HP from 2020. Today, there are 2,600 repos with 11 TB images on the 300 projects. Every day, there are about 30,000 images approved and deployed to internal and external platforms. OK, now let me hand over to Ludwig and share more about how we contribute back to the open-source community and what we have done to turn Harbor into an end-price with great Dr. Rajesh. Ludwig, your turn. OK, thank you very much, Ye. OK, so let's start with how enterprise can help in Harbor development. For this, we want to share our contributions to the Harbor development. Our first contribution to the Harbor development is disabling anonymous access. Our open-source and enterprise have produced great quality software, although the focus of both is very different. For open-source projects, the focus is more on collaboration and openness. Well, for enterprise, they have their own intellectual properties to protect. We contribute to the Harbor project by submitting a pull request that, optionally, this allows anonymous access to Harbor. This way, Harbor can work for both enterprise and as well as for open-source. Our second contribution to the Harbor project is OIDC testing. HP is one of the first companies that use OIDC integration feature of Harbor. Being an enterprise, we have access to authentication tools that have complex mechanisms such as OIDC. Also, HP has thousands of employees that can help in testing this feature. We help by submitting feedbacks and bug reports to the Harbor team on the OIDC integration feature. This is something that is very difficult to do if we just rely on the open-source community. Next, our next topic for the talk is turning Harbor into an enterprise-grade internal Docker registry. For this topic, we have three subtopics. First, the technical aspects. Second, the end user or customer aspects. And third, the security or compliance aspect. So let's go to the technical aspects. First, we need to understand the Harbor components. During our initial testing with Harbor, one of the challenges that we met is understanding Harbor components because it seems that every time we install, something different came up. For example, when we change the exposed type to load balancer, the Harbor installer will create an engine X controller. But if we set the exposed type to ingress, the engine X controller will not be created by the Harbor installer. So you need to be very careful. You need to understand how the configuration works because in the end, this configuration will affect how your Harbor architecture will look like at the end. Tip number two, leverage auto-scaling, okay? So for this tip, I'm going to show you a demo how HP uses HPA for Harbor. One of the challenges we have in maintaining Harbor is our customers are located all around the world. With a small team, we cannot have 24 by seven support. So how do we manage heavy workloads when they come suddenly? For example, those heavy workloads that come on a Saturday when we're both out of office. One of the solutions for this problem is to leverage Kubernetes horizontal pod auto-scaling or HPA. So let's start by showing you the current setup we have for this demo environment, okay? I'm going to show you the current pod replica configuration and as well as the HPA configuration. So let's do a K get deployment for the Harbor, okay? You can see that for Harbor core, we have two replicas and for registry, we have two replicas. And if we go get the HPA configuration, okay, let's do that again. We start with minimum of two pods for the engine X and for the registry, but it can scale to maximum of five pods for both the engine X and for the registry, it can scale to 15 pods. Now I'm going to run few test stress test scripts. These scripts will push Docker images and do multiple Docker logins, okay? So we're going to do multiple push and multiple logins. Let me run the script. Okay, there you go. This one and finally this one, okay? So since HPA will take some time to kick in, let us proceed to the next slides, okay? Tip number three, don't ignore pod restarts. You might be curious, Kubernetes has a self-healing function, so why worry about intermittent pod restarts? Well, because pod restarts are just symptoms of a bigger problem. So if you don't fix this problem at the early stage of your deployment, when your deployment gets bigger, more complex and there are more users, chances are you will have problems every day. So let me share to you one of the experience that we have, like for, we have like rare occasion where in our Harbor pods crash. And although it automatically restarts, we still want to fix this problem, okay? After long investigation, we found out that the cause of this was because it ran out of memory and after the pod run out of memory, it crash. Although Kubernetes was able to self-heal, users still notice it. Our next tip, don't treat containers as block boxes. Although Kubernetes provides layers of abstraction for your containers. To make your Harbor installation more robust, you need to understand the configuration of these containers. Different cloud providers, node architectures, instance types and container runtimes affect how your containers are created. Low limits will cause performance issues and in the worst case cause a pod to fail. Look at the screen that I'm sharing. You can see that both of these are for Harbor instances, okay, on different Kubernetes cluster, but on the same cloud provider. You can see that for this one, it has an open file of 1024. This issue actually caused our Docker registry pod to fail when people are doing multiple push because the number of files that it can open are so small. So we fix it by increasing the number of open files, okay? Okay, next tip, pay attention to external components. Harbor uses the following external components. Number one, Postgres SQL. It uses the Postgres DB to store like user credentials, project information like the project name and the repositories that you have. And if you have multiple users trying to do multiple Docker logins, the number of connections will increase. So you need to check the number of connections, especially if you have many users. If your number of connections hit the ceiling and your Docker will say that Docker login failed, okay? Also, if you have engine X as part of your Harbor installation, you need to check the worker connection settings for engine X. The default worker connection setting of engine X is very low and it's not suitable for production workload. You can check this by editing the config map of the engine X pod. Okay, now let's look at the HPA results. First, we go and get the HPA again. You can see now that for the registry, it has five replicas, okay? From the initial to replicas, we now have five replicas. And for the engine X, it has now three replicas. So let's do a cube CTL get pods to see if those pods are indeed created. So K get pods. You can see now we have five registry pods running. Let's go and check at the engine X pods. You can see that there are three pods and you can see that the pod here, it was just created about four minutes ago. So this is the proof that autoscaling indeed work, okay? For Harbor. Okay, now let's go to the next section, customer and end user aspect, okay? Our next tip for you is you need to know your customers, okay? What kind of images does your customer produce? Different type of applications will produce different kind of darker images. For our, in our experience, we have a customer who reported to us like he's having issues doing darker push to his registry. And when we investigate, we found out that his Docker image just are about two gigabytes. It turned out that his application is a machine learning application. And it has a lot of like machine learning data in this image. That's why it took a long time for the image to push, okay? And you need to check like, how often do they pull their image, okay? The number of pools can help you in estimating the workload that your customers will be doing in production. When you know their workload, you can do in advance like the tuning for the database connections, for the networking, okay? For global customers, are they pulling their images at the same time from a different location? So you need to check this one because sometimes the problem might be because of Harbor's performance. Other times it might be because of networking. Let's say you're pulling your Docker image from an on-premise network. So if you know where your customers are pulling their images, you can help them in troubleshooting, okay? And the last tip is involve your customers in testing. Different customers use Harbor differently. And in the end, your customers are the one using Harbor, not the IT team. So you let them be involved in the Harbor testing even at the early stage. Next section, security aspect, okay? We'll look at the security aspects in security aspects of turning Harbor into an enterprise grade internal Docker registry. First, when you deploy Harbor, you need to work with your cybersecurity team. For HP, before we move Harbor to production, we had lots of consultations. We have lots of reviews, architectural reviews, cybersecurity reviews with the cybersecurity team. They check our architecture. They make sure that the HP's intellectual property will be protected. So for enterprise such as HP, the IT team has responsibility of making sure that the HP's intellectual properties are properly protected. So this is one of the reasons why we submitted a pull request to the Harbor team to disable anonymous access. Number two, integration with official or security tools. So usually big companies like HP have partnership with cybersecurity companies that sells product that protects their assets. Let's say image scans, antivirus and stuff like that. So in the beginning of the Harbor project, when we tried to adopt Harbor, we worked with the Harbor team, we worked with the anchor team in the anchor plugin for Harbor. We not only test the anchor plugin, but we actually submit pull requests and code to the anchor plugin for Harbor. You can see our contribution at the GitHub repo below. Next, intranet or internet facing. So for intranet facing or internet facing cybersecurity, your cybersecurity might have different policies for this. You need to check with them before launching the Harbor application. Okay, that ends our presentation for today. In summary, we show you how we apply open source software like Harbor to suit HP's need. We also shared our experience and tips on making Harbor enterprise ready. Yeh and I want to thank you for your, for attending our presentation. Thank you everyone. We hope our Sherry can help you in your open source journey. Cool.