 Hey everyone, today we will talk about driving continuous delivery, trunk-based deployment strategy with Pipecity in large scale. So first of all, let me introduce ourselves. So I'm Yoshiki Fujikane. I work as software engineer from Cybiagent and currently we belongs to the developer productivity teams and I'm a member of Pipecity team. My name is Ken and I'm from the same team with Fujikane. I'm software engineer at Cybiagent and maintain our Pipecity project. So first of all, let me express my gratitude. So this announcement would not have been possible without the cooperation of Abema's back-end teams. Thank you again. Okay, so for the agenda of our speak today, first we will talk a little bit about productivity and durability. Second, we introduced the Pipecity, a tool that realized the progressive delivery with off. And lastly, we send you stay, the chunk-based deployment with Pipecity. So first of all, I would like to introduce a little bit about the Abema, how you skate today. The Abema is serviced from our company Cybiagent. Abema is famous video streaming service in Japan and we handle various content such as anime, drama and sport. We have, we are leading traffic handling service in Japan. We serve around 30 million user activity and for the development side, we have a back-end engineer team with around 20, it's about 12, 20 members and we handle about 130 microservice of many guys. We still in the growing phase and the more productivity and the more reliability is the better for our growing as a service. So why productivity and reliability are important? Productivity is mean for delivering the new values, the new features, the new bug fix, new thing to user. Why reliability is providing stable for existing thing that's serving user already. So it's not about productivity or reliability. We think that it has to be both, it has to be productivity and reliability and we have to find a balance for them. So we think how we can actually ensure productivity and reliability for our service, how we can actually balance them to ensure that our service growing good. We investigate it and we realize that track keeping are required. We introduce chunk-by-deployments and the mean for ensure productivity and progressive delivery as the mean to ensure reliability. So for the chunk-by-deployment it's a well-known deployment method. We develop a small and frequent PR to update a core chunks or a main branch. This allows the significant reduce the time it takes to resolve the conflict between the revision from different reviews. We all know that the resolve conflicting is not an easy work and it slowed out our team. Secondly, for the progressive delivery is about gradually rolling out the feature to the selected user. Developer and DevOps team can increase incrementally roll out and deliver the new feature with fine gain control to a small group of users to minimize the risk of pushing new feature to the production environments. And it's not just new feature. It's maybe new bug fix, new things, everything that we face in the daily life of engineering. For the progressive deliveries, we have two well-known methods which is canary deployment and bugging deployment. For the canary deployment, we have full control of how many percent of traffic we do to the new feature and how many was still serving by the old workload. And for the canary for the bugging deployment, we have kind of switching things between new and old workload. And it's not just like that. We need progressive deliveries, we need canary deployment, and also we need the board for all of our services. It's kind of specific problem with our infra, but we have a requirement for the monthly platform. Our system consists of many different components and we have several elements for monthly platform. For instance, API workload, we are using QRities on the managed cloud like ZKE or EKS. And on the other hand, for the batch workload, we use serverless things like LLS Lambda. We want to realize progressive deliveries for monthly platform. That's why we choose PyveCD. So a little bit more information about PyveCD. The PyveCD as a vision is the one CD for all application, platform, and operations. It's a bit of style CD solution. Currently, it's support deploy application to multi-platform. Most of our using platform is supported by PyveCD, such as QMTIS and AS Lambda ASI. Next, PyveCD supports progressive deliveries in mice, so that it's really easy to get in. Lastly, PyveCD is already a CNCF sandbox project. So now let's take a look at some actual use cases. We will explain trunk-based deployment using PyveCD after a demo. So we know the session title is trunk-based deployment, but we also mentioned previously that we need not only trunk-based deployment, but also progressive delivery pattern to achieve our needs for the previous productivity and reliability for our product. And great free PyveCD can help us achieve both. We will show you right away. In our demo, following trunk-based deployments, we separate the feature branch from the main branch, and we will work on this branch to keep the changes as small as possible. Additionally, we speed up development cycles by encouraging culture to review PR as a priority. Up to this point, this is standard deployment development that follows so-called trunk-based. However, ABMA has taken another step to further improvement deployment development efficiency, and the point is to merge functions into development plots at the same time. Let's learn about this while looking at actual CI CD configuration diagrams. So this is ABMA's CI CD flow. Let's take a closer look at each steps. First, create PR. So developers first make modification to branch based on the main branch and create a PR here. And once created, CI will automatically commit the version fixes required for PRD release. And this will cause development to begin when this PR is merged. Once the PRs are merged, CI creates artifact for deployment to production. At this time, artifact information is notified to 5CD as an event. By adapting this architecture, CI and CD are root-free couples, and each one does not need to know authentication information. From here on, the behavior on the CD side, 5CD updates the application's manifest files based on the event it received. And this is a feature called event watcher of 5CD. When defining an application managed by 5CD, you can define what behavior will occur in response to which event. And specifically, in response to certain events, it is set to update the image name of the corresponding file and the commit. And this saves you the trouble of creating a fixed PR for your manifest version. And 5CD regularly checks for changes in the manifest. So after a certain period of time, it will detect the difference in and change and reflect them in each environment. The sync strategy is different between dev and broad, so I will explain it next. So 5CD has two main sync strategies to choose from. One strategy is called Quick Sync, which simply updates the differences to our application. Another strategy is called Pipeline Sync. And you can create your own pipeline by combining stages provided by 5CD. And Abema mainly use Pipeline Sync to configure a pipeline that can achieve progressive delivery. Let's take a look at Proto and Dev Pipeline. At first, Proto. Abema's production environment uses Canary release. And first of all, we will release a few percent of the number of replicas as new version Canaries. And next is the analysis phase. So check the error rate of Canary for about 15 minutes and make sure that errors above a certain level don't occur. And this enables progressive delivery. And this feature implemented as standard in PipCD is called Automated Deployment Analysis. Based on the metrics stored in Prometheus and Detoc, you can determine whether the correct funding version of the application is normal. And if an abnormality is detected, you can automatically roll back to the previous version. And next, if ADI determines that it is normal, we will change our application to the new version after obtaining the final approval of the person making the changes. So after all, release is finished. So on the other hand, in the Dev environment, the first step is to roll everything out to the new version. This is Dev environment, so there is no need to do Canary strategy. After that, we analyze it using ADA. And if there is a problem, we immediately roll back to the older version. So let's go back to the previous question. Why match to Dev Proto at the same time? So this is because the lead time during deployment can be further shortened. I have illustrated the flow from the actual match to the deployments of Dev and Proto. While we are checking the operation in Dev environment, we are preparing to change the Proto environment to a new version at any time. If we discover a problem with the Proto at the time of ADA, it will automatically roll back and minimizing the impact on the production environment and minimize the time to release. So ultimately, we have recently been able to achieve average 22 release per day. And this means that the members currently involved are releasing at least once a day. And furthermore, this is the result of checking the latest recording on the PIPCD console. We can confirm that the development cycle is running at a very high pace. So currently, we are implementing this for some projects, but we hope to expand the scope of applications in the future. So thank you so much. Thank you for listening. Thank you all for listening. We have official sites at PyCDR Dev and live demo at play.pycdr.dev. So everything is up online. Just give it a try. Thank you.