 And thank you for having me today. I talk about today scalable deployments. Today I talk about how we cook but deploy. And I will not talk about topics here. I will only talk just only deployments. OK. Let me introduce myself. I'm Sula from Japan, and I'm still the youngest of Ruby Coaching, I'm still 17. So I have worked hard to give my parents approval for this clip, and giving them a piece of mind, because this part of the clip is too far. And I work full-time for CookBot for the biggest famous recipe-sharing website in Japan. OK, so deployments. First, here's our legs up of my legs up. As I said, we are learning famous and biggest application in Japan. So we perform deployments of this huge application showing here, every eight. Like over 1,000 high 500 models, and live calls is overflowed. And our infrastructure is we have over 140 servers and ship over 10 times per day in peak. Because we are allowing all developers to perform deployments by their own and their tools. Code should be green on CI, only time working hours. Developers should watch hours for an hour after they deploy. And developers should roll back if they are led in places or having any troubles. Next, our deployment flow was like this. Once a topic bunch gets merged, CI runs a build. If it succeeded, CI creates new Git tag, then start deploy automatically to staging server. Developers are staging them, and finally initiates a deployment to get production. Previously, we had used CapiPhone 2 with our sync over SSH. We put deploy server for kick off CapiPhone in our infrastructure because Cap takes too much bandwidth and why so it can't run from developer's local machine. And we have chat ops, too, like this. He would deploy cookware. OK, do you see this flow good? But we have a problem for this flow. It's a time spent for deployments. We run text cases in the fastest weekend, about 10 minutes. In order to get that fast, we have to do parallel testing and more hacks to run this fast. So we can get more shorter because the CI run is the fastest weekend. So this can be shorter. Then, developer takes five minutes to check staging. And finally, CapiPhone takes over 10 minutes. It's clocked. So developer must take about 15 to 20 minutes after they have wanted to deploy changes on production. And we also have some issues, kind of bad smells on deployment flow. First, we were using all the CapiPhone version with complicated and super historical Cap file. So it was time to leave it from scratch. For instance, we have a lot of files showing here. And over 2,000 and 500 of lines for deployment. This is Cap file. It had to leave. And I couldn't know where is the important part of deployment code, deployment script. Next, sequential SSH is slow. As I said, we have many servers. It consumes high CPU usage and sometimes fails due to timeout. Sometimes fails. Critical. You know? This is the same diagram shown before. If there's no other deployment finish in about 20 minutes, that usually the error we encounter is a timeout. So when error occurs, we have to wait for a few minutes. For instance, 3 minutes late. And then we try it. In the worst case, we have to try again. So deployment step will be not 10 minutes. Take more than 10 minutes. Long time span from branch have get merged. And the developer want to get change on production. OK. Next, let me introduce my team. My team, development infrastructure, aims to keep and improve developer productivity, like keeping development fast and maintain and improve our test environment and something like that. Of course, we have saw this state is a problem, so I started working on improve it. So here's the plans. One is replacing with Capitron 3 with Scratch. I have here SSH problems have improved from Capitron 3. But I couldn't think SSH scales for future server growth because we are still glowing. So I expect more servers in future. So after discussion in the scene, I decided to create new tool. So here's introducing Mamiya. It's already on my GitHub. So Mamiya use a tool named surf to orchestrate servers. And Amazon S3 for file distribution, also having compatibility with Capitron 2 and 3 for easier migration from Capitron to Mamiya. Surf is an orchestration tool from HashCop. How many do you know? Raise your hand if you know. Nope. Surf is an orchestration tool from HashCop. It's decentralized, fault-oriented, and highly available. It uses gossip protocol for communication each other. And the gossip protocol is, according to Wikipedia, a protocol inspired by a form of gossip. So quick introduction. We have eight nodes here. Someone initiates an event and then found out to random two nodes first. Repeat again. Repeating. The nodes, blobs, events already proceeded, a process. After a while, all nodes should have same event. Blobs 2 now all have same event. That's gossip protocol. It takes some UDP band-wise, but it's better than SSH. I also have created another gem called burn to control surf cluster. Check this out too if you are interested and how Mamiya works. We have these five concepts, master node, engine node, package, and storage steps. Master node controls and watches our whole cluster and all agents. It controls the whole cluster, sends requests to agents, and watches agent status, and has HTTP API to accept requests from human. Agent accepts requests from master node and learns deploy tasks from commands and blah, blah, blah. Deploy script contains how to prepare and restart up, contains how to build package, and within Ruby DSL. Package is a table of files to deploy. Yes, your application. It also contains deploy scripts that use to build itself. Scripts will be used from agent too. Flage source packages. Amazon S3 is available out of the box, but you can write to your custom Flage adapter if you want. Then step is a part of deployment process. It can be run separately and call remotely from master node. Steps is like this, fetch, prepare, and switch. If switch has called first, Mamiya kicks fetch and prepare tasks first automatically because they are required to learn switch tasks. OK. And here's deployment flow with Mamiya. CI builds package when passed, and CI pursues the package to Flage. Then deployment starts. Master sends prepare request to agents. Agent fetches package. Then prepare it. Next, master confirms all agents have prepared, and master sends switch request to agents. Then finally, agent switches him links, and then reload that process. In a diagram like this, so by using Mamiya results, no dependency to SSH that is slow, but we can do more, more faster. So let's back to previous slide. Steps is a slice step of deployment, deployment like fetching package, preparing package, and releasing it. These steps can be run separately. So we can do application preparation earlier before developers say deploy, like bundle install and assets compile. So now flow becomes like this. CI calls prepare step just after completed push package. So preparation will be completed while developers is checking staging server. And this is the result. Using Mamiya with doing preparation earlier, deploy takes only 30 seconds for 140 servers. 16 times faster than Capicilano. And see this graph too. This graph records time spent for deployment in Cookpad. After using Mamiya, value is lower than under one minute. OK, let's get them. This is recorded. So Mamiya has example configuration in our report story. So I have started using Forman. And this is included in Mamiya report story and allows to testing local. So for testing, we have two directories in local and that used by two agents. And this is deploy scripts used for Mamiya. And both used from building package and prepare, then releasing. So this goes too quickly. So I have inserted a flip to get know. Then this is building package using deploy script built. And one table and one system for metadata syncing into push on to storage. You can inspect the cluster using Mamiya client command. It communicates to the master node, listing packages and list agents available on cluster. And you can inspect agent status detail of the agent. Prepare the agent with push package, getting the package name. That's what it means, application name. So as you can see, another panel running Mamiya is have something work. And you can see a status of package that has been prepared. The directories for agents have been changed with new package and releases. Clearly, this means it's for package that's prepared but not released yet. So finally, we can deploy using deploy command to release the package. As same as a capitalano, we have client synlink and release directly. And build the package again and push it. You can skip the prepare because you can deploy with skipping prepare because they initiate prepare if the specified package has not prepared yet. Let's check the directories again. Client synlink have been replaced with new package. And we have new package directly in agent directly. So that's all. And another demo that do rollback. This goes really quick than capitalano. Just run. You can see application status. And there, that have a common previous release and current package having a majority. And we are rolling back to the previous package. We have deployed. This is just called client rollback and this application. It detects previous release automatically and rollback to it. Rollback has just done using the same deploy process. So if some agent has not the previous package, it will start automatically fetch and prepare same switches. So I have a huge difference here. Mamiya is still not perfect. It doesn't have rich UI, better documentation, and rich alert walking. Cliently, alert walking has required to log into server and see the log in remote server. So I have to collect into one master node. So I have a plan to implement later. And we have some alternatives. Essentially, such code that was on Amazon Web Services. I have applied that and seems nice for me. I think Mamiya may be overkill for most environment because you have to start maintaining the cluster about like a clashing or some alarms. So use two double tools for your environment. Capacitorals still can be a choice for you. Because just for us, we want to deploy faster. So I have created and installed this tool. But using this is up to you. OK, that's all. Thank you.