 a large scale deploy over the 100 servers in three minutes. Nihar, this slide is first and the last slide in Chinese and Japanese. So, self-adaptation. So, my name is Hiroshi Wata. So, please call me Hiroshi. So, my internet nickname is HSVT. It is used on Twitter and GitHub. So, my job title is Chief Engineer at GMO Peppable. So, I have many commit bits such as Ruby, and Lake, and VGMS, and other, and the second Ruby bit, and it's that stuff. So, I am a CRB Committer. So, I maintain these websites too. www.ruby.org, and bugs.ruby.org. This is our issue tracker of Ruby Core languages, and Ruby CI, and RailsGun, and RailsGun JP. So, I advertise Ruby Core team. So, we are developing to Ruby 2.4 now. So, if you have any issue or future requests, please submit our issue tracker, and this site. So, we hold the core development meeting every month from last year. So, we will discuss your issue with our face-to-face listening. I'm one of the organizers at 6RB. So, it is most active we meet up in Tokyo every Tuesday. So, if you visit Tokyo, Japan, please mention to these members on Twitter. We will pick up you near less sessions, and I'm working for ZMO Paypalvo in Japan. ZMO Paypalvo is Tokyo, and Fukuoka based a rating tech company that has a strong advantage in technologies, including Ruby-related ones, such as C-Ruby, and Rails, and M-Ruby. So, Ruby-Kai 2016 is coming to Kyoto in September. You know? So, CFP is still open until Sunday. You have our two days. So, please submit your proposal to Ruby-Kai. So, we are looking forward to see you, Kyoto, in Japan. Please submit. I need to submit, too. Interaction, it ends. So, let's look at the global strategy for next generation. In November 2014, CEO and CEO said three years ago, CEO, we are going to promote our service on TV show and TVCM at February 2015. CEO said, to make our service to scalable and redundant and high-performance architecture in three months. Yes, yes, yes, yes, I do it. But our service status is simple, where is the architecture on AWS? So, we have only six servers and use Capacitoral 2 for deployment. There are mix of background job and application process and batch test. So, it is complex architecture. It's my year in this show. So, our service has many issues for the scalable architecture. TVMedia has a huge impact for web access in Japan. We need to find things. So, do scale, do scale with automation. Do scale with rabbit automation. Do scale with extremely rapid automation. I plan to make scalable infrastructure to increase application servers automatically. But our web operation is manual instructions at the time. Typical scenario of the server setup for scale following instructions. First, launch the next instance for a low image. Modify basic configuration without dependency of raise application. Provide an instance used by provision to prepare to deployment, deploy raise application and testing and attach all of the servers to a load balancer. We have been created, or it's limited called the golden events from running a server. It doesn't name is a snapshot. Web operations such as OS configuration and instance, says launch a manual instruction by the developer. It's working time is about four to six hours. This is that cannot scale out in TVCM and TVShow time. It's a broker for Razi scale out. We added no SSH rule into our team. I describe the background of no SSH rule. So in Razi scale service, one server instance is like one processes in UNIX environment, I think. So we didn't attach process using ZDB usually. It means we do not access instance via SSH. We didn't modify program variables in memory usually. It means we do not change the configuration on instance via SSH. We can handle, we decided to handle instance status using API only. How about no SSH? One of two is Puppet. We put out Puppet Muffets with automation. However, it is handlebox status not for production environment. We need to fix all of Puppet Muffets. Example issue are these things. It is based on all those scientific facts. Some Muffets is broken. Developer didn't know how to use Puppet in production. At first, I fixed all of the program manifest and the inability to deploy production environment. Finally, we can understand our server configuration via code. So our manifest over writing number is 5,000 and 500. We can use Puppet for standard provision too. However, one of its requirement is distribute manifest files each servers. So we cannot deploy Puppet Muffets each server because goal of our architecture is over the 100 servers. We choice master agent model for Puppet users. Puppet master D is a master server on agent mode with the Puppet. Next tool is cloud init. We discover awesome operation tools. What's cloud init? Cloud init is a de facto master distribution package that handles the early initialization of our cloud instance. In fact, we and you already use the cloud init for customizing to OS configuration at initialization process on IaaS like AWS and Azure and the GCP. We solved automation of OS configuration via cloud init. Cloud init has one of the significant programs. So cloud init has few documents for our use cases. No document. How customized basic configuration via cloud init is like this. So this is Magica Wars, shop called me and to wrap up the date and wrap up grad in box yum or after update, it is refresh install the packages. Package section is going to install under the names. It installs a gate and the CRN and and it. And okay, time zone configuration of localization. You can modify configuration after boot time with adding this text to cloud init. Our advice is do not use run CMD section. Run CMD are all invoking all commands like the shell script and gen2 and more. We separated responsibility of OS configuration and provision for a simple architecture. After that, we apply puppet provisioning for application specific configuration like this. And finally, image creation with itself. It's embarked by instance like this. In this case, we use IaaS API for image creation with cloud init, user data. User data can handle your shell script. We can automate to OS boot and OS configuration and provision and provision development. Next, do scale it with rapid automation. I need to fix code on Rails application before automated to deploy via Capistan. Applying Rails app, I like applying Rails application. Applying Rails 4, we still use Rails 3 as the time two years ago. So I hope to use new API and syntax and component of Rails 4. I decided and plan to upgrade Rails 4 before our promotion of TV show. The planning production was performed with my colleagues. He tried to four times, I was upgrading it every very, very, very much. He did two hard works. Moreover, we use Capistan 2. So we relied on Capistan 2 tasks to our Capistan 3 conversion because the same reason of Rails app was, Capistan 3 tasks are simple tasks. We can write our focus and task to write our Rails DSL. And our Rails application has dependency over host names like this. Host name, start, ways, job, and host name, start, ways, host name, start, ways. So it's a lot of conditional calls. It break service host name. It break service because host name is dynamically assigned when it scale up. We discarded dependencies of host name and IP address like this code. We use only value fetched from our IS API. Next topic is Rails bundle. We need to deploy Rails application without human deployment in booting time. We prepared to standard our Rails application files with Ruby Jam, library, and pre-compiled assets. We make this package via Capistan task like this. Like this. So it is part of Capistan task. End job directory is checking out Rails calls and invoking to bundle install, FN install, binary install. After arrival is installation, it invokes pre-compiled assets. Finally, compress power archive and upload object storage like S3 and clear up bugging database. Now it is just right to our ultimate architecture of Rails deployment. So we create Rails bundle via Capistan and upper-size build server, build our Rails packages and upload S3. And our instances and the image creation server, the fetch Rails packages without deployment by human. How we integrate the image creation and the Rails bundle? So we extract the Rails bundle when instances creates a self-image with a cloud in it. We added these instructions of the public provisioning. So we can automate it or split configuration and provision and the preparation of application and the development of the Rails code. We can do it. So how to test the instance behavior? So we need to guarantee HP status from instance response. We remove the package version control from our concerns with behavior tests. We launch instances from an image created by automation. We test of STP behavior on single instance manually. We focused only the outside of servers. After test passes, so we launch 100 servers from our image. We can scan out result or instructions by a human. After essential topic, we use toll for image creation and my page ISAPI. We create subcommands on toll-related instance operation like this. Moreover, the center command of instance launch is like this. So we can launch instance from a command of awesome tool instance launch only. We create all of the web operation by a tool command. So we have no SSC rule. So write codes and do operation, write code and do operation. All of the web operation should be implemented by a Ruby code. So instance launch and scale up, do scale up and do developer in the environment. So our advice is do not change my architecture first time. So example, I use Google power home. So software and service architecture is too complex. So we need to find things. Our real world instructions pick instruction for automation and after do automation. We can automate half instruction. Next issue is do scale up with extremely rapid automation. So we can scale up with complete automation but all of the time of image creation is a long time like one hour. I concern but to stop time. Typical scenario again, build, configuration and provision setting up, deploy and attach to load balancer. We need to enhance to boot stop time extremely. So I group these instructions. So operation is always built and provisioning too and deploy raise the applications. First operation is OS configuration and setting up Capitain and attach to load balancer. I set it checkpoint over image creation. These are checkpoint one is provisioning with profit. Checkpoint two is deployment or raise applications. I added a concept of two phase image strategy. I create these images first is option OS image. It is provided a platform like LOS, Azure and ZCP and OpenStack and etc. Phase one is called minimal image. Me and my image have only network configuration and user creation and installation of a puppet ship and platform CLI tools. Phase two image called load specified image. It contains only boot OS and raise application without modifying configuration. We need to use only phase two image for scale out and preparation. It reduces our working time for image creation. We cut these images. We found the packer for our image creation. I described to our image creation tool or packer. I couldn't understand the use case of a packer at first time. I had to use the cloud in it and the image creation is the IR CPI. After that, I understood use case of packer. It is inside of image creation with packer. Packer configuration is JSON format. We set instance size, block volume and CPU resource with a declaration of JSON value. Packer I was invoking cloud in it. We use basic configuration of operating system. Packer provision I was invoking your shell script after cloud in it. Finally, packer invoke image creation API itself. It is an example of packer configuration for our minimal image. So, cloud in it is here. It is same as our first configuration. Moreover, provision is here, provision. We need to set a provision tool. YAM install, Puppet and Python PRP and other CLI. This line, I was changing the user name over through users. This line, I was presubbed host name. If this configuration is true, cloud in it did not modify host name every boot time. It is first cloud need to modify host name use a specific rule like IP address or unique ID of cloud platform every boot time. It is an example packer configuration of web application image. Cloud in it is only this one. We need to rename host name on web application servers after boot it. Because we use over the 100 servers, it needs to have identical host name. Provision is here. It invokes Puppet agent and fetches a raise up bundle packages. Moreover, remove cloud in it ID file, this line. So, if you remove instance ID file like this, cloud in it invokes the processes and the next boot time again. I describe integration tests with packer use the shell provision. Packer allows multiple provision. We test results of packer running before image creation. We use server spec for tests. It's like this configuration. It tests to confirm installation packages, the location of raise application and the boot configuration middleware and more. So, I will describe detail of the server spec of the site. So, we can guarantee to launch from server image with packer and server spec. We create it share it with all before packer. Packer is simple, so simple. But we have much use case. Example, we have many role like web application server and batch server and worker server and the party server and etc. And we have two files image. So, we didn't hope to make instruction manual describe the packer options. We can run packer over toll code with RMS option or packer. Like this. Now, we can create image services for rocket automation via packer. However, we are not enough for do scale out. We can only scale out to raise application server. So, we need to scale out everything, everything scale out. What's broker for scale out? I think these are brokers. Depends on money instruction of human. Second, depends on host name or IP address architecture and tool. Depends on persistent server or worker like a paragraph job. And depends on a persistent storage. It's a broker for scale out. We need to use cloud oriented architecture. Moreover, adopt next generation architecture aggressively. One of tool named console. We use Nagios for monitoring to service and instance status. However, we have two issues. Nagios do not support dynamic scale architecture. We need to add host name to Nagios configuration each server automatically. Also, Nagios have complex syntax and configuration. I never pass Nagios configuration one time. We decide to remove Nagios for service monitoring. After that, how to monitoring for service status without Nagios? We use console and console are for process monitoring. Like this. So, it provides to discover new instances automatically. Use console. And a lot of mechanism will suck integration. We set up console. Use the puppet for design. When instance run to console discovers cluster on our service and join it, we can monitor new instance automatically. Like this. Next, we use Baccalaure. We use Moony for resource monitoring in tests. However, Moony does not support dynamic scale architecture too. We need to add our host name to Moony configuration and deploy it each server. We decide to use Makarai instead of Moony. Makarai is made by Hatena Cope. Hatena place the Kyoto is one of web service companies in Japan. We discuss and feedback together for cloud-oriented monitoring to Hatena Cope. So, it is a graph. This graph is all of CPU resources in our application servers. It is too heavy rendering. You only add API key and roll name to Makarai agent console. It works. Only works. Moreover, you can make your specific plugin for Makarai. It is simple convention and compatible for Moony and Nagas. Many of Japanese developers made useful Makarai plugin written by MROV and Golang and Ruby. Next topic is Florentine. We need to store access logs for our audits. We must handle all of the servers including scale updates instances. We use Florentine to collect and aggregate. Florentine is a popular log collector written by Ruby. Do you know Florentine? It has probable architecture. We can chain plugins for our use cases. Our company prepares an aggregate server for access log collector. This is a configuration of NZX on an application server. This is it monitors access log of NZX and pushes to aggregate server these sections, collect and push it. The aggregate server handles these logs and stores redundant local storage and sends service of Hadoop cluster named TorajaData. It sends TorajaData and stores redundant storage. Next topic is remove to batch server. Remove batch server. We need to use batch server for scheduled write tasks. We have to create some payment transaction or send a promotion mail and indexing such items and more. We use Venobar, Venobar gem and clone process on the persistent state server. But it could not scale up ruzzy and it is SPOF. I use side of the kick scheduler and console cluster instead of clone of above programs. Side of the kick scheduler allows a period of mechanism to start kick server. But if you have multiple workers all of the servers are promoted to NQ servers. We need to specify our NQ server in side of the kick workers. I selected NQ server use the console cluster like this. So I will extract this mechanism to Ruby gems after this conference. Next topic is testing. We need to test everything contains instance, images and the perfect and raise application. We need to test everything. We use content CI named Doron CI. We use Doron CI on our open stack platform. We use Doron CI with raise application. We need to separate raise stack to following containers. First is raise up only raise application. Second, we need to test this and MySQL and ASIC search. We involve concurrent test processes used by TestQ and T-Splm. I try to our application on Docker now. But it is not production status yet. We use only test environment. We call it to inference CI. We test server status such as this of instance packages and running processes and configuration details and continuously. We can reflect puppet myfests aggressively using Doron CI and server spec. Server spec is a test for your server's configure by project. We can test with server spec after provision of puppet myfests like raise application. Therefore, we can replace puppet master to puppet server. And we in our future parser, we fix all warnings and syntax error. So we add it and remove the manifest code every day like our raise applications. Every day, date, and edit it. And we can switch scientific Linux 6 to CentOS 7 used by inference CI. We add it case condition for our scientific Linux 6 and CentOS 7 like this. So we didn't feel fair expectation for this OS changes. Moreover, we change all of the processes under system V. We haven't used the demon tool or supervisor D to run background processes. However, we need to wait to invoke their processes before invoking our application processes like Unicorn and Sidekick and other. We use system V for invoking to our application process directly. It's so fast. So I finished preparing today's main camera for strategy development. We use stretcher and capital error for normal deployment. We use to wrap it scale out with pack and raise bundle. However, we need deployment for code deploy every day every time. Capital error couldn't deploy over the 100 servers because capital error needs to connect each server synchronously via SSH. Stretcher pulls assets file like a raise bundle and it receives the console event. We add stretcher to our application server and register console event for stretcher. We can deploy raise application to all of the server when we only send we only send event to one server. We send one server so console event or send and pull event and stretcher pulls our raise application automatically. We make capital error stretcher capital error stretcher. It is integration to with capital error and stretcher. It provides following task for process strategy development. We add our raise bundle archive file and put archive file to object storage like S3 and finally invoke console events each stage and each world. You can use process strategy deployment by capital stretcher and console and stretcher. I describe the architecture of process strategy deployment. We also invoke console a capital error on the local machine. Capital error hand to make our raise bundle and upload it and notify console events build and put our application package and send a console event. Think console has received that event and we have our raise bundle packages and extract it and restart unicorns itself. Console tells another server in your cluster. It handles the same instructions automatically and simultaneously. Process strategy deployment allows to handle much data center department. We choose to deploy open stack so open stack has widely used big company like Yahoo and DNA and entity group in Japan. We need to reduce running cost of IaaS. AWS is too expensive so we try to build open stack environments on our parameter servers in our DC but this subject is too large story so I have no time about this topic in last week. I hope to talk about this open stack migrations. Finally, we cut running cost by 15% rather than AWS and we made Yao calls and integrated Yao is Ruby client for open stack and Fog is already supported open stack but Fog has huge dependencies. We didn't need to do this. Yao likes simple clients like AWS SDK or AWS. We can wipe out computer users using Ruby with Yao like this. We can deploy multi-DC in 3 minutes now so I invoke capital on my machine in the same month. The capital 100 to make raise assets and upload and object storage is same. So console events send to customers so same happens each DC simultaneously. So we can invoke stretcher deployment all of the servers at the same time. The final topic is a burglary deployment. The basic concept is following instructions. Do you know burglary deployment? So run the instances using OS image created from Packer and join to load balancer create and join to load balancer and wait to change in service status wait, wait, wait and finally terminate all the instances. That's all. So it allows OS upgrade and Midori upgrade like Ruby version for us not downtime. We need to prepare dynamic upstream with load balancer for burglary deployment. I introduce our choices. First is ELB ELB or LBS it provided by LBS so it can handle only LBS instances but we use OpenStack we can't choice ELBs one so second is NZX and console templates it change episode objective in NZX configuration use console event and console templates we choose this one in our OpenStack third is NZX and Moriwi NZX Moriwi allows changing up story objective in NZX process and memory used in Moriwi we are evaluating it now console and console templates can integrate slack notifications like this so if up story changes so it notifies our channel of slack itself we can monitor up story status so we made burglary deployment on tall task like this it for the running it puts instance number of old instances we invoke new servers using image contained red bundle it number is save as old instance like this it loop wait to run until all of the instances booting like this like this wait wait loop loop loop and finally we remove all the instances from our load balancer and terminate there like this so we can look only new instances on our services so summary so we can handle TVCM and TVShow use our scale servers and we can enhance infrastructure every day we can deploy application over 100 servers every day so we can upgrade operating system and Ruby version and Midori version every day so yes we can that's all thank you