 Okay, we're gonna get started. This is the cloud containers and DevOps track. This is going to be our last session for the day. Up next, we have Roy and Akshat. They're going to be talking about Procter. So without further ado, I'll just hand the mic over to them. Hello and welcome. So a quick question. How many of you have shared secrets with your friend using a communication platform like Slack, WhatsApp, email? I'll raise my hand because I do that. Okay, congrats. Your secrets are shared with your friend, Slack, and obviously, shh, okay. Moving ahead, it's 2015. A small ride-hailing startup out of Indonesia, Gojek, is operating with 15 engineers and just 20,000 orders per month. This is the actual Excel sheet we used to share our secrets and credentials across organization. How many of you have seen scripts like these? GitLab, Elasticsearch, these are early attempts at automating repetitive tasks in a small organization. Going ahead, since Gojek was only running on 50 machines back in 2015, we were able to manage our infrastructure and run laundry tasks using those scripts. For example, you want to take backup of a database or you want to create a database. Moving ahead. So now when you have access to all the secrets and you have these scripts, you can use them to run your tasks. And the biggest headache with doing that is that you need to install dependencies. And it's a big headache. So every time this thing changes script, you need to update the dependencies. Fast forwarding to 2015, 2018. We realized that a driver who can deliver customers can also deliver food from point A to point B, courier from point A to point B, e-commerce drivers, e-commerce orders, and medicine and grocery. The number of products in Gojek grew from one to 20 in the span of three years. As our products grew, the business continued to grow. From just doing 20,000 orders per month, we are now doing more than 100 million orders per month. The services continued to grow. From two monoliths back in 2015, we now have more than 300 microservices. Team continues to grow. From 15 engineers back then, now we have more than 200 engineers. And machines continue to grow. From just 50 machines, to now we manage more than 20,000 machines across GCloud and AWS. So let's take a moment and observe the ratio growth. First, engineer to machine ratio was one is to three. Now it is one is to 100. Each engineer was responsible for 1,000 business orders. Now every engineer, half a million business orders. Five engineers managing one service. One engineer managing two services. Now with this ratio, let's see what happens. As Will Ferrell puts it in Anchorman, trusting 15 engineers is easy. Trusting 200 is not. You cannot have your secrets shared across the organization like we used to do before. You might be familiar with the concept of test-driven development. So at Gojek, we practice test-driven development. It is a philosophy where you first run right tests for the feature you want to implement followed by the feature. Now one step in test-driven development is clearing the footprint of your tests. All the data that was accumulated. You clear the database after the tests have run. And guess what? Somebody at Gojek has run tests against a production database that is deleting entire data in the production environment. Now when you have secrets shared across your organization, accidents like this are bound to happen. So we need a better mechanism for secrets management than trust. Going ahead. All the secrets and scripts we talked about were using powerful tools. So everyone in the organization had access to tools like GCloud, AWS. What happened? Somebody accidentally deleted VMs in production environment. So when you have your powerful tools shared across organization, such accidents are bound to happen. And as Spider-Man puts it, with great power comes great responsibility. So the one thing that can be expected of humans to do consistently is make mistakes. And Gojek has been the market leader in the transport, logistics, and payment platform verticals in Indonesia for the past three years. The stakes being too high, our stakeholders told us what Jack Nicholson told Tom Cruise and a few good men. And, okay, moving ahead. We are all responsible employees of our organizations and we need a better mechanism for managing secrets and powerful tools. You want to see how we do that? Okay, I see a few eager nods. Introducing Proctor, a developer-friendly automation orchestrator. The idea behind Proctor is that it helps developers contribute to automation and use it. Now, go back. Okay, now you have access to powerful tools taken away. So if you need to provision infrastructure for yourselves, like creating a database, what do you need to do? There are two options. You either wait, the option one is, you wait for someone with access to the powerful tools, come and provision Insula for you. And what do you do when someone asks you to wait for provisioning Insula? You tell them what Morgan Freeman told Christian Bale in The Dark Knight. Which brings us to option two, which is you can use Prox. So what are Prox? Prox are automation added to Proctor. The example that I can give you is creating a load balancer, creating S3 bucket, creating a database. This is how you execute Prox using Proctor. Proctor is the command line that we have, tool to interact with Proctor. There are several commands with it. One of them is execution of Prox. So what we want is a database to store Foreshia data with 500 GB size. What we'll do is, we'll tell Proctor to execute this Prox, which is creating a database and give it arguments. The name of the database, let's keep it Foreshia 2018. In the size of database, let's keep it 500 GB. And Proctor will provide you with the database. Let's see what happens behind the scenes. The Proctor CLI talks to ProctorD, a web service who is responsible for execution of all the Prox. And ProctorD pulls the Prox from Prox registry and runs them in a Proc engine. Going ahead to monitor the progress of your Prox, you can stream the logs from Prox engine to the CLI. And that's what ProctorD takes care of. Going ahead to execute a Proc, all you need is an API call. And you can build that API on top of Slack, UI, or even a simple curl request will suffice. Proctor scales infinitely. The engine Proctor uses to run your Prox is Kubernetes. And that helps Proctor run multiple Prox parallelly and scale to your requirements. The benefit of this, the edge this gives Proctor is against competing clients like competing products in market like Rundec. So where you actually have to pay them in order to scale your machine, in order to scale your product. And it's 2018, I believe scaling should come out of the box for every product. The Proc engine is pluggable. You can have multiple Proc engine to run your Prox if you want. For now, Proctor only supports Kubernetes, but if you wish, you can have any other Proc engine. There is a pull request to us. We are open to contributions on GitHub and we'll be happy to help you. Another benefit of using Kubernetes as Proc engine is running Procs in isolation. One Proc will not be affected by the side effect of another Procs. Unlike running all the automation in a VM, which again our competing product like Rundec does, this gives a edge to Proctor. And this is the most important feature of Proctor that you have controlled access to powerful tools. So you have access to GCloud, AWS previously, but now you don't. Whereas Procs behind the scenes use GCloud, AWS, Terraform or any other tool to create database for you, to create a load balancer for you. And you won't have direct access to the tools, but you'll only have access to indirect access of features to the tool. And we just established that to air is human, but to reduce the damage of error is divine. So how many people here are at a little high level? By that I mean you are accountable for not only your mistakes, but for the mistakes of subordinates under you. Okay, so in that case you can consider giving your teams Proctor. They have the tools indirectly and they will not be affected by it. So now we've just explained how you can use Procs. It is very, very, very easy. Now let's see about adding Procs to Proctor. Adding Procs requires a DevOps mindset. Remember, it's a mindset and not a skill set. And now we have my friend Roy, who will be talking on how you can add Procs to Proctor and demonstrating that. Hi, thank you, Akshay. Yeah, so now we just saw how to run a Proc. Now let's discuss about how to add a Proc. So in many organizations, there'll be a central infrastructure team and adding automation is a responsibility of only that particular central team. But using Proctor, you can decentralize that responsibility and you can empower your developers to add infrastructure automation. So let's say you want to create a DNS record in AWS route 53 and you don't have access to AWS route 53. So what do you do? So let's take a BAS script, for example. So it is easy. And what you do is we use AWS CLI. And for the AWS to create a DNS record set, you need to create the BAS data. So in the BAS data, you pass like record name and record type and record value. And you invoke AWS route 53, change the resource set, come on and you will pass hosted zone ID and the data you just now created. So managing secrets. So you cannot put secrets inside the script. Let's see how Proctor handles that. So what we need to do is we have to pull out all the secrets and arguments and put it inside a metadata file. So now we have a metadata file here. So in that, you'll mention the access ID. Basically you just define it. You don't put the actual values. So you just define the AWS access ID and AWS secret and then you pass in all the arguments it's gonna take like record name, record value and record type. And let's see what's happening behind the scenes. So Proctor D is a web service. So it will store the secrets for you in a secret store and whenever the procs is running, it will pass the secrets to the proc. And yeah, go ahead. And now the problem is in the script we have seen, right? We are not installing any AWS CLI but how does AWS CLI will work? So we need to create a Docker file and in that, if you see, you're seeing like install, we are installing like all the dependencies like curl bash and JQ, Python and AWS CLI. And after that, we'll create a directory and we'll copy the script, the batch script. Just now we created and we had an entry point. So entry point is basically to whenever you start the container, so it will trigger that script. Good, yeah. So to summarize, so now we added a Docker file for adding dependencies and the actual logic, the batch script and the metadata where we have our arguments and the secrets. So you created all the three files and you commit and push. So that will trigger CLI. So that's CLI will do a Docker build and it will upload the Docker image to the proc registry along with all the dependencies. And it will also tell ProctorD about the new job and it will update the metadata. So that's how ProctorD discuss about new jobs, new procs. And this packaging is a pluggable. Basically, Proctor now supports Docker. So if you want anything else, it's a pluggable architecture so you can create a pull request and we'll be happy to help you. And this is the last step. So we haven't passed secret till now so now we need to tell Proctor about the secrets. So we'll make a call call and we'll pass the secrets. So in this case, AWS CLI needs to have two secrets like the key ID and the access key. So once you update the secrets, so you can see here the create DNS record proc actually. And along with the description. So this is shared across your organization. So anyone can run this job actually. And to run a job, you can use this described command. So it will give you a detailed explanation about this job like what all arguments it takes and what all secret it takes. Thank you. So we are open source Proctor. So if you have similar issues in your organization, feel free to use Proctor. And we have time for the demo. Yep, good. So this is the adventure sports that I've done in Singapore doing a live demo. I've been told multiple times that live demos always fail so don't do that. But yeah, let's do a live demo. So we just demonstrated how Proctor, you can add automation to Proctor and you can use automation. I'll demo the using automation part. So I have already gotten the Proctor CLI and I hope it's visible to all at the end, right? Yes. So this is the Proctor CLI, a command line interface to interact with Proctor D and the available commands with HR just one proc. So let's take a help for procs. So you can do three things with procs. You can either list all the procs in your organization, you can describe a proc and you can execute a proc. Let's list all the procs in our organization. So here are a few, but for the demo, we'll be talking about this one, DNS creation. Let's create the DNS that we just discussed in adding the proc. So let's say you want to get help for that. I don't know how to run that proc. What I'll do is Proctor proc describe and the name of the proc. So here we go. What it does is it creates any DNS record for you and it takes record name, record value, record type and TTL as arguments. For now, let's forget the TTL part. So I want to create a domain name for FossAsia. It does not exist as of now. What will I do? I'll run this DNS command. So I'll execute this proc and give it the name. The record name will be FossAsia.gojek.com. Sorry, I can't type with two hands. The record value will be the IP address. Let's keep it 10.1.2.3 and the record type will be A. Let's forget about the TTL for now and let's execute this proc. So now procs execution is successful and the logs are being streamed from AWS. So this is the record change which has been submitted. FossAsia is now pointing to this IP address and ta-da, we have a DNS created. Let's do an NS lookup right now. Okay, it will take a little time. Ta-da. Okay, so that's how you use procs. And let's head over to the repository in case you want to use Proctor at your organization. Head over to our repository on GitHub, which is under the Gojek Tech organization and in the name of Proctor, you will find three things here. The first is the command line, which is the code itself. And to set up the command line, you have instructions in the readme and tests, running tests for them. Then you have the code for Proctor D and everything is present in the readme. Same stuff. And you have a bunch of example procs, how you can write them. And you can find the future of Proctor mentioned as feature requests in the issues. And if you have any issues using Proctor, please raise one over here. You can reach out to us over here or you can reach out to us here. Thank you very much for your time. Any questions? And we have a couple of stickers also. Feel free to take it. Yeah, great talk. Just had one question. When you say the registry, the Proctor registry, what is that actually in the backend? Are you using something like Artifactory or something to store artifacts? Like we mentioned, we are using Docker to package our image blocks inside images. So we are currently hosting it on GCR, Google Container Registry. But if you need, you can host it anywhere you want. The secret store is Redis as of now. But like we said, everything is pluggable. If you need to use Vault or something else, you can plug that secret store in, encrypt secrets all there and store. No, so the job that we have, the proc that we have written, if you submit it with a different, this thing, IP address, 10.2.2.3, then it won't do that. Let's see. Not as of now, but we are adding authentication on Proctor. We are looking for some open source tools where we can plug in the organizations team structure, and then you can have authentication as well as authorization for running Procs. But as of now, everyone across the organization can use your Procs. There is auditing, by the way. Every Proc that is run, it's audited with the arguments it was run. So you can make sure, basically no monkey business is going on. Yes. So the auditing happens in a Postgres, and you can actually query stuff from there. So like we just discussed, you have the features of a tool, you don't have the tool itself. So if there is a Proc for deleting databases, without any authorization, then of course someone can delete databases. But for now, we don't have any Procs to delete databases. We only have Procs to create databases. But someone with access to a tool like GCloud can delete your database in GCloud, of course. Well, quite hard. That is the Provision using communities, right? Hello. So the machines that run the Procs, they are spin-up using communities, correct? It seems to me that they need quite powerful access to run all these things. So how do you actually control the access of these machines that run the Procs themselves? Like, you know, if you need to run everything, do you need to ground them like all powerful access? Currently, we are using the Google Kubernetes engine to host our Kubernetes cluster for running all the Procs. But we've even run it on our local machine, that is Minicube. Of course it won't scale, but you don't need very powerful machines to run all the jobs. So it's a lightweight image and scripts. Access, yes. How? I don't know. Access, okay. So I'm sorry, but we're at time and the conference closing is coming up. So if you guys could just take this. Yeah. So with that, this track is closed. For anyone who's interested, the conference closing will be at five in the main hall, which I think is the same lecture theater we had the opening then. So thanks everyone.