 Hello everyone. I'm very delighted to be here today. I think the conference has been so far-reviewed. Thanks to ASCII for arranging this. With that said, let's get started. My name is Habir Rakhman. I'm working at ASCII. The topic for today is cloud means fog and pathage, a story of worship control. I'm sorry, I could not get the time any longer. Anyway, so this is what we learned at ASCII, automating large-scale cloud deployment. So I'll be sharing some details and some details about the movement process we were covering and what we learned in that journey. Before we start, I've been about my company. APIG is an API management company. We provide free tools for developers. We also have enterprise model API management product, which is used by different enterprises across the world. Anyone using API developer tools? Okay. So, nice to do so. Basically, we make APIs better for developers and developers. That's our idea here at ASCII. We also have a recently acquired company called User Grid, and User Grid needs to make building mobile clients for iOS and Android easier. So it's an open source. You can actually find all the details in the APIG website. There are a lot of videos, webinars, database, APIs, and User Grid out there in aphix.com. You're welcome to visit and take a look. In fact, APIs are very relevant to our company. So, if you look at cloud and any providers, like Amazon Web Services, or App Space, or Lightning, the way they provide services is by giving you APIs. So, if you look at a lot of cloud maps we've published, I'm sure you might have heard of these names, like the ride scale. So, what these guys are doing is they're making some of these APIs that they're working on. So, that is the API echo space which we have right now, which the world is going to right now. So, in the API, one by one, you add different storage, then the API is like that. So, how are you using AWS? Amazon Web Services is in production. So, just to give a, just to set the stage right, if you look at AWS, Amazon Web Services, it allows you to create instances. Cloud completely resolves, you see. You can either go to Amazon's Web Console, you can use any of these cloud management rules, or you can use Amazon command line rules, or you can use any of these libraries, like the Java, there are libraries, Ruby, there are libraries. So, you bring up instances, you assign idea addresses, you add storage, all these things. All these things are, we will do it manually when we start. So, you know, this was our thing, we were using a third party. And it was good, but the problem is it would not stay. If you are talking about cloud servers, it's not easy. And we will have to do it manually. Maybe you're back in five or seven weeks. So, it gets boring also at times, right? So, if you look at this, you know, this gives you a small idea about the different things we were, if you know Amazon, there are Amazon machine images, right? Which you use to bring up instances. And there are, you know, packages, you need to install licenses. You create these volumes. These volumes are nothing but network-attached volumes, which is another service by Amazon. Then you open up, you create security groups, you attach and attach the other instance. All right, so all these things we were doing manually. So, my talk is about how we moved from this number of manual tasks to largely, mostly automated, you know, state VR now. All right, so I'm not saying we are completely automated. There are quite a few things which we can still automate. We are working on it, but still we are good. So, this was our take two, right? So, I think we are, you know, we are back, one of our colleague in US. He started, or he started exploring cloud configuration management tools. And we checked Chef, Puppet, and finally we, you know, we went with Puppet. You know, we thought that it's better fit for our team and our culture, right? So, Puppet, I'm sure a lot of you guys are already using Puppet. Can you tell me how many of you are using Puppet in? Okay, so Puppet is nothing but a configuration management tool where you can actually take a base instance, your base computer to a desired state. If you want your base computer to a passive server, you can apply these manifest. They call that manifest. Just rules, you know, which is install this package, you know, have these users assigned to, you know, this instance. Then you apply these manifest. In Puppet terms, they call it manifest. And they apply these manifest to that instance, and you'll be in that desired state, be it a NGTex server, be it a Apache server, you know, whatever you want. So, we actually started using Puppet. It was good. We also used Git for version control. I'm sure you guys are using Git. So, what we used Git for is, you know, to manage the entire configurations and packages. So, we, you know, bring these packages and manifest, Puppet manifest, under Git version controlling. So, Git gives you the history. You know, what happened, when happened, and who did that change, right? So, if at all you want to rework, it gives you that visibility as well. You can actually go back to that previous version at any point in time, right? So, this is where we were using Puppet and Git. And if you look at this, you'll see that, you know, installing packages and generating server configs. For example, we use load balancer. So, we were actually using Puppet to come up with the server configurations. But still, you'll see a lot of these things are not automated. We still do a lot of manage tasks. So, you know, that is where we actually found Puppet Cloud Provisioner. So, we actually also got help from Puppet Labs, the creators of Puppet. And we use Puppet Cloud Provisioner. And Puppet Cloud Provisioner is, in fact, using a Ruby Cloud Services library called Fog. So, using this, this is where, you know, Cloud meets Fog. And we started using Puppet Cloud Provisioner. It is nothing but a wrapper around Fog to bring up instances. I know that, you know, we had a talk around Cloud Formation. At that point in time, Cloud Formation wasn't that mature as a product, right? It just came in. So, we started using Fog and Puppet Cloud Provisioner to bring up instances. And, of course, Ruby is the whole language we were using. So, if you look at this, we actually automated the whole thing, you know. Only a few things were pending here. That's, I think, we are working on. And I think as we go, we may automate those. So, you know, we, now, what we have is, we have, in our case, we are a managed service provider. You know, we manage the whole API management product for the customer in the Cloud, right? So, we have a customer in a YAML file. So, YAML is a simplified version of JSON or XML, right? So, it'll look like this, right? We'll say, this is our configuration for so-and-so customer. And just run this Ruby script. And we have everything ready. Right? So, even the puppet will be installed automatically. All these security group creation, port mapping, everything will be done. So, this is just, I mean, it's just that you're giving all these variables, saying that, okay, this is the key file. This is the image you want to. This is the instance type. Anyone who worked on Amazon know that there are different instance types, like M1 small, you know, C1 medium, M1 large, like that. So, we'll give all this information in this YAML file, and we'll run that script. That's it. We have our customer instance ready. All right, so, the beauty of it is, you can actually reuse this, you know? Once you have this in Git, tomorrow you have another customer, maybe you need to change few things here and there. You can pretty much reuse the same thing across different customers. Sure. Right, so... Yeah, so, actually, it's like we happen to meet Fog, you know? So, nothing against Lib... I don't think LibCloud or Dental Cloud, JClouds would match at that point in time. I was talking about Cloud Formation also. Cloud Formation was very new at that time, right? So, as of now, we are only using with Amazon Cloud. But, yeah, so, I'll come to that. That's one of the features of Fog. We can actually use Fog across multiple vendors, right? So, this is our, you know, YAML file where we can bring up instances this way. So, basically, Git, the version controlling mechanism, to manage both puppet configuration and cloud deployment. The YAML files and the manifest, puppet manifest, both are matched by Git. All right, so, this is a, you know, very high-level view of the whole thing, right? So, we put this YAML files in the Git, right? And the Git has this post-receive hooks, right? Which means that whenever there is a commit, you know, they call it commit, you know? Basically, you are saying that, okay, let this change happen in this repository. Whenever there is a commit, you can actually have a post-receive hooks, which will copy this repository to the different puppet masters we have in different regions, right? Now, then, we'll run that script from whatever region we want to bring up the instance. So, if you know Amazon, there are, I guess, five or six regions as of now. So, we'll have, for us, we have puppet masters across different regions. So, once you push anything to the Git, you know, it will go to the corresponding puppet master. Then we'll run the script. So, like I said, we have fork, and we also have puppet cloud provisioner. Then we have an easy deploy, which is a wrapper script around all this, which creates the whole thing, which keeps everything in sync, right? Then it bootstraps the cloud instance in Amazon. Once it's up and running, it pulls configuration from the respective puppet masters. So, this is a very high-level view, but I think this gives you an idea of what we do. So, like I said, we are using Git hooks to do two things. One is, before committing something to the repository, we need to parse the puppet, you know? I mean, we don't want someone to do a bad manifest committed to a repository, right? So, we do a puppet parser check before the update. We also do, we don't do this as of now, but there is a tool called puppet link. We can export further, which actually does these things a lot more better. So, and afterwards, a post update in Git that's a hook will sync these repositories. For example, the puppet repositories with different puppet masters across the region. Okay. And now, a bit about the puppet, I know that you guys know about puppets, so I'm not going in detail. But what we do, maybe differently, I'm sure a lot of companies are doing this, we use Git branches for puppet environments. So, we don't have a separate dev, separate, you know, test environment. We have a single puppet master. We use that, we use different Git branches to separate it out. So, for example, if you have a, I'll give you an example, you know, you want to add some features to existing manifest. You don't definitely want to do it in your production environment. So, we create a new branch in our Git repository, then we push it to the central repository, then when you bring up instance, we say use that feature X, right? Now, this new instance will actually get all the new configurations manifest, new manifest from the Git repository, then we come up with that version of puppet manifest. Okay. So, like I said, you know, with single master, puppet master, you can have test, dev, production environments. Then manifest and files, data. So, basically, if you look at puppet, you have the basic rules, the manifest. You also have data which says, you know, you want to use this configuration file. So, we separate it out. You know, we don't keep both things. You can actually keep both manifest and these data in the single repository or in, yeah, single repository, we don't do that. We rather separate it out for ease of management. Another thing is, like I said, we have puppet masters per AWS regions. Okay. Now, these things we have been, we have started using. I'm not sure how many of you have heard Hyra. It's a, this is a tool which helps you to separate out the data again. I'll give you an example word. As of now, we are coming up with our, you know, one of our versions or one of our products. There is a new update to that product. All we have to do is to go to this Hyra repository. There will be a FH version, you know, that product version, we just update it, then restart the puppet, right? So, basically, you don't touch anything in the manifest. It's completely separated out. So, this is something you can explore. I think, from next version onwards, they are trying to integrate this with puppet. Sure. Yeah. I mean, if at all you want the change to happen immediately. True. Yeah. If at all you want to change the immediately, yeah. Otherwise, of course it will run every 30 minutes. Or you can, you know, front-app it whatever you want. That's right. So, I said, you know, I mean, we restart it because we want to see the changes validated, then go to the next thing. That's true. So, like I said, Hyra, they are trying to integrate it with the core puppet from next version onwards. Okay. So, now, why fog? You know, we started, we happened to meet fog, like I said, and we started using fog. So, a few of the reasons why I like fog is, one is cross-service compatibility. You know, you can actually use this with, you know, different services like compute, DNS. We actually use, I'm actually, you know, we are coming with a version of this ECDeploy script where we create all these DNS entries as well using fog. So, you have multitude of services like storage, compute, DNS, load balancers, you know, MySQL service, everything, which you can control using fog. You know, there are APIs in fog. Now, another thing is avoid vendor locking. I mean, this is a debatable topic, but as of now fog supports different vendors. ECDeploy, Rackspace supports, Slicehost, Lino, these are different vendors, cloud vendors, fog supports. Now, another thing I like about fog is the power of Ruby. Right? So, we'll come to Ruby later. So, this is how you install. If at all you are interested in fog, you know, fog is a gem. You can say, if you have Ruby in your laptop or in your server, you can install using gem install fog. Then in your Ruby program, you'll have to require this package, require fog. Then there is a .fog file in your home directory. That is where you keep all your, you know, your, you know, AWS account details. So, the fog by automatically fetch your cloud details from that file. Now, one of the things is that, at first, there are not many good documentation out there for fog. But I learned that you can actually look at the test suits, you know, and actually learn a lot about fog. So, it's because of Ruby's, I think there are a lot of languages using this approach now, test driven development, TDD. So, if you go to this folder, you'll find the way they have, you know, you can actually use fog in your environment. So, it's very easy to get started if you look at all these files. Okay, and fog also has a command line. You can say fog test AWS, which means this will take the test AWS account details from the fog file. Then you can play around. You can create instances in the command line itself. And this is another example where you actually, you know, look at the different vendors. See here, we are using similar command to bring up instance in Amazon, also in Rackspace. And here, you say, you know, you're creating a, creating with this AMI, here you are creating with Rackspace on AMIs and, you know, flower IDs. So, basically the idea is that you can create, you can use the same command and create instances across multiple cloud vendors. And, you know, we were discussing about this. There are a lot of open source fog alternatives. Probably if you're interested in, for example, Python, you can use LibCloud. And, you know, I know in Amazon, they have a package called Boto. I'm hoping someone is using that. That's not vendor-independent, but just Amazon, you know, Python library. Then Java, there is JClouds. In Ruby, you have DeltaCloud. That's a new Apache project. I think it looks interesting, something to explore. So, again, you know, why Ruby? You know, I used to program in shell scripting. For some reason I didn't like pole. But the time I started using Ruby, I liked it. And this answers why I liked Ruby. I think you should, I mean, if at all you're not using Ruby, I think you should start using Ruby because it's protective and it's fun to use. So this is the code from Maths Moto, Maths the creator of Ruby. So Ruby is definitely fun to program and fun to work with. It's very protective. All right, and I also suggest you to watch this talk by Ajay Ghor. So this is a nice talk, probably you can watch this online. This also gives you the different, especially for us, the different areas it's being used, Ruby being used. So I know Rami's smiling. It's his ex-colleague. All right, so if you want to learn Ruby, there is this IRB, that's Ruby interactive shell. You can do whatever you can do with the program in IRB's shell prompt. And I know that there is an IRB option at you which is called PRY pry. I didn't explore much, but it looks very interesting. So you can actually install it using, it's a gem installed PRY. All right, so one of the things interesting about Fog and Puppet is the layers of abstraction. It gives you an abstraction layer. It's independent, for example, if you take Fog, it gives you the cloud abstraction. It's independent of vendors. So for example, tomorrow you want to bring up your DR environment in a different vendor. Fog-like libraries makes it easy for you. I'm not saying it's, you can just say use lagspace, but it makes it easy for you. So Fog gives you that cloud abstraction and tools like Puppet gives you voice abstraction. And again, it's not like, you can just switch to say Ubuntu from Send OS, but it makes it easy. All right, so I think we are reaching the end of the talk. So these are a few notes to myself. Hopefully you'll get some points out of it. So one thing I've learned is that when, I mean, if you look at where we were and where we are, I never thought that you'd be able to reach this stage. Because it was like a pile of manner tasks. So we actually started small. Then we actually iterated. So I think it's a very, you know, nowadays in software development, they call it agile. But in this segment, what do we call it? It's we need to start small, then iterate, reiterate, and reiterate. I think once we do that, I think we can do under this. Another thing is there are a lot of fights over these tools, right? They, someone says, Chef is the best and someone says, you know, Puppet is the best, there are a lot of fights. But I mean, for us, we selected, I guess we selected the right tool for us because we thought that it suits our team, our culture. So I encourage you to do the same, to evaluate not because something is right for someone else, to use it because it's right for you guys. All right. Another thing is, sure. See, for example, I'll give you our example, right? We were like just operations guys. We didn't have much experience in Ruby. So if you look at a lot of startups who are Ruby based, Ruby on range based, they use Chef because there's no Ruby and the language Chef cookbooks are written is Ruby, whereas Puppet has their own DSL, which is kind of easy, right? So to learn. So we started with, we weighed that point and we thought that, okay, Puppet is suited for us, right? Sorry? Exactly, exactly. Maybe you can learn. I know that Chef is also widely used. It's good. Probably we can learn, but it takes some time. So we went for Puppet. So another thing is, you know, something I learned that, you know, whenever you design a software and it holds good for softwares as well, we need to make sure that it's operations ready, right? I mean, you don't think of making a software product operations ready after everything is up and running, right? As in, you know, when you go into production. So the day when you design the product, you have to make, this is not something for the system, but, you know, which is something good to keep in mind and we can probably, you know, advise our architects and designers out there, right? So with that said, I think I'm done. These are the few pointers to get started. Yeah, that's it. And just in case interested, we are having. Yeah. Yeah. That's true. Yeah. So this is mainly because of scalability. We have like around 3,800 plus instances, right? So we don't want, you know, the puppet request to go across regions to fetch the manifest and the packages. So mainly because of that. Sorry? Yeah. And then we thought that it will be much faster, the whole thing. I mean, I've seen that sometimes if you have too many packages in your puppet master, even within the region, it's very slow, right? So we don't want to be in that position for, say, Europe region. So we wanted to make it faster. So from the day one, we had different puppet masters for regions. Textbook? No, we did not have that situation yet. So you might remain, you know, problem accessing a puppet master in a particular region, right? We didn't have that. But still, I don't think that would be a problem. I don't think that will create a problem because, you know, if at all you don't have to have a new change to be done, a new package to be installed, you know, it will run fine. Yeah, so one of the things is that we will try to bring up a new puppet master as soon as possible. We have all these things automated. As in we keep these git repositories in S3, right? So we can actually bring up this new instance. We actually puppetize the puppet master also, okay? So we know that how to puppet the initial thing, then we bring up the instance. So once it runs, you have a new puppet master. So yeah, puppetize the puppet master. Yeah, we do that. But initially, you know, a bit of things you'll have to do manually, but other things are puppetized. And so now we monitor the processes. We don't actually monitor the logs as such, but we monitor the process in each individual puppet, servers, and agents. We monitor the individual process. But yeah, I think there are better mechanisms to actually monitor puppet by looking at the one link puppet. There is, you know, a file which will be updated every time the puppet will be, you know, run. We don't do that as of now, but yeah, there are mechanisms. Any other questions? Sorry? Yes. We do all these things. How do you make sure that they're all fine? Yeah, so we run this. Once you have an instance, and then you install. For example, if you have a patch server, you validate that. For example, you access the site. We have a portal, for example. So once you bring up the instance, yeah, we don't have that situation. For example... Okay, okay, okay. We've never been included in that session, but... So probably I'll repeat that question. So what he said was, if you want to, you know, create some instance and you have to do it like 20,000 or 2,000 servers, how do you make sure that it's being installed in all these servers? Right? So probably you can even write a script. I don't think... Yeah, so yeah. So M-collective might be an answer there. Yeah, yeah. Yeah, yeah. So if you give, install this version, install that one link. If you're young, possibly you have that script, a point link, that you can add with your... ensure this... Install. It will put me in for that. It will put me... Just... Yeah. So if you're an environment player, after a thousand servers, there's no need to allow me to have it. There's no need. Why are you over it? Yeah. Yeah. So M-collective, actually, which node has that... which node has that script? Exactly. Yeah, so... Yeah, even... Yeah, that's right. So we are actually using M-C-O. So for example, you can say what is the version installed, package installing all these different servers? Yeah, we are also using M-collective for that. Yeah, in our latest proper deployments. We only see that with servers that are entered installed all the time. Right? So you have 500 servers. Right? 50 of them pay. That's new, yeah. And you want to see how on which servers... Right? So... It has proper install and implement, that's how it works. You can see... The final specific thing, you can actually tell M-C-C-I to check for which... which node has a specific... that script. And actually, what happens? Nothing like that. So you will... M-collective, yeah. So you guys are using M-collective? Yeah. Yeah, okay, you are using, okay. Yeah, we are also starting using it. So it's a nice tool. Like I said, we are also using... Say for example, you can say what is this version of this product? So it will show you all these servers which are in that group. It will show you the version of the product. Okay. Why is that? I know that... I mean, any of these... I can change... I can add some... Yeah. No. Yeah, that is what you... Yeah, that is... you are using it with everything, you are using... you are using it with all your resources... I can say a lot of things. Not just this design, but I can add some data. So in that, So any other questions, I think we can discuss. I think we are having a talk on M-collective, so probably, yeah. So anyone using Rendec, there is an orchestration tool called Rendec, I think based on SSH. So yeah, with that said, if no more questions, any other questions? Okay, thank you guys.