 Hello everyone, welcome to the Linux Foundations open source summit Europe 2020 and in my session, simplifying first boot experience for your cloud VMs with cloud init. I hope you're having a great time in the open source summit and you're learning new things. My name is Ashish and I work with Microsoft as a partner technology strategist to help my ISP partners embrace the cloud and solve the technical challenges that they are facing. That's my email ID and the Twitter handle on the slide if you want to reach out. We live in an age of everything as a service. And that applies not only to computing but to the other aspects of our lives as well. But of course, we are only going to talk about software and computing today. The two things, the two innovations that have made it possible are virtualization and the cloud platforms. With virtualization, you could harness the complete potential of the hardware that you already had in your data centers that allow the organizations of all sizes to get much better returns on their investments. The cloud platforms today have made it possible for the developers sitting everywhere on the planet to get their hands on the latest innovation in the computing and software. We have services like App Service on Azure, Elastic Beanstalk on AWS, App Engine and Firebase on GCP that allow developers to quickly and rapidly deploy performance applications on any scale that they desire. This was really unheard of just a few years ago. But one thing that everyone still tries to create the moment they get their access on a cloud service is a virtual machine. And there are many valid reasons for that for the virtual machines to stay here. They are one versatile piece of technology that are available on almost all cloud providers and other platforms. You can install your front end, your back end, your database and any other software that you need for your solution to work and you can replicate that stuff among cloud providers because the virtual machines, the virtual machine images are almost identical between the cloud providers. And Ubuntu VM on Azure is going to be same as another Ubuntu VM running on AWS. And we are going to see that in the demos that we are, that I'm going to show you after a few slides. That makes the virtual machines really simple and cost effective solution and a preferred choice for the development and testing stuff. But it is not without problems. The images that you get from your cloud providers are immutable, which means you own the activity to customize the VM to provision them properly and then make them ready with all the necessary software application, database, configuration and other stuff that you need to ensure that your solution works in a stable fashion. And then you can probably create a clone or a master image of the VM, which allows you to tackle the problems of replication and scaling. And that's where Cloudinate comes into the picture to solve these problems for you. Cloudinate is a cloud instance initialization framework. In other words, Cloudinate is the framework that you can use to gain control over how your machine gets configured and provisioned at the boot time. When you create a virtual machine, you need to install software, you need to tweak settings and do other stuff to make it behave the way you want to. You often probably do this manually or you probably have scripts that you execute. With Cloudinate, you can line those up as the actions that get executed when the machine is getting provisioned. Cloudinate is also cross-platform and multi-distribution, which means that the configuration that you put together for the Cloudinate to customize your virtual machine on one platform is going to work almost exactly the same with little or no modification. It also means that the configuration, the multi-distribution tenet of Cloudinate means that you can also take the same configuration from one distribution and apply it to another. It supports all the major public and private Clouds and also gives you the fine-grained control over how the VM gets provisioned and initialized at the first boot. And this magic is made possible by instance data that is provided to Cloudinate. It can be classified into three categories, the Cloud meta data, the vendor data, and the user data. The Cloud meta data and vendor data are provided by the Cloud provider and the Linux distribution vendor. It includes information such as what kind of network this machine is going to be connected to, what kind of storage is going to be mounted on the virtual machine, and things like SSH configuration so that you can access the virtual machine after it is ready. The user data is where you and I get control over the customization that we want to carry out when the machine is being provisioned. As mentioned earlier, Cloudinate provides you with really fine-grained control over how and when you can customize your virtual machine. The generated stage is an early stage during the boot-up process. The local stage is when your root file system becomes read-write so that you can start writing data and configuration to the files. The network stage becomes available when the network is up. At this point of time, you can start calling the services outside and get your software and configuration in source code and other stuff downloaded to the virtual machine. And then you can configure them in the configure stage. And at the end, you have the final stage, which is similar to RC.local. The scripts in the stage gets executed after the machine is fully booted up. You also get control over how and when you want to execute the scripts. You can execute the scripts only once when the machine gets provisioned. You can also target a different set of scripts only during the first boot after the provisioning. Then there are scripts that you can also run at every boot up. So as you can see, the flexibility that you get with Cloudinate is really versatile. So that was the context about Cloudinate and what it can do for you. Let's jump into the demos to take a look at Cloudinate in action. In the official documentation, there are extensive examples that you can get your hands on. But for the purpose of this talk, we are going to take a look at some simpler examples. So there's a Cloud config example where we are going to perform the upgrade of the package repository and install Nginx as a package on the target virtual machine. Then we've got a way where you can use a different directory to run a script from a remote location. And then we have this example where we are going to run a script on the VM. So in this case, you can see that we have got a script that creates an index.estimal file with plain simple hello world in it. And then it serves that HTML file using simple HTTP server using Python. So I'm just going to copy this into my clipboard. And I'm going to start with AWS. So I'm going to launch a virtual machine, an EC2 instance on AWS. So we have the Amazon AMI virtual machine. I'm going to select the T2 micro. And then I'm going to click on the configure instance details. And after all these parameters on the screen, you have the section that is called user data. And that's where you can copy-paste these simple scripts here. As you can see, there are three different options available on AWS. You can paste your script as text file, you can attach a file, or you can encode it using base64. We are going to go ahead with the text options. Let's go through the steps and launch the VM. I'm going to use an existing key pair that I already have downloaded on machine. And I'm going to click on launch instance. And as you can see, the initialization is done. Let's click on the instance and it's going to try to log in. I think that goes a little bit early because it still shows pending here. So let's give it a moment before it becomes running. And let's go and log in. So let me make a copy of the address. Let me go back to my command prompt. And I'm going to point my SSH client to the key that I've downloaded on machine. And I'm going to log in to this machine. In fact, we don't really need to do this. So this is done, right? Now, if you remember, let's go back to the script that we were using. You remember, basically, it's prepping up a simple HTTP server using Python and that only serves Hello World, right? But I don't have my port 80 available to the world. I'm not open it up in the security group. So what I'm going to do is I'm going to run this command, curl on this local machine and connect to HTTP local host. And you can see that the output is Hello World, right? So which means the script that we saw here, it worked. And the Python based simple HTTP server was running and serving the content that we actually put there. And if we go to slashware slash log, you can actually see the file where the execution log can be seen. So what I'm going to do is I'm just going to show you the output of the execution. So this is all that happened when the machine got provisioned and the script that we added to the user data part of the VM provisioning executed. There's not much here, but you can actually basically see that there was a national request and that got served from the local host, right? So that was the first demo. What I'm going to do is I'm going to exit from here and let's jump into the second demo that we have. For that, let me show you the cloud init YAML file. So this YAML file starts with a cloud config directive. And then there's a directive called package of bread true, which is equivalent on Ubuntu VMs. It's equivalent to APT update, which means it is going to just update the package registry so that you can get the information about the latest available packages. And then after that, it is going to install this package called Nginx. Nginx is a web server. I think that's something that everybody practically knows. So what we are going to do this, what we are going to do now is we are going to copy this. And this time, we are going to launch a VM on Azure. So I'm going to create an Ubuntu 18 VM here. My resource group is already created, so I'm not going to change that. I'm going to call it Hushes of Web 1. I'm going to put it somewhere closer to me. And I'm going to use a standard B2S, which is the burstable two core VM. I'm going to use an existing key that I have already stored in Azure, and which is already downloaded to my laptop. I'm going to practically keep everything as it is. But when we come to the advanced stage, advanced stage of configuring, advanced step of the configuration, what I'm going to do is I'm just going to paste that copy texture. But when we come to this advanced step of the VM creation experience in Azure portal, what I'm going to do is in the custom data section, I am going to paste the text that I copied from the code example that we were looking at. And then I'm just going to go and create this VM. The VM has been the deployment is actually in progress right now. And while that is happening, not to waste time, what we will do is we will actually go and create an Ubuntu VM on AWS as well. So I'm going to create a launch instance. But instead of using the AMI image, I'm going to search for Ubuntu. Ubuntu 18, LTS branch. The rest of the steps are going to be exactly the same that we did the last time. T2 micro, which is the equivalent of the size that we chose on Azure. And I'm just going to put this here. You see, it's the same text that I've used on Azure. I'm just going to use that exactly the same on AWS as well. And I'm just going to launch it. Same question, just going to use the key that I already had saved on my VM. And while that is happening, let's go back and take a look at the Azure deployment. So as you can see, the deployment is done. That's my IP address. And I'm going to do this from the command line again. So SSH, my key is actually stored in the same directory. So I'm just going to do this. And I've logged in, and if I run engine next here, it's installed, right? Let's go to slashware, slashlog, and take a look at the cloud init output log. So that is the basic stuff, the initialization stuff. And then at the end, if you see, this is the output of the APT update, APT gate update. And then after that, whatever you see on your screen is basically engine is getting installed. And if you remember the slide where we were talking about the different stages of the boot that you can target, anything that you put in the user data, or the config, or the custom data on Azure portal gets executed at the final stage. The thing about these stages is that everything that you put in the cloud init and target a specific state that basically blocks the boot process. So you have to be absolutely careful about what you are doing and also be careful about whether or not it is going to prevent your machines from booting up. So be cognizant about the fact, and everything should be fine. So that's one thing. Let's go back and take a look at our AWS instance. So yeah, that seems to be running now. Click on this. Is it the same VM that we had? Or yeah, that was not the same thing. So this is our open to VM. What we're going to do is the same thing that we did on Azure. So I'm going to log it from here. I'm going to use my AWS key and the username for the open to VM on AWS is open to. So that's what I'm going to use here. I think I am confused between my VMs. Let's take a look at this one. Let's copy this and try this. So that worked. And GINX command is available. And what you can also do is you can actually take a look at the output logoff cloud in it. And you would practically see the same output that you saw on the Azure VM because, like I said, the open to VM on Azure is going to be practically identical to open to VM running on any other platform. So we're done. So the output and similarly targets the final phase. So that was about how you could use the cloud init YAML format to execute stuff, right? There's another thing that we can do is that you can also execute a remote script through the cloud init. So while this gives you a lot of flexibility, what you could do with this, you can actually maintain a central repository of the scripts to target different roles that a VM is going to play in your deployments and not really change anything with respect to the VM in your automation scripts and just customize the script in the center location so that your deployment gets customized automatically. The directive of interest here is hash and payload. And then I provide a URL to a script that I've stored on GitHub. So that's the raw path. So what is going to happen is that the moment cloud init reaches the final stage, it is going to download this script and it is going to execute that on the VM. So let's try that with Azure first and then we will do the same thing on AWS. So I'm going to create a VM on Azure, same resource group, somewhere closer to home. And I'm going to use the same key that I've had that I used for my last VM. And going to jump to the advanced stage and put this script here. So before we do that, let me just show you that script. So let me just copy this URL and open it for you. So as you can see, what I was doing with the cloud init YAML file in our second demo, I'm doing that, but I'm doing that through the script here. So I'm installing NPM, Node.js, and Nginx apart from updating the app registry. So let me start the deployment here and then we are going to do that exact same thing on the AWS. All right, so that deployment is done. Let's go back and create a new instance. I'm going to use Ubuntu again. Going to use the same size that is allowed for my free tier on AWS. And I'm going to put the exact same thing from my example files to here. And just going to finish the process and launch my instance. And in the meantime, our VM on Azure was created. Let's copy the IP address and take a look at that. And we are in Nginx, it's not found. So what we can do is let's take a look at our cloud init file because I feel that since we are targeting the final stage, I think the installation is not yet done. And yes, I'm telling the cloud init output log and you can see that this stuff is actually still going on. If you remember, unlike the last example that we had, instead of installing only Nginx, what I'm installing is also NPM and Node packages. So that added a little bit of time. And that's why you say this delay, right? So it's happening in the background. And until it's done, we will not have the applications that we wanted to install ready on this virtual machine. And we are almost towards the end. Well, that's happening. Let's go back and take a look at our AWS instance. It says successful, let's refresh. It's initializing at the moment. Let's click on it. Let's see if we can log in. All right, this seems to be done. This was the Azure VM. And now if I run that same command that I ran first, we have Nginx running. And we can verify it's running by running curl against the local host. And yeah, you can see we have welcome to Nginx available here. So let's log in from this VM. Let's try to log in the VM that we created on AWS. We are here. Let's see if where we are. Exactly. So Nginx is actually ready on AWS, curl, as you can be called in slash, slash local host. And we have the Nginx web server running on this VM as well. And if you take a look at the cloud in it, output log in where log, you can see the information to that effect as well, right? So yeah. So those were the three things I wanted. So those were the three things that I wanted to share with you today. Let me log out and let's go back to our slides. So what we saw today in the session is that how cloud init really makes it easy for you to customize your VMs irrespective of which cloud provider you choose and with practically the same configuration. The one thing that would have been different in today's demo is when I wanted to install the Nginx package on my AMI images because the Nginx package for the AMI images on AWS is called Nginx 1. So I would have had to change the package name to Nginx 1 for the cloud init code snippets that you saw to work on AMI the way it actually works on the Ubuntu VMs. These are a couple of links that I believe is going to be extremely helpful and specifically the examples section. So what you saw today on my screen on this session in the demos were very simple demos but if you want to explore the broader capabilities of cloud init, you can go to the documentation and take a look at the examples that are there and with that you could actually see that the kind of customizations you can carry out for your virtual machines running on any of the public clouds and the private clouds is very extensive and that gives you really more power to ensure that your VMs, your primary site, your dear site are practically identical irrespective of how you are, what kind of images you're choosing and what cloud provider you are choosing. And with that we are coming to an end of the session. I hope it was useful and you learned something today. Feel free to reach out to me on my email and the Twitter handle and I should be able to help you with your questions and any doubts that you have. Thank you very much. Again, welcome to the Linux Foundation's open source summit Europe 2020 virtual edition and I hope you're having a great time and I wish you great learning. Thank you.