 Hi everyone and welcome to the image builder deep dive. I'm Moshe Imaman and I'm joined here with Tushar. So the image builder deep dive project is a sponsored project by the SIG plus the life cycle group. And if we go into the history of the project, there was a really a lack of consensus on approaches. So there are quite a lot of approaches to building images and we are talking about machine images. So these are the images that provide Kubernetes hosts not contain images. There's quite a bit of confusion there. And there was a document that was put out at the beginning of the project, which was over a year ago. And there was a lot of comment on that document and some people prefer building in the cloud. Some people prefer building locally. Some people prefer using bits directly and there really was a lack of consensus on this. And it was kind of agreed upon that the best way forward was to at least try and consolidate efforts around building images. So that there is at least a centralized effort and not effort split over many different places and repositories. And then once effort was centralized in one place, we can begin rationalizing how the different approaches work together and how to make sure everybody's needs and opinions are kind of catered for with a single tool for the entire Kubernetes community. So this is a tool or a project that was created for the community, but it has a lot of focus on the cluster API project, most likely because that's where a lot of the effort in the SIG cluster lifecycle group is currently. So when we look at the goals, so we wanted to provide this consistent tooling and approach for all different type of ways of building images. And we wanted to make it easy for downstream consumers. Once the group or a distribution created an image for a company or an individual to go and customize that image, their little bits should be as easy as possible. And one of the things that really isn't catered for currently is testing conformance of images. There are a lot of issues that can come up with incompatible Docker versions, container versions. So we really do want to have some type of conformance test for an image to say this image can run Kubernetes successfully. And while not a primary goal, so releasing images across different operating systems and cloud providers was kind of a stretch call at the moment this is handled separately by individual sub projects. So each of the infrastructure providers for cluster API is releasing their own images. And eventually we want to get to a point where we can have some some centralized mechanism for releasing and promoting images. So looking at what we have today. So there are three tools that were kind of consolidated into the repository the first one being an Ansible packer configuration. And this is where where historically a lot of the work by Heptio was done on their projects and and where today Microsoft and VMware and many of the other infrastructure providers for cluster API or spending time, building conformant images for cluster API so they are providers for VMware, Microsoft, Google Cloud, Digital Ocean, OpenStack or CUMU and for a variety of operating systems. Windows is in the work. So it's flat car. The second tool in the repository would be a cube deploy and this comes from the chaos project. And at the moment there is very limited support on the tool and more than likely it will be the first one to merge into one of the other tools. And then the final tool is a Golang based CLI that that is really designed to take a lot of the configuration effort out of Ansible and packer and provide a better user experience. And this is where kind of the future of the project is but it's not very well adopted at the moment and not very well documented at the moment. So that is what we have today and I will hand over to to Shar who will give you a more deep dive into Ansible and packer configuration. Thanks, Mosh. Let's start with our packer and Ansible system. For those who don't know packer is a free and open source tool for creating golden or base images for multiple platforms from a single configuration and packer builds and provisions and also does some post processing to output the final cluster API conformant images. The next major tool that we use is Ansible. Ansible helps us to configure Kubernetes images and binaries, relevant Debian or RPM packages on the on the system, as well as it also helps in hardening the image like cleaning up the logs and removing the users and privileges. This will all be demoed later. So basically this screen shows the high level diagram of what the packer and Ansible make file base system looks like the pipeline here describes the process of image building the configs feed into make system which invokes packer binary with requisite packer variables and execution files. And as you can see here packer is a three stage process where it first builds an OS image from an ISO or a base AMI or an Azure offer. The first base VM is ready packer passes on the execution to provisioning stage. The two major provisioners we don't know Ansible and goss Ansible configures the image installs various components and goss is a server spec style tool which tests the image, whether all the packages and the versions are are as expected installed as expected. And the last step of the stage is post processing where you can do a lot of things like generating a manifest, which tells like what all things went into this image. You can also like download the OVF and customize the OVF in case of OVF builder. So yeah, and at the end of the whole process finally we get an image which is a CAPY conformant. So a bit more about packer conflicts that feed into the system. So these packer conflicts are used to define Kubernetes versions, Kubernetes sources, container diversion, kickstart and pc files. Some of these arguments are passed down by packer to the Ansible provision for installing correct components and stuff. You can also modify these conflicts to build custom images. Coming on to the Ansible, we have basically structured all the Ansible tasks in four roles as the slide displays actually five roles and one and like X number of custom roles. So you can see like there are four predefined roles which set up the base components, installs container to Kubernetes and provide a specific stuff like your CLI, AWS, SSM agents and VMware cloud and it gets sources. And then there are like custom roles so where you can run and like install some specific things to your own enterprise or your own image like whatever some agents specific agents that you want on your image. And the last role is also a system defined role called sysprep where we basically clean up the image building logs and then do some security hardening like remove SSH users and remove pseudo privilege and stuff. So as described before custom Ansible roles actually enable you to completely undo the past four roles that we saw and redo them in your own way. And then so it's a pretty powerful feature which will allow you to like customize the image as you want. And then we run the sysprep role in the end which will clean up the things for you. So yeah, so this is about customizing the image and we'll see a demo later how we customize the image. So in the demo, we'll see that we'll add a custom Ansible role. And then I'll build a custom photon OVA and customize it to increase the increase the to change some kernel parameters related to GC threshold. And then I'll also install a package called Havage which assists in increasing entropy for random devices on headless servers. At the end we'll see how our Kubernetes node OVA got built and provision, and we'll also go through the powered on image to see like the packages got installed. So let's start our demo. So here we can see that first time doing the make help command which will list all the targets related to photon three, which I'm going to use for my demo. We can see that there are various builders here and we are going to use a local builder local hypervisor VMware fusion based hypervisor and a photon three OS. So to start with let's look at the packer config. So here we can see that this is a packer config for Kubernetes.json and you can see that various variables are defined for the type of Kubernetes version we want to install like we can see one 1711 here with all of these things can be over in. We can also see HTTP sources, Debian, Debian and RPM repository and sources. Another configuration we can look at is container D. If you look here like we are also like customizing the container D version we are installing, you can also customize the pause image here. So all of these settings again feed into packer system, which passes to in civil for installing the right container D. Moving on to the packer main packer json or configuration file we'll see that on the top we have defined all the variables that are being consumed. And then we'll have various type of builders it can be local builder v sphere based builder ESX based builder, and then we'll see the builder we are going to use is VMware ISO, which helps us produce local OVF. And then we have various configurations that we do here boot command and SSH username CPU memory. You can look into more detail on our upstream repo. And next looking at the provisioner we can see that there is an Ansible provision here we can also add some extra arguments for increasing the verbosity and debugging purposes. And then we can see that we pass the playbook file and the and the Ansible SSH args to be done. So the provisioner that we use is gauze. It's basically as I mentioned before is for server validation and a server spec side tools tool. So we are passing various configurations to it. And then the last section is our post processor, where we have a post processor of type manifest which generates a just like a JSON kind of manifest which details what all things got installed as you can see from line. All this information is captured in the manifest along with the OVF. Next we are going to look at the custom Ansible role that I have created to customize the image apart from the upstream image building that will take place on it. So I'm installing this tool called heavaged, as you can see here to increase the entropy, and then I'm setting some extra kernel parents. Next we'll go and look at the gauze package and you can see that I have edited test also for the package that I'm installing. So at the end of my packer builder run and then after in civilization when gauze runs it will be able to check that this package is installed or not. Next, as I mentioned before, like we can edit packer configurations in place or we can create some packer very variable files at the top level and then just use them to override override from there instead of editing the file. So I am installing the Kubernetes version one 19 one, and then I have modified things. And then in the custom ISO as you can see that I have added a custom role, which is the Ansible customize. Now with this custom role names could be basically an array of roles, but I'm just using one role here will clear off the output to make sure there is nothing there before. And then, as you can see, I'm putting the custom variable files for packer, and I'm using the make a target for building the local and photon three. Let's kick off this build. As you can see on the top of the screen that there are few hack ensure and see will ensure packer got run, we just make sure the right tooling is present on the machine. And then right now it's building the VM it's waiting for it to boot. And it's, it's, then it will wait for getting an IP. And once the system is on it will, it should start approaching it as you can see here. It's installing all the Ansible roles. And here it is right now, and then this is sis prep role cleaning of the SSH keys. And here I can see that we are running goss, and you can see that 48 tests passed and zero failed. So basically goss was able to verify that all the components that were required are there and running. For instance, you can see that service container D is running and matches expectation true. We can go and see the goss pack done by goss provisioner on our machine. And this is the exact spec spec or the test spec run by goss in order to verify the image so we can see that all the various packages are being verified here. And we can also see that various services are being verified here and we can also see that we are verifying the, the version of a container to install and what all images are installed on our container runtime system. And then in the end we can also could see some kernel perhaps. We'll start to power on the image and we'll try to SSH into image to see like, if those things actually got installed and everything is running perfectly or not. Before powering on the image will just inject the cloud in a data, which injects a user for us to be able to SSH. Once we start the VM it takes some time to boot up and get an IP address. So the image SSH might not work. As we can see here the images are installed, but the images that we preload in order to make the boot up of the cluster faster. We can see all these image processes are also running some of which are exited because it's in a state of reconciliation where slowly everything will be powered on and now. Next, you can see that we installed this tool called Harvest, which is installed and present on the system. Yeah, so this is our demo of the complete thing. I would also like to mention that demo was for OVA, but we support wide array of providers and OSes like AWS Azure, VMware, GIMP, GC, Digital Ocean, and OSes like RHL, Ubuntu, Amazon Linux 2, CentOS and also like FlatCard is in progress. I'll pass the slide back to Mosh now. Thank you to Sharon that was very informative. So the image builder is definitely far from complete. It is being used in production for a number of use cases. And for the future type of plans we are looking at releasing a CLI that will replace much of the makefile functionality and make it a little bit more discoverable and user friendly. As well as make some of the configuration auto-generated and merged. We are also looking at adding more and more testing as we go along. We already have Azure based PR testing and we'll add AWS, QMU, GCP in the coming months. FlatCard support is currently in progress and that PR shouldn't merge soon. Same for Windows image support with support on Azure and OVA. We are also looking at taking the cost based specs and expanding upon those to make it a generalized conformance test suite for Kubernetes images. And not even just for images created by the image builder, but for any image and used as a tool within the community to make sure that all images are consistent and version compatibility is tested for and verified against. We also want to do a lot more testing on the images themselves to make sure that they do things like rotate images and they don't automatically upgrade packages if they shouldn't. And also publishing test grids and up to the test grid for the results of these images for each version of Kubernetes that's released. And then finally promoting images for public consumption in much the same way that the containers are promoted today for many of the sub-projects. So in terms of the CLI, so the CLI is not a replacement. It's an addition to the existing toolset that encapsulates much of the make file based functionality as well as adding a few other non-packer based approaches for building images. And that should be released for consumption also in the next month or two. And likely by the time this video is or this presentation is running, it should be out as well. In terms of how you can contribute, probably the biggest way is use the tool, consume the images and tell us what could be better. Attend the office hours if you want and really report new issues, bugs and suggestions, anything is welcome. And where you can find us, so we have a bi-weekly office hours. We're on the main SIG cluster lifecycle Slack group. There's a get book as well. And you can generally reach us on Slack or at the office hours. And then next up the Q&A. If you have any questions, please feel free to raise them and thank you again.