 Thank you, everyone, for coming to the session. I'm really proud to be a part of the 101 track. And a little bit about myself. I'm originally from Eastern Europe. So my native language is Russian. And coming to United States, definitely, I had to learn English. And what is interesting, the transition for me from being a Java developer to being a GoLand developer was pretty similar. So I had to learn different language. I had to understood the concepts. And it was a pretty fun journey. And I would like to know a little bit more about you, like how many of you use Kubernetes in any way, as a user, as a developer? All right, great. How many of you write tools for Kubernetes? All right. How many of you use Go? Oh, I really like the diversity. All right, so this session is going to be about using Kubernetes API from Go. First, let's talk about why we all love Kubernetes. As a user, you probably appreciate the feature set. There are so many things that you can use from Kubernetes for the workloads, for the node management. And if you're a big company, you appreciate the Kubernetes ability to handle the big workloads and its performance. And if your Kubernetes is a contributor, you definitely appreciate the community, which is very friendly, very approachable, very helpful. But the one thing that I wanted to focus on is Kubernetes extensibility. So that was actually very useful from the developer standpoint. You can extend Kubernetes in so many ways. You can develop your own APIs if it suits your use case. You can use custom resource definitions. We're going to talk more about it on the next slide. And that extensibility, that's what actually makes Kubernetes a platform. So what platform means that is there is a core feature set and this pretty much set. And it can be extended by the third parties without actually disrupting the main code base. So you don't have to, if you need something extra, you don't have to go and modify the Kubernetes repo. You can write it and push it to your own repo. You can deploy it and use it. Even the APIs can be plugged. And this new APIs that you plug in can be used from the Kube control. And then you can write your own custom controllers if you want to control your own resources. And that what makes the Kubernetes ecosystem. All the tools on the slide, I was still easy to put them all, but there's like a bunch of them. They would extend Kubernetes. Those are basically the custom controllers. So they extend the login functionality. They provide you the DNS, the service mesh, the monitoring. And that is pretty cool. And if you think about the ways that you would like to contribute to Kubernetes, or you would like to extend existing Kubernetes functionality, you would like to make a better logging, then you have to become a part of the ecosystem. And there are several ways to extend Kubernetes. Like on this slide, I've listed all the main Kubernetes components, which are like the controller, the API server, the scheduler. And then on the worker, there is a kubelet and their kube proxy. And if you look at the kubelet, there is a way to extend the continue runtime interface, which just sounds pretty intimidating to me, like too complicated. I've never looked into it, but there is a way to do it. So you can do Docker. You can use Rocket. Then on the controller manager side, there is a cloud provider. So you know that Kubernetes supports multiple cloud providers, like a Google cloud provider, Amazon. And these components are all pluggable. And there is a big initiative in the Kubernetes. The sum of them are part of the Kubernetes core base, but they're working on moving it out. So it's going to be more flexible. Like if you're a digital ocean, for example, you can develop your own plugin, and you don't have to contribute it to the Kubernetes core. You can just keep it in your own repo and handle the deployment. And then if you look at the API server, that's where it becomes more interesting and more user and developer friendly. You can build your own API extension. So Kubernetes native API, they have a bunch of objects, like nodes, pods. You can design your own objects, like, for example, if you do monitoring, you can think about your custom object that you can use just for monitoring purposes. And then you can write custom controllers for these resources. That's what came to the left of the slide. And that's what we are going to be focusing on today, writing the custom controllers. That component that just talks to the API, it can monitor either existing Kubernetes resource. It can monitor your custom resource. Then it can act on it, modify it, do a bunch of stuff. And you'll hear a term controller a lot. And there is no exact definition for it. If you Google what is Kubernetes controller, you're going to be given a bunch of stuff. And then I try to think about it like, what would be the best way to explain it? So have any one of you use Kubernetes ingress? So yeah. So Kubernetes ingress is just a way to provide the entry point for your application and to balance it. But if you look closer, the ingress, Kubernetes ingress resource is just a shell. It doesn't have any implementation. So you create Kubernetes ingress resource, and then there is some magic happens. And what this magic is, the people deploy the ingress controllers, which is a separate piece of code that runs, monitors Kubernetes ingress resource, and configures the external load balancer or internal load balancer to provide the actual functionality. And then it modifies ingress resource with a load balancer IP address. So as a user, you create your ingress, and then you magically get the IP address back. That's what ingress controller does. That's what controller is. It monitors something, it does something, and then it updates the resource and gives it back to the user. So now, what are the ways to talk to the Kubernetes API? The most simple and the most user-friendly way is, of course, the UI, the dashboard. Then Qubectl is also very user-friendly. If you prefer CLI, that's a great tool. And then there is the programmatic access. So if you're going to write your tool in any language, Go, whatever, you'd probably call this API programmatically. You're not going to be calling the QubeControl. Although QubeControl has a bunch of commands that I actually appreciate a lot, like Qectl apply, it's great. So yeah, programmatic access using APIs. But then how do you call this API? So Kubernetes API server is just an HTTP server that accepts a request, lists the resources, get the resource, update, remove. But how do you call them? Is it just a raw HTTP request? No. So well, you can send it as a raw HTTP request, but there are a bunch of libraries out there that are developed for different languages, for Python, Java, Go. And as any language, you can think of. But those three are the most popular one. And today, we are going to focus on the top one, Client.go. So that's a great library. I've been using it a lot. I work for the company that does a lot of stuff for Kubernetes, around Kubernetes. We use Client.go a lot. We monitor Kubernetes clusters. And yeah, so I've been using this tool for a while. Why would I recommend to use the Go client as opposed to any other language? So Kubernetes is written in Go. And I always find it's better to develop the client in the language in which the main component is written. Because it implies the faster update. If the main component gets updated, like Kubernetes gets updated, Client.go gets updates almost immediately. I'm not saying that the other tools don't get the immediate updates, but the Client.go is probably the fastest one. Go is pretty simple. It's easy to use. And it's very easy to be used for applications that are meant to be deployed in the container. And that's something that we are going to do today. So the best way to learn something is actually to build something from scratch. So today, we're going to build a tool that is going to monitor Kubernetes nodes. And it's going to alert when the storage occupied by the images changes. So here's the link to the repo where the tool resides. It's a pretty simple tool, but you're going to go from the very beginning to the very end. You're going to do it like everything. So as a developer, I like to sneak a peek of what people use for the development, what makes their life easier. So here are some tools that I use for demo project. Those are all internal tools. Open source, you just use them in Rancher to make the developer life easier. So the Go scale, if you're not familiar with Go and you wonder, what should be my Go project structure? You can just run this tool, and it will create the basic structure for you. It will even create the main class that you can actually invoke and run without any compiler. Then the trash. So it's a tool that manages the third party dependencies. Usually managing third party dependencies can be very, very painful. And there are many tools out there, but I like this one just because it's very simple. I'll show you how I use it. And then there is a tool called Dapper. So if you have an application that is meant to run in the container, then you would probably like to build an image for it. I use Mac. But if I run the container, it can be a Ubuntu. So I need to find some simple way to actually build my application so it can run on Ubuntu. So I use Dapper tool for that. It will spawn the container from the Ubuntu image. It will build your program in it, and it will give you back a nice Docker image. So what would be the first step? The first step would be to add client.go dependency. So you have to make sure that the dependency is compatible with whatever Kubernetes version you're using. So I'm currently using the, and for this demo, I'm using the, I think it's 4.0. So it works pretty well with Kubernetes 1.7 and 1.8. But I think there is a newer version of client.go. It shouldn't be any different. So what you do, you just add this to the file, vendor.conf, and then you just run the trash command, the tool that I've shared earlier. What it's going to do, it's going to pull all the third-party dependencies to your project, and that's it. The job is done. The tool is pretty smooth. I haven't had any problems with that. Then the second step, so you define the vendor, you got all the third-party dependencies that you need. By the way, it's going to, if this client.go uses some other dependencies, transitive dependencies, it's going to pull it in as well. You don't have to define them in this file. So trash will do all the job for you too, figuring out what needs to be pulled extra, and that's it. Now, the second way, if you look at the client.go.doc, the first thing you're going to see is like, how you want to run it? Do you want to run your tool in cluster, or do you want to run your tool outside of the cluster? It's like, I don't know. What are the advantages of what approaches? What does it mean to run in cluster? It means that your application is going to run as a part of the container that is going to be deployed as a part in Kubernetes. What advantages it gives you? Kubernetes is going to manage your application. You can configure health check if your application dies, Kubernetes is going to bring it back. If you want to run it on every node, you can deploy it as a demon set. Then, why would you choose to run application outside of the cluster? One of the use cases I'm working on, one of my tools actually monitors multiple Kubernetes clusters, so it has to be deployed outside. So I deploy the outside, I run it as a binary, I monitor it myself. Then, another good use case for that is like, let's say you develop something, and you really want to test things fast. You don't want to build the image every time you make a change. So you just run it as a binary, just build run it as a binary, and then at the end, you can choose all right. At the end, my application is perfect. I'm going to deploy it in cluster. I'm going to use the first approach. For testing, I'm going to use the second approach. All right, demo time. I'm switching to my IDE. I'm using Visual Studio Code for code development. Is the size all good? Should I make it bigger? It's great. Before we move on to the IDE fun part, any questions about the previous slides? No? All right. All right, let's move on. Let's start from the top. So in Go, here are all the imports. It's all the third-party packages that I'm using in my tool. This one is my favorite one. That's a CLI. So my application is going to accept the config parameter, which is going to be the path to the kube config file, so my tool can run outside of the cluster. So if you've got a kube control, the kube config is probably going to be already there. So just specify the path to it when you run it. Then once you get this config, it's time to get the client. Sorry, I'm switching back. Here we go. So this is the part where we actually decide on how you want to run your application. Do you want to run it outside of the cluster? If the path to config is not empty, then you use this package to build the config from the content of the pile from this path. Or if the path is empty, then you run it in clustering. So what are the internals? So usually, people use service account for authentication in the pod. What it means is that this default service account, if you don't specify anything, is going to be assigned to the pod, and it means that the credentials are going to be saved somewhere in the pod. This tool, it will know where to look for them to initiate the in-cluster config. Once the config is created, that's where you call Kubernetes. She's a client.go package to create a new client for the config. So we've got the client, and this client will help to make the request to the Kubernetes API server. Now, before it actually starting implementing something, when I work with a new package, I just test basic functions. What are the basic functions for the HTTP server? It's crowd, basic crowd, create, read, update, and delete. So let's move to this. So the first basic function, we know that for our application, we need to monitor the nodes. So we need to get the list of the nodes somehow. For that, I'm going to call client set, which we just created, and then I'm going to call core Kubernetes APIs, because I know that my nodes are the part of the core Kubernetes APIs, and then I'm going to say, okay, so I need the nodes object, and then comes the app creation that you're calling for this node function. I want to list them. You can pass the list option parameters. For example, you want to filter the node that you want to check. In my case, I'm running Minicube, which is pretty simple to install, and I know that the node name for the Minicube is always Minicube, so I'm just saying, okay, give me the Minicube node, and what it can give you back, it can give you either the nodes or it can give you the error. So error, something went wrong, I'm just going to skip. If my nodes are fine, I got them, I can iterate through them, but I just choose all right. I'm just going to get the first of them, and then what you can do with the object that you get, you can update it. So that's a second operation that I demo here. So basically, you can reset any field on the object, and then you call client set very similar to the list. You can just, instead of list, you call update and you pass the reference to the node. Here are some commented code which demonstrate the removal. I didn't want to remove the Minicube node, so I commented it out, but it's pretty similar. You also call the client set core APIs, you call the delete, and then you can pass the delete options, which is pretty cool, like you can say, all right, so what is grace period? I'm passing the grace period, it means delete my object, but delete it only after 10 seconds. So, okay, so now we've got the client, we know how to do the basic crud. Now it's time to think about it like, how are we going to implement our node polling? So yeah, and that I usually start simple. I don't think about like complicated way of doing things. So I start with polling. Like the basic way to do it is to poll something. By polling something means that you list something and then you wait for some period of time and then you list again, and you continue doing it over and over again. But there's certainly downsize to this approach. It's like, you only need to get the information when something gets changed. Like, we are interested in the changes in the images storage. And what I would like to have ideally, I would like to get the notification when something gets changed for the node. So I want to watch for the changes and I want to be notified on this notification. I'll decide what I'm going to do. So that's where the informers come. And now, what is the informer? Informer, it's a very nice way of watching for the resource and actually passing the callback function which can be called when the resource is modified. So here is an example. Here we are calling the cache package which is a part of the client.go. Here you go. And then we say, okay, give me the informer and what we are passing to the informer. We are passing several things. So first of all, we are passing on the instructions of how we want our resource to be listed and how we want our resource to be watched. So we are passing the list watch function. So we want to get the nodes and well, nodes is not namespace. So we just say namespace all and we want to get all the fields. So we are passing the watch list and then we are passing the object that we want to get. So we want to get the API node and then we are passing our callback functions and we are passing the callback functions for to handle the add and the remove. We don't add an update. We don't care about the remove because once the node is gone, we no longer care about the image storage capacity. So I'm just saying, okay, call these functions whenever the node gets added and whenever the node gets updated. And there's one more parameter that you can pass in which is a rescind period. So what is a rescind period? Sometimes like, well, there are no perfect things in the world and sometimes the list of watch can break. It would be nice to have some backup mechanism which will be called periodically. Like what I'm saying here, like every 30 seconds, even if nothing happens to the nodes, please give me them back so I can do something. So even when the watch function fails by some reason, I'm still back top by this periodic polling. So I just, I made it a bit extreme like 30 seconds. You can do it five minutes or anything you like. Then, so you're passing all these parameters. What are you getting back? So you're getting back the controller that is gonna actually do the watches and you're getting back something else very nice. The store, what is store? This is cash. So you're getting back the cash which will be for sure updated to the latest objects. So your tool, if it's needed, you don't have to call the Kubernetes APIs directly. You can just retrieve the objects from the cache and that will actually reduce the memory footprint. So once you get this controller back, you have to start it to make it watch for something. And then you're gonna be notified on the changes happening to your nodes. Now, there is an informer and there is something called shared informer. So what is the difference? What is, where do you use one and where do you use another? So you can see that in the informer, you can define one function for every operations. Like you define one callback per add, one callback for update and you get the cache back and the controller. Shared informer allows you to actually add more than one callback for the function and it gives you back the shared cache as well. So let's say you have, your application consists of multiple parts and there are multiple, there are multiple, actually here, as an example, I gave just the same function but it's usually the different function, like different callback. So different callback, I interested in the node information and ideally they would like to share the same cache so to reduce the memory footprint. That's where I would use the shared informer. All right, and that before we move on, I wanted to share the way of accessing the cache that we got back from the informer. So here's an example. As our cache stores the node objects, there is a way to retrieve it by the name, which is gonna be the cache key and you get back and you get back the node and you can do whatever you like with that. So what we did, we created the config, we created the client, we decided on the most optimal way of how to actually watch for the node changes and well, now it is time to actually run the application. So here's my go project, it's called kubecon and the first command I'm gonna do, I'm gonna run go build just to make sure it compiles. All right, it works. And just to share the structure of the directory, it has some, so this is just one class, main.go, this is a binary that I've just created and this is a vendor folder that we previously populated with trash. So we can look at it, what is this directory structure? What is this third-party folder structure? It has a bunch of stuff in it and if you look at the kubernetes.io, that's where our client.go is. All right, so we've built it now, we run it, how do we run it? Here we go. So you run as a kubecon and then you're passing the config to your kubeconfig and then it starts doing its job. But no changes yet. So the very first changes that you see, okay, something got changed for my node but it's only because it's reading it for the very first time from the cache. Once it's read, there are no changes. So we have to trigger the changes somehow. I want to infer to be changed. Okay, so what I'm gonna do, I'm actually gonna deploy something using the image that is not on the Minikube node yet just to increase the size occupied by the images. So I have this, all right, so it's just a simple deployment. It's gonna deploy a bunch of 18.04. All right, and now we're gonna watch and if there are changes happening, let's check on our pod. All right, so anyone, give me the any image, like any of your favorite images that I can fool and use. Don't say NGINX because it already has NGINX. It's busy box. Busy box, all right. I don't think it matters. I'm just gonna remove this deployment because I want to use the same name. Line, yeah, I'm so happy. The tool works. Okay, but that's an example of how to run it outside the cluster. So you run it as a binary. Now it is time to deploy it as a Kubernetes application. And that's gonna be fun. All right, so what do we start with? We start with the Docker file. So here's the Docker file that I use. So the base is gonna be Ubuntu and that's the commander is gonna call which is my binary, basically. And now how I'm gonna build this image. I'm gonna build it using that tool that I've shared earlier called Dapper. And here's the way of how I'm gonna call it. So I'm gonna say, okay, my repo, my Docker hub repo is gonna be Alina1.08. My tag, all right, let's make it today's date. I always change the minor version because my code is never that good to change the major version. So V070 and then I'm gonna call Dapper. So what it does, it creates a Docker container from Ubuntu. It builds the application in this Ubuntu container because if I build it on Mac and then I try to run it on Ubuntu, it's gonna complain. So I have to build it in Ubuntu. I don't, and I don't wanna create like a virtual machine or something. So I build it inside the container. It's usually fast, move on. Okay, so let me just share, let me kill something. This is right. This is a different Docker file. Yeah, that's what Dapper does. It creates a container, it builds the application, and then it gives you back the image. The image gets stored in my, it doesn't push, it doesn't push to the Docker Hub automatically. So you have to do it manually. The output is like, it just gives the image. So here you see, it built the image KubeCon V070. So, and it's on my machine. So now I have to push it to the Docker Hub. No, no, it's on my machine. All right, so I've built this image and I'm gonna push it. And now to deploy it on Kubernetes, we have to define the YAML file again. Let me remove the, all right, so I'm removing the old stuff. Let's look at the YAML definition for the tool. So it's simple. It's just, okay, so we just have to change the image to the one that we've just built. And that's the name, the KubeCon demo, image pull policy, all this. Okay, I think I've typed it. Let's see what happens. Something tells me that, okay, so one is terminating. That's the one that we just killed. And this one is container creating. That's the one that is new one. That's probably it's still pulling the image. Okay, so this is the one. I'm gonna look at the logs for this container. Ah, it doesn't like something. Let's look at what it doesn't like. So looks like some permissions are set, which is a RBAC. I have RBAC enabled in my Minikube. So it means that by default, nobody can just run the tools and execute any random API calls. I have to grant the permissions to the tool to actually do something. So let's kill this bad guy and let's define the permissions. All right, so what are we gonna do? We're gonna create a cluster role, which is gonna allow to operate on the nodes object and it's gonna allow to get, watch, list and update. So all the calls that we use in the tool. And then we're gonna create a service account that can be later on assigned to our application. And we're gonna call it KubeCon. I already have it deployed, I think. So I'm just gonna call apply to the right stuff. All right. And now we're gonna deploy the proper application. Let's look at it. So the only one difference from the previous YAML file is here you have the service account name, which is a KubeCon and the same typo. Okay, let's see now. So we're getting the pod and we're gonna see the logs. Looking better now, cool. So now our application runs as a Kubernetes pod. And well, I'm done, thank you. No difference, it's just another definition and I usually like, okay, when I see a new definition, I think it's a new thing, but it's usually the same thing just explained differently. So controller is something that just does this custom stuff outside of the Kubernetes. So it can be deployed as a regular pod, it can be deployed as a deployment, anything. You can do a bunch of stuff. Like you can extend the Kubernetes APIs, but what if you just monitor the Kubernetes resources similar to the way like I do? What if you just wanna, yeah. Well, if it does some useful thing, you can call it a controller, yeah. Any more questions? So usually I use Alpine actually, but this tool that I've shared earlier, the Go scale that actually creates the basic structure, it uses Ubuntu, I'm not sure why. Alpine is my favorite one because it's so tiny and just for this, you don't need Ubuntu. You don't need Ubuntu for this kind of functionality. Yeah, it can be anything, it can be monitoring. It's basically bringing the resource from the actual state to the desired state. That's how they put it. And that's how like I explained it for the ingress controller, like ingress. What is the desired state of the ingress? Just give me the access to the load balancer. And what is the actual state? Like when the user creates ingress, he just says, okay, so what I want to do, I want to balance between service A and service B. My desired state, I want to get a load balancer IP address. And the ingress controller will actually populate this ingress with the IP address of the load balancer that it's going to create somewhere. Oh yeah, of course. And yes, I'm using PowerPoint for my slides. Here we go. No advantages. Unless your permissions don't let you list the resources, which I run into, yeah. Yeah. This is right. Is there something that kind of goes the other way and just kind of watches for events coming? Yeah, that's actually the Informer part. So basically this shared Informer is meant to watch for the resource. Let me go back to the, let me go back from this PowerPoint to IDE. No, no, I'm happy to go to IDE. I love IDE better than PowerPoint. Okay, so this is it. This watch is just an instruction for the Informer on how to watch the resources. So here what I'm gonna do, I'm gonna watch on the node changes. Like all the nodes in my system, please watch for the changes. Whenever the add or update happens, call these functions. And those are mind functions. Right, right. So. When the node changes by some way. So if you look at the Qubectl, can we get bigger? If you do, oh Jason, it will give you like the entire object because Qubectl doesn't display you everything. So you look at the Jason output. And what do you see here? You actually see all the images. We're looking for the images, right? Oh, there's so many, yeah. So it gives you the list of the images with their names and the respective sizes. So, and yeah, if node object gets changed, it doesn't mean that the image get changed, but we're like, we are making our best effort here. So. You can watch any resource. There is a bit of a difference if you decide to watch the custom resource. It's not as easy, but it's doable. But if you wanna watch just any core Kubernetes resource like pods, it's very similar thing, yeah. I just, the advantage of that is basically, yeah, cache. You've said that cache is very important. And then it handles all the, like a failure cases, like what if the connection to the API server is lost? Then it's gonna handle it for you. So you don't have to worry about it's like, oh, okay, if something wrong, just go this way. So the inform is gonna handle it for you. No more questions? All right, thank you everyone. Yeah.