 Hello, welcome to clean up your room. What does it mean to delete something in Cates today? I'm going to talk about everything delete how to delete things in Kubernetes and why some things get deleted and others don't My name is Aaron Alpar. I'm with castin castin as a cloud native data management company We deal with data management for disaster recovery as well as cluster migration My background as an engineer is mostly in custom database implementations Domain specific database implementations for order matching dating websites biological analysis and geodesics so Today I am going to be talking about delete which can be difficult Specifically, I'm going to be talking about what properties on a resource can govern deletion The two things I'm going to focus on are finalizers and owner references And I'll get into the details on how these two things can affect deletion of objects within Kubernetes I'm going to demonstrate how it works and this is going to be a short presentation. So I have 30 minutes plus Q&A So the goal here is to inform inspire and make you dangerous. I encourage you to take what you learn here and Experiment with it on a test cluster So Kubernetes has a lot of commands, of course, I'm going to be focusing on delete I'm going to be using four kubectl commands throughout this presentation I'm going to be using create get patch and delete to keep things simple All of my examples are going to use config maps and the entire presentation is basically going to be presented as a series of shell commands So the idea is to Give you the commands Just show you how they work or what their repercussions are results are And to give you some examples that you can take back and try for yourself. So Here's the basic delete all of my shell commands are in bold the output for those shell commands is Lines immediately following an unboldened unbolded letters Here we can see I've got a kubectl create config map my map This is going to create an empty config map. I can then get that config map to prove that it actually exists I can then delete that config map and then I can attempt to get it again And I can see that I get a for a four back on that last kubectl command, which means that it's not found This is the basic delete it is very simple We start off with a kubectl create which creates a live object. I then perform a kubectl delete Which deletes the object and it's basically the final state for the state diagram This is this. This is a state diagram. I'm going to have one more of these Within the presentation to basically explain to you how finalizers work So finalizers finalizers are resources on our Finalizers are keys on resources That control their garbage collection Finalizers are keys that are designed to be used by controllers To tell them which which cleanup operations have to be performed before the resources removal They are they are very simple they do not necessarily They do not necessarily Point or name code that needs to be executed. They're just a list There are certain dead finalizers that can prevent deletion Dead finalizers are basically finalizer strings that the controller does not understand And does not know how to deal with I'll be talking about those Knowledge of finalizers is particularly useful and when you start to get into specifics of deleting objects knowing which finalizer keys Mean what controller actions on deletion can be very helpful in debugging why some objects don't get deleted I'm going to start off this Demo basically with a custom config map which has no properties, but contains a finalizer Finalizers are on the metadata of the object. They are a list of strings in this case I am adding a Kubernetes finalizer to this object Kubernetes finalizer is a dead finalizer for a config map a config map resource Controller doesn't understand what to do with this is the least as far as I know Kubernetes finalizer is actually used on namespaces. So this is going to produce some interesting results. So here I'm going to create this config map With the finalizer here you can see that the finalizer has been created I'm done going to attempt to delete that and here. Let me show you this in code on what actually happens Get my shell up Here I can create that and Here I'll get the config map Show you that it actually exists and we can take a look at the content Here I'm using Kubernetes 1.8. So I get managed fields. I've taken these out of my demo my demo slides to keep the slides short As you can see here toward the top of the object, I've got my finalizer and If I go ahead and attempt to delete that Kubernetes is going to tell me that the object has been deleted It hasn't been deleted in a traditional sense. It is in the process of deletion and here I'll go ahead and I'll background that task and I'll go ahead and Get that object again, and we'll see that it has been modified Specifically the deletion timestamp has been added the object so here I've created a config map with a finalizer on it I've attempted to delete it and what's happened in actuality is the object's been updated with a deletion timestamp The deletion timestamp Kubernetes added that because it seemed that that the object has finalizers and it's put it into a read-only state The deletion timestamp signals that this object can only be read with the exception of removing finalizer keys updates to the finalizers so This will the delete will Basically hang in the background until I actually go ahead and edit that object Go ahead edit this and I'm going to remove the finalizer and go ahead and write that and we'll see what happens It can fit tells me the config maps been edited. I removed the finalizer key and notice the delete has continued When I go back and try to get that config map once again, it's not found so So just as I showed you I created the kid I created the config map with a finalizer the deletion timestamp I tried to could delete that In actuality it got updated the deletion timestamp got added to it And here's a demonstration of using the kubectl patch command If I want to delete that object what I can do is basically patch it on command line to remove the finalizers In which case the deletion that I had before Running in the background will complete and the object will be deleted When I attempt to get that config map it will be gone here's a state diagram for finalization so If an object has a finalizer on it And you attempt to delete it it will remain in finalization Until the controllers remove the finalizer keys or the finalizers are removed by kubectl once that finalizer Finalizer list is empty the object can actually be reclaimed by kubernetes once it's empty It will be put in a queue to actually be deleted from the registry And here's basically a slide that says what I just said Keep in mind finalizers are just keys They should be managed by the controllers, but they're not always managed by the controllers Especially in the case of dead finalizers that I just showed you Here's some common finalizers that you've probably seen at least the ones I certainly come across Most often there are the PV and PVC protection finalizers They're there they are there to govern the deletion of the resources that are in the back of Kubernetes the actual disks Themselves and the actual reservations on the disk themselves There's kubernetes finalizers, which is used in namespaces and there's a foreground deletion Which I will allude to a little bit, but I won't get to it in depth So that's it for finalizers finalizers basically control how a single object is deleted by the controller So owner references Tell how groups of objects are deleted owner references our properties on resources that Specify the relationship to one another so entire trees of resources can be deleted Finalizers rules are processed when there are owner references. So these are not They're somewhat orthogonal on how they work and you've probably already seen these on pods typically have owner references to the replica sets so when the Deployments stateful sets or replica sets themselves are deleted that the pods are taken care of in the process I'm going to show you a quick example of Owner references and how they work. I'm going to start off with a really simple example and then I'll work into progressively more complex examples Here I'm going to create my parent object first Since the references are from child the parent have to get parent first and then I can go ahead and create the child Here I have to do some shell through To get the UID out of the parent and included in the shot child and you can see that once again We have a very simple config map That contains an owner reference to its parent An owner reference Consists of a name and a UID owner references have to be within the name same name space So you'll never see a namespace on an over a reference But there does have to be a there does have to be the object name. I also need the UID for that reference to work Here is another example here we can see after I created these objects we can see them both You can delete the child and When an over references involved the deletion of a child typically doesn't really do anything or rather The child will be deleted, but nothing will do will be done with a parent things get interesting with Deleting the parent owner references really The change how trees of objects are deleted from their parents So here we create the parent child references in the config maps Here I go ahead to delete the config map the parent config map and Because if there's an owner reference from the child of the parent when I go to get those When I go to get the config map the config maps, I'll see that none are in the namespace And Here I'll do a quick demonstration of this Here I have a I have a quick script to make this easy on myself Here we create the parent and then we create the We create the child with the parent reference in it go ahead and create these I Can get the config maps and here we can see that we have the parent child relationship We have a parent and a parent and a child and that the child the relationship is represented in the child and Down here near the bottom. We have the owner reference here If I go ahead and do the delete It tells me that The my map parent has been deleted, but in actuality The parent and child have been deleted Telling me if I go ahead and try to get the config maps within the namespace the default namespace No, no resources are found because they both been deleted. So getting back to the presentation We can see that there there's sort of a cascading operation by default When there's an over owner reference from a child to a parent when you delete the parent The the children are automatically deleted This is called cascade the default for cascade is true cascade option can be specified and delete And basically specifies whether or not it should orphan children Now this cascading property can can change it can be The the cast there's the cascade option on the leak can be specified that basically Allow you to orphan children so you can delete a parent without actually deleting the children if you delete the parent With a delete command using cascade equals false What it will do is it will take the owner references off of the children but leave the children intact There's something that's also called propagation policy, which the cascade option links to which controls the order in which the The order in which the nodes in the order in which the nodes in the tree are deleted Before this I'm going to start up a proxy in the background in here. Let me go ahead and demonstrate the Demonstrate the cascade equals false and in the process I Will start up proxy So I've just made my tree again. I've started my proxy I'll use the proxy in the next example, but for this example I'll show you how the cascade works in this example. I have a parent child and a parent In this case, I'll go ahead and get the child as yaml And we can see in near the bottom here that we still have we have owner references To the parent and if I delete With cascade equals false My parent as you remember earlier. This is deleted when I ran this Without a cascade option. It deleted the entire tree here. I'll go ahead and delete the parent with cascade equals false It's come back with exactly the same response as I got before config map my map parent has been deleted But as we'll see My map child still exists and in fact when I get that You'll notice that the owner reference on the child config map has been removed. So This is done by setting the what happens in the back end is the propagation policy when you when you specify cascade false is It sets the propagation policy for the API call and that's what we'll be looking at next Propagation policy allows you To change the order in which objects are deleted within a tree Propagation policy cannot be specified on the command line with kubectl Other than or fending objects You have to specify it using a custom API call. This is how you do it You you create just as the previous object you create a proxy So you have access to the API server from your client And then you can execute a curl command with just the URL to basically execute that delete command This is the curl command for the deletion And in here I can specify all sorts of options one of which is a deletion option And the propagation policy for a background deletion So here's an example of that. I will show you this I will show you this running on the examples that I have in just a moment This is a background propagation policy. This sets the order to delete delete from parent to child so There's three. There's three different options for the propagation policy. There's foreground deletion Which is the post order deletion Or you can specify a background deletion, which is a pre-order deletion Foreground is children are deleted before the parents are deleted Background is parents are deleted first Parents deleted before their children You can also specify owner owner, which is just like spastic fine cascade equals false Which means that the owner references will be ignored And the parent will leave the children intact the owner references on the child children will be removed and let me go over and Demonstrate this really quick There are my config maps and let me get the command Here's a curl command to perform the delete through raw API call Here I'm doing this because I want to specify a Custom propagation policy and there I got a response back that says that the leech on was successful I go here Should see that I have no config maps left in my namespace because they've been deleted Keep in mind that when you When you delete an object and owner references have been specified that finalizers will be honored in the process so switch back to the presentation that Finalizers will be Finalizers will be honored so this can lead to situations where basically you have trees Of objects are That trees of objects that have not been deleted is basically you end up with partial deletions It's then you've got it at that point you have to dig in you have to look at the existing owner references on your objects with some knowledge If the owner references have been deleted what they were before as well as the finalizers To understand what's happening there. We will once again the goal here is give you the tools to figure that out So there's an exercise that you'll have to go through to do a little bit of research There's one situation that you might run into where you need to force finalization for a namespace this is a basically If you've deleted a namespace you've cleaned out all of the objects under it But yet the namespace still exists Here's the command to force the finalization. This is one of the reasons why I showed you the right PI calls before this there's a There's an API call off of off of the namespace called finalize It informs the it informs the namespace controller that it needs to remove the finalizer from the namespace and Perform any cleanup is basically a nice way of telling the namespace to remove the so So That's concludes the presentation so You take away takeaways from this our finalizers can get in the get in the way of the deletion of resources especially when there's trees involved of owner references Generally, there's a reason for a finalizers presence. You should investigate first before manually deleting it Owner references allow tree trees of resources to be removed specified and removed and once again Finalizers are honored in that process so they can they can result in partial deletion of Partial deletion of trees Propagation policy can be specified in a custom API call to change the order in which resources are delivered Deleted so if you're in a situation where you do have a partial tree You can control how those objects are deleted by specifying the propagation policy on the API call so Here are some additional references These specify a lot more detail about how the garbage collection process occurs Please take a look and thank you. That is it