 Hello. Welcome to the MAC Collision to talk. My name is Eddie. I'm working at Red Hat on Open Chief Virtualization. It's in UPSIM, it's called Kubev. And I wanted to talk today about MAC Collision, but in essence I more or less focusing on trying to think about things a bit different, in a different way. And this talk is a bit special because whatever we are going to speak here about alternative solutions, they are not really existing. It's still open. I like to do it, but maybe it's not worth it. Maybe there are better ideas, so you're all welcome to comment on it. The first thing that I wanted to say is that a lot of solutions that we are doing, I don't know, it's like an engineering practice to complicate things a little bit more than it must, and it causes a lot of afterwards complications. So it will be very nice to, after we have something running, working, we did it very fast, and after a few years we can look back and see if we could have maybe simplified a little bit. Especially after a few years things are getting, many things go, are added and it gets more complicated. So what I would like to talk today is about, first of all I will give some context. Who knows here about Kubernetes, everyone? Who knows about Kuvir? Most of you. So I'll go over very quickly. I already had a talk with Andrea that is here, and they did the same thing, so I will be very, very quick about it. Then we're going to talk about why or what is this MAC address management thing, and what we have today, how we manage it today, and what maybe would be an alternative to it. So the ecosystem, some background. We had in the beginning a virtual machine, it's a simple thing, then we had many, on many nodes, so we had to manage them. That's the regular thing that we do. And then came the containers, then we also had many of them, and then we managed them. And the result of that came Kubernetes, which manages pods, which are actually behind them, we have containers, and Kuvir came along, and we managed them together, so we'll have one unified ecosystem management system. And now a few words about MAC address management. So why do we need it? Can anyone tell me why we need to manage MAC addresses? It's unzod, because usually if you know what's a MAC address, usually the manufacturer of a card is just putting it on the physical nick, and that's it. You don't need to manage anything. Yes. Unique? Yes, unique. So this is usually not a... Most of the time it's not a problem when the manufacturer is doing that, although I didn't know it in the past, but manufacturers do create, duplicate MAC addresses, they just send them to the other side of the world, those things like that. So that's one of them, and can you think about why we need it in VM specifically? Yes, and... Yes, and... And one. So yes, that's a uniqueness one, right? But there is an in virtual machine that we want also to manage, because when we create the virtual machine and we shut it down and we start it again, it may come off with a different MAC address, so we want to avoid that. It's especially a problem in... when we run it in Kubernetes, because we run virtual machine in pods, and when you create a new pod, it gets a new interface, which may have a different MAC address, and as we somehow manage it or tell it to use a specific address. So that's the persistent part. The persistent part is that we want... once the virtual machine was created, with some one or more nicks, virtual nicks, we want the MAC address to be there forever, not to change. Also for migration, but that's a small detail, and the second one is that we want uniqueness, so we want to avoid MAC address duplication, and we'll talk about it a bit later, more in detail about it. So, what is MAC address duplication? Is anyone here knows why there is a problem with MAC address duplication? What's the problem having two MAC addresses? I will try to make it fast. Do you want another? Yes. If you... So I will show you here why it's a problem. It's like a very simple use case. Let's say that you have two pieces, MAC A and MAC B. They are connected to a switch. The switch has a MAC table, and the MAC table has ports on one side, and the MAC address that it learned on the other side. When a switch usually learns the MAC address, I'm talking only about switches. Hubs is like two generations behind us, so I will not talk about it. In switches, MAC address... Switches learn about MAC address when some traffic goes into the switch. It looks at the source MAC, so it just learns it and puts it in its table. In this case, we have A and B. It learns it on two different ports. Now, if we have... The two pieces have the same MAC address, so let's assume it to learn it in port one, and then traffic came from the second machine, so it learned it on port two, and then it goes like this forever. It's not able to put the same entry on both of them. Usually, the classic switches will not do that. Moreover, the regular switches that we have today will just block it. One of the ports... As problematic, and you just block that port, so you will not have a problem in your network. Because it may assume that you have a loop, like a cable, which will cause you a lot of trouble. So, what can we do if we... What can we do if we have this situation? We need to somehow manage this, not to get into this case. Obviously, we will manage it. We all have a solution to manage stuff. So, the solution that we have today is one of them is prevention. This is what we have today, and we will go over it today, so we will know. And we need to solve what we spoke before, like the problem with uniqueness and the problem of having it permanent. So, prevention. So, we have... This is a very simplification of a tool that we... Not a tool, it's a product today. It's called kumek pool. It is sitting in a Kubernetes cluster. It has a controller. Everyone knows what's a controller, by the way. Kubernetes controller. I will... No one answers, so I'll answer it. A controller is something that looks on the system in a loop. So, just check it, if the system is as desired. So, in Kubernetes, the concept is that you specify what you want, and you have what exists, and it tries to reach from desired state to actual state. To move it to be an actual state. So, that's what controllers do. They usually look at the configuration and at the actual thing that exists there. What's a webbook? Does anyone know what's a webbook in Kubernetes? No? So, a webbook is... It's an endpoint. A web... URI endpoint, I will say, that you can configure a Kubernetes that when you change the manifest of an object, it will just redirect it. You will be redirected to that webbook. That webbook can do two things, one is it can validate the configuration. So, if the validation is not okay, it will just drop it, and the manifest will not be preserved in the database, in Kubernetes database, and the other thing, it can change it. You have a way to change... You have another step that you can change that configuration, and then it will be saved. So, given that we have that and we have a pool of MAC addresses, we create a manifest, and the controller obviously looks at that configuration. This is the VM manifest that describes how VM should look like, and the webbook will trigger as soon as we created that manifest. So, it goes there, and it's tried to go to the validation step. So, it checks in the pool that if this MAC address exists, if not, it can register it, and if that pass, it does two things. One is it mutates the manifest to put the MAC address there, so it solves the problem of the persistency in that manifest, and it tells the... It causes the pool with the virtual machine to be created with that specific MAC. Now, the case that it doesn't work, when there is a problem of duplication, so someone can use the manifest and specify a specific MAC address. So, if someone specifies a specific MAC address, then it will get to the webbook. The webbook will check with the database, with the pool, and it will see that it's already there, so it will reject it, and that will cause it to just drop the... I mean, it will just fail the creation of that manifest. So, the pod will not be created, the VM will not be created. Now, this is what we have today, and it works pretty well. The problem is that when something else happens, it doesn't work so well. So, one of the nice things that I like about the existing solution is that it's not intrusive to the VM system. It's external. It just monitors what happens. So, if there is a problem, it will just stop it, and that's it. You don't need a special parameter inside your VM manifest. I mean, VM, control, plane, and everything does not need to know about this kubemek pool thing. But there are still our problems. So, the first problem is that we are... This mechanism is just blocking creation of virtual machines. So, if, let's say, a situation that you have one virtual machine created with a specific Mac and then you create another virtual machine with the same Mac, I don't know why, then if that's happened, immediately it stops and the VM will never be created. But it's a bit anti-kubernetes in the sense that you describe what you want. And maybe in 10 seconds the other VM will go down, but you could not do what you... in the actual state, because it just gets immediately failed. So, it doesn't allow you to say, I want this, and eventually it will happen. I cannot do that with this tool at this moment. Another limitation is that it works only for virtual machines. So, if you need it also for just pods or something else that works with pods, it doesn't work. Ordo, there was an attempt to do it, but eventually it didn't. It also has one thing that it has only a single pool. What does it mean? It means that we can have two different networks and each network can have different local address. It can have a broadcast domain of its own. So, in this case, I will not expect even if there is duplication, it still should work. But this one treats everything as a single LAN. And just managing the whole thing is a bit complicated, because if we look here, then once you pass through the webbook, you still need a controller, because the webbook may have worked and everything worked fine, but there may be another webbook that will fail your machine. So, it will fail the creation of your machine and you want to give the Mac back to the pool, but then you need the controller. So, it gets a bit complicated. And this is the main thing that I actually... The only reason that I did this talk is about this case. The probability for a Mac duplication is really, really slim. It's like we are doing all of these fancy things. We spend a lot of energy and a lot of code to make it happen, but what is the probability of really having a duplication of Mac addresses in a cloud system where some of the interfaces are provided through some SDN thing or some other means. The probability, I guess, is very, very slim. So, then, what can we do to... What can we do differently? That's the question. So, one of the things that we can do differently is to think about this problem from a Kubernetes perspective. We want to facilitate the collision, if there is a collision, which means that we can let it happen, and if it happens, then we can try to detect it and then do something about it. Now, there is a very, very low chance that it will happen, so we will not have the blocking problem that we always have. I didn't say before, but one of the things that we did experience is when this service that provides this webbook is down, then the creation of VMs cannot happen. That's also a nasty thing. This is related to this sentence that I really like, is... It's actually taken from... Many Pythonists are using it a lot. I did program in Python in the past, so it's easier to ask for grants than permission. Do you know this sentence? It usually solves a lot of races, like, for example, you ask a question, is the file open, and then you open it, right? But in between the asking the question and actually trying to open the file or doing something on it, someone may have done something else in the middle. It's a regular race condition. So, in order to try to... One of the options to solve the previous problem is to separate the two things about persisting the MAC address and look in at the collision thing. So, one suggestion is to do some sticky MAC addresses with what we have today. So, we could take the existing control plane of virtual machines of Kuvert, and we have here a VM controller that looks out on the VM creations. So, assuming we have a VM manifest created, it will look at it, and let's say it will... There is no MAC address involved here, right? So, it will create the virtual machine. Some random MAC address, in this case, A, will be created. And that's it. This is the... We finished. We created the VM. Maybe this MAC address is duplicate. We don't know, but we don't care at this stage. So, after the A is created, the controller sees that this is the MAC address, and it can go and update the manifest. And say, okay, you created the VM. As the MAC A, I'm writing the MAC A in the manifest, and from now on it will always be there. Even if you shut down and install the VM, you will have it. So, we solved the stickiness problem of persisting the MAC address program. And regarding the problem with duplication, if we also look only at on VMs, then what we can do is just have a simple controller without this webbook, and we can just look at the manifest and see that there is a MAC address in the manifest, and it will just write it to its pool. If it manages to write it to the pool, register it, or everything is fine. If it doesn't, then it will just react to that. It will say, okay, I cannot register because it's already in use, so I'm going to tell the virtual machine to either stop, I can shut down the link, I can do whatever makes sense in order to resolve the problem. So, this is in a reactive mode. Things already happened, maybe disturbed traffic, but eventually it will react to it. And eventually it will do something in the pool. The manifest will be reflected in the actual virtual machine. Another thing we could do if we want to make it more generic, by the way, the same thing we could do for pods, I didn't specify it here, but we could do the same for pods unrelated to VMs. But for, we could do something smarter, like, for example, real switches today. We have SDN switches, like OVN, OVS, and we have all kind of vendor switches, like Cisco, Juniper, stuff like that. They are already detecting duplicate MAC addresses, then you can also query them and talk with them and ask them what is the status. So, if we could talk with someone else, or we could monitor our system about what's going on, like we could run TCP dump or some monitoring tool to check all the ARP traffic and see if some are registering with the same MAC address. It doesn't matter what's the solution, but that monitor thing can look at the network and once it sees a problem, it will just go to the controller, like in this case, and the controller will go and update the manifest, the same thing we did yesterday. I don't know why I'm talking about yesterday. The same thing that we did before is to react on the VM itself. Either stop it, stop the link, whatever. That's it, most. Do we? I think I finished. Any questions? Currently, I think it's using it in memory. So, the way it works is that when it comes up, that controller comes up, it will just look all over the system. That's one of its disadvantages. It looks all over the pods and VMs that exist, collects them all, puts them into the register them all, and then from here on, it just reacts to what happened. Oh, sorry. The question was, what was the database used to save the MAC addresses? Yes. No, it's... Sorry. The question was, where is the MAC address? Is the MAC address somehow registered in the manifest or not? It's specified in the manifest, right? In the configuration? Yes. So, if the user does not specify a specific MAC address, then Kube MacBook, that project will identify that would just edit itself, like here. If the cluster is killed, no. If any of the components of the system are down, like this is Kubernetes thing, because it has in the backend all the manifest that it has is implemented using gtcd, which is distributed, so even if things are getting... nodes are getting shut down or stuff like that, it will still be available. That's Kubernetes given, yeah. So, one of the disadvantages of the... the question was that we are not managing, right? We are not managing the MAC addresses outside of the cluster, right? So, we are not managing the MAC addresses not only from outside the cluster, even in the cluster for pods that are not VMs. We are not managing that, so they can cause collision. And surely if we use, for example, secondary networks, if there are devices on that network in the same LAN, then we will not even detect it. It's like nothing that we can do. That's only what I showed in the end. If we want to do something like that, then we could use some other external tooling to make it better. But I think this is the point here, is that what I tried to make the point is that the chances of this to happen are so low that you should not bother, probably. I mean, I would be interested to see someone actually having this problem and then we can talk about it. No more questions? Yes. So, have I... the question was have I encountered duplication of MAC addresses in reality, right? I worked in the past with a lot of networking not related to this system. And there can be something, for example, in a virtual machine, if someone is cloning the virtual machine and the cloning mechanism actually took an existing machine and duplicated its image, then yes, the MAC address will be duplicated because the MAC address many times is written in the configuration of the operation system. So we will have two virtual machines coming up with the same MAC address, which is a problem. Now, even in reality it can happen because you can... many times someone can... today there are tools that knows about it, so they will not do it, but in the past you could have just taken... cloned the disk and put it like in 100 machines and distributed it. If they were sitting in the same land, they will just start to collide. But as I said, in the cloud the real switches are just blocking your traffic, so you will see it in the logs that just block the port. So yes, it can happen, non-intentionally, but there is also an option that one of the things is that someone can change the... someone that runs inside your guest will change the MAC address. That can also happen, but then there are tools like MAC spuffing, which says that if I created a virtual machine, it's supposed to have MAC A, it can only get out with MAC A, it cannot get out with MAC B, for example. So that's like a security measurement. Anything else? Thank you.