 Cut. So hi everybody. I'm Arikadas, and in this session we are going to see how volume popular's are being used when migrating virtual machines, and we will try to answer the question that appears in this slide of whether volume popular's fit also for a virtual disk. של וירטואל משין. ביט About Myself, כשאופה מהבחר לפגר, starting contributing to Overt, זה הופתרפוח באופרגי של רדת usual. פערתי כברט, הפסטרפוח של הופניים ולצירה אופרגי ולפסת קצת לעשות סקפות אופרגי אשתו על דתי הופניים. ובפני 2020, אני עשיתי בהלו ערב ואני עשיתי בסוף which is the upstream of MTV migration toolkit for virtualization. Forklift is an extension to Kubernetes that enables users to migrate virtual machines from traditional legacy virtualization management systems to a QVirt or OpenShift. In OpenShift 2.4, the available sources are vSphere, OVirt and OpenStack, and these days we work on adding more sources, OVAs, virtual appliance files, as well as QVirt itself in order to enable migration between QVirt deployments. Generally speaking, Forklift is targeted for mass migration and in more details, the idea is to simplify the process of migrating virtual machines at scale to OpenShift. We just few simple steps that are listed here, and let me show you how simple they really are. At the right hand side of this slide, you see the Forklift UI, the new UI that is integrated within the OKD or OpenShift console, and the process of migration using this UI looks as follow. We need to define a source provider. In this slide, you see the complete form for defining vSphere as a source provider. If we migrate to a remote cluster, not the cluster that Forklift runs on, then we also need to define this cluster. But in this example, let's say that we migrate to the local cluster where Forklift runs, so we will skip the next step and get to the creation of the migration plan. In this form, we need to specify general things like name, description, and also pick the source and target providers, as well as the target namespace that the virtual machines will be defined in within the target provider. Once we do that, we click Next, and we see a list of virtual machines that exist in the source provider, along with more information that users need to know before triggering the migration. We select the VMs that we want to migrate and click Next. This takes us to the network mapping view, where we can easily map networks that exist in the source provider to networks that exist on the target provider. With a similar user interface, we also define the storage mapping that maps storage domains, devices, types, data stores, depending on the type of the source provider, to storage classes on the target OpenShift cluster. Once we do that, we will see our plan in the Plans view, and we can simply start it by pressing this Start button. Once the migration is done, we will eventually see the migrated VMs in the target namespace within the target provider, and we can start using them. But what happens in between during the migration itself? So, if we migrate VMs that ran on a foreign hypervisor, not KVM, then we need to convert the disks, we convert their format, replace the drivers, replace guest tools, etc., and then we need to copy the rest of the data from the disks to the target storage. Then we need to change the configuration of the VM in order to adapt it to QVM. In this session, we will focus on the first two steps of the conversion and copying of the disks. So, that's about MTV. Now, let's talk about the volume populators feature, which is a recent feature that landed in Kubernetes in OpenShift. And the goal, the purpose of this feature is to import data from remote sources to our cluster. I will demonstrate how it works with resources that were shared in a blog post that was written about this feature. On the left-hand side, we see an example of PVC, and when looking at its spec, we see the first part of the volume populators feature, which is the data source ref. This allows us to point to another CR, in this case to a CR called example hello of kind hello, and we can see this custom resource on the right-hand side of the slide. And specifically, we see that in its spec section, there are two fields, the file name that is set to example.txt, and file contents that contains a hello world. When we try to use this PVC, for example, with the job that we see on the left-hand side, we can see at the bottom that it uses the example PVC that we saw before, and we also see that it runs a container that dumps the data of the example.txt file. We will see what we see on the right-hand side, that once the job is completed, and we look at its logs, we will see the content that we saw in the spec of the CR, in this case hello world. So this is from the functional point of view, right, what it does. But let's talk about how it's done. So when we post a PVC that refers to a CR, there is a controller that detects it, and this brings us to the second part of the volume populators feature, which is the library called lib volume populator that facilitates the implementation of such controllers. The controller detects the PVC, it reads the custom resource that is being referenced, and based on the data it takes from those two places, it creates a prime PVC, which is similar to the original PVC, and this PVC is used by a popular topod. The popular topod then writes the data into the PV that was allocated for us. Once that's done, the controller attaches the PV that we saw to the original PVC, and in our previous example, the populator wrote the content that we saw in the spec, and now that the PV is attached to the original PVC, the job was able to get access to that and dump it. In Forklift, we implemented two kinds of populators, one for Ovid and the other one for OpenStack. We read the data from there and write it to the PV, and if this picture looks familiar to you, we had a similar design in older versions of Forklift, and also if you use Qvert along with data volumes that refer to remote sources, in both cases we use CDI, and with CDI we get a similar picture. The only difference is that the pod is named differently, it's called importer pod instead of popular pod, and the sources are a bit different, we don't have support for OpenStack, but we have for others in this case I added the vSphere to the slide. So it looks similar, but let's talk a bit more about CDI. So CDI was the solution that was introduced in Qvert before volume populators were implemented to import data from remote sources. It is based on the data volume CRD, we can see a CR of data volume on the right-hand side, and we can see in its spec section that it connects to Ovid because of the image IO section, and yes, as I said, CDI also supports other sources, you can find them in the documentation of CDI. Before Forklift 2.4 we used CDI in all migration flows, and in Forklift 2.4 the latest version it was partially implemented with volume populators. So I said it looks the same from the diagram that we saw, but looking more into the details we see that there is a trade-off between the two. In CDI there is an extension to Kubernetes with the data volume CRD, while with volume populators we get a solution that is integrated within Kubernetes. This data source ref section exists in all PVCs since Kubernetes 1.24. We talked about the naming of the ports that is different, another point is multi-stage transfers, which allows us to do what we call in Forklift 2.4 migration, the ability to copy snapshots of the data periodically, so that when the user asks CAS, we can shut down the VM, copy just the remaining of the data that has not been copied yet, and start the workload on the target environment, and that way we minimize the downtime. So we have no support for this with volume populators. Next, when we want to add another source to CDI, we basically need to extend the codebase with more logic that is written in Go to support this new source, while with volume populators we have a pluggable design. We can add a plugin that runs a popular port that basically can run anything inside, regardless of the language that it is written to, and it even allows us to leverage native tools that are provided by the source system. And lastly, CDI is integrated a solution in QVirt, and it is tailored to virtual disks. I will talk about it a bit later. While volume populators are not yet there, there is a walk in progress to integrate them into CDI, but it's not there, and there's no notion of virtual disks or virtual machines with volume populators. The reason we chose to go with volume populators is two-fold. One, we wanted to leverage the pluggable design, and second, we wanted the ability to run native tools inside the popular topod. When we planned forklift 2.4, we realized that in order to be able to deliver the feature of migration from open-stuck in time, we need to minimize the risk, and that also means to try to avoid changing CDI. So we looked at volume populators, but we also realized that it will take us some time until we get to the point that we can transfer data, because we first had to create inventory, and that took time. So the way we approached it is by starting with the volume populator for Ovid that replaced some of the functionality that we had before with CDI, and let's see how that works. So there is a controller in forklift that posts three resources, a secret that holds the credentials and properties of the source provider, a CR called Ovid volume populator, and the PVC that will hold the content of the virtual disk. Then another controller, populator controller that is based on the lib volume populator library, creates the prime PVC and the populator pod as we saw before, and inside the pod we run the command that we see below that uses Ovid IMG, a native tool that is provided by Ovid to interact with Ovid IMGIO to upload and download disks, and then we write them into the PV. The open stack volume populator works in a similar way. This time we post a CR that is called open stack volume populator, again the same controller creates the populator pod and the prime PVC, and this time the populator pod runs code that is written in Go, it uses the golfer cloud library in order to connect to open stack, get the data from there, and dump it to the PV. With these populators we were able, let's start with Ovid. With Ovid we were able to accelerate the time of the data transfers, in our testing in some cases we also reduced the execution time by half, so it was significant. We were able to introduce another feature of transferring the data over an insecure channel more easily because we didn't need to change CDI, and we were able to introduce an alternative implementation to the one we had with CDI, and now we can deprecate the relevant code in CDI. Even more importantly, when it comes to open stack we were able to introduce this new functionality of migration from open stack in time without changing CDI as I said, and really it works very similar to that functionality of Ovid. Some challenges and insights that we saw in the process, so the first one was that we initially planned to use CDI to allocate the PVS. We planned to create a data volume with a source that is set to blank, and then the populator pod would kick in and copy the data to the dv that we get, and that didn't quite work. I mean the populator pod started and wrote the data into the PV, but afterwards the data was written by the import report because it tried to give us a blank volume, right, like we asked for with the source equals to blank, so that didn't quite work, and we had to skip using data volume, but instead to post the PVS directly. The problem with this approach is that we lost the logic that we had in CDI for this. First, CDI knows how to pick the right properties for the virtual disk in terms of volume mode, volume access modes, and we had to duplicate it, and also when the PV is allocated on a file system, the file system itself has an overhead, so this gives less space for the virtual disk, so we need to take this into consideration and allocate more data than we need for the virtual disk itself, and this is also logic that we had to add to the core code of Forklift. Next, let's talk about progress reporting. VM migrations can take time, even couple of hours, so it is important for us to reflect the progress of this operation. We tried several things. Our first approach was to make the populator pod update the CR, put the progress on the CR itself. The problem with this approach is that it required another service account in order to let the populator do that, and that simplified, sorry, complicated things for us. Another approach we tried is to push the updates from the populator pod to the populator controller. The problem with this approach was that we had to propagate the endpoint that the data will be sent to the populator pod, and that also complicated things for us and didn't quite work in all scenarios. So we didn't use this one, and we tried another approach of pulling the data from the populator pod to the populator controller. This approach that is similar to what cdi does, and we actually implemented it, sorry, in a similar way, works by running a metrics server inside the populator pod, and then the populator controller pulls the data from there, the progress as metrics, and that worked well for us. Next is dynamic volume provisioning. So when we tested, when we played with our implementation, it worked nicely in our development environments. However, we got reports by QE that on testing environments migrations sometimes fail. When we looked into it, we saw that it happens when the target storage class doesn't support dynamic volume provisioning, but only statically provisioned volumes. So we figured out that it's an issue in the library, in the lib volume populator library, and we reached out to the maintainers there to explain and to try to find a solution, but we didn't find a solution we agree on, and since it's not that common to have such storage classes and since it's not that important for us, we rather chose to block using such storage classes when using flows that use populator pods. Next topic is the conversion of multi-volume disks. So a bit of a background. When we convert, and I said it that the conversion applies when we might get from a foreign hypervisor, in that case, we run virtv2v that allocates a local overlay volume that is backed by a remote volume. virtv2v then inspects the data of the disk and modifies it as we talked about earlier. When the disk is composed of a single volume, it works well with volume populators. However, when we have disks that are backed by several volumes, then this design breaks because volume populators expect one PV per populator pod, right, that all the data will be written to a single PV, and in this case, we need more than one, and we also need to specify the information about the source volumes, and this doesn't quite fit. So again, we reached out to the guys that work on the LibVolumePopulator library. We filed a bug, we had some discussion, but again, we didn't agree on a solution for that. So our idea here, we have a plan to change the code base of the library in order to support this, but we need to play with this and see how it goes. When it comes to migrations to remote clusters that I mentioned before, in this case, we need to post the CR on the target cluster, right, that we migrate to, and the populator pod also runs there, so we need to track the progress that it reports. This raises two questions. One, who should deploy the CRDs to that remote cluster, and second, how do we get the progress from there? Which component should do it? Our first attempt was to let a controller that already runs in forklift on the source cluster to create the CRDs and also to get the data, but it appeared to be more complicated than we expected because usually CRDs are deployed by operators, right, and we didn't want to duplicate this logic, and also when it comes to progress reporting, propagating them to another cluster can be challenging. So we chose to avoid this one and rather try to delegate this to CDI, so that CDI will, that runs on the target remote cluster, will deploy the CRDs and also track the progress there. And that's a work in progress. We work with the CDI team on this, and I hope that it will get in forklift 2.5. And lastly, some more adaptations that we had to do to the LibVolume Populators Library. Usually we try to create as much as we can of the resources that are per migration on the target namespace. That is debugging, and it also allows us to set owner references to improve the cleanup code. The library doesn't do it, so we needed to patch it. We got feedback from the guys that work on the library that they chose not to do it because they wanted to hide the prime PVCs and the popular torpods from users, but it's okay for us, right? We actually do want it. So that's something that we did. Another point is limiting the number of restarts of the popular torpods. We don't wish it to keep retrying, right? We want, at some point, to say that the migration fails and to be able to inspect the logs more easily, so we limited the amounts of restarts of the popular torpods to three. Also related to the cleanup, we want to correlate resources that are per migration with the migration itself, and again, we needed to change the library in order to set this label. And last but not least, when it comes to VM migration, it might be important for users to select the network that the transfer will be done on. It can be because we want to isolate this from the data that is used by the VMs, by the other VMs. It might be because we want to use a better network, faster network. Either way, propagating the transfer to the popular pod also required modifications. So to sum up, when we look back at the original question of whether volume populators fit also for virtual disks, I would say that the answer is generally yes. It allows us to implement the original things that we planned, the basic functionality, and we also work with the CDI team to integrate our populators and volume populators in general into CDI. So that's a good indication that the basics are there and it fits. But when it comes specifically to VM migration, in that case we see that it's not completely fit. We see this by the amount of changes that we had to do to the lib volume populator controller and basically that made us eventually fork it and add the code to our code base. And also it was challenging to get the more advanced functionality, the remote migrations that we said, the war migrations, and also the conversion of multi-volume disks. So those are all things that we think they are possible, but they require an additional effort and we work on those things these days. And yeah, that's all from my side. Now your turn. Questions here, the PV itself. I will repeat the question. The question is how come that when the prime PVC allocates the PV and then we say that at some point after the write is done, then the PV is reattached, attached to the original PVC, how come that the PV is not removed in the process? So this is part of the logic of the lib volume populator itself. I'm not that familiar with the details, but basically it catches this point and it just changed the reference. So it's done before something is lost. It kicks in, do this change, and then we get it attached to the original PVC. Any other question? Okay, if not then thank you, everyone.