 So, let's start. So, the agenda for our time journey today, we're going to see the APIs that already supported for backup in Ovid. Then we'll present to you, do a little jump to the future and see the incremental backup API. Then we'll dive under the Lorian hood and see how it works. Then we'll see how we can use it using the engine and image.io backup API. So, how many of you used Ovid or using it today? Can you please raise your hands? Nice. Thank you. See quite a people. Thank you. So, this is Ovid from 10,000 meters above. Ovid contains a lot of sub-products and a lot of components, but this is the main component that's related to backup. We have the Ovid engine application, the management, we have the hypervisors that runs the VMs, and we have the VMs disks. Also, we have an external backup application that wants to backup the VM disks and communicate with the engine itself. So, let's review the old APIs. The old backup APIs, let's take, jump back to the past. So, the oldest API that Ovid provides is the backup appliance. The year is 2014, even before I was born. The backup appliance run inside the VM, and let's see how it can be done. You should take a snapshot for a VM. Then you need to attach this disk snapshot to a backup VM, to a proxy VM, basically, which is different from the original VM you want to backup. Then inside this backup VM, you will copy the entire disk, detach the disk snapshot from the backup VM, and delete the snapshot. We move forward a little bit in time. The image transfer API. So, the year is 2017. Now you can download and upload snapshots using the image I owe that was presented in the Ovid 4.0, and you'll do this by taking your snapshot for the VM, downloading the snapshot using image I owe in Qco 2 format, and delete the older snapshot. To restore, you will prepare the disk on the backup application, rebase the snapshots. For example, upload the disk or the chain of snapshot in raw or Qco format, and then you'll create a VM from the uploaded disk. So, the new incremental backup API. Fasten your seatbelt. We're going to jump to the future. The year is next year. It's not ready yet. We're working on it. Now we're doing incremental backup using change block tracking and NBD protocol. It will be available in Ovid 4.0 as a tech preview using upstream libvirt and QMO. In the next version, I hope that we'll have backup application using it. So, how do we do full backup using this API? We start the backup. We provide a simple API to start the backup. And then we download the disk using the same APIs that we had before, but with some improvements. For example, now we can download raw disk data, regardless of the disk format. And we stop the backup. And there was no step four. So, we provide a very simple API and we do the Everlifting. Actually, libvirt is doing Everlifting and QMO. How do we do incremental backup? In the same way, we start incremental backup using the same API. We specify the last backup that we want to backup to, we want to see the changes since it's dead backup. And then we can download the change blocks since dead backup. Again, using raw format. We don't care about the actual format of the disk. And we stop the backup. So, this is basically the, from the user point of view, it's very simple. And when you want to restore the backup application, we have to prepare the disk for restore. It can be an image. It can be done by streaming from the backup media. And again, we can push the raw disk data back to storage. We don't care about the disk format. And finally, we attach a disk to the UVM. So, what are the advantages of the new way? First, we can copy change blocks instead of all the disk. This speeds up incremental backup or actually enable it. And next, we have extents API reporting where do we have zeros and dirty extents on storage. So, this can speed up full backup. We don't need to copy the entire disk, only the data. Next, there's no snapshot. We don't need to create a snapshot. We don't need to release a snapshot. Snapshots are very complicated. And this avoids the trouble with snapshots. The next advantage is raw guest data. We always get raw data. We don't care about the disk format. And when you've got raw data, you can convert it to another format. It's easy. It's not easy to get Q2 format and convert it, because Q2 was not built for streaming. And finally, we have improved Mimigayo client that can upload and download any format for you from any image to any disk or from any disk to any image. The limitation of this solution, we support only Q2 disks. So, if you have all disks, we can do full backup, but it cannot participate in incremental backup. You will have to convert it. We cannot backup non-running VM yet. We should be able to do it in the future. And the extents API reporting the zero extents or dirty extents is not useful for a pralocator disk. We just get one extent. We can not do anything about it. Okay, so we have incremental backup. Let's see how it's done under the Lorian hood. So, in the base of all that, we have checkpoints. Checkpoints are, we can imagine it as lightweight snapshots. When we are creating a backup, a checkpoint is created in Leveret. Each checkpoint actually marks us which disks participate in a specific backup. And the checkpoint, of course, marks which backup is related to. So, in this example, you can see that the user created a backup and a checkpoint was created in Leveret site. Moving forward, the day after, the user created another backup. So, another checkpoint is created, but now that checkpoint has a parent, the first checkpoint. So, we are creating a chain of checkpoints. The same disk participates in the checkpoint also. Again, the user creates the third checkpoint. And now we have a full chain of checkpoints. And the checkpoints persisted on the host as long as the VM is alive. If the VM shut down or crashes, the checkpoint information gone. So, the over-tension will persist all that information in this database. And when the VM goes up again, we need to redefine the checkpoints. So, Leveret will know which checkpoint was. But checkpoints only give us metadata about the backup. What really tracked the difference between the bits. So, this is done by the bitmaps that the bitmaps created in QMO. So, when we are creating a backup, a checkpoint is created. And for each checkpoint, we are creating a bitmap. Okay? In this example, you can see that the user created the checkpoint for SDA, the disk SDA. And the disk no writes is done to the disk. The disk is clean. Since the backup was taken. So, the bitmap of the checkpoint is still clean. Each bit in the bitmap will mark each cluster in the disk got dirty. For example, we move onward in time. And you can see that the second cluster got dirty. So, the bit that marks us if the second cluster is dirty or not, was turned on. Moving on, the guest writes another data to the fourth and the fifth cluster. We can see that even that the fourth cluster not changed fully, we are still marked as dirty. Now, the user finished the first backup and created another backup. So, the first checkpoint, the first bitmap of that checkpoint is now deactivated. And the state is preserved. And since we created the second backup, we can see that there's no changes and the bitmap is still clean. So, now that we have the bitmaps, we can know which extent exactly got dirty during that backup. And when the user will start to backup the VM for that checkpoint, you will get only the dirty extents. For example, the first cluster is still clean. No changes was made. And the second cluster got dirty. So, the dirty will be true for the second cluster. Okay. So, we know it works under the hood. Let's see how we can use it using over it Python SDK. So, in order to start the full backup, you just need to fetch the backup services of a specific VM. Okay. And just use the add method and providing disks to the backup. Okay. The disk that you provided will be participated in the backup. In this example, this is a full backup. And this is the first stage. We will wait until the backup is ready for the backup status. And then we'll get the backup object. So, as we mentioned before, when we're creating a backup, a checkpoint is created under the hood. So, we need to fetch the checkpoint ID in order to use it later in an incremental backup. So, now we have the checkpoint ID. We have the backup ready. We can start an image transfer. This is an image.io. So, we will provide the disk ID that we want to download, the direction. And because this is a disk that participates in a backup, we need to specify the backup ID. And the disk format will be raw. We're supporting raw guest data. We wait for the transfer to be ready. And we fetch the transfer URL once the transfer is ready for download. Using the transfer URL, we can actually download the disk. And we will elaborate it more later on. So, we download the disk. We have the disk ready. We can finalize the transfer and finalize the backup. We have a full backup on our environment. So, now we can start an incremental backup. So, it's basically the same. But for incremental backup, the user must specify each disk will support the incremental backup. And those disks must be in QCO2 format. So, we will filter the disks and we'll fetch only the disk that supports incremental backup. Once we have the disk, we can start an incremental backup and specify the from checkpoint ID. So, we create the chain of checkpoints. And the disk will be preserved from that checkpoint. So, how do we download the backups? Basically, we can do a simple get request and get the entire disk. It works, but it's not efficient. We don't want to use this. So, the real way to do backup is first to send an optional quest to the server. And this will tell us the feature supported by this server. Or actually for this source on the server. So, here we see that this server and for this image, we can use the extents feature and we can use zero and flash used for upload. And we can also see that we can optimize the connection using unique socket if you are running on the same machine. And we can use multiple connections to speed up the transfer. Also a new feature. So, we want to get to backup 100 gigabytes image. How many gigabytes do we need? Well, we don't know because, but we don't know that we care only about the data. We don't care about the zeros. So, we can use the new extents API. We send a get request to the extents or source of the transfer URL. And we can specify zero to get the zero extents. And we get a list, just on a quest with a list of extents. Similar to what we get in QM image map, basically. We can see that the first extents is zero. So, we can skip it. We don't care about it. And the second extents is not zero. So, we want to download it. And now we saved a huge amount of data and time and storage space when doing full backup. And what about incremental backup? In the same way, we want to know where the dirty blocks. We have seen that before. We can again use the extents API. But now we specify context dirty to get the dirty extents. And again, we get a list of extents. And we can see that the first extents is dirty. So, we want to download it. It's one cluster. And then we have a lot of clean data that we don't care about. And we can skip it. So, again, we save a lot of time and bandwidth and storage space. So, how do we get the extents? We know the locations, you know, the offsets. We use HTTP range request. Standard HTTP. Just send a get request with a range. And we get this data. And we need, of course, to make one request per extents. So, how do we speed up the upload in the same way when you want to restore this? Again, we care only about the data. So, we can send the data using put request. This is not new. It was, it exists a lot of time. It seems obvious for all. Or even before, we specify the content range for the offset and the size with the content length. And again, we need to send one request per data extents. And what about the zeros? Can we ignore the zeros? Well, sometimes we can ignore them. When we create a new image on file storage, the image is always empty. We don't need to zero anything. But on block storage, the contents is undefined usually. So, we want to zero this range of storage, but we don't want to send zeros on the wire, which is not efficient. So, we have zero API implemented using patch. We send a patch request with JSON describing how we want to patch the image, which is kind of standard. And we can specify zero operation of certain size, and it will zero this range of storage efficiently using NBD or using our far back end. And, of course, when we finish the upload, we send all the data with zero, everything we need to zero. Are we done yet? Well, maybe. We really want to send a flash request, which can also be sent during the upload if we need. And this will basically call fsync under the hood. And when this returns, the data should be safe on storage. So, it is too complicated, maybe. So, we have a better solution, but this is actually what the cool kids are using now. In this example, these kids bought a hoverboard kit, and they only had to add wheels to its work. So, in the same way, the client library let you build your own backup solution. You just need to add the wheels, which may be complicated. So, what you can do with a client, you can build a backup solution on it. You can use it as a reference implementation to learn how to access imagery in the correct way, in the efficient way. And you can do simple backups using, we have an example backup script that you can use to backup a single image or store images as a testing or simple solution. How does it work? You basically interact with the client, and it does everything for you. It will run QMNBD, opening the image you want to upload, or the image you want to download into. Then using NBD client, we will get the extents from the image, send them to the image of the demon, which on the other side also use QMNBD. So, basically it's QMNBD all the way down. And for download, on the other side, we can have QMNBD, and we will use NBD protocol to get the extents, to get the data and write it to the destination file. Again, using QMNBD and NBD client. And this is example how to use it for download. One call, you import the client, do one call with the transfer URL you get from the image transfer from engine, and the name of the file. The default is QCAPTU format, because it's the best way to backup data, and it's compatible with both full and incremental backups, but you can change it. For incremental backup, you only need to specify incremental equal to, and this will go and get the dirty extents for you and download them. In this case, only QCAPTU format is supported. And for a store, if you use this simple way to backup, you can take your incremental backup images, rebase them on one or top of another until the full backup. And after the rebase, you have a complete chain. You can just upload this chain. Basically, mostly like your image convert, but using current with the underneath. So this will upload the entire chain, actually get the data extents, upload them, and zero, whatever is not data. So time for the demo. Okay, so now we'll show you a quick demo that uses this client solution, the new image of your client, and the Python SDK. So in this example, we have a VM, if we do a 30 VM, and it has a disk, we just mark it as an enable incremental backup. And we can start to perform a full backup in this case. We just open a console to create a directory to illustrate that this directory will be part of the backup. And we'll create another one later to see that the other directory that was created after the backup was started is not participated in the backup. So we start the script that we provided you using the Python SDK and the new image of your client. The backup was started for the VM. We are creating another backup after the backup was started, another directory after the backup was started, the after directory. We can see that the backup already ready. So we are waiting for the image transfer. We created it and it's ready. So we can fetch after that the transfer URL from this URL. We can download the disks. It will be downloaded to the following path. And as soon as the disk was downloaded, six gigabytes of disk, we have the backup. As I mentioned before, this is a full backup. We just, we can see it and we create another QCOW layer on top of it. And we can start QMO on that QCOW layer. So we have a Fedora 30 VM and we have only the before directory and not the after. Okay. For more information, you have links here to dive into this new feature and also a Libert and QMO implementation. Thank you for listening. Any questions? We are providing, the question was that if we need, we are need to orchestrate the backups or the CIS admin will have to do it. So all we provide the API, the CIS admin can do whatever he wants with that API. He can orchestrate the backup each day, hour, month, up to him. Yes, basically we provide an API, the backup vendor should provide the real system behind it, backup. It's more complicated. More questions? What you will see in the demo is working now in master. So it should be available in the next OVR44 build. We are working on enabling the mental backup and managing checkpoints. And this should be completed for a future for 4 build and should be used for vendors to develop applications for backup. And probably we will finish in the next version. Yes. The fact that the backup is done using Libert and when the VM is running it doesn't leave it. So we need to introduce a new special mode of running a VM, a new backup mode. When you start a VM, post and then we can use the old APIs to do anything we need. And we don't have it yet. More questions? Okay, great. I guess that's it. Thank you very much.