 So, you need to connect the phone to the laptop, and then it's important to have a connection? Yes. To be automatic? Yes, you don't need to. Just use the phone. Yes, that's all right. Oh, nice. Then you have to use the phone to connect to the laptop. Oh, great. So, you have to use the phone to connect to the laptop. What's the problem with the picture? It's not very clear, if you look from the back, it's a little fuzzy. I can run a second projector, but with the focus I cannot do anything. This is the best projector. Can I run the second projector? I don't think we need the second one. The second one is more crisp. I think it's better because it's smaller, but sharper. We can try. Hopefully it doesn't break anything. If you enter the room, please try to close the door quietly so that we don't disturb the presenter. And please let me know in the short time who's going to be there in the talk. Please welcome. Welcome to this. I want to talk about the second edition of the conference. We'll talk about the short introduction to Loviat and all the components. Then we'll look at the difference in the tradition of the storage. We'll talk about few issues that we have to support. The future of Loviat. How many people are using Loviat or know about Loviat? I should know, I guess. I guess some are about SEPT. So SEPT is scalable, highly reliable, software-defined storage. Loviat was just only the locks for now. To use SEPT we are using Cinder. Because we had two goals when we wanted to accept support. We also wanted to support more storage in the future. And Cinder was to access to a lot of storage that we are not using now. We may be using the next version. And we support only SEPT back in Cinder. So we don't care what others thought in this version. To talk with Cinder, we'll be using OpenStack Java SDK. Cinder supports the project. And I will focus mostly on the SEPT integration. And less on the Cinder integration. Because this project was kind of a similar integration. I will focus on the SEPT side. I will talk more about the Cinder integration. So of the traditional storage, what is the traditional storage? This is a simplified view of the architecture related to storage. We have the engine controlling all the hosts. We have regular hosts like this. And the special host called SPM. As you know. The SPM is the only host that can change the shell storage. For example, if we have lock storage, the SPM will go and create volumes on this shell storage. While other hosts will consume this storage. For example, this host will connect to the iStudy server. And enable the MVs for the VMs. Or do mounts to exhaust the shell storage for the VMs. But only this host can change stuff. The SPM does a lot of complex stuff. And if the Cinder puts a fail, we are working on it. Hopefully we will do something. What is lock storage over the visual storage? We support iScazio's panel channel. And actually this is a video. The volume over the volume is the VG. Inside the VG we have special LVs for metadata. Like this one. We have special LVs for locking, like LVs and lcents. Inbox and outbox for messaging. Host can send messages to the SPM. And here we can see regular volume. And two other volumes that we use for VG. This is basically a block storage. What is block volume? On the bottom we have some SCSI devices. Which we use only via multiples. So in this case we have two passes to the same device. Both of these devices are actually the same remote device. This is the remote device that we can access via two passes. And we have another remote pass field. Both of them belong to VG. This is a block storage. Inside the block storage we have volumes using LVM. So we have one LV. And on top of the LV we created a QQ image. The last part is what the LVM is consuming. The problem with block storage is that we cannot create any amount of LVM. It started to become very slow. LVM was not designed to create huge amount of LVM. It's complex and how to manage. We have to do locking to do any changes in the VG. Only the SPM can do the changes. We have a complex way to roll back changes. There's a problem with timeouts. Static timeouts and multiple timeouts. It should be much better on PL-7.2. And of course we have the quality of service for storage operation. We are using the IO class. IO noise to run our operations. But we are not using the CFQ schedule. Actually we don't have any quality of service. The CFQ schedule is much slower. That is something that we should be working on. OK, how do we do the same provisioning? VPSL is not on the data class. We provide the device to the VM. And the VM is writing data. And we are promising about the event data. So we monitor the gives and what about the ideas. When we find that the disk is too full, we extend it. We extend it via the SPM. So if I'm a host running the VM, and the disk becomes too full, we send the message to the SPM. The SPM extends the disk, and we send the message to the disk from the standard. When operation is finished, we resolve the VM. It was too slow on the VM board. And we repeat. So this kind of looks... We're not going to do it, but this is how we do it in front of us. But of course it's limited. And because we are doing polling, we cannot scan to each amount of disks. It becomes costly. It's very complex. It's slow. The VM we post before we send the disk. And of course we depend on the SPM, which is a similar kind of failure. Okay, let's move on to file storage. This is Maxi Implea. We support a lot of shared file systems. File storage domain is just a directory, a meta directory. Here we see a file... a mounted file storage domain. And inside the directory we have metadata files and images. And file volume is just a couple of files inside the image directory. Here you can see the storage domain, image directory and image directory. And inside it we have single volume. This is the data of the volume, list file and metadata of the volume. And if you have a lot of volumes, you'll have a lot of files. Sync provisioning is easy. The file just go and leave it. Problems. We have operators of the file system. The VM wants to block the files, and we put it on the file system, so there is author if not leave it. We have bigger issue of timeouts, because NFS can block for minutes, and it's not accessible. So we run all the storage operation. We have 200 and set the same amount. In another process, we are using the IR process, which does all the IR for us. For example, to the directory in the list of files. And of course, we don't have any way to define quality of service. For example, maybe for locking, provide better quality of service, or for other storage operation, best quality of service, so we don't disturb the VMs. And of course reliability. It depends on the storage server. If you have one disk with NFS, and if this is gone, then that is one disk. If you have a smart server, that's the application or something, you're good, but it's not. Okay, how do we do snapshots? All snapshots doesn't have the same volume or same volume, they're the same way. We have a QCAD chain. The VM talks with the top layer, which is using QCAD format. And it has a backing file. The backing file can be all format with QCAD. And let's see an example. We start with one volume. We create the first snapshot. So we create new volume, and then we switch the VM to the new volume using the libger, if it's live snapshot. And the new volume backing file is the old one. Okay, so this is, I'll take a snapshot, we can take more snapshots. We can see that we have the volume of scale because it's meant to point out the page. So the issue is our performance. For a second, if QAMO wants to read the data, it will try to read it from this volume. And if it's not there, it will try this volume, and this volume, this volume. So it makes it read, sorry. Rights should not be affected too much because all the rights are going through the active volume. Again, the problem of scale, because we don't have a lot of scales, we're reaching the maximum number of scales that should be on top of it. So people, maybe, should ignore sort of domain so they don't have too many scales on the same sort of domain. And of course, if this parameter is too much, we have to keep the QAMO files in sync and the ADS, but it's not a lot of fun. Like in Edge. We know about it, it's really complicated. You can check Adam talking about it. Let's move now to something much more simple. Set storage. Set storage works completely different. On the host, VDSM is not accessing the storage. It's not organizing how to set storage. Only QAMO knows about the storage. Only QAMO can see the storage very, very deep. Engine uses single to create volumes at the same shop, to create volumes at the same shop. And single is... The set backend in single is doing the regression in set. So one important example, there was no SCM in this diagram because we don't care about the SCM in set. We don't use it. So this is nice improvement. Storage domain. There is no set storage domain on the host. On the engine side, there is a single provider which supports only set for now. But all the operations are done there. Here is the single. What is set volume? On the host, there is nothing about set volume. We set a piece of XML that we pass to the build. We get all the details from the engine that we get for the signal. For example, the pull and volume name, the host, the monitor, on this volume, and the authentication details. Same provisioning, the general thing to do, such as everything, snapshots. Nothing on the host, such as snapshots. The host does not know anything about the snapshots. You'll see that at the end you create snapshots. When we take a snapshot, we have a VM using the set volume. You can take snapshots or another snapshot. There is no tendency between the snapshots. The VM does not know anything about the snapshots. There is no way... I mean, there may be a problem of scale on the set side. I don't know anything about other real snapshots. But it's not our problem. So, for our site, this is the state. Live merge, there is no live merge. It's just a little snapshot on the signal and it does. The VM does not know about it. So, what are the benefits? Honey available storage, scale is very constructive. Everything is afforded to self. And we don't have to manage anything on the host. It's very little. Problem, it's not finished yet. There are some features that are not implemented. And where can we see that it can be tough? Because we see that there is a bit of an idea about storage. What are the options? So, we have to fight it sometimes. Self may be too complicated for the whole scenario. But it's important because it's only one set. Probably one set of view points. And scale is the problem. And in a way, the signal is our newest VM because now it depends on the signal. So, if the signal is down, we cannot do any source operation. Okay, now let's talk about how do we connect securely to self volumes. The standard way to do self-advocation is to create a keyring file and deploy this file on the host. I think the self-advocation is very useful for you. But you need something more secure. We don't want to deploy a piece like this on the host. So, what we are using is liquid secrets. You can register a secret in Lizard and we are using the general secrets and private secrets. It means that the secret is defined on the memory in Lizard. And if you get three stars, the secret is gone. So, it's never persistent on this. And private means that you cannot get the secret out on Lizard. Lizard will give it away to KMM, but normally you can take it out on Lizard. But of course, you cannot define this on Lizard and forget about it. Because when you get three stars, you use the secret. So, we need to manage this. So, what we do is we manage now the self-advocation key. You can add a set piece to engine. And engine will register the secrets for you along with the secret. And then the secrets will be out on Lizard. To allow this, we add new APIs. There is a register-secret API, new video center, and unregister secrets to multiple secrets. And engine will register secrets when host is activated or storage is activated or when we reconnect to host. This is the same way that engine is managing the storage connections. So, in the same way that we make sure that the host can talk to storage, we also make sure that the host has all the relevant secrets. And when hosting is put to maintenance or sold to municipal maintenance, you need to remove all the secrets that are not here. And also, be the same with your secrets when it's strapped or shut down, because you don't want to be around secrets that don't exist. Okay. Life's natural. Sometimes you want to capture the moment. To do this, we need consistent snapshots. We don't want to keep snapshots that are not useful to store later. So, to have consistent snapshots, we need to flip the guest's participants. Then we want to take the snapshot. And then we want to tell the guest for a system. And now we know that we have a good snapshot This, of course, requires the guest agent, the QMU guest agent. This is also for traditional storage, there is no difference. Although we do it with old storage, engine creates a new disk, and then we take a snapshot. Bigger sandwich, we have split bills to take the snapshot. We need to squeeze the guest's participants internally inside the snapshot valve of Libre. And finally, Libre will switch the VM to the new disk. Now, this is, we're not well processed because this phase should be here. So, we cannot use the old port for this. We have to make it different. And now, engine controls the time flow. So, it will not depend on Libre freezing. And we make sure that we freeze the guest file system before we created this snapshot. Because at the point we created the snapshot, the VM is not writing for the snapshot. It will write to the next one. So, we want to make the file system consistent and only then we have to snap it. So, we added the new PI to VDSM, freeze API, that is going to be a Libre FS freeze. And to the snapshot API, we added force and flag. So, when we take a snapshot we can tell VDSM if the VM is forced and or not. And finally, we added the tau API, again using Libre API. So, now the flow is like this. Engine first freeze the VM, create this snapshot via single, take the snapshot in VDSM, which may use, the VM may have set this in original VDSM and then you want to take the new snapshot or header. And when you finish, we target the VM via VDSM. And when we are taking a snapshot without set VDSM, we just tell VDSM take a snapshot with force and force and VDSM do the same. Any questions? If you make the whole thing agnostic and do the freeing and throwing in QA mode per se, you suffer from time loss and the guest operating system, wouldn't you? Yes, but there is no other way. You must use the guest agent. Yeah, that's what we do now because of such implications I just mentioned, was there some other design flow going down that path in order to avoid guest agent presence or more of the different guest operating systems? If you open the guest agent, we want that to consist of snapshots. You cannot do anything about it. If you want to consist of snapshots, you have to install the guest agent. Oh, we know you can use the VDSM. VDSM is the limitation of sections. I can answer the problem is that the guest does not just pass up, it does not give you memory and it doesn't drop down. Right, I know. It was, and I presume that we have issues with that approach with respect to timeouts and the guest operating system, that you essentially free the QA mode as a central place, presuming that you could do the whole cycle quickly enough that the guest doesn't hit a timeout an IO timeout. Of course, you wouldn't have the perfect, you know, say, content because the guest would have data and memories, but he'd probably suffer from the timeouts anyway because he can't rely on the particular guest behaving, you know, with respect to timeouts. If you went through it quick enough, you could actually avoid all these, you know, guest agents, like I said. Yeah, the problem is that the guest has the workflow and the crisis that the guest has to crash. So yes, it doesn't make possible for the guest to crash. Essentially, your snapshot will contain the half-written contents and your snapshot will receive the rest, it might be free, but that's just an idea because I was afraid of the timeout implications anyway. If your switch takes too long, right? It will continue to the edge, like it's sinking there. Okay, so there are a few future directions. First, a missing picture that we don't have yet. This is not a completely these are the most important things. Move this. It won't support moving this from safe to safe. From safe to another storage and from another store to safe. Should be impossible. I hope using liberal media without involving the mountain or such a disk. Life-sort negation. It's the same I do this, but I do it live, authenticated. Here, hopefully, we will support it to switch from a lot of disks in my projects. If it does, we can do it. Disaster recovery comes here. We obviously persist all the evidence in storage, but in such we do not create the obvious store volumes. We probably can do, but we don't know it now. And we cannot connect to such a storage domain with the cover of the evidence from the storage domain. Like we can in a stash or in a test or any other storage. Monitoring is an important point. In the regular storage, we have a lot of monitoring of the power of the storage domain. The monitoring is not perfect for me, but we can know what happens with this storage and if the storage is not available, engine knows not to try themselves here on this storage. We set, we don't have any monitoring. We open the whole site and we don't arrange this site. Everything is always up and operation will fail. Secret management we want to do the secret management that I that you see before is still complex. We don't want to manage secret or host. What we want to actually do is just send the secret to the host that needs to run again or needs to not to ingrate the VM to the source. Define the secret that we are leaving on the source start the VM or finish the integration and we are done with the secret. All the management that we are doing is just a waste of time and effort. So hopefully this is something that we can do in the next version. Of course migrating and other because we need to ingrate the VM we need to make sure that the same XML that the VM is using in welcome destination. For the secret that we should define should use the same UID but this is the UID that the this is what we look for when we try to connect to a single signal is having different views on storage for example like to decide for you what kind of storage you want so if you want to set storage signal you can decide why don't you take a VM storage or a data storage or NFS and we are not prepared to handle this we are not know why we don't use it we are not exactly our storage but sender will let us configure it so we must depend on the specific configuration and we can be at all with this specific configuration so one idea was to provide our own signal and control this sender instance with our agent so we can control the configuration and we can bypass sender a different view of the world and we can also hide sender completely behind the engine line which is something that customers really like simply why and simplify the line instead of this is one idea I'm not sure if we are going to do it because there is another idea right of going to talk sender we don't need a middleman we can use people with ease it's very easy to create volumes and snapshot and everything even if it's byte on byte things it's really great but the problem is that we want all the nice drive of sender support so we will have to think about this and if we think a few years ahead maybe we should go on a business storage it's so complex how to maintain easy to break so why do we need it if this storage is good because we can support the stupid disk we don't need any smart servers on the server side but we can use smart server like this this is the direction that the world is going smart servers not stupid disk I'm not sure that we will have this in case questions the sender service where does it run how many of those what the sender service where does it run where is it something that you should deploy in your setup you can have what you want you have to install you have to install a sender and whatever other services of the sender and the wires same like said you are responsible for installing sender and some such and we consume it or we may be not good at this like the high stack server I can go to install nice sender or infest like I said the sender so the open sender someone has to talk you don't know what the configuration is you don't know what the version is you don't know what the version is you don't know what the version is you don't know what the sender applies to sender or tail or sender that is the tool that we make it easier than you install your own sender sender you are thinking about making any that you want to install a little bit separate no, in the traditional story I think you can have content inside of an awesome sender sender sender sender sender back in the days that's kind of different places and ecosystem back as partners to give us all the space but it's not a black feature that we have out in the world so it's a little bit different you can also do it yourself because the rest of the time the question is the same so what is the difference between the two if you need to install a bit that's right you can do it with the same feature you can make a feature you can do it with the same feature where is the sender where is the sender what is the difference with the sender sender sender sender sender sender Everything alright? Great.