 Hello, I'm Dennis. I work in Red Hat. I can't put that thing in my pocket for some reason And I develop overt the overt I Hope it works. Yes, it works and over it works, too So over it is an enterprise grade virtualization management system hundreds of hosts thousands of VMs External API, a lot of storage interconnections, integrates with all the stuff you can imagine The guy before me just told you, okay, we integrate with Kubernetes and it's called kubi. And Before I continue, I would like to tell a little bit more and how overt is made. It's not that important to really take it to tales like, oh, we have some stuff there But we need just one single thing and I would like to mention it So we can think about overt as a system made of three layers. The lower one The best one is a storage subsystem Here we store all your VM data your snapshots or VM team plates or ISO images everything stored here and the most interesting part that Storage subsystem is not a part of overt usually so we use external stories like storage area networks or NFS servers or something like that The only requirement by us is that we needed to be shared between all the hosts and hosts They make a second layer the host where you run your VMs We call them video Sam hosts for reason code video Sam is the name of our host agent So every host have a video Sam and video Sam and knows how to run your VM How to create a network how to attach a storage how to do stuff The only thing it doesn't know what how to do is what to do. He have no brain it's like a really stupid and straightforward application that only executes commands by The second layer the engine Huge and big and very smart application written in Java and running Guess what it runs inside of VM. Yeah, we have a VM controlling all other VMs and we have a special tool set for it so it's Fully redundant and highly available and engine controls everything Kenjin knows. Oh, this host have enough resources This host have enough RAM this host is overloaded. Oh, this guy wants to Exit that snapshot and have no rights It's all on the engine side I'm not going to talk about over too much today I mostly will be talking about storage pool and storage domain and other storage boring things But why I'm going to talk about that why we need to virtualization at all There are a lot of reasons for virtualization But one of the reasons and I know the guys who pay for all of that are loving it That one of the reasons is the resources Now in 21st century, I check that this morning the smallest hard drive for a server You can buy I mean spin hard drive about 300 gigabytes big Now imagine you would like to install say DNS server. So you have operating system 5 gigabytes probably you have DNS server with your data Some other stuff. So probably you're using just 10 gigabytes You have 290 gigabytes of a drive inside of your server. You paid for it and you are not using it and Your finance department is quite unhappy like why why we are not using 90 percent of our resource and they say, okay We can use virtualization we can have 10 virtual small service making small tasks and sharing the 300 gigabytes drive Now now you'll use almost 100% but this is really 100% Suddenly realize it's no No, no, no, no, no, you have 10 VMs and each VM Uses 30 gigabyte drive, but actually it writes like say 5 to 10 to 15 Some of them use in all some of them use just 5 gigabytes. Some of them don't use at all because you forgot to deploy them and Actually, you are not using your resources again. So then it's like, okay Maybe maybe I could squeeze it more. Maybe I could just give VMs a storage space that they really need and now you can put 25 VMs or 30, okay, 35 VMs or 40 VMs doesn't matter and then you say, okay But some of them they really need the 30 gigabytes but only two hours a day and Some other VM need them on the other hours a day and you say, okay I can put even more VM and Use some kind of time multiplexing so they don't feel be using the same data all the day So now you are overbooking and Everyone who's Made that in airline should start hitting it, but as I told you finance department loves it Do we have it in Norbert of course if we will not have it in Norbert, I will not be there and How we implemented in Norbert The first and really simple thing I believe everyone knows about it, but I have to start with it The same provisioning That's what I told you the idea simple the VM believes you have a really big drive say 50 gigabytes but it only uses 5 gigabytes of it and You only allocate 5 gigabytes of your storage domain So you have 50 gigabyte VM, but it only use 5 gigabytes and you can use 20 VMs of that size How it works You say oh come on this guy is telling us all those things. Yeah, I'm starting with all those things It will be more interesting later so Or weird works with two types of storage domains first one is a file storage domain so it can be NFS or any posix FS say Glastro FS and Here we use sparse files. I Don't believe anyone there doesn't know what a sparse file about I have to say that sparse file is just a file with holes If you have a big file and you haven't written anything here inside of that file yet It will not use any space on the drive as you may see you have holes and those holes are not on the drive But logically you have a big file and physically it's still small Second option Block device and block device is a little bit more complicated over started using block devices Earlier than a thin LVM arise So we had to invent our own approach and with block devices what we do when you create a big VM say 50 gigabytes We only allocate logical volume of one gigabyte and start continuously looking into it and checking. Oh How much data it's used One bite don't worry 10 bytes. Oh Eight hundred megabytes. So when you hit some threshold Or weird immediately Extends your volume to next gigabyte and start watching it again And when you hit the threshold guess what happens another one extension and So on and so on and until you fool the problem here with both Sim provision on files and same provision here on the LVM Is that we can't reclaim that space back? So if your VM created 500 gigabyte file for a second and then deleted it and actually allocate it and deleted that your stuff and your storage domain will be full You will have that five gigabyte allocation there forever. We can't reclaim it back sorry, especially with that that approach and We don't like it. The problem is that Usually over doesn't control the storage the storage as I told you earlier is external to us So we have some son. We have some NFS server. We don't control the whole storage stack Or do we ah, it's not working anymore that interesting thing Yeah, but mainly it still works Sometimes we control our storage stack completely from drives to VM Because we have a hyperconverged environment of overt and How do we do that? We need a shared storage We need a shared storage the only requirement by over storage domain in that each host in a Overt cluster have access to same storage and have a same view of that storage Can we do that? Of course, we can take Glaster Glaster Amazing Software-defined storage a file system running in user space It's like NFS, but much much much much much better and have a lot of features and faster and don't have NFS Sickness I would say what Glaster does actually you may have a lot of servers That's the typical commodity hardware. You don't need a Expensive hardware anymore just buy cheap servers and so on and on each server You can install a Glaster software and export some directories and Those exports we call them bricks can be combined to volumes and volume act at a it's like a Mountable file system so you can mount the volume and each client of that volume We'll have a same view for the file system. It will be shared same namespace all files will be available Doesn't matter how much do you have clients? You may have a lot of servers You may have a lot of volumes you may construct any system you like and The best part here that volume is not just a collection of bricks. It can be just a collection of bricks like you know Combining them together But no no no no no you may have more of them first of all you may have a replicated volume so Same data is copied several times and kept on several bricks and when some of the bricks or servers disappears Date is still here and you still have access for it You may have distributed replicated if I remember correctly. We have like nine or ten Flowers of volume. Am I right? We have a glass of developer here. That's why I'm asking nine or ten Flowers of volumes and yeah Relent on highly available scalable. It scales to petabytes. It scales to hundreds of servers it sounds like a great choice for overt and Yeah, we use it. So we use Glastor as a storage subsystem for overt and Glastor volumes are consumed by overt as a storage domains So now we can use same notes. I mean literally same hardware for running a storage For writing your VMs for hosting your engine VM. Yes quite special one VM because it controls everything and you don't need external Subsystem for storage. You don't need NFS. You don't need fiber chain We don't need anything now you have it on just cheap commodity hardware and it's scalable and it's highly available Thanks to Glastor and redundant and What is more important for us? We have a control of a storage stack from drive to VM Can we use it? Oh? With Glastor We use LVM and We use LVM sim pools. So when you deploy a hyperconversed environment of overt You deploy LVM on top of your hard drive and then you create a logical volume for engine We don't want engine to die suddenly. That's why we have pre-allocated engine volume and then you deploy two simple a simple and two volume for your data sharing that simple What it gives us You have 100 simple On top of it. You may have two Of size of 100 so in total 200 storage domains running some VMs and It's compressed and the best part here when you delete something on VM VM sends a trim command to Block layer and QVM is smart enough to use QVM and QM To pass the tree to trim command to Glastor and Glastor. Yeah, I will repeat it again. It's smart enough to pass it to LVM and guess what do LVM sim pool if you say hooray This block is not in use anymore. I can return it back to simple. So it works like a Sim provision but better. Well, I mean same provision with sparse files bar better now your space is reclaimed back so in my case you have two VMs of 50 gigabytes each using just five gigabytes on Top of 200 gigabytes storage domains on top of just 100 gigabyte On top of just 100 gigabytes drive and you're still used 10 gigabytes of it But you pretend to have more and that's not all I'm coming to the best part of my presentation that the core part Probably, you know that VMs are usually almost same. I Don't believe anyone will be installing hundreds of VMs of really different kind like Just curious did anyone ever installed All the version of windows on the same virtualization cluster Please say no, please say no Okay, nobody yes, so usually you have almost gomaganos VMs, so usually you stick to have same version or Not not really same but close enough version of your operating system use almost same software and When use when I say that same same at saying it comes to do the application. So you have a lot of redundant data repeating data all the time especially Yesterday someone no, I don't I don't see you there today Yesterday someone come came to our booth and asked can we use a Overt for the virtual desktop and virtual desktops and What they actually have He told me he works in university and he would like to start 200 or 300 or VM for two hours and He was really concerned about space because yeah, I have a template what he told me I have a template for 50 gigabytes and then I have one hundred of VMs and one hundred of 50 gigabytes will be a five terabytes. I don't have that much space. Can we save it somehow I? Told him to come to come there, but he's I don't see him. Okay, so We can deduplicate all that data and what we are doing. We are actually deduplicating We are using a virtual data optimizer a tool by Let me turn it on We are using virtual data optimizer a tool by redhead and pyramid pyramid company That does three interesting things it works as a device mapper driver so yeah, just integrates to LVM and all that subsystems and It can make Block device from block device Bigger block device from smaller block device. How it does it? It starts analyzing blocks written to the storage. So when block hits a storage It reads it and checks that a block contains just a sequence of same byte like block full of zeros Oh, yeah, it does drop it. We don't need it at all Have I seen any of the blocks earlier it keeps index and RAM it keeps Some information have I seen it? Yes drop it we can reuse the previous one and finally when it have less blocks it compresses them and Stores them back to device Saving your space. What happens next? Oh No, oh No, how do how do I? Oh It really sucks Okay, you will see a lot of things Give me a second. Oh So how do we use it? We put the application layer between the storage drive and LVM So pretending you have 100 gigabyte storage drive Real one you can have an logical volume group of one terabyte one to ten and On top of it you have you still have logical volume for engine And then you have a simple and on top of the simple you have the volumes you already seen But now it looks more and more and more Pyramidic what it gives us You have just 10 gigabyte drive and you pretend that your 10 gigabyte drive is 100 gigabyte storage domain and Don't sorry not what stores them in your 100 gigabyte simple and on top of it You make 200 gigabyte of storage domain and say allocate 5vm 50 gigabytes each writing 5 gigabytes. So now what you have you have 5vm's believing they have 250 gigabytes and you believe that they Allocated 25 gigabytes, but actually you are still using about four gigabytes on a drive 220 gigabytes up there Four gigabytes down there. Isn't that impressive? Is it working really working? Yep I actually 10 gigabyte and 205 250 gigabytes is not just example I made some calculations and I installed a demo system at my home and make that video and Actually use those numbers so When you install just the first VM and I use center S version 7374 and 775 in rotation first VM uses about 1 and 4 gigabyte of physical drive and 4 well about 5 gigabyte of logical space the second VM It's just less than gigabyte and still you have about 7 gigabyte of logical space But it's not looking and old and third VM fourth VM and fifth VM and I repeat It's a little bit different versions of center S. They add about 200 megabytes Not gigabytes, but 200 megabytes to the physical storage. So it almost doesn't grow I installed five VMs on just four gigabytes and they were working and And all of them believe that okay, we have 250 gigabytes But I didn't carefully switching to next slide to not to get out of presentation Yeah, it works so The best part is over now. We came to the ground and real earth The same provisioning and especially compression and data application isn't free and the price is high The first problem Using that data application and compression is very very risky It it have a same risk as a plane on forex market with a leverage because we have leverage there You have 250 gigabytes on this side and just 10 gigabytes on this side and if you suddenly Start writing data on this side You may overflow your small tiny pity storage drive and What will happen after that all your VMs on the storage domains on top of your small video device? Will be affected they will not be able to write anything and It may happen just Suddenly everything went fine. You were writing data Your VMs are fine. You have ten VMs ops. You don't have anything you have your cluster down. You're running out of space That's really really really bad. Please don't do that and we are going to help you with that You have to monitor physical space. You have to monitor physical space on your device carefully and all the time We actually call it a guaranteed space and there's a reason we guarantee That you will be able to write that amount of data Maybe more but not less And if you're trying to write something to take a look on a guaranteed space and think about that like I have 10 gigabytes It may be not compressible data and I only have two gigabytes of guaranteed space. It can be a problem how do we do that and I told you Video is a just a device mapper driver. So you can stack all the device mappers and You may have thin pull on top of video as we do or you may revert it So you may have thin pull and video on top of it or you can make a Oreo Simple video simple or you may continue it until you are bored So this kind all the devices we scan all the devices and We try to choose the smallest one simple device. I don't mean the smallest one in size the smallest one device in terms of how much space it have and reported to a brick You still remember that Gloucester bricks and stuff and The brick is a physical thing the brick is the mount point of some device So we reported to a brick and store it in a brick and then We calculate same value for volume and If the volume have a connection to a glass to a storage domain You have to check a special checkbox for that. It also will be reported for a storage domain and You can take a look on it. So if you have a brick On the brick properties You see how much space it have and the application compression savings There is no savings there because it's a storage domain and brick from a different installation. It's actually empty. That's why it's zero Same works for a main view of a storage domain. So when you open When you open storage domain dashboard, it's so important So we put it even on dashboard. You will see how much space you have and how much space is Available and the guaranteed free space is blurry this there. I have no idea why It doesn't support supposed to be blue And you can also configure a threshold. So if you're guaranteed free space Will be used will be less than say in my case 10% of the physically Available space of your simple device of your same device. It can be simple or video on any of them you will start getting an events and messages and it's a Just an overt message event in system you can Get a mail SNMP, whatever But that's not all that's not the thing you would just pay in that was the most important thing because it can break everything If you run out of space, it will just break and you can run out of space really really fast But that multiplicator works in both directions the other problem as That I told you the application the application compression isn't free You spend your CPU cycles for it Of course, you have to spend your CPU for checking the index checking the blocks compressing them In worst case It may reduce your IOPS up to 30% in best case It may increase your IOPS Yeah, I've heard that from drive space times like oh, you know drives are so slow and CPUs are so fast So if you can't compare the data It may be it may require less physical IO operations now in 21st century drives are still living in 20th century and CPUs are fast enough right now So it may happen. I've seen that on a really really slow drives compression in the application increases performance of your IOPS up system There another one thing you pay and it's really important because it's really important for in a virtualization environment You have to pay a lot of RAM you Use about one gigabyte of RAM for each terabyte of a logical space And as we recommended to be a smallest part so one to ten and just one terabyte You spend a lot of RAM. Sorry. We have to keep index somewhere and we also keep a copy of the index on a storage domain Not on a storage domain on a cluster volume on a physical volume. So yeah, we are stealing your physical volume. Sorry guys Lots of space is used for the application someone small drives. It's not that that good as it can be That the good news there The good news All the slides this one. Oh No, it's not visible. Okay those numbers They include that overhead So that one gigabyte 37 megabytes of a storage domain for VM It includes overhead of keeping index there for second VM. It's same for third VM. It's same. So it's still make sense and It's not that bad. You are not spending more Gigabytes for keeping index now. No, no, no, no, you can get more data of it Soon you're going to lunch What it's all about Why we spend a certain minutes with me? Now, you know That is overt and over hyperconverse You may believe or You may tell your VMs that you have up to 25 times more space than you actually have and you can even try to use all of that Unfortunately, it works only with hyperconverse code. We have to control the whole storage stack sorry, if you use some external storage, we can do that and everything comes with price and The first price is a multiplicator that works in both directions. So yeah, you are multiplicating your gigabyte to 25 gigabytes of available data and then you Multiplicate your 25 gigabytes back No magic here if you try to write tomorrow too much data, you will be screwed. Sorry and Yes, finally finally You have to spend your RAM. You have to spade your disk space You have to spend your CPU cycles for achieving that wonderful 25 times Improvement of space. Sorry, it's not free and thank you. Do you have any questions? Okay The question was Can we use external blaster and benefit from it, right? The answer is yes and no Yes, you can install blaster and you can install blaster and video externally configure it on your own and use all the features of blaster and No, sorry, we can't monitor it You have to implement it on your own. We only can monitor the blaster volumes. We are managing that Partial solution could be You know, you don't really need a hyperconvert system in case you would like to use blaster So you may have a blaster nodes in that cluster. I Mean an overt cluster managed by a overt but not running VM. So it will be like partially external Yeah, if it's managed by or weird, we will be able to get all the data from it and We will be able to link that data from blaster to storage domains and give you giving monitoring information But even if it's not you can still use it that you only have to monitor it on your own and configure it on your own Not via UI, so I believe everyone is hungry. Well, if no more questions, thank you and thank you for attending