 So yeah, David Von Thenen. I'm from actually a team called Code by Dell EMC. So the code team what we do is we focus on everything open source. We work with Meizos, Kubernetes, Swarm and kind of what we're known for is taking different container schedulers and enabling them to consume different storage platforms in the back end. So kind of focusing around the talk today, right? Talk to container scheduler software based storage and then the cloud. Prior to working on the code team I actually worked in back up and recovery solutions in the virtualized space VMware in particular. So backing up virtual machines vCloud director and vCloud Air when it existed. So yeah. So today we're going I'm just gonna do a really brief overview of what software based storage is and then we're gonna talk container schedulers for those just do a little quick review and then kind of go into depth for those that don't know much about it and then we're gonna talk about what happens when we combine a Meizos framework and software based storage and then what happens when we take that framework into the cloud and then hopefully we're gonna do a demo maybe to if time permits. So software based storage. So there's what are they? So there are a lot of definitions that are kind of floating out there if you look at the Wikipedia page there's just gobs of information I think most people kind of agreed on two things one software defined storage serves as an abstraction layer on the underlying storage. So you know we'll go into a couple examples in a few minutes and then they also provide software defined storage usually provides an API. So some some form of CLI or API rest API to make changes manipulate the underlying storage or the storage platform. So a couple examples I have here are NFS and BSAN. So if we look at NFS actually before we get into that. So what makes software defined storage unique? So operationally you provision the hardware or you provision your storage through those API's. So if you and it's independent of the data that's on the underlying hardware. So as NFS as an example right that's a very primitive software defined storage. Very very primitive. It's kind of like one of the earlier ones. But if you have a host system and then you have rotating disk underneath you have a just through CLI you can say create an NFS server to serve this up to some consumer and that consumer would be an NFS client right. And then kind of the there's a physical abstraction layer that that also exists. So you could have instead of having one rotating disk you could have two rotating disks that form an LVM group and you can serve that combined storage up to to your NFS clients. And there's also a form of a notion of policy. So you can usually on software defined storage say I want to have tiered levels of service. So I want to have like a gold tier that's you know that's maybe SSDs and then you maybe have a bronze tier that's rotating disk and based on your level of service that you want for your particular application you say I need to have the gold tier or I need to have the bronze tier because I don't really care about performance. And then the other thing that's kind of interesting about software defined storage is that the day two operations are inherently unique. So what I mean by day two is maintenance right. So once you you know throw up your software defined storage platform. How you manage that platform is very different than how you manage your typical array that sits in some data center that takes up an entire rack right because it exists all mostly in software. It you know how you handle like approaching full and in those types of events are very different than a traditional storage array. So I kind of already gave a little brief example of what NFS looks like. So this is kind of like just a visual representation of that. So two physical drives maybe they're in an LVM group. Then you have that software defined storage layer which is just your NFS server. And then that gets exposed out to the end user and you consume that NFS storage like we all know using an NFS client. So that's a very very simple definition of what like software defined storage is. We're gonna skip over vSAN because I kind of want to put more time towards the demo. But this is a very complex example of vSAN. Who's used VMware? VMware vSAN at all? So yeah. So it's rotating disk with a SSD front end that's aggregated to look like a global pool of storage that you can create logical volumes out so you can create VMDKs for your virtual machines in a nutshell. So I picked NFS and vSAN in particular. And the reason why and what makes them special is that they all exist in software. So there's no special purpose built hardware that's required to run NFS and vSAN. Yes, vSAN does have an HCL and all that stuff. You can roll the dice and bring your own SSDs and stuff like that. But again, there's no special purpose built hardware that's required in order to have software based storage. So software based storage, another way of looking at it is if you have a host system that's out there and you have a software based storage platform, usually the way you install that is nothing more than installing RPMs or devs on top of a host system and then exposing some sort of storage out from that endpoint. Questions so far? Yeah. So that's another good example. Okay, container schedulers. So who here has used Mezos, Kubernetes, Swarm? Okay. So we're just going to do a little quick overview and then we're going to kind of, because we're here at ApacheCon, we're going to talk Mezos and then go into a little more detail. So what is a scheduler? So a scheduler effectively does a fair and efficient placement of workloads. And they usually adhere to a set of constraints. So if you have a particular workload, maybe you have a Postgres database or even something simple like a little Java app, and you deploy that into your container scheduler, it makes sense that the container schedule is not going to take all your workload and put it on one piece of compute. It's going to try to distribute that through all your compute nodes. And it also does that quickly and deterministically. So if you have a particular workload that needs to be rescheduled, there's a deterministic time that's known where if a workload comes down and it comes back up on another node, that's a set finite time. And the reason why it's deterministic is if it wasn't, then it just becomes very difficult to predict the behavior of your application in your workload. And then it's also robust and tolerates failure. So if you have a particular compute node that fails, that workload is going to get rescheduled onto a different node so that your application is available again. So a scheduling work. So what kind of work am I talking about? So I'm talking about containers. So we're talking container schedulers. So containers like Docker, MazeDos has a unified containerizer. And then for those that are familiar with Rocket and CoreOS, those types of containers. I already talked about that container schedules are effectively a cluster manager. If you have a particular workload that comes down and it needs to be rescheduled, it does that. It's also a resource manager. So if you have n pieces of compute node, you have X number of CPUs that are out there and Y amounts of memory. And when you schedule a particular piece of work to be run on your compute, it takes into account how many CPUs you need and how much memory you need. And based on that, it will decide where to place your particular task. So as an example, if you had a particular workload that required two CPUs and very little memory, and you had another piece of work, another workload that required like, you know, 0.1 CPUs and like 60 gigs of memory. It might be advantageous to place both those workloads on the same piece of compute because those resources aren't competing with each other, right? And then there's also operational constraints. So when you schedule a work, you can say, I want to deploy this particular application. I want it to be scaled to three nodes. But maybe you want a constraint that says, hey, when you place these, you know, these three instances, don't place them on the same node, right? Spread them out on three unique nodes so that if one fails, the other two are up and running. And the other cool thing about container schedulers is that you can do forms of custom scheduling. So if you want to kind of extend the analogy of, you know, the example of placing on different nodes, you might also, if your application's highly sensitive, you like, you know, maybe using NoSQL and Cassandra, and you have a three node Cassandra instance, you might want to even do something like, hey, instead of placing these on three separate nodes, I want you to actually place these three instances on separate racks, right? So if one rack goes down, you know, that I still have my application available. And you might also have a particular workload that requires like, I don't know, like heavy amount of compute cycles. And if you have nodes out there that are tagged that say, hey, I have a GPU that's available, you know, high-performance GPU, you might want to schedule that workload on that particular node so that you could take advantage of the GPU that's there. So that's like a form of hardware acceleration. Yeah. So I kind of generically talked about container schedulers, and now we're going to talk about one in particular. So Apache Mezos, since we're here at ApacheCon. It's been around for some time, but the cool thing about it being around for some time is that all the companies that, you know, the, here's the logo screen, right? All the companies that are on this logo screen here are all using it in production today. So it's not like some hypothetical, if I'm thinking about deploying it or I'm kicking the tires, these companies are doing it. And as an example, so my VP, Josh Bernstein, he actually came from Apple. And so the Apple Siri infrastructure is all built on Mezos. So it's not like a hypothetical thing. It's like real is out there. And then if, you know, you probably don't even realize that you used some of these services. So if you took an Uber here from the airport to the hotel, you know, you're a user of Apache Mezos, right? So kind of gave you a little intro to Mezos. Now, one cool feature that Mezos has is a thing called frameworks. And what a framework does is it allows you to schedule a task based on your specific applications needs. So how a framework is implemented, it's really in two pieces. So there's a scheduling piece and then there's the work piece. So that scheduling piece is called the scheduler, ironically enough, right? That scheduler can accept and deny resources. And based on what's offered, based on the resources that are offered up to the schedule, it will either decide to take some of that resource and deploy the application or decide to pass and do something else. And when it decides to deploy that application, that application or your application or container gets deployed in the form of an executor, which is that second component. So two components, scheduler and executor. And what that really that kind of little those two components that make a framework are actually what it is is an implementation of the offer accept model. And we're going to go into that a little in the next slide here. But what a framework does just to kind of really nail the point is a framework is a very specialized, it's like tightly, super tightly coupled to your application. So tightly coupled that if you your framework, if you wanted to do something beyond deploy your application and configure your application, you could do things like monitor your application, right? So if you wanted to do health checks and stuff like that, they're very specific to your app. So as an example, maybe you have a Postgres database and you want to actually check to make sure that your database is available, like one of your databases under Postgres, you could implement a health check that does something like that. So a framework gives you that, you know, that specificity to your app. Now, if you didn't want to use a framework, who here knows what Marathon is for Mezos? Okay. So Marathon itself is a framework. So most people don't realize that. But what Marathon provides you, instead of having that very specific framework for your application, what Marathon does is it's a generic provider to launch containers. And that's all it is. But that provides you kind of like a kind of like a catch all generic way of deploying a container, whereas a framework is, you know, giving you something very specific. So that's kind of like the difference between what Marathon does and what like a custom framework does. Let me skip that. So we talked frameworks and we talked the offer accept model. So this is actually, so this was taken straight off of the Mezos, Apache Mezos website. This actually is just a visual representation of what the offer accept model looks like and what the actual workflow for framework is. I'm kind of calling it out here because towards the end we're actually going to deploy a framework. So I'm just kind of, you know, just to have an understanding of what's actually happening behind the scenes. So the way the offer accept model is is all the compute nodes, which is the number one box down there in the lower left. Yes. Is all the compute nodes up there down at the bottom there. What they do is they offer up all the resources. So if like agent node one says I have four CPUs and four gigs of memory, it offers it up to the Mezos master, which is effectively just a control plane that, you know, manages all the messages and stuff that's going on. The Mezos master then in turn offers up those resources to all the frameworks that are out there, which is that number two box. And the frameworks that are sitting out there so you could have multiple frameworks. What they do is they see the resources that are available on all the various pieces of compute and they decide to either deny it or consume some of those resources. So if the framework in the form of that scheduler says I want to actually schedule work, it'll say I want two CPUs and two gigs of memory out of that pool on like agent one. And when it says I want two CPUs and two gigs of memory, that's that number three box. It's going to say this is what I want to run and this is how much resource I'm going to consume in order to run it. So that's that number three. And it sends it back to the Mezos master and it's just effectively just going to be a control plane thing where it's going to go to that agent node, which is we said is agent one, and said I want you to launch this container, you're going to consume this much memory and you know have at it. And that's that number four box. So when it actually when it hit that number four hits the agent node, your container gets spun up, it consumes the resources and that's how it knows what resources are available at the end, right? Because the next time that agent node offers up that resource, it's going to be two CPUs and two gigs of memory less than what it currently has, right? So this is basically just an encapsulation of what the offer accept model looks like and kind of what and what the framework implementation actually is and for a Mezos framework. Questions? You can actually do it either way. Oh yeah, so when you launch the task in the form of that executor, does the executor talk directly to the scheduler or does it talk through the master? And you can actually do either way. There's a mechanism for doing either way. There's advantages and disadvantages of doing either. If you want, let's talk afterwards. That's kind of a complex question. So we talked, you know, gave you a brief background on software-based storage. We talked schedulers and Mezos. So this was about a year ago. I had this idea where like what happens if we create a software-based storage framework? And I built it. I first released it in September 2016. It's now on version 031. It's open source. It's available on GitHub. The URL is right there. Yeah, and kind of, kind of the idea, we'll go into the idea a little bit in the next couple slides down, but this framework, what it does, it deploys Scale.io and manages Scale.io and it's a software-based storage platform and talk about it a little bit. So Scale.io is a software-based storage platform. It is a scale-out block storage. So what's interesting about Scale.io is that it's all software-based. It's all you install rpms, you install devs on however many nodes. You can do it in a hyper-converged configuration or a two-tier configuration. As you add nodes, because it's all software-based, Scale.io and like the metadata manager automatically knows that it needs to rebalance the data. And if you take nodes out for maintenance or you have a hardware failure, any data that was on that node that's missing, it will get automatically get rebalanced. So why is that kind of cool? It's because all the maintenance operations are completely taken care of for you. There's no user intervention to make that happen, and yeah, and if you need to take stuff out for maintenance or hardware failures, you don't really have to think about it. And it also has an elastic architecture. So if you need more IOPS, you can throw more nodes at it and it will, you can, instead of like going through a traditional storage array where you have one controller that you're going through, Scale.io, when you, as a consumer of a volume from Scale.io, what it does is actually stripe the data from end nodes all at once, which is kind of cool. So that's how you get the, as you add more nodes, your IOPS increase. So yeah, the link for it, try it, it's a free download. Yeah, give it a try there. So the interesting thing is all the stuff that I'm kind of describing right now, it couldn't be inherently done with any software based storage platform. That's just one that I happened to implement with. There are plenty of other ones that are out there, so I believe Rancher has a, like a competing product to that. They just released I think at DockerCon, I believe, was their announcement for it. So yeah, in the end what I'm saying is that any software based storage framework, you could build something like that and have the advantages of something like this. So yeah, so if you deploy this software based storage framework, what ends up happening is if you had a pure Mesos, you know, like highly available three node HA cluster and you have various pieces of compute that are underneath, when you deploy the framework, it will imprint every single agent node with the ability to provision and consume storage, your software based storage platform. And why, so why is that cool? Well first, if it's imprinted on every single one of your agent nodes, you can provision storage from one and then also have that storage in the case of like failover, when that your work gets rescheduled onto another piece of, like another piece of compute, you could reattach that volume to that other node, so that you have all your data available, right? So what I basically just described is high availability for containers, right? And it's also, if you think about it, because you have this framework, when you bring new Mesos agent nodes up and that, so as agent nodes get gets registered to your Mesos master cluster, the instant that it registers its resources to that to Mesos, that agent will send up its resources up to that framework, that software based storage framework, and then immediately imprint that node with the ability to access that software based storage platform. So, because everything software based and then in this case we're talking scale I O, because all the maintenance operations are handled for you and it can scale out linearly, like you don't have these like this weird operational complexity of having you know a storage array where you have to worry about failed disks and this that and the other, right? If you have a hardware failure, bring the node out, it'll automatically rebalance. Once you fix your node, whether it's a hard drive failure or not, then you bring it, you reintroduce it back in. And because it's all RPM and it's Dev based, like RPM, Dev or whatever platform it is, it's, you can deploy this anywhere. It's completely platform agnostic. If you want to run this on bare metal, go for it. If you want to run this on hypervisor like VMware, if you want to run this in AWS, which is what the demo is going to be where that's, you can deploy that native us as well or Azure or whatever it supports Windows. So, yeah. So like, why do we care? So, if you look at, so, you know, you know, by containers nature, if you look at what they are, they're ephemeral, right? You bring a container up, you bring it down, any data that was written to it is gone. But if you look at, this is actually a snapshot of Docker hub. If you look at Docker hub today and you just go browse, it sorts it by popularity, like most use containers on Docker hub. And if you actually look at the containers that are there, 10 out of 20 of them are actually stateful. So if you look at, there's Postgres, MongoDB, Elasticsearch, Cassandra, Wordpress, I even think is up there. But they're all applications that have state. And so, what happens when you have a stateful application that's running in a container that is inherently stateless, right? So, traditionally when that happens, if you spun up Postgres, you write your data to your, contained within your container and you bring Postgres down, you've lost your entire database. It's just gone. Like what a lot of the container schedulers did, they recognized so, including so, you know, Kubernetes, Mezos, and Swarm and Docker. What they realize is that you're obviously running this, your container on a particular piece of compute. There is usually a, you know, direct attached disk to that piece of compute. So, let's just go ahead and rewrite, or re, reroute the data so that when we're writing to our Postgres database, we write to a local disk instead of within the container itself. So that when you bring the container down, you bring it back up, it can reattach and remount, you know, amount on the local disk to your container so that your Postgres database can reattach and then you'd have all your data and you'd have your database there. Now the problem is, is that data, is the data locality, right? You're writing all your data to local disk and what happens if you have a hard dry failure or what happens if you, your mother board on that system goes out. What happens to your data? Well, because it's all local to that particular piece of compute, you've lost all your data. So you kind of, you know, you've kind of won certain advantages of doing it like that, but then you've also, you know, unfortunately it's all tied to that node. And so kind of like what we've realized, you know, going way back is that if you want to have like an application that's highly available, you need to have that data live on some piece of external storage. And yeah, so yeah, just need to have it live, you have your data live external to your compute nodes. And I kind of glossed over it really quickly, but so VSAN and ScaleIO, they inherently do the same thing. So what ScaleIO does is it takes your direct attached disks, contributes them to a global pool. And that's how that pool is that, you know, it's basically striped and stuff like that across that global pool. And when you provision storage, your provision storage are volumes out of that pool. And because it's a globally accessible pool, if you need to provision storage or remove volumes from one node to the next, that's how ScaleIO works in the back end. And because it can be, even though you're using local aggregated disks, because it's accessible from every node, it looks like external storage, right? Even though it's using the local disks as a back end to your storage platform. Yeah. So yeah, it's not. And actually, I intentionally built the ScaleIO framework, the framework to install on bare metal, because if it installs on bare metal, it can install in a VM and it can install on the cloud. Yeah. What's that? It's not. I kind of just intentionally glossed over it. You can use any software based storage platform. That's the one I happened to implement against. Like I said, there are competing products like Rancher. They have their own software based storage platform. I also believe I can't remember the other one off the top of my head. But yeah, it is free. You can download it and try it. It's not like some stripped down version that's you know, that's like throttled or anything like that. It is the full version. So what's that? Yeah, that's so in the back end that's that's how ScaleIO works. Right. If you lose any one piece of note in one piece of compute with your disk that's there, it's obviously replicated to your data sits on multiple nodes in order for it to be highly to be available. So your data is available. Let's take that offline. That's a very complicated question. And I kind of want to get to the demo. So yeah. So this framework installs a software based storage platform. It also installs the following components Rexray, which is open source. It's a vendor agnostic storage orchestration platform. So what it does is if you have something like Docker or Mezos and you want to provision storage, this provides you the mechanism to provision that storage. It supports AWS in the form of EBS volumes, EFS, DigitalOcean, GCE and ScaleIO. So that's why it installs this. And then Mezos module DVDI, for if you're not using Docker workloads, Rexray, it implements the Docker volume driver interface. And so when you're deploying a Docker container that requires external storage, it goes through Rexray. Now Mezos module DVDI is a Mezos file system isolator. And that basically you hook that up. Well, the framework will automatically deploy and configure all this. But what it gets hooked up to Mezos on the agent node and it effectively just is a file system hook to call Rexray to go ahead and provision a volume for non-docker workloads. So using the Mezos containerizer. And actually for that project, they're both open source. The second one, the Mezos module DVDI, it actually happened to be because it was open source and it was actually quite successful. Mezos itself it was actually pushed upstream. And as of 1.0, it's actually available in their source tree. So you just need to have flipped the bit on there to enable external storage. It's listed as experimental, but it's there. So yeah, what does this mean for your applications? So if you have an application because you can provision and access the access the storage from one node and then move that access to another node, it means your applications are highly available. Right. Piece of compute dies. You can bring it up and yeah, you have your application come up on the on another piece of compute and you'd have your data all available. All right. So take everything we said and wrap it up and we throw it into the cloud. So yeah, what enables all this stuff to happen is all it's all about the APIs, right? If you're in the cloud, you have eight APIs that are there, you know, through the form of Amazon, AWS, so which we're going to show in a few seconds. And for software based storage platforms, they have APIs, right? To manage the actual storage platform themselves. So kind of where this is going is if we have a framework that deploys and configures and applications and then we have APIs to manage and monitor the application, we should be able to through this framework do health and remediation on the storage platform. So if you can do health and remediation, that means when you run into trouble, you should be able to fix your storage platform, your software based storage platform. So yeah, Johnny five is alive and then we're getting into, you know, the whole Skynet thing, right? So I'm going to do a demo in a few seconds. If we take a storage software based storage platform and move it into the cloud like AWS and then using the AWS SDK, which is available in 10 language bindings, that application should be able to do things like auto scale instances. It should be able to do stuff like dial in the number of IOPS required on your disks. And then it should also be able to provision new hard drives and stuff in order to expand capacity. So what I'm saying is a framework, it's kind of cool if you have one that's very specific to your application, it almost gives you the ability to like make your application do things that were like otherwise not intended. And how much time do I have? I got 20 minutes. All right, we're going to go, we're going to do both demos today. So the second half of the demo that I'm going to have is I that kind of the idea is, is so I have this software based storage framework. What we're going to do is we're going to have, you know, roll out this software based storage on all the nodes that are out there. We're going to inject a crap loaded data to make that software storage platform look like it's 98% full. And who storage admins out here? Anybody's storage admins? No. Okay, storage admins, the dreaded, like if I have a traditional box and I have a whole bunch of disks that are in there and people's, you know, I'm not managing really well and it's starting to approach full, that's like a nightmare scenario because as soon as the next thing you need to do is quickly add another shelf, quickly add a whole bunch of hard drives in there, then go to the storage array and say, hey, go rebalance the thing so that I have, you know, I start striping the data across all that. So it's a huge operational nightmare. And in this case what, because the, the software based storage framework does the auto balancing and the framework itself can add new disks, it can provision more storage, expand the capacity of your storage pool and then give you more capacity so you're not full anymore. And so actually that's what I'm going to demo. Configuration info, the slides will be available afterwards. So really quickly this is what's going to happen. Deploy the scheduler, the offers go up to the mesos cluster, the scheduler then says, I want some of that resource, it's going to imprint the software based storage platform onto every piece of compute. I'm going to inject full into the, into every, into the storage platform. It's going to make a call to AWS to create EBS volumes. It's going to send the volumes, reattach them or attach the new volumes to the software based storage platform and everything will be okay. So let's go ahead and do that. I hope this works. So here I have everything in AWS, three master nodes, three agent nodes and actually let's take note of a few things. So here's the three node HA configuration, typical mesos configuration. I have three 180 gig disks that are already hooked into there because we need existing storage to create the the storage platform. Notice there aren't any one terabyte disks. Those are going to get created in a little bit. And then this is what marathon looks like for those who haven't seen it before. What we're going to do is we're going to curl JSON, which basically says deploy my framework. And it's going to go ahead and imprint the all the mesos agent nodes with the with the software based storage platform. Let's do that really quickly. Hope this works. Okay, so the the schedule is getting deployed. I've embedded a little UI in the into the the scheduler in the form of a rest end point with a little UI on there. So I'm going to go to that UI right now and it basically gives you the state of what's going on with in those three executor nodes, right? Those represent the three mesos agent nodes that are out there. So it's installing the scale.io packages right now. Right. So this it's just installing in this case, it's RPMs because we're on red hat. It's creating the cluster. And that actually that first note on the top is the one that's actually creating it. And then we're initializing the the platform right. And we saw that they had we had three 180 gig disk EDS volumes. So we're going to add those to the scale.io storage pool. And then hopefully we should see the capacity right once that gets added in the capacity should be there. So that's how much capacity is available in our storage platform. We're installing Rex Ray and mesos module dvdi. So those are the two open source projects. Right. Rex Ray is the Docker volume driver. So for Docker workloads and it installs mesos module dvdi for non Docker workloads for using the mesos containerizer. And it's not going to need to reboot. So right there. So now everything's in the running state. So starting with the pure mesos installation without having any storage stuff on there deploy the framework and in in print all the nodes your agent nodes so that you have you basically have access to provision volumes from a storage platform software based storage platform. And then it's got that much capacity and because we have some time hopefully let's go ahead and provision a persistent application. So let's go ahead. Just going to do another curl and I'm going to deploy a go based HD to be server that's going to provision an external volume. I'm going to curl that right there. Let's go back. So a little web server. So now it's running and what I did was just like in the the framework case. I embedded a little UI and notice that it's got the lost and found so that we know that it's external storage. I have a little data file that I'm just it's basically scribbling stuff into. Notice that it keeps going up and up. So if I go ahead and I stop this if I kill the up kill the container right this web server and now we notice that it's not there. Now if I go ahead and I redeploy it I did my job right. We should see that all the data is still there and we keep appending to the end of the file right. So yeah so here's a good example. If I could have killed one of the nodes in AWS but I didn't want to risk it but if you had a node failure and on that wherever this HTTP server was running if you had a node failure that work would get rescheduled. It would reattach that volume to that other node and then so when you looked at this you know looked at that your data file you'd have all your data available. So if this were something like a Postgres database right that's like some serious stuff and this is a little text file but if you had a Postgres database and you need and you had a hardware failure or needed to reschedule the work for because of maintenance to pull the system out it would get rescheduled somewhere else and you'd have all your data available for it. So let's get rid of that guy. Now let's go back to the scenario of so this is the capacity that I have and what I'm going to do right now is I'm going to inject full and this is kind of why how a framework in the cloud can be super advantageous because you can start modifying the underlying infrastructure. So I'm going to do oops I need to take note of this guy here because the end point has changed. So anybody familiar with Postman it's basically a way to post a rest call. So header information the body to acknowledge and this is how much fake data I'm going to inject and it actually worked. Yay. So if we go back to well this is not going to update oh there you go. So yeah so now it's not 98% used out of 90% so the threshold was 90% and this is the amount of fake data being used. And if we go back to the console here AWS hope this will work in five minutes keep refreshing. So now the storage platform knows that it's 98% full and hopefully it's on a poll right now. So I think it's like 30 seconds or a minute. Okay so there's the first one the first volume one one terabyte and if we keep refreshing should create the second one and notice the first one's already attached now it's creating the third one and it's attaching it to the each of the three mazes agent nodes that are there should be done. Now what's happening in the back end because we kind of said that software based storage platforms they all provide APIs that you can do to manipulate the underlying platform. So what's happening in the back end is it's making AP the framework the scheduling component of the framework is making API calls to create the EBS volumes but it's also that's controlling the the back end the AWS side but it's also making API calls on the storage platform and to say hey take these three disks on these three nodes and add them into the storage pool and then once that storage gets those EBS volumes gets added to that storage pool we should hopefully see the capacity expand and then the percent use drop and this is on a poll too and it makes a few API calls here and there and hopefully any second now any who I can take some questions and I'm sure it'll kick over in a second cross your fingers oh there it is 14% use capacity is expanded so yeah the framework itself just modified the underlying AWS stuff took the storage introduced it to storage platform and recognize the you know the is full event and basically fixed itself right so that's kind of the idea of the talk is I wanted to float this idea that using frameworks and stuff like that you could literally have applications that kind of like manipulate the infrastructure right out from underneath it so I thought it's a pretty cool idea it depends on the storage platform right and how they do the data striping across all the nodes in this case you're only going to ever get like it's about 50% is what you're usable is but yeah like I said it depends on the stored platform and how they strike the data yeah and yeah but to kind of the point is is this going to be done with any software based storage platform right so it's just a cool idea any who yep yes well it actually there's a bunch of things that happen so I picked three nodes three agent nodes because it requires a three node minimum right for quorum but all those three nodes get installed with a a metadata manager so it's a primary secondary and a tiebreaker that's how it knows like what data needs to go where to in order to stripe it then some other packages that get installed is a server component which basically takes the the devices that were introduced or that's a direct attach to that system and provides it to be consumed and then there's a client component that gets installed that consumes and is able to consume and provision storage out of it the red hat so six six x seven x you've been to 14 1404 1604 core os I'm there's a huge list I'd go to the website and take a look any other questions the live demo worked happy about that yeah so what I deployed here is effectively a hyperconverged scenario right is because I have three nodes here I I'm also providing and consuming the same storage all my pieces of compute there's other ways to install this if you wanted to and most stores like software storage platforms do this but if you wanted to have like a two tiered type deployment you could have your software storage that kind of live out external like completely external to your mesos configuration and then just and then you're this like a framework a storage software based storage framework what it would do instead of providing storage all it would do is consume storage and you just point it externally and this framework actually does support that configuration too you can still like right now if I've decided to kill one of these nodes right now like all my data would be available and if I wanted to add in a new node the framework the scheduling component of the framework would automatically imprint the this new fourth node with scaleio and because scale does all the maintenance for you and it rebalances everything it would rebalance all the data and you wouldn't know you know it would it would just basically take care of that for you wouldn't know that the maintenance that stuff was happening in the back there is I don't so that's some of the stuff that is I want to add into like the next version scaleio does have its own UI that's obviously way you know way tracks everything in the sharding and all that other stuff you can go and view it there I think just like any other like storage platform would have some sort of mechanism for that but yet in the UI itself in this particular one in the framework UI it's not there but there is scaleio UI that does provide stuff like that any other questions cool well I hope you found it interesting and entertaining so thank you guys