 All right. Thank you guys for coming. My name is Albert Chen. I'm a system engineer from Swiss stack And I'm Adam talk them another systems engineer from Swiss stack So we're here to talk about Swift 101 today It's a new breed of storage system that we're working with so hope you guys enjoy So quick overview over the agenda is I'm going to go over what object is quickly Why do you swift and what are the different components of Swift and then Adam will go over a couple use use cases to give you an idea What it's used for So what is an object well an object? Consist of a piece of data plus its metadata. So here's a picture I took of the bamboo forest on my trip to visit Japan back in 2013. So you can see Along with a file name, which is a very long file name. I have it has the size of the picture the date taken the extra tags I attached to it and plus the GPS GPS location from my phone all these Information combined with the actual picture itself makes up for an object So, but why Swift so what makes Swift different than traditional object store or not even object storage is Just traditional storage systems in general One of the things that we traditionally associate with storage is the way that we access it How do we ingest and consume data from that storage repository? So we're familiar with SIF shares NFS shares Luns and all these various technologies that are used to access data some of the Limitations of that though is that there is a limited scope that you can access that data from it's usually accessible from within the enterprise or within the Organization and not much outside of your own boundaries. So Swift by its native interface is HTTP So by speaking the language of the internet it does exactly what you would think and it can expose Your data to various devices and endpoints that are both in your organization and anywhere on the internet So that changes the radius of that you can access and influence that data the other big difference is that Swift is a pure Pure software-based solution so it is as opposed to a hardware-based appliance Being software gives you the flexibility to be able to choose the hardware that you want to run it on and to be able to scale out that Hardware at any arbitrary increments. So you're not locked into having to scale by particular Racks or or even keep the same exact configuration, you know as new things come out you can start to scale into those things as well Other big points about it are the ability to be able to define Policies without getting into it too much the ability to have durability through erasure codes or through replicas Gives an additional level of availability for that data Let's take a look at that in more detail. Yeah, so Swift is consistent of four different pieces of service So I'll go over that in just one second, but the first things I want to talk about is Data consistency and I'm going to try to do this without tripping at my own town So the traditional storage system you're most likely used to are usually strictly consistent and what that means is when you write a piece of data out and it needs to write to multiple locations all Those location will have to be completed successfully for the right to be completed And if there is an error in one of the locations writing to then the entire right operations consider fail So that's what block file storage is is usually consist of Now eventually consistent system are what is slightly different What that means is when you initiate a right multiple copies will be written out simultaneously Now if there are five three pieces of right happening and two of them completed which is more than half of the rights The right is considered. Okay If that that last piece did not complete the system will Redo redo that right asynchronously afterwards So you don't need to wait for all three pieces to complete For example, if that third data center you have is having a network outage or this drive It's trying to write to is having a bad desk and needs to be replaced the system will self-heal at the end of the day There are some limitations to Strictly consistent and eventual consistent systems depending on what your use cases and what your scalability is So we've listed those out on on the slides Okay, so now that we got the consistency piece out of the way Let's go on to the different services The first one I'm going to touch on is called proxy service This service is responsible for directing all the incoming HTTP API traffic And it's going to direct that right to part of the cluster. How does it know where to send those rights? Well, it will take it will compute the hash of the incoming objects file path and Take that value and it will use that value to figure out where that right will be located on Where that services on the back end? So if you have a hundred storage node, it'll take the incoming file path and then it will figure out Oh, it's going to be stored on node 10 17 and 25 so a swift will do his best to spread out the right in a As uniquely as possible type of situation What that means is I'm going to take if I had to write three copies for example If I only have one hard drive is going to write it to three different location of the hard drive If I have three different hard drive, it's going to write it to three different hard drives So on so forth if I have three servers is going to spread them among the different servers And then if I have three different geo locations, it'll spread that among the different location as much as possible And the idea is so you'll get better durability in case you have a drive failure or a server outage when disconnections The proxy service is considered stateless. So if you want to increase your performance, you will just add additional Proxy service. So you have more parallel worker that could process the incoming request for read and write Okay The next one is the accounts and containers service the account service is responsible for the accounts metadata such as username permissions and it has a listing of the containers that's owned by this account and There's also the container service which contains the container metadata has the listing of the objects underneath and the storage policy such as replicas or racial codes We usually combine these services together because they're very similar We deploy them in on SSDs and part of the reason is because when you have an incoming request all the requests will need to touch on these accounts and Container service to know where if you have the rights to read these containers What objects in the container and where the objects are and then before it could go and retrieve the object So it's think about it as an indexing service You would increase the number of accounts container if you have a large number of object stores or you have a lot of users and you have a lot of Frequent accesses so and you just add additional services to them We will also deploy these on SSDs to give you a faster response You could put them on spinning disk, but giving the cost of SSD and how much space it actually use We generally provide a disk per two per one of the services notes Then we have the last month at least but the object server So the object service is responsible for storing your payload So if you saw example earlier my picture would actually be stored in the object server This is where all the disk will be it would usually consist of a large amount of hard drives The metadata for the object will also be stored in the object service as well And there are two other subservices I would like to let you know that's running on that one is called the replication service It is jobs is to be responsible for making sure you have enough copies of the good data If it find out that you have a policy store three copies of this picture But it can only find two because one is being replaced or being corrupted It will put into the queue and replicate another copy to another location. So that that Service is called the replication service There's also the auditing service that work with the replication service Its job is to find data in your system. That's not that's been corrupted and it's been modified unwillingly so Its job is to look through all the data and then flag anything that's unusual Check it with the incoming some that the checksum that we have written when the data was first ingested into your cluster and Do a comparison if that does not match it will quarantine this piece of data and said to the replication service Hey, this piece of data is bad Get me another copy from another location the most General scaling out use case is when you need more storage or if you you you run out of space Okay, well you add additional object storage services in your system usually by adding additional disks as well Okay, so Adam will cover the next part All right, so So what do we do with an eventually consistent object store? We want to examine a few different use cases of Specific implementations that people have done and some common things that we run into out in the field First among them is backup I know everyone here backs up their data. That's where I give that that glance at everyone, right? You everyone backs up their data, right? so So we have a number of partners that are mentioned at the bottom as well as a Two different primary cases in which we see this one is Enterprise backup scenarios so you're backing up the various workstations within an enterprise or in a managed service provider context Where you're actually providing a cloud backup service for customers and in either case you can You can leverage an object storage to do really two things one is you want to be able to have a very high Throughput so as he was mentioning the ability to be able to scale out the proxies becomes very helpful Especially in the MSP scenario if you've got a lot of different endpoints that are backing up at the same time and then also having the ability to To have those rights optimized so that they are They're coming in with again sort of again Re-emphasizing his point around the eventual consistency having the rights more guaranteed reduces the number of retries Which obviously improves the number of successes and decreases the overall throughput necessary So I gave it a as a couple of examples here of some specific high-density configurations We provided those just for reference if you're interested Another another use case. This is one that we get asked about a lot. This is one big data slash map reduce so this is where Customers come to us and they say well, you know, we of course we have HDFS That's being used as the underlying file system for Hadoop and it's starting to grow and it's starting to grow and it's starting to grow Well, why is it starting to grow? Well, it's starting to grow because we're storing all this with this warm data and this cold data is just over time starting to accumulate on HDFS And well, it provides replication. It has some you know data durability guarantees But as we begin to continue to scale this out first of all, it is strictly consistent so then going back to that earlier conversation and then It's not really what it was made for right. I mean HDFS was meant to be a Really good file system for being able to very quickly deliver data where you need it And it's really that data locality property. That's what makes HDFS unique. So let's let it do that All right, let's let it do what it does best and then for once that we have the data That's already computed and put out then let's put that over into an archived here like a swift Swift then gives you the ability to store that data with all the guarantees and durability that comes along with it as well as being able To have a centralized repository for all sorts of kinds of data So as you're trying to share those results with other different departments within the organization you have the one place that everybody knows to go to whereas HDFS doesn't really provide that same type of capabilities and And also what it allows you to do is just shrink the HDFS down It doesn't ever fully remove it because you're going to need it for you know for the internals of what that MapReduce job is doing but having it as a Import-export relationship is really sort of what we're shooting for in this scenario So at the very last I included a link to the Hadoop OpenStack project It was formerly known as Swift FS and what it does is it's actually part of the Hadoop project now and It provides an interface where you can literally just natively go in and pull data right out of Swift bring it in to To a Hadoop job do your processing and then when you're done archive it back out Media and entertainment so this is a fun one because it really it highlights some some very interesting capabilities inside of Swift So the first part of it is the media encoding and ingest now This is one that so I stepped out a little bit of Swift Specifically and mentioned a piece of middleware called storeless that was developed by IBM and So what media entertainment companies need to do is they have to have a way to be able to first of all get unencoded Media into their file system, but then they also need to encode it into various formats So having ability to be able to manipulate objects on the fly as they're being either read into or written into or read Out of the object storage system would be really useful and so that's where this piece of middleware comes into play where It will basically gives you the ability to trigger on and on objects as they're coming in and out of the system So they can say well whenever we're going to put in a new You know movie of whatever it is right in a raw format Then we want to automatically kick off encoders that are going to go in and write in the various formats And then write those back so that sort of thing is now becoming possible This is really on the very cutting edge of of things that are available within the Swift community In content streaming content streaming is of course very important being able to Stream if you have a hundred gigabyte movie uncompressed Let's say being able to stream that back out to users and not have to have them buffer the entire thing and play it Is is key and that's intrinsic to what Swift does. There's absolutely nothing we have to do to support that Large objects so one of the limitations of an object storage system is that the object is the unit that we look at As far as the the minimum amount of stuff we can move around we're not we're not block storage So we don't break it into blocks. We look keep it as a unit So we put a five gigabyte limit which is the same as Amazon s3 on the size of an object So when these objects are routinely larger than that then What often we see customers do is now that they break it up like at the five gigabyte limit and then start another five gig segment And so on so forth which you could do the other option that they often go for is to break it up According to minutes or if seconds in this case usually 15 seconds or 30 second increments Because by doing that then that gives them the flexibility to be able to after the fact Modify that stream so they could then go back and modify the manifest that's created So maybe I need to step back a minute and it's So when each of those different parts are uploaded there has to be a manifest file that says these are the parts that comprise this larger object So then going back and being able to modify that so we can play with the manifest a little bit and say Well, let's insert a couple of objects here at this particular point in time and a couple of over here And now we've added commercials to the movie for better or for worse, right? And so it's a way to also be able to modify and manage your data In Swift without having to make copies, especially not personalized copies for every user Life science. So this is one where Swift is gaining a lot of traction as a content repository it's It's not that the use case is Incredibly unique you can imagine a lot of different Applications for this where you have a lot of large files that need to be shared with a lot of people But in particular some of the unique things that they do have are that there's a lot of laboratory instruments that are running So these could be gene sequencers or mass spectrometers or a variety of different any types of sensors or or things that are gathering samples Uploading that data into Swift And so there's a large concurrency requirement that needs a very large throughput to be able to handle all of that data That's coming in and then have the durability requirements the ability to spread that data across multiple regions Then also gives the life sciences community really everything they need to be able to you know read their data in Process it and then publish those results all within one place which Which again keeps coming back to the same theme of that one common centralized repository All right, so some of the question that you might be asking is this sounds pretty cool What is it gonna take me to get a Swift cluster up and running? So I'm gonna talk about the hardware a little bit So what kind of hardware does Swift run on as you can see here? These are not brand-name hardware. It's just x86 hardware the support requirement is Ubuntu sent us as long as the operating system supports the system that you're dealing with underneath That's all we need and be able to see Obviously you have to be able to present driver for disks Driver for networking adapters and what or whatever connectivity that you choose to to run through So As you see the logo on the right we work with pretty much anybody and we've got customer for pretty much all of them Depending on who you are mostly working with who you're familiar with Who is your preferred vendor? So I'll give you example this year. I may be getting very good deal from HP. They're having a sale I like their sales guys Next year they're like, oh Albert your your discount is only last year. I can't do it this year again Well, I can go and talk to super micro and Or Cisco and say, hey, you know, can you give me a good deal this year? And I could mix the different type of hardware together Swift will figure out how to put all those weighting into calculations and store the data evenly We put in on the slides here a couple of general purpose servers some of them have 60 base 36 base drives and 62 base rise the UCS servers actually two nodes compute combined into one so Depending on what your work workloads are you could pick the number of disks number of disk per server, but as for our Recommended three copy minimum. We usually ask you to have three if not four or more location to put that data So Swift can satisfy is as uniquely as possible locations If you have a backup archive, you'll say, well, I don't need that much compute power on retrieving this data I can I can wait another two seconds for my data to be retrieved We got even dense for solutions from 60 drive to 84 drive to 90 drives per box 90 drive what we're eight terabyte 720 it's very new that's that's the server Dell announced last week. So they're very proud of it So we'll give them a we'll help him amplify that signal a little bit. So yeah, so you can get 720 terabyte raw per box times out by three that's Several petabytes. So depending on what you use case and I think Swift works best when you're dealing with large quantity of data I'm not talking about five terabyte ten terabyte because that runs on two disks We're talking about minimally 50 plus To a hundred to 500 to a petabyte to multi-petabyte range scales Being software only it scales quite well on whatever hardware you give it But it's it's meant to play in the larger side of things. Okay, so As again, my name is Albert Chen. I work for Swiss stack and this is Adam. We're both from the Swiss stack here telling you about Swift and We also have a couple other sessions coming up in the next couple of days So please feel free to come and visit them. I'll put this back later, but Being this is a Swiss session. You said hey, this sounds good. I want to know more. We have links on Give you detailed guides and and overviews to if you want to find out more information There is also on open stack itself There's a Swift on one step-by-step guide on deploying how to how to deploy Swift itself That's a good practice if you want to find out what it takes to deploy Swift and do all the calculation by hand It's written by John Dickinson are one of the Swiss stack employees There's API if you're a developer for contributor. There's also access to IRC Now if you want to trial Swiss stack product and find out what the difference is I also have a link up there for the free trial I see a couple of you guys have books out from from Swiss. So that's the the book written by Joe Arnold our What's his title? The chief product officer. So he will be doing a signing and give away free books tomorrow at 1045 to 1115 at bouti 41 come by if you want to copy it gives you a Very detailed a good read on what Swift is. So if you want to find out more come grab a free copy of books and Of course, thank you very much for for coming and we've got questions. Now will be the time to put your hand up. So We'll grab you a mic so everybody can hear Yeah, you talked about the hardware requirement. Is there any DRAM requirement for the size if I want to implement a Swift on a 10 terabyte What's the DRAM requirement? For 10 terabyte. Yeah For that size, I probably would suggest you run it on virtual machines Generally Swift needs access to bare metal for trial purposes. What we find is people can spin up a virtual machine quite easily To virtual CPU 4 gigs of RAM now if you're going a little bit bigger I would say 4 v CPUs 8 gigs of RAM and you have a couple of nodes and you can run those in for for functionality testing But if you're doing any sort of production environment, make sure you put it on hardware Part of the reason Swift is is designed for failure And what that means is it needs to know when things are not working properly either is networking or disk or The entire box is not responding virtualizing in a virtual machine running it through ray controller Will actually remove that ability to see it. So we say run it on bare metal if you're gonna do it in production Do not run it through ray controller and there's a whole article online to say why but the short of it is We need to know when a drive is broken Okay I think generally speaking to the memory usage and maybe this is some what you're getting at is maybe caching You know caching requirements things like that. There is no caching inside of Swift So so the memory requirements you'll find are very modest There there are certain minimum requirement by the XFS file system that underlies it So it has a need to be able to cache I nodes which are you know are extremely small So if you had a really large number of objects We started talking about billions of objects then I would start to really care about your memory And and if we're gonna have you know, I I know faults and things like that which which in and of themselves are fine To if that happens it just means it has to pull it off disk instead of having it memory is not a big deal But yeah, that's really it you know So beyond that there's not a lot driving a really high memory requirement on anything in Swift What is it that Swift stack brings to Swift that's not there already Well, so a stack is a company that was built around OpenStack Swift and well really we're bringing two things into the picture One is the ability to easily manage the deployment and configuration of a Swift stack of a OpenStack Swift cluster Fundamentally and then also gather statistics back about that cluster So we're continuously gathering metrics not just CPU and memory But also how are each of the nodes doing the health of the services? You know are we starting to fill up the disk all these sorts of things And so we provide a series of metrics along with thresholds for each of those metrics so that you don't have to basically do all that work Yourself and figure out what's important to an object store and what isn't what's indicative of a failure or a failure about to come What you know, how do we know that we're about to encounter something so we handle all of that and then if any of those Thresholds are exceeded and that's an MP trap is sent and then that can be sent to you know a centralized monitoring solution. I Think there's one other one that Adam misses when things are not working I have a phone number to call and there'll be a person on the other side help me with it a Lot of time what we find with open source project is that it's a very powerful engine But that's all that is is an engine for it for me to actually use it I have to know every little thing about it. I have to be in the chat group. I had been the IRC I have to Contribute patches. I had to be in the weeds of things I have to know everybody to get this to feel comfortable to deploy this in an enterprise environment, but not every administrator have this Amount of time or energy not every company can dedicate multiple headcounts to Develop their own open source cluster and what happened that person leaves one day It says I wanted to go do a better do it find a different job What happened to your production environment? Do you feel comfortable doing that? No, so our job is to say we're here to provide you a Source of comfort so when you have your environment up and running and you need to call somebody and said hey My cluster is doing something funny. It's can you help me figure out? What this means or help me figure out how to write my application to take advantage of this cluster We are able to do that for you But I think it's very important for us to be here in the community and tell you what the product What the engine is about and if you want to know more of what we do We provide the steering wheel. We provide the wheel We provide the seat you sit into to make this car drivable for for your environment Okay, so I do have a question for the audience does can you guys put your hand up? This is the first time you've heard about Swift or object storage Hands up That's what I think you guys You guys are all veterans. You guys know what what object storage is okay. That's kind of interesting. So oh, no It's a survey. So if you guys have any question Adam I will be here afterwards to help you answer them I also have a demo for a Swiss stack I can run and show you guys after after the session outside to see what our product does But if you have any question, please feel free to come up to us and come and say hi. I Can two minutes. Yeah Give me one sec though Wait, is my internet fast enough for this Yeah, so I I don't have I don't have a re source running on my machine. So this will take a little bit of time to do but once the once the the installation starts to happen you can kind of get an idea of what what The system will kind of look like So one of the key things that he's going to show here in just a moment is about is around the controller So the controller node is one that we haven't talked about it Previously in this talk because this was an open stack Swift talk So we really try to keep the Swift stack things stuff to an absolute minimum But but Swift stack does make a controller It is an entire entirely closed source product that that we provide that is what does this centralized management administration and monitoring of the cluster and it also works To manage the ingestion of nodes that's the terminology that we use to ingest a node is to Configure it with Swift and bring it into your cluster and so so that process is really very straightforward once you have a Server that's set up like any of the ones that we mentioned in the hardware section the only requirement is that you have either The latest version of CentOS or Ubuntu or Red Hat installed We actually request a minimal installation We don't need any really other prerequisites other than just having access to the storage itself having access to the network and having the network configured appropriately Beyond that it's a matter of running a curl command to then download a script the script runs And it'll install a series of packages that we provide That comes from your controller So your controller also acts as sort of a package repository for being able to push out versions of Swift and Swift stack out to the cluster Okay, so what I have here is a Ubuntu node It's got the basic IP address configured hostname and That's about it. There's no Swift service installed basically empty so What Adam said earlier is you're right you initialize the installation by running a curl command towards the controller which is a So stack product and it's going to pull down this set of commands for you and This set of commands will basically say okay Well do a test on the node and then it's going to download all these packages for you automatically, okay? I'm going to pipe this to bashing Hopefully my internet is fast, but I'm crossing my fingers right now So what you see on the screen right now is going through the installation process to pull down the different Package dependency that it needs it's going to go and do an installation And then configure these packages So the node will be done. This is all done automatically One of the things you're gonna put up your hands say yeah, I can do this myself too. Yeah, you can But the thing is you got to think about Every single update you do you have to go and make sure you update your dependency new operating system comes out new version The Python console you're gonna go and check all that you can do that or we can do that It's up to you right one major part of that people overlook is what to do in the event of a failure I mean so deployment is one thing ongoing administration is another so if you have a failed disk What do you do? I mean with Swift with the open stacks with product? Well, you have options, but you're gonna have to manually do something You're gonna have to go and remove that drive from the ring Re-push the ring out or replace the drive and have it rebuilt those are kind of your two options with Swiss stack You don't have to really worry about that. It will recognize that the drive is failed it will automatically push the ring because that's your configuration and Then you'll see the cluster rebalance and nobody has to get up at two o'clock in the morning I don't want to wake up at two in the morning. Yeah, I don't either. I hardly go to sleep by two o'clock in the morning Okay, so while that is doing the installation right now I'm gonna switch to the controller so Who the people who have you Swift know that is a very command line based configuration? So this whole web interface that you see here is Swift stack only Basically, it'll give me a good indication of how healthy my cluster is and I can look through my My existing nodes. It'll tell me how much data. I've stored on it. How many how are my rings balanced? How well are they balanced together? I can even look on the server stats and these are different ones thinking about CPU memory usage across the different nodes and I believe I can also pick up the different nodes to actually look at I'm gonna find the buttons, but the Swiss stats is also the operational of the switch. So how much Throughput I'm getting what is response time? How long it takes before I get ahead a Post request completed. So these are all aggregated presented to you in a Easy to read format that I can track for multiple of weeks or years These data are all sitting in the controller itself There is one very important thing to keep in mind is no actual user data is sent to the controller We do the hosting of controller, but you're only sending back stats For for your system. No actual customer payloads and back to us. That would be too much data actually So then for those or who are running service provider type of model if you're providing storage for your different department You're gonna do charge back. We also have a utilization API by accounts To tell you how much to charge everybody per month This is a GUI interface, but this can be accessed through API as well And the last part is planning Now that you know how much data is being used you also want to know when you need to add additional Stores to your cluster so you don't run to the wall So we also give you a chart of an overall understanding of how utilize your cluster is So there is no surprise check in every now and then to make sure that hasn't gone to gone to the red part Okay, I'm gonna look back on my cluster right now So it has finished that Doing that automated installation right now. So it gave me back a claim URL. I'm gonna copy to the browser My mouse far enough I'm gonna paste this into my browser and it's gonna give me a claim URL and Something to keep in mind is the only thing I've done on this Machine through the limit through the command console is to type this curl command and that's it And I copy the URL out So if you are not that familiar with Linux or you're still learning about Linux and you're worried about whole can I do this? Yes, you can it's very easy Makes it quite simple for you to use so I'm gonna go ahead and claim the node It is going to establish a reverse VPN connection What that means is you don't need to open a firewall port out if you have the standard ports open and outgoing connections are allowed You can basically set up that connection automatically You don't need to go and ask your network command to say open port 9550 for me We don't need to worry about that. So now the communication has been set up. I'm gonna hit claim And I'm gonna add that extra node to my demo cluster and It's gonna ask me. Okay. So this is the host name. Here's the MAC address. Where would you like to put it? Let's see I'm gonna put it in Hong Kong to for today just for the fun of it And I'm gonna say this is gonna run the Swift node and Swift node is the one that has proxy account Container and object services all in one for demo purposes. I'm gonna leave it like that But you have also options of running it such as accounts container objects or the different Different options such as Proxy objects so it the different combinations are endless depending on what you want This is Swiss stack UI not integrated into horizon Not at this point. It's it's strictly within Swift can run very much as a separate standalone product So the dashboard is separate, but it will work with open stack other components as well to accept as a object store Yeah, it's a question that comes up Every now and then but Really here's the real reason right the real reason we don't is because the vast majority of people who deploy Swift are not open stack customers so we see a lot of Swift being deployed into VMware environments bare metal environments anything you can think of and so Maintaining that separation just kind of makes sense. I mean, there's some easy way to plug it in or something You're sure, you know, we look into that but But right now that's sort of the reality of the market, you know unfortunately open stack still tread and water right against some some Uncertainty in the market and so but hopefully we're starting to see that turn around now I mean, I think we're definitely finding more and more open stack deployments So so we should start to see some of these integrations get deeper over time. Okay before I keep going so Just as the time Adam was talking I just set up the network Interfaces Swift runs on three separate network interfaces. This is the outgoing which it talks to the application And then there are two ink internal facing communication, which is does replication and ingest I can explain that a little bit more, but we separate it out for administrator so they can Direct or modulate the traffic depending on their needs. Okay, so I'm gonna go ahead and Manage my driving on a sec. Here we go So I also formatted the drive and I'm gonna add all those drives into my storage policies So this one's in the Hong Kong. So I'm gonna account container multi-georeplication reduced. I'm gonna add this to Asia replica, but not anything else. I'm gonna add it to the policies and I'm gonna add it gradually What this does is it's gonna do the rebalancing of the ring by itself when you have a large amount of data to say 500 terabytes and you add into the 500 terabytes of Storage space into the system. It's gonna migrate Proximately 200 to 250 terabytes of data around if you say immediately it's gonna do that right away and Generate a fair amount of traffic on your network that your network admin may not like you very much at that point so Swift will it suggests that the best practice to do that a little portion at a time, but I don't really want to do it by hand. So I'm gonna ask Swiss tech to do that for me I'm gonna hit enable and I'm gonna enable this node And I'm gonna click on deploy to deploy that to the rest of the node and this is gonna run it will take a couple minutes And that's it for me for deploying another node to my sys stack cluster. So That's 10 minutes Any questions all of this too can be automated as well. So if you running Ansible is one that we use in house a lot that we've developed some playbooks for also works with puppeter chef We do have an automation Sort of a plug-in for this that we can enable that will allow you to use a configuration management utility to run through it Can you say that he wouldn't be heat directly he would call puppet right and puppet would do it and yes, you could do it from there Okay, so I already see the increased storage so the other questions Different monitoring I think I cover the monitoring the management so with Swiss stock it still runs on Swift Object storage engine itself. So has all the functionality underneath. We just made it easier to use. So you guys support multiple versions of Swift with this We usually tell you to upgrade to the latest because you want Yeah, just keeping it to the latest is probably the standard Yeah, we bundle a version of Swift with each release of Swift stack that comes out every two weeks and so You really have two options if you use our hosted controller offering You'll always get the latest thing all the time so you won't have to worry about having the controller updated It's going to be upgraded for you and you'll always have the latest stuff to be able to run All you have to do is go and say all right. I'm ready to pull the latest Swift down onto my nodes on the releases where Swift is actually Increased but The the other option to would be to be able to Deploy the controller to your on-premises within your your firewall doing that gives you two things One is the ability to just sort of keep it in-house be able to manage your own your own configuration But it also gives you the flexibility if you wanted to run an older version or didn't want to take every two week Updates you could have that option to do so Okay, so I think we're time. Thank you guys for coming if you got questions. We'll be right outside. Thank you