 Thank you. Welcome you all very astonished. That's so many people over here My talk is as introduced better bad self-efface for satellite data processing. How has this talk been generated almost exactly one year ago. I met Gregory and Sasha at the meet-up session here at this dev conf and They said they always like user stories and since we had one to provide we said, okay, let's do it. Let's do a Let's talk about what we use self-efface for since everybody always is talking about safe and doing the rather spark device thing and Most of them now also do the object gateway stuff But the surface at least how we feel felt that's not really much in Public so short words to me. I'm my name is Martin Spiegel. I work for Jaffa. That's a rebrand of catalysts game Behar individual software development in North Korea and yeah, I'm master of the classroom And I'm also not afraid of touching pay a P PHP sometimes I have basically three chapters for you including final words and I tried to find the sense of race from a song a very much like Imagine from John Maddon's old courtesy for that with him imagine doesn't happen the key word here is heaven And there is actually having a bath was and a lot of companies actually do something there So we have a lot of satellites there We have that's actually a picture of a satellite called sent in a 1a It's part of the Copernicus program of the ESA European Space Agency Which has some instruments and board in that case specifically their radar instruments so they're taking radar pictures and They produce a lot of data also airplanes which fly around and have here an RGB camera on it and here a laser scanner They basically Do something that's then called photogrammetry. So you take two pictures of the same area Roughly overlapping a little bit and then you can calculate out of that if you know the exact height and the Different angle you have you can do three D point Mesh walk clouds. We will see later something of that So and what did we start to do this data processing? We met the academic research group in France that Doing a cat algorithm that was called grass that's that generalized material of our soul and Surface properties. So basically what do we have in the air and that algorithm was apparently very great They did find much more to get much more out of the satellite data than everybody else did but they had one problem They were academics. So that's not a certain problem but academics tend to do at their great in Developing algorithms, but they're not so great In developing cold that's fast. So they had a quite good code base which worked but it took them very very long to process such files and then we came into play and we Improved the speed of the algorithm at the first stage about the factor of 100 so and Somehow that joint venture came started and then we met other companies which had basically the same problem They had a good idea. They had good techniques. They had good algorithms But they didn't have to know how and didn't want to build it up. That also came up with the time the knowledge to do this in a large scale so and Yeah, what then happened? We did some processings for USA and omets out and other stuff and You know to get you an idea what data do we process is here. These are concrete pictures. So the upper one is from Sentinel-2 image. That's actually an image of criminal Sentinel-2 has optical sensors in it so you can do basically for the graves and you can then use that Here we have a 10 meter resolution at the ground level To calculate something out of it. So you can try to detect particles in the atmosphere which fry over Brennan the other thing is here we have so that's the base material and Give you a roughly an idea of what we're talking about if it would take Every day of the satellite sentinel-2a mission That would be roughly of every instrument that satellite has on it. We would have about roughly seven terabyte per day So and what we need to do is so that we can process all that stuff We need to massively parallelize that so we built up also a compute cluster with a lot of compute virtual machines and they do the Concrete processing in parallel, but that also needs Some kind of storage which they can access in parallel so they can share that in between products Normally generate so normally it's something like we do a first round on some picture sequences Then there are some mid-term products We then shared that amongst the others because the other processes need that information to further iterate and gain speed and so we Apparently had something where we need shared storage The next thing is most of these algorithms even those we own basically so we have the We can influence the code They're not very object storage oriented So that's not that easy to Rewrite them to use that and there are also a lot of algorithms where we don't even have the source code for us So we can't even say yeah, well, okay Physis them accesses bullshit. Let's do it in object oriented way Which would be most of the times better, but we can't do it and we can't use it there So this wouldn't make sense to go that path So and here I tried to list from a s5 p-satellite So there's a sentinel 5b and a sentinel 3a to give you an idea What the difference is between what we have here we have that's for exactly one day in July 2019 this satellite produced 14 files exactly 14. We will see could you can see here They are all Basically roughly the same size all over the day and then we had another set the light the s sentinel 3a Also, just a random day in that case in the year 2018. We have 150,000 files and we have 150,000 files not only large amount, but also range from 40k to 30 megabyte something like that Why is that important because normally you tend to do optimization also on a Block size level, so you want to say okay, it would be optimal if all of my files. I don't know 64k then I could say and trim my strip size in the south of s to 64k and then that would be quite good That's not the case. That's not easily possible here so not only that we have Really a huge difference in the products they we have to consume, but they are also in The products itself quite different Before we go any further. I would like to show you some products. What what we make out of that. So that's actually This is a grasp Processing which shows three points in that's where is it the Cape of Rome Milo's and soccer What you can see here is the darker the blue the less the air is polluted and If it goes to red or yellow then air is very polluted. That's concrete for project where they plan to do Certain research Not a research Certain locations where people with a lung obstructions can Have a medical therapy and there it's very important to not have very polluted air So that's something very concrete or here we have the smoke plumes in the atmosphere and South America during the fires in The Amazon so you can see it quite good here. You have a lot of dust and pollution This is a thing the digital twin city of Frankfurt. That's something that's generated out of photochromatic Stuff, that's the thing where I said to overlapping pictures and then you can create Point clouds out of that. So here that's Still from the picture and here you can already Recognize that's the generated point out of that. We did that as you can see here for Frankfurt We did that even for Munich You can use that for what can that be used you can use that for development of Country areas or city areas you can do it for concrete calculations You can now really measure by heart that a point from here to that roof something like that forever. What do you need? So Another points in the challenge as I said before already Most of the time we need to process the data every day since it's such a huge amount and we wouldn't come up or To say with with that if you wouldn't do the processings every day We are not always in control of the algorithm source So we can't easily say okay, let's keep the file system part and use something else so that also doesn't work out for us Since I said we almost get seven or roughly about seven temporary per day into our storage We need something that's K level up to a very large level because Project did some time point in time said okay. I always need the last ten years of A lot of satellites and a lot of satellites mean a lot of data It's all because written so that's another thing We didn't want to buy I don't want to name some concrete vendors, but large boxes where we have pictures afterwards We shall be lean and Asia so it should be easy. So this was the main idea To change things So don't have buy a product where you can just do things just in one way and in the water so That immediately led us to it's easy if you try it Key word here is trying We didn't say well Let some vendors get us some sales representatives and then we just buy what they What what has the best price we said, okay? We want to evaluate it and what did we evaluate first of all is of course we had Think about classic or new age. What do I mean by that classic is for me the direct storage you buy from some Wenders on large and buy one two three four of them and live with the shortcomings they have The new age would be soft to define storage with all It's inclinations and we from from the beginning said okay We want something that is at least able to do block and file storage object store We didn't care so much about Just has to write an excess path in terms of blog and file storage low cost and any hardware is also fine because we didn't want to get a Vendor login we didn't want to get into the situation where at some point in time We suddenly recognize oh my fuck we can't own we can't buy anything else than HP or something like that We have we wanted the flexibility to access it in multiple ways and of course the scalability Was also very important fact What we took a look on so what we evaluated on the paper that really was on paper just reading through the dogs and trying to See if that fits first where last year fs. It were IBM spectrum care. That was actually gbfs code back then There were a cluster and okay. There was so that were the ones that Came up for us Then we had the discussion do you want something like close source that would have been gbfs gbfs better great Basically does it shop quite good as far as I'm told I mean it's expensive Yeah, you just can scale if you'll buy more licenses and these licenses aren't quite cheap But at least it does it shop. Why good, but then we said okay we everywhere else we build an open source So why should we start doing it here in our way? So that immediately led to the extinction of of gbfs Then we said okay, we saw there's exactly one product That was the main difference of self between all others at least for us which says or claims But the integrity is an vital part of the solution It's not that we say okay. We do the the distribution thing and stuff and that would have been last row also Okay, so but we and the for about integrity at the end of the day the rate controller is responsible or something like that That was the only solution that did that we needed posing compliance and the file system level Yeah, and that led to the fact that we wanted to use self what we did then as we said, okay Let's establish a plan. We at least let's try one more one other file system What I meant is then forget about plans life doesn't like them I Don't know exactly the time frame anymore after the roughly about two months of testing So we built up a test set up where we could Throw away everything in a few minutes and build up a new cluster. So that was what was done for safe and for cluster But at the end of the day to be honest, we just tested said It was okay, it worked out now we had some colleagues Which already did a cluster affairs before and we consulted there and then they said okay well for that What do you need go with that? So This is just Gregory thankfully already explained the basics of set for itself. So I don't have to explain that anymore This is roughly how does set a fast work? So we have a data pool data pool is a lot of people unit in the self construct, which has a crush rule behind it Which determines where objects should be placed? We have a major data pool and we have here the client and we have the things that's called Meet the data service. These are actually the ones which know where files Live so basically that's the idea To put it quite frankly This was just needed to tell you something else then we had the evaluation of hardware So what can you do there? You can read white papers. There are actually a lot of winners right now out there Which already have something like Yeah, that's the perfect self-solution We have we can sell you a whole safe rack for that and stuff like that that has Network suppliers like Melanox or so they provide you with a lot of white paper So that's our basic idea of an adult self cluster stuff like that I mean you can read through all that you can check the prices you have to find a way of trade-off So there's basically something where we said okay, we need a lot of storage. So this is one of the main ideas It's not that easy to create a safe cluster. Not Doesn't mind if it's just a block storage or something else, which is the Multiculti animal which can do anything so you have to find a trade-off between Performance between high IOPS between a large storage amount This is something you have to decide basically for yourself and at the end of the day What proved out first or the only thing to really work out is try it Buy or talk to your vendor if you can give you a pre-sale or a demo object and then try it. That's it These pictures you are here to show you what we have right now in the class is actually HP and super micro devices We tried both of them Yeah, as said before we don't want to get an event along in the work. That's basically it's it's not like that I would have say yeah, the one is better than the other don't care really much What we did actually as I said before we did a lot of load and performance test That's that's really something you should do and the most important thing is you should identify your workload So how will your access pattern be? That's crucial Because otherwise you might run into the situation where you say hey I have a perfect cluster and it's quite good spices to have a distributed one and your users say yeah Well, that's fine and we have a lot of storage but actually the performance. Yeah, it's not that good So what we did is we created an automated sweet Roughly can look something like that so fat is a Python thing which can Python SSH scheduler he can schedule jobs for you and different masterings executed over SSH And there we had as you can see it is just for one scenario a lot of fire jobs so fire is a utility which just can measure in a large scale and with a lot of threads and stuff like that multiple scenarios for reading writing files So that's what we did what we said, okay? That's what we want to use and what we also didn't just test it from one note We said well at the end of the day there will be 20 30 40 Compute notes which will access this file system in parallel, so we have to test it in parallel So we had to build up all that stuff and Yeah, and then run the test suite And the outcome of such test suite is then exactly such a graph where we can see yeah Back then to that point in time where from where this graph is is we had the class to build up with form storage nodes because we needed the other ones for the virtual machines to provide the Compute VMs And as you can see it's quite impressive already with a certain number of threads accumulate that you get around roughly 3500 megabyte per second Which is good because it was a 10 gigabyte internet network connection between them So no help below us How I wish that sentence would have been true a lot of times What were our challenges when we got that stuff into production first of all I have to say we Immediately switched basically with the final testing cluster into production. So we didn't rebuild it Not a real rebuild which I said, okay, so it's working fine. The settings are good. Let's get that stuff running Actually It would have been wise we would have done that from scratch But well it was like that because we already had some 100 terabyte on data on it We had one thing when we started all the trial phase We were in a data center where certain environment circumstances were not optimal. Let's say They had problems with the power supplies. They had problems with the air condition So that actually led to a lot of outage of our notes And the outage actually happened during a lot of ride and read scenarios. So and Now a thing to say we used we didn't really use redhead self storage We used the upstream for a project and we used it in a way back then when even their surface was not marked as Production ready. So it's not to blame surface for that. That's something happened back then But it just happened We didn't actually notice it until some point in time some colleagues said, hey, well, I did I did start a processing, but it didn't work out. There were some errors. Can you take a look at and then we suddenly noticed? Yeah, actually we have some loose leaves in our file system tree where we have a director in fragmentation Yeah, about half of the facts first of all They didn't care all too much because these we are as you can see perhaps on the director name something 10 files We didn't need them. So that was basically not that problem But we had somehow to deal with it because otherwise they got problems the colleagues which did the processing and It's roughly the half of that we could create could Fix with a ball with with safe provided utilities and the other half We actually really had to to kick manually some objects So we had to identify which object means that in middle data pool and had to remove it Of course during the middle data, so we'll stop and then start it again One thing then we also had to notice. Yeah, you should choose some settings wisely up front The layer tries a layer free size and range You will like to use for the front and network optimally in an optimal world You have a power front and then back in the network in a self-cluster The front of network is that that is facing to the clients. So also for surface. That's also true And we started with way too less. So we had to afterwards blow it up to a slash 18 we have right now that works until now Actually started with a slash 24, which after I don't know exactly more And more think about after one month we had too much lost for that and then we then Somebody had a great idea. Yeah, well, that's then that's the rocket Yeah, this doesn't work out in terms of performance very bad. So That's not the best thing Pull assignments to directories. So if you're like us and say, okay, you have different Important products. So have to in between products as said before you have to end product Which are very valuable because you Invested I don't know three months four months of processing time into that and you won't do that easily again Then they are very well. So we created some different pools where we said, okay here We have a higher application factor here. We can work with erasure coded things Different importance levels and we assigned these pools to different directories So there's something you can do in self-affirmation. You can say, okay this director or even this file should be in that pool That's quite cool. I think and then you mean can even do more you can not only can say I want the pool You also can define the stripe unit the stripe count the object size The thing only is if you do that and that Is used in a recursively ways or the sub sub sub director of this Directory would also have to set in pools FFS data as Long as I don't define somewhere in between the path at different setting for that The only thing is if you change it afterwards it Doesn't have an immediate impact. So if I would change for that directory Okay, now go to a different pool that would not happen by magic that in the background Somehow self manages to move all the objects from pool a to po b It's only applied for new files in there So if you want to do that, you would have to copy that file You set it on the directory first copy that file remove the old file that would work out Then you would have it in the file in the new pool Which is not quite easy if you have a few millions or billions of files So choose that up front you definitely want to have some monitoring So we have basically everything in there. We started with in scope. That was at the time of in final is something like that In scope is a project or was a project. I think they are not active anymore which had something like a quiet mix of just dashboards and Graphics and a little bit of control of the cluster also We don't use that anymore Then we had me established a home room for Mita's Grafana stack, which is still in use today Then there exists also the solution provided stacks since not the loss. I would say it's also usable Before that it wasn't it so this screenshot you can see is from the solution provided sex or from self itself That is is our main dashboard of Grafana for that thing And what we also do did is we integrated that into our chat thing. So we have everything rules in the parameters which Tell us if something is going wrong. So I just made a sequence here It says yeah cat is always D is dangerously full Or we also have something like the this compute node or data that storage node uses too much memory stuff like that That's quite fine because it directly get gets into a chat Channel where all of the team members can react to that your issue coding Yeah, the gross net ratio versus performance thing We definitely have some product trees or five system trees or directory trees in there Which we want to keep for a longer time, but we don't want to spend all too much in terms of Grow gross ratio gross net ratio on it So normally we would say okay for that importance of that we would do at least the Replication factor of three or four Some sometimes we talked about five, but then said to my project the other actually well You know what it means that means we have to buy five times the amount of storage You really want you really can use I said, okay, that's that's not that good. So we have to use a ratio coding Yeah Back then when we started with all of that it was not doable to use directly and a ratio coded pool for it some Saffa fast directory what we had to do back then was We had to add a cash tire Which is in in terms of again a replicated pool and that again have immediate impact on the performance of all of that since luminous I Think yeah, I think it was luminous. You can directly use a ratio coded pools But only if you're on blue store That's also important then you can use and we do that right now Then you can use a ratio coded pools directly for some Saffa fast directories Another thing we didn't start with we had to learn it also the hard way was you will like flash storage And you will love it At least for the meta data pools and the journaling if the journaling if you're on five stop But even if you're using blue store, you will love it to give you a roughly an idea They show measurements of a concrete Processing run when we had no flash storage for the meta data pool and we had That was the commit latency in average of 16 milliseconds That's quite high When we established the flash storage we drop that to 70 microseconds. That's a fact of 228. It's really impressive and you immediately know that is it it feels way more fluid if you use it under the Common line and stuff like that. That's really great Yeah This was something I was talking about performance influencing parts and we have something Well, it's a usage pattern in that whole cluster we call making a Holter Holter is the name of a colleague Which actually had the gift to bring the the cluster to its edge, so Inside a joke and we had to react on that and what we did was simply to separate Performance influencing parts away from from other stuff. That's important. So we created different crush rules Which make a separation between block devices and surface We also will have different crush rules of different pools with different crush rules for whole directories of the users on on certain Virtual machines whereas the satellite data and that's what you can see here basically so we have the surface data pool That's crush rule zero which should know that's not that's the default rule still Then we have said if it's made a data that's crush rule three. It's that one here where we use a replicated pool based on enemy storage and This notation here is a class notation which tells the crush algorithm to use only devices of that class and then we have again here for the Rattles block devices for shift for all the shift instance where we use it, but also for the center volume since we are also using open stack Crush rule four. That's that one here where we say, okay We separated some heart is some always these just for that usage so that the poor performance impact or When somebody does a Holter on the file system doesn't impact the virtual machines I know is that Maintenance, yeah You have to deal with version updates and upgrades the cluster size changes. We started with one petabyte We're right now at three and a half petabyte Next round is coming this year. So then it will be something about four to five petabyte It's nothing you should take Easily and you should think about it at all. So if you just would do it in a plain way get the new notes Stuff it in start it up. Your cluster will immediately work and it will everything will be fine Basically, but you will notice performance impact. So we did that in a steered way so that we can control During night relate we use the cluster balance also for that the up-map feature so that during nights We're not a lot of people are working there except the processings that that are running We can do the rebalancing and stuff like that another thing We had to decide a little bit is to rebalancing per placement groups or by by usage the volume usage it would be ideal if We could use by placement group. We can't actually use it since we have a Quite large gift then of the usage on this on certain OSDs Yeah, then we had some out there chess I told you before about the The things with the power supplies and with the cooling systems But beneath that we also had a semi broken network cable that was really nasty Since it actually was working a little bit But when you put load on it, it suddenly started to generate media errors But not in a way that you would have seen it on the TCP a P stack You just could see it in some certain proc subsystem assist subsystem of the current where it says the Manamox OVD thingy starts to count up the impact on the performance was was was really large We had some active MDS and crashes again not fall of the product itself This was during a time when it still wasn't marked a stable and yeah We had other problems with the hardware to disk sprake mine not like that But every you have to be aware and rotating this Will they are used quite heavily and at some point in time they will tell you now I've done I'm at the end of my lifetime. You have to replace me General half of it so our advice is if you start to do something like that and that range and that scale Have spa spare parts at hand It might be that in two or three years when your clusters is running And at least if you're like us and you didn't buy the super expensive window stuff and had Not a support contract with ten years with 24 seven stuff like that. Then you should put some spare parts Beside and lay that stock Yeah, what still do we have here. Oh, that's a picture for friends like it Blue store rooms It's another thing that's that's quite important. You want to use blue store. There's no reason at least we believe so to use file store anymore Sadly these graphs here. I have not the same scale factor here We go from zero to 250 and here we have from zero to 12,000 something like that That's the maximum latency we notice these three nodes are still file store nodes during a concrete processing run Where ZFS is heavily used and these three nodes are already on blue store So you can see we also have a fuck factor of 100 something like that. Oh, it's not 100 But very high So you definitely want to use that use cases I Can't really tell you how much time I spent talking with all the guys who will use that cluster in a specific set of s In the next time and how they told me yeah, we need that we will have that workload We will have that access pattern and we will need that was the best thing. We will just need that much in storage forget it, I Don't know if they do it on purpose or not, but it's basically a lie to you a few days ago we had an appointment in an academic environment where the IT leader guy there was telling us Give an academic guy some storage. He will fill it up in no more than two days and second thing is you will not be Satisfied with the performance he gets so that's the bottom line you can take with you Always think a little bit Even for them and another thing is of course also the use cases evolve over time. We had a Since one and a half years something like that We're working with a company from Germany, which has also acquired a good algorithm doing in this photogrammetric stuff But as I said before already they have some really weird access pattern for example, they have some What are four hundred bytes in size control files Which are dynamically created only at the beginning of a processing run during the processing run these days The same but what they do is since they we have performance their problems They under analyze them and the song well actually They are opening and reading these four hundred bytes during a sequence of ten seconds twelve thousand times well This is not really good on the on the file system of that type Last thing is here security implication and consignments of self-efface here I would not actually use it as a public file store for anything There is what is the thing that I know is us the most is the loose coupling of pool and pause explanations So if I have I am the user who has basically some kind of a keys So there is an access management and staff that allows me to access the pool a Then I can if I know what I'm doing read anything in there at the end of the days all the data that is in there Objects and as long as I'm allowed to read the pool which I apparently have to be otherwise I couldn't use it as a suffice is then I can read also data Which I should not be able to read in terms of pause explanations So that's that's a thing. That's not that good I'm coming a little bit to the end of both us on this guy is the current solution So what do we have right now? We have as I said before open stack It also uses the same cluster they are apparently we're using the block devices Why do we use open stack because we have quite dynamically be able to Throw away all that compute virtual machines and regenerate them This works quite well with open stack. We have some terraform receipts in place for that So to give you an idea of 272 Processing virtual machines which actually take half of the cluster resources and that's already something about 1,700 CPUs and 9 terabyte of Random access memory Are generated in five minutes? So that's quite cool since we can provide the colleagues which actually do the process things a way to say okay for everything The way let's start from scratch and you can do use some number of algorithms stuff like that Yeah, we use the ansible thing for Setting up open stack on itself then we use as well also for provisioning or setting up the Virtual machine after the terraform thing has created them You're using form and for inventory management and bootstrapping the whole note. So there's also a thing we can Get a new note doesn't matter if it's a storage note Or it's if a carpet note or something else into our cluster system in roughly about 10 minutes So that's the time it takes just for formatting the disk and setting up the operating system and stuff like that And then we have our colleagues that created a glue cold user interface the processing portal Which that's some quiet unique features I think it can integrate into other independent computing facilities So we have Companies which are part of us which do the same stuff basically as we do on different Products so they're using GPFS or and and we emerald stuff like that Whole thing is container-based Slurm is used slurm if somebody knows it is a job scheduler on the in the Unix world Largely used in an academic environment and Yeah, the thing with the different points of files already told you before That's how you can see a this Stranger to have here because we can see here one thing what all of this Portal does it's heavily really heavily uses the set file system So under the mount point slash media slash slash scratch. That's actually set of s and CFFS is because it's working that good was released also used to do the orchestration stuff in there, so That really works out. Yeah, you get nice graphics out of that Yeah, you may say that I'm a dreamer We are at my final words basically This is a picture of the cluster in the upper part So exact most exactly half that thing here is the self storage and the other half is the compute nodes Resiliency we survived really multiple outages with almost no interaction immediate manual interaction necessary So there were one through season power strips. There were the cooling system failures There were discs dying disc crashing OSD's We've had once the the part where exactly in half of the cluster since we were talking about Front of stretch clusters was kind of way because the power system failed Of course at that point some point in time. It didn't work but the cool thing about it was We didn't have to do anything after park came back I don't know the time from exactly animals. I say something like two or three hours something like that everything was green again No admin doing anything I Had a similar situation at another company where we had the classic old school rec storage solution They had to work five days until everything was good again so that's quite impressive and Both does that resilience is definitely a factor. Another thing I want to give you on the way is documentation Don't just rely on the documentation. I mean staff is a really great product The documentation does not really reflect that in my opinion But I told you that last year I mean On the other side to be honest a lot of people always claim that it's better to use open source software Because you have the source and you have the power and stuff like that and how do they use it? Do they ever read one line of code? So we have to take ourselves on our nose and say okay if you're using that when we claim it That's cool then really use it and Don't blame the dogs if there is something working different than it actually should I mean, okay, dogs should also be up to date, but yeah, and last thing is don't underestimate the complexity It took us quite a while to really tame that beast In that scale mean the fellow world thing is done in a few minutes. That's working. It's cool But in that scale you will most likely have other problems Yeah, since I like to And my talks always with some meaningful phrase and you might already have noticed I'm a little bit older. I'm from the last millennium and I also like the old stuff and I thought that a little bit a thing a phrase from from the song Imagine and the storage will work as one So that's it from my side, thank you for your questions. I hope I can give answers Sorry Okay, the question was do we have any background of the data actually to be honest no That's by by concept. This is all we've said If we would really talk about backing up free thought so what the satellite data actually is roughly is free thought one Peter bite right now It would not be restorable in a sense full amount of time So you have to trust in the system itself and choose your settings your reputation your ratio coding stuff in a way That you're safe and the failure domains and stuff like that So it's we don't use a failure domain of always deal ever right now We are still on host level and the next iteration will we will move the field domain to rack level Then it's gone So that's the this year. We will deal basically with that so Tom on a Fold question was what happens if somebody accidentally delete something Yeah, well, there is there is stuff you should not be you should really not delete since it's important since I don't know six months of Possessing are in there These stuff is put into a place automatically where it's just read only for everybody else except a few That's the way we deal with it mean We could also talk there. Yeah, we could talk about doing a classical backup system. You can use a burials or I don't know How all the others are cold? There's the company which now owns redhead which also has a product in place the storage manager Backup manager. She will she will read something. I don't know where they want With that amount of data, it would be possible since we're talking there about roughly 50 terabyte or something like that but yeah Everything else this simply won't work out What exactly so the question Okay The question was what is exactly our hardware specification and if there is a server with only and we meet drives Right now we don't have us. That's the first easy part. We don't have a server with just and we meet drives We're right now in right now actually Organizing them But they won't be just and we yeah, they will and we mean you to trust it will be so yeah Right now we are thinking about the pure flash storage part of the cluster since we have some workloads with databases and stuff like that where we Definitely can benefit from the high IOPS thing of that and the cool thing is again We still have to see if it really works out in a theory. It should work out to be let that be also part of that cluster For the other nodes. It's a set before we have basically get roughly the same to the storage nodes are 256 gigabyte of memory CPU with two CPUs with 14 real cores, so that's 28556 vc bills with around 2.7 gigahertz something like that And the one notes on the left side we saw have 24 discs normally roughly about 10 terabyte And the other nodes have 36 discs. You shouldn't put really more than 36 and that's also only for a cluster with Which is volume-based should be volume-based Yeah more than one it was but I don't know exactly the calculation for my anymore quite hot at least Yeah Yeah So the question was what has this qualified cluster fs Well We as said before one demand was to have different access paths and we really Wanted so that was that? hard demand a block storage and file storage and At that point in time we didn't trust the block storage component of cluster very much So this was the disqualification basically and the other thing was that we also said, okay, we're getting We're getting the files is there and not the files is then the swift S3 compatible layer for free So yeah We will definitely need that some point in the future now. We have open shift for very much We were very much using that we even use it use it don't think after one week of last year's death com So we hacked it together that it somehow got installed and then we will need There is a colleague out there, which already should be working on that that we have the S3 part running in next week or something like that So, yeah, it made sense So some more Fine, then thanks for my side still have a great