 So let me get on the grandstand. Hi there. I'm Turk Peterson from Seattle Fred Hudson Cancer Research Center. I will introduce myself today. There's no person who introduced me. I will introduce Bobby Minier from Swiftstack. We're both going to give a presentation about how to bring an object storage system into a real life research environment. And Bobby plays a key role in one of the access methods we use to store this system. So let's just get started. So what we'll cover today is who we are, why we did what we do, what we did, what were the requirements for our project, what else did we try, what we deployed, how we manage, what it costs, and how does it really work in day-to-day life. And what was the timeline? We saw that in implementation projects a couple of times that I assisted. The timeline is really something that in cloud goes from sometimes it's a couple of weeks, sometimes it's a couple of months. But we are talking enterprise, and that is often measured in years. So let's go ahead. What is Fred Hutch? We are a cancer and HIV research place. We have about half a billion dollar budget. We are largely funded by the federal government. And for the purpose of our talk, we possess multiple data centers on one campus. That's really important here. And we also are subject to this explosion and genomic data sets. And we introduce charge bags. So the charge bag is a researcher has to pay. They get a certain level for free. And then they have to pay when it exceeds a certain threshold. And that's what we implemented in July 2014. So it's really important for this talk. So why did we do it? Our researchers were faced with charge bags. They said, OK, we made up NAS charge bags. We didn't make them up. But we came to the conclusion that NAS charge bag would have to be $40 a terabyte a month. And a little bit is free. But when federal grants are declining, you probably heard about sequestration and everything, then that's a really tough nut to swallow for researchers. We had a lot of discussions about it, whether that would be feasible or not. And at the end, we came to the conclusion that they said, OK, if you charge us, you need to give us something really cheap, really inexpensive, where we can store older and also bigger genomic data sets. And of course, there's always this misperception about the drive from Best Buy. I always wonder if there's another company that sells like cheap drives that Best Buy is often mentioned as an example. Finance, we maintain very good relations with our finance department. And when they see that they have to flip a bill for some storage system, they're not necessarily concerned about the cost per se. But they're really getting worried when you come up with like $500,000 or a million dollar at one year, and then a couple of years later, you buy another one of these things. And they say, OK, you have this thing here with lots of yellow lights. And now you bring on something with green lights. But in reality, they all live side by side for a long time until you have your migration down and everything. And so they don't like that. And they know how to use a web browser. They go to Amazon, and they click a couple of clicks, and they see what storage should cost. And if that is fantastic out of line with what you want to offer, then you have a problem. So then we had, of course, the usual stuff that enterprise have always problems with data protection and archiving. In many cases, data protection is an afterthought. Why do you buy a fancy new storage system? And then you come to the company, no, you also have to pack it up. And then the backup performance is not as you expected. So that we've lived through several cycles of that. We do have a very nice backup system, but still it runs into scalability issues. So we did look for a solution, but we were not necessarily looking for an object storage system. We're going to talk about object storage today. We're going to talk about Swift. But we were not necessarily looking for that per se. We didn't know about it, actually. So users wanted a mountable file system of any sort. They wanted to use their Unix tools. They wanted to continue to use their standard Unix tool. And we also had this thing that you really do this for only a subset of your users. 90% of the users are just fine. They can use any storage system. But the 10%, they're really the data users. And you have to be careful that if you invest larger amounts of time and effort, that you not focus only on your 5%. So performance needs were not too high. We were very nimble about that. But we wanted to grow about it. And we had concerns about disaster recovery. Seattle is a place where earthquakes can go up to 9.0, like the one that we saw in Japan, potentially. And that gives us some worry. So we wanted at least the first step to replicate our storage to multiple buildings that are a little bit further away from each other. But in the next phase, we wanted to get out and maybe do a DR site at the other side of the Rockies or something like that. So a couple of key characteristics. It was like permanent static data. We wanted like POSIX permissions. It's supposed to be optimized for economy, supposed to be self-protecting. We didn't want to have a dedicated backup system for it. So a project, our architecture team typically like dissects every project in like must have, should have, nice to have. In our case, there were lots of like must have. There were some nice to have. But one of the nice to have was this thing that we wanted to do like a broader storage strategy. It wasn't necessarily a hard requirement, but we had really the idea that, OK, you do this for like only a subset of users who use like POSIX file systems. And I wanted to like jump the genomic data in there. But it would be really good if we could also like leverage it for other things in the future. So object storage, scaling, manageability, no rate systems that sounded like that was like kind of like music in our ears. The resiliency, but also predictability. We ran often into performance bottlenecks when there's a rate rebuild in our storage system. So the perceived predictability of an object storage system is lower performance in our requirement, but at least that low performance is very predictable. That was something that we really wanted. So then we heard about Swift and Swift Stack. And at first we said, oh, wow, it's proven. Some really big clouds use it. They have, I heard rumors that there's installations up to plus 50 petabytes. So that's more than we ever will need. But if we want to get there, that would be great. Manageability, low cost, longevity. We have some requirements where we store data like for 10 years and longer. And some researchers want to store it for 20 years and longer. And they make up, or they define those requirements. And we deliver then the technology to serve that. And if you have a proprietary piece of equipment, storage equipment, and you have to do these upgrades, then that's really not longevity. So with Swift, the promise could be it's open source, it runs on commodity hardware. You might be able to run it forever. Forever and ever. And then the durability part, data centers fail. We can have, we said we can have earthquake, we can have fires, we can have disasters. That was an important component there. So conceptual view of how this will work. We have like three, basically three storage buckets, or we call them zones, and each of them in a dedicated data center. And then on top of that, there's a SIFSA NFS gateway. So traditional file access to serve our standard community. The research community we saw that in the previous talk is rather conservative. And a lot of object storage systems are not like brought to the end user yet. So we need to have some bridge technology to serve them in the meantime. There were previous talks about Swift, so I may not have to explain how it worked, but in general, the Swift system has three replicas, or three replicas are recommended, and they're stored in a unique as possible way. So it can either be like a single node, it can be a small cluster of multiple nodes, or it can be like multiple data centers that are distributed in multi-region across the country or across the globe. And so looking further into this, we really felt like, ooh, look at the underlying stuff. What is Swift? Linux, Python, R-Sync, those are all tools that we were already using. And we said, wow, the first concept, just bring it up. It looked really easy, it looked really great. And so we felt a lot of comfort in the resiliency here. Here's an architectural overview of where we see this tier will fit. So we have, typically we have a POSIX file system, this is an Isilon scale out NAS system, then we have some high security stuff. It's basically a fully audited NetApp system. We have some block storage here. Block storage is rather small, so in our case, it doesn't really matter what we use. You could actually buy anything there. And here's our economy filed tier with Swift. Swift stack, right? And so this is like $40 a month and this is $3 a month. So this is gonna be an enormous economic pressure to move as much data into this tier. So what were the other, I often, or you'll get this asked, what were the other options that were out there? We were a long-term user of Gluster FS for scratch spaces and high performance computing. We did really like it, but when we looked at this, there was kind of the different level of resiliency that we wanted and we didn't really have any proof of scalability. We also, the replication features that were in Gluster is probably now more upcoming, but at the time, they were not really widely in use and we couldn't really find any production deployments that did that. And still you have the rate that you have to manage underneath Gluster. I'm not sure if that's still the case. Probably it's still the case. We looked at these magic HSM solutions where basically you take a file away and replace it with a little stubby stub and then when you hit the stubby stub, nothing happens for like half a minute and then it tries to get it to the tape from the tape system and then move it back. And we've had that in the past and it really didn't work so well. And it also required a forklift upgrade at some point. We also looked at several commercial vendors with who were touting erasure coding. And erasure coding is really good at first, but if you have to sustain a complete data center drop out, then the erasure coding algorithm has to be more conservative. So you're losing some of this cost advantage that you initially had with it. I'm also often asked by folks from the open source community how does it compare? And why don't you use, Seth, we saw this talk before. It's very similar use case, high performance computing, some genomic data. But in details, perhaps a little bit different use case. And Seth was more targeted towards that use case. In general, if you just want an object storage system, you can, for example, look at the lines of code that each of these projects has. And you can go to OLO, which is a website that analyzes lines of code of each open source project. And you see that Gloucester is really 10 times as heavy. And Seth is still heavier than Swift and Swift is a really, really, really lightweight piece of code. And there are some people, I'm not saying that, but there are some people who say that the numbers of errors per line of code is always the same, no matter what you do. And if that is true, then, you know, there would be not necessarily more bugs in a system that is like heavier, has more code. So it's just a, you know, that's just a developer view. And not everybody, I think that's plenty controversial and I'm happy to discuss this further. So what are the public cloud? We are about like a quarter mile from Amazon AWS headquarters. We know lots of people there. You know, why don't you just dump there? And quite frankly, yeah, that's what we will probably end up doing at some point. Amazon is in use at the hutch today. And you know, people like it. There's still some fuzziness about government policies. You have to have business associate agreements. And so we are moving slowly towards that. And we will also be able to prepare for this day when it comes. At some point, we believe we will live in the public cloud. It may be like 10 years from now, 15 years from now, five years from now. But by actually using Swift, we can use the S3 API to adopt applications and programs to actually use S3 compatible ways of storing data. And that means that when the day comes, we will be able to switch over right away without any or without lots of hassles rather than having to migrate all like legacy POSIX file systems to the cloud. But there's still some truth about the cost here. So we've just seen recently some cost reductions at Google and Amazon. We see our NEST here is about like $40 a terabyte a month. And the public cloud is in the neighborhood of 25. That's without transfer cost, it's just storing it. And Swift, TCO, backup, everything included, except for the data center, which we already have, it's $13. Including energy cost and everything. So as long as we don't find it less expensive to move to the public cloud, there's no reason to do it. And maybe the day will never come and then we stay there. In some cases, we will need to storage Switzerland. That means people who wanna share data with us don't trust necessarily me to store that data when they're not in our institution. They want like kind of a Switzerland and then they will use the public cloud. But for many types of data, that's not true. How are we doing on time? Okay, we're good. So what we deploy, we deploy one zone per data center and we have the Swift management, the Swift stack management node in the cloud. And that is something, so here we go. So here's the bird eye view on Fred Hutch Google Earth. We see there's three zones, zone one in one building and they're about like 1,000, maybe 1,500 feet apart and zone two and zone three all in separate buildings. So if there's an earthquake and the highway falls on this building, we still have these two available. And that will be without any downtime. So we would be able to sustain a building failure without zero downtime. An architecture view, again, three storage zones. There's a low cost switch in each zone and then that switch is hooked up to the data center switch and it's hooked up to the building and the core router. So this is how it goes through the entire campus. Some building blocks. We have a super micro chassis. It uses desktop grade hard drives and for about 130 terabyte usable space, you pay about $13,000 plus tax. This is a discussion that for cloud people is, because why would you use anything else but consumer grade drives? Google published this paper in 2007 that said, those drives are just fine. They fail a little bit more often or they couldn't even prove that they would fail more often. They found them reliable. But for enterprise folks, for enterprise storage groups, that is a tough call to say that you're using some quote unquote junk drive. But the folks from Backplace, that's an online backup company, they really have collected some data over the last five years and they saw how actually does it look like when failure rates go up? So here's at year three, you see that these drives typically seem to fail more here. Five and six are projections. So they assume that it will go there, but at year three really see an increase. So enterprise storage, people say, oh my God, we need to replace all these drives and the sky is falling in year three. But cloud operators say, oh wow, after six years, you still might have 50% of the drives in the system. That's great. Continue to use them. They're not broken. They work. So that's a little bit of a mind-change in the enterprise. If drive failure rates are in the name of 8% per year, which is pretty high, we estimate that at the end of the build out at three petabyte system, we would have 20 drive failures per month. And the beauty about SWIFT is that you can just replace them all at once, like once a month. You don't have to run because it's not a RAID system. You'd have to run into your data center full of sweat to replace that one or two drives. You just wait. Over time, the storage capacity shrinks. And then when the day comes, like on the first Monday every month, you rip out the old drives and place them with new drives and you're good to go. And you could actually replace them with a bigger drive, which has an impact on your storage costs. So if you had like three, two terabyte drives before, you can suddenly use maybe four terabyte drives. This is the Netgear switch component that we use, $1,400 for a full 10G switch. It has about, we did some performance testing. It has about 75% of a Cisco switch. So it's pretty good. It took some time to have the Enterprise Networking Team getting familiar with it. At the end of the day we came to the conclusion it's just like an unmanned switch. It's a throw away piece. We put like one end storage and then if it breaks, you just throw it away and replace it with something else for $1,400. So basically, I think it's the price of a Cisco G-BIC. So that's not too bad. Architecture view. So here you see the super micro systems, the Cisco, the Netgear switches. And then you have the data center switch up here. And there's a NFS SIFS gateway. So the totals on storage is like 260 usable for $83,000 and the monthly cost is about $5. So how do we manage it? The initial deployment took about seven days. What's really important is that you have to make sure that you have a vendor like Silicon Mechanics, for example, who offers you these desktop grade drives and drive carriers. You can't ask your storage team to get out a screwdriver and put them into drive carriers. I think that's too much to ask. And then it's about 0.25 FTE. We calculated that. It's perhaps a little bit less. So open source Swift is functional. You can just deploy it. It sounds all very easy. But there are a couple of things that are really not so easy, like maintenance and upgrades and just the general monitoring. And we've seen people who have actually used like an FTE or one and a half FTEs to maintain a Swift environment. And perhaps it's even like large or maybe three or four FTEs in some cases. So that's why we came to Swift Stack. We said, we are not people who could do that. We want to keep the cost low. We can't just have this storage adventure at our place and then not pay anything for hardware and software, but then invest like two FTEs in maintaining it. That would be rather absurd. So we said, we need to have something that does it for us. And so then Swift Stack basically came to the rescue. Provides a cloud-based management console. You have enterprise features like delete protection. Really important in standard Swift. When you delete your file, it's gone. And if you give that to your end users, well, what happens if you don't have backup? You can't get the file back. So Swift Stack actually offers a middleware layer that puts files that are deleted in a trash can for 60 days or so, and then they're expunged afterwards. So that's a very critical enterprise feature. And then the Swift Stack controller lives in the cloud. So you don't have to upgrade that. You log in and sometimes it's there's new features. They're just popping up and you use them. And it's a Swift Stack is a real 24-7 global company, which I didn't know initially. I always wanted to, at some point, the staff was answering email at like 2 a.m. in the morning and I said, oh, you know, he probably doesn't have really good sleep. And then he continued doing that. And I just wonder what that was when he was actually in Taiwan. So Swift Stack provides all this control and visibility. You can completely monitor your cluster with almost every detail. And the rollout with Swift Stack is literally 10 minutes. You have your Linux boxes up. You have them in front of you. You type a couple of commands. They are all there and it's like 10 minutes and you're done. Then the one feature that really made us like, we were just like speechless was the upgrade. So we had a full blow of like files right into the cluster, you know, multiple hundred megabytes per second to it. And then at some point, an upgrade button showed up. And you can just like this, nothing more. You just click this upgrade button and silently it upgrades the entire cluster to the latest version of Swift. You know, not a single ping missed. Nearly full performance. This thing is degrading like by 20 something percent. But that was really a spectacular experience for us. Upgrades otherwise eat like lots of time and we have like project managers involved and whatnot. It's really intensive normally for storage upgrades. So that was really good. Swift Stack Controller. I mentioned that. What's the difference between like Swift and Swift Stack? So the Swift Stack Controller provides these features on the left. There's also a runtime on your local node and you know, you just have open stack Swift. That's the standard stuff, Linux and your standard hardware. Pretty straightforward. So now I'm gonna talk about one of the most, Bobby's gonna talk about the most critical component in our case. We said we can't use today, the objects store directly most of our users are, you know, use legacy NFS and SIFS. And Bobby's gonna talk about this new nice technology that's available today. Thank you. And again, I'm Bobby Maneer, Director of Technology at Swift Stack, specifically focused on access methods to the object store that are not the API. And have been working with Fred Hutch, specifically on the SIFS and NFS gateway. And just gonna give you a little bit of a quick, you know, technology deep dive here and just spend a few minutes talking about it. So if you're not familiar with a gateway that it's effectively a means to getting data into an object store other than using the native object store API. And its role in life is to effectively lower the barrier to entry to be able to get data into the object store. And use familiar IT workflows, paradigms that your operation staff, you know, sort of know and love or hate, depending on who they are. But just allow the object store to be utilized without having to take the time to immediately start recoding applications to use the native object store APIs. And in fact, in some cases, you know, we've specifically heard from customers that, you know, they have certain applications that they're never going to rewrite. At some point in the future, that, you know, that use case just goes away but the data persists. And if you look at the spectrum of gateways that are out there in the marketplace right now, I like to really kind of come up with a taxonomy of two types here. One is, how does the gateway effectively treat the data as it's being ingested into the object store? There's certainly a class of gateways where they're taking the data, the files that you're wanting to upload, chopping them up into, you know, very small segments, potentially performing some compression, some deduplication, and then shoving all of that, those little blobs of objects into the object store. And, you know, it's a very efficient means to do it. It, you know, reduces the amount of data you actually have to store in the object store. But effectively, it's now taken your data, your objects, and has, you know, morphed them into something where you have no idea what form they're being stored in. And, you know, what, you know, I like to talk about with, you know, the gateway that we provided is, it's a no blobification that we tried to preserve, or we didn't try. We have preserved the one-to-one mapping of the object as it's, you know, produced by the applications and how it's stored in the object store ultimately. Because that's just, in our view, hugely important for mixed use cases. So, you know, you don't want to be able to, you know, push something up in, you know, via one of these file system gateways, and then want to later access that data with a different application. And, you know, you don't want to have to backhaul that data through the gateway in order to then, you know, reconstitute the data and make it accessible. You just want your, these new applications to effectively be able to hit that data in the object store and utilize that without any other piece there. So, effectively, that is, you know, no technology lock-in. That the data that you put in the object store is exactly what you know and expect. And, you know, I think that point gets lost on people a little bit, but it's just something, I think, very important to consider. Another element of the gateway that, you know, it's an ideal point for, you know, possible data transformations or data extractions. And there's a lot that you can imagine in terms of having, you know, a multitude of gateways deployed that are all, you know, pushing data into the object store. And they can perform these types of extractions. And I'll talk, you know, a little bit more about the metadata extraction in a minute. But basically give you that consistent view and enable different workflows here. You know, one that Dirk had mentioned or at least had on the slide, was this notion of write, want, read, mutt, many. So the gateway, again, is giving you a control point in your storage network, whereby you can ingest an object and then coordinate with the other gateways to make sure that that object is then protected from future modification. So I'm just gonna touch really briefly here because this is obviously a case study about Fred Hutch and not on, you know, gateway deep dive. But just some of the ideal use cases that, you know, we see for a, you know, non-blobifying gateway, I'm sure I'm gonna patent that at some point. So one is active data archiving. So when you're pushing tons and tons of data into your object store, one of the things the gateway can do is extract metadata that's inherently embedded into the files and associate that in the object's meta tags or even potentially vector that information off into an index server. So that you can form these catalogs of the data that's in the object store without having to go back and try to pull or, you know, search the object store in ways that are not currently exposed, you know, in the API. And, you know, I think that's something that, you know, we've started here a lot of interest in. And really the key is having an automated way of being able to pull that information out of the files and get it directly into these servers. Touched a little bit on this, but multi-site data sharing, whereby you can have these gateways deployed at all your remote sites. You have your one core centralized object store and you want to have a unified view, again, via SIFS and NFS, your standard file system protocols. And the gateways have the ability to, you know, coordinate their file system view of the information stored in the object store. And then anybody in a remote location can access these objects just like they were coming out of an AS file system and can do some specific things to improve performance there. And then finally, if this ever clicks, maybe. Data migration. So because the gateway is providing a SIFS and NFS mount point, it's just very easy if you have large amounts of data that you want to start pushing into the object store, just mount, you know, a container as a mount point via the gateway and just start a copy job or an r-sync job and it just gets all of your data up into the object store without having to, you know, fiddle around with, again, you know, tools or anything else. It's just, you know, emphasize that the IT workflows are preserved. The IT shops are very, very experienced in dealing with gateways or with file systems. And we'd like to just preserve that functionality. So where do gateways go? Ah. You know, right now the gateway itself is utilizing the object store APIs directly, again, saving, you know, the potential for having to rewrite the application right now. Ultimately, I think there's better performance that we can get out of the gateways with tighter integration with the cluster. It's something, again, we're working at, you know, very hard at Swiftstack and are going to continue to look at these ingest points for other types of data grooming that can be done on the objects to, you know, open up the potential of, you know, accessing the metadata and being able to, you know, work with the data directly. All right, thanks. Super. So let me just... Oh, if your phone was a pointer. What's that? Oh yeah, that would not be good. Five minutes. Okay, so then let's run through it. Oops. Okay, so one more. So with the assistant NFS gateway, I said initially that our performance requirements were not very high. We actually have achieved about 300 megabytes per second. And if we use, you know, the native Swift API, then we can actually get gigabytes per second throughput. So that is really good. We have enough performance to get started and we can grow later and we can have people who are in the same research group who use Swiftstack NFS. And then we have advanced programmers who can access the really same data. And that's kind of unique. If you buy like a standard file system gateway into an object storage system, then you really wanna drive that home, then you necessarily can't access it in different ways. You have to either choose Swiftstack NFS or you do object. And that's really the benefit, the long-term benefit that we see here from the system. We have a couple of limitations. Because you map object to file, it's more like a Swiftstack NFS bridge, if you wanna call it. There are some limitations that you have to be aware of and you have to discuss with your stakeholders. And some links are not available. That's for some people, it's tough to swallow, but for most folks, it was not a problem. You can't easily rename directory structures. You need to copy and delete them. And the renaming of the directory structure hangs other IO in that certain directory. So for most use cases that we looked at was an elimination at all, if you look at it really. And the DU command doesn't work on Linux. So we wrote a replacement for that to make that work. Overall, we've really implemented with Bobby's help, basically everything that's needed for an enterprise. Like we have like esoteric POSIX permissions that we can do now. We have, Bobby mentioned that Active Directory integration has been worked on. It's improved, but it's production ready today. We use Active Directory integration today and it works. So a little bit about the timeline. We started two years ago. That's for an enterprise. It's probably appropriate for many other use cases. You will find that amusing. But two years is the time in that we took the initial evaluation of Swiss sector only two days and said, hey, it's a good tool that took us two days and implementing it two years. Lessons learned, we really need to communicate with everybody at all times about pros and cons and limitations and what the system does. And you need to be ultra responsive during your test period, ultra, ultra responsive. Then you get some buy-in from people. Prioritizing requirements, do you really need symbolic links? Is that a requirement? And if it is, we need to work on them. And if not, then we put that at the back of our priority list. Oops. So tape backup is a big step for people to give up. Very big step. And with SwissSec help, we really think we pull that off, that we can have data secure long term without any tapes. That's a big thing. So I mentioned that already, the POSIX gateway. And I have some things that I want to do in the future. Where do we really land? Or what's the next step? Once we have implemented this use case, I mentioned initially that we want to use it for other things as well. And we have an endpoint backup. There's a product called Druver. We use CloudBerry backup. We can use that. Then we have, there's a high performance POSIX file system front, and it's called Avere. That's available. CloudBerry drive, expand drive. There's all sorts of tools that we want to investigate in the next year. So can you fire it up again? I'm sorry. I just want to mention this as a final comment. I should probably not push too many buttons here. Shift F5. Oop, no, leave it there. No. We're also working with Seagate on their new Kinetic platform, which is an Ethernet-enabled drive that really allows you to use a hard drive as key value store. Basically, you can remove several layers from the Swift architecture and just store data directly in the drive. I mean, you saw initially before that I was really a big friend of Nimble systems, like 70,000 lines of code. Well, we've taken outlines of code. We reduce it even further. We simplify it even further with these new drives that come online from Seagate. So I don't know if Seagate is actually here, but it's a great project, and we hope that we can make some good inroads on that. OK, Doug, thank you very much. That's it, enough. If you have questions, two questions? There's a protocol that is being utilized by multiple gateways to communicate and synchronize their file system view, so that's how that's done. Well, the file system itself does not have snapshotting. You can keep versions within Swift. So Swift has the ability to keep snapshots, but it keeps the trash can that I mentioned before, and it also has versioning, which basically delivers you the full data protection. There's one more question here. We are currently in a project to use a Fraunhofer file system, FHGFS, which is the highest performing HPC file system that we currently know. And we use that for our Scratch file system. But it's not for this use case. This is really like archiving long-term, what we call an active archive. It's an archive that is not going away, that you can always get to immediately. All right, thank you. Thank you very much.