 Hello everyone. We'll introduce ourselves here. I'm Brad Posorek. I'm the Systems Administrator with Oklahoma Medical Research Foundation, OMRF, as you'll hear us call it. I've been with OMRF for about 16 years. I've been in most of the infrastructure roles on the infrastructure team of IT, enterprise IT at OMRF. I'm Stuart. I am essentially a CIS admin also with a more of a background in Linux. I work in the Arthritis and Clinical Immunology department. Which basically means I'm actually in the labs doing computer work for for the labs instead of actually working for IT. Which basically means I do anything and everything computer related that is not handled by IT. I've been doing that for about 18 years. And we're here today to kind of share our story about SWIFT. So a little bit about OMRF. We are located in Oklahoma City, and we're an independent nonprofit biomedical research institute. Dedicated to understanding and developing effective treatments for human disease so that others may live longer, healthier lives. Just a little bit about OMRF. We're about 500 full-time employees in size. We run a pretty small lean IT shop. We have three system admins. Two of them are right here. The team of five dev apps, three help desks, security officers, CIO. That's it. We all wear many hats overlapping a lot of areas, help each other out a lot. A small fact about us here is that some people have a hard time believing that our IT department has actually run on a $150,000 capital budget annually. It's all we're given to bust it up, divide it up. We fight for it. Networking wants it. I want it. Everyone else wants it. AV wants it. Everything else we have to beg for. In the science side, we have about 45 lead researchers. They are grant-based funded. They bring their own money in, and they really want to do their own thing. This self-funded nature of them having their own money, some of these labs have big money. Some of them not so big. It's really created an us-first-them environment over the years. We're just constantly fighting administration versus science. My role in the IT role, we're focused on our core applications, accounting, procurement systems, payroll, networking, email, etc. We bring with that availability, business continuity, and a support model behind it. Many times this often brings project management and a longer deployment times to the table, much longer than what science wants to deal with. Science, like I said before, they bring their own pool of money. They know what they want. They need to do an experiment. They want to get it going so they can move on to the next step, find their discoveries. So they just go and buy their stuff, stand it up, move forward down the road to science. Oftentimes, this sets up what we call shadow IT. This stuff gets thrown up on lab benches and closets under desks, any place they can find to stick their equipment. For me, it's the typical usual business train model. I run Dell Equalogic Sands for all my stuff. I got two groups, a legacy group of PS6000s that I migrated off of. And another group, my primary group, is also Dell Equalogic. And it's dedicated to iSCSI on one pool and SMV NAS on another pool that runs Dell FS. Right now, I'm down to about 11 terabyte free on my NAS system, and that terrifies me. I could get a call this week that that's gone in full, or it might last me another couple of months. I really don't know in today's science. I need to add something to that. And on the iSCSI side of it, one of my main roles is the convolt backup system, where all my volumes are mounted, two terabytes of volumes into convolt. Well, I run disk to disk to tape, and really one of the key issues I had was my primary data and my backup data are all on one array, one nightmare away from being a full tape recovery. And I just can't fathom that, to be honest. So I need to find something I could grow into, expand as I needed it, and be able to do it in a cost-effective model. So from the other side, what I'm bringing to the table as a shadow IT group, just kind of giving you an idea of what I had before. Being the largest shadow IT presence on campus, mainly around the idea of supporting genomics workflows sequencing, mainly in the labs that I'm in right now. I run a pair of scale-out NAS systems. One of them is newer, smaller, faster, and under-support contract. The other one is older, slower, larger, but not under-support contract. This leads me to spending a lot of my time copying data between it as it ages off to get it off the smaller, faster ones. So we have more space there to continue with our analysis. So I say I have 200 to 400 terabytes of data. I don't really know how much I really have, because it's moving around all the time. On top of all that, I have to back it up somehow. So I actually run a third NAS system that I basically built myself. ZFS box sitting in one of these closets, just another building on campus that I then replicate some of the data to, again, as my supposed backup. Because the amount of data we have doesn't fit into the combat licensing that we have. And again, like I say, all of this I'm doing on my own, separated from IT. There's little backup for my actual data. There's no backup for me. There's no cross-training. It's all on my own. And like I say, I'm just a small portion of this. I'm the largest portion of the shadow IT, but there are other groups. You know, there's another lab that has close to 100 terabytes on a RAID 5 array. There's another lab that has another RAID array that has a drive in it that's failed that they have no idea how to replace the drive on, because they don't have the administrative password to that interface. It just hits there and blinks with a red light. So in getting this talk ready, I actually came across this little slide I had from one of our shares. It's pretty old, you know, 2010, 2011 timeframe. Just to kind of give you an idea of what all this stuff is there for. You know, this share, you know, we started out 2010 under 5 terabytes on this share. That's, you know, nothing. You can see the, you know, big spike there. We get some of our first sequencing equipment in and start running experiments and start running more experiments, you know, just keeps growing. Basic story on this, you know, in about a year's time, you know, this group generated 35 terabytes of data. Contrast this with this little graph from one of the shares on our new NAS system. And the thing here is literally last month, another lab called BradUp and told them they have 20 terabytes of data sitting on a bunch of USB portable hard drives. They got it from running one experiment in one week and they needed to do something with it. And so now what took us, you know, about a year to generate, you know, we can do in, you know, less than a week without anyone knowing. It just shows up. Put it somewhere. Yeah, we have to put it somewhere. So, you know, Brad's comment of, you know, 11 terabytes free on his system, you know, he considers that low. That's not low. That's empty. That's no capacity. So what is all this hardware there about and that, you know, it's all about data. You know, that's, that's what it's all about. You know, it's the hockey stick, you know, whatever you want to call it. It's always growing. You know, again, this is all about doing science. You know, we get funding to perform experiments to try to solve diseases or whatever. And you know, like I say, we get that funding from our science side to perform these experiments, to generate these data. But that funding may come with money for me to buy some of these NASA rays that I've had in the past. But it doesn't give you the ongoing funding to maintain that NASA ray when that support contract goes up or, you know, you need to fix something that's broken in it. And so, you know, it continues to grow because, you know, the other fun thing of sciences is they don't want to get rid of data ever. It's just not in their vocabulary. You know, they've spent time, they spent money, effort to generate these data sets. Why, why get rid of them? What's the point of that? There is no point of that. So we're basically holding on to it forever. You know, I have stuff on my NASA system with timestamps before 2000. I have no idea what the data are, but they're there and I have to continue copying them around to different systems. So, you know, this leads to, you know, this mix of sizes of data and then types. You know, I have big data, I have small data, old data, new data, raw data, you know, final papers, anything and everything. The backup data, you know, virtual machines, it's all there. And that's what we're doing for science. All our science is data driven now. You know, they were talking about the keynotes, you know, everything is becoming computer science. You know, the computer science stuff is coming to the bioinformatics space. That's what it's all about. On top of all of that, we have to protect this data somehow. You know, Oklahoma Medical Research Foundation, that's Oklahoma. You may think of, you know, a covered wagon, but Oklahoma is also known for a documentary that came out a few years ago. Some people think it's a movie, Twister. We are in Tornado Alley, so that is always a threat. And we've also learned of a new threat recently, of earthquakes actually. So, you know, we have to do something. You know, back to my older graph, there are under five terabytes. At that time, you know, my disaster recovery plan was actually, I told my investigator in my lab that, you know, if I see a tornado coming, I'll just drive down to the office and grab the drives and, you know, drive out away from the tornado. That doesn't work when you have a couple racks worth of NASS servers. So, you know, what all this means of all this data is we basically grew into a mess. You know, we have this historical warring, you know, on the bad days, just not communicating on the good days between the different departments and the different labs of everyone doing something on their own. You know, IT runs a SAN and a NAS system and they have their own backup, but that doesn't really work for the research computing side of things. I run a NAS system and need backup, but that NAS system doesn't really work for the IT side of things because my NAS system is too expensive for them and my backups are too large for them. So, you know, we're competing and duplicating effort and not working together on that. And then you have, you know, tape. You know, Brad is using tape for his backup. I actually looked into using tape for my archive data for some of my science data. I had to wait in line behind the business backups to get my stuff onto the tapes. I had to wait in line behind the business backups to get my data back off of the tapes when I needed to use that data again. I had to lose my data when both of my tapes failed. Even though we did this thing, you know, write everything to two copies on two tapes. Both tapes failed and so I lost some data there. You know, it's just, it's stupid. It's what tape is. And you know, on top of all of that, you know, we're both running out of space. You know, at one point last year, a year before last, for Brad's disc to disc to tape, it came to he had no space left and he needed some more for that. So I had to actually give him some of the space off of my science NAS. And then this question of, well, wait a second, he didn't pay for any of that. He charges me for storage and I'm giving him storage back for free. Shouldn't I be getting some money back somehow? Great model. You know, I don't, I don't really know on that. So it's like Sarah said, we both in IT and science ran out of space. We needed storage and we needed it quickly. I started down my ecologic path and we'll just buy another array, slam it in, expanded a couple hours later and I'm good to go. He went down his path, buy more Iceland, slam it in, expand. He's off to his science quickly as well. But what are we really doing there? We're paying and maintaining two independent systems. The cost is enormous to the foundation, to a nonprofit foundation. Stuart and I came together. We started talking. It's like, why can't we do this together? There's got to be something out there that we could both put in on that it could be useful to both of us, could be usable both of us, still perform it for him, cost effective for me. How do we move forward with this? So we both, we went off, got our, brought our normal traditional quotes to the table, eliminated them quickly, looked at some other scalable systems, and really OpenStack Swift quickly became the front runner in utilizing Swift Stack as its front-end management system. Some of these initial wins that we had on it was, my Commvault environment was very easy, very simple to deploy. It accepted it very well, formed very well. And Swift Stack had great Commvault documentation already out there for me. Provided an excellent page to grow model. We, as we fill it up, we just put in more commodity disk drives, expand it as we really need it. My CFO loves this idea. And the COO really loves that science and administration have come to the same table and are working together. Instead of fighting each other so much. One of my other big wins here is, it has 3x replicas helping to eliminate my tape environment completely. And really for the first time ever, I have a third location off-site in an actual DR facility that I can set up a Commvault DR replication center. Something I've never been able to do without having to buy a whole other system down there, and array storage down there to replicate too. Science immediately got it online and we're able to move off 50 terabyte of data immediately and archive off of their Isilon system to immediately ingest more science and get running fast. Kind of how this came together was in August of 2015. We put in our initial order. It was for six nodes. We did reuse one JBOD system to help initially save some money up front in our third backup region. There's about 250 terabytes total of the disk and the license purchase from Swift Stack. Roughly six months later, Stewart got a new project online, a new grant awarded for 100 more terabyte of disk and license and two more chassis for a separate genome project that his lab was running. And then our third major expansion was another 50 terabytes, just earlier this year, three more chassis nodes again off of a grant that he was awarded, and then I brought to the table the drives to start filling up those chassis. And then over the course of the whole year, as we've approached the best practices guidelines or needed more expansion, Enterprise IT has been able to fund that license and those drives, sometimes even sneaking drives in as office supplies. So a little bit more details on how that's all done hardware-wise. Like I say, we're using Swift there, standard 3x replica, so we have three zones in two regions. I say regions here in quotes because initially one of those regions was again in a closet in another building on campus. You know, standard white box, super micro things, you know again, nothing fancy, save money just for you chassis, couple SSDs for boot, couple SSDs for your account container, and then slam full of drives for everything else. We actually run a mix of drives. We reused some SAS drives we had previously. We've bought various SATA drives over the time and the last time we actually expanded some drives, the SAS drives were cheaper for us at the time, so we actually slammed in more SAS drives. We honestly spend more time when it comes to expand drives, bitching back and forth over what drive to buy, what's going to be the cheapest, what do we think is going to last the longest. We spend a week or more going back and forth looking at that than it takes to put the drives in when we expand it. We're using for the most part there all commodity consumer drives, which can fail obviously as anything and everything does. The extra thing we're doing that is using the full disk encryption, the Linux full disk encryption, so the drives themselves are fully encrypted. We did this for a couple of reasons. One, when we first did our install, Swift did not have native encryption. Two, it's just easier to keep it that way now and we don't have to care about the drives at all. They're throw away. Everything on there is completely protected if you don't have the key. Genomic data is considered personal identifiable information, so we have to protect it. We pull that drive out of the chassis. It's useless to anyone else without the key. For our one failure so far of a drive, we actually for fun went ahead and did the standard consumer RMA replacement on it, mailed it back to Seagate or West Indies or whoever it was, and they sent us another one. It didn't matter what was on there. For performance reasons, we're using read-write affinity, so we're targeting the local nodes in our local region just to maintain the performance and then everything else. Obviously we'll replicate off-site eventually. We started out with just simple round robin DNS between all the nodes because we were going with a full proxy account container on each node. That was done to meet some price performance and capacity requirements. We couldn't afford to do a dedicated proxy tier and we're continuing with it without the proxy tier just because it's easier to have every node do everything. Everything's the same. Later on we added in load balancing through the Swiftstack controller, which is what we're using to do all the management of everything. For anyone kind of graphically minded in the audience wants to see a pretty picture, here's some lines of everything. We have our two regions. One is in our main office in Oklahoma City. We sit on an Oklahoma City campus of the local university. The main campus is about 20 miles away down in Norman in Oklahoma and they actually have their own fiber between the two and are nice enough to let us utilize that. We actually have 10 gig down to our off-site now since that's been moved down there last month. Our two local zones on off the sides of the data center, that's all plugged into my high performance computing 10 gig network, and then I'm able to bridge out to the OMRF network now so I'm not separated by a little Linksys NAT router between my connection out there and then obviously the Swiftstack controller manages all that for, so Commvault and everything else then talks in. So what is all that fancy hardware actually get us? These are a couple graphs I pulled from the monitoring system shortly after we actually stood it up and the top one here is our HPC doing some test analysis and we were able to pull out a gigabyte per second of our archive data to reanalyze it and then we were able to push in about 600 megabytes per second, which for me really kind of turned it from just a cold archive to an active archive tier. It turned into something that's useful enough to do all our normal analysis from. The other side of this collaboration with the IT department, which is a traditional Windows-focused IT department, whereas like I said, I have a Linux background, open source, none of that stuff is a big deal for me, but to get something like this into the IT department was a little bit of a sell. A couple things that made that easier, some Ansible scripts up front, just a couple playbooks to do the basic stuff to set up your Linux and do our work for encrypting the hard drives. So Brad can run one playbook that will add a new node, get it all provisioned ready to go, and then when he adds drives in at any time, another playbook will find those new drives and throw our keys on it. The other big thing with that is the Swiss stat controller actually. Like Brad said, we all wear mini hats. On our hat rack, none of those hats is full-time open stack operator. None of our hats are even full-time storage operator. There's too much other stuff to do. So having the Swiss stack there to actually do all the work of installing, configuring, managing, and then most importantly, actually have a support number to call if something goes wrong or there's a question, which came up last year when I configured a couple things a little strangely, adding in some drives, and then immediately went on paternity leave. So having a number to call and take care of that made things easier for the IT department. The other thing that the Swiss stack gives us is we're not running the full open stack suite, so we're using just the standard off from there, but they have an integration with Active Directory, which then just from a simple couple of radio buttons lets the IT department manage all of the access to control straight out of Active Directory like they're normally used to using. There's no other different tool. So this is a quick shot of some of the accounts we have set up. So we can define all of our accounts to access our different scientific archives straight in Active Directory, manage all the control. Nothing weird for anyone to have to do. So as Brad mentioned, we got it set up in a week. We shoved the Commvault data in there. We shoved my initial archive data in there. That solved our initial pain points. But any storage can do that and just some data. There's nothing big there. And so what comes next is start using some of the fun stuff that you get with Swift. Do some programming against it. So what I started playing with was first start using some other stuff that can talk to Swift. Stand up a quick Docker registry. Everyone Docker, Docker, Docker. Everything's in a container. And for its persistence, straight in Swift. So that was easy enough. From there, it started to... I put on my developer hat and start to actually program against the API. So the first thing I did was just kind of get familiar with getting and putting. And so I actually wrote a quick back in for HashEvault for some secret stores. So again, that's not too fancy. Get and put objects. Nothing fun there. Next thing is start using it for some real business cases. For me, in the sequencing core, we regularly send 50 to 500 gigabytes out to customers. Previously, I was having to, again, manually copy this over into another share that was on a publicly accessible web server, send out a link to someone, then remember in a couple of weeks to go back and delete that stuff from my share because it was sitting on my NAS system. What does Swift have that can help me with this? Swift has a thing where you can expire data. So another quick script that I can tell it, here's a set of data, send it to this email address. It will create a container for me, upload all that data to the container as it uploads, set a delete flag in whatever timeframe I tell it, and then they get a link to a randomly named container and they can download the data. And I don't have to think about it. It just gets cleaned up afterward, and if they don't get their data, that's their problem because that's our policy. That's a policy thing, so it doesn't matter for that. So the next thing I started playing with in Swift was the temp URLs. And for that, what I started doing was I actually wanted to start backing up some of my other systems, but I didn't want to go tie-in to a Commvault license, and I just had a couple of small files on systems I wanted to back up. And normally, that would be okay, there's different things that can do back up and do it to a cloud object, but you have to give it credentials and save credentials in a credential file. That then means if someone get those credentials, they could read whatever data you've put in there or delete whatever data you've been in there. I don't trust anything, so I don't want my credentials sitting out on another system, even though it's in my own firewall. So I actually wrote a quick service that basically would hand out temp URLs on another authentication token, so I can just one-off back up things to a temp URL. I get to put URL for five minutes that's completely random based on another key. And again, it was just something quick and a day worth of hacking something together as fun. From there, like I said, we have the ability to use this as an active archive, pull our analysis data straight from it, and if you remember back to the Active Directory image, I have a lot of groups, and that was just part of it that was there, which means basically we've taken the approach to break up all our archives into separate projects and groups so we can more easily identify them and control access and know how much space they're using for any sort of charge back stuff we may have to do in the future, which means my users need to know what they have access to, and more importantly, they have to be able to access it with their Active Directory credentials. Again, I don't want them putting... well, I don't really care, but Brad doesn't want them putting their Active Directory password into some sort of environment file. So again, a quick little script of a program to interact with the list of tenets that are available and set up your basic auth tokens so you can then do some analysis, which then leads you to... okay, I don't want my users to manually have to download the files, run their analysis because a lot of our analysis software doesn't actually speak HTTP. It wants a standard file system, so we still have to pull these objects out. Again, I don't want them to manually download them, run their analysis, forget to delete that file because now I have the file sitting on my file system again, so I don't want them to be able to access it in the last space. So another quick script that basically you can name an object or a set of objects and a standard Unix command. It'll pull it down, pipe it through, run the stuff and delete the file behind you and just a quick thing of counting the lines in a file. So all that, just a couple of days of playing around with some development stuff. So it's all sunshine, rainbows, great stuff, but there are some troubles along the way. So, familiar with the idea of objects, how to use that and how do we want to use it, what's the best way to set up our containers and projects. That kind of then bleeds into some user error. Like I said, I had user error from the admin sign of adding stuff too fast, but I also have a user error from the standpoint I live in fear of yes, the data are 3x replica, but if I accidentally do a curl delete to the wrong thing, I could delete all my data. So we're getting around that right now with different accounts that have read and write access. The other problem was, like I said, the offsite. It took us over a year to actually identify a location that we could A, afford, B was far enough away and C had enough bandwidth. This little picture here is when we were doing some experiments on site using virtual PF sense to see if we could encrypt the data enough and have enough bandwidth, and then how slow we could get it. The virtual PF sense did fine encrypting with an IPsec tunnel for what we needed to do, and as we were slowing that bandwidth down, we found at about 500 megabit we would fall behind on the replication too far, so we knew how much we needed to have to actually do that. Going forward, like Brad said, the CFO loves the cheap and he wants all the stuff in the NAS systems now, so we're looking at, can we tear off of that? My ISLON has a thing where it can automatically tear, which would be nice, and all the users could use it. It technically only supports public clouds like S3, which then makes me think, maybe I could plug into the Swift S3 stuff. But then I'm tied into using that proprietary system forever, so I'm really looking forward to what comes along with proxy FS to just skip the NAS altogether. I'm looking at the data aspect of it. All the scientific data I put in there has tagged with the investigator, the sample, the disease, the project, the grant number, anything and everything we may ever want to look on, and so to now put a search engine in front of that so other investigators on campus can discover what data are already available to them. The last thing I want to start to kind of developing is play with some middleware, actually. Like I said, I fear myself that I'm not wrong, so I want to just do a simple, simple worm like middleware of if you do any destructive operation, just have to have an extra HTTP header on your client's call. Just something real simple. And so really what has that brought us to now that we're collaborating, we're doing everything together, it's no longer us versus them, it's us and them working together going forward. We're fleshing out our shadow IT departments, giving them the 3x replica, the disaster recovery, the business continuity aspects and hopefully saving them money in the long run but not having to buy this stuff up front and putting more money back into their scientific research. Some of the takeaways, real quick, so far we've got about 420 terabyte of object data sitting in three regions. We've come together, presented a viable joint effort using OpenStack Swift and SwiftStack as the front-end management and proved not only can it work and science and administration can work together but we did it providing a more cost-efficient model providing more benefits that can be achieved again like I said, with more traditional storage efforts. I will say personally up front that this was a very daunting project initially as it brought it to me being traditionally a Windows environment traditionally trained but Swift has not been so bad and really the SwiftStack portion has brought a great sense of ease to me. Thank you. Any questions? Very interesting talk. Thank you very much. Can you say a few words about how you manage the distribution of resources between operations and the mission side in terms of like are you working this as a condo or how are you dividing the costs for long-term maintenance and provision of capability? On this OpenStack model we kind of who bought the last round who gets the next round type thing? It's a mix. The initial purchase that was kind of a 50-50 split. We both needed it and we were both going to get half of that 250 terabyte that we initially bought. Since then some of those expansions again just kind of the way some of the stuff works it's easier for the science side with their larger grants to buy the chassis you have to get the whole chassis at a time even if it's empty which is a sizable cost and so we've gotten some of those and then we get capacity at that and then like Brad said it's easier for them to kind of buy drives one off on their own I can bury those in operating expenses he can put those under his budget and so long-term what we've worked out with the administration is they'll maintain the licenses going forward is the idea and then what we're presenting to our other investigators is like the guys that had 20 terabytes of USB we can either just take those USB drives and slam them in there or you can buy other drives bring your drives to us, we'll put it in there we'll cover the rest of the cost again to try to get these data out of the labs out of the closets and something that's protected okay the other guy and this maybe goes back to the same question of how do you break up costs do you use the accounts to sort of group like projects or departments can you see like how much storage is in accounts over the past month or six months or something like that yeah actually that's again kind of why I have so many accounts set up accounts from the SWIFTAC SWIFT level is with the SWIFTAC interface it can easily give a report back in terms of the usage buy the top level account without having to go in and run extra accounting against containers individually so we can say okay this project is named the first investigator for this purpose so we can know exactly okay you know investigator X you're using this total against these you know five accounts that you have in there for your five different projects thanks for your talk you mentioned that you have I think you've termed it an eternal retention policy have you have you used different storage policies to kind of create like cold tears for moving data to doing that for most of the stuff at this point no I do have some different SWIFT policies for the customer delivery I am using just a 2x replica for that one because I don't care about their data I probably shouldn't say that in case anyone wants to use our services and so that you know I don't need to make that third replica because again it's a short-lived thing that doesn't go out so I'm only using 2x on that but for everything else we haven't done much of that I don't want to go into the exploration of other policies you know erasure codes or anything like that that's one of the things I am actually waiting for is multi-region aware erasure codes so maybe we could move that for some of the really old backup data but still have be sure that there's enough offsite to fully recover if it comes up good stuff we're kind of looking at something similar and one of the questions that we have is and it may relate to the relatively high load on the environment and then you lose your link to your offsite and your main site goes down and you rely on your offsite and then the offsite and the online they both eventually come up and have to resynchronize and how long something like that takes or is the amount of data that you're loading low enough that they can get back to sync relatively quickly our daily churn is low enough at this point that it hasn't been an issue like I said it was just three weeks ago, four weeks ago that we actually did get the stuff offsite and so that third region was completely offline for 24 hours as we drove it down I had unhooked the day before so I could get everything on track and we had to throw it all in trucks because we were cheap, we had to do it ourselves and physically drive it all down there and hook it all back up and so it was offline and everything came back and resinked relatively quickly, it wasn't an issue our daily, the Commvault stuff is the biggest driver of daily churn usually and it hasn't been an issue I don't know if we'll get another big whole genome project and have 70 terabytes show up in a week or something and have to put that in but it's supposedly a 10 gig link and we can use it all so we'll just let it churn for a while so at this point all the data we're putting in the scientific side is technically private to our own investigators but that is something that would definitely be something to look at is turning it into a repository and make that wider available if it can be or agree to by that investigator Thank you