 I guess I'm on. Well, hello, everybody. Thanks for staying till Thursday and coming here. My name is Jim Merritt. And I'm speaking to you this afternoon as a systems engineer at Burton Snowboards. And like many small shops, I wear many hats. So I'm also a unique systems administrator. I'm the sand administrator. I am the backup administrator. And I'm also the storage administrator. I have a colleague who also is the network engineer. He says I'm only certified for layer one, so he doesn't let me do a lot with a network. With that, I want to say a little bit about my background. I came from genomics research in financial. I spent about six years in the research genomics field. So I wanted to say that because it gives a framework for how we ended up where we did with OpenStack and specifically OpenStack Swift. So I want to start off with this question. We've all heard about big data. If you're playing buzzword bingo, that would probably be one. But really, how big is big data? I came from genomics. Genomics, we were talking about petabyte scale. We were talking about exabyte scale at one point. That was big data. That's traditionally what you see and what you read about. I say that big data is any amount of data that becomes a problem for your business. That could be moving gigabytes of files, terabytes of files. But if it's problematic, it's big data. So this is where we started from. Burton Snowboards has about 250 or so, maybe a little more, unstructured and structured data. Of that data, the structured part databases really make up a small portion of that. The large portion of our unstructured data is really the video, photo content. We do video presentations. Burton presents. We do snowboard videos. We do snowboard imaging. It's part of our marketing. All of that data has tremendous marketing value. And we also have to retain that for a lot of years, tens of years, because we like to go back to that old footage from the US snowboard event. Events around the world. And we'll leverage that back in as new content. So we have to retain all of this video. And we had to protect all of this data and allow for that data growth. Because we had to save it all, we had to protect it, and we had to grow to keep that content. And because of that, our traditional methods of data protection for that were coming very expensive, both administratively and monetarily. Costs a lot of money to keep this stuff around, even for a small company. Probably more especially for a smaller company. And then if we sat and said, OK, how do we recover this data? Because that's the end game, right? We back it up, that's great. But at the end of the day, you've got to recover it. That's the end goal. When we ask that question, it started becoming more and more difficult to come up with a straightforward, simple answer to recover our data. So here's the infrastructure we started with. We have the traditional santa rays. We use NetApp. Pick your storage poison. There's a bunch of them out there. We had two disk arrays. We did the traditional disk to disk to tape. We had a tape library, 250 slots. LTO5, we had five of them. We still have that, by the way. We had an offsite storage facility for tape. So disk to disk to tape. Tape goes out to offsite storage. And then we actually had a company that we contracted to save us space and potentially hardware if we defined it correctly. So in the event of a disaster, we would go there and spin up all that equipment, hopefully. We have a mix of server hardware. We have Dell, Cisco, UCS. We have some HP boxes. I think we have a Sunbox kit around, too, somewhere. And then we use Commvault for data protection software. Again, insert your backup software here, because this whole thing is really agnostic to that. And then operating systems, we have Windows, Linux, Solaris. We're a pretty typical shop. We even have several distributions of Linux. We use SUSE. We use CentOS. We have some Debian in-house. We are also an SAP shop. So we have a very large SAP infrastructure as well. And we're heavily virtualized in ESXi. So the old way. So we used to use our storage systems, both as primary storage, provide LUNs to our databases, file shares for our end users. And we also used it as a backup target. So we had another array sitting there that was our other disk of the disk to disk to tape. And then inside the primary array, Storage Array 1 in this, we also had all the snapshot functions. So we're snapshotting in an array. We're the other array is used for both Oracle Databases, RMAN, we would write RMANs, NFS, out MSSQL, right out the SIFs. Or we would use agents, go through Commvault. And Commvault would write it to the second disk array first and then go out to tape. We relied on shipping of these LTO tapes out to the tape repository. And we had all these protocols, NFS, SIFs, that were all in play, including the FedEx protocol. Guy comes up, ships your tape off to off-site storage, worked well for a lot of years. Again, when we started to think about disaster recovery and looking at what we had is we grew extremely difficult to come up with a procedure with all of these lines. Where does everything really go? And this thing goes here, and that thing goes here, and you've got to click this button over here, and you've got to do it in the right order, or it doesn't work, you've got to start again. So we had some technical issues and not so technical issues. I think you heard at the keynotes earlier in the week, the soft part of this is really a large portion of the storage problem. We had the traditional data protection model. We had lots of tapes in flight. We were doing copies of copies of other copies and multiple tapes and shipping them off. We had lots of tapes to manage. We had the library maintenance. I don't know if anybody's maintained tape libraries, especially some of the newer ones. They take a little care in feeding. We had little deduplication in use. And we use it in place, but we have little in flight. Get to that in a little bit here. Besides video images, movies, they don't dedupe well anyway. Again, we'll get to that in here in a minute. And then we had the integration timing between all the pieces. And I kind of alluded to that before. Had all those lines going, they had to be in a certain order. We would do an R-man to a disk, to an NFS volume. We would have Commvault would come in behind it. You had to make sure that Commvault didn't come in. Before then, if a DBA, say, ran a job later or earlier, or maybe there was some glitch, an R-man took longer, now your Commvault hits it, it's not done. You don't have your data backed up. So there was issues around there. And then really, the soft skills, we had little data management and curation. This is a big one, and I'll come back to this, too, is we just had data everywhere. And I don't know if you ran into this, but users are really crafty. If you give them data and they run out of it, they sometimes won't call you. They'll find another place that they have access to, and they'll start putting data there. They do this all the time. And that also becomes kind of a management nightmare. So as you fill up and you don't grow fast enough, you run into that issue as well. And then you've got the administrative effort. It's getting complicated to back up these systems or protect them, and it was even more complicated to recover them. So we decided to make a plan for change in all this. So we leveraged past experience in traditional big data genomics field. I've been looking at object stores for six years. We had, I've had to deal with this petabyte scale data management issue before. And I used the concept of raw and intermediate data that helped our governance. Raw data being that initial data that cannot be recreated any other way. Intermediate data being that data that can be recreated at any time. It comes from the raw data. So at the end of the day when you're doing data protection and disaster recovery, I can forgo the intermediate data. I can recreate that from the raw. It may take a day. It may take an hour. It may take a day. It may take a month. But I can recreate it. In a disaster, I can recreate it. The raw data is the important data. So it was a good concept that I brought with me from genomics and applied it in this circumstance. And the organization was very open to this. And then when you look at that, it turns out the problems with our data volume are the same exact problems we had in the big data. Same exact problems. Data governance, where's my data? How do I get it moved? How do I protect it? Same thing, just a smaller scale. So we came up with a solution. We're going to deploy OpenStack Swift as a backup target. So now all our backups are going to go into our object store. We're going to utilize a remote site. And we're going to get that site, get that data off site. And it's built into the OpenStack Swift. So it just happens. We're going to utilize commodity hardware. And networking, we're appropriate. And you'll see we kind of went outside the normal vendors for this. We're going to archive old unstructured data. So anything old, anything intermediate or raw data that we need to keep, we're going to archive that. And then, as I put here, because we used OpenStack Swift, we inadvertently had a disaster recovery plan. It goes off site. It was a little more complicated than that. But basically, it's built in. It's there. And then we drastically reduced our tape management. And well, we basically reduced our tape. We don't really use it anymore. And that was part of our plan to get rid of that. And then we utilized Swift Stack to actually do our deployment and ongoing maintenance and management. Made it much easier for us to manage that and made it even more transparent to our organization, especially our admin staff. Let's see. So what we implemented is we have an OpenStack Swift implementation. We have two regions, three zones, three object nodes per zone, and two proxy count containers. I have a picture of it here in a minute. Each zone is its own rack, separate power, AB phase, all those that are infrastructure guys. We have one region that's located in our main office and our data center, and one region at a COLO facility. I'll touch on this a bit. Because we had the luxury of certain availability to COLOs. We're not going very far away. We're going across town. We're from the state of Vermont. Across town means you're going through a couple farms, some cow fields, and things like that. So it is pretty far away. But it's not traditionally states. And you would really see region when they do the map in countries. We made our standard Commvault Cloud Library in Commvault. We made one for Ddupe, one for non-Ddupe. Very easy. Have a picture of that. And then, again, we used Swift Stack for the cluster deployment and management. And we also used the Swift Stack Gateway to do SIFs and NFS Gateway for applications and some end users. And we'll get to that here in a minute as well. So the server hardware we got was all silicon mechanics. We went to them. We went to several different vendors and basically did an RFP and said, hey, this is what we want. What do you got out there? And we literally chose the less expensive. We said, we don't need big HP boxes with all those built-in stuff. We don't need RAID controllers. We don't need anything. We basically need a box, CPU, a bunch of disks. That's it. And that's literally what we bought. So we have the three silicon mechanics. I said three. There's actually a lot more than that. I did three and six. I'll get to this, too. So we started off with a smaller cluster and then expanded. And I'm going to have what we did with that. So we bought silicon mechanics boxes. We've got CentOS 7 on them now. This is what we currently have. And we have two proxy account container nodes that I just call pack nodes. Netgear switches, less expensive, probably the least expensive ones I could find. They're 10 gig capable. We utilize HA proxy. And HA proxy was really put there for SSL offloading, not so much for load balancing. Although I do use it to switch between the pack nodes when I'm doing maintenance, or if something were to happen. But it was really there for SSL offloading. And then I have the Swiftstack gateway. Both of those are virtualized in our ESX. So there's no hardware behind those right now. So here's a photo of what our infrastructure looks like now. When we first deployed, let's see, everything on the left-hand side was what we started with. And we started on that about a year ago. So we brought that in-house. We actually had both pack nodes are over there. We had them all together. We have the HA proxy. We have the Swiftstack gateway that has Sips and NFS access. We have a 10 gig cluster replication network that was running across the Netgear switches, non-rotable network. Nobody can get in and out of it. We have a one gig outward facing network. It's all we needed at the time. And at the time, our infrastructure was only one gig capable across the entire infrastructure. Nothing at the core. I think I had four 10 gig ports on an old 6509 switch. And we were moving them around so that we could cook our Netgear switches into them to get 10 gear, at least to the Netgear, two Netgear switches. So we jockeyed a bunch of stuff around to get enough 10 gig to make this make sense. While we were doing all this, we ripped out our entire core and went to Cisco UCS and Cisco Networking. We'll get to that here in a minute. So now we have 10 gig in the back, full 10 gig. We do have 10 gig available. I just haven't flipped it over yet. So in this infrastructure, Commvault and its media servers all talk Swift. We started working with Commvault about a year ago. It took a while. They had some little few niggly issues. But it's been working great. So they come in through standard HTTP, through the HAProxy, and into the cluster. Our archive storage plan right now is going through SIFS and or NFS. And we haven't really completed that yet. We're still working on that. It is not a technical issue. It is purely a governance and data management issue, which I'll touch on at the end here a little bit. User access, the users like SIFS because they're used to it. Right on the thing, they can do amount, windows, people like it. So they all use SIFS for the most part to get to the archive on what's there. And then I don't have the Swift Stack controller that helps manage and do the deployment because it's hosted. Better let somebody else have that part. So all of this took a bit of a paradigm shift for us. And this gets into kind of the governance piece. Before data was Wild West, we had data going everywhere. What we found out in the archiving was, and I don't know if you've seen the presentation before about container size, and we learned this. When you build a container, you don't want to build a container, one container, and just start putting stuff into it. And that's kind of what we did. And we had lots of objects, lots and lots and lots of objects. So what we've learned is, we had to step back and look at our data and come up with a data structure that was more appropriate. And we could build these containers yet let our end users have easy access into that. So we built what we call dated containers. And we went to the end users, and they had already started doing this pretty much themselves before. But we got them to think about dated containers on a file system level. So they started going, oh, I'm working in 2007. All my 2007 work goes in this 2007 container. 2008, 9, 10, 11, 12, up to 17, 16, or 18, whatever. So they're starting to get used to dumping that in the container. Now, we just take those containers, and they say, we're done with year X, move it to archive. We still have to manually move it to archive, but it makes it really easy for us, because we just take that whole file system in its entirety, move it. They start writing into the new one. If they need anything to come back, anything that comes out of archive is a recent raw data, and they will modify it and put it back into the newer container. We only needed to put a primary copy of our backups through COMBOL. So traditionally, COMBOL, I don't know if Syving Semantic is the same. You write data, then you take the data, and you make a copy of it. You ship the copy off-site. Well, my underlying object store is not taking care of that. So I don't have to think about that anymore. So I just go in once. So I don't have to worry about any of the tapes, where it's at, what data made it off. So it made it really easy. So noogs, copies, or whatever your backup software calls it. The databases, we said, well, why can't we leverage the gateway and just tell, for instance, Oracle R-Man right to this NFS share? And now, I don't have to back that up at all. It just goes into Swift. And the beauty is, when it comes time to do disaster recovery, it's there. It's easy to get at. I can literally go in through the website, bring my data over to the if I have to. I don't need much infrastructure to start my recovery process. So we leverage that. We do the same thing with MS SQL, which sips. That happens all the time. My vCenter backups. Don't even bother with native vCenter backup. I just dump the database into Swift. And then, yeah, the Commvault DR and DDoop database, which that's going to be different depending on your backup software. But we put that into Swift, too. So we have a recovery mode off-site ready to go. So I wanted to show what the configuration for Commvault was. And in the circle, that's the only thing I did. You go into Commvault, and you tell it I want a cloud library. And as of the recent version, it actually says, I think it says OpenStack Swift now. It used to say Rackspace, if I remember right. So but now it actually says OpenStack Swift. You click that. You put in your credentials. Just so you know, we use SW off. We do not use Keystone because we don't have the entire OpenStack. We don't really need Keystone. So we just use SW off. You put the v1 credentials in. It goes in. You're done. I create my containers ahead of time. I don't let Commvault do it. And that's really based, if you went to the performance one this morning, if you let Commvault do it, it picks the default storage policy if we ever wanted to do EC2 or anything like that, RacialCoding, I'm sorry. It would default back. So I create the containers ahead of time. So if there's 20 of them, I create 20 of them. It's a little bit of time ahead of time, but I know exactly what's going into the container, what storage policy I'm using. Your mileage may vary depending on the backup software you're using. But the three boxes underneath the tape library are it. It looks just like that tape library. Just like the SDK box. It's used just like it. The storage policy pieces, everything else is combo. You don't have to do any modifications. So this. I wanted to bring this up. So if you remember back, we have two regions. I apologize. I didn't do a good job describing that picture. So we had two regions. We had object nodes on the other side. We had a WAN link. That WAN link is one gig. I'm now going to talk to you the little bit I said before about we had the luxury of certain availability. So in Burlington, Vermont, they actually rolled dark fiber around the city. So when we talked to our Colo people, they said, hey, you've got strands behind your building. Great. So we actually have dark fiber that we run between our two locations. And that allows us to actually run layer two across our WAN link. So we don't have to go out our internet good layer three and all the routing. We started out with a 200 meg connection. Don't start out with a 200 meg connection. Um, we did it because it was easily changed. And we were gotten for punishment. And we wanted to see how it worked. And you can see, I think this is from this morning, 200 and or 124 megabytes a second. If you do the math, it's 996 megabits a second. So we flood that. Um, but you know what? It's OK. It just works. You get a few, you know, if you're doing pings, you get a few ping drops. But the storage just works, writes, gets an error, writes again, replication. It knows it needs to replicate. It comes back through. Auditor comes through. Hey, I'm going to replicate. Replicate process. It just does it. It's awesome. All the little bumps are our oracle and SQL backups in the night. So you can kind of see every week we have a big job that runs Thursday morning at 1am. And it may run for a couple of days. Some of our volumes that we're backing up are 20 terabytes, 22, 29, and we run those. We don't have Ddupe. If you have Ddupe available, this is where Ddupe really helps. Because your first backup will look like this, you will flood it, but your subsequent backups will only be the changes, the things that didn't get Ddupe. So it will lower your network. That's what we're working on next. So you're after initial deployment. So we have some initial issues with Commvault, not big ones. As of Simpana 10, SP11, I think 13's out now. We're on 13. No issues with it whatsoever. It just works. Really, it's the configuration and doing policies. And that's not a storage really issue. Again, governance part. We currently are backing up on a cycle about 160 terabytes of data weekly. We currently are archiving 25 terabytes. That was the first chunk of archiving we put in about eight months ago. And then said, hey, we have to rethink this and break things up into smaller pieces. So we have to get back onto that project again. And then we now have a much easier and much reliable recovery process. Right now, the plan, we can just spin up our Commvault in a remote site and attach it to Swift and read out. We only have the one zone there. So I'm going to be able to read out of it. But that's all I need for recovery, for whatever our secondary storage is to bring them over. Or to get the data back to primary storage once we get that in place. So now that question of, well, gee, how do we recover things became much shorter? So instead of these three-page list of how to recover things, we're down to one or two. So next steps, racer coding. A lot of people talking about it. Two use cases for me for racer coding. I don't know how many people do it. We still back up desktop laptops. I don't like it, but we do it. That data does not need to be spread across multiple regions. If we have a disaster, sorry, I'm not really concerned about your desktop laptop data, but I also don't want to spend three to one protecting it local. No matter how small it is, I'd rather take the 1.7. And here's a more important one that we just ran into last week. So Microsoft Exchange, we do mailbox backups. We somehow have people that delete whole portions of their mailbox. I have no idea how they do it. Well, we're going to take Commvault, make another container that's a racer coding, and do the mailbox backups into an eraser coded. I'm not taking a three to one hit. I do one to five. I already am doing my database backups. So in a DR, I do my database, but I don't need to do mailbox backups over there. In a disaster, I bring up my Exchange server, back it up again, doing the mailbox wherever my object store is again. But I can keep it in the one region, local, and I can get that very fast recovery of somebody's mailbox, because it's usually always someone important that's just getting ready to get on a plane, and then they've deleted half their mailbox. Now we can click it, put it out, ready to go. Archive, we need to archive more data. The importance of archiving is the more we archive, the less we have to back up. So our archive strategy is we get all the old unstructured data, we put it one time into an object store, and we never touch it again. So I don't have to buy any more Commvault licenses. Sorry, Commvault. I don't have to expand it. I can use the licenses I have on more important things. Metadata search, you've heard about that. I'm huge into the metadata search. Sorry, Joe, I saw your thing this morning. Yeah, love metadata search, big on that. And that's really leveraging our Elasticsearch. We're also Elasticsearch user, Elkstack. So I want to leverage that to put metadata in and search our archive data based on metadata tags. So now that fixes a little bit of my governance issue. Now I can find these files that are in my object store. And then the other integration point to Elkstack is for auditing. Who's writing files? Where are they writing them to? Who deleted files? When did they delete them? That kind of thing. So that's the end of my presentation. And thanks. I think we have time for questions. Five minutes for questions, except for you. I think you've got to hit the button. Testing. Hi, Jim. This is Mario from SwissTech. Which one of those people in this photo are you? Mario, that's a good question. See that roof of that building down there? I'm down there having a beer. No questions? Thank you very much for showing up.