 Okay, let's do the introductions agenda first and then we'll cover it We're gonna cover what pack 12 networks is because I'm sure a lot of you raise your hand if you're familiar with what pack 12 networks is Oh good. Oh, it's that's great Good. Oh, I see that's an Arizona State video you had up there, but I'll I have some duck stuff, too We'll cover the unique challenges that media content creator faces with regards to archiving And why a traditional model versus something new had to be discussed We'll cover budget planning the stand-up Martin is with Swiss stack and we'll do introductions in a second But he helped me stand it up and then what happens when it all inevitably goes wrong I have a great story for that one How one-click expansions are probably not the way to expand your cluster? And the takeaways and lessons we learned after having Swift stack for almost two years now So my name is Scott Adam. It's I'm the director of system architecture and technology at pack 12 networks in San Francisco In my former life. I helped launch big 10 network in Chicago and in the middle I worked at Groupon during their IPO phase. So that was exciting And I'm Martin Lanner. I'm with Swiss stack. I'm an engagement manager and I do exactly what Scott just described Which is help our customers stand up their clusters engineer their environments so that they are well Done and can expand in the future And Martin they worked with us very from the very beginning all the way to today We still have problems that we're dealing with and he's helping me solve my issues that I cause So let's cover what pack 12 networks is. This is one of those short little propaganda films that I'm supposed to play I like it gives me the tingles. So see if it does for you So as you can tell it did launch in August of 2012 I'll give you some stats on the network itself. It's made up of the 12 most prestigious universities in the world in my opinion We've won 469 NCAA championship titles, which is more than any other conference by about 200 So it's a very it's a winningest conference in US college collegiate sports We are available as a network in 60 million US homes and an additional 33 countries through a YouTube distribution deal internationally We produce 850 live events every year and I can't tell you how much content that is We were actually founded through a 12-year three billion dollar deal with Fox and ESPN. That was a genesis of our network So let's talk about the challenges. This is the number that keeps me up at night Ratio that is for every one hour of live content We store and record eight hours of media and how you're asking could that be possible? It's one game. It's three hours long. It should be three hours of media. No, there's front halls back calls There's b-roll of coaches interviews. There's Content that they use to make packages promos and features They'll do cut-downs of the game because people don't want to watch it through our long game They want to watch it in 60 minutes. So we'll take all the action put it together into a shorter clip That's a new piece of media that has to be stored So this is a challenge that we face every day and unfortunately We underestimated what we'd need when we first launched the network back in 2012. This is what we are launched Sam We had two tiers one tier was 15,000k sass drives about a hundred terabytes usable Tier two was 200 terabytes of slower SATA discs and the idea was we'd have 3000 hours Available for recording editing and playback and then once it's not needed in other words football season's over We'll go to basketball Because those are two different seasons and they don't overlap at all And they do and so what we found out is we filled up our tier two because we kept punting content to it And had no place to go from there So some stats on how we ingest on a busy weekend between football and basketball seasons as we are in right now We'll ingest about 12 terabytes of archivable media each weekend. So it's it's not small The reality of this model lasted six months before we figured out we needed something else And this is where we looked at traditional archive strategies We could expand our tier two add more discs We had enough in the chassis to pretty much double it problem was that was going to be about $1,600 per terabyte to expand Yes, it's always online and it's extremely fast, but it's a sand it doesn't scale and it's not meant for archive So it should never have been an archive in the first place So that obviously doesn't work What next this would be what everyone's thinking LTO and I talked to a lot of people and including a very high-level person at Turner Sports Who shall remain unnamed? And I asked him when we were in the planning phase Would you go LTO today if you were building a greenfield network? Would you go LTO and he said emphatically? Yes? Absolutely tape is never going away Yes, it's cheap and you can restore locally, but unless you make a copy of it It's not safe and unless you have a body that's going to load the tape robot when it inevitably runs out of slots And you start to have to put things on a shelf It gets quite expensive Not to mention. I just hate it. It's rust on tape. We're 2014 at the time 2012 we can do better So we decided to look for something else and you know the next obvious answer is to the cloud We'll just go to the cloud. Oh, yeah, I wanted to burn LTO tape. So it's gone. I Really don't like tape So the option was let's try Amazon We're close to a data center that's affordable at the Oregon data center But there's there's there's benefits here. Obviously. It's durable 11 9's for both s3 and glacier which makes me think they're the same thing It's geographically diverse and it scales infinitely But there's a problem s3 is way too expensive to store my long-term archive I did the math for this presentation if I were to move my archive to s3 today My first month's bill would be two hundred and seventeen thousand dollars. That's one month So logically you say glacier it's one cent per gig per month and they keep on lowering the price It doesn't work. It's too slow. I need my content back Fast and I need my content back often. This is not a What's the term what banking where we have to push it off and keep it for ten years a regulatory backup? This is not regulatory backup We push this off so we leave room on our tier one so we can bring in new content And then once we need it like let's say we are in summer and we're about to go into football season again The producers will want to bring back all of last year's football games So I can't pay that amount coming back from glacier. It would be too expensive. So those don't work So this led me to find something new These were at the time I started looking my requirements and you'll notice there's two extra boxes because it evolved I need it to be durable. I need it to be scalable has to be fast and has to be affordable And as you can see none of the options that we just talked about really cover it for LTO to be Durable I need to make two copies and now it's no longer affordable So take one check and move it from the bottom to the top for S3 and glacier. Yes, they're durable and scalable But once fast and one slow or once fast and one's affordable So object, this is where we were I started Object kind of solves all this doesn't it? Problem is that not all object is the same So these are all companies that claim to have object storage and we did not want to get locked into a vendor And I didn't want to buy their hardware because it's needed to be affordable So not object. I need it to be I need I need whatever I'm going to put in to be open and yeah, so something open and Obviously swift and this is actually where we started with our archive. I spun up my own swift cluster native Before the swift that guy's got involved and it worked. It was wonderful. I was a I saw the light I was an early adopter. This was back in the Essex Folsom days. I think so not not the earliest but Early is to put it in production But there's a problem here. What happens if I leave Dana wants me to say this the team that we manage is a team of four We've launched this network with even fewer and we brought on more people as we go The systems team is for people myself included So I can't have something that isn't supportable if I end up leaving or something terrible happens to me So I need this to be enterprise grade with support and SLAs So obviously swift and it's with stack in particular Became the option. So I was going to jump into it now and show you what we built and I'll tell you before We start it's working wonderfully We built two tiers Which is unique. I think people thought we were just going to build one and put a local object store It doesn't satisfy all the requirements I still need a dr location and I still need the ability to scale infinitely and I don't really want to scale my local cloud Infinitely, I'd love to be able to have a specific amount in there So what we decided was two to three years of cached media content stored locally in my data center It's fast. It's a right click actually To bring it back into my system and to my media asset management system It scales unbelievably easy with one click and I'll explain why you don't want to do that And it's inexpensive. So a stack was wonderful. They brought in Hardware guides and said this is what we'd recommend and I took that and Kind of added my own little special sauce to it And we beefed it up a bit and made it fit in the budget And then the greatest part is that it integrates natively with or not natively Integrates with my legacy man application, which happens to be this software called the let So tier four is also involved. This is why I had to do this I needed a dr location and Because I was already an object and because I was already going to have a workflow that copied it to assist gateway I figured why can't I then fork that media to both my local cloud and my public cloud? so by doing that I actually was able to meet a Monetization partner who takes our content as you guys love a bunch of ducks fans in here People want that content and we're the owners of it pack 12 conference and pack 12 network zones it so we found a Third party that we've used before that actually licenses that content and then pays us a dividend of it So we meet them in s3. They're already there with their transcode farm and their catalog So we give them access to our s3 bucket read only for seven days And they're able to pull as much as they want after that it automatically ages into glacier And I'll explain that as we go So this is how my media workflow is today. Oh actually is It's similar. I'll tell you the story That green arrow is that onslaught of media the 12 terabytes a weekend that keeps coming in It lands on tier one, which is very fast and then from there I needed a way to get it to both my private cloud and public cloud And I'll throw it over to Martin to explain where Swiss that came in and solve that piece for me Yes, so so like Scott alluded to earlier we brought up the the switch that cluster in there and There are two file system gateways Where the delet software can send the data to one gateway or the other and What happens is that? Correct me if I'm wrong here now, but And it's evolved Yeah, so so some of the the data that Scott mentioned goes up to the cloud where they can be met by their partner and all the other data also goes into the private store which is sitting on premise in a Incredibly beautiful data center. Yeah, well, we have pictures actually and then you'll talk about actually the physical specs of it And the power in this is that I keep it local I can push content to it But I can delete from tier 3 if I need to we never delete from tier 4 tier 4 is our posterity archive It's the achievements and accomplishments of Pac-12 athletes for the next hundred years And when we started Pac-12 that was one thing we were missing. There was no archive It was it was a big challenge when I launched big 10 network back in 2007. They had a huge archive I mean it was warehouses full of Everything from real to real tape to film to nitrate I mean there was so much content going back that we didn't know what to do with it all Pac-12 unfortunately didn't have that because they grew in a different way So that was why we wanted an archive first of all and why it had to last forever And this is why tier 3 and tier 4 made sense. So physically, let's look at the Swiss stack cluster that we have in San Francisco Actually Martin you cover the specs on this because you were the one that told me to build it. Yeah, okay, so yeah so it's to begin with it was about 1.1 petabytes raw and Roughly 360 350 terabyte usable meaning one copy is the usable portion of it so that then translates into three replicas and Every object is the max file size or object size is about five gigs so in some cases the large video files gets chunked up and gets placed as Several different objects in there that gets put together as a manifest So based on the Model of as unique as possible that Swift uses We designed it together with Scott to create five zones Meaning that every zone will have For not every zone, but every object will be placed in three different zones So if one zone which in this case happens to sit with a top or x-witch And have different power supplies if one zone goes away. There's still durability of the data and accessibility So you're looking at three of our zones right there and there's another two behind it that you aren't seeing That the first rack has all of the management controllers the proxy nodes and the file system gateways And then you see the big honking for you drive chassis. Those are our object storage targets, and there's ten of them And it you know it it was a great model for us When we first started because Martin explained that our expansion is very easy in Swiss stack But you have to plan for it ahead of time And so if you notice there's space underneath every one of those racks or every one of those big servers We always put a blinking panel because we want it to look pretty But below that is for you have space and if you look at the bottom We have the same thing so there's an additional ten for our you Bayes completely empty ready, and we Predicted the power usage for them before we ever got to phase two. So Let's talk about why this model ended up working out for the best They'll never forget the day it was October 28th 2013 actually all of you were probably in Hong Kong In fact, I know you were in Hong Kong because you got the email while you're in Hong Kong I was not I was in San Francisco getting a phone call 5 13 a.m Our tier two exploded It lost a storage controller. It didn't beep. It didn't have a red light. It didn't send in a trap It did it just went stupid which put us to half speed and no no parody And this is a DDN if anyone has a DDN or wants one we have one to give away We replaced it, but we basically figured out that we Monday this was a Monday one of our controllers failed of two So DDN shipped us out a replacement controller the next day because that's the next business day So on Wednesday, we actually received it. We put it in racked it up. It was broken. They shipped us a faulty one So Thursday they ship out a second replacement controller Friday it arrives and by the time we racked it up installed it and brought it up We it did worked this time. We lost one bit in our tier two. So the entire Tire volume was corrupted It would have taken weeks to go through because there was no journaling because you have to have journaling across two controllers So with having only one controller the journal was not being written. So we couldn't restore from journal So because of that this is up in our data center and will be for the rest of time if you can't see it Let me make it a little bigger This actually got back to the company and they were not too pleased about this But we lost a hundred and seventy four terabytes of media or did we we had finished our copy to Swift stack a couple days prior So we really didn't lose anything. We were able to restore from Swift stack in a matter of days It was beautiful. We weren't freaking out. We made them stress, but we weren't freaking out So it works the archives already come into and it's paid for itself and it's wonderful and we're very happy And thank you very much and I'll hug you after this. So let's talk phase two Immediately after that we decided we need to expand this because it works and it's easy and it fits into our workflow just great So let's put more in so this past summer with Martin's assistance we More than doubled it. Yeah Yes, so like Scott Told you there the blank panels on there. They are they were there for a reason He had already planned this to begin with there was an extra HBA in in the The storage nodes so that it would be really easy to just buy another j-bot plug it in on the back side and These particular j-bots have 45 slots in and so at This time there's 80 discs behind or 81 to be exact but behind every object node so that one way we're done had it and injected all that all of it Ended up being 1.3 petabytes Additional raw yeah, yeah, and these connected with two cables and power obviously, but two mini sass cables I mean it was the easiest that I've ever done. We just they were heavy I mean these things are hard to move, but they shipped with drives in them, but once we got them racked up It was very easy and now let's talk about how not to expand your archive This is a pro tip for anyone out there when you're adding 450 Brand-new completely empty 350 3 terabyte drives and your cluster is 98% full already, which I didn't know Click the orange add gradually button that will do it To measure slow Methodical process it will usually take a few weeks especially at the size, but it works and it doesn't take away the Any of the processing power from incoming requests or anything like that So I like to live dangerously and I click the big red button that said add it all now so And let me show you what that does It worked by the way it worked amazingly This is Nagios networking graph from one of the nodes. So just realize that there are 10 other ones This is immediately before I click the button and if you can note the scale. It's a 200 Mega bytes per second. We're not in bits. We're in bytes This is immediately after I click the button Notice the scale has now jumped to 600 megabytes per second and this is on one node Again, there are 10. So if you look at that number, we were pushing Close to 5 gigabytes per second throughout the cluster moving it from discs that were completely full to discs that were completely empty And this took several days if you can notice that's those are in days So when all most of Wednesday night all of Thursday all Friday and then we actually throttled it because we realized we couldn't restore So we turned it down a little bit So let's talk numbers so you guys can see that this does work with the budget guys in the CFO In phase one we added about 1.1 terabytes phase two We added 1315 if you look at the cost per terabyte if you remember that first slide early on when I said that to expand Our tier two would be about sixteen hundred dollars per terabyte just to go from 200 to four hundred and fifteen I'm I'm a quarter of that and that was with phase one and as I added more and more drives and using JBODs and planning ahead with beefed up CPU a bigger SAS backplane I'm able to get it down to about three hundred and sixty one dollars per terabyte So the takeaways and this is where we open it up for questions because I think there's a lot here that we can talk about This technology is stable as long as you don't click that big red button. It works great And we're very happy with it. It's saved our butts at least once and I don't doubt that it will save us again And I do I will say that the the biggest caveat or the biggest takeaway I would say is look for opportunities that object brings We would never have been able to meet our archive provider our monetization partner in the in Amazon web services If we didn't have a reason to push it to glacier So I could just put it in glacier, but I can't let them touch it because it would cost them so much to build Pull it back. So that was just something that just the light bulb went off and it's worked perfectly They pull all of our content. We've been licensing like crazy So and then plan ahead Trust these guys. They know what they're doing They walked us through every step of the way and we planned for a future expansion before we'd even put in the first phase So thank you guys And there's our info if you guys want to get in touch Yeah questions For K is easy 8k is actually the harder one. So what we do is It's a great way to say this instant replays if you guys have seen sports You always want to be able to see it from every angle and you want to be able to zoom in on the play Was his was his knee down did the was he out at the base? 2k which what we have right now doesn't really help you do that because you zoom in it looks like web quality from old cell phone days It's really bad for 4k helps 4k gives you a wider field of view and you can leave the cameras Fixed and then zoom into it But 8k is where it's really at because now you can take an 8k camera and they exist I've played with them They're amazing and take a shot of the entire baseball diamond and never move the camera And then you just move a virtual window within the 8k image and you can get down to just 2k native So you can go down to 1080p in a 4 in an 8k image No, the analogy a little demo I always do is take four credit cards and put them in a little grid a little rectangle and then take a fifth credit card That's like moving a virtual camera within a bigger image So what you used to be able to only see with one you can now see with four and then with eight double it again And now you're getting 16. I mean now you can't do math. I'm an engineer It's it's it's that's the hardest part for us and we have to record that constantly So yes, it's here. It's coming our tier one sand as you saw again If anyone needs a DDN sand if you can get it if you can come take it. It's yours We replaced it with an EMC vnx2 about a petabyte and we are planned for 8k at this moment so It would have done that it would have done that so that was my goal I wanted to be able to have one gateway copy the file once and then let it send it to two different object storage locations and Oddly enough the technology was possible. We had actually had custom development done for Maldivica Which I think you guys acquired some of the IP for that That was planned to have to have it work Now the wrench comes in with our media asset management system called the let Delet is not capable of saying if I put you in one place. I'll consider you in two places Delet has to physically move the file once and then say okay It's online on tier 3 and now I'm going to copy another file. Well, it's online on tier 4 So we were actually limited by the delet installation and the delet capabilities We had to do two physical copies. Luckily, we had the bandwidth to do it I mean, it doesn't really slow us down that much So we bought two gateways instead one goes to tier 3 one goes to tier 4 in order for delet to understand Yes, this media is online and healthy in tier 3 and tier 4. Yeah, that's a good question We think alike No, no, no, we're so that's a great part about pack 12 is we are 100% owned and operated by the pack 12 conference Yeah, which is not for profit. So we are we do not have a broadcast partner. So everything you see that we've done We built the network in five months by ourselves Yeah, oh, yeah, I mean we share content with them. Yes, and I lots of friends there and it's a great place They are owned 50% by Fox So that was I got to see it one way and now I get to see it another And if you look at the third option the sec network is a hundred percent owned by ESPN so half-and-half wholly owned for Kind of like a wholly owned but by a broadcast partner. Oh Yeah, oh, yeah That was ESPN right that was a hundred percent Hey Dana Okay, I knew that was gonna come We are in the middle of a very big push into China And I was not aware of this until I started at pack 12 there are professional sports in China They all follow collegiate sports and most of the collegiate sports they follow around the West Coast There are actually 32 UCLA stores official UCLA stores in mainland China So you can walk up and there's a gap and a you know banana Republic and then a UCLA store You can buy a peril and you get no joke I've seen pictures So with that with knowing that we have a huge fan base there and we end up bringing a lot of their students here to attend pack 12 Universities It's our goal and our mission Larry Scott's goal to push our content into China as much as we can in as fast as possible So what we're looking at now is the possibility of having another location another region of our Swift stack that is based in mainland China or as near as possible This would be able for this would be great for us to be able to stage content there And then use it in place instead of having to make the long journey back and forth We are live streaming or will be live streaming to China At least one of our networks by the end of the month Which is actually extremely difficult to do And we are delivering VOD assets to China video and demand They have over-the-top cable system that is amazing. I wish we had it here But we have to get content to them and it's not easy There's that great firewall and it does exist and it does make things hard So that's our next plan is expand this and put it in another location Absolutely, yeah, that's that's the plan when we launched correct me if I'm wrong We didn't have the concept of regions. So we were a Yeah, that's correct. Oh No, I agree. I completely agree and what we can do is we have 12 universities They have data centers that are unbelievable. We have access to them We can partner with them and we could put it into a in fact all of our equipment that we use for our digital Transmission is all in university owned and operated data centers. So yeah, we could buy rack space We could stand up another one down at pick a school. You want to do that tomorrow? Sure Let me go find some data space and some money But yeah, no, I think it would it's it makes a lot more sense Especially when you look at how much s3 would be if I were to put it there 217 thousand dollars a month. I barely spent 300 and I have it almost three petabytes. I mean, it just doesn't make any sense And we are currently in Glacier Under management of about seven petabytes of content So it just keeps growing In time. Yeah in time. I think it'd be smart for us to have our own infrastructure Yes, as long as I can again meet my provider there as well. So we Yes and no Our we're kind of hamstrung because our media asset management system Which if anyone's coming from broadcast, you know that it's the glue that holds all of your pieces together It triggers records it helps you edit and it plays out the content It's the lifeblood you can't not touch it and any any of your workflows Ours unfortunately doesn't speak Swift native. We've asked we've inquired Unfortunately, it doesn't sound like that's going to be something that is going to happen anytime soon or without a massive spend So knowing that that we have to use these sifts gateways And we've got to find a way to dumb it down to the least common denominator for our media asset management system I can't throw more workloads at it just yet but What I'm building essentially is a vault I'm building a vault of Pac-12 content that there's an API for so if I want to at any point Tomorrow and say hey digital team. Let's do a Pac-12 vault. Let's let everybody watch content of all the Pac-12 things going back to 1915 when we were founded as the Western West Coast Conference Western Athletic Conference Yeah, we could do it Again, the other one is we don't have archives. We didn't have it when we started No one saved them or what we have is piecemeal. So there really isn't that I don't have the content yet I have the dream and the vision, but I don't have the content yet, but I couldn't have pictures That's one of our production trucks That we use to produce our events that rack of servers right there on the left is the let So his and then that's our football or a basketball championship game that we hold every year at MGM Grand in Las Vegas. You guys are all invited We fill that thing up and it's a beautiful beautiful event. It's really fun This year actually we'll be doing our football champ game at Levi's Stadium in San Francisco in Santa Clara So if anybody is available in the Bay Area, please Contact me get you down there And I have actual numbers and specs and there's a workshop that I want to tell you guys about Yeah, tomorrow morning Starting at nine we'll be doing a time-warner cable kind of Use case and then follow it up for the rest of the day by a full-on swift Swift stack workshop on how to deploy things what to think about when you deploy it Why you should be thinking about all the things that we just talked about now working load balancing things like that And we'll walk you through a lot of nitty-gritty details and Caleb as it's there in the audience will be leading it Okay