 All right, good morning everyone. It's officially 11 o'clock, so I think we're gonna go ahead and get started So hey, welcome. Welcome to the first session of Drupalcon, New Orleans 2016 And we're here for score superior digital experiences with MLS And I'd like to first throw the mic to Brian to introduce himself Hello, I am Brian Avner director of engineering at major league soccer And hey, I'm Stephen Merrill. I'm the director of phase twos DevOps engineering practice So to start off, how many people here know of major league soccer? All right, how many people here have been to a major league soccer game or watched one? All right, so yeah, so I guess maybe then I'll go on how many people have been to an MLS game So maybe we have some avids here like this is a pretty good distribution of fans. Oh Oh, hey Red Bulls. Yes, I Have you know what I have to actually admit? This is maybe a bit of a shameful thing I've lived in New York City for eight years and the first time I ever went to Yankee Stadium Was to go to a New York City football club game not to go to Yankees game. Yeah, I know I know Yeah, all right cool So, you know this session score superior digital experiences and what we really want to talk about is just sort of some work That we have been doing together over the past year. So to start out with Brian's gonna give you some information about the league and kind of about the group that he works with inside of the league specifically Yeah, so Real top level on on the state of the league We've got 20 teams with plans to expand to 24 by 2020 All along the way, we're adding state-of-the-art soccer specific stadiums around the US And it's it's noteworthy that our average stadium attendance actually exceeds NHL and NBA We're expanding our reach in the US and around the world with great new TV distribution deals and one of the really interesting Aspects of our league is that our fan base is very young and multicultural and very digital hungry a good a good Line share of our traffic is actually not coming from desktop website So we're really focused on building out our digital products, whether they be web mobile app tablet CTV whatever it may be and the the services that power them is is is very important to us So in terms of our fan base, we really bucket them into two categories. We've got avids and casuals we define avids as Fans that are regularly watching and attending MLS matches. We've got these these fans are also regularly consuming soccer rated related content and They're regular regularly visiting our properties, whether they be lead the league site MLS soccer comm the individual club sites or Any other soccer specific sites our casuals we define as those that are watching and Attending matches less frequently occasionally basically and and you know one of our Objectives is to grow bells so turn convert non casuals to casuals and also get casuals to become more avid That's this is like our operating principles in terms of our ecosystem we've got 20 club sites each each club has a website and One league site all running on Drupal 7 in a multi-tenanted Environment so one Drupal instance many sites Steven will talk more about that We've got mobile apps iOS and Android for mobile Sorry for for phone and for tablet. We've got connected television Devices Roku Chromecast and Apple TV And then we've got services which power everything a content API We've a series of micro services that are that are powering our Video and stats ingestion so we our video and stats comes from third-party providers and we have a series of Services which which pull this information and send them process and process them through a pipeline And then they eventually end up in our Drupal CMS and All of this is running within AWS All right So a lot of what we want to talk about is that as we'll say we've actually been working together for a while But for the year from March of 2015 to March of 2016 MLS digital in phase two have worked quite closely together on a pretty large project We want to tell you about and it actually bridges a lot of different areas from infrastructure and DevOps to the actual platform itself to even some kind of management and delegation challenges that were there so We we had one goal in 2015 and that was really to modernize and stabilize the MLS web platform both the back end and the front end and this really meant in practicality successfully migrating 21 sites from Our colo and Drupal 6 to AWS and Drupal 7 so a lot of things were happening at once and also You know part of this is also a redesign so we were accomplishing a lot at once Yeah, so and and obviously you know as Brian said migrating 21 different sites with 21 different sets of stakeholders Required a large multidisciplinary team So in terms of the folks who who work together to accomplish the goals of this migration and stabilization from about March of 2015 to March of 2016 I work for phase two obviously phase two was founded in 2001 We have major offices in DC, New York, San Francisco and Portland And then we also have a bunch of distributed folks throughout North America. So we also have a very remote friendly culture as well You know you you are at Drupal con You've probably heard of us in terms of some of the large Drupal sites Drupal contributions that we've made things like atrium our collaboration focus distro and product And so we are very committed to open source technology We do so much work in the open source space and we use open technology to help our clients succeed and You know I say you're at Drupal con you know of us as a Drupal shop But we're always moving beyond that like as you'll see quite a bit of the work that we're talking about with Brian Is about infrastructure and scaling and DevOps, you know We also have a rapidly expanding experience team of content strategists UX and UI folks So you know in addition to doing large Drupal platform builds We're always expanding kind of our capabilities around the edges. So in terms of phase two and MLS We've actually worked together for a while We've been working together for a little over three years at this point Brian had mentioned that there was a legacy Drupal 6 site and Platform and a new Drupal 7 one we actually helped to build out the Drupal 7 platform Originally the Drupal 7 platform was going to be used for a series of Mexican soccer leagues There is a property called medio tiempo Which actually kind of reports on soccer news in Latin America And we originally built out the D7 platform to be the home for that to actually just be home for soccer news And not necessarily for the league itself Over time though as the D6 platform stagnated and we'll talk about some of the challenges with the D6 platform We said well Let's actually just use what we've built here and make it the home for the club sites and the league site So yeah as I said in March 2015 We actually kicked off this effort to do the migration to what we called mp7 in the words the Drupal 7 platform and to move into AWS So a bit on the team for 2015 We had in terms of capabilities and practices. We had a we had Drupal developers We had full stock full stack JavaScript developers that were developing and react backbone node.js and go We had DevOps And the process we followed was scrum bond which for those that don't know it's it's it's a murder It's kind of like the best of both worlds from the scrum and the Kanban methodology. We we pick this specifically because we felt it really maximized our development velocity and our ability to be responsive to to business needs and 2015 also marked the the beginning of a plan to centralize our development team and What does what this means is that we were moving from Separate teams one for Drupal's platform development and another you know the other for other technologies and we're Trying to start to consolidate. We start what we started with DevOps Because the first point in consolidation is to get our deployments are our management of the of our of our stack As unified as possible So one way to one way to start and stop things one way to triage from from the high level And this kind of is starting our process to to try to get as much centralization as possible So everyone knows as much about about the infrastructure as everyone else So the challenges and solutions for 2015 We had an outdated UX Originally our sites were built for desktop viewing only we had a and we had an adaptive site M dot site that that did not share The same functionality for the most part different experiences We we had inflexible authoring the authoring as as Sort of it relates to how the the pages would be rendered on mobile and desktop We had wasted server capacity so our our because of our high variability in traffic It's very spiky based on games and events. We needed to plan for the worst So that goes to our fourth challenge, which was high fixed infrastructure costs All a while there was some there's poor performance poor performing modules code You know, we all know that the scalability issues that could be challenging with a with a triple site and so we were looking to address that and technology soup this is more of a Just the nature of the environment which makes things hard It's not necessarily that we kind of address that in 2015 But we took steps towards that and this is going to be an ongoing challenge and that we're always addressing as we move forward And the last but not least we had an aggressive migration 21 sites No downtime each of each club and the league has a series of major 10-pole events throughout the year and the we had to migrate and and have no Affect to the users of fans and the stakeholders All right So, you know now I'd like to kind of dig in one by one into these challenges that we were talking about and talk about how we kind of looked to avoid those so The first one that Brian talked about was sort of an inflexible UI like when when the initial Drupal 6 platform was built. There was just the desktop site There was an actual different adaptive M dot site that really had like stripped-down content So it was a very different experience Even for a while there was an even like lower-grade mobile site to at one point which was turned off So the whole idea behind the the D7 platforms are just kind of bake responsive in like in the and so I'll talk more when we talk about how the actual Drupal site is laid out But there was one common mp7 base theme and that base theme had all of the responsive breakpoints already built in So he basically said, you know in order to make this easier and in order to make it easy to have a consistent Visual theme between all the different clubs and the league and in order to make sure that the responsive stuff worked And this actually happened before we got there But Hans who worked at MLS was the front-end lead and actually said hey we're going to make a an mp7 theme that just works at the breakpoints that we want and so You know, we had to make sure that all new features were designed for that But then kind of out of the box we got to the point where all of the clubs just have a sub theme off of the mp7 theme And as a result, you know the colors change for each club the logos change for each club But that way you're not reinventing the wheel, right? You've got this you've got this theme that you know works there was actually also a Kind of internal test club called digital fc that we use is like the testing ground for this as a way to have a club But yeah, this way like kind of out of the box You had a theme that was responsive that worked in multiple breakpoints. So you didn't need an m dot You didn't need like a different tablet site You just had the same content that was being served up and then the layout optimized for all three of the breakpoints that you worked at another big one was editorial flexibility so The way that we built this was designed to give editors the maximum amount of flexibility While still making sure that they didn't do anything that would jeopardize the responsive nature of the sites, right? As Brian mentioned, you know the majority of MLS's traffic does not come from desktop browsers at this point So it is very important that everything that is available, you know The experience is appropriate no matter what device you happen to be viewing it on So yeah, like all the building blocks that we put together the grid of articles on a section landing page The videos lists all these things we had to make sure that they were you know Building blocks that editors could go in and put content into And that there wasn't a way to like break out and make something that wasn't gonna work in a responsive fashion So, you know and another big thing about MLS's platform the Drupal 6 platform was pretty monolithic like everything revolved around Drupal But as Brian said MLS has got a ton of different services now And really the MLS tech team builds a lot of the stuff that is more real-time That deals with things like streaming stats things that you want to have live updates for using node.js And so really, you know Drupal is the CMS It is the core linchpin of where all the content is created and where it's distributed But again, you know this build was started probably like four years ago And so Drupal 8 was still really really in early stages So MLS has a content API That uses Drupal's database, but is itself a node app for example or there is a live Match center experience where avid fans can go and keep track of stuff that's going on during a game And so that stuff, you know that itself is written in note. It uses web sockets so that you can get updates on everything as it happens Using web sockets, but still everything goes through Drupal as the CMS like Drupal is the canonical place Where editors will Will make articles or they will see that the video system has delivered a video Write up some content around it and then publish it so Drupal's kind of the you know the digital hub in all of this And then it feeds out into the connected TV platforms via the API. It's really kind of the gatekeeper So some of the other specific tools in addition to just being able to work with all of these content entities That are being fed in through their pipeline also includes just you know the ability to do really nice image cropping Because of the responsive nature you want to make sure that if you have an image that it It looks good in all the different breakpoints that you're gonna put in the site And certainly something that MLS is perhaps looking forward to in the future Might be to look at taking advantage of some of the better editorial UI in Drupal 8 like I think in some cases You know editors would like to be able to use the quick edit feature for example and just be able to change a Field or two certainly there's there's there's pros and cons of that, but something that's you know, maybe being thought of for down the road So Brian mentioned spiky traffic I think MLS definitely gets very very spiky traffic So there are big traffic spikes whenever big events happen when news about players comes out Honestly when games happen, you know when games are going on Usually the league site itself as well as the club sites for the games that are going on get a big influx of traffic The league site itself represents about half the traffic on the platform So at any given time like that's one that will be getting a good majority of the traffic And so as a result this actually MLS is a great candidate for auto scaling So this also ties into the problem that Brian was talking about of having high Infrastructure costs so when we got to MLS I really like this term that Brian used they were they kind of had one foot in the Cloud right there their Drupal 6 platform was in rack space on physical servers You know no real burstable capacity because again when the Drupal 6 platform was started rack space was a great option and certainly You know they offer 24 by 7 support. There are some really nice perks to using them So Drupal was over in rack space and then the new services that MLS was building the content API the stats ingestion pipeline All these things were being deployed to Amazon web services So yeah Drupal was running in this big fixed set of servers that were Configured and set up so they could handle the high end of that spiky traffic amount because obviously like with rack space They can't like really rack servers for you quickly if you say oh man We're you know every team in the league is gonna play next Sunday Could you have a servers in a week? They might be like oh that'll be a lot of money to get that set up You know there they're not exactly as agile as you might like them to be in that case and for the physical infrastructure in Rackspace you also had things like redundant firewalls redundant f5 load balancers Net app physical storage servers to handle the the files directory So basically it was a large infrastructure that really had a fixed amount of capacity in it So that really led to paying for all this capacity Even when there wasn't a game going on or even worse when the season wasn't going on certainly MLS soccer comm is a great hub For news, but you have you know less visits to the club sites when it's the off season So you know this was a big driver for moving to AWS to be able to have everything in a single scalable cloud infrastructure You know one of the big benefits is auto-scaled instances and we'll talk about that in a second But honestly, you know there are a lot of different providers these days that will give you Linux servers At the click of a button right and certainly they're great for some purposes, but like yeah You could go to digital ocean you could go to linode you could go to these places and get Linux servers that you scale up and down the real operational benefits to your business through using something like AWS come in their Ancillary services like how many people here manage my SQL servers? on your own Okay, not a ton which I mean I I certainly don't want to and so you have things like Amazon RDS, right? So Amazon is not just we'll give you a bunch of disks. We'll give you a bunch of instances They also have things like Amazon RDS, which will save your bacon. I have had Amazon RDS Save me pretty handily because they will just handle like point-in-time restores I mean, I know how to do that in a physical data center, but they just have Amazon manage it and some you know Somebody accidentally deletes 500 articles you can restore to an hour ago, you know That's all just built in so in addition to being a great platform for auto-scaling and running Linux instances There are these other things that offer you a huge cost and time reduction to be able to run your business Like Amazon RDS, Amazon ElastiCache, Amazon S3 to store your files in All right, so auto-scaling How many people have ever like sat on the phone and just looked at a server graph when something was going on and been really Stressed out about it. Okay. That's not a place that you want to be right So auto-scaling has some awesome technical benefits But I think the number one benefit to auto-scaling done well is reducing stress Like we actually did get to the point where we could come in on a Monday morning and look at our monitoring system and say Oh, hey, look, we had some games on Sunday We auto-scaled up to five instances and then at like 10 o'clock at night the traffic had dropped off and we were back down to two Instances so I really think that like the peace of mind for you for your engineers for your ops team is one of the biggest reasons to go to auto-scaling Another benefit even before we talk about kind of the Scaling part of it is the automatic healing if you have two servers at least in an auto-scaling group and one of them just dies for whatever reason AWS will bring it back up. So, you know, I mean certainly stuff happens in data centers You know, you might get a notice that your host is going to go away where one of these servers is running Once you get to the point that you have your setup so that you can do auto-scaling It is a fantastic way to also make your infrastructure self-healing So then the reason that most people use it though is for the performance the fact that you can scale up and down and You can automatically scale based on certain criteria and that's mostly what we did at MLS We would say based on a certain CPU threshold if the average goes over this amount add a new instance into the pool for Drupal sites CPUs are pretty good indicator, but you can actually use anything that Amazon can monitor Another great thing though is that you can also do manual scaling, right? Like if you know we have five games going on with the five biggest teams in the league You can also be like let's just scale up to six ahead of time Pay Amazon an extra 30 bucks this month and then scale it back down in four hours So again that kind of goes back to the peace of mind You can preemptively and proactively scale in addition to doing it in response to monitoring events And of course this all leads to cost benefits If you're only running like two small instances and then you can burst up to say six small instances when you need it You're gonna pay a lot less on average So a way that people like to use this is to talk about cattle and not pets That you should build cattle and not pets and the reason behind this is that it is a shift in how you think How many people here have ever had a pet? That you really love and you lavished attention on okay You if you're going to use auto scaling you don't want your servers to be pets because of that connection Like you have a server that's a pet probably has a name You're really attached to it if it went away you'd be really sad like your primary database server is probably a pet and that's okay Like you don't want that to go away, but your web servers should be cattle, right? They should be ready to be shipped off for processing when their time has come, but This is a real term that's used I didn't make this up but if you ever read something about cattle not pet basically ideas is that it's a mind shift that for at least like The application servers in the case of Drupal like the servers where your lamp stack or lamp stack is running You have to know that they're ephemeral right because the thing about auto scaling is that at any time a new instance could come up Or an instance that came up in response to a traffic spike could just go away So it really does change how you have to think about like doing your builds So when we got to MLS MLS because they were looking at moving to AWS had already kind of You know gotten to a point where they had an auto scaling setup, but it was not necessarily Working all that well one of the big problems with the setup that was there was that it would take a new Amazon instance from completely scratch like nothing installed on it and install everything using its config manager And then put the build of Drupal on there and then at that point you'd be ready to go The problem was that sometimes under load it would take more than like five minutes to do that And if you really need new traffic, you know right now you need new servers to handle that that just you know That isn't enough time so phase 2 came in and said hey Let's move to a fat build where we build in all the software the Drupal code itself And we do that every time that we have a build you know ready to go to master and that way an AMI can come up even if everything else is down like if the config management servers down doesn't matter We have an AMI that has all our dependencies if we can't get access to our Drupal source code repo Doesn't matter because we've already built it into the AMI basically if anything else is down or there's some kind of problem It won't matter and you'll be able to access it So I am actually giving us talk tomorrow that is about immutable infrastructure and how you can run Drupal Using Amazon Auto Scaling or using Docker and a scheduler So if you'd like to see some more about that I'm presenting tomorrow So here's some examples of what we're looking at we turned Auto Scaling on for MLS The top line on this graph is the load average So that's basically the number of processes that are running at any given time and the bottom line is the number of Auto Scaling instances, so You can see that the bottom line for the number of Auto Scaling instances is two And it sometimes ticks up to four when there's traffic you can see that actually like for the first couple days in this graph There's basically an average of two Auto Scaling instances Then for a little while it jumps up to needing three to handle the traffic then it goes back down at a couple points Those are probably weekends when there's like games going on it has to go up to four instances But again, this all just happens automatically we could come in on a Monday morning and say oh cool You know we went up to four the average on this graph. It's kind of hard to see but it's like 2.3 So this means that you know MLS on average is paying for like 2.3 instances and not four of them Which is what they had before Then we later did some other performance tweaks and we got it to the point where you can maybe guess where the performance tweaks Rolled out because the number of instances just goes to two and so generally even for some pretty spiky traffic Like that's a whole week at the end of the graph we could handle it with just two So this is some really great peace of mind All right poor performance how many people have ever had an editor come to you and say Drupal is dogging. It is so slow when I hit submit on this page Okay, it's a thing that happens sometimes right and we certainly had that too, you know We got to the point where our performance was pretty good for visitors to the site but editors had some problems and I will say another thing that MLS toyed with and I say this just kind of as a cautionary tale at one point And you can still read the blog post about this MLS was actually serving traffic out of two different regions. They said hey, let's serve stuff out of Virginia and out of Oregon That eventually ended up having some problems just mainly due to caches right like Drupal's full of caches And then if you have like a whole setup over here in Oregon and a whole setup over here in Virginia So that actually resulted in some extra like cash clearing stuff being put into the code base Which later came back to cause some of this slow performance So we actually found one case where like when an editor hit save it was clearing all of Drupal's caches three times It's taken like 10 seconds and we managed to get that out of there And we actually at the time when you came in some Drupal pages were taking more than a second to load If you didn't like and we managed to track that down. So a Couple of the quick things that we did when we first started working with MLS is we just wanted to say hey Are we using the right AWS instances? MLS was using M4 instance types, which are kind of a good balance of memory and network and CPU Drupal loves CPU in general though. So we said hey, let's use the CPU optimized the C4 class of instances Combining that and upgrading to PHP 5.5 from 5.3 actually got a 40% increase in performance With a slightly lower hosting bill because the C instances are just a little bit cheaper And we also did a couple things like ripped out one of the four caching layers that was there because it really wasn't Helping out. So this is really just to say that you know, it's possible just by like upgrading a PHP version Then you could get a pretty decent increase in performance for your platform. So don't count that out We also did a lot of load testing and using application monitoring tools We had at one point Enabled a thing that was trying to do DB read write splits was actually help hurting more than it was helping We looked at increasing the size of their memcache pool and killing off some of that legacy code that I was talking about and so at one point we were able to reduce the instant sizes by half and also Decrease the page render times by a pretty big amount So here's actually some stuff that Brian put together for MLS management to talk about the work that we did so this chart is from our application monitoring tool New Relic and It kind of Summaries as our our efforts towards optimization and performance And performance increase so this arrow here represents the way he actually installed a monitoring tool We did not have one that monitored actual real-time application performance like how the health of the server is doing at any moment currently in the past So once we installed that we had great visibility into how Poorly our application was performing, but specifically what was slow inside of your app During this time period that we were doing our analysis these red bands represent degraded server Performance that that may or may not correlate to UX Experience so these red bands are indicators that something's wrong and as you can see we really were hovering between Like a like a one to two seconds basically so at the end of this during this process We are we were doing our analysis and we came up with it at the end here with three root causes these root causes Can kind of be summed up as follows we had AWS cloud caching configuration that was suboptimal we had And that I just equated to our memcache settings basically And in an Amazon that's elastic cash we had we saw that there was There was inefficient database access happening through our application database libraries and third was there was a handful of top destination sites that were Responsible for the overall slowdown of the entire site And so these were pages that were frequently accessed things like our schedule page or things like our stats page so by addressing and focusing and prioritizing our Optimization efforts towards those top worst performing pages. We ended up Greatly reducing a response time so each one of these arrows represents a build that had a fix for the aforementioned root causes and The net net is that we were at between 250 milliseconds to 500 milliseconds of server response time That's not the greatest, you know, I've worked on sites that were much faster, but this is a fact two-fold and you know decrease and it's it was pretty it was pretty Pivotal and getting us into the direction that we wanted to go cool. So yeah As Brian mentioned, you know MLS has a lot of awesome tech going on and and as Brian mentioned This is still something that MLS is kind of working to alleviate is the complexity But really I think you know the biggest thing and one of things that MLS is working on now Is that you've got like, you know Drupal world over here, which is Drupal 6 and 7 PHP 5 1 or 5 5 now using Jenkins or build salt for deployment building Amazon AMIs for doing that auto scaling then on the other side You've got like the services side which are go and node based Those were already set up using Docker and using core OS and fleet So basically, you know one of the problems was that like in order to be able to fully understand the platform you had to know salt and Jenkins and team city and Docker and AMIs like there was a lot of different stuff on different sides of the stack So although we didn't finish, you know getting that addressed in this year. That's something that MLS is working on quite a bit So the biggest challenge I think was migration When we got to MLS there were actually three clubs that were already in AWS even without auto scaling that were on the Drupal 7 platform And editors were using them Seattle actually had originally been running their own website up to this point So they actually were the first to be migrated and then Orlando and New York City football club were new to the league So they actually were on mp7. There was not a legacy Drupal 6 site to migrate from but figuring out how to do this migration was really quite a big project management lift You know, we had to talk with the stakeholders from the clubs from the league We ended up actually deciding that we would try to have four different groups of sites to be migrated and that you Know basically for each of the groups we'd get a timeline where we would do overnight migrations Whenever a site was in migration you would continually migrate from Drupal 6 to Drupal 7 Using a migrate that was built on top of the migrate module that kicked off using Jenkins And that way we could continually be putting the new data into the Drupal 7 site So the editors would have the stuff that they had in their old platform in the new one And then they would get training on how to use the new site and the new system Another thing we had to figure out like with this group of four different sets of migrations was how to work around club schedules Because certainly like you've got big games. You've got big events for each one of the clubs So that actually took a lot of work to do in that case as well a really huge part of this too was that MLS has a a fairly small tech team in terms of developers But then MLS also has a group called digital club services and really digital club services Was a fantastic help and their group during the course of this actually doubled in size just because they were such an important asset So that you know, they were not triple developers, but they were really You know, they knew the CMS and know the CMS inside and out So, you know, they were the ones who did training for the clubs Like when we had a group of three clubs that we knew were going to migrate We said hey, okay Digital club services is going to help you by giving you training by showing you how to set up your You know, your schedule page your tickets page all the you know The standard pages the clubs wanted to come in in addition They're also really deeply integrated into the like development team and the development workflow So the thing is like that they kind of act as a tier one support desk Like if an editor in Vancouver Houston has a problem, they don't actually like go right to the developers They will actually go to digital club services and say hey, I'm having trouble doing this thing What should I do and the really great thing about like as Brian said of like building the centralized team inside of MLS Digital is that usually if someone has a problem, you know an editor has a problem that digital club services is going to say One of three things one will be oh, here's how you do that and then it solves which is great The second one might be oh, that's a bug We'll help you work around that and then we will make sure that this gets addressed as soon as possible You know, so if it's a known issue, they might have a workaround or then yeah They might just say okay, you know that is a bug But it's planned to be released next week or next month So, you know, they were the ones who actually helped us to kind of figure out what were the pressing issues that editors were facing Which is important when you have 21 different like editor communities that are using this digital platform Another big thing about this how many people here know about Drupal multi-site All right, and how many people have heard have heard both good and bad things about Drupal multi-site Yeah, it's kind of a polarizing thing and there's good reasons for that So MLS the Drupal 7 platform was not originally a Drupal multi-site when we got there And again when we got there there were really only three clubs on it and the league site was in beta But we knew that we had to like get it a little more stable before we actually flip the switch on the league website. So We took a look at the site itself and said, you know what this actually would make sense as a Drupal multi-site So when you're considering Drupal multi-site, you do have to think about your audience about what level of You know differentiation your different stakeholders will have in terms of like do they need different versions of modules? Are they gonna want to have very different themes? But honestly given that MLS digital basically runs this as a service to their clubs, right? None of the clubs have web development resources like it's not like Seattle's like I want panels 5 and context 4 and I want to do this like those are the kinds of things that make Drupal multi-site hard when it's like Every site wants like a different module or different versions of contrib modules But in the case of MLS like MLS digital really is a service provider to the clubs And so we said hey this actually makes perfect sense as a multi-site because you know We even talked before there's like a an mp7 theme that everyone Inherits from and that makes sure that you are on brand that your stuff works To all these responsive break points that have been defined. So with that we said, okay We'll migrate the first couple sites. We'll do them not as a multi-site But down the road it's actually gonna make sense to run as a multi-site and by going to a multi-site You got a couple things It helps save server resources for one Drupal caches all of its well PHP caches all of the Drupal code into memory And so if we were gonna have like 21 different sites We figured that was gonna be like a gig of memory that we could otherwise get back if we used a multi-site It also just made the deploys smaller like certainly all these different themes With different node modules folders and things and then different copies of the Drupal code base and the same contrib modules If you repeat those like 21 times, you know I think our builds were like 400 megs when we got started and then when we rolled out a multi-site like a build Was like 30 megs. So it just really made a ton of sense Yeah, and as I said that is because there is no bespoke development in each club And it really helped to like centralize future development And another huge thing about this is that so the MLS site platform has 21 different club or sorry 20 clubs and the league and also just six people support it like there are four digital club services folks who Are tier one who work with the editors who work to help prioritize development and at this point There's two Drupal developers that are supporting like a 21 site platform for a major sports league in America So I think this centralization Both of the code and as Brian said of the teams that are working together has really made it so that you know The day-to-day operations and the work on the platform itself can really just be done by six people Which is pretty great So a quick overview of our our milestones. We started in March in April. We launched the New York Red Bull site They were the first one to Be the ones that we migrated in July. We turned on auto-scaling We worked from April and November to actually migrate the sites over to the MP7 platform in October the lead MLS soccer comm site launched on D7 and in AWS. There was also a huge high traffic high Fan impact event that was new for 2015 called decision day that happened in October that the platform stood through In December we managed to turn off the legacy hosting provider in January of 2016 I'm actually very happy that we Brian and I worked together to help hire an engineer who had DevOps skills Into the MLS organization So we actually did kind of help hire our own replacement on the infrastructure and DevOps side and we're still working with MLS on like the Drupal platform side and then this March We all sat in a conference room at MLS digital headquarters and we had a successful launch of the 2016 Season with all of the infrastructure in AWS and completely on the new MP7 platform So a slight step back and just to go over the tooling that we used to accomplish a lot of the performance Optimization with without which none of we really could not have gotten the site to perform the way it does Basically three tools new relic for application performance management I've mentioned this before I'll go a little bit more into depth about what that looks like and how it helped us blaze meter for load testing and Data dog for our infrastructure metrics. I'm essentially the you don't have to have these exact tools But in my opinion you need Something that is that that accomplishes what these tools do so something that is look that is able to report on how the health of your application in real time And if you make changes to address Performance you need to test those changes with some sort of load testing tool and then you need some kind of Monitoring of your infrastructure and in totality to see how it's doing with regards to things like disk reads CPU utilization server health So new relic I'm kind of Breaking this down a little bit more just because of how crucial it was in in in our operations So what it is in my opinion is in it is table stakes Tooling for the modern technology team. So whether you're using new relic or one of their or a similar APM tool like I think I believe app dynamics is another one It's absolutely essential to understand How your site is performing at any given time and also how your site might have not performed in the in the past What it is is it provides real-time and end-to-end client and server metrics and performance measurement so This is really the ability to look into everything from how your code is performing in a user's browser to how the Drupal modules are performing in the server to even how the database is responding to Two queries and the slow performing queries So a good example of really like what what we get out of it in terms of like a user story is We can use a new relic to show us that an article page was slow Yesterday for a user in California using the Chrome browser at 9 p.m We can also find out that the slowness of that particular user was 70% on their browser So it was JavaScript that was slowing down that user's Experience or the opposite we could say well, you know, it was actually server lag that was responsible for that slowness and That kind of visibility is is crucial to really Figuring out what your application problems are I'm gonna go over just some kind of things you can get out of New Relic So right what we're looking at now out of the box after you install New Relic was minimal configuration I might add you can get a per of you of your entire stack application Server I'm sorry your client code The middle tier there is a server and on the right there that is the Repository so that would be the memcache instance the Eos API that's our API or web our web service which has our Drupal sites rely on web services and then on the bottom there. That's the the database instance now You can also drill down And so if you expand each one of those I can see oh, wow look here's how my browser How is here's how the average performance of my code is running on a browser? Here's how that relates to the server and here's how that relates to its External web service calls or the database or the what have you now? This is this is an example of when we had a performance hit on our server so as you can see the our the the middle server Module is red and that's signifying that there's a problem there not on the browser not on the database But on the server code and that will happen wherever New Relic Locates bottlenecks Here's another view this new relics also self-aware of the application that it is in that it is Interespecting so New Relic knows. Oh, this is a Drupal app Immediately it gives you a section called Drupal with a breakdown of modules Hooks and views so from here you can see I can see what's my slowest module that's performing or what is the the Slowest running module so one is you could have a module that's called by everybody and so therefore that will be slow the another Scenario is you might just have a module. That's not called a lot, but when it's called it is dog slow and You can do you can look at this with modules hooks and views no configuration was done to get that Here's an example of what the high-level Metrics from our multi-site look like so this is an aggregate of all of the clubs in the site So we can look at our aggregate of all of our editorial Instances our admin instances in our roll-up of our 2021 sites in in our in our Production instance and you can see the average is about 5.6 seconds on the end user experience The average is about for you know below 500. Sorry around. Oh actually we're in a bad situation right now at the 1.4 So It really does work So we're you know we're about like above a second on our on our app server You can also then look at individual sites within your multi sites out here This is the Red Bulls and NYC FC and I can see specifically how that code is operating with the NYC FC and the Red Bulls traffic All right. Yeah, so in summary just to go back to kind of what we talked about By the end of our year, which is basically from March 2015 to March 2016 We had gotten to the point where all of MLS digital services were mired fully in the AWS And this in turn helped to significantly lower the hosting cost just mainly because you know rack space or any like legacy host Is pretty inflexible so both by using Amazon services that took less money like auto-scaling in the average case or less time to Maintain like RDS That helped actually free up MLS to like actually build new features for their Dribble platform or Build new digital products like build new things into the match center experience into the connected TV experience We also really helped to reduce support load So as Brian said like kind of a big part that was happening not even related technology But still a very important part was the growth of that centralized digital club services team and integrating them Very deeply into one combined development organization so that you know You've got stakeholders at MLS that are driving new feature development You've got the digital club services group who are really kind of like tier one of supporting the editors in all 21 of the different, you know instances of the site and then they helped actually like drive development to say like hey We need this bug fixed before we migrate or then like this club really wants this feature And so yeah like not every change is technical certainly quite a few changes that will help your organization be more efficient Can completely be about governance and about policy and again? I'll plug that tomorrow. I'm talking a little more in-depth about immutable infrastructure and how you could make your Dribble instance suitable for running in AWS or in Docker So yeah more specifically MLS address the outdated front end UI by having a consistent mp7 theme that was sub-themed by all the different clubs and skin to keep a consistent design language and to work in responsive Giving club editorial teams kind of a couple more tools to work with but still very carefully making sure that that all still fits into that responsive platform Getting to the point where MLS was doing auto scaling so that they could automatically scale up and not worry so much about Spiky traffic and also help to lower their overall hosting bill by using those kinds of services Still in process, but trying to get to a point where there is really kind of just one technology stack like there's always going to be I think at least in the near term there's going to be Drupal as the main CMS and JavaScript whether in the browser or on the server as the main component to all the other services part of the platform But trying to move towards a kind of coming together of deployment strategies Yeah, and you know in the end we actually managed to get all 21 of those websites live on Drupal and in Amazon web services So what the future looks like for us? 2015 is really just about getting the D7 and getting in the cloud and we did it 2016 It's about making further performance is always going to be a goal in my opinion, but yes increasing performance, but Now focusing on the editorial usability of that you know of that migration and really looking at all the modules of forms The editorial experience and and how we can facilitate them to tell the stories. They want to tell and not feel so confined by the tools that are at their disposal, so that this is a major focus And you know as I keep mentioning the consolidation and simplification of the tech stack is Important to me now just to clarify what what I mean is by this is Every problem has a that's a tool that is that is best suited for it And by no means do we want to have like one language on one platform, but we do want to see what? You know use the right tool for the right problem. So in terms of content management Drupal is the right tool in terms of Video ingestion and data ingestion we were happy with with both node and go but in terms of deployment there's really should be only one technology for that in terms of configuration management the same thing and Consolidating here is where we started and we were continuing to to to You know work on and this is going to have pretty big Organizational impact as it allows us to be continue to be light nimble and get a lot done with very with very little and Fine and actually Docker as a sub as a sub note There is is kind of one of the the crucial technology that's enabling that And finally preparing for omni omni channel omni channel is is major for us like I mentioned before The desktop is one of the many products that that our users can see more content on so we have to make sure that that everything from our content model to our Distribution of that content is is really optimized for for the consumption on any form factor whether it be connected television or or a Chatbot we know whatever it is. We want to make sure that we have us We have the ability to get our content to that to that device to that to the right channel and at the right time the right moment for that user and You can check us out that we've got a we've got a technology blog where we constantly talk about the new tools We're using what what's working for us at labs at mls.com the league site and You can visit phase two at phase two technology.com Questions none So if you have a question, please go to that mic and turn it on You just want to walk up to the mic and ask the question. Yeah, you might have to turn it on too If it's not working, I can just repeat it for the recording. Yeah about that. Yeah, I think so. Yeah Well, I was just curious as you talk about the future and Omni-channel devices and internet of things you got stadiums like I was out at a bias stadium last year Trying to make it like this whole digital experience and interaction like about hyperlocalization and stuff real-time delivering to different, you know, and I'm not kiosks and make jumbotrons or whatever tracking player Well, definitely player tracking definitely fan tracking would probably be the high priority of all of what you mentioned in terms of IOT I think that's something that's that is that's that's very interesting to us However before we can even get there It's you know, you definitely have to have the foundation and the pipeline to To transmit information to said devices respect to the stadiums the our product Directors in the back there you can speak more to that but really each club and each stadium kind of has its own little world And they have their own sort of technologies that that they're using and so Back to the Omni-channel. I mean if we can provide a consistent And consistent access to our content then that would enable any button whether it be the stadium or a Partner a content partner to to get our content and there has been talks here and there to sort of integrate And augment the stadium experience with nothing specific and we don't have anything in the immediate plans But in my opinion, it's really about getting our services to be quite performant and be fully robust and Expose every kind of content we create through an API and that will I think will set us up for things like that Hey of the 21 teams were there any that were kind of special snowflakes, you know And they had their own ideas about the creative direction and the platform and weren't really on board And you had to kind of navigate that a little bit and bring them along or were they Everyone was sort of on the same page Wade in the back. No, no I will say this so so because so I think a consequence of having a difficult Request for service that you're providing is what is it? What are the capabilities in the said customer, right? So if you if they have technology a staff, they're gonna want things that are gonna be difficult, right? As it happens, there there is no real technology staff at any of our any of our clubs currently at the moment other than Mobile some mobile competency to create ancillary about web apps for said Club like Seattle has its own web app. That's kind of its own thing Although it does use our API to consume its content, but because so because of this because of that situation We didn't actually we were able to move it lockstep and kind of roll out changes and everyone kind of Accepted it. I'll reiterate what Stephen mentioned having a centralized group We call it DCS or digital club services But a centralized group of power users that are really the face of Drupal and really the tier one to technology It's kind of crucial because the whole time they're managing these changes and they're kind of Communicating and and to the to their tour clubs and and making them feel at ease about the changes of one way or another And and that really does set us up to to move in in in a multi-tenanted environment Which I think is kind of what you're getting to because that's a that's a problem with multi-tenanted. I know anything else Thanks for the presentation and definitely thanks to MLS for sharing you guys. I've seen you before at other Drupal events and it's been really very useful for Great. So my question is about multi-site So when you push out a release and you have 20 sites running the same code base And let's say there's a scheme of change Do you do anything special to handle the up DB's and the feature of earths that brings them all down? Yes, this is the maybe hidden or not so hidden downside of multi-sites is what happens when? a contrived module like Renames a field let's say because then you got a big problem because if your old code like old code before the scheme update Happens will work and the new code will be like this field's not in the database crash Then the same thing happens when you update the scheme and then the old code base will be like Oh, the old fields on the database crash. So what do you do? So when we initially designed this I was a little bit lying We didn't we didn't do this as a straight-up multi-site out of the bat We had one extra thing in place where during a deployment We would deploy the old code base the new code base Then we would do the scheme update and flip sim links one by one so that we could like essentially We were still doing that deployment where we said, okay And we do that on the editorial tier of servers which are kind of pets They're not autoscale They're a separate group because they're like, you know behind a more restrictive firewall only accessible to club So in that case we could actually do that like flip the sim link run the scheme update go flip flip Flip and then actually make that live so that certainly is a problem though And you know It would be nice if contrib was a little better about like well That's not just like rename a field that's instead you know like change the semantics and maybe like provide a more graceful path But you will occasionally still that's probably the biggest downside of Multi-site is that you just will look at your scheme updates carefully because yeah There's nothing that could stop a mod from just been like I'm gonna delete this field And then the code will be like I have a PDO error because I can't find this so it is not perfect I will say that like we have our stopgap solution there of being able to at least for the big scheme updates like to run them And then like flip the new code base and run it on a site by site basis And then like kind of let that you know catch up as we deploy the new AMIs out But yeah, that's definitely the biggest thing and I think that that's probably the thing where you have to kind of be the Most careful Question you mentioned doctor, but it wasn't clear to me whether you're actually using it now or that's something you're moving towards Yeah, I yeah, so Yes, and yes, we're using it now any new technology that we introduced to our stack is fully doctorized We the only thing that is right now not doctorized in production is Drupal And we already have a doctor doctorized Drupal version ready, and we're just waiting to release it during some downtime so but everything like our the Apps that run our Apple TV or API our ancillary websites which provides stats everything is doctorized and Drupal will be doctorized very soon And that will will have everything doctorized by the end of the year Actually, I'll throw a plug out for my favorite container scheduler, which is kubernetes MLS is using kubernetes We did quite a bit of work to kind of decide on that and one of the biggest reasons for that is that MLS has a bunch of node apps and node apps are great except that they only use one CPU core right because node is single threaded So there were several cases where node where MLS had services that would take very high traffic And eventually just that some traffic threshold like one process would not be able to handle it The cool thing with kubernetes is that it has built-in kind of service discovery So you can say like spin up a service that handles our web socket traffic And then you can literally scale just that service you can be like oh Hey, we need to go from five pods to 20 pods and that way you can on the same physical machine Be running like 20 copies of a node app that can use it say 20 cores of the machine as opposed to like having to build in the Node cluster module and do all this extra stuff So it actually has an additional huge benefit for a lot of the node-based apps that MLS runs kind of that kubernetes Like has service discovery built in and has load balancing based in which then also means you have horizontal scaling Baked in like you can just say hey, we know it's going to be a high traffic day Let's go ahead and just scale up to 20 of our web socket instances And there is even in kubernetes one to like an experimental Autoscaler for pods like for containers themselves You can say like if the average CPU of these containers is high add a new one in so that actually I think fits Really nicely with some of the node-based services that MLS is running I wanted to know if you could tell us a little bit more about the video ingestion You mentioned some of the tools you're using. Where are the sources for the video? They come from the editors you have outside feeds. Do you allow fans to upload photos or? Great question, okay So we have two main sources of video one is our live videos provide provided by a vendor new lion our encoding in our Vaude video is housed in a video CMS and a service with Uyalla So we don't use bright the alternative to break up So that's the two points where video assets would be introduced into the system The way that the ingest works in a nutshell is we've got a series of microservices Which is constantly pulling these systems and sinking them down into Drupal and then also bi-directionally Sinking them back and it's these it's this pipeline of microservices Which which allow us the ability to to focus on in you know On on getting things from one system to another in in localized way so that if one thing breaks not everything breaks So the transcoding the conversion is done with our video provider So I know that there's a lot of people a lot of people deal with their own ink They do their own encoding we were we kind of go with You know in terms of bill adverse buy if the vendor can do it and they do a decent job at doing it Let them keep doing it until there comes to a point in time where we do have to take it on ourselves And at this point we're not so yeah the encoding happens with Uyalla. Oh And then in terms of UGC we we have integration with our Twitter vine Like everything Instagram and that is integrated into our our match experience our match site called Matchcenter.mls.com and that that is primarily where users are uploading content So we don't use Drupal for UGC. We use social platforms. All right anything else All right, thanks guys All right, and then I think I'm contractually obligated to say That if you go to the session page you can leave comments or questions and let us know how we did and again Thanks for coming to our session