 Good afternoon Welcome to the great consolidation. This is the entertainment weekly migration case study So this is a co-presentation across two different organizations. My name is John Peck. I'm a senior engineer at four kitchens I'm joined by Matt grill who's an engineer at four kitchens and Preston so from time incorporated He's the tack lead of entertainment weekly So What did we do and who did it? So entertainment weekly is Entertainment news magazine. It's print magazine has been around for quite a while. It's the 40th largest magazine in the United States EW.com serves 13.7 million consumers per week. This is a cross variety of devices And these numbers are a few months old. It's probably a bit bigger now There's a lot of moving pieces involved into migration and we're gonna be talking about you know How each one of them work together and the scope of this migration is kind of interesting Coming from a number of disparate sources including wordpress sites specifically. There were 10 blogs that we were coming from all Hosted on wordpress VIP That included a 52,000 Individual terms that were all in this one flat, you know monolithic taxonomy 101,000 posts are basically articles over 90,000 images and Then the other system was a vignette six, which is content proprietary content management system vignette v6 Dates back to 2007 And it's been kind of creaking along and there's a reason why that they were looking for an opportunity to modernize and move off of it So there was a hundred and seventeen thousands posts on that over 200,000 images 97,000, you know taxonomy terms that were actually split off and many different areas as opposed to like one monolithic taxonomy and Almost 5,000 photo galleries which drive a lot of traffic to the site So as you can imagine there's a lot of moving pieces a lot of things coming from many disparate places all going into one monolithic Drupal 7 instance So a little bit about the development teams for kitchens when we started the engagement, you know post the Post-discovery phase started off with three developers Scaled up to six developers at the apex and then started drawing back when we were getting closer to launch time incorporated started off with one dedicated developer and Moved up to about four developers working full-time on the site And part of the whole process was this kind of handoff or empowering them and making sure that they're acclimated To the system that was built and also building it with them Embedded within our team, so it wasn't really kind of an us versus them kind of thing It was we we're all working on the same project together all participating in the same sprints You know using the same issue cue communicating with each other as if you know, we're all physically in the same organization We used zoom, which is a video chat clients, which is You know has many advantages over a system like Google Hangouts You can support you know more than 25 individual video streams simultaneously and we use what we've referred to as kind of the Brady Bunch view Where you can see everybody at the same time this really kind of humanizes the experience when working with a team of developers as opposed to Say if you're just using a system like hip chat, which is good for day-to-day communication, but It depersonalizes the experience you can't see the people behind them So when you are participating either in a meeting or a daily stand-up actually being able to see the developers really puts that kind of You know human touch to it. You know that these are people that you're working together with So yeah video chat I think is essential to the success of any type of project Especially if there's a problem is one thing to pick up a phone it's another per you know thing to actually face a person and You know talk out an issue or talk or have the meeting in person Another thing that we used is github selfies Which sounds a little bit silly, but like bear with me. It's this Chrome extension that Allows you to use your webcam to Basically take a picture or an animated gif of yourself and add it to github comments so This is very helpful for morale because and also just you know kind of like When you're when you're doing a code review and you say it's like well, this is bad. It's like people take that personally. It's like you know Whether or not it's intellectually correct You know it's like you are being critical of somebody's work But if you can like make a little bit more light-hearted and say and say you know just say like ship it and actually like you know Give Give the symbol for ship it or actually draw a little boat and make it dance You know it's like these like 30 seconds worth of effort like you know really makes it you know more personal and You know here's an example of one that was used when a Particularly a pull request past code review And it's like all right. This is cool So timeline of the project, you know, this is the actual you know github repository That's how much code was committed over the duration of the project the first commit started on April 29th 2014 the launch date was January 29th so 17 sprints two weeks each so we had like this kind of like regular sprint cycle in which you know You do the story estimation You know to you know determine the team's velocity do the prioritization and so forth Make making sure that people are tasked and also like giving you kind of a sense of like when we're going to be able to launch So Across the two teams there were focuses and some and also there was also some Delegation responsibilities across the organizations for kitchens was responsible for the project management of the of the actual implementation of the site for the data structures that were used in the architecture folks come on just to Feel take a seat. We got a number of spaces available We also implemented the migration And implemented the design and advertising now implemented is not the same as actually creating the design or the advertising We were just it's like here are the goals. Here's what it's supposed to look like and so forth So time incorporated they were obviously the product owner they were the people who had like the most interest and also the The business context to be able to know what they needed they provided the design in the form of comps And a lot of the comps. We actually used a third party called notable Which is actually a fantastic service for? Producing annotated screenshots, so it's like here's the photo You know here's a screenshot of what it's supposed to look like with individual areas Highlighting you can go to like say areas 17 and you get a little pop-up that will describe the functionality that is intended behind it You know what fields are being used or what's the what's the semantic intent of this particular piece? They are also in charge of providing the editorial workflow components we there if you look at last year's jubil con at the there's the 2014 session actually goes pretty deeply into the editorial workflow and Kind of some of the prototypes that were being used. We're also going to discuss it a little bit further in this presentation as well and they they hosted the final product and provided the local hosting environments that developers used that mirrored the configuration of the production environments and Managed the build process which you know using Jenkins on the for deployments on their hosted infrastructure and using thing to do the the actual builds of the site So next up we're going to have Preston talk about the standards and documentation and the process that was used for the site Thanks a lot John So what I'll be covering next is sort of our approach to the process that we undertook and documentation things like coding standards So in any sort of agile methodology, it's very important to have a very clear definition of what done means and what ready means What does it mean for a story to be ready to begin work? What does it mean for a story to be done and mark done? So at the very outset of this project we really focused on Producing a very robust definition of what these things mean in terms of a definition of readiness We needed to you know, we required that every ticket before a developer would start on it needed to have a Very robust set of requirements acceptance criteria so on so forth that needed to be slotted into a sprint and tagged for and And also categorized for a release as well. So this was very important We approached all of our stakeholders and made sure that all of our tickets were written in a way that would allow developers to Work at a very high velocity without blockers and For developers we had a very robust definition of what complete means what what does it mean when your task is actually complete? and what this involved was a very Very organized review process that involved both a code review and a functional review Within github poor quests and we also had several product owners who were able to accept these stories during our sprint retrospectives while we demoed them and Finally, of course the importance of documentation for different features if you you know add a new feature that wasn't there before and might require some sort of You know help text or or a read me file those sorts of things were absolutely essential for the story to be marked complete But you know again as with and you know any sort of agile Approach to a project. It's you know that we define this framework at the beginning But we constantly iterated on it throughout the project and it was an it was an evolving framework that That worked quite well for us and and Helped us to really advance our approach to process So all of our code and style standards were algorithmically enforced We used task runners and various tools in order to provide the robustness of code that we required from all of our developers The first was PHP code sniffer, which is a package with the Drupal coder module And we required that any PHP that was written would need to be run through PHP code sniffer in order to detect any potential syntax errors or or So on so forth We also used JS hint to great extent which gave us a very good Look into syntax errors in JavaScript And ensured that all of our JavaScript would run appropriately And finally we use JS CS for code style enforcing things such as to space indents, which is part of the Drupal coding standards enforcing things like Making sure that there's spaces after on I'm as functions so on so forth and finally in order to ensure parity between our various ID is given that we have such a Distributed team and many different approaches to development We really wanted to make sure that you know things like tabs and spaces and so on so forth were matched across all of the IDEs that we were using in terms of our commit and poor quest workflow again, we leveraged github for our repository and One of the things that we really focused on and honed in on it at the beginning of the project was to ensure that all developers produced action based descriptive commit messages that were very clear And that would allow us to very easily cherry pick or cherry pit whatever commits We needed by solely looking at the message itself and not looking at the diff We also required that all developers provided on every opened poor quest a very strong set of testing instructions Which would allow anyone to basically complete the test And this is very useful when you are very siloed into different tasks and for example John will be working on a migration I will be working on some very front-end tasks and he would ask me to test a migration and well I haven't touched migration. What are you kidding me John? But John would provide a very very clear list of steps for me to test and this facilitated a very easy process And what actually happened is that we actually had a lot of cross-pollination where we learned a lot about each other's code by using this method And finally we also had a poor quest labeling scheme provided by Matt in order to you know have color coding and Make it very clear. What stage of the review process each of our poor quests were at In terms of code review we asked several questions of developers whenever they completed a task The first is does it pass code and style standards? You know, did you run gulp? Did you run PHP code sniffer? You know, these are things that are very important to instill at the very beginning of a project Next does it accomplish the task gracefully? Is this the best solution for this problem? Is this the best approach to this problem? And finally does it perform at an optimal level both from the back end and the front end perspective? Performance was a very important consideration for us throughout this entire project In terms of our functional reviews, we you know again We also tested our functionality of each of the different features that we implemented And some of the questions that we answered where well does it fulfill the intended story? You know if the story is asking you to make all of your hyperlinks blue It's not asking you to put an animated monkey jiff after every link. It's asking you to put you know to actually limit that to The the the blue color of it of the link itself and so Ensuring that scope was respected was a very important portion of the functional review process Does it use best practices and this goes back to the graceful solution idea, you know Is this is this something that was written? Well, is this something that you know other developers can point to and say that this was something that that you know Would have been written the same way had somebody else done it and Finally does it avoid technical debt? We really wanted to avoid a situation in which you know after launch We would have to spend two or three or four sprints really sort of cleaning up the project and and so all of our pull requests and all of our our tickets were You know had a lot of scrutiny on this notion of Ensuring that we wouldn't need to do a whole lot of sort of cleanup development after the fact In terms of our development environments, we used we leveraged a virtual machine provided by our timing infrastructure team and We actually provided a great deal of documentation that was project specific and allowed Many of our developers to become on boarded at a very fast clip At the beginning of the project it would take about a couple days two or three days for developers to be on boarded Just because of the documentation was in a very sort of fluctuating state By the end of the project it was reduced to a matter of you know hours or a half day And and this was really greatly aided by our focus on documentation We also had weekly meetings with the infrastructure team and and the virtual machine team at timing and provided very detailed feedback about sort of you know some of the some of the ways to improve or Dance the the the virtualization In terms of our branching standard, so this is a very very you know sort of off-to-discuss topic in In In the end as far as branching goes so in order to Provide a very robust branching standard. We actually used sprint based Branch names and what this means is that we would use a sprint prefix and then the number of the sprint and what this did is it? Really reduced the amount of clutter that we had and made our code base a lot more navigable It it really improved communication because it was very immediately clear. What ticket your branch was reflecting What was the sprint that that branch is going to go into? And so we also had project prefix and then the ticket number afterwards and what this did is it really ensured that? We kept all of our branches at the same very synchronized with our with our with our ticketing system Every time you immerse in the poor quest you would delete the sprint brand or sorry you would delete the Particular feature branch and then once the sprint completed you would actually tag Those sprints and what this did is it allowed us to really target different? Points in our in our code base so that we could you know for example present to stakeholders a very particular state of the code and Now Matt is going to talk about our front end Talk about deeming performance and advertising so We's Aurora as a base theme for the site and we built the mobile theme first In this project separate themes was a hard requirement from the business so we used a edge server to detect the actual device that the client or the customer was using and set a cookie on That incoming request we used the theme key module to switch the Template right and so again or a space theme Here we go. We just took advantage of HTML 5 sass and compass like most people So then again the mobile theme is a sub theme of Aurora and the desktop is a sub theme of The mobile theme and this enabled us to share a bunch of resources that we developed for the mobile theme up to the desktop Themes we don't have to repeat ourselves or do anything more than once So what we use gulp as our task manager, it's great really fast you guys should be using it this compiles We use gulp to compiler sass And also run the JavaScript checkers the js lint and js cs that press and mentioned earlier This also watch for changes in our build system, so they're not to restart or recompile our sass make sure you've got everything So Again, one of the biggest things we built was this JavaScript global Like management system. There's a lot of pieces on the site and we wanted to keep everything from polluting the global scope So there is one global JS object for entertainment weekly that contains Metadata and all the functions attached this gives us a centralized point of entry We can talk get anything about the page from this one object. You don't have to know about anything else and there's not confusion of about where something might live so easy access So and non radioactive functionality So basically with that global object each Page got everything was broken out into different files, right? We only want to load the things that are needed. This is very important on a site that has a lot of JavaScript You want to keep it as slim as possible? So I mean we're using aggregation, but things were conditionally aggregated together only when they were needed And this drastically improves the front end performance of the site So pre-processing You know we are basically taking no template no template data is building the pre-processing Everything is super clean and reusable So this enabled us to have some series of functions on that helped Build data and expose it to the temple air in something that's very clean and reusable so that we can build templates faster And in a standardized manner Helpers and abstraction functions that were built that allowed to take sort of complex actions Get things out of the database in a sane manner So yeah, another really big piece of built with this global metadata object this Took it's attached to that global EW object. I mentioned earlier This is all the metadata about an individual page and it's attached in one location. It's available in one location one scope This avoids repetition every time you build a feature It doesn't always need to have metadata rebuilt for each individual feature This also facilitated a really great access by like crawlers One place to look for something about a page That was very important so beacon performance So beacon performance is the only thing Things like tracking an additional like advertisers We wanted to delay these as late as possible in the page serve content first, right? So we use the defer objects on JavaScript And this put everything sort of low blast when I get the content first to the client and then you know load any additional interactions after that so Great, this is the best part so ad performance right as you know ads are potentially slow and Ads can make your site really slow. And so we wanted to really improve this aspect of the site So in the template traditionally you would do inline JavaScript right for rendering your ad We got rid of that whole entire concept. So there's no ad specific logic in your templates now You don't think it's contained or a bunch of data a trip tags attached to the div that's in the page This provides a starting information for the ad that's rendered avoids in line JS and Ads are then picked up after the page is fully done and in the footer There's sort of a some logic to figure out where the ads are read those data atribs And then add the ads after the page is loaded and this enabled us to render the ads last but still provide ads Rendering before they were previously and this is a huge win for us so Preston's gonna talk about the editorial UI and workflow Thanks Matt So I'm gonna introduce you guys to a little bit of time ink jargon right now So packages and channels are basically glorified aggregate views they were two content types that we leveraged a great deal of in our project and Basically the one of the hard business requirements that we had through developing this project was the opportunity for editors to Provide automatic dynamic content Well, what that means is that editors would define rules for what sort of content would be selected by a query and inserted Into these packages and channels these aggregate views But we also needed to ensure that certain things could be manually curated as well So if you wanted to have let's say one story at the top that was specific You could also and then also have backfill that would populate the rest of the view So initially what we did is we We used timing features in order to Provision our content types and we actually ended up extending these featureized content types to a great extent and they underwent a very robust series of product owner and editorial feedback and one of the big keys to ensuring Adoption across the editorial group was Usability and so we wanted to make sure in these rounds of feedback and these rounds of reviews that editors were actually finding that They're that the step-by-step process whatever they were doing in the node creation process was actually easy to use so what we did is we Our team member Patrick coffee put together a custom module That produced dynamic entity references which allowed us to put in any reference fields that would provide these aggregate views that editors wanted And so what would happen is that you would go into the node creation page And you would have complete editorial control of these views, but you could also provide overrides And this is a custom module that would actually backfill The empty parts of a view based on its context And so as I mentioned earlier there were rules for selection and you could also have manual Manual curation, but the backfill process would actually be automatic So for you know for instance, you know if you have a view that lists 15 articles or 15 nodes The editor could define five of those nodes that would appear In those first five positions the dynamic any reference module would populate the remaining So here were some of the challenges that we face when we implemented the solution You know as as you might have noticed we decided not to use views and and that was because of some Concerns that we had and some of the the the major attention that we focused in terms of performance So the challenges that we faced were really making it very extensible and reusable because this is a this is a system That was used in many different places across the website And also we wanted to make sure that any of these nodes that were that that were dynamically curated would actually be accurate And wouldn't actually select the the wrong nodes And also another challenge that that we really grappled with and well mostly Patrick grappled with was a performance and Ensuring that this feature performed at the level that we wanted The other major concern that we had was just editorial expectations for caching We really wanted to ensure that high-velocity publishing by editors would be facilitated by this process and that was one of the the the major concerns that we addressed during this process and Finally a timing we we are We chose to use a workflow system known as state machine and and integrating with that Particular system was a real challenge as well And now John we'll talk about our content migrations Thanks, president Okay, so as I mentioned earlier We came from a number of disparate systems wordpress and vignette and we're all consolidating it into one place Drupal so wordpress is its own little bundle of wax so Digging into it you have ten different blogs Each one of those was spun up by a different you know different teams Some of them were literally clones of each other used as the same kind of starting place others were set up from that Started from the ground up Each one of those had Different structures, so some of them were just like a complete Stock wordpress installation with just you know a different look and feel of them Some of them I actually had some custom content types in it and also Trying to combine all these disparate taxonomies into one you know into one repository Also, whenever you have humans working on a system you find mistakes also people write Imperfect code especially with what you see is what you get kind of editors So there's an invalid markup that we're trying to clean and and prevent Problems in the future Also each one of these sites had their own set of custom short codes and filters so in addition to all the wordpress short codes and filters that that are provided there was a super set of like Time incorporated a specific brand functionality So It's like well, why don't you just use wordpress migrate wordpress migrate? It is a fantastic module if you were going from like just a plain Jane wordpress site to a plain Jane You know Drupal site With no content types. It's like this is an article. This is an article. This is a comment. This is comments now kiff You know it's not The problem is that as soon as you start getting into custom content types or like fields or like trying to use logic as Part of the migration it it doesn't work out of the box now when we were working on this project This is back in 2014 Wordpress migrate 2.6. We weren't the latest version wasn't available yet. It was still in beta. So we were stuck on migrate 2.5 and It provided us a framework that allowed us to actually create a custom migration by extending the classes that are available within wordpress migrate and Building your own custom migration classes on top of that Wordpress migrate is also fantastic for normalizing the wex the their proprietary WXR structure, which is if you look at it is just kind of like a wacky XML fee feed that is generated by their RSS system and By using wordpress migrate is like a starting point. It really kind of accelerated Development and allow just to focus on the custom logic that was required to make you know all these pieces work And I just won't move to the wrong place So how did we handle this wordpress? We did a lot of pre-processing on the content even before we started the migration itself. So We stripped the comments out so one of the one of the things that had taken place Across the lifespan of these wordpress blogs is there was a transition of the commenting system from Wordpress's built-in commenting system, which you know had a lot of spam and had a lot of problems Moved it over to discuss which is you know asynchronous you can Basically delegate the responsibility for the commenting system in the singles line on and so forth to another system. So But wordpress doesn't really care is like here's this export that has all this garbage that you don't really care And also with like invalid content and some actually malicious content. So it's like we wanted to get rid of that So stripped out the comments also transforming identifiers. So when you have Multiple sites that you're migrating from into the same system that either are clones from each other or it doesn't matter He's starting everybody starts from one and so everybody has a post ID one It Migrate will look at that and say oh I've already imported post one. I'm not going to do it again. So what we had to do is Basically read all the files transform the identifiers basically pre-pended them with the name of the blog That it was coming from and you know to make them unique. So you had one that was a style watch You had another one style watch ID one and you had another one. That was Ken Tucker's TV ID one and so forth So, you know that way migrate would be able to differentiate where each of these pieces of content came from and had the context Necessary to be able to combine them into one monolithic system. We Separated out the authors which is kind of repetitive information within the WXR imports Separated out the images themselves into a separate Jason files. So it was so you didn't have to Traverse literally gigabytes of XML content. It's like, oh, you know, here's a few megs of just cleanly structured Jason content And also move tags onto their into their own files as well So WordPress has its built-in shortcode system and filters which work really really well They also want you to continue. We're using WordPress WordPress by VIP makes it very easy for you to get this dump But it will leave all the shortcodes in place. It won't render them which is useful, but it isn't Because Drupal doesn't render WordPress shortcodes. There is a shortcode module back in 2014 it wasn't that robust now in 2015 is actually a little bit better. It's worth taking a look at now but What made sense for the context of the migration was to take kind of like a subset of these Shortcodes actually implement them in Drupal, but like render the rest of them as part of the migration Process because you know, here's like this old Older deprecated functionality and made sense once, you know But it last time in this shortcode was used just three four years ago No sense taking developer time to reimplement it in Drupal if you're never going to use it again So just rendered as you know straight HTML So most of the shortcodes and filters weren't migrated And so what do we do when you render them as HTML and pre-processing? How did we do that? Well, who knows best how to render WordPress then WordPress so actually took the WordPress Right source code kind of like separated out created this kind of like mini WordPress Pretending to bootstrap kind of installation and actually using WordPress native functions and the the actual you know shortcodes and filters that can't come with WordPress and the custom stuff and You know fake the bootstrap So it would not actually execute in the database But it would work well enough to be able to execute the code natively this happened really really quickly again without actually having to bootstrap It's like hey, is there a caching class return null. Yes, there is good job move on it's and That greatly decrease the complexity of the point, you know of the process rather than having to implement say I think it was between a hundred and fifty and two hundred shortcodes rather than doing that We only focused on about a dozen I Keep pressing the wrong button. So also one of the problems that you have when you have all this disparate content is Normalization of the data across all these places people have fat fingers people make mistakes misspell things. There is a you know Actually one of the tags that I remember Specifically it was cool to use YOLO once they decided they don't editorially they didn't want to use YOLO anymore So now how are you going to be able to you know, take care of like renaming tags combining them? using logic also by Separating them out into distinct vocabularies. So you have something like Miley Cyrus. That's a person And there's a person content type. How do you map it to that? it doesn't make sense to Make a program that has all that logic But editors humans have that knowledge. So how do we translate that knowledge into something that is actionable? So what we did is created this CSV spreadsheet. It's like, okay. Here are all the available terms Here's the context where it came from. Here's the blog that it came from And here's a series of columns. It's like, you know, there's a column for you know Is this a creative work? You know such as a movie or a book? Is this a person? Is this something that should be renamed? Is this something that should be ignored like YOLO? And we took we took those rules and on import basically just It loaded a term if that term is found in the spreadsheet use it use whatever rules are applied by that Editors were happy because they didn't have to do anything that was programmatic They could just use Excel or open office to edit the sheet And then we just took the CSV just threw it in the code base imported it and away we went Vignette is also one of those systems that you know, it's like they're not going to make it easy for you to get out So the TI infrastructure team actually had to write their own exporter for it And used XML once again for the structures Now most of the content was rendered as HTML again similar to WordPress there was a the Had a system called procs, which are basically like short codes. It has the same functional equivalents The short code module that I mentioned before didn't handle those that syntax In the same way it didn't have like an open and closed tag. It was just in one place So we implemented our own short code system again. We Rendered a number of the tags You know things like there was a br tag and a bold tag. It's like, okay Just render that as HTML But for something that has more specific functionality such as embedding an image that will link to a particular celebrity and so forth That's something that makes sense to actually implement that as a short code We also You know took some of those taxonomy terms that When contained in vignette that had like this very specific structure and actually Transformed it into content took the kind of the metadata that was associated with it So something that was a movie for example Was uses a taxonomy term is like, okay, this is now a node that has you know duration a length You know, here's the actors who are in it. Here's the mpa rating and so forth So migration performance especially when you're dealing with hundreds of thousands or millions of objects Is really important because you want the migrations to go as quickly as possible. You want to iterate Often you want to also minimize the amount of downtime or the need for double publishing When actually making the making the switch over so there's a number of things that we did to improve performance Validate what you're ingesting PHP's native XML functions bogged down as soon as they Hit any type of error states similar to you know, PHP warnings and errors If there's a problem it slows down to an absolute crawl. So we used XML lint you know upfront and And basically worked with the infrastructure team at any time that there was any kind of syntax errors like go back fix it You know, let's normalize it with the WXR files. Yeah, guess what wordpress is going to actually Give you some unclean XML, which is kind of a pain in the butt. So there was a List of like five files that actually required hand editing because of that of invalid markup Also in eliminating all PHP warnings and errors No matter what the level is as soon as PHP Hits an error of any sort it puts it into a kind of a recovery state Even if that message is not printed to the screen. It still slows it down It's still gathering information and logging it internally Also avoiding redundant migrations. I mentioned, you know separating out the images into their own file separating out the authors and So forth again, so you don't have to traverse the same data over and over again And also making things as granular as possible. It's like, okay, so you have You know, for example, it's like all the articles are in this one file We'll we'll separate all these content types into their own files So again, you don't have to traverse the gigabytes of data Also, my great out of the box does not cash counts if you're Loading say a thousand items. It doesn't really matter when you're loading 500,000 items. This makes a huge difference Which would you rather, you know five seconds to get a response or 15 minutes every time? It's kind of a no-brainer, but it's not on by default because it you know It takes time and to actually compute that Also, we did our best to reduce the migration overhead WordPress. All right. I'm sorry the migrate module version 2.6 I think in above. I know definitely it's in 2.7 You know gives you the ability to selectively disable hooks migrate 2.5 did not so we disabled The like the solar integration, you know for the search back end path auto meta tag and a number of others During the migration is like okay, you know, we'll get everything in there turn those back on run cron Do yeah, and basically get things caught up because there's no reason to do all those operations Like indexing for example on each individual piece of content. Just do it in a batch at the end Also indexing a lookup tables. So when you have legacy content And you want to do associations you need to have you know, you need to know where that content came from And so you can add it as a field. It's like great. I've added the legacy identifier to the field Those aren't indexed you have to actually go through you know manually put an index for it So when you do a lookup, even if it is just an integer If it's going to still be slow until you add an index to it Also, we scripted out a system that would allow some of the migrations to run in parallels We could run them literally simultaneously on the same system. So you're running WordPress migrations and the vignette migrations at the same time And something that is always you know interesting kind of concern when you're Getting ready for launch making sure that there's a sufficient hardware resources that are going to be available to do the processing because this is you know A very intensive data-driven operation, it's a different kind of load, but it is still a very high load. So if you You know if you're trying to get everything into a tiny funnel is like, oh, no There's no production data or there's no production traffic right now We're only going to give you one server you're going to kind of have a bad time You want to be able to run those things on as robust resources as possible, especially like a database as a you know Prime example is they can want to make sure that it's on a beefy machine as you're working the redirection strategy obviously when you're working when you have a lot of content you want to ensure that you Don't lose any SEO and the transition so you know to make sure that there's You know proper redirect strategy in place and making sure that links to old content still are able to get to the new content location So Drupal is you know can perform redirects. It sucks at it. It's really slow You want to be able to handle that on the edge? So time incorporated has a redirect form You know they're using Akamai and they're using some other layers, you know in there and that you can use it basically pattern-based You know redirects you can also do one-to-one to redirects on a particular URL DNS changes obviously is like hey that if you go to the old Wordpress blog location since they they manage the DNS that they're just like nope. That's now pointing to EW Redirects from the migrated servers themselves. They just write you know for example Apache you just say well guess what everything goes over to this other domain and Yeah, that that got the vast majority of it So the last piece is going to be a performance and caching and this is from that So front-end performance which I was in this a little bit earlier. We'll go a little bit deeper now So css is sass and compass optimized. I'm sure you know this So basically we really wanted to Limit the CSS that was loaded when we were building the site right don't send down your entire set of styles if you don't need them so again JavaScript is linked to very strict standards we wanted to reduce the probability that you do an error if you an error then the whole rest of things blocked and you're going to have a bad day So again chase is in the footer. We put this is the last thing that's going to execute We don't want to get to the situation where you're blocking the rendering of your content with JavaScript And if there's an error then your whole page is busted So front-end caching strategy Basically, we had an editorial one minute published alive goal We really work to ensure that the casual headers are sent for each request And we set a shorter TTL on Akamai than I've rn-ish this give us a nice ability to keep content fresh So and when we were publishing content we were sending purges and this allowed us to keep things in cash More persistently and then clear them when they're updated. So we're not automatically expiring things in generating a more predictable load So again back in performance Again, like John said eliminate all these PHP errors. We did our best to eliminate everything And then for our custom queries Caching everything as much as possible and making sure those details are set accurately And explaining custom queries figuring out, you know, what might be slow what we can optimize And minimizing unnecessary overhead don't do things you don't need to do Be cautious don't add a rocket ship when you just want to make a link blue like Preston said So and we reduce the module count, right? Don't Just take a look at this my preferred side out of module. This will help you out with this So benchmarking again, we tried to analyze everything we were doing to make sure that it was actually the right thing So again for performance testing We really we load tested once we got the Production data in this was a big deal for us because the site did actually behave differently once this happened And so we learned a lot So we use blitz.io and some custom tooling to make sure we were able to provide an accurate Replayable load tests and that's a really big deal is something that you can Load tests that you can repeat over and over again and make a change and run it again and get some like standard benchmarks So we definitely use New Relic for introspection on the server side. This is very valuable especially running right up to launch we found a really Slow thing that we didn't think was going to be slow and then turn out to be terrible And we were able to fix it thanks to New Relic So and we exercised different content types That exceeded the TTL's so let's see you have an article and it has a TTL of five minutes And we're gonna sustain a constant load for six minutes. What happens when that node has to be regenerated We wanted to figure that out and we were able to do that. So We use web page test org It's pretty standard for front end performance filmstrip view and the horrors you can get which can be replayed in Chrome this give us really great insight into things that we're loading and Third-party resources that we're taking a really long time to load and blocking our site We're able to work with our vendors to get that faster so great Thank you all for coming I think thank you very much. There's a microphone in the middle of the room. Do you folks have any questions? Are there's a switch there? We're shot it out. I'll repeat the question. This is Actually a lot of it relates to advertising right and how ads are sold right now And add placements and add sizes between desktop and mobile themes And so and the technology honestly isn't really super there yet for responsive ads And so we wanted to build a system that allowed like two very different displays For the content and then the advertising so and it means it was a requirement from the business in the end So I thought you're gonna ask my great question. Yeah, absolutely. We were definitely using the RDFA and To add an additional metadata attached to the module for or attached to the article for like Google search results An additional like includes Look it up If I'm after this I can tell you exactly what we used to get that done so thank you Thank you. I appreciate it When you say back in the question is how are we syncing back and changes? Can you clarify a little bit? Oh, okay? So we use features across the site and also in addition used the master module to be able to enable and disable Modules across different Environmental tiers so you have dev test in life and you also have a local environment. So Yeah, so we feature the feature eyes the content From the development perspective we basically had a running database that was used and regenerated once per sprint You know basically doing like this bare metal approach But over the course of the sprint we were trying to treat it similar to a production environment where you don't have the option to you Know reformat and redo everything every you know every release. So, you know part of the testing process is okay, you know take take the the database snapshot that was created at the beginning of the sprint that has the configuration up to that point You know run, you know run whatever the deployment process is if it's features if it's the master module make sure that everything is enabled make sure the structural changes, you know take place and You know and also this prevented like you know weird kind of regression errors because field is I'm sorry features is Fantastic at adding things, but it won't remove them. So you got to do hook update and remember to do You know remember to do that so that enforced kind of like developer best practices and was like Hey, this is the real-world kind of expectation that you're gonna have once the site's gonna live So you don't have that kind of like developer. Oh, well it worked locally, but you know, I don't know what the heck's going on in production Not not in not internally Certainly the core team was very familiar with features and you know over the course of the project We added some developers who were a little less familiar with Drupal and obviously there was some sort of Training that we had to do and it's a mentoring that we had to do with that. So yeah other thing that I Didn't mention we created a subset of content to be migrated And so if you wanted a setup system that was had all the features reverted and had everything enabled And you wanted to have some sampled data to test with we had a sort of a slice that was a good representation of all The content that was going to be migrated, but not the entire full set. Thank you That was a manual step In some of it was we didn't have direct control over the build process We could only you know influence, but we weren't actually in control of it So it's like so instead, you know, it's like we made it as simple as possible So there was actually a gulp task that would just you know run all the linting. There's another one that would run You know PHP code sniffer either on the entire project or just on a particular module that you were working on So we wanted you know whenever you Provide tools, you know to developers and you just like use this to enforce the standards You want to make it as easy as possible to use so yeah pre-commit makes absolute sense We weren't in the position to be able to do that So we did the next best thing which is to empower the developers with easy use tools. It's like just run gulp phpcs and And it'll do it'll run the coding standard checks over the entire php code base for the most part I Can speak more specifically to Towards like the the run up to the actual release The code that was put in there up to release, you know was you know clean as possible There was a couple last second you know fixes Once it got past the initial You know initial sprint. I can't speak to that But I you know Preston has been you know championing you know both best practices and ensuring any you know developers who are you know Both supporting the site and continuing to add new functionality still adhere to the same kind of coding standards I have the same you know tool set and knowledge you know available We focused again a lot of effort on Documentation and process and just making sure it's like hey There's like this one place that you can go that you can learn how to work on this project You know where things are what the expectations are you know it takes maybe 15 minutes to read It's not you know not monolithic, but it just kind of sets the expectations and also the poll review process has been You know that That has been maintained by you know time and you know is being used on other projects today We did not use panels on the project and a very logical question is why and This short answer was it was not approved There's been That decision has since been Changed, but by the time that change had been made we had a different process in place Other brands and other projects that are going now are using panels and aren't using views Is there any reason why you didn't migrate the content to the legacy URLs and leave it in the same place so that Was Yeah, that's a that's actually a really interesting question the particular reason we were a Lot of the content was coming out of vignette which had some really really ugly URLs And so the idea on that particular project was to Move it to you know basically clean URL structure slash article slash your month day and so forth with you know with a logical name You know this did have some SEO Implications it did increase the comp you know complexity of the redirects other migrations that have taken place have taken a slightly different strategy and You know for just the reasons that you've you know implied Yeah, especially when you're dealing with a compressed we had we had some kind of the luxury of well time not to overuse a word On on EW on other brand implementations I don't necessarily you know you have an accelerated timeline. It's like let's focus on an MVP The MVP didn't include maintaining those here didn't include a new URL structure So it was been maintained one-to-one. So the TLDR is both approaches Were used and in different situations for exactly the reason you identified We're gonna have time for one more question. I'm unless it's really really quick man. I'm Could you talk a little bit more about how you built your JSON feed for images? It's actually really simple. It was just iterated over the content built a giant array and dumped it using JSON and code to a flat file That was it sorry Yeah, actually we got time for one more Oh Okay, I see what you're saying. All right, so Yeah, so images came from you know just a number of different sources some of them were embedded using You know inline image tags some of them were embedded using short codes and prox and So we took you know kind of a nuanced approach for you know things that were in short codes it was actually really easy for us to Add the add the logic in Drupal to look up You know basically scan the content For that legacy URL if that legacy URL is actually attached to a imported piece of content rewrite that you know to be the new canonical location and apply the image styles to it We can This is kind of the end of the time, but I'll come up and we can definitely I'll give you some more context Thank you all very much for coming. I definitely appreciate