 Howdy, y'all. Welcome to the very last session for today, the one that everyone really wants to be at dinner to already, and I don't blame them for leaving. So we're going to talk about media ads and performance today, and it's a Vox story. So hi, I'm Ian Carrico. I'm a web performance artist at Vox Media. You can find me online at I am Carrico on Twitter, GitHub, Drupal, IRC, my website, et cetera, et cetera, et cetera. Also, this presentation was half written by Jason Ormans. He's the other performance engineer at Vox Media. He is Okor all across the internet, and he is actually on paternity leave right now, so we're all hoping for the best. So what is this presentation going to be about? This presentation is all about two performance devs working on a legacy code base. We have a code base that's been around for a very long time, the Cora CMS, that we have been charged with making the entire site more performance, or all of our sites more performance. It's practical tips on what we have altered to make the site more performance and what we've done to enhance performance across all of our verticals, as well as some overarching lessons that we have learned in this venture. I kind of want this to be a frank discussion of the successes and failures that we have had, as well as kind of what we've taken across. A lot of performance talks, including all of mine in the past, have been based on the idea that when you're starting anew, you can have performance budgets, and when you're doing a redesign, you can do this or that. We don't have that ability anymore, and a lot of people in a lot of sites don't, because we have code that we're going to be stuck with, whether we want it or not. So let's first talk about the constraints that we face when we're working on performance. The first is that we cannot do a redesign. We have a bunch of different sites, and while occasionally we do do redesign for sites, we have not the ability to start from the ground up, migrate to a new system, or to start everything from scratch. We have a huge amount of users and a huge amount of sites that need constant attention, and we can't just stop supporting them for some way, shape, or form. We have eight verticals. So we run Vox.com, Espionation, Curved, Racked, Eater, The Verge, Polygon, and Recode. All eight of our sites. Anything that we do needs to work for all of them. We can't easily single out a single site and say we're just going to work on performance of this. We need to work on performance on all of them. So at any point in time when we release code, we're releasing it to all of those sites for the most part. And finally, we have advertising. I'm sure a lot of people use ads in their sites. Our business is primarily advertising driven, so we don't have the option of just saying, oh, no ads. There'll be a few accountants coming at my door wondering what I just did if I chose to do that. So we have our money pixels that bring in all of the money for the, you know, what time it is. There we go. So our plan. We stepped in and we started almost a year ago. Actually just over a year ago. So we started April in 2015 and sat down and made a plan on how we want to go and attack performance for all of our sites. We decided to kind of do a triage as you would in many different situations. So we start with the big ones. We start with the things that we felt we could make a quick impact and something that could easily, you know, show both the higher ups and other does, you know, the impact of what the foreign team can really do. And that started off with images, redoing how we work with images across all of our sites, how we do fonts across most of our sites. And then after that, we went to the more time consuming options, the ones that we knew we're going to take a bit more time, a bit more finagling, more work with other teams within the organization. So things like ad calls where we had to work with all of our revenue team, all of our sales team, all of our design developers on many different fronts. More image work. There's always more image work to be done that will take a bit more finessing. And also updating our JavaScript and how that is being run. So we're going to start with images. I'm going to slow you down right now. I did not do any of the image work on our sites. This is actually all Jason Orman's work. And if you're interested in learning more about this, again, his Twitter handles Okor, and he has gone deeper into images than I knew was possibly available. However, at first, we looked at delivery of our images and the different ways you can do that. So the primary methods that you can send off images to a site. You have a regular IMG tag, which we use for a lot of things. You can use lazy loading or JavaScript where you load it as it comes to a viewport, or you can load it after the page load happens, but something where you pull that image in with JavaScript after the initial load. Or you can use the picture element, which allows for various different sizes or types to be pushed forward to the front end, depending on your viewport and a bunch of other various variables. And we use, we actually use a little bit of all three of these, depending on the type of image and where the image is and why we're doing that, which we'll get to in a little bit. But we also want to make sure that the images that we're sending to the user are the best images for that user's screen, et cetera. We have opted to do a lot of real time processing. So almost every image you will see on any Vox property has been sent to a server, resized and reformatted to be right for your screen or screens very similar to yours. So there are a couple of options doing this. You can use software as a service. There are some things that do this for you. We use an open source project that we run on our own servers to resize, reformat a bunch of images. You could also, in theory, do pre-processing and have images as they get uploaded to the server, your editorial team puts it up there. Then it just creates a bunch of images on the server and sends those up. We don't do this because we have hundreds and hundreds of articles going up every day. Thousands of images getting processed every day. Most of our articles get looked at within the first 48 hours. So we don't want a huge amount of images stored on a server that won't be used past the first week or so. So, nope. So the open source solution we've been using is Thumbor. This has turned out to be amazing. We can send off different sizes through any single image, different formats. We're working right now on, I think he's doing JPEG 2000 support for this and making sure we can get that for some browsers. The cool thing that Thumbor can do is it can send off either a WebP or a non-WebP version depending on if the browser supports it. So this is kind of the before and after. The left is loading the Verge homepage without WebP on mobile, 3G. The right is loading it with WebP. As you can see, we save, you know, 24 seconds of load time just from loading WebP images. And Thumbor does all of that work for us. We do a little bit of script on the front ends to confirm whether or not a browser can use WebP, but that's it. Everything else is done completely by Thumbor to deliver smaller, faster, more efficient images to our users. Look how you can hear my straw. This is a very hot mic. And then we also looked at the timing of our images. How each and every image loads and when that happens within our load order. So we want to load our critical images first. We want to load the logo. So the logo is usually an IMG tag. We want to load the hero images as fast as possible. The hero images are usually picture elements because that is handled by the browser and sent off immediately. And then secondary images are delayed. So things that images within articles are all lazy loaded. Those are done by JavaScript when you're close to the viewport. So that, one, we don't load extra images that users aren't going to see if they don't scroll down all the way. And, two, we don't accidentally have an image that's further down, possibly halting an image further up that we definitely want a user to see. Then we also use the preloader meta tag to connect to the right subdomains for our images so that when a user comes to the page, already a connection is being made to that server so that we know those images will definitely be downloading as soon as possible. And that's just a meta tag in the head. So just altering the timing on espionation. Again, the left is before, right is after. But we save visually for that first paint. Let's see here. It's six seconds, roughly, more or less. And it's not stopping the full page load will still take longer if you're including the images down below. But by lazy loading the images down below, by making sure that we have that preloader, and by making sure that we use picture element and the most important tags showing up on the right side, the user feels like they're having a much faster experience, a much better experience. And it's a whole lot better for UX. And this is our first overarching lesson, which is tackle the straightforward issues first. We found this is the best for a couple reasons. One, we showed the executive team right off the bat that we're useful, which really helps sell our cause for later work. They were already on our side and that performance is a good thing. But this is a great way of showing them. This is exactly why it's a good thing. But also it kind of builds up team morale right immediately off the front of, look, we can make an impact fairly quickly. So let's let's hit those ones that images were one of those things that we have so many of them changing a few things around like using the picture element for the here images was fairly straightforward, compared to other work that we could nail out within a month or so. So the next thing we did is fonts. And this is what I did it first. We use font face observer by Bramstein. It's a wonderful library that will check to see if fonts are loaded within the browser or not and alert you if they have. You can also use the document fonts, the font events already in native browsers. However, we use, I guess we'll get to this second, but font face observer is a small little JavaScript library that we send off the fonts that we're loading for a page. It returns back with whether or not they've been loaded with some promises. So we set that up to monitor with whether or not fonts are loading on Vox, Eater and curbs. We haven't rolled out some of our other sites because of the fonts forge that we're using doesn't support. They do things weird. Then we set up with Google Analytics, where we would send out to Google Analytics whether or not fonts loaded on pages or not. And we just sent off a quick event. Fonts loaded, fonts failed, or fonts are available. And we looked at the analytics data of, okay, what browsers are failing, which ones are succeeding? And we had about, with just font face observer, I believe it was 95% of the time, fonts were loading fine. There was no issues. But then we use that Google Analytics, and we sent it to Bramstein. We figured out what browsers were failing the most. And we did some debugging with font face observer on some older browsers on how to fix those ones. But also we found out the one that was failing the most often passed, like IE7, which was Google Chrome. Oddly enough that font face observer was failing most in that browser. So this is where we then figured out, okay, if we're failing most in the most modern browser in this, for reasons that we cannot necessarily ascertain, let's use native font events. So for any modern browser that has native font events within the browser, we use that first. And then we fall back to font face observer for all of our scripts. This allows us to load our scripts asynchronously. It allows us to push our font loading further down the critical rendering path to make sure that we have content loads first and then fonts get loaded in whenever they are available. And it also allows us to avoid, for the most part, the flash of invisible text where fonts go away and then reappear on your page sometimes. I say for the most part, we have a very minor issue that's popped up on curbed recently because we have updated some other things. But then we also worked with our design team in every site that we implemented to have better fallback fonts. I'm sure many people who are used to just fonts loading up on our page immediately because they're part of the rendering path don't always look at what the fallback fonts look like. And our design team hadn't, because we weren't loading the A-Sign or A-Sign before, like they kind of sort of thought about what they should be, but not what they look like. But we did a lot of work with them, a lot of QA work to make sure that all of our fallback fonts are as close as possible so that the UX jar is as small as possible. And that took coordination between several different design teams as we're working on several different verticals at the time. But the end result is that things like the Verge and Eater and Curbed and Vox all look pretty good without the design fonts within, with just using the native browser fonts. And as close as possible. The nice thing about Font Face Observer is we set it up to add a class to the body of fonts loaded once it's there. So when that's not there, we can alter the font sizes. We can also like it's not just we have a good fallback font, but we can change the size to be a different pixel size, not just changing the fonts, making it so much closer for all of our main fonts. And this is the next overarching lesson. We could not have done this font work if we did not have buy in through not just our team, but design team editorial. I actually did have some editorial people freaking out when they heard that fonts might not be there immediately. Then once I explained to them, they're like, Oh, right, that's logical, but editorial people don't like the design or the visual aesthetic to be off. The design team doesn't like that. But once we have that culture of performance is an important thing, reaching out to these teams has been so much easier. And to this day, I have a bunch of key people in various teams that will reach out to have like, Okay, we want to change this or that or we want to make just recently, I was talking with editorial teams about forcing HTTPS on our site. And like the buy in for that is so much easier once you have that culture of this is just as important as design. This is a piece of design. So you'll find a huge amount of talks all over is that culture is something you need to do. And this truly is in our experience, an important thing. Also, even with culture on your side, there are still some things that we can't or it's harder for us to do, because it's a little too much change, I guess. So the next thing we attacked, and this is something I just finished a heavy bit of work, actually, last week, Monday deployed is advertising. Our advertising system was built. I forget the original, like for a system pre DFP, that was all synchronous. So we moved to DFP. And before another team did a lot of work to make sure that DFP could load asynchronous eventually. So this has been a long heavy bit of work to go from a very synchronous, slow version of loading ads to a more efficient way. And there's still a bunch of work we have yet to do. So first, for ads that we built internally, Vox not only creates content, we also have actually an announced advertising agency for our sites that we build sites. We have a bunch of people who are creating ads and they use templates and systems that we've built internally, but they don't, they're site builders effectively that don't necessarily know, okay, is this a performance ad, is this slow ad or what have you. So what we did is we have a set of a personal web page test server that runs a test on every single ad as it's published. And then the creator of that ad will get what we call our speed rocket, which will be either green, yellow or red. Green is great. Yellow is okay, but could use some improvements and red is bad. And what ends up as is if it's red, then they come to an engineer and say, okay, something's wrong with this ad, can we take a look at it? And sometimes it's, oh, there's some template work needs to be changed on how the ad is working. Sometimes it's, oh, let's just optimize an image a little better. But we use that system for an easy way for our site builders to know, is this something that needs to be looked at again, or is this something that's good to go? And generally speaking, most of our ads already were pretty good. And this helps us find the little ones. We also use header bidding. Has anyone used header bidding on their sites? It's a relatively new technology. Okay, I'll make this section short then. So header bidding is a way for third parties to compete with Google's DFP servers. So it's a way for us to increase our CPM, our revenue from ads. But it means that in the header, we are sending off requests to three or four different exchanges for ads. So not just DFP, but also other ones. The problem with header bidding is you have header scripts that are going off to third parties. When this was first started, and when it was first added to the page before I was focusing on this, it was all done synchronously. So we have a bunch of header scripts that are just destroying our page performance. And if any single one of those were to fail, or their CDN were to fail, then our site would be delayed heavily. So we have since reworked that heavy amount to be mostly asynchronous. We have like one main script that is synchronous, and the rest are all asynchronous. We set up timers to make sure that if your response time is over, such a thing where you just don't care about your bids. And a lot of this is done by Tecnerati, but we have a lot of work to do with this, and internal work to this as well. And we have some more work hopefully coming out soon to make it even better. But also our ad calling. Our ad calling again was synchronous. So the work that we just pushed rewrote our entire ad JavaScript from the ground up to be instead of one, because it was synchronous in the forest reliant on jQuery. Because it was synchronous, there was a bunch of other little mini libraries that we did. We rewrote it so that it could be run entirely independent of any other scripts on the page. So that all of our JavaScript on certain sites are now completely in the footer. With the exception of our ad JavaScript, which runs in line. I'm not quite sure if it will stay in line. It may move to be something else sooner rather than later. However, having it in line means that we have no internal JavaScript calls in the header until, at all, the only JavaScript calls that are being called in the header are all done asynchronously via our ad cs scripts. And our results are, this is one of the videos that we shared, before is on the left, after is on the right. But we save out 45% roughly on the start render time. We're currently just baiting it on curbed alone. But as you can see, we have a much faster site just by moving our scripts of the footer, by loading our ads more efficiently, and stepping back and rethinking how we did in moving code that was legacy or may not be needed anymore, or there might be a better way of doing that. Now, I joke this is also where you can see we have the flash of invisible text is coming back because we have finally made our critical rendering path so small. Now we actually are having some faunce issues again. So we need to go back and improve that a little bit more. But this is the biggest lesson that I was very appreciative of. For a long time, we were kind of focused on quick result tasks, things that we could show come out. The entire ability of moving our JavaScript to the footer and rewriting ads came out of allowing the performance team to tackle the projects that we want to tackle. Doing this took about two and a half, three months from start to finish. And it was a specific decision by our managers after a spay and we had a lot of work to do. So what we wanted to do was to step back and say, okay, you have all the time in the world, what do you want to do? And this allowed us to tackle work that other front-end developers have been wanting to do for years. Other teams have been wanting to get done, but allowing a single developer to have that freedom, it allowed us to actually go out and really make a change and very large changes that will also actually improve the performance team. There's another team that is so excited that we've done this because it will allow them to write JavaScript better. And the last thing we have is testing. Is all of this actually working? Our very first meeting of the performance team was actually looking at how to test our sites and how to set up testing across the board. We've looked at a variety of vendors. We've built some internal tools to do testing in very certain ways we can look at to know is the change that is being actually a good thing or is it something that's not good. We've settled more or less on speaker. Speaker allows us to monitor all of our sites. It's a wonderful service. I personally love it. Because we are a site, we are somewhat unique in that we're looking at not just one site, but I mean in reality hundreds of sites, but eight main verticals. It's a little hard to get all of our data organized in a logical way as well. But I personally love it. But the biggest issue is finding the signal through the noise. When we have as many third party scripts as we load between ads and analytics and other targeting systems and stuff, and you see one of those things could be causing our site to be faster or slower on a given day. So we have been working heavily on creating some custom analytics. We use the timing metrics within the browser to know when ads are loading, when ads are done finishing, when third party scripts are done loading, all so that we can find a signal to know where along this path are we actually having issues. Which section of scripts might be causing us slowness or is it something that we've done, is it something that we just committed, or is it something that's completely out of our control? We actually started with a friend. I ignore the entire page load time because that could be altered immensely from just a single ad pinging a server repeatedly. It's a useless metric when you have as many third party scripts as we do. But our start render time is something that I look at very directly and that's what I look at heavily. That being said, data is really hard sometimes. It's incredibly important to look at the data. It's incredibly important to make metrics of how you're doing, but looking at straight numbers on a database basis can be very difficult on any site that you're loading, anything extraneous. Do it, look at it, heavily look at it, but recognize that this is an issue, especially with media sites that is not solved in any way, shape or form. It's a very, very difficult problem. So those are all the main things. I want to say thank you. I want to lose some time for questions today. But yeah, do we have any questions, concerns? Anyone working on media sites who found something interesting? Not supporting what? Define road blocking of ads. Yeah. Okay. So the question is if we have, for using DFPA in asynchronous mode, do we not support road blocking ads or ads taking over the entire page? We do. And what we end up doing is if I remember this correctly, we have it in asynchronous mode, but we don't actually run a call to get the ads until all of the ads have been defined completely. And then once DFPA has all the information, then we do a call to actually call the ads. So we, I forget what it's, we're still using single request mode. There it is. And that is something that we have to have on our site. Because those are great ads and they bring in a lot of money. Mm-hmm. Constant more. Mm-hmm. Yeah. No, and that's one thing we don't have infinite scroll in any way to reform. The ads that we defined at the beginning that we are in the markup are the only ads that we have. We've been looking at altering that possibly in the future. But at the moment that's not a concern that we share. But yeah, I don't have a device on how to, oh, yeah, that's impressive. For devices, so, repeat, how do we monitor performance for devices was the question. Trying to get better at that. So our speed curve will monitor not for different devices, but for different speeds and for different browser windows. So that will at least give us a view into relative speeds on devices. But actually for, we don't have any standardized testing set for different devices. Something that we do do is we have a device lab and I personally use our sites. And I personally use our sites not as someone who wants to use our sites, but as someone who travels enough that I will go in different areas with different speeds and purposely look at how our sites are just stealing. I'm a huge proponent on data and numbers and making sure that you have quantifiable data behind what's going on. But at the end of the day, none of that matters as much as how your sites actually feel. So using devices is what I personally recommend over strict data. The question was if we've noticed any changes in site usage. So as far as the things that we have done for like image and stuff, those were all kind of sporadically pushed out. So like one small piece was changed and another small piece was changed. So we didn't notice anything directly because of those. The ads change that we made, we just pushed it last week and next week we're going to be sitting down and looking at the data impressions and traffic and things like that. So I can't really answer that question yet. Hopefully once we have all that data done, then I'll be writing a blog post about it. But we just, using a single week of data isn't enough to know if we're having more or a different type of viewer or if just the latest curb article was really hot that week. Yeah. So for our programmatic ads, we use a system called the media trust. And the media trust goes through our DFP calls. And they, I forget the exact percentage, but they do a subsection of our ads and they run them for not just performance but a bunch of different metrics. One of the things that drives us most is we had a small stint of ads that would redirect users to a different page, which is obviously like the worst thing on earth. So they go through and they scan a subsection of our ads and anything that does not meet our performance or other security standards will immediately be rejected and disabled from DFP's perspective. And I don't know how deep that goes into if like that entire publisher will get removed or what have you. But yeah, we are constantly scanning all of our, all of our creatives, third party creatives. Yes sir. Were we using to share content? What modules? So this is the fun question I get to answer. We do not use Drupal. I'm a reformed Drupal developer I'm going to say. I used to do a bunch of Drupal work but this site is on a Rails. So we built our own custom CMS many many moons ago that we're still using called Chorus. And that is a Ruby slash Rails thing. Although none of the, none of the specific work that we, or I advocate for is specific to a CMS or any back end language. It can be done with anything. If you have questions on specific modules that can do something like this, all of these guys out in green right here could answer the latest, anything else? Well hey, thank you all for coming. I'll be around all day all the first day. And tomorrow if you guys have any more questions I want to come in and talk. Also I would be remiss if I didn't say the slides are at imceco.com slash media perf. They'll also be on my GitHub in a week or so. I need to make sure some back end changes are okay. And then there is feedback on the page for this for how well this session went. I'd be happy if you filled that out. There's a short link but it's also on the New Orleans Drupalcon site. And yeah, it's a presentation by Vox Product. Thank you so much.