 If you've ever attended any of the past MOSCons, then I don't even need to introduce this next speaker. In fact, she would be the one doing the introductions that I'm doing this year. You'll recognize Brittany in many of Maz's best whiteboard Fridays. In a more recent one, get this, she had a real Python while she was teaching an intro to the programming language. Python. Brittany was previously a senior SEO scientist at Moz and a founder of Pride Marketing. These days, Brittany is an SEO consultant and data science nerd working to make machine learning accessible to everyone. And naturally today, she's here to help you use data science to identify why your website performance has changed. Take it away, Brittany. What's up, Mozcon? So excited to introduce a few examples today on how being a data skeptic and thinking differently has saved lives, created our favorite TV shows, predicted future events, and hopefully will allow you all to be more effective SEOs. So to kickstart these examples, we have Weg Dodge, a bit of a legend in the Forestry Service. Weg was one of a team of 15 smoke jumpers back in 1949. And they were parachuted into this Montana wildfire. And unfortunately, high winds picked up setting the fire ablaze in their direction, sending all 15 firefighters up this super steep hill. And Weg did something totally out of the ordinary. And there's a lot of facets to this story, but one of the most memorable ones is that Weg stopped in the midst of all this and he took out a box of matches and he started lighting them and lighting the grass in front of him on fire. Now the other firefighters saw this and he was calling to them and they all just thought he sort of lost his mind. And so they continued running up the hill and what Weg was doing was he was burning out the fuel for this fire that was coming behind him. So when the fire finally came a few minutes later, he wet a towel or a t-shirt, put it over his head and sort of breathed into the earth and was one of the only three survivors. It's an incredible story and he became a legend and this example helped set all sorts of new standards. So it's really, really powerful. Another interesting example of thinking differently was in the early 90s, 400 households were recruited to evaluate this new TV show. And now this is interesting because as SEOs, we praise each other on real surveys, on real customer information, that real company shit stuff. And this was interesting because the results of this pilot were overwhelmingly negative. I mean, people hated it. These are just a couple of the highlights of the participants that watched the show. You can't get too excited about going to the laundromat. Some of you may be guessing as to what this show was and sure enough they ended up launching regardless and it ended up becoming one of the most successful TV shows in history. So what happened here? How could data lead us astrayed this far? And this is an interesting case where it was so outside of the box from what people typically saw and experienced on television that they weren't really prepared for what it was. But it made me think of these other examples and how this parlays into our work. Some of you may have heard of super forecasters. This is a group of elite people who predict future events within 74% accuracy. That is insane, right? That is absolutely wild. And how do they do this? They have all these techniques and we can all learn a lot from these individuals. So I want you to just kind of mull on this question of what do you believe the most important driver of those super forecasters success is? And we'll get back to that. So again, it's very easy to be led astrayed by data. One of my favorite examples of this is the scientific published paper that within statistical significance proved that storks deliver babies. Obviously, we know this isn't true, right? We know that's not the case. However, with statistics, we were able or Robert Matthews was able to prove that you could publish a paper with data that is, you know, it's a spurious relationship of sorts. So two or more variables, they might be associated, but they're not causally related. And we talk about this in search all the time, right? Correlation does not equal causation. Now, I think a lot of you at home are probably sitting there thinking, yeah, I'm see where you're going with this, but I'm smart. I'm not as likely to fall for these things, right? I fall into this fallacy too sometimes. And I challenge you on this because research shows the smarter you are, the less mental dexterity you actually have, meaning that if you score higher on an IQ test, you are more likely to fall for stereotypes or different biases because you're quick at recognizing those patterns. It's absolutely wild and really important to consider in the work that we do. And another reason that we do this is there's this binary bias. It's this natural tendency to seek clarity from complex information. We want to close that loop in our brain. It feels good. We want to do that. But oftentimes, things are a lot more complex than they seem. There's also deep forces behind our resistance to rethinking and being skeptical. It makes the world more unpredictable. It requires us to admit that facts may have changed. Think about search, even the last five, 10 years and how things have evolved. And the third and probably the most powerful is that it threatens our identities. If you remember that WEG story I just told, the other facet of that, that's really kind of shaking is that half a dozen of those firefighters could have likely survived is what researchers estimate if they had just dropped their equipment. But it's a firefighter's identity to carry those tools. And in the rush of that situation, when you're tied to that identity, it's understandable to not have the thought process to drop those things. And I sort of challenge you all today to consider what data should we be rethinking? What belief systems that we are tied to should we be letting go of? And I have some really interesting examples that some of you may have dealt with as well. So this first one has to do with click-through rate. So this is a real example, modified for NDA. By the client, mid-meeting presents this finding that a page we had been working on, click-through rate went from 5.5% to 1.9. And your gut reaction tells you yikes, that's not great, right? And there were a lot of things in the air that day that we had to get through in this meeting. And my gut, I almost tried to kind of, you try to rationalize things in the moment. And this struck me as something really strange. And I had the wherewithal to just say, let me look into it. Thank goodness, right? Because what I found blew my mind. And that was that clicks were actually up, that's not a typo. Over 5,800% impressions were up, over 18,000%. What had happened was the data that the client had been looking at previously were such low numbers. Think of even, you know, 10 impressions a month in Google Search Console with five clicks. 50% click-through rate, right? So with the lower numbers, data can be more easily skewed one way or the other. And this was a case of where they just, they started to rank for a larger number of new keywords. That's a great thing. But it brought the previous low impression, low click way down. But you know, it's sort of this example I like to raise with people is, you know, would you rather have two impressions and two clicks a month and 100% click-through rate or a thousand impressions and 300 clicks, right? Which would you prefer? Now, your gut instinct should say case B. And then, you know, you can challenge yourself again to consider, well, what if those first two clicks are really qualified? And this is a unique purchase. So these things are multifaceted and they require us to maintain a level of skepticism that the situations really deserve, right? And this puts us on better footing with our work. Website traffic falling can be a great thing, right? Let's break some brains today, you guys. I can just picture some of your faces. And this isn't a common belief, but I've seen it happen. I've seen it happen multiple times. One of my favorite examples was this website, I won't name. They were ranking for the color wheel featured snippet. And their products and services had nothing to do with the color wheel. They randomly kind of got this spot in search and were seeing tons of traffic that they didn't want. It wasn't qualified. And they reached out to help get that featured snippet dropped. Super interesting example where once it was dropped and their traffic fell, their data was more accurate and they felt a lot better about what they were doing. And this is one of my favorite quotes by Adam Grant. We laugh at people who still use Windows 95, yet we still cling to opinions that we formed in 1995. It's okay to change our minds. It's okay to change our beliefs. And it's a great thing when it leads us closer to the truth. Another example, even though bounce rate can be a terrible metric, is going up can be a great thing. I've worked and seen several restaurant websites bounce rate go way up. And business had never been better. And when we looked at actual user behavior on the site, users were simply coming to find the phone number or the address to arrive at the location. They didn't necessarily need to browse a bunch of pages of a restaurant. So there's all of these things that we should really be thinking about. And one of my favorite sort of to put all this in perspective is this old story about this horse. How do we determine what's good and bad? So it's the story of this farmer and his horse and one day his horse runs away and his neighbor comes over and says, I'm so sorry to hear about your horse. And the farmer says, who knows what's good or bad? The neighbor's confused because this is the most valuable thing that farmer owns. And the horse ends up coming back the next day and he brings with him 12 feral horses. And the neighbor comes back over to celebrate. Congratulations on your great fortune. And again, the farmer, who knows what's good or bad? And the following day, the sun breaks his leg, taming one of the wild horses. And this story continues and continues as does real life and as does client work and website performance. Things can lead us astray, but how do we determine what's good and what's bad? And so I challenge all of you to consider, we'll keep it specific to Google search console data. What does your site-wide click-through rate really tell you? We saw that previous example and you don't know what kind of keywords you're ranking for. You don't know the quality of the traffic per se. What does that really tell you? What does average position really tell you? What if you just started ranking for a bunch of keywords that aren't really qualified or aren't ranking that well? What does the number of ranking keywords really tell you? Again, we live and breathe by these metrics, but sometimes I wonder if we don't think enough about the root of them. What does all of this mean? And sometimes we often favor feeling right over being right. I've totally been guilty of that. Feels good to feel right. But let's try to view these metrics in a different way today. And there's different tools that help you do this more effectively. And so if you just take a little bit of time, we can get some better perspective. So let's try to view this stuff like WEG. What aren't we considering? And so these are some of the questions among others that I provide answers to in a Colab notebook of sorts that's available to you today if you want to follow along. We're going to use sample data from Google. But these are the two bitly links. I had to split it up because it got too long. But we'll start with that first one, MozCon-21. And then all of the URL performance metrics will be in that second one. And so this is going to look a little intimidating. And some of you might be thinking, why would we bring data over here? Let me show you. So these notebooks simply allow you to write and execute Python in your browser. Super nice. You can share it with other people. You're not going to break anything on your computer. And it's really fun. I promise you, it's really, really fun. And the best part is that I set up these notebooks that once you've downloaded and uploaded these CSVs, all you're going to do is shift enter. That's all that is required to flip through these cells. And then you get to experience and learn what's happening. I made notes throughout so that you can replicate this on your site. In this particular example, we're pulling from this, and I shared this link in the notebook, this Google Data Studio view, where we're actually pulling two different months keywords. And so this will give us a better idea of what's going on. That's the most common question I will get from SEOs is traffic dropped. How do I figure out, how do I get a grasp of what happened? And this is one of kind of the most fun ways to do that. And you learn a lot about what's happening with the website. So, and this is not super conventional. I know there's easier APIs you could be using. You can get really savvy with some Google Data Studio stuff. But in my experience, majority of SEOs aren't totally comfortable using APIs. And so if I can provide a solution that's just a simple CSV export, I want to do that. But feel free to modify however you'd like. So you pull this data in and you've got the two different data frames. And so you can quickly evaluate, do we rank for more or fewer keywords this month? Now not to be confused with actual new keywords, right? This is just volume. You might be ranking for a lot more new keywords. And then you've lost some, right? So again, we're being skeptical. We're kind of considering all of the, all the things. And what keywords have you gained? So I have all of this code information set up in the cells. You just have to run it and you can start to explore this data. What keywords have you lost? This is a really interesting one. And it's really good to sort by clicks, right? So you look at those lost keywords and look at, well, what are the highest clicks for the previous month that we might want to get back? In this example, they're pretty low, but I've seen different client examples where you definitely want to sort of go back and sort of optimize for certain lost keywords. They've got these great magical tables in collab notebooks that allow you to do SQL like queries. So within those lost keywords, you can search for branded terms. You can search for whatever you'd like. You can do some really neat kind of deep analysis on the change of the data. This is one of my favorites. You can really quickly pull all questions from Google search console. So this is a line of regular expression that I wrote probably not that great. If any of you have ways to improve this line of projects, please help me. It was a huge headache, but it really gets me most away. So I'm pretty happy with it. And I hope that it can serve some of you well. It's just a great, easy way to quickly pull all of those questions. So again, this is in that notebook, and you just swap this out with your data and you can do the same thing for the websites that you manage and then optimize for these, right? You can very quickly create a brand versus non-brand column. This is a really fun one to analyze because you start to quickly gain insights and awareness on brand term performance versus not brand. And so you get to see the entire ranking distribution of all of those keywords within a bar plot like this. And then again, you get to apply that brand versus non-brand filter and look at what's going on here. Where are our opportunities? And here it's pretty apparent that there's a stack of keywords lower on page one and just off onto page two that could be great opportunities for this particular website. So putting a little bit of effort behind those that are already showing performance, they're already ranking and helping them rank a bit higher would be really great. You can do really cool interactive visualizations. I have a couple of clients that love these because they use them in meetings. They're very helpful when they have questions. And if you can teach your clients a little bit about some of this data science stuff and give them the power to do some of it on their own, it's really exciting and wonderful to see them use it and continue to evolve with sort of this rethinking of metrics. So this visualization is really neat. You can hover over all the dots. They're colored branded, non-branded. You can do all sorts of things with this. You can scroll in and this is using a library called Altair. You can also really quickly identify keywords that have cannibalization. So keywords that you rank for but have competing landing pages for. Sometimes that's an okay thing. It gives you more real estate, but if you really want to send traffic to a specific place for that query, you should be exploring these insights. This is a really fun one. You can break out categories by URL structure of the landing page. And so I did this for brand versus non-brand clicks by category. And yeah, you can really quickly see the differences and how that's sort of shaping up. And if you might want to shift efforts, right? Are your branded queries going where you want them to? So some of you may be wondering, okay, this is all great, but how do we instill this habit of rethinking or skepticism within the process that you're currently involved in? And I would like to sort of challenge the fact that research has shown that learning cultures innovate more. They just do. And learning cultures, they thrive under a combination of psychological safety and accountability, right? With a balance of those two things, you can really cultivate some powerful rethinking and innovation. And how do you develop that habit yourself? You think like a scientist, right? You define your identity in terms of value and not opinions. Again, we get so attached to these beliefs. I see it on Twitter all the time. Don't even joke like you haven't. So funny, but seek out information that goes against your views. And I challenge you to find joy in being wrong. We're getting closer to the truth and this space is going to evolve. And I'm so excited to see how that progresses. Another way to cultivate learning culture is to continuously ask questions like, what leads you to that assumption? And again, this is in a safe environment where people feel comfortable receiving and asking these questions. And I tell you what, you surround yourself with people who are comfortable doing that and your work and efforts will take off. It's really, really fun and exciting. All right, back to that super forecaster success. I'm curious what everyone thought the most important factor was of their success. Feel free to put it in the comments. But the factor, the most important driver is how often forecasters updated their beliefs. Isn't that amazing? So typical predictors will update their predictions around two times per question. And super forecasters go back and forth on things up to four plus times per question. They're considering all of the things. And this leads them to better predict outcomes than arguably anyone in the world. Super powerful. So something to consider, you know, example of thinking again can make you a more powerful decision maker. It will make you a more successful SEO. And so I'm very curious to hear what will you rethink? And also what can help me rethink even with this talk, right? We should constantly be learning and evolving from one another. These are some of the books that inspired this talk. There's, believe it or not, there's a common thread throughout all these books. And if you're interested, I would, you know, highly encourage you to explore that. These are some of the most brilliant people I suggest you follow. I wish I could fit even more. There's so many amazing people in this industry. I can't speak highly enough about the community that we get to be among today and every day. It's super wonderful. I wanted to make this just a woman's slide. And then there's a bunch of great guys too. So I put this one together and big shout out to that top right. Totally going to butcher his name, but he put together a bunch of the notebook work in that second notebook. And he's been doing really powerful stuff with Python and SEO. So go give him a follow as well. So thank you so much for listening to this. And I can't wait to hear what you all think again.