 Hey, everyone, my name is Sharon, and I'm here today to talk behavioral analytics, what it is, why it's important, and how you should be using it to build really great products. A little about me first. So I've been a product manager at Microsoft for almost four years, working in the metrics and latency spaces. I'm currently a PM on the insights team at Microsoft Clarity, which I'll be getting into a little more later today. But for now, what I wanna do first is introduce you more broadly to behavioral analytics as a field. Let's get started with the basics. So in the analytics space, you can track two kinds of data. The first bucket is quantitative. This is countable, measurable, usually the box of what happened. The second bucket is qualitative data. This is more description-based, more observation-based and descriptive, usually of why it happened. Most companies and businesses rely a lot on the quantitative data aspects to make business decisions. As a PM or someone who really cares about products, you might have at some point tracked some of the following traffic data, sessions, page speed, pages per session, bounce rate. All of these are your classic traffic analytics, which is all in the bucket of your quantitative data. When it comes to qualitative data though, it's a little more of a black hole. Not everyone is familiar or knows what to do here. And that's a problem because when you don't have the qualitative bucket to explain the why behind the trends you're seeing, people start making up reasons or ideas that might not really address the issue at hand. And it's hard to build a really great product like that. It's like saying that you know revenue needs to increase, but without qualitative data, it's hard to understand the root issue. It's hard to know what your users want and it's hard to prioritize the right solution. And to make matters worse, sometimes you even have the hippo, aka the highest paid person in the room calling the shots, meaning you might end up investing in features not because the decision was based on data, but because the decision was on the whim or the idea of someone higher in the team who just happened to be there. That was just a meme, but let's take a real example of how this might manifest on your site. So let's say you just shipped a big change on your site, you're really excited, you're watching the metrics. Instead of all of them moving in the direction you're expecting though, your traffic metrics, aka the quantitative data, they're all moving in skewed object directions. Conversions are down, balance rate is up, page speed is slower, but then also total sessions and pages per session are higher. What is going on? How do you even start to investigate and root cause? This is really where your behavioral data, aka your qualitative data comes in clutch. We can actually watch an actual user session and see what they're seeing. Let's take a look. What appears to be your standard website experience, also standard cookie experience asking you to accept, quickly turns into something that seems problematic. There's probably a bug. The buttons are not working or being pressed as they expect. And when users trying to exit out of the overlay, nothing is happening. So the bad news in this case, user was frustrated, left the site and you lost a potential conversion. The good news though is that with behavioral data, it's now straightforward, what the problem is. We can go back to the team, recommend the fix and then see the members return back up. This is an example of both behavioral and traffic data working in tandem. Your traffic analytics data is the one that gives you the general sense of what's happening on your site. It's a quantitative data we all need. It'll tell you when your KPIs move, it'll let you know when you're all up metrics move. But behavioral analytics is the qualitative data that game changes how you understand your user's experiences. What parts of your site does the user hate? What part does they love? And really only with both analytics together can you get the full picture of your site experience. Now to orient our conversation since we are at product school, we'll focus here on maybe the main cases for product managers. PMs determine the roadmap and strategy for the product, what features need to be built and invested in. And to do that, PMs need to fundamentally understand the users. How users feel about the product, what the users are expecting from the product. That user understanding should drive and be the heart of everything, including things like determining the ship bar for when a feature gets launched. My goal today is to take you through how exactly behavioral analytics answers these key questions for product managers. Let's start with the behavioral analytics tool set. So you'll typically find three core tools. The first one, recordings, show you how your users navigated your site from their mouse flow to clicks to abundance. Heatmaps are the aggregated view of recordings across all your users. The three key heatmaps are click maps, showing you where your users are clicking, scroll maps showing you where users and how far down they're scrolling, and area maps, aka popular aggregate elements on your page like headers or flitters of your site. And to round it all out, user experience metrics, these are typically signals that show frustration like rage clicks or JavaScript errors. And they're a really useful way to isolate sessions with big user pain points. So let's see how all of these actually work in action with a few different case studies. So our first example is how behavioral analytics really optimizes for visibility. Traffic boom is a marketing agency focused on the traffic, sorry, not the traffic, this travel sector. And their big business question to answer with their site was why conversions were so low for one of their clients. They tried AB experiments, they looked at traffic analysts and just weren't able to debug the full reasoning behind it. They found the key though with behavioral analytics. They specifically looked at the scroll heatmaps, which if you remember, show how much content users are actually scrolling to see. And the key piece of this is that less than 50% of users to the site are actually seeing the main call to action button, which is the shop now section that is in purple here. And the reason was because it was so low on the page, the client hadn't realized that this button was out of view for most of its visitors. So naturally the fix is to move this button higher. And with that, they were able to move it into the area of the page that most users are seeing without even scrolling. And it made all the difference. Now visible to all the site visitors, the CTA was more often click and amazingly, travel boom saw a 30% increase in conversions for this client. And it doesn't just end there. We see similar results when improving product education. Hurley is a DIY bubble tea kit brand. They are a product that launched during the pandemic. So you can just imagine how heavily they rely on online sales for their business. Their business objective here was similarly to improve conversions. And they use the click heatmap to actually notice that users are clicking on their learn more button more often than the actual main conversion button to add to cart. And when they looked at session recordings of users who clicked on learn more, they saw that folks were actually trying to figure out how the product works. So these were great indications that the users had more questions and existing product education wasn't enough. Hurley took this insight. They built a quick animation module directly on their homepage that taught users how to use the site. And the result was a 35% increase in conversions. Another scenario where behavioral analytics shines is with simplifying the critical customer path. So ProProps is a cloud-based SaaS company. They offer products related to training and customer support. In trying to optimize their own site, ProProps had created a full traffic funnel based on the site's customer journey map. And what they found was that the largest drop-off point of their users happen when they are asking these potential customers to sign up. There was good conversion once the signup was completed, but the conversion rate during the signup process itself was not meeting the expectations. They watched a few session recordings and finally figured out why. Perspective customers were dropping the signup form in the middle of completing it. It was too long and a little complicated. So ProProps then reworked that signup form. Instead of one long glow, the form was now shorter and spread across multiple pages to make it easier to complete. So you see they segmented it. And what is the result? You guessed it, 27% increase in monthly signups, 70% increase in conversions. And even beyond just signups, behavioral analytics can really streamline your all-up user improvements. So if you watch Shark Tank, you might be familiar with this startup. HelloPrinup is an online platform that allows couples to create their own prenups without involving any lawyers. And what they're we're looking for with behavioral analytics were two main questions. What are the general user pain points with our site and how do we make our experience as seamless as possible for a very complicated topic? So HelloPrinup first focused on sessions with rage clicks. They found pain points, which like you can see in this example, where there were confusions around where the input areas in the form are supposed to be. The users were thinking about getting, putting in their input on the bottom, on the top, and there was not too much understanding of how that should work. They had prenup also looked at really long sessions to find out that users were reading text by mousing over it. This helped them isolate experiences where copywriting, legal jargon would have been way too confusing. So they took all of these findings, they made changes to address these issues. And astoundingly, once they shipped those changes, they saw a huge increase of 32% increase in revenue month over month. That's amazing. And again, you probably know what I'm gonna say. It doesn't just end there. Your products, even if it's a landing page, also benefits a lot. So Adapt Worldwide is a UK digital performance marketing agency. Their client Middletons was using Google Ads to drive traffic to their site. The catch though, the additional traffic was not actually converting into sales. So Adapt Worldwide was brought in and tasked to improve the conversion rate and reduce CPA, which is cost per acquisition. So how they actually did this was they looked at user sessions filtered to their ad campaigns and then overlaid the quantitative data from Google Analytics with the qualitative behavioral data. Adapt Worldwide found user frustration. As a result of that, they saw rage clicks, which shown here is a set of excessive clicks and a small static part of the page. And with those rage clicks, they were able to pinpoint a set of areas to improve. And once they did that, they saw incredible conversion increase of 86%. And also a reduction in CPA of about 25 to 41%. So coming onto the topic of these user experience metrics, in general, they're really helpful and I wanna introduce you more deeply into dead clicks, which are static clicks on static parts of the page. So the common scenario here is you click an image thinking it's gonna bring you somewhere or show you something. And when it doesn't, you're confused. This is a great signal that your experience could probably use some tweaking. And we actually see this pretty commonly in our econ sites, specifically on the checkout page. So on this checkout page, the left-hand side of it, you can think of as your classic, this is where you put address, your credit card information, your personal information to buy. And on the right-hand side of this checkout page is where you would typically see your product. A little snippet we included on the top, you can see what that might look like if you had the product. That would be what you see on the right-hand part of the screen. So when we're watching this, we see that user seems like they were trying to potentially go back to the main website. They're clicking on the image for the product. They're clicking on the title of the product, thinking that it might take them back. And you can see they're moving on now to the logo. They're expecting the logo to be clickable. They're expecting it to reroute them somewhere else. And as we all know with shopping sites, the goal is really to streamline the process for a purchase. With every additional effort it takes, the site really risks losing the customer. And even if the customer converts, the odds the customer will come back again is going to be much lower. These are the kinds of user expectations that dead clicks helps capture. And another user experience metric to call out is the rage clicks. So this we previewed earlier in the ADAPT case study. These are clicks similar to dead clicks that occur on some static part of the page and don't actually have any action associated with it. The difference though is that rage clicks typically express a little more user frustration than dead clicks. They're more repeated, they're more aggressive. So what we have here is an example of this on Bing. And the GIF, users are clicking on the margins of the search box expecting it to work just as accurately as if they had clicked the inside of the search box. So the rage clicks here are showing us that users aren't really satisfied with the lack of clickability. We take a step back, think about this more broadly. In a search engine, the search bar is the most important part of the page. It's how the user does what they came to do, which is make a search. If this primary functionality doesn't work as they expect, then we would definitely expect to see some dissatisfaction. Now knowing how important that is for the search site, we can show that the Bing team later investigated. They saw what was happening was that the outer margins of the bar, as we suspected, were not getting clicked on. This is what's in orange. The section in blue is what's actually operating as expected. So the fix for this wasn't actually that difficult. It's pretty easy. We expanded out the margins. And what we were able to see as a result, there were no more rage clicks. The improvements in our top line satisfaction metrics, which are quite hard to move, actually improved. And more importantly, Bing was able to streamline the user experience for a really crucial part of the search engine product. Now that we've seen all these behavioral analytics in action, I want to sum up the key takeaways. So firstly, behavioral analytics tools really work together with web analytics tools. Both of them together give you the full, complete picture. It's not choosing one or the other. They tell you different things. Web analytics tells you the what and behavioral tells you the why. Combining both lets you become more data-driven and really stop guessing with where your priorities should be going. The second is that behavioral analytics really helps pinpoint a variety of issues. And the case studies we saw today, it tackles everything from UX customer frustration points to optimizing for a variety of things, above full visibility, product education, critical paths, campaign landing pages, and much more. It answers even the questions you didn't know to ask. And finally, behavioral analytics is accessible to even non-technical folks. It's all visual, all really easy to understand and interpret, and it truly democratizes analytics. So by now, I'm hoping you're sold on the power of behavioral analytics and you're also interested in starting to use it. The good news is that I can actually give you some clarity here, pun intended. Clarity is Microsoft's own visual behavioral analytics tool. Our product is fully free and we're on a mission to democratize behavioral analytics for everyone. So because it's free, because it's really simple to use, I wanna be able to give you some examples of how you can get started with here. Our feature set consists of the three core behavioral analytics tools that we just talked about. First one is session recordings. These are what we recommend using to find product opportunities, like user pain points, to identify target audiences. And also if you're running AB experiments, recordings are great to validate the product changes and make sure that the treatment is ship quality. The second tool, as we talked a little about earlier today, is the heat map, which aggregates recording data. So this is particularly useful to show impact to stakeholders, to optimize landing pages and add placements, and also to see where people are dropping off, where they're starting to not scroll anymore, where they're exiting the site. And one really cool feature I wanna call out is the heat maps compare feature. This makes it really, really easy to do a side-by-side comparison of two heat maps across different time frames, different experiment variants, different URLs, even different kinds of heat maps. You can see a side-by-side overlay. And closing it off, the dashboard and metrics are also really insightful in finding clusters of user problems and frustration pain points. So this helps you narrow down quickly to promising areas of improvement. So now that we've looked at the core feature set, I want to provide some context on more of the founding inspiration for clarity, a case study on how we used it internally to bug bash, build better products before we ended up shipping it publicly to what we now know as clarity. So a few years ago, Bing received some reports of mysterious purple ads showing up on the page. Bing wasn't intentionally setting these, so it was kind of weird. And internally, Bing went through the mental checklist. Was anything shipped? Was this an experiment from an internal team? Was there a known bug? The answer was no, no, no. And given how ads is, everyone knows, these are how search engines make their revenue. So you can imagine how important this question was to resolve. But because we had clarity data for Bing, we were able to actually replicate the issue just as the user saw it. So I'm gonna show you an image here. You'll notice all the text is automatically masked. This is because we care a lot about privacy and security. We aren't collecting any of the personal data and we're just trying to look at the user patterns. So what we're seeing from this site is that the sessions where this purple ad appeared were also correlated with sessions where there was an unexpected icon next to each search result. Taking a closer look, the team dug in and saw that these icons were actually being rendered by calls made by Kaspersky Lab, which is an anti-virus program. So the issue was only present for users who had Kaspersky installed, but this number was pretty substantial. And even though it was a third party issue, any user who's seen this on Bing will likely associate it with Bing, not come back and potentially go to Google. So once Bing found the issue, they made a fix and we consider this as a successful application of behavioral analytics. The whole process was a really, really great learning experience, especially in that existing tools in the market weren't good enough. They weren't fast enough and they weren't free enough to adequately debug this issue. And the main takeaway for internal was that, this tool won't just be useful for internal use, it'll be useful for everyone out there as well. So how do we make that possible and accessible to everyone? Fast forward and clarity was launched. It was launched with the mission to serve these behavioral analytics to everyone, to democratize the space so that even non-technical folks could access the value. And as such, we are free now and we plan to be free forever for as many sites you want and however much traffic you want. Privacy is also hugely important for us as a product, for customers and for product owners in general. We are GDPR and CCPA compliant and we even by default mask personal data. So we allow site owners also to customize their sites masking as needed. And all of this is done on the client side so nothing ever comes to Microsoft servers. In the realm of performance, clarity was also created by many former performance engineers. It went through rounds and rounds of optimizing the script for performance. So rest assured, our product does not make your page slower. And lastly but not least, our product is built on open source and you can actually access all of our code on GitHub. With the mission of democratizing the behavioral analytics space, we want us to encourage others to learn and to contribute with us. So before I conclude for today, I want to mention, we focus primarily on product managers today but behavioral analytics is truly for everyone. Building great products takes a whole team and this tool isn't just something that only a PM can use. Engineers, designers, content creators, anyone in any role is able to find value from different answers it'll help them solve. And it's also easy to use and easy to pick up by anyone even if you're not technical. The list goes on, applications are endless and in any role or scenario you're looking to start using clarity on, I want to leave you with a handy process to get started. So when beginning your analysis, you'll always want to have some goal in mind. What is the main metric you're looking to optimize? From there, work backwards, figure out what the components and pages that feed into those main metrics are. Then look at the heat maps for those pages and really try to use that to understand the high level user behavior. Are you seeing rage clicks? Are you seeing other frustration signals? If you're running an experiment, how does the impact of each variant compare to each other? And once you have a general sense of some user issues, take a look at a few recordings to watch them in action, see the mouse flow, use the filters to narrow down where people get stuck. So with the hypothesis now in mind, you can build a feature to counteract the user pain points and reuse this flow to confirm things are working as expected for you. All in all, I hope this session has been really helpful for you and understanding behavioral analytics, what they are, how you can use them to build great websites. If you have any follow-up questions, you can reach me at shareanduppaying at Microsoft.com and I look forward to hearing from you. Thanks everyone.