 Greetings. Hello, everyone. I am Seth Hodgson, a VP of Engineering at Udemy, which is an online learning platform. I have the pleasure of meeting with you today to talk about how we are using Amplitude at Udemy. Focus of my talk is going to be specifically on how we are connecting our inputs to our outputs. Connecting the dots from our customer journey and our lived customer experience up to eventual business outcomes that we care about. Too often, there is an unhealthy gap that can grow between the folks that are working on the inputs, working on the customer experience of a product, and those in the company who are focused on critical business outputs higher in the org. And as teams and companies are shifting away from a small number of aggregated, lagging business meaningful output KPIs and shifting toward measurable inputs that are customer-centric, the number of things that we have to measure grows. And with this, it means that our teams need access to self-service insights and flexible analytics and analysis. So I'm going to walk you through how we are using Amplitude to unlock our ability to drive customer-centric product delivery and strategy and how this fits cleanly into a larger ecosystem of data analysis and tooling that we use. So I'm going to start with a real-life example. I have a dear friend who talked me into signing up for Half Marathon in May this month, Zion, at night. I'm not a serious runner, but it's going to be a lot of fun. And I'm not going to break any land speed records, but I want to complete this Half Marathon and I want to enjoy it. So those are my goals for myself. What should I do to achieve this goal? Well, I don't have any direct control over, you know, I don't have a magic wand. I can't just magically be at the end of the Half Marathon happy and healthy. So I need to work backwards and I need to set a prerequisite goal. And so a useful prerequisite goal for me to set is to achieve a target body weight so that I can run this comfortably and effectively. So I can ask myself, is my body weight measurable? Yes. Is it directly controllable? No. And this means my body weight is an output. It's not an input that I directly control. In everything that we do, whether it's our personal lives, the goals we set for ourselves with our friends and family, goals with our team and goals at our companies, we need to focus on managing our inputs, not our outputs. We have to keep an eye on our outputs. So I need to weigh myself daily to reach my output goal. I need to keep an eye on it, but it's much more important for me to work on inputs I can directly control. This is much easier said than done, but it's absolutely essential. It's critical in order to make believable improvements that ladder up to the outputs that I want to achieve. So I can ask myself, what are some inputs that I do directly control that relate to this output, a target body weight that I don't? And some that came to mind for me are diet, exercise, and sleep. So let's do the same exercise. Let's walk through each and ask ourselves for diet. Is the cleric density of my diet measurable? Yes. Can I directly control this? Yes. All kinds of tactics I can use to manage how, when, and what I'm eating. Therefore, my diet is an input. I can directly control it. How about exercise? Is the duration and type of exercise I do measurable? Yes. Can I directly control it? Yes. Sleep is interesting because a lot of things that we might control in our lives are things that we can do more of, or we can change how we do them. Sleep is interesting because we don't actually directly control sleep while we're tuned out and sleeping, but we can still directly control sleep, because we're not actually actually controlling sleep. So I can just give you a little bit of a little bit of a description. We don't actually directly control sleep while we're tuned out and sleeping, but we can still directly control the quality of our sleep, for instance, by not doing things. So I can not read my laptop in bed late at night, right before I go to sleep. And so here as well, my sleep is an input. So if we think about how these things connect up, the inputs that we choose to work on that we directly control that we can focus our work on, these need to be measurable, they need to be tracked and they need to be observable. And in my case, I use a handful of apps on my phone to do so. By doing this, we can start to connect these up to outputs that we want to influence that we do not directly control. And through analysis and experiments, we can validate that an input does in fact causally influence an output. So in summary, in my personal life, I have limited direct control over the output, right? The number that I see on the scale every morning when I check in on the output that ultimately I need to be managing, but I do have direct control over inputs. And as I work on those inputs, I can influence the output. More specifically, and outside of my personal, my personal hero quest to run through the desert night with my buddy, let's walk through a software system example of how this can work. So at Udemy, we have a few different lines of business. One of those lines of business is a direct-to-consumer transactional marketplace business. And for any company that's doing direct-to-consumer work, SEO and SEO performance is absolutely critical. Google, Bing, and other search engines are constantly adjusting the algorithms that they use to rank and order search results that they return to users. And so at Udemy, we oftentimes have goals effectively to achieve target SEO performance. And this could be just in terms of growth of the business. It can also be in response to algo changes that Google and Bing and others are shipping. So we can ask ourselves, is SEO performance measurable? It is, albeit indirectly, we can measure it by observing how our site is performing in search results for head queries. Can we directly control our SEO performance? No, not at all. Therefore, SEO performance is an output. It's something that we want to influence or improve in a positive direction, but we have no direct control over how we rank. So let's revise our goal and let's identify inputs that we do directly control. In the case of SEO performance on search engines, a great proxy to this is Google's lighthouse performance score for a given web page. This is interesting, though, because here as well, do we directly control our lighthouse performance score? Not directly. It's a weighted average metric based on several inputs that Google has defined. And there are three crucial inputs that Google refers to as core web vitals that feed into the lighthouse performance score, which in turn is the best proxy to great Google search results, which is the ultimate output that we want to drive. Can we directly control these inputs, Google core web vitals? Great news is that we can, and these represent a web page load performance, faster is better, web page interactivity, more responsive is better, and visual stability of the page. So as it loads and paints on screen, less visual shifting is better. And Google has validated that these core inputs are key to a positive customer experience with a web page. And so here, just like in my personal example, we can see how Google core web vitals as inputs are measurable, trackable, and observable, and causally feed into an influence, a great lighthouse performance score, which in turn causally influences great Google search results. What's interesting, though, is we wish that reality was perfectly understandable and measurable, like a gray machine. And it is, but only in part. And so we have to be humble about this whole process. And my body way, there has many inputs, beyond diet, exercise, and sleep. And myself and my buddy, Patrick, who's running the race with me, different inputs, even in this subset of three, might be more or less meaningful. In my case, diet might be more important, and in Patrick's, sleep might be more important. And so every business, every site, every company, has to be thoughtful about the inputs that they choose to focus on and drive. And in the case of Google SEO and SCM results, the outputs, driving positive changes in these outputs are the same. Lighthouse performance score and Google core of vitals are a causally connected and meaningful input to focus work on, but it's only part of the picture. So in the case of search results, Google also defines a framework called EAT, which stands for expertise, authority, and trust. And these are characteristics that apply to the content of pages or native app screens. And these are also crucial inputs, entirely different than the technical inputs that we just looked at. In any case, an improvement of X% in any of these inputs is not guaranteed to yield an improvement of Y% in a desired output. But through analysis and experimentation, we can establish a causal relationship and be confident that when we drive positive motion in an input, we will expect a positive motion in our target output. So what about the Udemy product and platform more specifically? Google core of vitals were effectively a cheat code like God mode in a video game. We didn't have to figure out good inputs to directly control and manage the output that we were trying to drive SEO search performance. The case of a product though, such as Udemy or any of the other products that you all are working on, what are the measurable inputs and outputs for your product and platform? This is tricky and it's on us to figure out. The way that we do it at Udemy is we consider our platform at four levels of detail. And you could imagine zooming in progressively from high to lower levels. So at the highest level, 10,000 foot view, we think about our ecosystem. And this is what our executive staff, the board and other folks are focused on primarily. These are our company-wide outputs and goals. And this really is about our ecosystem. Traffic flows in and out of our platform on the learner side, the instructor side, the business side and key touch points. If we zoom in on that, we get into the space of customer journeys. And inside of Udemy, we organize ourselves into portfolios, portfolios handle different customer journeys, different customer segments. And at this level, you can think about establishing portfolio-wide or customer segment-based outputs and goals. This is the next level of business and product strategy. And we approach this by defining customer journeys and customer journey phases. And visually, you can see that the entire ecosystem decomposes into a set of meaningful customer journeys. Zoom in again and we can decompose that down into product flows. And this is at the level of our actual pods, our teams and their tactics and goals. And so this is teams ideating, designing, building and analyzing the product flows and platform capabilities they're putting in place to drive those higher level portfolio or company-wide or ecosystem-wide outputs. And product flows are understood at the ground level with events and we use amplitude to do this. And everyone here is a stakeholder. These events are the input measurements that we have the most direct control over. This is fundamental to our entire system. So let's look at how outputs decompose the inputs across these levels. At the ecosystem level, we can imagine we have a company-wide output or a KPI. At the customer journey level, our customer journey is divided into phases. And so if we think about a learner, the learner oftentimes is motivated for some reason to learn and they become aware that they can do so on our platform. They land in Udemy in the native apps or on the website. And so that land phase has its own set of essential outputs and KPIs. After they land, they move into a discovery phase and that has its own set of key outputs and KPIs. And perhaps they identify learning activities that they want to engage in. They move into a learning phase that has its own outputs and KPIs. But these are still outputs. They're higher level, they're lagging. We don't directly control them. For example, in the case of learning, we have a metric, monthly active learning days. And so this is not something that we can force users to do. But we can influence in a positive direction at the next level down, which is our product flows. And so at the level of a product flow, the teams are looking at what are AB test or product treatments and interventions that they can build that will help take a customer from novice to expert, from the step one of their journey toward their next goal to completion of that goal. And the product flows are instrumented with events and operationalized in amplitude so we can see what is happening meaningfully, semantically. And our events, we try to approach this by having our events expressed as something in our system either a key entity or abstraction and then an action verb. So for instance, the topic page was viewed or a lab was launched, a lab was completed. So with this as a framing, the important thing here is that it lets folks at different levels of the company from executive staff to portfolio leadership to team leadership up and down operate at the right level of abstraction and understand how the inputs that are being built feed into the outputs that they're trying to drive. So more visually, let's look at a little visual representation of how this looks. Again, we have these different customer journey phases, landing, discovering, evaluate and choose, purchase and sign up for a learner. And you could look at all of these different dots as representing possible user interactions or system reactions. These are events. And let's assume that this sequence of highlighted events captures and describes my personal journey on Udemy. So I enter the site, go through a flow for site entry. I go through a search flow to discover what's available. I get into a course preview flow that helps me start to evaluate and choose whether or not I want to engage. And then perhaps I get into a subscription trial sign up flow followed by a checkout flow to start my subscription and get access to that course. Using amplitude, we can connect these dots to understand how our inputs are working and how they ladder up to the outputs that we care about. And not only for me, but for all of our customers. So recall that we can define meaningful outputs, KPIs, lagging metrics for each phase of the customer journey and then for higher level business goals and our product flows and events are the inputs that we directly control to influence our desired outputs. So based on this, a few other tools that we use are KPI goal trees or driver trees and event storming. And this lets us work backwards from the outputs that we ultimately care about at a high level in the business to the inputs that our product teams are working on delivering to our customers in a customer-centric way increasingly. KPI goal trees or driver trees are nothing more than a causal tree where the root is a key lagging business outcome. In this case, monthly recurring revenue for a subscription product. And you can imagine how outputs in earlier phases of the customer journey for learners can be measured and how these are prerequisites to later outputs happening. So for instance, in that left branch of the tree, visitors need to be offered a subscription. And then there must be an offered a trial rate conversion at which point they start the trial. And then there must be a trial to paid conversion rate at which point we have new subscribers and so on. This ladders up the tree. Each of the nodes in this tree are effectively outputs that we don't directly control, but we influence them with inputs that are our product flows. The product flows we map out with event storming and in event storming what you do is you model the cause-effect flow that takes a user from start to finish in some task or job. And this was done with stickies before the time of COVID and remote work. And now we use Miro. But in short, when you're modeling a flow with event storming, you represent user commands with blue post-its, system reactions or product reactions with purple post-its and events along the way that are recording the meaningful user interactions or system reactions as orange post-its. What's nice about this and why events are represented with orange post-its is they visually pop, they stand out. And so when we model product flows, oftentimes people get very hung up on the details, the implementation details or UI UX details on Apple or Android or in a web experience. And we try to zoom out from that a bit and understand semantically, slightly more abstractly. When a user is engaging with a lab, what are the things that they are engaging with and what are the touch points? What are the events that matter? And you can imagine in between those orange events in the flow, all of that is UI detail that can and will change as iOS, Android and the web evolve, as core services evolve and as we AB test our way to better experiences. But the events don't have to change. And so the events become a stable framing and understanding of the customer experience that can hold true across client platforms and across time as the rest of our system and product evolves. And by doing this, what we're aiming to do is close this air gap that too often grows between executive view of the business, which is up at the top of that KPI goal tree and the work that our teams are doing in close collaboration with our end users, which is at the bottom of this tree. So in summary, I want to encourage you all to really focus on managing your inputs. These are your product flows and your events focused on specific customer goals. These are things that you actually have control over that you can directly influence and improve. We use amplitude for this and it is an amazing tool. This ladders into key KPIs and outputs along your user's customer journeys. And you can imagine decomposing your customer journey into a set of phases and for each phase identifying the key outputs and the key KPIs that represent completion of that phase. Again, Amplitude is an amazing tool for this. And these ladder into trailing final outputs, which your company KPIs, things that your executive staff, your board and Wall Street perhaps care about. These are outputs that we don't directly control but we can influence. And so using this approach and this framework and using Amplitude at the front end will let you manage your inputs, notch your outputs and increasingly improve your understanding of how this is all connected. And I like to finish with a quote. I have worked in a number of companies and I believe observability and customer centricity is absolutely crucial to delivering trailing business outcomes. But too often we look toward the trailing business outcomes rather than toward the customer. It's one thing to talk about being customer centric or being data informed. It's an entirely different thing to live it. And so to quote Richard Buckminster Fuller, if you wanna teach people a new way of thinking don't bother trying to teach them. Instead give them a tool such as Amplitude the use of which will lead to new ways of thinking.