 Hello. Hi everyone. Pleasure to be with everybody here today. Hi James. We're excited to dive into this topic, how product and marketing teams collaborate on experimentation to drive growth. It's a subject that we are deeply interested in and we think you are too. My name is Colin. I'm leading Chameleon here in North America. I'm joined today by one of my friends and colleagues, James. Hello. Hey Colin. That's super exciting to be involved with this project with yourself and Chameleon. Yeah, just I guess quick introduction to myself. So I've spent a large part of my career being really fascinated in what I term the digital intelligence market and specifically around the experimentation practices that large enterprises have kind of adopted and it's been interesting to to watch how experimentation has matured over the last decade or so from a kind of a niche, nice to have kind of think into the strategic imperative that drive growth today. So and a large part of that was during my time leading the research at Forest as a principal analyst there. Nowadays, I'm an independent growth advisor, working with some of the leading brands and technology firms out there. And also I lead an awesome marketing team at a software company called CreativeX. So that's me Colin. Over to you. Yeah, thank you. Well, for everyone on the call, I just thought it'd be helpful if we took a big step backwards and explains a little bit about, you know, the research that we are aiming to present to you today. I'm not sure if this paradigm looks familiar, but you know, essentially what we see here at Chameleon is we talked to a lot of teams where they say like there's this wall in my company where I work really hard to like optimize acquisition funnels to generate interest and brand awareness and to create leads and demand. And once I build the prospects and customers are into that funnel, and they convert the hand over into product teams and cross this wall or this line in my company and often I never see them again. And that is a that is a story that we hear quite a lot here at Chameleon. And you know, it's a story I think that's a little bit of an old fashioned one, you know, this is something that that we see a long time ago, speaking technology so just for the last eight decade, but really the world has changed a fair amount. You don't have to live in this world where there's this line and this wall. The world has changed to a point now what what we're seeing is that it's less about just marketing lead growth and just trying to stimulate as much demand as you can to taking a more balanced approach to understanding that you can grow with marketing lead growth but you can also grow with product And this is an important point that we want to try to highlight for everyone on this call today is that you do not need to choose between marketing lead growth and product growth. In fact, the world has changed. What we see from these leading companies is that they are combining marketing lead growth and product lead growth. And one of the things that has a strong overlap between these leaders that work together on both types of growth is that they're relying on experimentation. This experimentation for them is is not necessarily just about, you know, conversion rate optimization. Like I said, improving those acquisition funnels, but really about trying to be as data driven as possible about where they should invest what sort of business decisions they should take. And if you look at these type of companies that are doing this today, the names are going to be pretty familiar to you. Right. These companies that we know about and talk about, you know, they're organized around how to support growth overall and often product led growth or people who have strong capabilities to affect product are leading those sort of teams, but they very much are collaborating with marketing. So this is the story that we've seen at Chameleon for, you know, the last few years. And we just thought it would be helpful to sort of begin to understand it in more nuanced detail. And that's where we invited James with all of his experience and experimentation research. We thought it would be really helpful to dig deeper into this trend. And I'm going to hand it over to you, James, sort of like present. What did we do here and what were some of the key takeaways? Yeah, I mean, what do we do is we ran a survey and I think produced with you and your team Colin some fantastic results that it's a kind of a one of a kind survey that certainly I've seen, you know, research around this before but not specifically on this topic kind of looking at growth and growth strategies and experimentation certainly not in this way. The title of the survey is, you know, the 2023 experimentation of growth survey, obviously sponsored by Chameleon, led by myself with involvement of many people at Chameleon and others. What we did is surveyed around 160 managers and above in the UK and within the US, you know, from like leading companies. And from the graphic, you can see that the roles that we surveyed were roughly split between product, marketing, engineering and experimentation. And from six verticals, for instance, BFSI, healthcare, media gaming, retail e-commerce and software as a service technology companies. And what we actually wanted to do was actually enter to your point Colin was to see what industry leaders are doing differently from the rest, especially when it came to the experimentation and growth strategies around, you know, market and product led growth strategies. So the first thing we really needed to do with this study is to identify these leaders. And we did this by viewing them through two lenses or via, you know, two attributes. One is the perception of themselves as industry leaders and their expectation for growth over the last 12 months. And what we actually found was that around 20% one in five of respondents came from industry leading organizations that we identified as organizations that identified as the only one, the one and only leader in the industry, as well as they expecting to grow faster than any other organizations in their sector within the next 12 months. So two very strong signals that indicate leadership within the organization. So what are we fine? Well, I'm going to go into that now. But before we go too much into detail, I want to maybe iterate a little bit what you kind of said, Colin, which is, you know, an important outcome of the race to digital transformation over the last couple of decades is the ability of companies to measure and manipulate digital engagement to understand and deliver great experiences. That's what essentially one of the outcomes of digital transformation, right? And this has driven huge opportunities for marketers, particularly around their growth strategies as they improve engagement within their brand and their CRO or conversion marketing efforts, right? However, an even bigger benefit factor over the last, let's say decade has been teams that we now call product teams that deliver digitized engagements within the products and services that many companies now offer their customers. And hence we've gone through this huge, you know, this kind of era where we've seen the rapid rise of product led growth strategies, which are now maturing. And these product led growth strategies now sit alongside these kind of more traditional or longer standing market led growth strategies in many of these companies, countries, sorry, companies. And what our survey shows is that leading companies tend to line, not only have both these strategies in place but are strongly aligned on both of these strategies. As you can see from this first kind of result that we show you right where we say that 90% of industry leading organizations have product led and market led strategies that are strongly aligned. And this contrast quite significantly where companies that are leaders or not leaders, should I say, and laggards have a 50% or less chance of lining on their strategies. So of course, you know, a fundamental characteristic of any organization or organizational strategic approach is how these teams work together. And so it's no surprise really that organizations where product and marketing teams are aligned have an 81% chance of growing significantly. So that's kind of parallel kind of result to the previous slide where it says the alignment of the strategies. This is actually the alignment of the teams. And you know organizations that are kind of of average success are much less likely to be aligned on their teams, the teams as well. So now we've, now we've looked at the kind of growth strategies of leaders, let's now look, have a look at how leaders are investing in experimentation to support this high growth, right. So the big hypothesis we had coming into this research was that great aligned experimentation strategies support these kind of holistic aligned growth strategies. Our survey clearly shows that experimentation supports both the product and marketing teams with with the growth approaches. So for instance, product teams benefit from investing in feature experimentation. And if they do, they're almost, you know, they are three times or almost three times more likely to come from high growth organizations, as well as the marketing teams benefit from investing, for instance, in web experimentation. If they do invest in web experimentation, they are over actually around three, three and a half more times likely to come from high growth organizations than those that don't, and will be growing significantly in the next 12 months. So investing, so investment in experimentation is critical to the success of both product led growth and market led growth strategies, particularly in the year ahead. So a few slides back, we reported that there's an 81% chance of a significant organizational growth for organizations where product and marketing growth strategies are aligned. The same alignment story is true actually for experimentation, where we see the alignment of experimentation metrics is critical to business success, at least so say 87% of the respondents. So let's talk a little bit more about the approach these leading organizations take to support multi team alignment on experimentation. We touched on the people and process part of experimentation strategies, but what about the technology part of it, right? So to bring people, in other words, teams together to drive enterprise wide processes needed for holistic growth approaches requires the right technology approach, such as a single platform approach. Our survey shows that around 90% of leading organizations are using one platform to execute and collaborate on experimentation, as well as the maturity of using mature proven technology. And by that we mean off the shelf technology provided by great vendors such as chameleon that have the scale capability and stability needed to support sophisticated experimentation programs. And what our result showed was that 63% of leading organizations realize that off the shelf experimentation products are the way to go over and above the kind of build it themselves. And actually what we find is that there is a great combination of buying these sophisticated technologies that we see today and building on top of them as well. So there's this kind of dynamic equilibrium between the two. So I want to take a little bit closer look a closer look at the technology approach. And what we show is so let's take a look at what the leaders are saying and what we see is that critical to growth is the existence of a fully integrated platform with the organization. So what does that mean? It means that so what is a fully integrated platform is one that brings together web and feature experimentation within a single platform. It's not only a single platform, but it also has all these integrated features as well so slightly different from a nuanced perspective inside from the previous slides. And so we need this fully this full integration of web and feature experimentation and other features to support the needs of all the teams. And what we find is that firms that have a fully integrated platform are 70% more likely to grow in the next 12 months. And this is what our surveys are showing us. No, I just want to I want to add touch on that point there and give a little bit extra context if we can. James, because what I see is when I talk to these teams is that it's not enough just like it's one thing to say like okay well we've got a web experimentation solution and we've got a feature experimentation solution and okay fine it's great that they're on an appearance maybe that they're on a single platform. But often what is happening is that the web experimentation solution has its own KPIs, its own events, its own segments, its own team management, and then the feature experimentation has an entirely different data model. And what happens is that it may look like it's an integrated platform but in reality there's still this bickering intention about what data model are you using what data model am I using and then that's that benefit of having like a single source of truth is missed, and it just bogs down the experimentation efforts. And that's not what the, I think I just wanted to touch on that point that's not what we're seeing the leaders do. They know they have a single library for all their events and KPIs and segments. They have a single dashboard where developers and engineers can see these flags and these experiments and marketers can see these over here. But importantly they're all trusting that the information that they're using is accurate and not worried about apples to oranges problems which I think a lot of like immature or laggard stuff. Yeah, I mean you got your, you said something that's key there, it's like they share the same data model, the same measurement engine, right? And if you, I mean, just notionally if you're going to have a marketing and product or two teams driving the growth strategy and that is a merged or integrated growth strategy driven by experimentation, you need a single system of insights and experimentation to really do that. I mean, it's not possible to do it, right? To have two measurements or two experimentation systems will essentially drive two different programs and never the train shall meet all they can, but it's really difficult. So, yeah, totally, yeah, totally get that. So, shall I take us to our final insight then, you know, that I just wanted to share. You know, it takes us back to the people part of the people process and technology discussion, right? And it really brings us to an age old debate of the involvement of technology people such as developers, right? So, you know, I'm kind of old and gnarly enough to kind of remember where developer was King or Queen and she, you know, ruled the roost when I came to driving the adoption and enablement of technology. And we've gone through a couple of decades where, you know, the business person is now kind of in the driving seat when it comes to leveraging technologies that drive many practices, not least experimentation, right? And we've almost sidelined the developer. We don't want the developer there because she or he slows things down when we want to implement, we want to do stuff, oh, we need to get to the developer. And so, you know, hence the rise of business friendly, low code and even no code technologies. And, you know, chameleon has again with the rest of the experimentation community chase that kind of that mantra of like we want just the stuff in the hands of the people that the operators, the business people, etc. However, you know, to really maximize the ability to innovate and scale experimentation within enterprises requires at least some involvement in of the developer, right? Why, for instance, if we're going to integrate a your experimentation platform with your, your, for instance, your engagement systems or your experience delivery systems as your personalization system, or even your, your CDP or other data delivery systems, you are going to And you want to do that in ways that perhaps you're competitive on doing, you need that secret source that the developer can do to kind of do that, as well as you need that, you know, the dev person to customize and tailor capabilities, you know, processes and outputs that are unique to your own business so that your experimentation is supporting your business processes rather than altering your business processes to the needs of the, the, you know, the system itself. And so that's why experimentation is important. And, you know, hence no surprising that leading organizations rely heavily on developers when supporting their holistic strategies with experimentation. You know, this insight that we see here from the survey shows that 59% of leaders see developers as critical to experimentation. And that's about, you know, one and a half times more likely than the non leaders. So I thought that was a really interesting kind of final insight Colin from from this this research and so I'm going to take a breath now and maybe did you want to summarize some key takeaways. Yeah, you know, but I do want to stay on this point because I think it's so important for everybody. You know, the bottom line is that if you're going to be optimizing the product, you're going to be working in the back end. You're going to be working with full stack developers and engineers so it's, you know, if you want to be a part of this trend of taking marketing led growth and product led growth and collaborating to become a leader in your category. You have to have engineering support. But what we see is that it's not the zero sum game where it's like, okay, if engineers have to be involved, then everything when it comes to experimentation needs to be engineering heavy. Do you need engineers to have code back end experiences? Absolutely. Like no one's going to, you know, no one's going to tweak the algorithms or, you know, build the on the back end without an engineer. But does that engineer actually need to be involved in like allocating how much traffic your experiment gets or choosing the KPIs that are going to be used to measure the experiment. Because the developer need to be part of, you know, picking out the targeting in the segments. No, this can all be handed off to somebody that may be dedicated to experimentation. It could be the product manager that knows their segments much better. So I just want to highlight there's a nuance to that point. Developers absolutely need to be involved to be able to reap the benefits of experimentation, but they don't necessarily have to be involved in every single step of the experimentation launch. And I think that's just a key point everybody should hear. So I know it's, you know, if you try, you know, there's different ways of like compressing that into some report, but the way that thought that comes to mind is that, you know, developers are really important to create innovation and integration and make it relevant to your organization to kind of create that differentiation. And then they need to kind of take a step back as they've created this enabled feature these integrations and let the operations and business people scale it right and to scale those experimentation. So this is exactly an dynamic. Yeah, and I think that's the real sauce of success is that when you look at those leaders again, you know, I can kind of pivot over to some of the key takeaways because I always just like to think like, how, what do I do here? I want to roadmap, give me the table. But, you know, before I, you know, before I dive into this list, I just want to highlight the one big thing that's, you know, seems apparent, but it's important to like continue raising is that these teams that are aligned, they're relying on a common set of metrics that and they use those metrics to build these flywheels and you can see that is what these leaders are doing. So to kind of get into that more detail, they do not choose between marketing led growth and product led growth. They go on both. I think that's very important. If you're going to pursue both methods, then you have to again understand experimentation is an engine of growth for marketing led growth and product led growth. You're going to have to give the marketing side, classic web experimentation. James, it speaks to that like no code, no tech, non technical teams being able to like easily optimize UX and messaging to help, you know, optimize their acquisition models. You're going to have to give the product and engineering teams the ability to gate features with a flag control release control rollouts rollout go back easily. But what's so important is that flywheel point here, which is that these teams that understand, okay, well, look, wow, when this particular cohort of users really is sticky. We see great engagement, a lot of retention from them. How can we get more of these type of users that information is fed back into the marketing led growth teams or the marketing teams, and then they use that information to build more refined campaigns. So essentially, they're optimizing their funnels so that they can deliver more of the types of customers that product team knows, love, and will adapt to their product. And again, you can see that benefit that flywheel just like feeds itself again and again and again. Yeah, you know, I thought you said it really well. So when you look at laggards, what they what I commonly see is that they're, you know, I can't tell you how many times, you know, call the day with somebody complaining about in house and built in house technology that is super old nobody said and wish they had never done it to begin with. But again, it's there's more nuance to this where you shouldn't just go off and say like I'm going to build a robust unified experimentation platform from scratch. That's a very hard thing to do. But you don't have to say to yourself like I'm never going to try to like integrate something that's off the shelf with my tech stack. I think you can have both essentially today, which I think is really important. Yeah, that's a fundamental shift. I've seen literally in the last like three to five years right that kind of this sophisticated hybrid approaches, only because it's been supported by you know chameleon and similar types of technologies that have enabled to do that right. So yeah, yeah, yeah, you mean again, you can have it both today, which is super exciting pursue appeal gene emoji pursue in house and sorry off the shelf technology and then build customized stuff on top of that. And then we already talked a lot about engineers so I think that points just welcome welcome your engineers don't be that you know, be kind to your engineers because you can work together definitely. Yeah, yeah, you know, so you can invite them into the process by giving them the tools and the methodologies that they want, but you can simultaneously guard how much of their time needs to be in experimentation by being thoughtful about the solutions you're working with because again, not all solutions need to be so deadhead. So, you know, everyone on the call. Obviously, we have a big stake in this. It was important for us to see this research really kind of confirmed the vision that we had for chameleon, which is that hold on to be successful today you have to have a single platform that's going to work for all time teams and all types of teams. You have to make sure that you're giving all the solutions that those teams are looking for, you know, do they does marketing need a whizzy wig. Absolutely. Does product. No, they don't as engineering. No, they need feature flag future experimentation. And then, you know, each one of those teams is going to require that we work very well with their preferred analytics tool, or their preferred CDP or whatever kind of third party martech that's important to that team to make sure that you're getting ROI from experimentation from we have to integrate with. So, yeah, I just wanted to thank you for the this research and because it again confirmed the sort of path of the vision that we have for experimentation. Yeah, it's been fun. Huge fun. I love it. For everybody on the call. We just touched on some of the key takeaways from this report. It's really interesting if anyone is really struggling to communicate to senior management about how experimentation needs to flourish outside of whether it's marketing or in product. And how it needs to come together so that you can become one of the leaders that we highlighted here. I really encourage you to have a close look at the research. You'll see the full report there. Thank you very much for your time. Thank you everyone. Bye bye.