 Hi everyone. Stand here. Can you hear me? I just want to do the test that you can hear me, okay? No, I'm Horancik. It's hard to say. He knows what I'm talking about. How do you say yours? So yeah, I work at Lead Space and I'm gonna talk about simplifying complete data solutions. Just the agenda of what we're gonna talk about. I'm gonna introduce myself, introduce Lead Space. I think it's important to understand what we do to explain why it's complex and how we try to simplify it. The data science team, what are the modeling use cases that we have? What are the analytics that we currently show our customers? And I'll talk about a feature we're currently developing which is called model drivers and then just kind of general PM advice on this whole idea of simplifying the products. So a bit about me. I'm from Israel. I was I served at the intelligence unit in the Israel Defense Forces for three years. Then I traveled a bit in Australia, New Zealand, Japan a bit more than that. And then I went and studied bachelors of management and psychology in the Hebrew University. And then I joined Lead Space in 2013. So I've been at the startup for four and a half years already. I started as a data analyst, which was that role is the one that actually creates the profiles for our customers. You'll get that when I talk about what Lead Space does. Then I became a technical customer success manager, a lot more interaction with the customers realizing what they actually need and getting feedback on the product. Then solution architect, which means that I was matched with a CSM with a customer success on each account, providing the more kind of sophisticated side of running complex projects for the customers. And now I'm director of analytics solutions. So what I do is I serve kind of a I call it the bridge of context. I made that up, but I like it. So between product, our customers and data science. Okay, so we have the data science team I'm going to talk about them. And I like to snowboard to play soccer and drums. If anyone talk about that as well later. A bit about Lead Space. Okay, so we're a B2B audience management platform. What the hell is that? That is a very bombastic. I don't know what, but I'll explain. It comes with three different solutions. The solution in blue is the one I'm in charge of. I'm actually in charge of these two, but the solution in blue is the one we're going to talk about today. It's the most complex one, and it's the one that we're trying. We have these challenges of simplifying. So the idea is that customers usually start by purchasing audience data management. What that does, we actually integrate with Salesforce, Marketo. If you know those kind of systems for marketing and sales, and we take care of data hygiene, we make sure that the data is full, it's segmented, and things like that, which helps route the leads to sales and marketing to be more effective. Then what happens with our customers? They're saying, okay, the Lead Space data is pretty good. It's really good. It's actually better than the competitors. Why not model based on that? Right? Because if you have good data, you can model better. You have better data to model on. So we kind of came up with this audience modeling a couple years ago, where we actually create predictive scores for customers on the lead and on the account level. There are other things we do as well. We do persona fit. So we actually tell you, is that person the one you really need to talk to and which content you should send to that person? And we provide the insights, which is the main part of what we're going to talk about. And then they say, okay, so you clean our data. You have good data. You created models based on that good data. Now give me net new people that I should go after based on those models. So the model tells me, here's an A lead. Give me all A leads. Here's an A account. Give me all A accounts. So it's kind of a flow that goes for most customers. It's like a land and expand. We land here and we expand all the way to the bottom. So before I move on, I kind of just want to understand here who works with marketing people or sales people. Okay. Anyone here is a product manager? Okay. Some aspiring product managers, assuming. Cool. Weird question. How many people here think your solution in product is complex? Okay. Most. And do you think you're doing a good job simplifying it? Okay. Good. Good. I like that comment. The atmosphere. So yeah. So if I say anything that is very specific to marketing or sales that you don't know, just ask. Because I'm used to the terminology. So sometimes I think things are a bit... I kind of know the terms, but most people just they never encountered those terms. Okay. So data science at lead space. There are four members. We have a head of data science in Israel. Two that work under him in Israel and one in San Francisco. Just move here is actually sitting at the back. And the responsibilities are for the data science team is this is the overall responsibility. Own the audience modeling solution end to end. Which doesn't mean much by itself. But the idea is that anything related to the predictive models, the insights. It all comes down to this team to manage both from the customer facing side, which is me. I'm kind of work with the team on the more contextual side with the customers. And they actually develop and produce the models. Okay. So develop and produce and refresh the models. Most of our models are propensity for an action to happen. I'll explain in a second what that means. But we also do some other things like advanced classifiers, which we have a customer that asked us to help them understand if an inbound lead that comes in, if they come from a marketing agency or not, just based on their social profile. Which is not a simple task. You have to come up with a yes, no answer quality versus quantity. All those different things that come to play. We have another one of like, is this lead valid? So then you have to look at the contact details. Are they valid? Is it Mickey Mouse at Google.com? You know that's fake. All these different things. So we work on those as well. We create the supporting analytics, very important part of it. And we continuously improve the lead space products. So we have our team kind of working on all different core lead space products to just improve them with data science, which everything can improve from that. The modeling use case does that we have. So we have inbound, outbound and analytics. I'll explain what that means. Inbound is from a marketing perspective is scoring anything that comes inbound to my website to my systems. So if you fill out a form that shows up in the company's Marketo, any marketing automation system, and then they, that is called an inbound lead. And then what do you do with it? Is it a good lead? Do I pass it directly to sales? Is it a bad lead? Do I try to ignore it as much as I can? That depends outbound. That means that I proactively reach out to companies. So the people here that are into marketing, you know what ABM is, account-based marketing. This is the whole new era of ABM that you really understand who are the accounts you want to talk to and then proactively reach out to them. You don't wait for them to come to your website and fill out a form. And analytics as a use case, that's a question mark to be honest. We are not sure yet if analytics as analytics is a valid, has valid value proposition from our business. Like can we actually sell analytics by themselves, which is a question that we can talk about. And the idea is that we create multiple model versions. We have a person score, which is that persona fit of telling you how you can personalize your marketing sales efforts or find the right people if you're going outbound. What we mostly focus on are these two in the middle. The first one is the lead score. You can see it's a person and company. So what we do here in the inbound scenario is that we kind of calculate the propensity of a lead to convert to a certain action. It depends on what the model is fed with, but it can be convert from a marketing qualified lead to a deal or from an inquiry, just someone that made it to the website, to a marketing qualified lead. All of these are supported by our infrastructure. And we just need to work with the customer understanding what they're actually trying to optimize. Okay, company score. It's similar to the lead score, but its focus is only on the company. That comes with the whole ABM era of let's just give me a clean score. Is that company worth going after? And I'll find how to reach that company. Okay, that's both for inbound to tell you who you should send to sales because they are the right account or to find things that you have in your database saying, okay, lead space is saying this is an A account. You should probably talk to them. Make sure you have your best sales rep on and calling them all the time. And we have a company local like list. That's a pretty different algorithm. No need to get into that at the moment, but we can talk about that later. So the analytics. So all of this hard work that we do, we create these models in different ways and we have all this machine learning and all the buzzwords you always hear about big data and stuff like that. It all results in a score. Usually it's zero to 100, right? So we do all of this work. Don't try to understand this. This is from Google. It's not us. We do all this complicated work and what it comes down to and you don't have to see it from there, just one field in your CRM that says company score with a number. And that is frustrating because you're saying, well, all this hard work goes into that number. How do I explain what I did? How do I show them? How do I make them trust it? That's a problem. So what are the current analytics that we show? And I'm going to show you a couple of slides of what we currently show. And at the end, I'm going to ask you what's missing, right? There's something that is missing in what we present at the moment that we are working on. That is the whole discussion. So I'll go over this kind of quickly. What we show is a model performance. We show on the x-axis the predictive score from 5 to 100. It's a bit cut off. And then we show you how, what is the lift relative to the conversion of each score? So essentially that means that if a lead gets a score of 95, they're expected to convert 2.2 times better than the average, right? So they can know that first of all, the line is pretty, the slope is pretty good. So you can see that every score basically is better than the one before it. And you know what to expect when someone gets a different score. But it's aggregate, right? So you're saying all the ones that we're going to get a B, it's cut off. Everyone that's going to get a B is going to convert 1.4 times better than the average, okay? Then we show them, you know what, we'll open the box. We don't want it to be black box. We want to show you the different signals that go into the model that make the score. So you're seeing different signal types like company size, revenue, industry, there's technologies, there's a lot of different signals. The actual signal, so if it's company industry, it's going to be financial services. What is the incidence in your data? Just so you know kind of how many we're talking about out of the whole population. And then the lift, anything in green means it converts better than average. Anything in red means it converts less than average, okay? That is still not enough. We're saying, okay, this is just one dimension. Like if you look at this here, financial services, 10% of the data, very long green bar. And manufacturing, 6% red bar. If you just look at this, you're going to say, oh, financial services I'm selling well to, let's just sell more to financial services and let's avoid selling to manufacturing. But that's not true because just by combining two dimensions, you already get a much, much clearer picture. So what you're seeing here is the industries on the left, a bit cut off, sorry about that. And company size here, number of employees. And then you remember I said financial services was the greenest one, was the best one. You see that it is green for the high company sizes. But it's actually red for the low company sizes. So essentially what that tells me is that if I only had this slide, I would say financial services is great. But essentially it's only great 250 or above. And manufacturing, it's a similar story. So manufacturing is, you can't find it right now. I think it's this somewhere. So all of them were red and then there was one green at the end. Oh, there it is, here, manufacturing. So all of them are red, but there is a green dot at the 10K to 50K. So there is some sweet spot in manufacturing and the model will pick that up. The model knows that manufacturing by itself is a negative, but manufacturing with a large company is a positive. And we show this to customers just to make them understand how complex this is. This is just two dimensions and we already had a full story to talk about. If you start introducing revenue and technologies and social sources, all of these different things will come into play here. And there's so many ifs that a person just can't do it. That's why you use an algorithm to tell you, here's that number. But again, that number is kind of, it's hard for us to just provide that number. So we show you what goes under that. So you can see that different companies in different scenarios reach different conversion. And this is just a scatter plot we show, just to kind of show you all the different signals that go and kind of choose specific signals that are either green or red just to give you a sense of what converts and what doesn't. We show some other stuff, but this is kind of the gist of it. And my question is, what do you think is missing? And what are they selling? They wanted you to pull the selling product. You're penetrating some industry, but what would you do selling? Yeah, so for that, the thing is with each data we model for each customer anyway, so they know that the data that we model for them is their own company data. So all of this analytics will be for that company's data based on the product that they sell. So the sales team knows what they're selling and they know that it's tailored to what they're selling. Feedback from the customer on the success of how your specific application has helped them meet their metrics? Yeah, one of the things. So that is a challenge. We'll talk about it later. But the idea is that, yes, we need the closed loop, right? The whole idea, we create one model. We have this whole model performance. The idea is that we want to see what converts and we want to feed the model for me. We do do that. We rely on the customers to provide us with that data and we're actually working now on a solution that we'll be able to take that data as it closes and feed the model automatically. Anything else that's missing? Yeah. Can you predict what will be the cost of that application? In what way? Yeah. Depending on the company profile, if I provide you my infrastructure or my costs, will you be able to tell me in that, in using all of your model, what the cost of acquisition for each customer will be? Yeah. So the question was, can our models tell you what the cost of acquisition will be? Yes. We can essentially digest any data that you give to us and use it in the model. So we have a lot of customers that use, not necessarily we don't see too much of the cost of acquisition on the top of the funnel. We see more at the bottom of the funnel where the actual revenue ended up coming from these customers and we have models that actually model on not just is that company going to buy, but is it going to buy for more than X? Or is it going to have a lifetime value of more than $50,000? So essentially we can take any field and digest it into the model and make it affect the scores and the threshold that we choose for ABCND. So one of the things that's missing is record level insights. So all of the things I showed, all of these different slides, they all show aggregate information. They don't tell me what is the story of that specifically that got that 90 or 50. And we've seen that this is a problem and then that is why we are generating this feature called model drivers. By the way, terrible name, it's an internal name, and we want to, I'm actually looking for help here naming it in a better way. Just so that you know. So from a model driver's perspective what we mean when we say model drivers is that we want to show the most significant signals. I'll explain what categories, types, and features mean. That influence a specific predictive score on a record by record basis. So when I talk about categories, types, and features a category is a thermographic data. It is demographic data. It is technologies. It is web presence. It's all these different things that you can use as attributes, as categories. Under thermographic you will have industry size revenue. Those are the types. So the type is company industry, company size. And the feature is under company industry it will be software and internet. So we go from this is the most granular to the most generic. In the way of things we want to show the customer this is what affects the score. So this is what we're doing model drivers and this is the example I'm actually going to walk you through for the rest of this session of how we're developing it. We are developing it in this next month. So everything I'm going to show you are things that we are thinking about right now. So when we talk about what we're trying to solve sales people don't understand the scores and therefore don't adopt them. That is a problem. And we understand why it's a problem. We show them all the analytics but eventually it's one score there. Sales people don't have patience whatsoever and if they see a couple of scores that they don't understand they're going to say what the hell is this? I'm ignoring it. That is a problem on the customer side. It's obviously a problem on our side as well. Why solve it? So sales metrics are just not improving. Someone paid for lead space to come in create a scoring model score all the leads and they want to see improvement in results and it's not happening because they're not using the scores because they don't trust it. So all of this is happening and marketing who usually pays for this aren't showing ROI because everything that I just said. So what are we doing? So one of the things we're doing is we're developing this model drivers feature where we explain more to the sales team and hopefully we expect the adoption to increase and all of these to be fulfilled. Okay? So milkshakes. I have actually a good story about milkshakes it's not my story it was did anyone here attend product con? You have? Of course. It's your conference. But that makes sense. But I think it was the product manager product manager for draw box that shared the story at product con of this guy that was hired by this milkshake store to help make increased customer satisfaction and increase sales. And then he had a very creative approach of instead of just going say oh just add strawberry banana or add some sort of flavor he went it from a very creative direction he kind of tried to understand who are the customers and he created two cohorts of customers two kind of groups that he identified that one of them were people that are driving to work and one of them were parents and the people that are driving to work he asked what do you like about this about this milkshake what is the purpose the use case of that milkshake and what they said is I drink it while I'm on my way to work and for the parents they're like why do you buy this milkshake to please my kids and then he thought about he went to the drivers and he comes to them and he's like if I create a smaller straw so I don't change anything about the milkshake I change the straw it will take longer for people to finish it they will be happy and that worked and for the kid for the parents he's like okay the parents the kid is crying I want milkshake I want milkshake but you don't want him to get a giant milkshake so you're going to give him a small milkshake so what he actually did in the product he changed the straw to be smaller and the size is to be smaller which makes which is very different than the way you would think when you're hiring someone to increase customer satisfaction this has probably nothing to do with what I'm talking about I just like this story but hopefully I'm going to be able to connect it so the customer problem from going back to our world not milkshakes for salespeople this is a happy salesperson so what do most people salespeople care about? money selling yes what else these are all definitions of selling from the dictionary that's all they care about most I'm not going to talk about all sales rep but essentially they want to sell more that's what they care about so I'm really I'm simplifying this on purpose in the way that what do they really need information that will help them sell they don't need me to brag about my product they don't need something too complicated or too sophisticated they need information that will help them sell even further than that they need information that will help them feel confident that they can sell marketing so what do most marketing people care about? leads well creating a healthy pipeline for sale hitting marketing KPIs and then what do they care about implicitly that is kind of my view of the world but I think it is true not getting yelled at by sales they really care about that because they get yelled at by sales and most of the times they don't rule with our product at the current situation they can't defend themselves a sales person comes why is this a 90? I don't know I know that this combination of industry and size creates a high lift and the sales rep will be like what the hell is this and getting promoted to be honest okay this is kind of why I chose this so what do they really need need and not again simplifying it completely information that will make the sales team happy will make them yell at them less and information that will make them look good okay so when you think about it this way in a more like instead of thinking about from a product perspective but more from the pain point and the use case that they have then it makes you think more creatively and differently about the problem you're actually facing so I want to talk about what we talked about the why and the how I want to talk about the how a bit deeper so I kind of separated to the PM owned how and to the dev and data science owned how okay so I'm talking about what PM owns on the how side it is how do I solve the customer problem that is the question I'm asking and it's not an easy thing right when we come back to multiple drivers because the way machine learning scores generated is not straight forward and the way it's generated is very different than what they used to I separated it out to two bullet points for a reason it is kind of the same thing but you need to think differently when you think about these two things because one thing it's not straight forward how do I make it easy and for this second bullet point it's not what they're used to is how do I create this like change management for them because sales and marketing people what they're used to they're used to five points for downloaded white paper and seven points for failed out of form this is not how machine learning works it's not like oh industry has five points and size is nine points it doesn't work that way so you have to understand that that is what they think about and they hear about a score and then things I'm currently wondering and kind of figuring out is the signal considerations so which categories, features and signals I'm including because again I'm going back to what they care about they want information that will help them sell so I don't necessarily want to show something very advanced just to say oh lead space is cool they can give me this very unique thing I just want things that will actually help them I have this consideration of simplicity versus truth so a funny story about this a month ago we started working on this and one of the leads I went to my data science team I'm like what is the number one feature that influences the score I want the truth and the truth was machine learning company country cluster there's no sales rep that would say ah okay okay now I know why this lead will buy it's machine learning company country cluster so that doesn't help me it's the truth but it doesn't help me I can't use it I'm thinking about how I make it simple but keep it pretty similar to the truth same for variety versus truth most of the models are mainly affected by the big things like industry size department level but I want to have more variety I want to make it interesting I want to show them a specific technology that the lead is using when I can even though it might not be the one the one thing that actually influences the score another thing is positive versus negative signals okay so what do they care about from a positive perspective and what do they care about from a negative perspective I don't think they would care about that they use a certain technology that just has nothing to do with conversion they would want the negative to be things that actually can prevent them from closing the deal and edge cases right when we can't find a company so what do we do all these different things other things that the PM owned how is UI considerations so what is the wording I use it's very important that I use kind of words that sales will react well to and they will understand I need to think where that view should live if it's in Salesforce is it on the accounts page it's in the contacts page, opportunities page where do I put it what is the layout of what I'm going to put and how many signals to show and again edge cases of there's no information to show so I came up with this wireframe very very initial wireframe of kind of thinking what I want to show in this case I'm showing kind of this is Google for example a predictive score and I'm providing a sentence pretty clear sentence Google is 3.1x more likely to convert from account to close to one than your average right this is what the sales person wants to know and then why and I break it down to two different pieces of the page facet breakdown which is more kind of a holistic view of what influence the scores that was it more from a graphic or technology these kind of things and then the prominent predictive signals of the things that we actually want to show which all the considerations from the previous slide are kind of relate to this thing okay now I want to talk about what PM doesn't own but helps helps with so this is the dev data science owned how at least it leads space so how do I create the solution that the PM designed and it's very different right all these different considerations I won't go into all of them but it's like how you generate the signals how you extract the feature which variables you choose but how do you store the data that you have all these different things are very complex and that is what dev and data science are kind of in charge of and I kind of added a general PM tip which is never assumed that you know this better than them if you that is kind of general thing because the I heard about the people that are like come to their dev and like oh this is just a button why does it take two weeks that is a perfect way to get people to hate you perfect way away so because you don't know what are all the dependencies just don't assume just that's not your role your role is to design to answer the customer problem and to help them so how do you help them instead of just make them hate you so what is your role the way I see it is short feedback cycles so I'm the source of context for my data science team for my dev team I'm the one that talks to customer to customers I know what they want better than them that's just the truth so whenever they create something we just they want okay I did this little step tell me if I'm on the right direction and if you and then if you do that very quickly and you're going to save you a lot of trouble later that they did all this work and they're oh we're changing the direction that's actually not what the customer wants so I should want to do with this little kind of exercise this is a real situation of a lead that got a score of a hundred okay the highest score possible and all the things where I'm showing here are positive signals of why that lead will purchase okay you can assume that this company sells it stuff for example so these are all different options that I got from the data science team of features that they can extract in different ways we don't need to get into that but different ways of extracting extracting those features and the order matters so if you're thinking from a sales rep this is the order in which you will see the different features the different model drivers that will that is supposed to help you sell okay so you can see we're talking about company size industry department tenured company website technology this is jQuery sorry and then there are five options so I'll give you kind of a minute just to take a look at those five options and then I'll kind of ask you which one you you think will be the best and again your perspective is not is it going to be complex what to do you're just thinking about what a sales rep would like to see and remember the order matters there's and by the way there's no right I'm actually hoping that you help me because I need to I need to figure option for presenting the features for a certain sales rep so a certain sales that rep sees a lead that gets a score of a hundred and they want to understand why they got that hundred what are the things that will lead to them converting yeah oh yeah yeah okay okay so yeah yeah these are actual signals that we take out and we yeah okay so anyone chose one okay why okay okay okay okay okay okay cool cool you also chose one you want to yeah okay okay cool anyone choose option two okay option three yeah why I chose option three because I can't tell a good story and a sales person may want to defend themselves so if you how about company size information industry is the same thing but some industry is something deeper level okay okay cool option four no takers for option four surprise option five most people anyone you want to say why option five well I guess the difference between five and one being some industry that the sales person once they establish the sort of the size of the company gives some initial indication of the ability to purchase yeah more of the compatibility okay okay anyone want to add all of five people yeah okay so my preferred one is actually five so I'm happy that the majority and I'll explain why it's kind of you touched all of the reasons why I'll go one by one so with the one yes I agree one is one is good I don't like this the website technology it's jQuery it's so generic it doesn't tell me anything from a sales rep and the industry software I actually like the fact it's e-commerce software because it has the word software so I'm saying well why do I need the industry if I can go all the way to the sub industry and say okay it's e-commerce software I'm kind of answering two questions instead of one that would be my goal when it comes to option number two you start with the technology I don't like that like I'm a sales rep I wouldn't like because again they don't have too much patience they're gonna read maybe the first three or something you're starting with the wrong thing that would be that would be my take option three is I don't like this again too generic I would just remove it and again you start with the technology option number four is good option number four actually has company social presence medium which is just redundant it doesn't tell any story it will just confuse people and you're gonna get questions it doesn't help the sales rep doesn't need to know that that company has ex-followers on Twitter they don't care doesn't help themselves that's why more is not necessarily better in this case and five the way I would read it out from a sales rep company size 10k to 50k what does that tell me from a sales rep they have money to buy I can sell more sub industry e-commerce okay cool it's e-commerce and software now the person department is their IT I sell IT products I'm happy that's the person I want to talk to they're in the company for more than five years they probably have a power to make decisions that it all looks good oh and they're looking they have HPE hybrid I have no idea what that is to be honest but if I'm selling into IT that would probably mean something to me and this is a simple example and you see how complex it already gets just think about if the square is 50 what do you show three positive and three negative what negative what positive and also things like this industry software and sub industry this is redundant right because this one kind of has this and technologies that you're saying well I don't want this to even take any place in my database no one cares about that so all these things like how the feel how much how populated that field is right sometimes you will see things that are just not populated all these different things we're thinking about now to make this a success and again you think about the solution and the whole product it's a complicated product we just have to simplify and make it easy to digest for sales reps and for marketing people so what we're thinking actually is to have a story like an assistant in a way that sales rep actually kind of get as if an assistant is sitting next to them and kind of reading out what they did from a research perspective so if it's an A okay from a graphic attribute suggest that it's specifically industry and software and company side it's gonna be like highlighted also the company uses this and that and if it's gonna be a B it's gonna give you the good things and then it's gonna say however and like something less good and for C it's gonna be the opposite but just kind of phrasing it as a sentence on top of that UI so there will be this UI kind of showing you this the kind of clear thing of what's going on and then you might also want to have a sentence just explaining to you in plain words yeah and kind of just the last things I want to talk is general PM advice when you're designing these kinds of things we talked about all of this so don't underestimate it don't ever tell your dev team it's just a button why does it take two weeks and just also think about all the different things you need to you need to work on and they need to work on to communicate with your data science and dev teams the more context they have the better right because they don't know how to answer this question they might have a hunch but you're talking to customers all day you know the customer problem you need to answer this they just need to give you all the data and tell you what's available what's not available encourage fast feedback cycles reviewing the output from a customer-centric view and never lose sight of the customer problem right sometimes we just find ourselves oh there's this cool signal we can include like company social presence no one cares right people the sales reps don't buy lead space they buy a solution to their problem so you don't need to dazzle them with all these excessive things when all they want is to sell again going back to that simplicity and focus on simplicity actually went to the fifth bullet point that's what I had any questions how much time did it take oh cool yeah in your data finding do you use any general products such as Tableau or is it all sort of in-house that you leverage yeah so the question is do we use Tableau or do we use our own analytics yeah so that is a good question we you saw the visuals you saw here well have a lot of clicks are all Tableau yeah so I'm still going back yeah this is all Tableau we did use other things with Microsoft Power BI click and we're actually working now on making this online so at the moment we're still in the situation where we send it to customers but the idea is that we understood that there's no real most companies at least in our size there's no real reason to generate your own analytics when you can purchase something here that's already fully baked and they know how to work with startups they know if you want to OEM it or if you want to use it for yourself usually we find this is an easier solution yeah yeah go yeah go that way can you talk about the rest of your technology stack what part of the technology stack how well is your visualization what are you using for kind of machine learning technology the warehousing the question is what do we use for machine learning and data warehouse to be honest I don't know the data warehousing part I know that we use SVM models with proprietary additions that we added but I don't know so if we go back to the options that you had is the machine learning finding this like granular enough for you to click on like for instance there we go I got it if we get to option one like this industry is the machine learning algorithm like granular enough so that if I click on it I can go to sub-industry and then under that like further granularity you know what I'm saying so that I can keep pushing on to any one of those and then get deeper and deeper because if I'm a self-person and I want to know even more detail within the sub-industry like if the algorithm like really general and it goes all the way to sub-industry we'll zoom all the way in so the question I need to repeat for the the question is can you zoom into these different signals or are they all basically the same layer that's what you're talking about so no you can we didn't think about it to be honest that's a good point of kind of allowing it in real time for you to actually have this software and maybe have like this plus sign that says so you do have software that is built above industry built above sub-industry and size range is built above the size exact so essentially you can do that it brings a question of do you want to allow that because do you want the sales rep to actually start playing with all these or do you want to say okay these are the drivers trust us again trust us that we're doing what we're doing what we want what you think is good instead of kind of making them focus on time get to what they actually really want but you can create it that way if you want like one is all yeah so the question is do we make it model custom for each customer or do we have something on it so the data is custom for each customer that's obvious right so we take the data for one customer and we don't do anything for another customer from kind of creating it at scale we do have the integration with Marketo and Salesforce so once the model is ready we upload it in the same way for all customers and leads start flowing and getting the scores so most of it is that is our goal because we want to scale this and we need to have as much automation as we can so we do except of the data we're thinking how can we automate everything we even have kind of internal tools of okay you get this data from the customer that automatically creates the tableau so all of the different reports are being created and the integrations most of our customers do use Salesforce and Marketo when you start talking to customers that don't then you kind of have this customization but I would say that the majority of it is very standard and we're working on standardizing it even further except the data okay Have you considered for instance or have you used like information on the reason the cost for like losing a deal in let's say Salesforce or models also what is the competition doing and how you're losing opportunities for like the reason for our customers of losing so the question was if we're using the closed one or closed lost reason in Salesforce right so yes so actually whenever we scope the data that we get from the customer we actually look at all these different stages on the opportunity and we do multiple testing seeing should you tag them as negative or positive and the funny the weird thing is most closed lost instances we actually tag as positive because we're saying a company got all the way through an opportunity and then when became closed lost you know it might be someone left the company or some legal issue or lost a competitor there's still a good company to go after because they reached all this throughout the stage so we really tested out so there's all these stages in Salesforce custom stages so essentially what we do we have this way automated way of doing it is that okay let's take stages one to four as negative and stages five to seven as positive how does that look like now let's look at one to three versus four to seven so we always look at kind of thinking about maybe all the way they got to stage four actually means that they're good or not so we do take that into consideration we don't take the text we don't look at like a reason and analyze the reason that that's not something we do okay yeah what kind of methods do you take on in order to interact or interface with your customers get feedback about the products that they're looking for are you doing focus groups doing surveys or are you doing one on one sessions with them what's that feedback look like for you yeah so the question is how the feedback looks with the customers so we don't do user don't do surveys we just meet with them on a regular basis just this morning we had a two hour we have a two hour scoping session which is painful but it's important because we have all these questions we need to ask and then every quarter every six months depending on the refresh cycle we get new data from them we talk with them understanding did something in the strategy change in the last three months that we need to know about that will affect how the data will look like right so we just these are we're a B2B company we have 140 customers it's not like something huge like Salesforce with 3000 so essentially we have weekly meetings with every customer that has a predictive model that we actually look into the trends and getting the feedback and every every three months when they pull the data we kind of get the actual quantitative information but the qualitative information gets added all the time so you're saying well not all 140 of them have predictive models but the ones that have yeah there's a CSM assigned to them yeah yeah yeah so I know you mentioned the CSM but just in generalities like how many algorithms do you use because I'm not sure she is a very popular one her Eichelian future so is it is there three or four or could you name? I don't know yeah that is the we have you can probably talk to Philip at the back when after this and he will be able to answer yeah at the back so what do you mean by signals the data points that we gather yeah so the actually the data points that we gather we work with 40 data data sources in real time so essentially when we get a record we ping all these 40 data sources that are third-party data sources they don't belong to any customer of ours and then aggregate it to find out what is the most accurate company size most accurate company size is the most accurate industry yeah yeah so the question is do we use do we consider using sales feedback definitely but you also think about the scale of the sales feedback we mostly focus the sales feedback when we implement the model so we actually there's a score running and we just want to validate that it's working so we will meet with a sales rep on the customer side look here are 20 scores with companies does it look good or not and they will say oh yes the scores look good or not sometimes we even do blind tests we have a sales rep that goes through 20 records gives the score the way they see fit and then you see how it is compared to the score but gathering feedback from a certain sale usually there's not enough meat in that right because you will have one sales rep if it's someone that sells enterprise customers it's not going to be enough for you to actually use that as a feature in the model okay personality test could be that could be so basically doing a personality test for all the sales rep in that organization and using that as a feature in the model interesting yeah yeah yeah so you are if you're like calibrating your models by comparing their scores versus other reps do you worry that you're just replicating rep thinking automating it maybe maybe yeah so the question is if we're kind of afraid that we're actually just replicating what the rep is scoring and just doing that in an automated way so we're using it as some feedback to and not to influence the actual score but more to understand the sentiment and things that matter to them more so for example we can score you learn from all of the data and then you score in a certain way but then a sales rep looks in that like oh this company has less than 50 million revenue I'm never going to talk to them so that is insights we don't necessarily have from the data maybe there's all these companies that have less than 50 million that people tried to sell to but now we understand okay that is a strong threshold a strong threshold that we can introduce into the model let's say okay anything less than 50 million they're just not going to talk to so there's no reason to score them high obviously you have to I understand what you're talking about the fact that you have to be kind of careful that you're not just doing what the sales rep says but we usually just use them as kind of a source of feedback to influence the model rather than just trying to say oh they think it's a 70 how can we tweak all these different things to make it a 70 you had a question right so how do we measure success of our products not an easy question right so what we do is essentially when we when we get the data for the refresh a couple of things we do one we refresh the model based on this new data right so we just have more data and we have closed loop for a lot of records plus we actually show them how the score could have if they used it influence the performance we're telling them we scored in August all these leads is a and we told you to call a more and now we can see that the highest conversion is for a so if you use if you listen to us you just got better results that is that is a simple way of saying okay lead space tells me who to go after I go after them and I actually sell more okay so that is kind of how we how we measure success when when the sale cycle completes the problem is that we're working with enterprise company customers that have a sale cycle of 18 months what do you do you wait 18 months until you're finished no that's when you're actually using quantitative feedback where you're going to the sale you go into the sales rep and asking you like the scores you just getting a sentiment of you know or maybe even measuring a step a stage higher up in the funnel you're saying okay I know the model predicts lead to closed one that will take forever so let's see how it's performing for lead to mql that's not necessarily what the model supposed to do it's not necessarily going to work right it might you might have weird logic for mql it's not actually going to predict lead to mql but it's going to predict lead to close one so that is a challenge we are facing of how to measure it it's a good point