 Yn ychydig, mae'r cyfnodd Kevin. Mae'n gyfnodd Cymru, ac mae'n gweithio'r gwych ar y cyfnodd Cymru o'r ffordd o'r cyfnodd Cymru yn y cyfnodd Cymru o'r gweithio'r cyfnodd Cymru. Rwy'n gweithio'r gweithio ar yr A typ yw BC. Yn yw'r gweithio, roi'n gweithio'r gweithio'r gweithio, mae'r gweithio yn y jannu ar y cyfnodd, a'n gyfnodd gan y gyfnodd Cymru, felly mae'n rhaid i'r gwybod a gwelio i. Rhywodd yn fwy o ddwy i. Felly, mae'n gyfnod ymlaen, mae'n gweithio ymlaen o'r cyffredinol. Felly, rydyn ni'n gweithio ymlaen o'r cyffredinol? Roeddwn ni'n gweithio ymlaen. Felly, mae'n gweithio. Felly, mae'n gyffredinol a cyffredinol. Felly, mae'n gweithio ymlaen i'r cyffredinol a mae'r cyffredin iawn i'w gweld o'r cyffredin iawn o'r cyffredin iawn. Rydych chi'n siaradau o'r productau, ac mae'r brand wedi'u helpu yn dilyn yw ddweud y Facebook a Twitter, a byddwch chi'n dweud y ffordd. A byddwch chi'n cyffredin iawn i'r productau yw Crowds, mae e'n gweithio i chi'n gweithio i'r cyffredin iawn, ac mae'n gweithio i'r byddwch. A dyna chi'n gweithio i chi'n gweld i'r productau a llunio cyfeiri. I started in community management a while back in the video games industry where my job was basically to get people to be happy and play nice when really they were angry all the time and basically killing each other. So that gave me kind of an insight as to how to create group dynamics and maybe help me work out a way to have people work towards goals in a way that is fun for them but does not involve a direct hierarchy with the people that they're interacting with. So I thought I'd start with this because I think it's a terrible, terrible definition. Took it from Techopedia, sorry if anyone's from Techopedia. But B2B to C, what's B2B to C? Business to business to consumer is an e-commerce model that combines business to business and business to consumer for a complete product or service transaction. Which is all cool. That was probably very, very accurate ten years ago. But in the meantime a few things have come along where you can have a B2B to C system or model that does not include or selling stuff to people in any way, shape or form. So what I would like to just you to remember is the second part of this definition from Techopedia is it's a collaboration process that in theory creates mutually beneficial service and products. They think product delivery channels is important, I don't. But just for the sake of clarity there it is. So crowds, what is crowds? I said it's a bit of a different beast. The mouthful definition of crowds is that it's community powered peer-to-peer social resolution on Twitter. So that's the one sentence. It probably doesn't mean much to you right now. So in a nutshell what it is is this. Imagine you're a company, let's say a big tech company and you've got loads of people on Twitter who've got problems and you want to find a way to interact with those people. And we're not talking about people who are reaching out to you. We're talking about the average person who might be going, ah, why doesn't this work? This is such and such company's product is the worst. Right now you've got very few ways of interacting with these people. Well what we do, what crowds does is work with a brand to define what kind of content they want to find on Twitter. So for example a number of searches that will say, well we want to pull in questions that include these terms, do not include these terms in that language and so on and so on. And once we do that we start, we turn on the engine and crowds are pulling in content, giving it good shake and then finding the relevant content, the good questions, the things that people can help with. So taking the questions from this guy and giving it to a group of people that are the crowds users and which we call experts, who are people who are attached to your brand in some way but do not work for you. So they are your brand advocates, they are people who are really into what you do and that way you can address those problems at scale very easily and in a way that's very organic and simple. So of course it's not, if you're a major tech company and the question is, my account's been hacked, please help me here on my credentials, that's not for crowds. But if someone's going, does anyone know how to do this? I don't know, use it for that software or if someone's got a, oh I don't know how to do this, does anyone have any ideas? We can help you with crowds and crowds really lets you go below the waterline and address the rest of the iceberg or the stuff that you don't usually see. So that's crowds in a small nutshell. And this is why it looks like, so this is just one steal, I'm not going to walk you through all of it, but just so you have a bit of an idea with this being a navigation between all the different types of content, this is all the stuff that we pull in and this is where your interaction will happen. Very easy, it's dead simple, if you're an expert all you have to do is sign up with your Twitter account and say, well I mentioned all of these out of a choice. I speak those languages and that's it, we will make everything else really simple for you. And so the weird bit, actually I'm going to go back one, the really tricky bit about crowds and the bit that is usually the hardest to explain is that the people, the crowds users are not employees, they are people who are just happy to help. And so bearing that in mind, so where we have that relationship with first the clients, the people who actually pay us to set up crowds for them and the users, they're too completely separate, there's no relationship of power, there's no, one can't have the other what to do. What would you think the main one or two challenges would be for product manager when we're working in that kind of situation? Does anyone have any suggestions apart from you guys? OK, well the challenge is this, how do you make it work when clients and users define value differently and have completely different requirements? The clients are the companies, they are companies that have a problem and are looking for a solution, so they're in this as an economic agent. Users are people who want to help but will never see the admin section on the back end, they just want to help and they will see bits of the app that the client will never use. Those two groups are completely separate, have different agendas, different motivations, different everything and yet we have to make this work for everyone. That's what we've been working on for the past three something years and that is what I'd like to talk to you about. Tonight I'd just like to share a few thoughts as to how what we went through, what it's taught us and what you might take away if you ever want to design a B2B2C app that is something different than an e-commerce solution. First one is how do you break ground? How do you manage your chicken and egg? The question being this, if you don't have a good app that meets your client's need, you don't have a client. I'll go yeah that's very nice, I'm not interested. If you don't have all the features that you need to make it cool for the users, you don't have any users to answer the questions. So would you rather have a really cool but empty room or a room that's not so cool but that's full of people that are really having fun? Well that's the dilemma that we had a while back and here's a few things that I took away from it. The first one is where do you start and I would say, and this is just me, I'm not representing Converse Social here, this is just my opinion, I would say start with a client. For the simple reason that the client has a defined and existing need, that means that they know what they want and they know that they want it. They have a clear end game with wind conditions and hopefully they've got resources that you can use for your MVP. So for example, just to give you a little bit of history on crowds, crowds sprung originally from discussion from people at Google. Google are clients of Converse Socials, they use some of our other solutions and they have a wonderful community of experts who help Google users on forums mainly and if you've ever used any Google products and search for solution online and someone just popped out of nowhere and gave you wonderful help, it's these people. And Google said, wouldn't it be really great if we could find a way for these people to engage with our customers in other platforms? And we said yes it would and no idea how to do it but that's how we got started. So we had deep conversations and we thought about it and long story short is in the end because you're talking to a rational economic agent, you can come up out of this with a business model canvas, some wind conditions and assumptions register. So the business model canvas is how are you going to be charging this, who are you going to be selling this to, what do you need to make it work and so on and so on. The wind conditions is, well you know what, our clients want to reach either take some of the load that goes to the agents and transfer it to other people or they just want to make their clients happy or they want to reach people they were previously unable to reach. And that builds your assumptions register where you say, well all of this at that stage is this is about the only thing that we know for sure. All the other stuff is what we think our best guess is and based on our research we think this and that. And so you put all this in your assumptions register and you say well from the client's side this is what they want, this is the goal, this is what we're shooting for. We want to create this tool that does this and then you go see the user and that's where it gets interesting because the user is key. There's no user, no value. You can have the best tool in the world if there's no one using it. Well tough luck. And there's a couple of things about that that I'll get to in a second but it makes for a very interesting problem. And you have to create a completely independent set of documents for this new, for these users that have to be completely separate from what you did for the clients and you need to do a product vision board. So basically a simplified business model canvas where you explain what the product is but you don't have to worry about who's going to be paying for it because it's not the users. You do your ethnographic research so your personas and all that. So who would be using this? Your use cases, how are you going to be using it? Your use a case matrix which is basically once you start matching things what emerges as our key demographics and our target users. Key expectations on the user side because they don't come in with no expectations. There's plenty of solutions out there where they can help people and they've been doing it very well for years and years. So we're not going in there as no say you're saying hey this is what you've been waiting for forever. We go in there with look at this shiny new thing that we haven't built yet interested. And that's as you can guess it's not something that people react to immediately. And finally you can build a second assumptions register where you say well here's a different bunch of assumptions that will need to prove or disprove. I completely unrelated to anything else. And at that point you're ready to get cracking and do and make the magic happen. Because you're going to have to take those two things and work out a bunch of solutions that are at that stage hard to prove or disprove. The first thing you're going to do is you're going to have to create a product backlog with three types of things. So first the important commonalities for example the brand is going to go. I want users to help clients with problems because that's what I want them to do. And the user can say well I want to help people because that's something I enjoy doing. Great that's a commonality and there's a bunch that are really important. Secondly you've got all the stuff that only the clients care about. So I want data. I want an admin section. I want a back end. What's your API? All that kind of stuff that the users really don't care about. And finally you have what I call the user environment and I get to this in a minute in more detail. But it's basically all the stuff that the user cares about really deeply but that the company doesn't care about, not directly. And once you do this you can start building, so your product backlog, you can start ordering it, the usual I don't think I need to get into too much detail there. But you have to start creating a feature value matrix which is basically a simple way of explaining the value of one feature to different people because you're not going to be stressing the same things to the same people all the time. You're going to have to explain the same thing to people who have a commercial background, people who have a technical background, people who've got no background at all in any of that but are users. And finally you can start testing those assumptions that you had in your register. This was the hardest part is we've got all this but where do you decide what to build first? So what we said is what we're going to do is we're going to build an MVP and the MVP is going to have the smallest critical loop that we can. The first thing that we built is how can a user answer a question in a way that is more enjoyable to them than it is to do on another platform. So it's more fun to use crowds than it is to use a forum. That's it, that's the smallest thing. You bring the question, give it to the right person, that person shoots back. That's it, that's how we built it. Nothing else, a little bit of a... So I'm looking at one of my programmers over there who's going to kill me when I say nothing else. There's actually a lot in the background but we really kept it as simple as possible building a previous experience from other products. But then you get the question of the environment which is the essential additional user requirements. When I said that there's something deeply, deeply fun is that when we were talking about users and people who actually answer questions in crowds, there are people who are external to the company. The company doesn't pay them, doesn't reward them directly, and you have no way of forcing anyone to do anything. When you have your regular customer support software, the manager buys it, says, I love that one, and then turns around and tells his team, you know what, start using it because that's the one I want to start using. There's no discussion, everybody else just has to use it. In this case, no, each individual user will have to make a conscious decision to start using crowds. That means that we have to be... We are competing with everything else in that person's life. That person is answering questions on their free time where they could rather be spending time with their family playing video games, going to the movies. That's kind of like tough competition. All of you think about it. After work, do you want to just go on a lap and start answering questions for strangers? Most of you, myself included, not necessarily, especially since it's all people that you don't know. We had to find a way, and one of the major hurdles that we have had when explaining this to potential clients is they look at us and go, who are these users, and why do they do this? It doesn't make any sense. It doesn't if you think that the user is an economic agent. It does if you look at the research that exists as to why they use forums or crowds for helping people. I know there's a former Google expert in the room. I know who he is. I don't want to point your fingers. If he's comfortable to raise his hand, I'd like to ask him a couple of questions. Thank you. Sorry, I didn't want to call you out. You were a Google TC, a top contributor to an expert, and you were part of the Google programme for just short of two years, and you used crowds, the early versions. Why do you help people? I think there's a human nature, isn't it, to be someone who helps other people. Especially if you're a client manager, you get knowledge over time. You want to give back to as many people as possible. I think that's why a lot of client managers go into mentoring, consulting, and finding their own businesses because they want to give back what they've learned. There were all sorts of other parts. I'll go into that in a bit. Do you want to community as well? There was a community of top contributors. I think it was one of about four in the UK. It was good. I love your answer because it's exactly what I have on my next slide. Thank you very much, first of all, for agreeing to tell us a bit more about this. As you might remember, I said earlier that I usually worked in video games. There's a lot of video game theory that goes into this. It's kind of old news in video games, but it's something that's slowly tricking down into more mainstream tech. Exactly what you're talking about, the research shows that people do this for four things. The first one, altruism, which is the first one you mentioned. People like to help. Some people will volunteer with the RSPCA. Some people will just help the elderly lady next door, whatever it is. People like to help. That makes them happy and that's why they're doing it. Some people like economic gain. They help people. They're not being paid, but as you said, tript San Francisco every now and then with Google, not bad. Some people like the status. One thing that we have is experts across all our clients are usually people who have a connection with a brand. Some people like Google experts are people who really feel strongly about Google products, but we also have clients that are video game companies. People love those video games. People who have an interest in a matter subject and will answer in questions, not because necessarily they like the very serious big company that treats with that, but because they like the idea behind it. They want to be part of that. They want to be part of that world and they want to have something else that you were talking about, is you want to be part of that community and it's a community that makes sense and to you it's something that has value socially. Finally, one thing that motivates people is people want to be the best at helping. Those are the four things and everyone who helps other people is somewhere on that quadrant. It's usually more than one thing. As you said, it's usually a bit of a mix so you're somewhere in the diamond, but those are the four things. That means that when we have to explain to people to our clients or potential clients why users would be using this, people go, I don't think so. People are like, yes, here's a number of companies where this works perfectly and it's not a good fit for every company because, for example, if you are a company that makes washing machines, you might not have enough people who are passionate about washing machines to do this, but in most cases, for a lot of big clients, this is a very viable solution. That means that once, if companies accept this and they accept this dichotomy, they can start to understand and accept the challenge that we are faced and that they are faced with, which is namely, we need to keep our clients happy and we need to keep our users happy. And so, another oxymoron, the superfluous essential. As free agents, your users have requirements outside of the narrow scope of basic client requirements. For example, the client say, all I need is that loop where people can ask the questions for me, that's all I need, don't need anything else, don't bother, that's the bit I want. The user is going to say, no, that bit's good, I like helping people, but how about the other stuff because right now I'm not having fun and if I'm going to do this, I'm going to be worn out after 15 minutes. Fair enough, it's our job to make it so that it's enjoyable for you for as long as you stay and hopefully, if we do a good job, you're going to spend more time answering questions every day, helping more people because we are going to make it easier for you. And so, we have, for example, a number of features, I don't think I need to delve into these too much right now, but we have a number of features that we think are very important, the users think they're very important, brands in general, they think they're nice, but to be honest, it's not necessarily going to be the top of their backlog if we'd ask them solely. So, for example, leaderboards, where do you stand in the community? You score points when you help people, you know if you're at the top of the leaderboard, bottom of the leaderboard. This is not something that has any economic value for clients, but it's something that people care about. Incentivisation, part of the programme is if you want people to help, you can give them like a reward. So, it's not just trips in New York and free tech, but it just might be a mention on Twitter or something to reward people for a job well done. And finally, it might just be something as simple as built-in communication channels. People want to, our social animals, as people say, you're part of a group, you're part of four people in the UK, but it's nice if you can speak to those four people without having to use other platforms. So, something that's really important for us is striking that balance, and you will only be able to do that if people accept this condition. You cannot be a crowd's client if you don't accept that this is not just some kind of, this is not just your run-off-the-mill support solution. This is B to B to C. The C being the users, obviously, if you don't appeal to them, the rest is completely irrelevant. You cannot power through, you cannot do any of that. You have to accept that. We have to build some stuff that, quite frankly, as a client, you don't necessarily care too much about it. Some clients do, and are very good about it. Some clients just kind of go, okay, fine, do it, but no. Let's move on and get cracking and do other stuff. As you can tell, I like big words because they're fun. Something, and this is the bit where the designers will, their ears will perk up, asymmetric simplicity. Because you've got two sets of people to satisfy your clients and users, you need to have possibly two completely different approaches to design. Because you will not be able to make everyone happy with just one solution. So don't be afraid to really look at it in two completely different ways. And so it hasn't seen any of this. So I think right now she's a bit worried. But what this means is basically this. When you're designing for clients, so they're the people who see the economic gain behind using crowds. But these people, A, they can be trained or supported individually, which means you can teach them how to use crowds. If something is not quite there yet, or a bit complicated, or a bit obscure, or not quite optimised, that's fine. They're at work, they've got plenty of time, they're being paid to be there, they're happy to be trained. And there's not too many of them. They also have the economic incentive to use your tool. So don't want to do the sales pitch for crowds here. But helping someone on crowds is three times cheaper than helping someone through your regular Twitter agent thing. And it's, I think, eight times cheaper than email. So there is something there. So that will make people happy. And finally, because they expect a high degree of customisation, they also accept some pain points. So for example, at one point, the admin section was not very intuitive. It wasn't very pretty. Some stuff was buried a bit deep, and it was hard to use. But that's fine. Users are the other end of the spectrum, because you cannot onboard your users individually. Basically, what happens when a client sends people, so some of the experts of crowds, is they usually invite them directly, say, hey, here's the link, click on that, or they put it up somewhere on their website, where they say, hey, if you would like to be an expert, here's where you can go. And that's it. There's a help section in crowds, obviously. But we cannot onboard those people individually. And they've got no external incentive to use this tool. They already have their forums, they have their Twitter things. So really, if we don't catch their eye in the first few seconds, or at any point in the first five minutes, they kind of go, this is a bit of a chore. What's this? I've never come back. And that onboarding is the first challenge that we have. And the second one is, whereas a client will have a longer user flow, so, for example, your average client might interact with the admin section maybe once a day, twice a day at most, your users will do the same loops a lot for quite a bit of time. People spend a lot of time in the app and they decide to use it. And if something's a pain point, they're just going to stop using it. So don't worry, I'm not going to ask you individual questions. But out of curiosity, how much time do you think the average crowds user spends in app every day? So who thinks it's less than 20 minutes? Every day they will spend 20 minutes of their free time helping other people. So less than 20 minutes, we've got four people. 20 to 40. Yeah, about 10. 40 to 60. Yes, six, seven people. More than an hour every day. Cool, congratulations. Three people just won the quiz. People across clients, so this is not just Google or that company, people will spend on average 67 minutes a day helping people. That's 67 minutes each and every day. That's an hour per day helping people in app. In that time, because we do a good job, people will help a lot of people and go through the same loops potentially 20, 30 times. The average is, I think right now, it's just above the average expert helps a little bit over. I think it's seven people per day on average. But we've got some outliers who've helped literally 10,000 people in the past year. We need to make sure that these people are having a good time. Simplicity and ease of use is our goal here. This is something I cannot stress enough. If you're doing B2B2C, you have to make sure that the people who are on there for the money get the royal suite. The rest, it doesn't mean that you shouldn't be making an effort, but that's where the trick is. This is what crowds look like when it was an MVP. I know it looks like 2004. It wasn't. That was the first year, but this is what it looked like when we validated the core loop. It's terrible. You can see all your questions are here, a couple of filters are done that way, and then you get my bunch of stuff here. It was not the best experience. I think that's the one that you used. I'm not going to ask if it was suboptimal, but I'm guessing it was. Maybe, possibly. But what we did is we decided, once we validated that, to take it to the next level, add a whole bunch of features, and make it really cool and slick. Our design team did a cracking job, and now it looks like this, as you can remember. I know it doesn't really show, but we added a lot of features, a lot of cool stuff. The really cool thing is we still managed to get a SUS score of 92. For those of you who aren't familiar with how SUS works, it's not a percentage. 92 puts us in the top, like 1%, or 0-5% or something. This is AAA experience. It doesn't look like much, but once we figured out how to make it simple and beautiful, it's obvious. There's a lot of work that went into this. We got really good feedback from all the experts across all our clients. This was really the key. This is where you will make or break your app. When you're trying to appeal to people, this is where you need to focus your effort. If you've got limited resources, design resources, this is where you make it snazzy. The clients don't care if it's grey and dull. The users do and they should. This is the simpler bit. Recently we released a new feature, which is a clipboard, which lets people create and manage clippings that they can reuse really quick so that they can literally answer questions five times faster. If it's something that they've seen before, there's a bunch of cool features, two clicks, done. That's, I think, our most complicated feature to date. We released it. It won't mean much to you, but this is to show you you've got things like filters over here and you can manage. You've got tags and you've got counters and you can do things per category and we've got quick search and suggestions. It's all smartly done. We still have a SUS score of 89 and we're not done yet. This is something that we released three months ago and our design team is already working on iteration because we want to make it better. It's good, but not good enough yet. This is where we focus our efforts. This is where we have... If you want to do B2B to C, this is, in my opinion, where you'll have to concentrate all your design energies. Force thing, an informed impulse. I like those words, but basically it means balancing listening with guiding clients and users. The advantage that you have when you have a kind of app that people are familiar with is you can explain it very simply. For example, you can say, well, it's like delivery, but for this, it's like Snapchat for cats, people, three words, four words, people get the idea. When we started talking to people beyond Google, who were part of the equation from the start, we went, so it's a bit like, but not really, and so you've got these people that you don't pay who do stuff for an hour a day, and people are like, what they... Yeah, not interested. We need to find the balancing act making sure that people who want to help you can by giving them freedom and what they need to do their thing and then giving them guidance. If your app is something that's truly trailblazing, so it's something that you can't just say, well, it's Snapchat for cats, you will need to find who your early adopters are and within those early adopters, for those of you familiar with the chasm theory, you will need to find the right people within the company, and that's not easy, because if you remember at the very start we were doing ethnographic research, you will need to identify what kind of positions will be the people who recommend your tool and are there the people who can actually sign off to buy the tool or who do you need to convince? So you'll need to find the right people, make the short introductory content available for key questions, so if someone wants to hear about crowds through a colleague from another company they need to be able to have a look at the web page and understand the idea in less than 30 seconds, but then you also need the comprehensive follow-up content as well. For example, when we were talking about the four reasons for people to help using crowds, people were like, what's that based on? Who came up with that idea? Well, actually here's the research, so if you think it's a bit far-fetched you know there's a bunch of guys with PhDs who've been working on it for ten years, so there might be something there. And you need to educate your clients because this tool works, I mean crowds works very well, makes a lot of people very happy, it's not for everyone and it does come with its own challenges. So you need to educate people and that means three things. What do they need to know about external constraints? So you need to tell your clients what we'll do our very best to help you and make you happy, but we also have to think about our users who have other requirements and we also have external constraints, for example, we use, this is on Twitter, so we have technical limitations from what we can do through Twitter, so there's a bunch of things that people need to be aware of. But you need to do it in a way that makes sense to them and those they report to. So you need to make sure that you can define what you do for this idea and define it to different people. You need to be able to explain to potential expert why they want to be an expert. You need to be able to convince a CMO to spend some money on crowds, but you also need to be able to talk to a customer support person saying why this is amazing. And if you cannot do that you will have a very hard time finding your early adopters for your really cool idea. But then you will also need to be open and transparent about your product backlog and prioritisation. That you don't see with too many companies where people just accept that it's there. For example if you take an ubiquitous tool like Jira, pretty much all of us have heard of Jira. We use Jira. People use it because it's there and it's a solution that's been there for a long time and it's quite good. But how many people here understand the constraints? Need to be convinced day in day out to use it? No one. Because it's just Jira and that's what people use. If you cannot make people understand those constraints at one point they're going to go well I'm the client, do this because otherwise I'm not going to be your client and sometimes you're going to have to go that's the way it is. And that's because you are the voice of the user. The user will not be talking directly to too many people. They might interact with their customer representative at the brand they're experts for but that's about it. So that means that you are the one with all the knowledge that you collect from the users and that means you have to tell a wide variety of people you have to understand then tell a wide variety of people what people use to think and why things are important and why you should do it that way you need to be able to back it up and that's the tricky bit because some of it is very easily quantifiable. When we did this we released this last time productivity went up 10% so we're releasing this which is similar so we're hoping to get another 10%. Sometimes you're going to go 90% of users want better leaderboards so we think it's important there's a demand but we can't tell you if it's going to go up 2% or 25%. No idea but let's give it a go. Once you educate those clients about your user needs and challenges you need to balance making your users happy with making the clients happy and that's the tricky bit. As a product manager I'm the guy who's in the front lines I'm travelling the world all the time talking to all the big clients and saying well here's what we have we have this special thing that we need when can we start it in sometimes we can so sometimes we say cracking idea let's do it let's do it like start next sprint 10 days from now which Devs aren't very happy about but that's the perks. Sometimes you kind of say good idea not straight away sometimes you can't tell people that's a terrible idea we're never going to do that it makes sense for the client it makes no sense for the user and you have to be able as a product manager to identify that early and you have to understand that that's where you stand that's who's voice you are doesn't mean that you have to count out to users but you need to understand that these are the people who's voice you need to be. The last challenge that we face especially early on is that crowds is a very versatile tool you can do many many different things with it some people just want to answer questions some people want to reach out to new markets some people use it to say we don't offer support in Mandarin because we can't afford agents who speak Mandarin but we have a bunch of users who speak Mandarin and can support our community and that's really cool so there's all these things that you can do and we've got so much data that you can use to track your success and that at first when people said so what does crowds do exactly and we say what do you want it to do it can do all these things and we'd give them this buffet of ideas and they'd look at us and go I don't know you tell me I've never done this before I've got no idea what I could do with this and so we have to do this buffet but you have to tell people you have to help people and you have to provide leadership and what you have to need to do and the problem that you will have is that when you do that you will be facing different expectations from all the different clients and users because it's not just one client and one user it's many clients and many users so you need to juggle that who've got different levels of understanding so for example the people who run a crowds programme for a company day in day out will have a deep understanding of the tool they understand exactly what's happening and then once a year they need to go talk to their CMO about this little quirky tool that they use and they have to explain it to a guy who's got exactly five minutes to understand what crowds does and who doesn't care one bit about making people happy he just wants to know how much money he's making how much money he's saving us so you need to make that and people have different levels of buying so that means that some people you will need to be able to juggle people who are really into crowds and people who give it a go but want to see what happens you have the best overall understanding and that level of understanding is what gives you the position of leadership you don't get to tell people what to do you do get to suggest people not because you're smarter not because you've got the epiphany and that's your word is law but you have that information that people don't have you have been thinking about this programme for three years I've been thinking about this programme for three years I've seen what works, what doesn't work I've met a lot of users a lot of clients I've talked about it all over the world I've been thinking about this these people met you five minutes ago and they don't know what you're talking about is nonsense so you must show your leadership by increasing your general knowledge so then general knowledge and everyone's knowledge of this you need to be able to frame expectations and this is very very very important I cannot emphasise this enough is you need to say here's what we can do here's what we will never do you can try it as much as you want it will not work and here's some stuff that you can try but who knows if it will work and I don't recommend it and we've got clients who have extensive programmes where day in day out hundreds of experts help thousands of people and it's really this is what we had in mind when we started some users told us we've got a bunch of experts who we don't really interact with but what we do is when there's a problem we activate them like this because for example let's say you are a big tech company and one day your servers come crashing down and you've got hundreds of people on Twitter going like what's this is it down, is it just me this is rubbish, I paid for this people are angry and you've only got like 10 agents because you cannot have foreseen this so you can't answer everyone whereas you've got crowds this company said literally click our fingers and activate all our experts who can then just for a few hours be very very active and answer this and they said that's what we want to do to be fair before they told me about it I was like that use case never even crossed my mind it's maybe not what I would suggest but if it works for you do it but building on that experience I can then talk about this experience to other clients and provide each of the other people because I've built that knowledge using those clients and so you will be putting people and clients in touch and you will be building this understanding and slowly you're gathering more and more understanding and more and more people and you're moving away from being this curculele app that's used by two companies to having people from companies call you and say hey I'm from this big company or this big bank and you're like okay we're not talking about tech anymore big banks and you will be moving and you have to cross that chasm and you can only do that but you can only do that if you frame your expectations early and often and if you suggest solutions over possibilities you've got a buffet that's probably what you're looking for you want some help customising that's fine we can do that and the conditions for leadership because you know it's easy to say be a leader but no I'm not I'm just some guy that they might or might not listen to you have to build that position of leadership and that means mostly five things the first one is you need to be accessible when you're building a B2B2C proposition especially a curculele one you need to be involved at every level which means you're not just talking to other product managers because product managers love talking to other product managers because we all speak the same language we understand each other we're talking to users we're talking to people who are CMOs talking to people who are customer support representatives we've travelled to Asia to America, across Europe talking to people with different backgrounds different cultures, different languages and they all have a different understanding of how things should exist and should be run and I can only do this I can only gather that intelligence to be accessible but you can also only build the goodwill necessary for people to want to talk to you and to use this if you're transparent that doesn't mean that you tell people everything but that means that you have to make all the information possible available for example we have public Trello boards where you say these are the bugs that have been reported that one, yeah we're fixing it's really bad that one, no because honestly it's a lot of work from not much he's our product backlog he's our product backlog and it's really cool and people might agree or disagree but people cannot and have never faulted us for not sharing information we are transparent as to what we do and why we do it and sometimes it also involves admitting when things go wrong and sometimes they do I go for example to the Google conference every year for experts and I'm in front of a sea of experts there's something like 200 or 300 of you who are very nice people the nicest people you ever meet they're all there to help and they're all very altruistic but they will also roast you like you wouldn't believe and I'm like why is this not fixed that's rubbish did you mess up and you can't just say well we think that KPIs like no sometimes you kind of go yeah we did mess up how can we fix it and that's something for that curcule B to B to C you need to be able to do that with everyone you need to have the goodwill A you need to have the confidence to do that you need to have the goodwill built up for them to accept that you're going to fix it and that you cannot just say one thing to one side and one thing to the other side it will come crumbling down if you have to be very very honest because that's your strength and that's how you're going to cross the chasm and you have and finally before something that's important too is you have to balance expectations and you have to balance that so I've talked at length about that so I'm not going to go over it again but something that you have to do is you have to balance utility and visibility because you are servicing so many different people users and clients sometimes something that you're doing for one group is completely visible to the other group sometimes no, you're going to go we need to redo a bit of the back end because right now it's not scaling properly so we're going to do this bit and the user is going to go it's been four months since you released something for us what are you guys doing, you're still here so we kind of have to say well we're doing that bit but they don't care about that bit because that's not their bit or it's not something that affects them directly so you have to maintain the balance between what's visible to all and that means that you have to think about ways where you can bring value and transparency to everyone for example when we designed the refoundation of the app when we went from the MVP that looked a bit 2004 to the new thing that process took from start to finish a few months but instead of just saying we're going to do our thing we're going to start coding codes are going to know we'll be typing away what we did is because we did a lot of user research and UX testing and you can talk to Zuzanna about the whole thing because she's the expert here but we spent a lot of time talking to people we made sure that that was visible so that we shared our insights we shared our trials, we shared our errors we updated people, we had we had hangouts every month with communities this Thursday Chris and I would have been talking to Google users next time it's going to be users from other companies saying this is what we're looking at and here's the challenges, what do you think and that means by doing that we were able to maintain visibility as we're doing utility and that gave us a lot of room to breathe because people don't just if they're subscribing to your quirky B2B to C originally if they're the early adopters they're the people who will help you that are interested in what you're doing not just a final result but chances are that they're thinking this could go I believe in this but I want to be shown that I'm right I want to know if I was right to trust you or not and finally a big part I think of navigating the early months of building something like this is that you need to make sure that people know that you're a positive force someone who's there to enable my clients I'm there to enable my development team my design team pretty much everyone sometimes I'm going to have to say no we all do this as product managers you've got your backlog, your priorities and constraints you have to make sure that people from different backgrounds understand this because sometimes you're going to have to say no to people who have no reason to accept that you said no if you say if the users who are volunteers and adjust no there because they're having fun and they think that you're a cool guy and that they're surrounded by cool people if once they say no because we think that we can't do it or quite frankly it's not a good idea then you go is you really that nice, do you want to use this if people know why you say no and they know that it's coming from a good place and you can explain why that will help tremendously and that's a condition for leadership I'm not just here to say yes I always try to empower people but I also sometimes have to say well no that's the limit, that's where we stop we're not going there and that's it so that's me that's all I have for you right now I hope it will shed some light as to how we did things in the early stages and how we're still doing things so if you've got any questions for me directly about what I've shared or anything else that you would like to discuss about anything tangential to this or if you've got any questions for our UX designer or for our coders they're over there, they're just saying no but you're trapped in this room so does anyone have any questions hey so yes so you have to understand what your value proposition is to the user but you're not presenting it that's the tricky bit because when you go to the client you're going to say use crowds because our value proposition is this when you're talking to the user mostly you're not the one who pitches this new tool you have to go through a third party so the QT manager at that company will say hey guys here's what we have but also you have to make that value proposition clear without ever stating it so the best you can say is it's cool it will help this and here's what it will do for you but you cannot formulate it as a proper value proposition even though in the background when you do your product summary you do know what that is from a product perspective I mean just in this context who would be your customer within an organisation is it more the customer support guys or marketing because it can be looked at from a brand perspective as well so that's a very good question because one of the hurdles that we had was early on we would go to customer support because our other tool is a customer support tool so we have a lot of people who know about this and been working with us for years but we would go look at this snazzy tool a lot of customer support guys would say this is really good I want this but they're not the ones with the checkbook so then they would go to talk to usually it's marketing or someone higher up then we have to so we had to formulate a different value proposition for them we had to make it understandable to them why they would want crowds and why it's useful and that's what I was talking about earlier when I said you need to be able to explain this to different people you need to empower the people within those companies who will on your behalf talk to the CMO and that will be a very very tough hurdle to jump so I'm not going to lie and that's so right now with crowds it's mostly customer support people those are the people who are most involved sorry I'll come back to you in a second sorry we do recommend to use like a tool to analyse all the data and the user's behaviours to take your next decision symptoms of how to modify the app how to iterate the app or would you say you would just travel meet your clients and get their feedbacks so that might be a Zuzanna question but what we did mostly is talk directly to users and we did a lot so we did do a lot of also a lot of surveys so asking questions but we also do a lot of user testing while we'd be there noting hesitations, deviations and we would create good prototypes that people could click through all that stuff that we did earlier on then made it easier for us to go into production because we knew that we already had the kind of data that we needed but we do use some software for tracking other things but I think to me the most important thing was really the interaction with the users and getting that kind of feedback directly but on Do you have like two separate packbooks for your users and your clients so they're all together and then how do you prioritise your users' needs versus your clients' needs so we have one backlog but every every item has two user stories how do you pitch this to the client and how do we present value for the user how we emphasise this is so how we order this is primarily because when we did our MVP we identified a bunch of pain points so for example repetitive questions so we did the clipboard so we already identified this and what we would do is where can we how we prioritise it is how can we increase productivity in a way that we can track results so for example at some point we need to find a way where we can say if the increase in productivity is due to this or due to that so that's and that would drive how we did things is this is the biggest pain point that we can identify we need to address this and in a way that we can track where we can track what we're doing and then we work what we do is we cycle down the the items and we iterate so for example we released the clipboard back in October I think and we will be iterating on this fairly soon because we've got feedback and then it will rotate at one point and say we're happy with this feature it will either go back into the bottom of the backlog as a V2 so sometime in the future we've got a bunch of ideas that we're going to do or we're going to say we're done we've done the best we could do having said that how you juggle that is also based on it's not just what you think the feature should be is how can you sell the idea I'm not talking about saying in a monetary fashion here but how can you explain the value to more people because sometimes for example users are going to say one discussion that we had recently was they say when I want to help someone I want to see their twitter profile because I want to know more about the person that I'm helping because it might inform the support that I offer that's something that users wanted and we implemented that's not something that the clients cared about because we cannot demonstrate value but we could say we've done this we've just delivered this for clients and this added this right now we're doing this and on average a person checking a twitter profile natively will go out of the app for like 20 seconds whatever it is so if you take this away people who spend this amount of time can do this much more and this is the value that we're creating and you can track that so that's how we went from client user, client user but usually the fun thing is you can actually find that there's a happy medium where pretty much all the features that we do are something that I have interest and that we can explain to both sides I'll just throw them off the back of that if you haven't launched your app yet so you're picking which features you want to launch with as a 5050 split because you obviously don't have results to test on it's kind of like a melting point so what we did it's like a proper MVP it was very bare bones and we were lucky enough to be able to identify earlier doctors amongst the user community where we could say hey users, it's not funny yet it's what we have in mind and if you can find that core target the people who stick with you while it's not fun it means that you can test those assumptions that we had in the very I think the second slide of my presentation and you can move from there but that means that you need to find those people before you launch the MVP or as soon as you launch it because that will let you build things until you have a viable solution for both sides but in the beginning the client is heavy some came back to you in terms of the customer for our clients for the fellowship now is it more kind of beneficial for established clients or like the startup would also be can take advantage of this startups like typically on the other side where they're still trying to build something so having experts of what they haven't built might be different so yes, so you're right so two things, the first thing I want to say is I'm talking just about crowds because we've got these other offerings that are conversational so those are like separate things but for crowds most of our clients are medium to large companies a lot of them are tech gaming even though we have more brick and mortar businesses joining us you can use it as a startup but even if you don't have your experts, yes what some of our users do is that they if you want to have customer support through Twitter most solutions and it's not just conversational will say well you have to buy a seat a seat is this many thousand pounds and such and such it's not very flexible what you do with crowds is we say pay for this much for this many searches you can have as many users you want and it's quite cheap and what startups do is they just say well we're going to use our own employees put them on crowds and they use it as a filter to interact with people on Twitter so it's not just customer support but customer engagement and they can do customer support so that's how a smaller company would do it so they basically would use for discovery and then as you grow and you start having experts if you have a product that people are passionate about you can transition them and you don't have to change a program because it's very flexible you don't have to just drop it and pick another solution just either scale it or just shift the population of experts away from being employees to being proper users like most companies sorry it's a continuation possibly of that but you talk a lot about these clients who have experts how involved are you in finding and creating because obviously if you don't have them then you've got the notion of users and everybody needs a user and I understand the good work that you've done to keep them once you've got them but how do you not build those bases? We do not build it directly as in I'm not going to go in most times and find those users what we do have though is a number of resources explained to people who don't have that resource that pool of available experts how to create that community and then we can accompany them and provide guidance because it's my background so I can say hey have you tried this oh god there's many people have you tried that and so on so on but I do not get involved I'm not going to do the heavy lifting myself the basic way of doing it if you've got zero experts is to find a pool of experts with similar interests for example if you've got a music app you don't have any experts but you know that there's this other music app where people would have the kind of right profile we might say you know what you might want to see if you can approach some of those people and state your value proposition and talk about what we're doing then you attract a few people and then you just start putting a bit of string until you get like a pool of experts the trickiest bit is you need to have a product that has some appeal to people as I said if you're building chairs regular chairs not design chairs and you want experts it's tricky but if you're a tech company if you've got an app that people are really passionate about or if you're into games you already have those people you just need to find them and empower them and there is a bit of setup so once you have that crowds requires fairly little maintenance so you can have 10,000 experts or 50 experts it's the same amount of work but there is a setup face where you need to reach that stage where you say I'm comfortable with how many people I have or I want more and that's where it can be tricky for some companies but we do provide solutions as to how to do that or suggestions or literature as to how to do that sorry one and then two yes maybe this is a question for your designers like what software do you use when you're prototyping your designs and how easy is it to create a new version so it is I mean you can I can give you the short answer or you can then go to Zuzanna who will tell you exactly what we use and do throw stuff at me if I get it right wrong so for the rawest of designs we use Balsamic and then when we want to make it prettier we use InVision for like very linear prototyping InVision we don't usually affect too many interactions we're going to be up and we have to do the same inVision and we usually we test early and we test often even with just very basic pro top and then we have high resolution mock ups where everything is clickable and it's pointed when some bits don't do anything because they don't realise it's not the app but that's what we use for design software just going back to the kind of user client just have you experienced an internal lens as well that you then have to balance against your kind of external requirement so I don't know how do you balance that against the external that's so in a way that's very tricky because usually if someone internally tells me that we can't do that because quite simply we literally cannot do it not because we don't want to but we can't usually it's very non-negotiable so it's tricky but at the same time that's where we have the most flexibility because I can go and badger my devs until they do find a solution because they're much smarter than I am so we can find something that gets actually do it the only problem that I've run into in that capacity was how do we how do we get enough devs to develop fast enough because it's very competitive market out there, a lot of devs a lot of people want devs and so when that I think that was the only bottleneck apart from that we've always so far received excellent feedback from all our clients about what we do and what order and there's only been a couple of times where we had to say no for internal reasons where literally we cannot do it there's been a bunch of times where people say oh you should do this and we say no we cannot because we are based we work through Twitter so we have to pull stuff from Twitter it's just that they don't do it but that would be we can't do it because of an external factor and only a couple of times it's been for internal and it's usually gone down pretty well because we were always very transparent and we came to them with the no recommendation and usually either work around or like an alternate solution Joshua here are the early stage of the development how do you define your personas like the costumentary is going to be able to see how do you define the eggs understanding so we could personas at the early stage before I mean going to the stage of the MVP for the next stage so we were lucky enough that the people that we need to make personas for we could talk to them before we built anything because we this sprang originally from a conversation with Google who we've got an excellent relationship with we said hey can we talk to all your users your experts and said yeah sure go so we could talk to those people and have proto personas and have all that fleshed out before we built anything so that was the advantage the tricky bit for us was not doing the persona for users because we had access to those people the tricky bit was making proper personas for our clients because as I said usually when we reach out people who reach out to us are customer support people we said oh this looks really great can you give me more literature then if they come back to say this is cracking stuff can we how can we take this further well at one point will you talk money because you need to buy this solution and they say oh it's the person with the checkbook and the problem that we have is customer support is one of those departments that sits in a variety of other departments usually sometimes they're part of marketing sometimes there are their own things sometimes they're part of live ops and we had to create personas for all the different C level people that we might have to explain this value to because if you if the person who has the checkbook is the CMO they're going to be thinking of marketing so we're going to say well here's what it does and it's really cool added bonus if you're a marketer because it's on Twitter a lot of people will see your users doing good things because if it's an email thing it's like person sending the email and person answering the email that's the only two people who will ever know about the interaction but if you've got these really cool people doing really cool stuff on Twitter it gets retweeted, it gets viewed it gets a positive exposure for example Google's got this hashtag G help where people can use and more and more people are turning to that because they can see that they're getting good answers quickly and very simply from Twitter so the trick for us was really the tricky bit was creating those personas for clients not for users however I'm very conscious that's something that we were fortunate enough to have that pool of users to talk to early on but I would argue that it will almost always be harder for you to get time from someone who's corporate than someone who's passionate about something because if you've got a music app you can find people who might be potential users but how many CMOs do you know who would give you an hour and a half of their time to be interviewed, you might know one might know two but do you know ten that was the challenge for us and I'm pretty sure it would be a major challenge for anyone who's not part of a global company that has that kind of reach and can talk to other people Z did the personas so again when I start saying stupid stuff but what we ended up doing is we ended up identifying the three main types of people that we want to talk to I think seven or eight that we identified as potentials and then from there what we did is we were lucky enough to have our previous relationships with people through other products that come to social so for example because we also offer this agent support solution we could say hey we know we're not using crowds and you might not be interested but could we talk to you and so it was kind of hard but we did manage to do that and we've got the proper personas like really well researched and documented for I think it's three or four people and yes but again we were building on something so it would be tougher and I'm not sure I'd have a good solution for you if you're starting without that kind of pre-existing pool of people you can talk to I'm going to give you my card that way you can email me I mean keep them coming we don't have a dashboard but we do make a lot of data available directly either through the API or through reports but what we do is we make recommendations or they come to us with a clear idea of what they want and then we say well based on what you think is useful here are the KPIs that we suggest because we provide that kind of because some people are going to say what's very important to me is the volume so they say we received 10,000 questions we answered 5,000 of them through crowds some people say no no what I want is first response time and then I want to break it down some people want c-sat some people don't care about c-sat so what we can do is we work out with people what they want to see how they want to do it and then we make it possible for them to get that information and sometimes it happens that we do not have we do not currently track that because some people have really creative ideas and usually Chris we make it possible to start tracking so we say if they say we've got this idea we don't know how to track it we can say well here's how we would do it and then we work together and then we implement and it's happened in the past where at one point people said well something that's very important to me I don't care how many questions they answer I want to know how much time they spend in that aren't actually doing stuff not just no idling so we had to say well okay some people came to us saying we do care about how many questions are being answered and how many questions we got so let's say we got 10,000 questions and we answered 5,000 but did people see the other 5,000 so we need to track what people actually see and we're like okay it makes sense it's cool stuff but nobody before you had ever asked for it so we had to work on that so that's how we as we go we kind of build the number of things that people can track and that allows people to be a bit more creative and it also kind of challenges us because we had to be able to demonstrate value in a variety of ways and what's your current like lead time lead to pleasure for clients I've got no idea we talked to a lot of people and a lot of companies what happens is when we first reach out this is cool and then there's like a long period we kind of keep in touch and send stuff but it's not that we're not trying to close and then usually it kind of goes like this and there's a number of companies where when people tell me hey we've closed this gaming company it makes perfect sense at some time they say out of the left field they'll go oh we signed that bank the big bank with all the buildings and you kind of go what? who? where? how? and that's so it is but it kind of depends on because there's also a strong word of mouth so people will just say and reach out but again then once you start that conversation the problem is the person who wants crowds is usually not the person who's got the checkbook so that's the so then do you meet at times like help these guys who are actually needing the product but can't sign a checkbook like the business case justification what we kind of what usually how it goes is we've got those personas that we talked about earlier and we say usually the CMA we want to see this and this and this and this oh your boss is a CXO well here's what we can do but once we've got that what we can do is we can say well here's talk to your boss and kind of tell us what they want to see and then if you don't know or you can't talk to them so that we can suggest and then we support them but we do pretty much every single time have to help them cross that line because at one point they're just going to be stuck most of the time and I think you hit a question I'm just wondering about your clients put a lot of trust in these experts and it's potentially their brand of risk if the experts are as expert as they could be is there an escalating process or something within the app there that raises particular problems up to the client yes so that's one of the questions that we get the most is people say what happens if a user goes rogue so there's two things that you need to consider the first one is experts answer using their own Twitter account and people on Twitter will never know no that's not what I mean is you choose who you let in some brands just say here's the link anyone can sign up and that's it and some companies have literally 9000 experts some companies say it's a club you have to prove yourself and you will be let in and they know and trust those people but in both cases what happens is they sign up with their own personal Twitter account and crowds is completely invisible to anyone on Twitter people on Twitter will never know that crowd exists so what they see is someone who found their question and using their own Twitter handle to answer them and if you want to be that rogue agent you can do that exact same thing with that crowd if you're a disgruntled user for whatever brand you can go on Twitter search for a couple of terms it won't be as good as crowds but you can do the exact same thing and because it's their account your brand is not at risk not more so than if some guy on Twitter goes on the rant and so that's the first thing and we've never had issues there the second thing is we did have that question we had one client in particular who was very very concerned about this and so what we did is we added a functionality that's called report so if you're a user you can see all the other answers by users and you can just go through it really quick and there's a button that says basically report to admin click, send something and then the admin can intervene but then again they can pretty much the only thing they can do is kick the person out of the program block their access and then take it up with the Twitter having said that we implemented it and it's been used I think three times and one of those times is me testing the feature live and that's across clients over two years so there's nothing major there and the last bit is the companies who do use CSAT it's kind of funny because they were majorly concerned about this and CSAT was higher on crowds than on any of the other platforms because if you look at let's say you've got a problem and let's assume you're computer literate your solutions right now would be you find the FAQ assuming you know what things are called and you don't just google why is the thingy flashing so if you can do that you've got the FAQ but it's really hard to find and so on or you can reach out to forums but a lot of people find forums intimidating and it takes some time and you find the right thing and it's consuming and so you've got all these other solutions or you're on Twitter and you kind of go how does this work? how do you do that? which takes exactly 5 seconds from the comfort of Twitter and you get an answer that's in your language that's immediate the language is natural and you do not have to step out of your comfort zone and if you've got any follow up questions you've got a real human being someone who's deeply knowledgeable about what they're talking about who's answering you and one thing that was really fun was that we talked to clients and we realised that in a lot of companies you've got employees asking questions to experts because the experts live and breathe those products and they know so much more than sometimes agents who might be very smart and very proactive but they've only been there 3 months and they don't know how that old version works so that comforts means that people are happy because of the system that I'm aware of so I don't follow every account all the time it's very rare for someone to go rogue and then basically just go like kicks and that's it I was more actually thinking more about the person asking the question in the first place because some of the time it's going to be quite annoyed customers as well and Twitter is again known for ransom and if they don't get the answer that they want is there a way to share the question I thought you meant to escalate the violation so yes and no so right now we are thinking about a right now what we do is experts will at mention support and support know who the experts are so they can then go and have a look one feature that we are that is near the top of a backlog is called escalate where what you do is the question comes into crowds and an expert has a look and says it's a valid question but it's not something that I as an expert can answer because there's a question about a specific account or something and we want them to be able to click and it goes to the right person in customer support and because we offer that customer support solution as well we can do crossover conversely another feature that's been requested by clients is a bump to agents to expert because sometimes agents get questions like someone will say where's the button for this you don't have to be an agent to answer that anyone can answer that but if your agent answers that question because you have to pay the agents and all that the cost per resolution will be four dollars or something if you send it to an expert the cost per resolution will be a quarter of that so it is a feature and sometimes also people say on the agent I quite frankly don't know so I can either go and poke the expert to get that expert to give me the solution and then post it again what can basically just say hey expert can you please help this because I'm out of my day but no so we don't have that feature yet it's not built in as a full-fledged feature it's something that we're looking at and right now the work around is the applications from users just that answer the question cool so how basically how do we calculate cost per resolution is that the is that how do we calculate cost per resolution when I say a resolution through crowds is no like a dollar something versus that much is that what you're asking so I mean I think this might be when I try not to use the word customer because usually the customer is someone who's into it so clients we're talking about the company that starts the crowds program and has experts is that how do we how do we balance value between them and the users user it does it does it's just because I need to make sure I understand the question to not repeat the the primary work that we just had where I misunderstood the users don't care about value their value they get something out of out of crowds but they don't get a monetary value what's the question what you mean then apart from growing up that we don't have motivation but one of those times there are some users that spend six to seven minutes daily that's so users do not do not consciously say well I'm worth this much we can if we want we can you've answered this many questions therefore no this is the value of what you've done but it's like asking if you go to the RSPCA how much is each volunteer worth there is a reward but the reward is depends it's on a per client basis some clients like like the big clients like Google can afford to fly people places and no they never pay because you're not a paid employee that's very important but they might say you're doing a cracking job here's the new pixel phone or if you're a video game company you might say oh we've got this new game here's the new game or this new expansion or this new this new that but it's a token of appreciation and that's because the users while they appreciate that token of appreciation it's just that it's not they're not being paid it's not a salary people don't make a living from being a user it's just it's a tool for people who want to do what they do and it makes it easier for them so we don't it's not something that has ever come up for us it's never been a problem if you ask me we could find a way of the what each person has created but it's not something that we've ever done because that's not how we look at it because after having talked to users that's not how they look at it and that relationship of potential rewards or tokens of appreciation is entirely between the client and the users we provide data if they want to see things and that but maybe after a section but yes we can talk about it I guess you can come from a slight value into some more measurable metrics I can actually think about how many clients actually use it and if they can use it and if they can use it as an expert so you can also measure what's your engagement ratio if the same people keep coming back and also if your expertise keeps growing so you can basically I'm just kidding right so you can help to translate value into something that we can actually measure and you can say that people find value if they use your product and what experts and clients oh so you mean how do we know if they're happy with the product and they assign value to your product as opposed to their actions yes so that's what we do is how active are people and how often because some people will have and some people will do this day in day out some people spend like an hour every single day some people know like two hours every second day some people just try it and then they'll never come back so we do track all that and we can say but if this person is doing this then it's that they find value in what we provide there's a bunch of features that we have also where we can track what people are doing so for example some users used to be very active on forums some of them are still active on forums at the same level but also do crowds and not active on forums anymore at all and only do crowds and some people just know there's a bit of a balance so that's where we can say we can demonstrate that this person offered with this value proposition and prefers this value proposition and by returning you prove that you're using it in a way where we provide value to you the user sorry I misunderstood the question originally