 So Ben Finn OBE – he's the first of our OBEs today, his story is he co-founded a company called Sibelius with his twin brother and that was when he was a teenager – he wrote this piece of software for musical notation. In the years that followed, Sibelius became the world standard that was used by artists like Sting, Ray Charles, symphony orchestras around the world. Professional and amateur musicians alike. Once he got to his 30s, Ben exited for $23 million or Ben and his brother. He was then awarded an OBE for his services to the music industry. He's got some experience of, like I said, building up a software company. I first saw him at an event for StartUp a while ago and he had this very good presentation about How to prioritise features when you're building a piece of software because you get all the stake holders and other constraints pulling your software in different directions and Ben gave a very good talk about how you prioritize and choose which features to build fitness and when to build them. So without further ado, I'm going to let him tell you exactly how he did that. Thank you. Hello, can you hear me? Yep, good. So, conventional wisdom says I should stand up and wander around in some, but I'm actually just going to be lazy and I'm going to sit here. So, I'm going to be talking about developing products, not necessarily desktop software like Sibelius, it could be any kind of software, it could be a website, it could be an app, anything like that. Whatever product you develop, you have to choose features. You have to decide what you're going to actually develop. If you do it right, you can make a lot of money. If you do it wrong, you don't make so much money. People very often do it wrong and don't realise they're doing so. So, in my company, Sibelius, once we've been going for a few years, I started introducing a very rigorous economic method of choosing features that decided what we were going to do in order to get us the maximum value out of our development. Development is extremely expensive, or can be extremely expensive. We have a small development team, but by being very careful about what we decided to do and what we didn't do, and by understanding what customers wanted and what they didn't want, we were able to produce a lot of very successful versions of Sibelius in a very short time. We did a major new version about every 18 months, which in each new version had a whole lot of big features and lots of little features. And we did these for Windows and for Mac and in multiple different language versions and all this kind of stuff. So, we were able to be enormously productive with a small development team by being very careful about it. Now, of course, there are always more ideas available than development time. We had a database of several thousand different feature ideas that had come in over the years, a lot of little tiny ones, but a lot of big good ones. So, we used this method in order to choose what we would do. As I mentioned, this applies to any kind of product. So, if we're talking about software, it could be desktop software, it could be web-based, it could be an app, anything like that. It might be something which generates revenues which you directly sell, like Sibelius, we would sell it for, you know, it might be £600 a goat professional, but you might be developing software which doesn't directly generate revenues, but it's generating users or usage or appeal or traffic, something like that. However, the same thing applies, whether you're trying to increase or revenues or increase some other parameter by choosing the features right so you can do it better. This methodology I'm going to be talking about also applies equally to other products, physical goods, services, because all of these things contain features. Everything has features. Now, what do I mean by a feature here? A feature is something you develop that customers want, that they want your software to do or want your software to have. So, the obvious thing you think of as a feature is some kind of functionality, something which does something. But that's not the only features we're talking about here. User interface can be very important, very important feature. People may not feel that they're paying money for user interface, but if you have a good user interface, it makes people find the software easier to use, it makes it look more attractive, so people do in fact effectively pay for it. Compatibility is another thing you might not quite think of as a feature, but supposedly developed through an app, let's say, for iOS, and it's quite successful, and then you decide you're going to convert it to work for Android as well. Now, that involves, of course, a huge amount of development, but it's also effectively a vast feature because releasing an iOS app for Android is going to multiply your revenues or your number of users by a huge amount, so that is a big feature. Another thing you might not think of is speed. If you've got some software where speed counts, then making it faster is a feature, and it's in fact potentially a marketable feature. This may not be quite so important now that computers are faster than they were, but if your competitor's software is slow and clunky and yours is fast and slick, then something people will pay money for. So, you've got a whole lot of features you might be able to do, but you've got limited time. How are you going to choose which ones you're going to do? Well, let's look first at how people normally go about it. How are features usually chosen? So, you have to start off with a long list and a load of ideas for features. Where do these ideas usually come from? They come in from customers. If you've already got customers, you will get some kind of feedback from them. If you don't have customers, there will be at least potential that the customer is out there who you can find out what we will tell you what they'd like software to do. Now, customers, and in fact all these other people I'm going to be talking about, are useful sources of ideas for features, but they're not good ways of finding out priorities. They're not necessarily good what's important and what isn't, and that's the crucial thing. In the case of customers, the trouble is the ones you hear from most may be highly unrepresentative of the market. Let's take, for example, Sibelius. The most demanding customers we have for Sibelius were often people with highly specialised requirements. So, for example, our software was used by all kinds of amateur musicians, schools, universities, hundreds of thousands of those, but there would be a small number, and we're talking maybe in the dozens here, of people like, say, music publishers producing very high-quality printed scores. Now, we designed Sibelius to be able to do that, but they had the most advanced stringent requirements of anyone. And so, they would bend our ear and they would shout very loud and say we wanted to be able to do XYZ features, some of which features no one else would want other than that. Now, that's not to say these publishers aren't important, and in some ways, much more important customers than your average customer, because they are influential. If they give you an endorsement, that counts for a lot. If they recommend your software to other people, that counts for a lot. But nonetheless, they're highly unrepresentative. So, if you only listen to people who shout a lot, or people who seem important, you'll get a very skewed idea of what you need to be developing. In fact, it's rather an inverse of how it should be, because the people who matter the most, who in fact, in our case, turned out to be the education market, turned out to be schools and universities and people like that, who we hadn't thought of at all as being important initially. They ended up being about two-thirds of our revenues. And you don't get teachers turning you up, or emailing you, telling you what they want, because they don't have time to do that. They're too busy teaching and that kind of thing. So, if you only listen to people who speak to you, who contact you, that gives you a very highly unrepresentative idea of what people actually want and where money is going to come from. The other thing I just mentioned briefly is that customers, when they suggest features, they're not good at solutions. And what I mean by that is, they know what kind of thing they'd like to do, what kind of problem they've got. But when they suggest a solution to it, when they say, I want a feature which does X, that's often actually not a very good idea because they're not experienced enough to think through what kind of features would fit into the software, what a good solution is. You're generally much better than that. So, suggestions that come in from customers, you often have to slightly ignore precisely what they're saying and try and think what they're trying to achieve and try and come up with your own solution to that. Right, the next source of feature ideas and potentially bad advice is your CEO, some of you might be CEOs, your CEO or indeed you. Now, the trouble is that these people think they know best and they're in a role to tell people what to do and they're used to being listened to. But again, they don't necessarily quite have their finger on the past or they have their finger on part of what's going on, but not all of it. Your CEO will very often have quite good knowledge of the market, hopefully very good knowledge of the market, but will not be an expert often at development so he or she won't know what is practical. The CTO, on the other hand, will have a very good idea of what's feasible technically but he'll be very much more interested in that than directly in the customers because he may not have much direct contact with customers. And you, whatever it is you do, or anyone else who might be in a position to make decisions about features always think they know best. Finally, the final frequent source of feature proposals is developers themselves. Now, often, particularly in small companies, developers are, to some extent, left to their own devices, for example, coming up with specs and that kind of thing. So they often get quite a big say in how features are implemented. Now, that's not to say that developers don't have very good ideas for features. I mean, all these people can have good ideas for features, but in terms of, again, of prioritising in terms of knowing what's important, developers, not surprisingly, are interested in programming things that are interesting to programme, interesting in developing things that seem cool. But that is not the same agenda as customers may have. Customers often want very mundane things to be developed. What they want is something boring, but that's what makes you money. Developers may not be interested in that. The other thing with developers is that it's not in their interest to shorten development time. Developers may well be very interested in doing a big, complex feature that will take years to develop. And when you say, sorry, we want something that does a third of that and is much less interesting, they're not so interested in doing that. Understandably. So, again, you have to take any suggestions for importance or prioritisation with a big pinch of salt. So, nonetheless, these people will come up, typically, with a load of feature suggestions. But how are you going to decide which of these you're actually going to do or which ones you're going to do first? First of various charts and graphs. So, here is a chart of potential features. Now, each blob is a feature. Can you see my mouse? Each blob is a feature. And I have to explain these axes. Things over to the rise are bigger features. So, these bigger blobs are big features. These small blobs are small features. And by big, I mean roughly the same as expensive. So, bigger features take longer to develop. So, that's the x-axis. And then going upwards, we've got more useful features. So, features that customers actually want more that benefit customers more are higher up. So, we've got all these different possible features. Which ones might you do? Well, I've just highlighted a couple which might often end up on a feature list. So, we've got here a big feature, a nice big feature, a big chunky feature. The developers want to do it. It's quite popular. The customers will quite like it. Big and chunky, very marketable. The CEO likes a sample of that one as well. Okay, that might often get hit. And let's look at this red one here. This one is not so big, but it's the most useful feature. This is the feature customers will want the most. Let's suppose you know that. So, that one, the customers want the most, might also often get the balance. So, those are released now. It's not completely clear whether those are the most important features. But those might well end up on your feature list. Yes? Yup, everyone agreed. Or at least they're very well seriously considering those two rare features. Okay, we'll come back to that one later. Okay. So, now, of course, people don't even have a graph like this. This took a huge amount of work to develop this graph. This is actual data, which I'll talk about later. They don't have a graph like this. All they've got is a list of things and a bit of a hunch as to which might be more useful, which might be more important. So, how do people actually go about selecting from their long list of features? Well, they use this method. This is the human gut. They use gut feel. Now, gut feel is what most people, most companies, even big companies, more or less rely very much on gut feel for selecting features. Gut feel, in my experience and analysing how this goes about, is a very unreliable and very poor way at selecting features. They're good at digestion, but we all know what comes out to be your gut, and that's what comes out when you use it to choose features that way. So, the reasons are, who's gut are we talking about? Who's doing the choosing here? You've got different people with different agendas here. Who actually gets the final say? Well, it depends. It might be a group of people, but again, you've got the CEO, you've got the CTO. They've both got quite a skewed view of things. You might have a product manager who hopefully has a better overview of both the technical side and the marketing side. But ultimately, someone makes a bit of a judgment call without doing it in a rigorous numerical way. What happens in practice is, people tend to choose features that excite them, that they are personally interested in, which the actual customs may or may not want so much. Or they pick features which will take a long time to develop, and they don't probably take that into account. So the features that end up being developed are not cost-effective very often. And this is actually a key thing that I think gut feel is very bad at, is balancing the benefit and the cost. Balancing how useful a feature is going to be to a customer with the cost, i.e. the number of days it's going to take to developers. So, how should you actually go about choosing features? Well, this is what I'm going to be telling you. This is what I'm going to be talking about. And I'm going to be going into immense detail, I'm afraid. Of course, I'm going to be telling you all about how we did this at Napsa Vegas. So I'll take an example. Suppose you're developing a word processor. Suppose you're a Microsoft about 30 years ago, developing a new version of Microsoft Word. And we've got a list of potential features. I'm going to suggest three features to you. One is a spell-checking feature. Now, this is an example of a big feature, let's say. Let's suppose it's a big feature. Many users want spell-checking a lot, but let's suppose it's hard to develop. So it's a big feature, but it's expensive. Next feature you might consider is a smaller feature. This is something to do footnotes at the bottom of a page. Now, a small number of users really want this. Let's suppose academics, they're publishing academic papers. They must have footnotes at the bottom of a page. But let's suppose they're only 10% of the market. And most people, people writing letters, that kind of thing, never ever want footnotes. So it's essential for a few people, the rest don't care at all. And let's suppose it's quite hard to develop, not as hard as spell-checking, but it's quite hard. Third feature on our list, we've got something a bit small and boring. So let's suppose page numbers go in the middle of the bottom of the page, but you want to be able to print double-sided, and you want numbers to be able to go on the left side or the right side, depending on whether it's the left or right-hand page. It's a slightly boring feature. Now, many people would actually use it, but they're not really that bothered about it. You might, maybe as many as half the people, might have to use it, but they don't care that much where the page numbers go. But let's suppose it's actually quite an easy feature to do. So this is your classic dilemma. We've got limited development time. Which of these is higher priority? Which of these do we do first? So we've got a big, difficult feature, a medium-sized but highly-specialised feature, and we've got a small, easy, but boring feature. How do we balance all of that? Now, fortunately, simple economics provides a solution. Anyone who knows anything about economics bear with me, this seems unbelievably basic, but people just don't do this. Benefit cost ratio. This is how almost everything in economics is worked out. There's a ratio. If you can work out the benefit of a feature, now, what that means typically is how much revenue you'll get from a feature. If only you could know how much revenue you'll get from a feature and the cost of developing it, well, you can do an estimate of that, that's a bit easier, then the benefit divided by the cost tells you how much, how good a use of development time that would be. That is effectively telling you how many pounds generated by the feature you get per pound spent on the feature. It's the same thing as return on investment. So each pound you spend, how many pounds are you going to get back? Now, this seems obvious, but to actually try and work out what this is for features is actually really very hard, because it's hard to predict the benefit of a feature, that it's hard to predict how much money people will actually pay you just for that feature. And people, as I say, very rarely try to do an analysis like this, but they should. So how do you actually come up with these numbers, the benefit and the cost? So in a bit more detail, this is how I think we should do it. So, now, we don't need to actually work out the actual revenues you'll get. We just need something that's a proxy to it, something that's proportional to it. So the more people who are going to use a feature, the more money you will get, basically. If people are going to use it, they will be prepared to pay you for it. Also, if people who use the feature really want the feature, they will be prepared to pay you more for it. So if you multiply those two together, those two numbers, that gives you something which is proportional to how much money you would expect that feature to make. Now, it's useful to separate these two factors of the benefits. It's useful to separate these out. So if you think about our academics who want to use footnotes. Now, let's suppose it's 10% of users who want footnotes. But if it's essential to these academics, let's suppose they give it a rating of 10 out of 10. You see how important it is to separate those two. These academics might be shouting very loudly for a feature. And similarly, our publishers that survey this, they will be shouting extremely loudly for some feature. They say, we must have it, we must have it. It's essential. And they're trying to make you think it's essential. But they're not telling you how few they are in number, however few they are. So you have to split them out. Now, the cost is much easier to estimate. And a good proxy is simply the number of days taken to develop the software. And that is something your project manager should be working out anyway or should be working out in advance anyway. Now, of course, there are other things involved like testing and bug fixing, but you can assume, it's reasonable to assume, those are proportional to the number of days taken to develop them. Now, or just a little, anyone who knows about economics, I'll just mention a little detail here. I haven't, I'm not expressing these in actual pounds. We're doing these in kind of arbitrary units. There are certain advantages to doing it in pounds and certain cases trying to work out actually how much real money we're talking about here and how much real cost we're talking about here in pounds. Sometimes that's a better thing to do, but this will do for now. I'll also just mention that I've put a little approximation sign here. This is quite a good approximation of what's going on, but then there are other adjustments you may want to make, which I'll talk about later, going into a mental detail, because this doesn't show the whole picture of what influence is, how much money you will. So I'm going to get very boring now. We're going to work through this example of three features. We can actually put numbers on this. So if you just bear with me while I put numbers. So spell checking, we said most people want to use it. Let's say 75% of people want to use it, and let's suppose they want it quite a lot. They would give it a score of 8 out of 10, let's say. But it's a big feature. Let's say it's going to take 100 days to develop. So that's a big feature which lots of people want a lot. Footnotes. Let's say only 10% of users actually want it, but they really, really want it. They want it 10 out of 10 is what score they would give it. And that's going to take a medium amount of time, 50 days. And then finally we've got our left and right page numbers. Let's say nearly half of people would actually use it. But they're really not that bothered. They don't give it a score of 2 out of 10 on average. I mean, some of them might really want it. Some of them might hardly want it. But it's a really easy feature. It would only take, let's say, 5 days to develop it. So then let's do our maths on it. So we calculate the benefit and the cost. That times that is the benefit divided by the cost is that. And that gives you your priority. So we can see from these numbers that our highest priority feature is left and right page numbers, followed by spell checking, followed by doing footnotes. Now, this is called cost-benefit analysis in Wales, what this little calculation we've done here. Now, this tells you something which was not at all obvious from the previous slide or two slides back, that actually every pound you spend developing page numbers is eight times more effective than every pound or every day you spend on footnotes. It's really much, much better use of time to do page numbers and to do footnotes. You can't tell that unless you actually put numbers on it and do the calculation. Now, it doesn't mean you'll never do footnotes. It means that you should do the page numbers first, followed by spell checking, followed by the footnotes. So let's suppose you've got about 150 days of development time, then you might be able to do all three. You start doing your page numbers, that's done in five days or so. You then next you do your spell checking and that's another 100 days. But suppose that overruns, it turns into 130 days or something, then at least you'll get that finish and you can drop footnotes and that will have given you the maximum revenue for your development time. If you did them in a different order, if you did say footnotes first, then you started spell checking and then you just ran out of time and you haven't finished spell checking so you'd have to drop that feature. So then you've got a version which is missing a major feature and you'll only get a third or something of the revenue. So that's a bit of a disaster. Okay. Now that's it in theory. I'm going to go into more detail now. I'm sorry. That's in theory what you do. How do you actually do this? How do you actually do this? And the difficulty is how you find out for a very large number of features how many customers want them and how much they want them. That is not an easy thing to do. I'm going to show you the full process we did at Sabanus. Now don't feel you have to do this. You can do a much simpler version of it. You can just use a bit of gut feel to come up with some numbers but at least plug them into the formula and then you'll get so much better results. So you don't have to do everything I'm going to show you but a lot of it might be useful. So this is another kind of a process we use. You start by coming up with a list of all potential features. You then get a load of feedback and you get a load of information. You get a load of feedback and information and that will tell you how many people want them and how much they want them and then you do a cost-benefit analysis like I've just shown you. So we'll go into each of these. So how do you come up with all your features? Well the key thing is having a database. Now if you've got a little product just a list will do like a list in Excel or a list in Word but we had a database and anytime anyone thinks of anything that could go with a product you get an email, you have an idea you see a competitor's product anytime any idea comes along it goes in your database. Even very unlikely ones. It doesn't matter how unlikely they all go in there. So they've got a repository of everything you might ever possibly do. Then before you do a new version or before you do a new product you talk to a load of people. First of all you talk to internal sales and marketing people. Sales and marketing people talk to customers all the time they have their finger on the part of what people want. So ask them what are people saying they want and how much are they saying they want it and what kinds of people are these. Development you talk to them. They will obviously come up with all kinds of good ideas of their own. They may not have their finger on the part of what customers want so much but they will have good ideas. Technical support, if you have technical support like sales and marketing people they are speaking to customers all the time. Of course they're particularly good on telling you bugs that need to be fixed but also on things that people are trying to do and finding it difficult to do and they're having to give work around. So that's enormously valuable. For people what features they think people want. You can also consult experts if you have customers who are particularly experts in the field you can ask them. You should definitely look at all competing products and see and list all their features as candidates. It doesn't mean you have to do them but have them in the database. Finally you can do a whole load of brainstorming in-house with customers trying to come up with novel ideas that no one's had before. Those can become USPs unique selling points for your products. So just a bit more about just what kind of features you're going to include here. Lots of detail but this is quite useful stuff. Now if you're going to do a big feature of course big features can come in different sizes. Typically if you've got some big huge thing you can do a big fancy version you can do a small simple version maybe a medium version. So for a big feature you should consider having at least two different sizes of it and maybe list them separately in the database or have some description of what the different sizes might be. At the other end of the scale tiny features on their own are not very marketable in fact not marketable at all. Some little tweak isn't marketable but if you group a bunch of tiny features together and make something which you've been described and people can understand you can put it in a brochure on the website and people can understand. So it's useful to give tiny features together. It's also useful from a cost perspective not to try and work out whether this is going to take one hour or two but to have a bunch of them say we'll spend a week on this bunch of little features. Changes to existing features are obviously hugely important and very marketable. If you make a big change to an existing feature you can even give it a name and you can call it a new feature. I've already mentioned user interface, treat that as a feature compatibility, treat that as a feature maybe even speed. Now this list can end up huge like as I say our list was thousands long so you need to exclude that to make it practical. I think bug fixes should treat as a separate thing so don't try and do a cost benefit analysis on those. Similarly other internal code work like refactoring and also you can eliminate early on features which are clearly unfeasible and when I say clearly unfeasible ones that have a very low demand and a very high cost. If it's clear they're never going to end up anywhere near the top of the prioritised list you can drop those early on in the future. Now you've got a big, big, long list what are you going to do with it? You need to get feedback on this list this is where you start finding out how many people want to have it so for each feature you're trying to find out who to use it, how much they want it people will also give you feedback on any improvements or other suggestions that pop into the head when they read your about this feature idea and you also need to find out obviously how much time at least initially roughly how much time each of these features is going to take to develop. That's the bottom of the fraction that's going to cost. Who do you get this feedback from? From all your internal staff you've spoke to before you can give them your long list and ask them those questions which of these do they think different customers would like so I think at this point you can start spilling the beans slightly to a few external customers like if you've got beta testers you can ask them, you can survey them you can maybe have a focus group of beta testers for all of your users where you describe the features then maybe have a mock up for big features you might do a slide mock up like a PowerPoint or something say what if we did something like this would you pay for it, would you like it how much would you use it if you've already got products out there and if you're selling in a traditional way like maybe software, box software like Sibelius your distributors might have some idea about what their customers want so in addition to that a final load of information you can get but ideally you should already have is information on your customer breakdown how many different customers do you have of different types which features of the software if you've already got software which features do they currently use how much do they use it now you can get this kind of information if your software is registered if you have a website which people register on you can ask them profiling questions and that will give you some idea of the breakdown of your customer base so this gives you a huge amount of information this just briefly this is one of many different kinds of information we've had about Sibelius users for example this was an analysis of what different Sibelius users use the software for and we found this by doing a survey so for example if you're doing a feature which has something to do with homework which would let you submit homework or something like that it's very useful to know that 29% of Sibelius users use Sibelius for homework that's really valuable information so on to with all this feedback on all these potential features how much different people want them you then do your cost benefit analysis and that is just a very much wrong version of that little table I showed you before so you go into a room with some senior staff maybe these senior staff with Excel for several days probably ideally certainly quite a few hours you probably need a sales person that's being an expert on the market you need your CTO as an expert on the elements and they try and fill in these numbers so for each feature as before what percentage of customers would use it, how much said what is it in the number of development days now these numbers you're not guessing anymore because you've actually got hard data here you've got survey results you've got to get almost complete agreement on what these numbers are for example as I say if you're doing some feature which is relevant to homework you can just plug in straight away that number whatever it was 29% and plug that straight in how much they want it, well you should have a fair idea of that having got all this feedback so you plug all these numbers in and then you can calculate the priority number for the feature now I will probably go rapidly over this next bit there are various adjustments you may want to make to that I'll just mention for example some features are intrinsically more marketable than others some features are just flashier and more exciting and so they will probably generate more revenue regardless of merit and conversely little tweaks to existing features may actually be very useful to customers but they're really quite hard to sell people won't understand why they want them until they've got them a classic case of that I think is new versions of operating systems like why do you want them the next version of Windows it's impossible for most people to understand it's got thousands of little tiny changes they're used to the odd one why do they want these changes it's very hard to explain and this is, I don't know if anyone remembers this Windows Vista it's big USP it's big selling point which was this view of the desktop where everything would go all 3D no one ever used it but I think actually it did make sense it was highly gimmicky but it made sense to include it because it was one tangible feature that people could identify and thought that looks cool and it probably made some people buy it even though they never actually used it some other adjustments you can make which I won't talk about so what do you get after all of this you end up with a feature list which is sorted by priority it's a very long feature list of course because it includes all kinds of things that you'll probably not get around to but in practice you can truncate that you can chop off the feature list once you've used up all the developer days all the days you've got assigned for the development you should expect to develop in the order that features are listed on the list at least approximately although pragmatic concerns mean that that may not make sense just because of what different people are doing who's on holiday that kind of thing things can be shifted around to some extent and of course you shouldn't check the numbers that come out completely literally there's obviously a margin of error but it's not that big a margin of error in fact if you do it properly and I will show you that in a minute I'll show you it right now how well does this all work how well does this procedure work well after one of our versions we did a very thorough paste of mortem on it we did a survey of all our users and we asked them the questions we were trying to guess at we asked them this new feature we added how much did it make you want to buy this update want to buy this new version how much influence did it have how much do you use this feature and those questions that we didn't know before and we compared that with our predictions which we made like maybe a year before and this is how it came out this is the real data at the end so these are our top ten features now ignore the first one I'll talk about that if you have a look we've got we've got nine features and seven of them we predicted with extremely close accuracy thanks to this process how much people would want them we had two mistakes where we grossly underestimated how popular the features would be but all these other ones we got it pretty much bang on now this first feature is an interesting case this is one of these big huge features which we started out on and then we realised partway through the development it was going to go on forever so we truncated it we did a rather smaller version of it and recalculated the priorities so we predicted a big, very popular feature it had ended up being a somewhat smaller popular feature but the cost shrunk as well I haven't shown the cost here so it does make sense if it looks like your cost prediction is completely wrong during development so redo the calculation redo the estimate and then be prepared to either truncate the feature should make it smaller or to drop it entirely if it should be disappearing way off the bottom of the priority list so just to spell out what this tells you if you can predict accurately how much people will want features and you can if you do it properly then you know you won't be wasting development sign on features which people won't actually want ok, so I haven't quite finished, we've got a couple more charts to go this process is all such a big palaver surely surely it's all too much hassle surely you can just rely on your gut back on your gut well let's see go back to this, remember this now these blocks here this was in fact these were the shortlisted potential features I think in that same version I talked about so this is in fact real data so we've got here the predicted cost of each of these potential features along here remember we've got the predicted benefit so more useful more beneficial features are up here bigger features are over here and we actually express this all in pounds now let's look what these ratios are so this big feature which looked like it was quite plausible came out 2.7 so for every pound spent on development we'd expect to get £2.70 back this feature which is the most useful feature is the highest one the most useful feature for every pound spent you'd expect to get £9 back this is just dividing that number by that number sorry that number by that number now are those good features? people thought those might end up on their feature list let's have a look at these ones 152 that feature is more than 50 times a better use of time look at it that little feature there is almost as beneficial to the customer as that huge feature there they care as much about that feature as they do about that feature now because this is big and it's huge about the development and it probably sounds exciting to the developers to the CEO and so on you'd be very strongly inclined to pick that feature you could have done 50 features equally good instead so picking features which are over here is a bad idea the best features tend to be up in this segment up in the most useful least cost those who are the best features are so even this one the highest priority features so even this feature here which is the most useful feature of all is actually a very low priority feature as I say this is all real data I'll just give you an example of a little feature like this we added in one version in this one a little feature where if you selected some music you could copy it, control C and you could paste it into say Microsoft Word let's suppose you're writing some homework or something, writing a report writing an essay or something you want a little bit of music in you could just do control C and control V to copy and paste now a little trivial thing of course you could get music out of Sibelius before that you could save a graphics file and then load it into Word so it's a couple more steps so we added a little feature where you do copy and paste now this only took a day or two to develop that it turned out was a completely massive feature people absolutely loved it it was a really big deal we didn't predict that so that was a feature like this which people absolutely loved and we just assumed it's so trivial it doesn't matter, it doesn't matter if we don't do it it turned out to be as big as lots of much bigger features so clearly now you can build your product entirely with features from over here because you're going to run out of them but it will spread across to these bigger features which have a lower benefit cost ratio which have a lower numbers here but none the less you'll get a very high value result how much better is it in fact than picking the features by gut feel well again I crunched the numbers and all this and I found that the prioritised feature is you end up with on average three times the overall effectiveness that is three times the revenue than if you just pick features at random now think if you're using gut feel you're pretty much picking features at random in fact I think probably most people are doing worse than random using gut feel because you tend to go for big features you tend to go for highly beneficial features and not balancing the cost of the two together now as I mentioned at the start if you're not working on software that generates revenues then you may think this doesn't matter to you so much but the increase in effectiveness is equally high for other things like for example the usage of the software, the number of users the amount of appeal all of these will benefit commensurably if you use a process like this and again as I mentioned earlier I've described a very elaborate process that took us quite a few weeks to do the full process but it was an extremely good use of our time even if you don't want to do that at least do some mini version of it at least put some kind of numbers on how many customers want to feature how much they want it how much it will cost and actually do the sum it doesn't only take you a few minutes but the results are often extremely surprising so just to summarise I don't want to go through this but just to cover the points I've made don't rely on the gut feel make sure you get a feature list make sure you get feedback and actually do a cost benefit analysis thank you thank you very much Ben it's always good to hear from someone who's done it before and I think everyone today has done it before in their own way any questions from anybody? a lot of what I've heard today reminds me of lean startup do you read anything about lean startup lean startup before? I'm just interested in it no, we've probably predated it thank you that was really brilliant I think it's amazing how you can sort of overlook something that's sort of so simple and so beneficial I have one the question that I guess with this approach you're kind of following the majority of the money for the majority of the customers the majority of the time in a competitive environment you might end up sort of copying but never quite being as good as a competitive software and perhaps so if you're a market leader that's maybe the position you want to be do you think there are other strategic concerns like maybe you want to make that bit of software for the publishers that really does what they want but then you're kind of off and you occupy a niche in that market do you think there are some other strategic things to think about as well? Absolutely, actually that's one of the slides that I slightly skipped over is one of the adjustments and one of the big adjustments is desirable to make in the cost-benefit analysis is to take account sorry, yeah can you do that on? Is to take account of USPs, unique selling points and gaps so for example if you've got a product or by a gap I mean something that a competitor's got that you haven't then I think you should certainly put extra benefits, boost the score or filling a gap. Conversely if you come up with a new feature that may not be that big but is at least unique then that's a unique selling point and so for people who want that thing they're definitely going to go to you and not to your