 I'm Claire Parker, CEO,olwgrorunolwr yn ymdweithwyr cymaint. I've been doing product management now for good three years I think. And I've been at the FT for about five and a half years. So a little while and I've done a lot of different things at the FT. But I want to talk about how we develop products at the FT. So I'm going to first of all start talking about how we measure as part of our product development. That's quite a big piece of the talk. Ac rydyn ni'n gweithio bod ni'n cwrsio i fit yn dda o gwnaeth y broses yn y gallwn yma Ac rydyn ni'n gweithio bod ni'n gweithio bod ni wedi'n gweithio'r broses ac drwy'r colli yn y broses ar y gweithio'r cwrsio'n ymddangos ac fod y broses yn mirwm ag y non-n cleverau ar i Japan ac rydyn ni'n gweithio bod ni wedi gweithio'r cwrsio Ac mae'n gweithio ar yFTM, rydyn ni'n gweithio ar y FTM Felly fod yw'r FFD a bod yna'r ffordd o'r FFD. Mynd yw'r mord. Mae'n gweithio'n gweithio. Mae'n ffordd o'r dda i'r ddweud o'r dda o'r dda o'r dda o'r dda. Ond weithio'n ddweud o'r 930,000 ddod ar ddweud o'r ddweud. 730,000 o'r ddweud o'r digidol. Felly mae'n gweithio'n ddweud o'r digidol. Mae'n cyfrofiol o'r ddweud, mae'n ddweud o'r ddweud o ddweud o'r digidol. Our Daily Readership is around 1.9 million. We operate with very strict payable and we are basically a subscription-based digital company. It's obvious that as a product manager, measurement is important, metrics are important. What isn't obvious is sometimes what are the right metrics and how do we pick the right metrics to ensure they're given value to us as product managers in our job. Mae'r gwerth hynny'n meddwl. Metrach efo'n aeth yn proxies. Mae'r cyllid iawn yn mynd i fy ngysyllt hŷr, ond byddai'n meddwl yng ngllianiaeth ar ddeudol. Felly, warehouse ein leddwhaeth, a'r meddwl efo'r meddwl efo'r meddwl yn llwy. Mae rhai 30 oes yr yma, mwy'n gwneud yn llwyth bwysig i ddechrau o'r meddwl yno. Byddwn y drefydol yn meddwl i bai mas. Mae ymellodol y gallwn ei gweithio i gyd yn ddau'r content. felly mae'r metrwch yn eich bod yn fawr, efallai yn ystod yn ei ddweud, mae'n meddwl. Mae'n meddwl. Yn ymhellfa ymhellio'r metrwch yn ein cyfrwg yn fawr yn ystod ystod ystod ystod, mae'n meddwl â'r cyfrwg mewn cyfrwg mewn cyfrwg mewn cyfrwg. Ond, felly mae'n meddwl yw gan myfyd ac mae'n meddwl ei fod yn fawr, ond mae'n meddwl, mae'n meddwl gan ychydig ar y cerdd, mae'n meddwl ymhellio'r cyfrwg mewn cyfrwg. The second pitfall with metrics is that there are many truths, so at the FT for example we are trying to work out if people are actually engaged with us, but that is such a buzz word. You could measure engagement by looking at the time spent that readers take when they're reading an article. We could look at our Twitter followers, we could look at our Facebook lights, we could look at our app downloads. There are lots and lots of things that we could look at. So what out of all of the things is the most valuable, that's what you also need to find, because if you pick the wrong one then you're not really actually learning as you go. And then the third pitfall is that there's just an overwhelming sense of so much data. So we have loads of qualitative data and loads of quantitative data. We have built our own data platform at the Financial Times so we do our own data tracking on our website. We pump it into our own platform, it has a nice API and we can literally find out anything. We also talk to customers really regularly so we try and get them in the building, try and understand what they like and also test new ideas with them. So if you have an overwhelming sense of too much information, if there's too many data points, what that can actually lead to is a lack of focus. So for us what we try and do is we call it our North Star so it's our kind of like guiding metric. We have found a single metric that we use. It's simple to understand for us, it wasn't simple to explain it but I've given it a shot. And what we do is consistently prove the correlation to our real business goal and I'm going to explain our North metric now and how it works. So a bit of background around news industry, so historically it was all about digital advertising and print advertising. People then stopped buying newspapers so print advertising pretty much fell off a cliff. Digital advertising, again Facebook, Google, big players kind of came in and they've taken a lot of that chunk of the market. So we at the FT kind of thought to ourselves how do we have a sustainable business in these kind of environments. So we switch from an advertising model to a subscription model but it's really hard to do that. So for example 50% of digital news readers actually come and read content via social media and two thirds of those people do not actually remember the source of the news that they read which is quite scary when we're trying to ask someone to pay for the news because it is really difficult to acquire someone. We typically use the term it takes five times the effort to acquire a new reader than it does to keep an existing reader if you look at cost and time taken. So in this kind of environment we've made the switch to a paid publication but it's a very very hard thing to do. So this is why we used data, really our data intelligence on customer behaviour to help us answer the question how do we best optimise purchase so people actually buy our subscriptions and then how do we best keep people happy with their subscriptions so they're more likely to renew. So how do you kind of get more people coming into the funnel and less people dropping out. So for us we looked at subscription and we felt that it was all about habit. So if you've built up an FT routine so you keep coming to the site you're using it you're getting value you then build up a habit. And then if we can be clever and prompt you to come back in non annoying ways that are useful you will continue to come back so your routine builds. So for us that is the focus of our business goal so our business goal is to acquire people and to keep them but we feel that habit is how we are going to do that. So we looked at data and usage so how many people we use in our product and what we were able to do was positively correlate it to revenue via cancellation rate. So the more people used our product the less likely they were to cancel and if they were an anonymous user so someone who hasn't got a subscription the more likely they were to actually come and kind of like read our stuff the more likely they were to convert to become in a subscriber. And that was the case for both B to C and B to B. So the financial times you're able to buy a single subscription for yourself which is our B to C model but we also have a very strong B to B model so typically how we sell it is that you could have an army of people in your organisation who know all the stuff about the financial times sorry all the stuff that's happening in the world by expert journalism at the FT. So if you've got account managers going out to clients and if they're going out to those clients saying well I know exactly what's happening in your industry I know exactly what's happening with you and I know exactly what's happening with your competitors and I've got that from a quality news source that is something that they pay to have. So what we did was create this single engagement score and this is based on the recency frequency and volume. So recency over a time period is how long has it been since you've actually come to us as a site as a product and that might be on our app or that might be on our responsive site. During that time period we also look at the frequency of how often you come and then when you do come to the site we look at how much you've read. So how many articles did you read per visit. So for us that is the three components of habit and what we did was create a very complicated equation by our data science team and what we found was that for us there's a tipping point because not everyone is a digital news junkie so not everyone is going to love reading the FT come every day, read loads of articles every visit and come very frequently during the day. And we came up with a score of 18.2 and I'm not a data scientist so I can't tell you how we came up with that score but I promise it's real and we did come up with that score with accurate data and what we found is if we can get people over 18.2 as a subscriber they are more likely to renew. And if we have an anonymous reader and we can calculate how often they come in during one of our trials for example you can buy an FT subscription for £1 and try it for a month. If we can look at how you use a product in your trial and we can safely say you get to a certain level of engagement we are pretty confident that you will actually then at the end of your trial buy a subscription. And from a product perspective what this enables us to do is if we've got someone who is completely in love with the FT really is getting loads of value we always ask the question should we focus our product development on those people or should we focus our effort down here on people who are really unengaged or should we try and get more people in the middle and push them over that tipping point. So I'm going to give a few examples now of how we use that metric but I just want to reiterate a couple of the key points. So for us at the FT we are a subscription business therefore we need to be able to acquire people and retain them. We know that if you are engaged in using our product as an anonymous user you are more likely to acquire and as a subscriber you are more likely to retain. We broke that down by looking at the recency, frequency and volume of our readers and kind of calculated them a score so then we could work out how likely they were with data to cancel or convert. At this point I'm going to pause and just open a question out to the room. Is there anything that you'd like me to clarify on that at all? Bear in mind I'm not a data scientist but I'll try. So our product vision for FT.com so this is our main site is to help readers make better decisions to advance their career or business because we are predominantly a very professional read and we do that by providing them the most relevant information but we do not obscure the FT view so we give them the facts but we also give them the editorial views from FT journalists and what we try and do is build a product that saves them time so there's so much news out there and there's so much news on our own site how do we get people to the content they care about most or the content they need. So one of the things we've done is we've built my FT. What it is is kind of like a personalisation tool so we have lots of topics so for example Brexit there's a lot we're writing about that at the moment. Technology, China etc so we've basically got lots and lots of topics that we write about and what we enable people to do is follow a topic that is of interest to them and this is something that we started with a real MVP back in I think 2016 and what we've done is we were able to look at what we did with my FT and found that if a user is following a topic and receiving alerts on that topic by email when something new is published or able to go to the site go to my FT and really find quickly their topic and their articles about that topic we found that that feature helped increase engagement by 86% compared to people who were not my FT users of a product. So what that enabled us to do was spend a further three years developing this as a product feature. So we had a team of five to seven people building this feature for year after year after year and optimising it and that's kind of how it's grown now and for us we were able to make that product decision that that was worthy investment because we saw such a big uplift in engagement and because we know that engagement ties to cancellation ties to conversion ties to revenue we know that it's worth doing that product investment. So another example was speed. So before we... when did we launch my... I think 2017 we launched our new website FT.com and it's a lot more responsive but when we were in our beta phase we were allowing subscribers to opt into our beta to try it out before we launched the site and there were much less content on it and much less features so it wasn't a really... it was kind of like a bare bones experience I guess but we wanted to get customer feedback we wanted to see what people were doing so we kind of invited people on and explained it was beta. What we found is that 5% of our loyal customers became immediately more engaged and we were like okay so why is this the case? So we built a hypothesis that the reason they were becoming a lot more engaged is because they were really engaged anyway so they knew the FT, they were literally coming just to read content but because our site had less stuff on it, being polite it was a lot quicker so what we actually did was a test so we purposely slowed our site down so we had the control which was the fastest site Variant A was one second slower Variant B was two seconds slower and Variant C was three seconds slower and what we found is over seven days there wasn't much... there was about a 5% drop and there wasn't much between this but a 7% drop here and after 28 days of people using our strategically made slow site we could really see significant drops in engagement so then what we were able to do in product and tech was effectively again because engagement and I'm sorry to repeat myself links to retention which links to revenue we were able to basically to our board members put a pound sign don't know why we've got dollars on there as well for a second so what this enabled us to do was argue to our stakeholders that we need to invest in the technology of our site so we spent a lot of time building product features making the journalism a lot better on our site but we also had an entire team of developers focused purely on infrastructure and speed and as a result we won you know we were... when we launched the fastest compared to some really good websites so Guardian, New York Times etc and then we also won the best website in 2017 when we launched and we also won best use of technology and again we were able to be given the space in product and tech to focus on speed because from a data perspective we were able to prove that it was worthwhile doing so and then one more example so we had a responsive web app and we were going into the iOS store and what we did was look at the medium engagement of all of the different readers before the app launched so the people who won our old app and what we did was when we actually launched was able to measure their engagement so we found the people who were using our old app were way more engaged when they used the iOS app we found people who had never used our app before that took out the iOS app were more engaged and we were able to compare that to non-app users so again that signified to us that going back into the iOS store was useful and was beneficial to us and it made sense for us to continue investing in our app and lastly we also did this again as a metric for when we looked at now we have an iOS app how can we improve push notifications so we were able to look at the control group we put a group which we were receiving around 15 and a group which was receiving around 30 and we were able to measure their medium RFP so a medium engagement score so we were able to tailor the amount of push notifications that the newsroom or that we would automatically send again by looking at engagement so for us at the FT measuring things is really really important what we try and do is find a single metric to focus on one that the whole business can understand and get behind and we consistently ensure that it actually correlates to our goal so our goal is driving keeping our subscribers this is business goal keeping our subscribers by renewal and acquiring more subscribers by anonymous traffic we also try to pick a metric that helps us focus on the most important thing there are many many things we can do and for us we know that engagement has that dual effect it also gives us one single version of the truth so for example our marketing teams if they target low-engage users if we're doing something on the product side that's also targeting low-engage users we know that we're working with the same users and it enables us to consolidate effort conversation so we've been able to prioritise our product and development work based on this score as we've seen with the kind of examples but what we do do is continue to challenge it so we always ask ourselves have users actually gained value have their numbers gone up as expected and is our north star correlation valid and accurate so this is a algorithm and a calculation that we continuously do rerun just to ensure that we are still making sense for us as a business and so far that is the case so that's our kind of approach to measuring but I wanted to talk about our product approach in general so we always start with business outcomes so we are quite lucky in that when we look at, when our board ask product and technology to do things in the year what they do is ask us to move business outcomes so that would be engagement, acquisition revenue, quality or making things more efficient so that's kind of the level of where we start we don't tend to start with can we have feature requests can you do this feature, can you do this feature we always bring it back to our board level outcomes that we set and our investment board is what we call them they kind of oversee the technology and products spend and they all are aware of those outcomes and they are across functional groups so that's newsroom, B2C, B2B advertising, probably forgot someone, data for example so they are all aware of what we are trying to do as a technology and product team so what that means is that we have outcomes and we have a strong product vision setting the clear direction for the teams and when we use measurements we always do it to learn so when we, you know, with the examples I gave with myFT we learnt that it was a useful feature so we go back to the investment board and say we need more funding to develop myFT so we can continue the process and we can do that at that level which is quite like a higher level funding discussion or we can do it more in the detail if we are building product features so like push notifications for example and what we also do is try and work lean to deliver value quickly so instead of building features that are perfect what we will always try and do is look at the minimal viable product so what is the thing that we can test straight away because we do a learn, build, measure, learn cycle so if we are going to develop a new product feature because we think it's going to improve engagement we will look at the data we have at the moment we will come up with a hypothesis we will get some customer data in there we will build something and we will try and make that as small as possible we will then release it we will measure it and then we will learn again and then iterate on it and I will talk a bit more about this if you don't follow this as a product approach but for us it is something that again we have been able to persuade the newsroom B to C, B to B that this is a good way of working so we are not striving for perfection we are striving to try and do things quickly because stakeholders want so many things so when they say what do you want they say we want this and we go great that's fantastic but what outcome are you trying to achieve what problem are you trying to solve and then how can we build something very quickly to test that hypothesis that actually solves that problem that actually meets that outcome so in April this year I joined a project which was not following any of the principles really that I've just talked about at the FT the FT was acquired by Nikkei in 2015 and they are a huge news publisher in Japan so they are massive in Japan not expecting anyone to be able to read that and they have a global business publication that is in English language and it's called the Nikkei Asian Review so the idea of Nikkei Asian Review is that publications like The Financial Times The New York Times, The Guardian they were absolutely right about what's happening in Asia but this newsroom is based in Asia itself it's got correspondence in Vietnam in Philippines everywhere so they can give you a real Asia insight and expertise that even at the FT we struggle to sometimes get just the way Asia is and we formed a partnership so we created what we called Synergy Projects and the aim was to share the FT learning so we've learnt a lot of things in our newsroom digital first we've learnt a lot of things in our marketing teams both B2C and B2B and we've been doing product and technology a lot longer as well so we wanted to share what we had learnt over our 10-15 years of becoming a digital first news publisher to accelerate the digital change across Nikkei Asian Review they were still quite print focused and that was kind of in the newsroom it was true in marketing it was true on their customer facing products so we built them a new website which we are continuing to improve we have improved their newsroom toolings and next year we'll also be looking at their marketing systems too but a product approach was not being followed at all so it was like trying to push a big boulder up a hill so back to pitfalls the first one was that we had no real clear direction so we were absolutely delivering things but they were big big chunks of work so it was like here is a brand new website and there wasn't a product vision for our website so we were literally looked at this grey version of their old website and kind of did a copy paste job but made it blue we improved certain parts of it but it was very much there wasn't really like a customer focused vision of what we were trying to achieve we built a CMS this summer we did a very similar thing we used some definite improvements but a lot of areas we kind of just did a copy paste because we just had no time we were rushing to deadlines and that's because we had abandoned Aline principles really pretty picture but not pretty working in this way we were working in of Waterford I joined in April when we launched the site at the end of the month so I joined at the start of the month and it was like ok we need to get a website out and I was like this is crazy and then we then was like ok now you need to get a CMS out and the CMS has to go four weeks after the website launch and I was like well that's not possible but that was kind of the mentality and that really wasn't a fun place to be and then it was a long distance relationship I've never been in one and I'm now happy I'm not because we are all the way over on the left and they were all the way over on the right and that is really really difficult at the moment with time difference they are nine hours ahead so our realistic crossover time is between eight and ten am so you've got like two hours and all my stakeholders are over there and I'm here with my team and in the summer it gets a little bit better we have a luxury of eight to eleven but that's not perfect and I'm going to talk about some of the challenges in a minute so how did we basically go in and say right you need to put in a product approach when you don't have things and data was exactly one of the things we didn't really have so we had some tracking on our website but it wasn't really properly implemented so this was the kind of situation that I was thrown into in April so what we desperately needed was a common purpose and we didn't have time to do that in April when I joined because it was very much get the website live and then we had to build a CMS casually as you do very quickly but in a trip in May I think, yep we kind of used it as a reset session so I was like what are we actually all here trying to do and we did a session where we had people from Japan people from UK we had people from NICA we had people from the FT but also we had from then which was quite interesting newsroom, commercial and product because even in their organisation I don't know if you have this but you have silos that appear across the business and that was exactly this kind of same thing there so we created this vision statement which was an occasion review provides original quality insight that again enables professionals with an interest in Asian business to make informed decisions and gain competitive advantage so it's even more of a professional read than the FT is probably but what this did was give us a kind of just like a common goal of what we were trying to achieve and this enabled us to kind of then be one team because what we were able to do was then move from a bit more of a client and supplier relationship to like can you deliver these things to an actual partnership because we kind of got everyone to agree on a product vision and say okay what are we going to do to make that true for our customers so I cannot under like I cannot overstate how important a product vision is and I just thought it was always something that should always like it's always there you kind of look at it you kind of do things and you kind of like it's very in the background but when you land into a project that's chaotic and you are someone why are we doing these things and they're not able to answer that's not a good place to be and for us it was a really valuable exercise to have the second thing I did was introduce a learn, build, measure, learn cycle which was really really hard to do when the data was in a bit of a bad shape so what I started doing was established business outcomes and completely nicked to the FT ones for ease because they work so we took acquisition, engagement, revenue quality and efficiency as our outcomes so they were the five things that we were trying to do this yeah, if there was a piece of work that did not meet any of those outcomes not going to happen that's just not going to happen and then what we did was set up metrics within acquisition, engagement the other three that I now can't remember and we didn't have an engagement score but and we didn't really have enough data we're still trying to create one and our data isn't in a great shape so we're still trying to work out if we have enough of a sample size to actually work out what the engagement score is for this publication but what we were able to do was know at the FT recency is important frequency is important as is volume so I had those metrics sitting underneath the engagement outcome so what we were immediately able to do was measure what we were doing so when we kind of came up with an idea we would say what business outcome and then we'd be able to say let's see if improving the onward journey on the article page enables volume to increase so we were kind of what we did in an imperfect world was look at I've got an engagement score but I know these metrics are important so we'll just measure on them individually whilst we create that score we persuaded them again that it would make more sense to work in smaller iterations for Japanese culture that's really hard because they are a little bit more in business a little bit more detail orientated than us naturally the idea of not doing something that wasn't completely perfect was a lot more uncomfortable for them than it is for us so that's something that really took a lot of persuading we basically got there because we kind of said if you want all of this we're not going to meet your deadline so that's where Waterfall actually helped which is a weird saying but that kind of showed them we had to do stuff after launch and then what we did was use data to learn from our work and then help use data to adopt a more continuous approach we had those outcomes defined and those metrics we were then able to measure what we were doing and seeing if it was working and then the last thing we did in Japanese culture group consensus is really important so group harmony is important and this is particularly important when making decisions so what the previous delivery manager was trying to do was get decisions from about 25 people in the room over a Google Hangout and those people were newsroom, they were B2C they were B2B, they were senior they were junior and that was just a nightmare nothing ever ever got decided which was obviously slowing us down so what we did you can't change culture but you can look at it and change your approach so we kind of created subgroups so I created a leadership team which consisted of myself the delivery lead on the project and the tech lead because at the FT we call ourselves the free amigos so we always work together loads we're really cool so they never really got that concept we had like a dance in Mexican free amigos and it was just like tumbleweed in the room when we presented that on the fly deck and for them it had their two senior leaders from the newsroom and the two senior leaders from the commercial team because we were kind of forcing them to be one leadership team to actually make some joint decisions so what we do as a group is prioritisation so we look at our outcomes we look at our data look at our problem areas and then we prioritise the features that will achieve all the problems we want to solve to achieve what we're trying to do we then created a subgroup which we called experts slightly to flatter the egos and so once we say right this is a problem we're going to solve so for example our navigation usability is poor we then create like an expert group and we only work with them on that solution so we kind of get them involved with the process and we are saying how can we better improve the nav let's look at how people are clicking on it and then what we then do is go back to that group of 25 people and kind of say they prioritise it we've worked with them here's the designs please don't moan kind of thing that's how we've had to break it down I think actually that's pretty much the end of the talk from taking a project approach across cultures my belief is that a product development approach can completely work across cultures that's fine it's a really good approach and it's something you should really do what you should do however is tailor how you work together so for example I've never had my prioritisation list in a spreadsheet it kills me every day I love Trello but it completely works with my stakeholders so tailor that approach I would never want to go through so many groups to review a design and get sign off but I need to do that due to the stakeholders for example and what we do on our programme is we have for our product development learn, build, measure, learn we do exactly the same for ways of working so we've done loads of things where that has completely fallen flat and missed the mark and our stakeholders are like we hated that so we take that feedback and then we kind of continue to iterate