 Kia ora koutou everyone. Thank you, Fi. I'm very nervous about this presentation. This is an original piece of work and I've done a lot of talking internally at Te Papa around it and I've started sharing it a little bit wider and it's really interesting, the feedback. I'm going to go through it pretty quickly. It's a reasonably decent sized model and you probably will have lots of questions until we won't get through it in half an hour, but I'll do my best. OK. Not working good. 1.8 million. 1.8 million visitors in 2015. That's a big number. So it must mean that we're pretty successful. 12 million is an even bigger number. 12 million people passing through the Wellington Railway Station a year. So that must mean that the Wellington Railway Station is more successful than Te Papa. It's pretty important. I mean there's no denying it. 500,000 visits to collections online. So less successful than the building. 400,000 visits to Rainbow's End. Is that comparable? Are we competing with that kind of thing? It's a similar kind of number. 700,000 sky subscribers. So collections online is less successful than sky by the numbers. 250 likes is the highest we had in Instagram for a post. It's a picture of the Sunday markets. Nothing to do with Te Papa at all. 520,000 visits. Is this something we should be comparing ourselves to? Visits to Citizens Advice Bureau in a year. So numbers are important, but they're not the most important thing that are out there. We need to find a way to differentiate ourselves from the numbers that are pouring through the alternative museums and the other things. And so we need to start telling different stories. But numbers are important because we need them to understand market share and reach and plan our budgeting and all of these sorts of things. But it's only the beginning of what we can actually do to measure our success. We do wonder and awe and emotion and those sorts of things actually fit in to what we do and how we measure. What does the funnel of engagement look like beyond the numbers? What is the value of what we offer and what is the impact of what we offer for our audiences, not for us, for our audiences. So this is an attempt to move away from big dumb numbers. So there's a bit of a theme there. Big dumb numbers are important and they're good storytelling, but they're not the only thing. Traditionally, Te Papa has talked a lot about feet through the door. That's our main metric. Paid views is something that we report to the Government for some reason. And dollars earned, which is really important. We have to have money and we have to gain that kind of stuff, but these are the things that we've traditionally measured as success. But as the National Museum and Art Gallery, shouldn't we be doing more? And we thought, yes. So a couple of years ago, when we had a digital directorate, we were setting up a number of tools and frameworks that made sure that we were building the right things for the right people in the right way, for the right amount of money. We had something called the Digital Product Development Framework, which I'll talk to soon. We had a customised lean canvas to make sure that we understood the problem, the audience, the goals that we were attempting to do. We used design sprints to make sure we were focused. We had a really good focus on user needs and how we tried to identify problems to solve. But a massive gap that we had was a consistent and meaningful way to identify success beyond numbers. This is a new version of the Digital Product Development Framework. You can... See, you start with an idea. You write up some stuff for a lean business canvas, maybe a short business casing like that. You make a decision when you put it up against other products that might go through. Do some solution validation, which is a short, intense, maybe design sprint or something like that so that you can try and come up with a problem or a prototype that you can say, we think this is the right direction to be going in. Then you get a goal and no go, so you get the rest of the money to actually go off and build something. Then you launch and then you go into high care so that you look after this product for a while very actively. You move into sustain and grow, which means that you don't just walk away from products once they're launched. You make sure that they're being looked after and maintained. You look after your communities. You look after these products and you may decide to retire it or you may decide to move it on to the next iteration. Just a nice life cycle to make sure that we're covering all the parts of digital product management that we need to. In this area here is where we notice the real problem of not having a good way to understand what success of fixing a problem looks like beyond numbers. This is the modified lean canvas that we use. The lean canvas is incredibly useful for very quickly checking all your biases and making sure that you have a problem that you're trying to fix and that you understand your audiences. It's modified from a start-up lean canvas. One of the things that we did was we modified instead of revenue, which of course is my start-up thrill in goal, is we had value and success metrics, but we didn't really have a good way of actually filling in that box. So that's all that stuff. The way that we started was using Dave McClure's Pirate Metrics, and they're called Pirate Metrics because they are because they're acquisition, activation, retention, revenue and referral. And they were good, and they were better than what we had before. They were a model that kind of worked and worked for a lot of people, but we're not driven by revenue most of the time. So we looked at a bunch of other models. We do kind of have an ad hoc way of doing things when we are coming with exhibition visitor goals. We'll come up with what we attempt to achieve in the long term, but it's usually pretty inconsistent and there's not an actual framework behind it. Google has a model called HEART, that's Happiness Engagement, Activation, Retention and Task Completion. That's more UX-focused, so at least it's based around the visitor, not the business, but again, still not quite right. Task Completion, we don't just want people to complete tasks. We want people to complete tasks with impact, with some kind of benefit. That's Morris Hargraves McIntyre, which is a cultural segment company and they have an engagement spectrum that talks about the traditional ways that museums have done stuff in the past, which is labels in cases and that kind of thing, at one end of the spectrum, all the way through to co-creation and participatory design and things like that in exhibitions. So there's a spectrum there. MK Hayley is an imaginary at Disney and she's got a model that I'll talk to in a second. And what I found was a really simple common flow spectrum. It's not a one-to-one relationship across all of these models at all, but there is a common flow. How to move people from initial attraction. Most of them start with, we need to get people. Obviously you have to get people before you can do anything. Through to something more. Depending on the model, what that something more was varied. At the beginning, always the user commitment was pretty shallow, but at the end there was a more deep commitment from the person who was engaging with you. Time frame started off from medium to long term and value exchanges minimum effect through to maximum impact. Looks a little bit like this. I've also included the tupapa changing hearts, changing minds, changing lives because it kind of happened to fit in there as well. Changing lives is if that's our goal, how do we get down there. Again, the spectrum is kind of similar. Start off with the lighter stuff, end up with the heavier stuff. I've since removed the MHM engagement spectrum and the mapping of this because it's more of a how to get impact rather than what is the impact. Every time I share this with someone new, they go, have you thought about model? I go, no. I didn't know about it. I couldn't find it. What usually happens is it doesn't prove that I'm wrong. It actually gives more evidence that there's something there. A recent one was the Kirkpatrick model which is a model for assessing organisational training. It starts with, do we get people coming to training? Are they engaging? Are they learning? And then are they actually using that learning back at the organisation and there is the impact from that learning? And as of yesterday, we've now got the Ethan Zuckerman when you look at the symbolic impactful light through to heavy. So we're only talking about the X-axis here. The thin and the thick again is more about how rather than what. So I've added those two in here and you can see again if there's flow from lighter symbolic acquisition, grabbing people with that sort of thing through to the heavier the idea of changing lives, of tar success, of having an action of results impacting the business, of impactful engagement. So I started developing a new spectrum for digital products that started with the lightest engagement and moved through to more successful impacts. Came out with a high level of attention and motion action. Pretty basic, pretty straightforward. And nine steps underneath the user's scoring system. Lots of testing, lots of talking, some yelling, lots of feedback. I'm looking at Rob here for the yelling bit. And lots of discussion around what we found was, people liked it but it wasn't quite right. They didn't trust it, it was one of those things that I saw and they found too many ways to pick it apart. So I tried again. An attempt to be much more in a sweet spot. So it's now a five-step model with a 1 to 10 ranking for further clarity and more tangible scoring. And it really, I think, focuses more on the idea of having qualitative and quantitative together. And using the quantitative to support pulling the qualitative measures up higher. And it provides, really importantly, a structure for storytelling. So these are the five steps. The tension, we get someone. Reaction, they react to whatever it is that we've provided for them. Connection is they find a reason to actually want to engage with this. There's a connection between them and the product or the story or the experience. Insight, they learn something. Action, they actually go away and do something. There's some kind of something that actually happens. And it looks like this. So we've got the 1 to 10. The five are broken down into 10 more tangible, scorable spots. And that looks incredibly boring and incredibly dry. It's when you actually put the real data in that it starts to make a lot more sense. I know you can't read that so I'll pull out a few of the highlights. Most of this is real data. Some of it is anecdotal. But it's based on experience and some of the data that we're gathering. So attention. We're good at attention. Tepapa is very good at attention. That's getting people. So that is getting people to click on our social media. It's getting people to read a label. It's getting people to walk into an exhibition. It's getting people to click on a website. Immediate response, we put a lot of effort into this. This is where a lot of the design originally goes. This is getting people to actually enjoy something or feel an emotion or have a reaction. It might be sad. A lot of people come out of Gallipoli feeling sad. That's a great response that we might have wanted to do. You might have jumped with surprise. You might be repulsed by a spider. That is an important part of actually moving further down the spectrum. And then there's personal connection. So my grandfather was in the war. I wonder if his gun was like that. Thinking about the stuff down in Gallipoli. We're developing a new nature exhibition that will open early next year. And one of the things that we're talking about is river health and the amount of ills that are around now. I remember when I was in Wai Paoa growing up in the Wai Paoa River there were places you would know would be ills. Now there's hardly any. So that is the reaction or the emotional and personal connection that I can have that will pull me into wanting to go further down. Well, Tepapa has a journal on replay station game. That's why we have some of these things in the collection and we tell these stories because it is about the personal connection. Simple learning. We want people to learn basic stuff. So learning basic treaty facts. The idea of finding out learning that the two versions of the treaty were not the same. That's really important. It doesn't tell you a lot but it gives you the beginning of understanding sure, that's okay. Wow, look at all those kiwi feathers. It must take a lot of birds to make that kura wai, that cloak. I learned a word of te reo. I learned there was more than one kind of moa. I hadn't even really thought about that before. So really simple learning. We're good at that. A lot of museums and libraries and archives and all that kind of stuff, we're good at that. Then it starts to get a little bit harder. Contextual learning. Is where you actually start to question something about what you thought before. So imagine that you're a man in his 70s who maybe grew up in Wellington and around time when Rupe was around. But now, when we look at her story, I just had thought of her as a drag queen before. I hadn't thought about her as a hero of her society and that she was a business woman and all of these kinds of things. Start to question stuff a little bit. Seeing where your opinion fits in relation to other people's opinion, actually understanding that. To level five. Level six, applied personal learning. Actually starting to change the way that you think a little bit based on what you've learnt. Changed my mind about something, understood how the treaty fits in my job. Never really understood why it was relevant. I understand a bit more now. I had a talk with my family about war and we talked about what we thought, what was right, what was wrong. Able to see art in a way that I haven't before. Moving on, applied empathetic learning is actually starting to think about how that learning and what you've learnt affects other people. I'll read the labels or other things. I'll read the labels on seafood products more carefully now and I'll look for sustainable ways of fishing. I talked with my kids about the impact of pets on wildlife and what we thought was appropriate as a family and with our pets. Felt confident at work in a discussion about when to rail shouldn't be used. This person might feel more empowered to actually have that discussion. I'll look at how Māori are represented in popular culture more carefully now. We're not into action yet but we're starting to get people to empathise with others and share that empathy. Then we get into actual action. With the nature exhibition one of the things we want to do is get New Zealanders understanding the impact of pests on native wildlife and that sort of thing and actually doing something about it. We might sell pest traps down in the store and we want people to leave the exhibition be moved enough to go, I'm going to take action, I'm going to buy a pest trap, I'm going to install it. That is something that would be a significant success for the exhibition. Committed to using a few words of te reo in my daily life and actually doing it. Going to keep my cat in it now. Again, at night. That's an action that we would be an amazing benefit. Better earthquake proof, that water cylinder I guess. This is a real one where a couple of years ago we made the Minecraft shaker mod in earthquake simulator in Minecraft and a lot of things fall over if you don't stabilise them. We had kids playing the game and we had parents to go and do stuff to the house to make sure that it doesn't happen in real life. And so again, that's personal action. Made, wrote and created something. We don't record that kind of stuff enough as a measure of success of what we enable. So that's personal action. I'll come back to, I think I've finally decided to go back to school because that's a real one. Then moving on to nine which is group or community impact. What we've done has enabled somebody else to do something for others. Has created an impact on more than one person. So this is a real one. Created their own guide for an exhibition for others when the art galleries were closed when we were redeveloping toy. We put a guide on the website that says there's still art in the building. This is where it's scattered around and you can find it. A young man who was on the spectrum came in and he did that. He really enjoyed the guide because he was able to find his own way around. He's next to a lot of other people. He could do it all himself. He liked it so much he went home and made another one for his classmates. Started a stream restoration project in my street with my neighbours. That would be pretty impressive. Why aren't we designing for that kind of stuff more? This is a real one to use for Māori myth and legends component of our treasures of Tane's Zoo School programme at Orana Wildlife Park. Again it's making something but it's making something for other people that has ongoing educational benefit. That's from collections online as a result of our high resolution downloadable images. We can actually talk about national impact as to Papa. There's a number of things that we do that can enable national impact. We might provide some content that someone is able to turn into some really successful movie or something along those lines and they generate and come for New Zealand. Secondary and economic benefit in the money doesn't have to come to us but we've done something that's actually affected the economy. Aided research to find the original pink and white terraces. We're doing that again with the content particularly using our historic content to help digitally recreate and also locate the pink and white terraces. And affecting policy change at government. There's lots of ways that all of our organisations can do that kind of thing. These are all very general but what it shows is that there are real stories that show that we can move down this spectrum. But it is a funnel. We're not going to get everyone who comes in the door doing national impact. That's just silly. It's just a really indicative funnel. If we get a certain number of people into the exhibition, into a digital product, whatever this could be the user flow that we have of losing people as we move all the way through. We might only get 0.01% of that first number thinking about doing national impact. There's still one. One out of 10,000. This isn't going to apply to every product or anything along those lines but it just goes to show that when you think about the funnel or the design for it, there's a lot you can achieve. So some real examples. Artwall is a little kiosk where there's about 400 paintings and that sort of thing on there where you can scroll through. You can pick the one that you like. You can say where you like it, where you're from and you can put it up on the wall and everyone can see it and see why you liked it. It's just come down but it's in toy art. It's actually on display in the trade stand. We wanted to make sure that this wasn't a digital projection. So we actually put some measures against it. 500 user sessions a week, we thought based on the traffic that we have, some testing and observation that we've done, we thought that might be about right. We want the average session length greater than three and a half minutes. That means that people are actually moving through, they're looking at all of the stuff, making some decisions. This is still at level one though. We want three or more artworks that you can scroll through thumbnails. Click on one and go, no, not that one. Close it, keep scrolling. Again, that shows that people are really actually using the work. Immediate response. We want 50% of sessions resulting in artwork being successfully sent to the art wall. 50% of submissions include comments so there's an actual action. It doesn't matter what those comments are, we want 50% of the people to make a comment because that shows some kind of response. We want 20% of comments to indicate emotional response, draw, intake. This is all real data because the comments are the data. We want... I'll just jump ahead because I'm running out of time. We want 5% of comments to indicate new art confidence. So this is actually a target for it. We're not just going, this is a shiny thing. We need to design for these targets when we're doing it. We'll move on to another one really quickly. We were talking about making a digital discussion platform for Suffrage 125. This is the year to do it obviously. We thought this is something that maybe we could have National Retro on. You don't have to come to the building but we would have a building kiosk that did it as well. So based on the traffic to level 2 we thought, okay, we want 80% of people to level 2 to see it. We want 60,000 campaign homepage views online over the year and this is all based on the... We kind of know the numbers. We want 10,000 on-floor visitors to start a campaign view. 10,000 non-textual responses left. That's there because we know that not everybody is willing to do a textual response or put their voice to something that they may be a little bit nervous about so we need to design a way for them to have access into this conversation. 30% of surveyed visitors we want to have a new perspective on Suffrage beyond voting. We want 5,000 comments and it moves down to, if we just jump to say 7, 5% of commenters report having had a follow-up or ongoing discussion with others since visiting. 2% of commenters report having made volunteer mental or charity contribution as a result. 1% of commenters having made a change in their lives, e.g. asking for a pay raise they weren't going to applied for a job that they were too nervous to starting a sport that they thought was too rough or changed a home situation or chore. These sound lofty but the point is that they're supposed to be because now we have to design for these comments. We're going to do it. How do we get those kinds of comments, those kinds of impacts to happen? We have to design for it up front. Collections online has 500,000 visitors also a year. This is all real data. We gather lots and lots of data about use and reuse every day from people who are using it. We've got all this stuff about I liked it. I'm using it as a nice picture for my phone wallpaper. I wanted to find out what the hell the spider in my bath was. That's great. That's our most popular task completion but it's only a level 4. Lots and lots of level 4s, so that's great. Lots of other things moving through but we've got lots of examples of people creating things as a result of using our collections and our images. We've got comments indicating teaching use and again the idea of reconstruction of a national site. We have gathered all of this data from people who are using it. This is the example where first of all I realised that this wasn't just a digital spectrum. This was about anything that we do. When nature closed the natural science exhibition that had been open for 20 years. There are people in Wellington who had grown up with that exhibition and had come in and seen the giant squirt, the birds, the wetas and the trees and all that kind of thing. That had hidden in the trees from the parents. We were shutting it down so we held a closing party and there was disco, there was DNA themed cocktails, all of this sort of stuff. You might think for that kind of stuff we would only design for, and we did to be honest, a four or a five. Simple learning, maybe contextual learning. There were curator floor talks about this is what this bird likes and all of that kind of thing. So you get access, what we didn't necessarily design for but is what we enable through the one-to-one access to curators and that kind of thing is we convinced an adult student to feel that her passion was worthy now and that she wasn't afraid to go back and do it. Why can't we do with this one-on-one stuff when we have people start out on the floor talking to our public? We've got a lot more opportunity for impact there. Giliplee is a really interesting one. This is all retrospective. One of the things I found interesting about this is when you think about personal action, most of the stuff that is in there is sort of level four, level five. I learnt about the war, I learnt about the Turks. There's some empathetic stuff here, so the idea of during the commemoration thinking about the Turkish side, not just the Allies side and the impact that it had on those as well. Everyone was people. But also the idea of potentially negative impact. Where do we go with negative impact? So I decided not to join the army. War is horrible. What if we convinced that they did need to join the army to protect the country? Is that okay? I don't have an answer to that. Everything is based on use of success, not on to Papa's success. Reaching the higher levels is difficult and it's difficult to measure and it requires new ways of thinking and I think it's one of the reasons why we haven't got there yet. Not every product will hit level 10 but this model asks how far can we go. We were happy designing Artwall to get to level four that fits well with what that product was. Obviously the different products are different goals. Allows for the shiny to sit alongside the deep very obviously. And the models definitely designed to be aspirational. The models also designed to be applied at product level, at exhibition level, at programme level and at organisational level. And what you can do is come up with things like this is how are we doing organisationally across our programme. Are we happy that we've got this much impact and we've got lots of volume down here? Are we happy that product C has got hardly any impact but it goes, sorry, hardly any visitors so the numbers are very low but it's getting all the way down to lots of personal action that might be through an education course. It's a design tool, a prioritisation tool, a testing tool, a success measurement tool. The idea is it should be used not just at the problems of validation stage it should also be used as a decision making tool it should be used when you're validating your solution any decision and point it should be used should be made as your building and testing it should be used as your in-high care when you've just launched it, how well is it doing and it should be used throughout its lifespan and it should be used to figure out is it still doing what we thought it would do. Measuring success is hard how do you measure the impact if it's five years after the visit and again this is one of the reasons why I think we haven't done it. We can use our channels, we have people Te Papi has amazing networks and so does most of the sector so we can use the Ministry of Culture and Heritage, Ministry of Education Schools, Universities, find out the stories that they're hearing about us and about the impact of visitors so art schools, how many people were influenced by, how many new students were influenced by Te Papa in choosing their school and use those channels we need to do more social media monitoring we need to do follow up surveys and repeat visitor interviews when people leave Te Papa they do an exit survey that says how did you enjoy your visit? Why don't we add into that how did you enjoy your previous visit and what impact did that have on you in the last six months or whatever we need to automate the earlier stages more so that we can put more effort into the later stages but we just need to be more proactive, we need to ask people and get their stories and not rely on the numbers it can be used in all kinds of things and there is this is a movement that is happening everywhere this is a major science funder in the UK and they want to know what economic and societal impact is as well not just academic so there are some weaknesses possibly not backed by enough no-one is that linear in their motivations it's a pretty simple flow but it's better than nothing different cultural groups and audience segments reacts to emotional engagement in different ways and there is research around that some levels are open to interpretation and it can be cheated so you need to keep an eye on it and it defies a complex need and yet at the same time it's too challenging for some people it exists, it's pretty simple it's designed for our sector we need to ask people more about what their impact is going to keep working on it going to make it easy to use it's not that simple at this stage I know it's at the beginning but we are going to start using it to map to Treasury's four capitals so when we report to the Government we've just been reporting numbers and that kind of thing the Government in general wants to know more about what real impact do you have not just on the finances we give you money and you tell us how many people came how are we actually starting to be good for the nation and as of yesterday it's like why can't we even just map to the UN's Sustainable Goals because this is the kind of stuff that we're working to as part of our work sorry that was bad