 Tool and Confluence, all right, cool, excellent. And yeah, we're based in Sydney, San Francisco, Amsterdam. And there's a couple of us here from Sydney. So it's great to be here. Thanks for having us. So just a little bit about me. I'm an engineer-turned-product manager. I've been doing product management for about 3 and a half, four years now. And I spend most of my time in Elassian working on our collaboration tool called Confluence. It's a content collaboration tool for teams, and mostly helps them create documents like requirements, meeting notes, specifications, that kind of stuff. So that's where I spend most of my day working on that. I also created team calendars at Elassian. It's a commercial add-on that we have on top of Confluence. It helps teams plan, track, and manage their events, their team events, and their projects. So I'm also the product manager for that, half a day a week. And last year, I spent quite a bit of time working on this project that was across all our tools, and it was cross-product notifications and tasks. And the reason I tell you this is throughout each of these experiences, I've learned a lot about getting feedback. And it's been quite challenging. Some have been quite the hard way of getting feedback, and some has been the easy way. And so, for example, on Confluence, we completely changed the way people create content inside our tool. Previously, they had this code-like syntax called wiki markup, and we completely replaced it with a wissy week editing experience. So a huge change, and we learned a lot about how to get feedback during that change, lifecycle, and different techniques we would have to deploy. Team Calendars was something that was new. Alassian was launching a new product and was something that was new. And we deployed and learned different feedback techniques throughout that project. And this cross-product notifications and tasks, it was one of those features that we had a big blurry vision for. And we went through quite a lot of product discovery, and we were unsure who would find this feature the most helpful, what kind of personas or users would use this the most. And so we learned a lot about feedback through that project as well. So I'm going to be talking to you throughout experiences that I've had in each of these projects, kind of like an experience report. And we'll be sharing with you guys the different techniques that we did. So I hope by the end of this presentation you'll be equipped with at least five practical tips that you can take back to your projects or teams for building effective customer feedback loops. Just out of curiosity, how many people you actually work in product teams? All right, three quarters of you. OK, excellent. So this is for you guys. But before I jump into the actual practical tips, I'm just going to spend a bit of time setting up some context on a topic that's been discussed a bit for the last couple of days. And it's about the build, measure, learn lifecycle. Well, he's read the Lean startup book by Eric Reyes. It's been referenced a few times in a few presentations. I highly recommend this book. I learned a lot from reading it. But the premise of the book is fairly simple. And I'm not going to do it any justice in my explanation right now, so I still encourage you to read the book. But it's all about optimizing for learning. And the faster that you can learn, the more sure you can be you're building the right thing for your customers. So you're a product team, and you've got an idea, and you're building something. And then when you ship it, you learn. And there are different ways of learning, typically measuring through analytics or usages of your features. And then from that, you have ideas. You apply the learnings, and then you go back, and you iterate over and over again. And the faster that you can learn, the more effective your team can be in making sure that you're building the right tool. And a lot of learning these days happens through number crunching. I spoke to a product manager of a very large organization, who's very famous. And they said to me, we don't talk to customers. And I said, what? He said, no, no, we don't talk to customers. We spend most of our time just number crunching. They're just huge, hive data stores, looking at analytics, and all that stuff. And he was shocked that we spoke to customers on a very regular basis. And I think sometimes we forget to get that unsolicited feedback from customers and have that verbal contact. Or as Jeff had earlier, meeting them face to face and bringing them in your office. So a lot of the feedback techniques I'm going to be talking about is not about number crunching stuff, is about that unsolicited feedback that you can get from your customers for the products that you're building. So let's dive straight into it. Five tips for building effective feedback loops. And the first one I have is all about reducing friction. Friction, friction, friction. You need to be obsessive about removing it. And I'll give you a couple of examples. Some practical real life examples. Like a couple of months ago, I was coming back from a conference in San Francisco, I went back to Sydney and picked something up from Judy Free. And when I arrived in Sydney, I gave the lady my boarding pass so she could get my goods. And in front of me was this screen. And it was 15 seconds to give feedback. Yeah, sure, why not? I'll just give them feedback while I'm waiting for my goods. Great way to reduce friction. They didn't give me a paper or a survey that I had to fill out and then bring back later or something like that. So as I picked up my goods, the person next to me was already filling out their survey and giving him feedback immediately about how he felt their service was working. Or I had to put this photo in. Two days ago, I was in Singapore on the way here. And I came out of the toilet. And there was this big LCD screen in front of me. And it said, please rate our toilet. And so I pulled out my phone. And everyone was walking past. And I'm trying to get the photo of this screen. So it was a bit embarrassing. But talk about reducing friction. You've just used their product if you use that analogy. And you're walking out. And they're asking you to rate their product. And I presume this poor gentleman here would be notified of the different ratings that people would have rated the product that they just used. Sorry? The last line is clear. What was the last line? Oh, the screen is. I did think about it just before I touched it. I was like, oh, is this clean? So but an example that we did at last, and I told you about this earlier, we completely changed the way people create content inside Confluence. This is the tool that I work on every day. So this is the main screen of the tool. This is the editor screen, where people create content. And what we did was we put this little Got Feedback button up the top. And it was in context. We wanted to capture feedback about their editing experience. It was right there. And when you clicked on that, you were presented with this screen. No login screen. No redirect to another website. It was all in the product, in the dialogue. It's a modal dialogue, so you didn't have to switch context. And believe it or not, we debated and we agonized over these three fields that appear here. But it was well worth it. We ended up deciding only one in another tool where they need to log in again. You really do want to reduce the amount of fields. Like debate each field that goes on in that feedback form and make sure you have only what you need. And try to automate things where possible. If you need to capture browser information, technology can solve that pretty well today. As you saw in the example of the toilet, I thought it was a great one. Just if that's all you're after, a yes or no, or how do you like this feature, then make it really easy for the end user to express themselves to use your tool. So how can you do this? How can you build these feedback forms? Well, there are plenty of free services out there that help you do this. This costs you no money. If you are a JIRA customer, which I know half of you here, we have the JIRA issue collector, and that's free. But there's WiFu's free Google Apps. You can use that for free. Hips of free stuff out there. So it's not hard to do this, and you guys should be doing this and all your products. It's amazing the unsolicited feedback that you get. My second tip is all about making the experience fun. And I actually don't know, but do you get the biggest loser in India? Have you heard about it? Yes? No, some sheep. So for those of you who haven't seen it, it's a reality TV show where people lose weight and they've turned it into a game. And in Australia, we have a terrible habit of copying everything, all the reality TV shows from the States. So we have it in Australia as well. And what amazed me is the amount of different editions of the show that they can make. So it was the biggest loser couples, the biggest loser singles, biggest loser celebrities. And someone told me that was actually a biggest loser for pets, for your obese pets, which was kind of freaky. But I came back home from work one day throughout this Confluence 4.0 release, and the whole changing of the editor thing. And I was trying to think about what we can do to encourage people to give us feedback. And I turned on TV, and the biggest loser was on. And I thought, hey, why don't we create this competition called the Biggest Winjer? And I have to explain what Winjer is, because it's an Australian slang. It's someone who complains a lot. And so hence the baby having a sook or complaining, another slang term, sorry, complaining about this. So we created this competition called the Biggest Winjer. And we said, hey, whoever gives us the most amount of feedback, we're going to give you a prize by the time we ship Confluence 4.0. So because all that stuff was coming into our issue tracking system, we could easily report on it and graph and work out who is giving us the amount of feedback. And we gave them gold class movie tickets and said, hey, thanks for taking time out to give us feedback. We really appreciate it. For team calendars, I told you earlier that was a new product that our company was launching. We said, hey, whoever gives us the most valuable feedback will give you a free license. You will not have to pay for this product ever again. We'll give you a free license. And I was amazed at the amount of feedback that we received from customers that took time out of their day to give us feedback. Another technique we used, which we copied out of the Xerahinota Balsomic. They're a mock-ups tool. They do this really well. They're very personal with their customers. They actually call out all the customers that give them feedback in their release notes. And they say, hey, thank you for that suggestion. I made a huge difference and we implemented this feature. And it goes a long way for that user feeling engaged. And they're more encouraged to give you feedback a bit later. So think about how you can make the experience more fun by incentivizing through prizes or awards. Take time out to say thank you. It goes a long way to show some appreciation for the time that people give you to give you feedback on your products. And where possible, think about some game mechanics. Make it a competition or a prize. Give some different incentives for people to give you feedback. So we've talked about reducing friction. We've talked about making this experience more fun. My third tip is about getting personal. And I'm not talking about finding your most attractive customer and asking them out on a date. I'm talking about really getting a deeper understanding of your customers. So on my way here, I emptied out my wallet. And I have this habit of just emptying out my wallet before I travel. So I'm afraid I'll lose all these cards that I never use. And some of these cards came out. And I was looking at them. And they're just consumer rewards cards. You know, the things where you collect points for shopping or frequent flyers or hotel, that kind of stuff. And I was thinking about the other end. They learn a lot from my usage. As I use their products or services, I'm giving up some of my privacy. In exchange, I hope they're going to get a better product in return. So they're learning a lot about me as a customer and the different things that I do or the things that I like or don't like. And so I talked to you briefly at the start about this cross product notifications feature that we're building at Atlassian. I'm not going to dive into details of that. It's kind of irrelevant. I guess the point is we had this project that we were doing a lot of product discovery. We were not sure on the types of person, types of people that would be using this tool. What kind of role are they in? What kind of features would they like or not like? And so Atlassian is getting quite big now. And we're now 600 people. I mean, it'll be big for some of you. I spoke to some of you here that worked for a company that was 23,000 people. But so one of our engineers came up with something quite creative. And internally, because this was all internal, it was not as many privacy concerns, we had this giant wall board. So we got this LCD TV where the team sits and we plotted a picture of the usages of that feature of that day. And we rolled the feature internally. We didn't tell anyone about it. And we wanted to see who uses the feature. And so we had to take a closer look here. This is not this guy's hot or not rating. It's his actual usage. So we had a picture of the user and the key features metrics that we were interested in about that particular user. And so at stand-up, we'd be like, oh, well, you know, there's John. He's using it again today. Why don't we reach out to John for an interview? He seems to be using tasks a lot. Let's ask him why he's using tasks. What's he finding helpful about it? What's he finding painful about it? And so we used that a lot to get an understanding of who this feature stuck with and who the persona was so we can learn quickly. In fact, their engineer went even further. And I haven't got a screenshot of this where he drew some pie charts. And because this is internal, we graphed by department who used the feature the most. So we quickly understood this feature was used most with business people that had teams working in JIRA, but they spent most of their time in Confluence. And we'll be able to zone in on that customer of ours so we can learn a bit more about them. So putting a face to the staff, that goes a long way to getting some feedback. You get some engineers. And it's amazing what it does in terms of engagement. They can see the people that use their tool every day. And what we did is we got those people that we wanted to talk to that had interesting usage metrics, and we reached out to them for interviews. And we had one-on-one discussions with them. And we said, hey, what don't you like about this? What do you think this is supposed to do? So it helped us learn quickly about this product discovery phase that we were going in for this particular feature. So my fourth tip is a bit boring, but you should really think about this. And it's not that hard. And I want to show you that it's not that hard. Have a think about writing a feedback strategy before you ship your next release. I want to give you four basic criteria that you can think about. This probably heaps more, but this four works pretty well for us. So it may be relevant to you guys. The first one is what you're building or what you want feedback for, a new feature or a new product? Or is it something that's existing? Because how you get feedback for those two different ways is quite different, actually. Or are you after something internal? You're deploying to internal customers or external customers? Because there's different privacy concerns, and getting feedback can differ quite drastically between these two. Are you after feedback for a specific feature or are you after general feedback for your product? Because if you're after something specific, you may need to think about having some contextual feedback mechanism for that feature. Whereas general, you'll have a general placement of your feedback button. Are you driven by a deadline? Or are you just after ongoing feedback? Because you may deploy different techniques if you are driven by a deadline or if you want something ongoing all the time, you don't need to incentivize people as much. So a quick practical examples for team calendars. This is the new product that we launched. This was new. And so we had to find different techniques to encourage customers to take time out of the day to install something new. And so that's why incentivizing went a long way. And we found some potential beta customers and we built a relationship with them and we checked in with them weekly. So we had to make sure there was probably a bit more high touch to get feedback on those examples. Was it internal? Was it external? Well, this was both. We captured we had internal customers and external customers. So we learned a lot about feedback through that way. And for internal, we're a bit more slack with the data that we collected because we could collect a bit more than what we could do external. It was a new tool. So we're after general feedback. We went after feedback for a specific sub feature of that tool. And so we had a big floating kind of button at the bottom that said, hey, give us feedback. You can kind of close it off or whatever. And we arranged casual interviews for the tool. For us, we had a deadline. And so I talked to you earlier about we gave customers a free copy of the software. And so that went a long way to incentivizing people to give us feedback with a deadline. And we had a follow-up plan. I told you we just talked to these people regularly and we checked in with them weekly. So right out of plan of attack, it's not that hard. It's probably just a table on a page with a couple of paragraphs. It's not going to take a lot of time. But what that helps you do is if you do this regularly, you can start to set a baseline on the amount of feedback that you receive and then set some goals. So you can say, hey, last release, we got so much feedback. Next release, we're going to try and get this much more. Or it's a bigger change, so we should expect to get more feedback. So you can start to use this iteratively as you release more software. My last step is probably my most favorite tip. And a few people touched on this already in the conference. And just the idea that you can get feedback well before you write any line of code. And I think it's that gentleman Owen. Yeah, he spoke about building the MVP for the enterprise yesterday morning. And a lot of similar concepts with his talk. So if you were his talk, a lot of similar learnings, and I found myself nodding, and we deployed some different techniques to get feedback well before we start. So a little story to this. I was in Bali. Anyone here been Bali, Indonesia? No one? Oh, well. Or one person. And I was walking down the shops. If you're not wearing a surf brand, you're not cool. That's what they tell me all the time. And I was walking down the road and some shops on the side. And I found these great surf t-shirts, Billabong and Quicksilver. And they were like $5. And I was, this was great. This was my Christmas presents and birthdays for the next couple of years. I'll just buy a whole heap. And I'll be the coolest uncle they've ever had. And I got a whole bunch of shirts. And I went back to the hotel. And I was so proud showing my negotiation skills to my wife. And I was like, hey, check this out. I got like 10 shirts for this many dollars and all this stuff. And she said, did you read these shirts? I said, yeah, it's Billabong and Quicksilver and all these cool brands. She's like, no, no, no. That says Billabang. It's spelt wrong. And I was like, oh, no. What am I going to do? So I risked it. I went back to Sydney and gave my little nephews their t-shirts. I was the coolest uncle in the world. They did not notice. OK, I thought maybe they didn't say anything about it because they were trying to be polite. But they're not very polite kids. So I'm pretty sure they didn't notice at all. But you can go a long way faking it. And I certainly was fooled thinking I found these shirts with these cool brands on it. But if you can fake it, you can save a lot of time and money to get your job done. And so Atlassian, I want to share with you some techniques that we have done internally of a toolkit that we've built to help us fake it. And Owen talked about how they built a static site that had no back end. And we have done similar things. And we have done similar techniques at Atlassian. So it costs us no engineering time. As customers come to us for features, we'll do these techniques before we'll even engage engineers just to get feedback, make sure we're on the right track. So the first tool that we have is we have a user interface library. So if you build products, you probably generally have a set of components that you use for your user interface. And we have this static thing that's called a flakpack. It's a massive HTML file with all the user elements, user components, dialogues, buttons that we use in our tools in one big file. So we can grab that file at any time. And if you know some basic HTML, you can quickly mock up a screen, put some buttons in a dialogue, and get some feedback on is this in the right location, that kind of stuff. Fairly cheap to do. You need to know a little bit of coding, but fairly cheaper to do. So that's what we call a flakpack. And that's what we have internally. And you might want to consider doing something different for your software. We have one of our engineers built this really cool framework internally. And my best analogy for it is think of browser extensions, but for a specific software, specific web applications. So he created this plug-in architecture to allow us to create user-specific extensions. So they're really just hacky bits of JavaScript, HTML, and CSS that manipulate the page. It's great for the type of feedback that you might want to get is, is this button prominent enough, or should we do this, or should we prompt someone after they do this task? It's all stuff that you could do with some really basic coding without deploying new software. So he's built it all so anyone can go into the tool and create this. And there are HTML, CSS, and JavaScript. It looks like this, a really basic screenshot, but this is confluence of our product. And in my username, I've got an extension section, and I get a directory of all the extensions that are available. And people have hacked the product to try and do different things. And we've actually found the really popular extensions kind of almost validate themselves. And they're like, OK, we should take that and bake it into the product and make it a core feature. These are really basic things. You can edit the source code in the product, so you don't have to download and do all this funky stuff. You can move pages around, that kind of stuff. Really basic. This actual framework is open source. It's called SpeakEasy. We have it internally. So you can go check it out and see if you want to deploy something for your products that you do like to do something quite similar. It's changed the way we prototype. It's really cheap prototypes for basic visual changes. So if you put this on a kind of graph of speed at the bottom of how quick you can communicate a concept to get feedback on. And fidelity, whether it's high fidelity or low fidelity, that SpeakEasy, HTML, flat pack stuff, probably in the middle for me, but it's different for each person. I'm not the greatest coder, so it's probably in the middle top. We have designers and designers love Photoshop, so they can produce good fidelity mockups. They're not quite interactive, but they're visual designs. If you use a mockups tool like Go Mockingbird, Balsamic, there's heaps of those free ones out there as well, you can crank something out really quick, but it's really low fidelity. So over the last year, I've been thinking about this space a lot because I'm trying to always validate concepts and get feedback as quickly as possible. And I found a tool that for me sits in this top right hand corner, fairly quick and medium to high fidelity. And so to give you an example of what we've done to use this tool, I wanna show a video that we showed customers for this cross product notifications and tasks feature that I've talked about a few times. This is a video that I recorded for the feature that we were building for customers. You know, you click on a notification, you can drill down and here's some notifications here and you can reply to the person that sent you a notification all in app and you haven't had to switch out in and out of the box. This whole video is fake. So no, not a single line of code, completely fake. We can reorder things, that kind of stuff. And we showed it to plenty of customers and as we were showing them the video, they were already telling us, hey, this star thing doesn't make sense. I don't understand it. What is that? What doesn't work for me? How do I add personal tasks? Okay, well, let's let you add personal notes up here. And then we also took the video and we showed it to plugin developers because our products are kind of like a platform. We said, hey, your notifications can appear here. Check this out. You can send notifications in here. This is what it could look like. What do you think? Do you have any questions? And they were like, well, what happens if I move this there? And so we actually started getting a lot of feedback before we wrote any code at all. This whole thing took me about two hours to do. And after doing this, we got much faster at doing them over and over again. So that tool is Keynote. Does anyone use Keynote or PowerPoint to prototype? Three, three people, four people. It's changed the way I work. I've found it extremely valuable. I do lots of presentations, so I'm proficient in Keynote, so that works quite well. If you use Microsoft PowerPoint, try Microsoft PowerPoint. It's amazing the kind of stuff that you can do. So I want to give you a quick guide on how you can do this yourself. This is the Keynote edition. I'm sure there's a PowerPoint edition as well. This is the Keynote edition that I demo internally to our other product managers and designers, and they use this themselves to prototype stuff pretty quick, or get feedback well before we start writing any code. First step, take a base. Make your base like you're making a cake. You need a base ingredient. In our case, it's usually a screenshot of a screen that we want to start working with. So we'll take a screenshot of that, and then we'll mix and match some Keynote goodness. So that flat pack that I showed you earlier, that big HTML file, I got that whole HTML file, and I recreated a lot of those elements inside a giant Keynote file. So headers, buttons, toolbars, form elements, dialogues. I literally just drew the shapes. It wasn't a lot of effort. And as we keep adding to it, everyone shares this library and we keep contributing to this library. It's got tables, colors, messages, all that kind of stuff, lozenges. So this is a shared library that we use internally. And you can come back to this and use elements out of that in your prototype, and I'll show you how in a sec. And then last, you just want to apply some desired icing, like you do in a cake. So you might want to transition between slides. You can set click areas. For those of you building mobile apps, Keynote is excellent for that. I'm almost out of time, two minutes. So I'll give you a quick three-minute example of how you can do this yourself. First step, make your base. I talked about this earlier. So I'm going to take a screenshot of this tool that we're going to do. This is our support portal, and I'm going to paste it here. So one of our support engineers had this great idea of, hey, what happens if people could tweet their support experience? What if they could share their support experience? Okay, so I'm going to go back to my Keynote library that we created for all our front-end buttons and components. I'm going to take a button here, and I'm going to put a button inside our product that lets people tweet. So after you're working in a support ticket with one of our support engineers, you can say, hey, this didn't work for me. Can someone from Atlassian help me out here? So we drag and drop the button. We can rename it. We can resize it. It's really simple. I'll just fast-forward a bit because I'm getting someone signaling at me. You're crazy. Jeff, this is your fault. So I add an image here, resize the button. So I'll go back to my Keynote file, and I need a dialogue. So we've already drawn one up with a rectangle. It's not rocket science. I'll literally highlight that from my source Keynote file, and I'll go to the one I'm prototyping. I'll paste it, and I'll just move some buttons around. This is all three minutes, by the way, not fast-forwarded stuff that I just did. And, okay, well, we got some text there to eat the support experience. We can kind of communicate the concept that we're trying to build here. I'm going to pretend that someone's typing so you can get a feel that someone's actually filling out this form. So I'll write up some text here, and then you can use transitions to communicate that someone's typing. And so Keynote lets you have this typewriter transition so you can make it look like someone's typing and filling out the form. Let's do that now. We'll just apply a transition. So select it, and you go apply some transitions. You can go, and now we'll actually set a click area so it looks, it's a bit more interactive. So this button, we want it to be clickable. So we'll click that, and Keynote lets you link between slides and have clickable areas and that kind of stuff. So we'll make it a click and we'll make it go to the next slide. And that's it, that's three minutes. Well, it's not three minutes. I fast forwarded a bit, but you get the idea. I hit play. This is the concept. Oh, there's a button there that we've added. When I press that button, oh, I get this dialogue. Okay, cool. I want to share my experience with the support engineer. I'm going to fill it out and get some feedback. Done, three minutes. Three minutes to communicate a concept, get lots of feedback from it. No code, no engineers engaged. Really cheap, really easy to do. It's okay to fake it. So don't feel bad about it. It's great to get feedback. We saw some great examples of that yesterday as well in some other presentations. Build a toolbox. I showed you some of the tools that we have. I haven't got time to show you all the tools, but those are some of the tools that we have. So have a think about the tools that you can build for your products and use the right tool. It's not always good to use high fidelity designs and prototypes, because if you don't want feedback about the design, then don't do that stuff because you'll get someone going, oh, why is that button the light shade of red and they want it blue? And if that's not what you want feedback for, maybe use a low fidelity tool. So have a think about the right tool for the right situation. So in case you forgot, conclusion, build measure alone. The faster you learn, the better it is for you guys. The more you can make sure you're building the right thing. Reduce friction. I would give you several tips and techniques for reducing friction, making the experience fun and engaging through game mechanics. Get personal, spend some time getting to know your customers. Spend a couple of minutes running a feedback strategy. It's not really that much work. And think about how you can get feedback well before you write any code way over time. So thank you. I'll be at the back for questions. Thanks guys.