 So, thank you guys for coming today. All of you, you're awesome. What we're gonna be talking about today is kind of like how we use feedback throughout the course of a product to make it better and that can be before the product launches and it can be after it launches as we iterate in as we're here at Jill Sprint's. But before we start, I thought I'd tell you a little about myself. I run a company in New York City called Charming Robot and we are a product design studio. So we work with really large companies helping them kind of strategize what it means to be a business on the internet even if they've been around for 50 years or 100 years. So I've thought about like what does it mean to be a media company in 2019 for like the New York Times or the Wall Street Journal and creating an experience that fits what their audience wants but also what you know, they need to do for a business. We also work with smaller companies that are startups things like Hulu, which is kind of a similar thing to Netflix in the United States and Japan. Things like Foursquare, which was a big thing about 10 years ago and now it's probably a little less relevant. And I also do a podcast called Story and a Bottle, which is about learning from people who are practitioners in the world of tech and engineering and user experience and design. And why I mentioned that is that it's also something that is a product that we've been working on for about four years and are constantly, you know, getting feedback for and hopefully making better the week episode that we launched, which is about every Wednesday, give or take a few depending on who we have in the can. But definitely worth listening to if you want to learn more about kind of the product process and how how some great products are created. One of the things that's going to be a little different about today as we talk about feedback is that you're actually going to be giving me feedback. So over the course of just to fit it past these back to each other over the course of this workshop, I'm going to stop intermittently and ask you guys what you may have written down on a card. So hopefully you have pens. If not, we can get you pens. There's some of the table right here. And so if you have a piece of feedback for me, it could be I don't like your shirt. It could be I don't agree with you on this thing. It could be you're talking too fast or you're talking too slow or whatever kind of feedback you want to give positive or negative, it is welcome. And I look forward to picking on you and listening to your feedback and hopefully applying it as we go. If you don't like my shirt, I have one on underneath. I can change it if that is your feedback. I'm sure these guys will love if I do that with a mic. It'll make really good noises. But I thought like when it comes to feedback, there's so many different kinds of feedback we give right and we get. We get feedback from our peers, from our clients. We have clients or our bosses. We get feedback from the audiences we develop for. We get feedback from our parents. We get feedback from all these people who sometimes we want it and sometimes we don't. So what I'm going to do is break this up into different kind of stories of ways that feedback can impact a project. So we'll start with Hulu. Hulu launched in the States in October of 2007 and my role on it was I led the strategy and user experience and design side of it. Someone else was in engineering, someone else figured out the business model, but my team and I kind of figured out what does it mean at the time to watch long-form video online, which back in 2007 was not something that existed. You had YouTube and that was about it. And so the background here is that NBC and Fox, two American TV networks, had the idea to create a site where you could go and watch TV. The thing was is that most of the tech media and most of the people in tech thought we were trying to create a YouTube killer. We weren't. We were trying to create something that allows you to watch TV anywhere, any time, anywhere, any how you wanted to, which is different than watching short-form video, meaning like to lean back experience as opponent goes to leaning in during the whole process. We're not trying to get you to watch something else. We're trying to let you finish watching what you're watching and then maybe lead you to the next thing. But the challenge with this is that you had a lot of people working on this project that didn't know anything about the Internet. You had NBC and Fox TV executives who know TV, but don't know your experience design, don't know visual design, don't understand how people think about the Internet. And so you get a lot of bad ideas. We got a lot of bad ideas and during the course of the process, we kind of, I think we made about a hundred and fifty or a hundred and eighty wireframes with things like friend-casting, which is appointment-based viewing for TV shows. The problem with it was, think about this, now the idea is watch TV any time anywhere you want to. So we have this thing called friend-casting. It's like, okay, you know, Aunt, you and I are gonna watch a show tonight on TV, eight o'clock, turn on your computer, I'm gonna turn on my computer, we're gonna watch it together. What if you're late? All of a sudden the thing breaks down, but the TV executives didn't get this. And so they, we kept giving feedback to them saying, we understand this audience, we've done the research, we're inventing this for a different type of experience, and they said, no, no, no, we have to do this, we have to do this, and they didn't listen. And so we spent about six months doing this and we came up with this prototype, which is nothing like what Hulu looks like today, is not what it looked like when it launches. But one of the things that you can't see in here, you kind of can see in here, but if you could play with it, is that there are little buttons everywhere, over here, down here, and all these buttons to do is try to get you to watch something else. So there's like, now playing, so the Simpsons, you can watch more Simpsons, there's related episodes, you can watch more related episodes, there's other details that might give you the other shows. The whole thing was antithetical to what this product was, and it was because no one at the high level would listen to what this product was all about. So what halfway through the process, this guy named Jason Calar gets hired to be the CEO of this thing called Hulu, at the time it was called Sierra. And he comes in and he looks at all this stuff and he says to me and a friend of mine who was running the project, he's like, there's a lot going on here. How did we get here? And my friend and I look at him and we're like, well, let us tell you, because it is crazy. And we went through all these things and he's like, okay, it's interesting. And we said, but look, this is what they should do. Just whatever you do, this is what this product should be. It's okay. And the next day, it was probably the next week, he comes back to the team that was working on this project and he fires everyone. He says, get out NBC, get out Fox, get out your entire team. He says to me and my friend Kevin, he's like, you two need to move from New York to Los Angeles and we're going to start from scratch. We're going to build this whole thing in six weeks. And in six weeks, we, the way we did this was we had a team now instead of 40 people, we had a team of four. It was me, my friend Kevin, the Eric, who was a CTO and Jason, who was a CEO. And we, every morning would brainstorm, what should we do for a player? What would search look like? What would a show page look like? And we would argue and we would argue because Jason was from Amazon, so he was very much about like people commenting on shows and we were like, no, who want to watch shows and you know, we'd have these battles. But what was really interesting is we listened to each other. We draw things on the board and kind of debate things back and forth until we come up with something that was stronger. And so when we launched it later on later on that year in October, this is what the first version of who looks like. Now what you see is that there are some buttons in here. Don't get me wrong. But there are all things that relate to what you're watching. You can pop it out and watch it in the corner of your screen if you're at work and want to be able to quickly get rid of it. You can dim the lights if you want to have a more movie-like experience. You can go full screen. But all the stuff that you can do about watching other episodes is either down here or more importantly at the end of this, we show you other things. The best example of this, though, is one day we were trying to figure out how to create search results for the site. And Eric, the CTO, asked a very simple question. He's like, well, what happens if they search for a show that's not on who? Fairly standard question. A null result set. And the initial reaction was, well, we'll just show them other things that are like that. So if they search for a popular show at the time was lost, if they search for lost, we'll show them Star Trek or Star Trek the next generation or Battle Star Galactica. We all kind of sat there like, doesn't feel right. And I think someone actually said that because that doesn't feel like that's not what this should be. What could it be? And we started brainstorming and no one shot the idea down. It was the most common result of the time. But then someone had this other idea that was like, what if we just told them where they could go to watch Lost? What if we could just say, you know what, we don't have this. But you can go to abc.com and watch Lost and more power to you. And at first everyone was like, no, that's stupid. What a terrible idea, which is not good feedback. And then we all kind of thought for about it. We're like, wait a minute, if our whole purpose is to let people watch TV when they want to watch it, where they want to watch it, let them go watch them on ABC. The next time they want to watch a show, maybe we'll come back here and try to find it on Hulu. And then we'll have done our job right. And so like that kind of like positive reinforcement, that kind of feedback where we were helping each other, we were listening to each other help make this product succeed in the very beginning. Whereas if we had launched with, you know, this piece of garbage, that was chock full of technology without thinking about how people actually want to use it, without thinking about the real mantra of the, of the product, you know, I think I think it would have failed a lot more quickly. Now you could argue that it had content that people wanted. So it would have been around for a little bit. But every detail from, you know, how do we surads up, you know, what do we do about search results? What do we do about show pages? How do we get people to watch other things they might want to watch? All those, that thinking was about brainstorming together and listening to our feedback. So we're going to do a little exercise right now. Does anyone have any of the pads that they give out out there, any paper? Or you can use your index cards, actually. We'll get some more. What I want you to do is I'm going to do an exercise where individually, oh great, individually, you're going to draw and put your phones away, put your computers away. I don't want you cheating. You're going to draw Facebook, as you think about it, from memory. Is everyone here on Facebook, by the way? You're not? What, you're not? Okay. Is anyone else on Facebook? Okay. We'll think about it. For those of you who are on Facebook, it's going to be interesting to see you try to get feedback here. But, or draw, if you were on Facebook in the past, draw what your Facebook experience was like. So we're going to give you one minute. Like this should be something, if you aren't on it right now, you're never there. But if you are on it, you're probably there every day. So give it a shot. There are mostly not wrong answers here. So you'll probably be all right. That is okay. Okay, no worries. Oh, it's totally okay. So shot in the dark, you figure, you know, you get most people on it, it's probably fine. The point is to kind of draw from memory as to what you see. And, you know, this doesn't have to be a work of art. It doesn't have to be pixel perfect. We're looking for sketches, you know, vague ideas. If you want to put detail in there, great. If you want to like quote with on your timeline, that's fine too. It's a little weird, but you know, whatever works for you, you have about five seconds left. And then we're going to say about pencils down. All right, pencils down, pens down. Stop drawing. So here's what we're going to do. You guys are, you guys are pretty honest group. You guys pretty open. You guys talkative? Good, you're going to be. Because what we're going to do is we're going to pair up. And you're going to share your drawings with each other. And you're going to tell each other, hopefully, what's good about it. But what you're going to do is try to help that person improve their drawing so it's more accurate to what Facebook accurate looks like. Whether it's mobile or desktop, doesn't matter either way. Based on your memory. And now, again, no peeking in your phones. This is about thinking from memory and trying to, you know, help someone do a better job. So you have five minutes to do both of your drawings. And you can start that now. Did you draw anything? Oh, no. Well, maybe she can help you. Oh, it's not Facebook, right? Well, join into another group. It's fine. There can be three people. Yeah, join with them. You guys draw Facebook? Okay, to start notes. Okay. And do you have one? Do you use it? Do you use Facebook? Do you use it? That's okay. Why don't you help him? Yes. Oh, great. Perfect. Great. I want to see that drawing at the end. Me too. I had to take screenshots. You know, are you guys working together? But on mobile or you don't use it at all? Okay, that's fine. But you guys help each other out a little bit though. Perfect. Great. Awesome. Are you guys doing is working it out together? Are you guys helping each other fill out things? Okay, you got a good start there. It doesn't look a little bit like LinkedIn. Maybe LinkedIn would have been good for this conference. I don't know. How are you guys doing? Yeah, you guys helping each other? Great. Perfect. I might call on you. How are you guys doing? That's okay. But this is going to be interesting then. Oh, really? Okay. Well, this is interesting to call it by memory. I mean, I think you guys are you're kind of getting it, though. I like it. You help each other out more. This is great. You got about a minute left, guys. How are you guys doing? You guys helping each other out from the drawings? Are they all starting to look similar? This is going to be interesting. Alright, I'm like bringing you guys up then. This is good. This is good. You want 30 seconds to wrap everything up? Aunt, did you learn a lot about Facebook just now? Good. Good. I bet. All reasons why you don't want to join it. Oh, good. Good. All right. Time's up. Time's up. Who here drew desktop versions of Facebook? Who here drew mobile? Anyone got mobile? You guys? Okay. So let's do this. I'm gonna have you guys come up and we're gonna talk about desktop. It's a workshop or not just not just about me. So this is what Facebook looks like today. This is a screenshot I took yesterday. So it's pretty accurate, I hope. And so let's talk about your drawings. How how close did you guys get to to this? Okay, okay. Okay, okay. It sounds like we had a lot of lousy Facebook users in here. We're just gonna make this super fun. Because we're just talking about Facebook for the next 85 minutes. What did you guys talk about when you you're looking at each other's drawings? How did you guys critique them? What other feedback did you give? Okay, but I remember that the feedback I wanted to give you is that exactly what you're saying, what thoughts that I will kind of remember. Got it. No, no, that's great. Perfect. Love it. Excellent. Good. Good job. Can I have you guys come up here? You guys look like yeah, you two, you're working very closely together. You guys did mobile, right? Or one of you did mobile? Great. This is gonna be great. Okay, so just for those of you don't know, Facebook mobile, this is what that looks like today. Very different than desktop, obviously. So let's start with the mobile stuff. So when you first looked at her mobile drawing, what did you what was your reaction? What did you say to her? That's totally fine. Like I said, this is not about being pixel perfect. And it's about helping each other. So what like what kind of stuff did did you change in your drawing based on what you guys talked about? Okay. Okay. Now what your desktop drawing? How did you guys work on that together? So it was a lot of details that just came out of it. All right, great. Awesome. Cool. What? Well, for you guys, was there anything like you when you look at each other's drawings, you're like, Oh, my God, I want to put that into mind. Yeah. And did you do that? Okay. So the whole point of this is like to listen to you, like you said, you guys listen to each other and kind of even if you hadn't been on a while, you kind of thought about those details, like that kind of feedback, where you're open to it and able to like quickly adapt to it. That's something that we do all the time. Like we'll often sit there and like, I'll go in with a with one of my team members, like a UX person, and we'll be whiteboarding things or that we'll come with drawings, and we'll look at each other. So we'll like be like, I want to steal that I want to steal that. But it's all about like positive reinforcement. It's all about like, I really love this thing that you did. And I want to you want to use that. And that or that thing that you did really fits with the strategy. I think that like that would work well over here. And during the product process, or even after product launches, being able to kind of stay positive that way and really listen to each other is something that makes things work. Because you know, who doesn't like to listen? Facebook. Facebook in 2009, this is what it looked like. And the big thing was that they had this newsfeed and everyone was like, you know, a gas that this this invention had happened. Now the thing is that Facebook likes to make changes. They make them all the time. And it changes their look again, and again, and again. And again, here we are today. And you know what what happens is Facebook changes it so often that everyone who uses Facebook gets mad at them and sends them all this negative feedback. And people for the most part, maybe not people in this room, but people for the most part keep using it. And they keep using it because it's just something you have for a lot of people have to be on their grandkids are on their kids are on their sisters, brothers, friends. And so it's kind of become this thing where Facebook can kind of do whatever it wants. And doesn't ever have to listen to people. In the kind of a Steve Jobs kind of way where Steve Jobs didn't like to do research, he was like, I know what people want, which is none of us are Steve Jobs. Like, like, it's a dumb way of doing things like listening to people can be really effective. And in Facebook, not having to do that has created this this theory out there in the product world, we don't have to listen to people either. And we can just do what we want. There's a cycle of tech crunch in that covers the tech industry was founded by the guy named Michael Arrington. And they used to do a lot of redesigns. And the thing about redesigns is that everyone hates them. Like, every time it was redesigned launches, people are like, this is the worst thing ever. I'm never coming back to this. And so tech crunch would do a redesign. And people would be like, I'm never, I'm never coming back to this. This side is garbage. It's terrible. And how dare you change how I read my news. And so Arrington once put as like 2012, I think, published all these comments from people in an article, like this is what everyone said about our redesign, why everyone hates it. That's all those comments that I kind of just quoted in the stream. And at the end of it, he says, and those are the comments from our last redesign and no one left. And so like all that negative negativity didn't really make a difference. And I kind of think it's true when it comes to feedback in general. Negative feedback doesn't help. It doesn't make someone inspire, it doesn't inspire someone to want to change something to iterate. And we're working digital. Like if we're not iterating, we're dead. The internet is never done. And so any product we work on should be poising itself to get better. And the way we get better is by listening to each other and to our audience. And if our audience only has shit things to say, then we're not going to be able to make any improvements. Speaking of which, it's time for some critiques. So did anyone write anything down on a card that they would like to share about me? You don't have to have written it down. You can also just shout it out. One's very quiet right now. Speak a little silly. Thank you. That's a really good piece of criticism. I get that often. I will definitely start speaking more slowly. And by the way, if I don't, call me out on it. I deserve it. So mostly in this talk, we're supposed to be getting at what happens after you launch. How do you use feedback to become better after you launch? So first of all, let's just put it out there. Launching anything is hard. So if you put something out there, who here has launched a product ever? Yeah, right. Good job. It's hard. It takes a lot of work. It takes a lot of people sometimes. It takes money. It takes whatever. Like, you got it out there. We work with so many people who often don't ever get it out there that that's the first step. But once it's out there, then what do you do? Like, how do you move forward? Because I don't know about you guys, but some of the stuff I put out there on the internet is pretty terrible. And I think what Reid Hoffman says is that like, if you're not embarrassed by basically the first time the first thing you make, then you're not doing it right. The internet is full of first attempts. The internet is full of first attempts I've been working on since 1995. And they're out there and they're in the wayback machine and they're embarrassing. And sometimes I look them up and it's a really good way to remain humble because they're not good. But the thing is that when we put things out there, they have to evolve. And one of the things that we often find happens is that people get kind of these mindsets like, well, I want my product to be this thing and I want it to get here. And I have this clear way of getting here. Therefore that's the only way I can do it. But that's not necessarily true because one of the great things about launching something is you immediately put out in the world and learn what worked and what didn't work. And one of the things that we always tell our clients on our kickoff meetings were like, listen, if we are 70% right, we are winning because we are going to be wrong about things. It's so true every time we do a design that like we do a lot of research, we look at data, but sometimes we're going to be wrong and let's learn from that and let's adjust accordingly. One of the biggest things about product design is that people's habits change. Tech changes all the time and so we have to adapt to that stuff too. So we have to be willing to look at our road map and this is like agile is such an important process so that we can say, okay, in these two weeks we're going to do this. This thing is scheduled for six weeks from now, but you know what, we can move this earlier because it's all of a sudden become a priority. And sometimes that priority is because someone up top, that was a really important thing. And sometimes it's because we realize, oh my God, users just want this thing so bad. So let's talk about how we do that and how we do that from a building block standpoint. So I have two stories about this company, the Block. It's called theblockcrypto.com. If you're into cryptocurrency and blockchain, it's supposed to be one of the go to new sources for that industry. So the Block is an interesting product in that it's a brand new news site, new brand launched in September of last year. And we did UX design branding coded, my company did all of this before it. But one of the problems was the founder of this, for better or worse, he wanted to launch. Unlike the founders who have launched paralysis and are constantly adding features, he was like, I just need to get this out there, I just need to get this out there. And so one of the things we had to learn was how to cut. And the way we learned how to cut was by listening to what people out in the world really wanted from this, because he had been, he got a huge Twitter following up from starting this brand, he got he created a private Slack group. So he we already had kind of feedback coming in about what people want from this. But we designed, you know, this is just the homepage, we designed a lot of things to put a launch with. So on this one page, which is kind of a read, the whole the whole design is to be like, you don't have to give or go into an article page is like you have, you know, your your coin prices up here and a link to a marketplace for coins, you have different types of posts long form and short form, you have recirculation, you have sponsored posts with that orange one there, you have coins tied to stories and you have related stories, a lot of stuff going on here. And what we realized, because he wanted to launch, you know, within five weeks, as opposed to like 12 weeks of when we finished design, we were like, we need to figure out what matters. And when it came down to it, and thinking about who would be the way we had, watch anything you want, anywhere you want. This was all about consuming love about the content. You know, it doesn't matter, like half the stuff on this page. So when we launched it, you can see side by side, we just took away stuff. We took away related coins, we took away with the kind of navigation that you can't really see right here, we took away this top hat, and just simplified. And oftentimes, that's what feedback is really all about. It's about simplifying and our audience is just like, we just want to know the news, all this other stuff is great, blah, blah, we can get it elsewhere. But this is what we want. And so we created a plan where over the course of, you know, three or four months, we could slowly start adding things in. But, you know, the road map also changed. So, you know, a month later, we did add in that top hat, we added in, you know, some navigation. But you don't really see us adding in those side pieces. Instead, we prioritized something else, which was a paid product, which came up in the middle of the middle of the night, I think, when I got an email from the guy who founded this being like, we need to make money. We need to start a subscription service. We need to get this up in two weeks, change the road map. And so that was feedback he was getting from his investors. Like, what is your, what is your product going to do? How are you going to make money? You have no advertising. And you have a really big audience. So within, you know, three or four weeks, we rebranded this, this thing on the top here, which is basically their paid product. It's called the Block Genesis. And it launched at a thousand dollars a year, which I thought was ridiculously expensive. And, you know, within the first couple days, they had like 60 people paying, you know, a thousand dollars a year, which is not a lot of people, but that's 60 thousand dollars a year. And then that went up to a hundred, then went up to 200. And, you know, it's stuck with his strategy of I want to make this above the news. We could, all the bells and whistles, we can think about later. But that feedback from him was really helpful. Here's an example of something that was not helpful and how not to get feedback. He, I was on, I was at a conference in Arizona in like, July of last year. We're in the middle of the branding process, designing the logo for this and the brand and identity. And I get a call from my creative director, my business partner. He's like, you need to go to Twitter right now. And I'm like, Oh, and then Twitter, I want to, I wish I could show you this tweet, but I went back on Twitter yesterday looking for it and it actually was deleted because he had posted up on his, and he tweets all the time. He gets kicked off Twitter all the time too. He posted up of decisions, decisions, and it was the final two logos that we were down to. Now, that may not seem like a big deal, but in the world of design, like, you don't share out the design process because everyone has an opinion and not everyone's right. You know, and they don't know the strategy behind why this logo might be better than another logo and maybe the other logo would have been better. Who knows? But you don't crowdsource that because if you do, you lose the whole point of doing an exercise like that in the first place. And so I called them on the phone and I'm like, listen, man, like, you can't do this. Like, you can't crowdsource your logo. He said, yeah, but I'm so excited. I'm like, I'm excited too. I'm really want this to launch. You're going to get people, and he did, start having conversations and start to like break down, you know, why this logo is like this and why, why is it blue or why is that shape? And I don't think it should be that shape. And then they're going to start offering their own logos. And suddenly you get this like huge cycle of feedback on Twitter that is not useful. It's just noise and it's noise that gets in the way of understanding, you know, what, what should this product be? Why are we going this direction? So, you know, he, he, by the way, he did this twice. I want to be honest with you, he did this once about logo. And then like a month later, another call from my creative director is like, you have to check Twitter, Twitter. And I'm like, okay, here we go again. And this time he had posted our entire brand book to Twitter, being like, I'm so excited to have the brand done, like, look at how great it is. And the problem with that is a couple of things. One is you shouldn't do that. But the bigger one is that in the brand book, which is a document that is internal to the company, something that is meant to like, I can give you and say, this is everything you need to know as an employee. It has things like, here are our competitors. Here's what we think about our competitors. Here's why our competitors are suck or whatever, you know. And, you know, you don't want to put that out there on the internet for everyone to steal. And so he's like, I'll take it down. I'm like, it's already out there. Like, don't you understand how the internet works? So again, not, not a good way to really get feedback from your competitors when you're just telling them how bad they are. But I think that the idea of, of taking a, taking feedback from an internal team so that you can launch more smartly, even though you've done all the work is something that we also try to think about. And oftentimes, by the way, this comes from the engineers who look at our designs and they're like, oh, no, what did you guys do? Like, this is too much. We can't launch them in this date. We have, we have a, you know, a deadline. The way we sell that is we bring engineers into all of our design meetings so that, or we review the designs before with engineers so that they can give us all the feedback so that when we go, we go to a client or we go to something internally, we're not presenting them that can't be done. So that kind of feedback internally before we get it, can we get out there? It's so important because the last thing I want to do is make a promise that we're going to deliver all this stuff. And then the engineering team gets like a stack of designs and say, this is going to take six months longer than we thought, we thought it would. But after you launch, there are different ways of getting feedback as well. And so one of the things I've divided this into is analytics and people. And so analytics, this is just a handful of examples. I'm sure you guys have used Google Analytics or crazy egg or sale through or whatever. These are all just ways to measure success. And these are really only good if you know what success means. Right? Like you could go, this is one of my big problems with Google Analytics that they've gotten better is when you go there, they're like, here's how many pages you have. Okay. Is that good? Is that bad? I guess compared to last month, that could be better or worse. But like, what do I need them to be? You know, with something like chart feet, you can see how far someone got down the page or with crazy egg, maybe I can see where they tried to click. And I've used these kind of things for in this one type of feedback for redesigns. I worked on a redesign for Billboard magazine a couple of years ago, their music magazine been around for about 100 years and they have a challenge in that they're not very relevant. So they're very much kind of quick bait now. So you go, you find them on Google or Twitter or whatever and someone wrote an article about Lady Gaga. So you go to this article page of Lady Gaga. So they wanted to increase their page views by, I think it was like 50% or 60% and they didn't want to pay for user research. So all the users on the side over there that we haven't talked about yet, didn't care about them. They're like, just figure it out through the data. I'm sure it won't be a problem. So we use crazy egg, which is a heat mapping tool and Google Analytics combined to figure out like what, like what can we find here to make this happen? What we did find was that they had all these articles on Lady Gaga and if people went to an article about Lady Gaga, there would be a big, big glowing click cloud around her name in the middle of an article. But there was no link there if you were just trying to click it. So we did some more testing and we found in other articles, everyone's trying to click on celebrity names and there's nowhere for them to go. They had no artist pages on a website built around the music industry. So we're like, here's an idea. Perhaps you should make artist pages or just tag pages for artists and then, you know, put links to them in the stories. And they're like, well, that's, that's a new type of page. We're like, yeah, it's a little bit of effort, but let's just try it. So we redesigned the article page to have links in it and actually highlight who is in the story if it was more than one celebrity. And just that change alone, through analytics, through the feedback of click, click clouds, we were able to change their business. Their their page views went up 98% from what one change, which is ridiculous. But so that's one thing on the user side, you know, we go and do a couple of different types of research, we'll do ethnographic research where we go and we we talk to people and in their houses and their environments and try to understand kind of what they're all about and and see their frustrations firsthand. Sometimes we do usability analysis and we like, you know, we look at them using a site and see specific specifically how certain paths go. Sometimes we'll do surveys, though I'm not a huge fan of surveys, because they're binary and you don't always get the right questions. There's a wrong way to use a research as well. And some people like it. Some people like, I guess it's a guerrilla research. I am not a fan of this because I think the lack of context around design can be problematic. It could be an example of that. A few years ago, we had a client in the personal finance space, very not warm, fuzzy personal kind of product, very much a banking kind of product. And they, they were really obsessed with the site being beautiful. They were like, we just want to have the most beautiful financial services site in the world. What the hell does that mean? Like beauty is subjective. And so we got through the UX process. We got through half the design process and we go to a meeting with them. We say that and we're like reviewing designs. And it's very tense. And they're not like being mean and they're not being insulting. They're not giving any feedback at all. They're just sitting there silently nodding like, yep, as we prevent things. And finally, my colleague says to them, I'm sorry, is there something wrong? Like everything was really going well last week. What's, what's different this week? And they said, yeah, you know, we did some testing or the week to see if, you know, the designs resonated with our audience. And, and you know, we just, we don't feel like it's really going in the right direction. We go, okay. What was the problem? What kind of research did you do? And they said, well, we did the caribou method. And, and then, you know, we talked to people. The problem is that I've been doing research for let's say 15 years. My colleague, about 10 years. Neither of us had heard of the caribou method. We also were too afraid at the moment to say, what is the caribou method because they didn't want to feel stupid, you know. And so we, we listen to them for a little bit. And finally, my friend says, look, I'm so sorry. I apologize. But I just don't know what this caribou method is. Like, could you, I'm embarrassed. Like, could you tell me what it is? I'd love to know. And she's, and the client says back, well, what we did was we took the designs that you gave us, and we printed them out. Now here's the first problem. You don't print out designs meant to be looked at on a screen, they will never look the right way. Then we took, said printed designs, and we went downstairs into the lobby of our office building, where there is a place called caribou coffee, coffee shop. And then we went up to random strangers and said, is this beautiful? To which the random stranger, having no idea what this was, was like, no. And you know what? They were right. Because it's a personal finance site. It's not supposed to be beautiful. So we got fired. And that actually is true. We did get fired. It was better for them. But that kind of feedback isn't helpful, right? Because with no context, either you're going to say, no, it's not beautiful, or you're going to try to find something. People want to be helpful. They want to look at something and say, yeah, I mean, I guess it could be beautiful if you did this or, you know, that, that image is beautiful. The rest of it, I don't know. And so without context about what you're trying to test and what kind of feedback you're trying to get, you know, I, I think that, that we want to, you know, we want to kind of set ourselves up for success. So thought we'd, we'd try this a little bit where you've listened to me going to go on about this. I want to you guys to pair up again and do some little have a conversation talk about some things that you've worked on where either analytics or user research either helped. I love it anecdote about that or where you would have liked to have that feedback and why it would have helped. It's five minutes, jot down a couple of notes. I can't wait to hear what you guys love. Talk to each other. It can work either way, but it's something that you've worked on where either you've used analytics or use the research to make it better or where you would have liked to have. Yeah. Yeah. In any particular example you can give. Hopefully you're each telling each other stories. So should we wrapping up your stories about now? I am going to call on some of you. Don't you worry. I was like, I was like, you really don't want to be called done. I'm good. Thanks. Oh, right. You guys are talking a lot. I love it. Love it. So you guys are talking a lot. I want to we're going to start with you guys. What were you doing? What kind of came up around analytics and people that has been relevant to you guys is branding right there. Yeah, totally. Yeah. Yeah. Great. Awesome. And went back there. That's that's that's totally right. Back there. You guys are talking a lot as well. Yeah. I always say that like the analytics can tell you what's going on, but they can't tell you why and the people always kind of do give you that y axis. Yeah. Right. Right. That's a great example. Absolutely. That's why I love usability testing for that reason. You you the hardest thing with usability testing is watching somebody take a usability test or go through it. You just want to be like, no, it's right. It's right there. It's right there. Come on. And when you're moderating it, it's even worse. This is great. You guys you guys totally get the value of both of those things, which is something so important. We we we often see one or the other and user user research is always something people are like, well, we don't need to do that. Like I I use the internet. It's like one of the things we I think it's somewhere later in here. I say like, you are not the user. You know, we are users. We are we are not the user. We could be a user. Right. So looking at how people use things and think about things and what their motivations are, like, how can we make decisions without that? And we'll talk a little about that later too. So one of the hardest things is in your own head when you're creating something is looking at what's hot out there. So over the years, there are things that have been hot that like people come to us and they say, I want my home page to look like Pinterest. And I'm like, why would you want that? That is ugly and not usable, but but works for them. And right now, the big thing is Bitcoin or actually any sort of cryptocurrency. Everyone wants to have a coin. And it's like, why? Is it is that fit your brand? When I was seven came out, you know, everything went from scumorphic to flat. And that was which I think was a good thing personally. But like, it was very much like chasing these trends. And, you know, one of the things that it's important to do as a team member when working on a project is to is to check ourselves. Are we doing this because we're doing it for the right reason for the product? Are we doing it because someone else is doing it? My favorite example of this and this is where so many organizations did not listen to feedback is is this one right here? The carousel? We've all seen that on a site where like you get to a news site or something in it. You have one thing and then slides to the next thing and then it slides to the next thing. The carousel is the biggest failure in the history of the Internet, my opinion. And everyone tried to use it because someone was like Yahoo! News or something created it in the mid 2000s because the worry was a news organization that nothing not enough stuff was above the fold, you know? And so like well if we cram all these things behind a carousel then you know every every type of article could be represented. You could have a sports article, a news article, an entertainment article. The problem is that a carousel doesn't work. The carousel assumes that someone is looking at their computer like this. Oh shit I clicked the wrong thing. And the reason why people don't do that is that the Internet is an active media. So all these news sites were like I get my stuff above the fold and what would happen is people would click on the first article 80% of the time. Maybe like you'd get a drop-off down to like 20% of the second article and then nothing after that. But everyone kept doing it because everyone else was doing it. And this is where like looking at feedback is one of those interesting things because all the analytics told you the story. You didn't even need the why. It's like yeah the drop-off is here on every single carousel. It's very very clear analytics. But people kept doing it because they thought it was working for someone else. It's like oh if New York Times does it then CNN should do it. If CNN does it then you know Time Magazine should do it or whatever. And that the case is that looking at what other people are doing doesn't help you. It's not feedback at all. In fact it's much like the negative comments from the tech country design. It's just noise. Focus on what you're doing and look for feedback that's going to be useful and actionable. The only thing around design feedback is that it's those little things those little iterations that we can learn a lot from. And unlike you know Facebook who doesn't like to listen to its audience there is a company that does and it's Amazon. Now Amazon has many problems. I'll admit this. But what Amazon did really smartly was use analytics to understand use analytics and usability I'm sorry to understand how they can evolve. Just one little piece. And for those of you don't know Amazon does about 98 AB tests on each page of their site every single day. Like every pixel being tested. So I want to talk about one part of Amazon. One part that has changed significantly in the past 20 years. And that is the Amazon navigation. So when it started actually it was even less navigation but let's you know give them some benefit. So in like 1999 or 2000 Amazon started selling things beyond just books right books music videos whatever. And they had this tab system and I don't know who was around at that point. But let me tell you on the tab system that Amazon created. Everyone thought it was amazing. And they tried to patent it. It did not go through. But they tried to patent this design. And the interesting thing about it was it works. You know you have you know six items. Maybe eight items and I'm pretty good. And then you start adding the more things to get a little more complicated here. Adding rows and then rows. It doesn't scale right. There was at one point. I don't have this on this slide anymore. But at one point there are actually three rows of tabs here because they had so much stuff. And then they were like OK we're getting a little busy. And this is you know this is from feedback internally that you actually will really talk about this. And and so they there we go. They went to a one tab system. Now I don't know if you guys remember this but if you clicked on all categories back then this was a terrible idea but like click on all categories. You get this huge modal of like 30 things. You're like well I don't know what to do. Close it. Let me close it. I'm just going to search. So they realized very quickly some analytics that wasn't working for them. But search started to become more and more important. But what they did was they realized they needed a more scalable navigation. So then they went to a left handed navigation. Great. You can stack things put them in bigger categories have dynamic menus that come out seems like a good idea except this got longer and longer and longer and longer. And people said I don't I'm never going to use that. It just it doesn't work. So then Amazon went and said you know what I'm just going to do this. It's going to be a Google just a search bar. That's it. That's all you need. And you know what's great is Amazon could do that in that was 2013 or whatever. Because at that point they'd been around long enough. They'd heard everything from everyone about what worked about their product and what didn't. But they said we can we understand our uses well enough to know that if we have a search bar here. We don't need anything else. And that's pretty much how it stayed till today. Same thing. You go to Amazon. This is probably a month or two ago. Same thing just a search box. Here's remember that tracing trend slide a couple of minutes ago. Here's what happens now if people come to us or anyone want to create a product like well I'm doing a store so I don't need navigation I just need search because Amazon does that. It's like yes Amazon can get away with that. But imagine if in 1997 Jeff Bezos launches a bookstore online with no categories and just a search bar buys and books look for what you want. It wouldn't have worked. It had to evolve and evolve through iteration and that iteration came from understanding you know how people want to use this listening to how people use it watching where they're failing that big model for example. And then moving on speaking and moving on let's try this again. Anyone have any feedback from me. Am I speaking better now or am I going too fast still. You're you. Fair enough. Any other feedback. Good bad. Thank you so much. Great. Awesome. I like positive feedback. We should be more positive on our feedback. Shouldn't we. Isn't that nicer that you put something I designed things for a living right. You can probably relate to this like you put something in front of someone and all you're doing is looking to be criticized. Like you spend hours and hours working on something and you're like cool. Tell me what's wrong with this which is good. It's good to group it. But it's nice to have a positive comment too. I always say like my clients and like any lead with the positive and we get better. So now we're going to talk about the process of feedback. I think that feedback is such an important part of the conversation and it can come from anywhere right. You know we I've mentioned engineers earlier when we're in design meetings. I want project managers that are UX people designers clients subject matter experts like I want people there because everyone is going to have a lens onto this product in a way that I don't. Right. So I might come out of from a design standpoint I might come out of my usability standpoint. But I don't know the difficulty from an engineering standpoint. I also don't know what you've done before. So a good example of that I was designing a fashion site for this fashion designer in America called Nicole Miller. She's known for like kind of pattern dresses. Doesn't really matter. But it was an e-commerce experience and we had designed this like one page checkout and we were so proud of it. We were like you know you can get through it in one page it's like next next next you're done. Check out boom. And the engineers like that and they're like yeah we can build this but it's not going to work. We're like what do you mean it's easy. Like yeah we we built that before we tested it. It fails. This is the one that works. This is the checkout process that always succeeds. And so we looked at theirs and it was more than one page but it was simple. It was very easy to get through and they were totally right. And so not having them in that room would have let us down the wrong path. So having those different opinions are important. The child the challenge is that like most of us when we have opinions think that they're think that we're right. Right. We have an opinion because it's what we're trained to think. So it's understanding who to listen to when to listen to and how to apply that that we really have to do. And that's someone's job. It could be my job as the as the UX lead on a project with the client's job. But oftentimes it's someone who has to understand this feedback makes sense at this time and this feedback does not which is hard sometimes especially like you have a bunch of really smart people in a room. One of the other challenges is that everyone has ideas and we can keep spinning our wheels until you know we don't we don't actually get anything done. My my favorite example of this is some I worked on this project got a muscle in 10 years ago. It was a virtual don't really virtual worlds like yeah. Yeah they were good the second life whatever. So I worked on this virtual world. Here's the thing. It started as an online poker game which you wouldn't expect but it was just a poker game like any other online gambling poker game. And you could go there and play against other people bet money simple enough but the people who were working on this had a lot of ideas and they were like we love online poker but everyone can play online poker over here. What else can we do that's kind of fun. And so they said well what do people do when they play poker. Well they talk to each other. OK so we'll add a chat function but that's not really interesting. What else do people do. Oh they listen to music. Let's put the ability for people to DJ with each other while they're playing online poker. So we're playing poker. I put on some Lady Gaga and you're like yeah I don't really want to listen to that. I'm going to put on you know some some Bauhaus and I'm listening to Bauhaus and and then you're like I don't like Bauhaus too negative. I want some Britney Spears whatever. And so that was interesting and they add that in and that should have really been the end of it. But then someone said this as this question this gets down to focus and feedback. Not what do people do when they play poker. What do people do when they listen to music. And all of a sudden the virtual world was born where it's like will they dance. So we got to let people get out of the poker room and go into the club and dance. Obviously. And so they built a dance room and then they're like well what do people do when they dance. They want to hook up. This is true. And so they made like male avatars and female avatars. You can filter out the male or female depending on what you're interested in. And then you could go talk to them. And then someone said well what if I don't like the music in the club because you like Bauhaus and you like Britney Spears and I you know like everything. I just want to see it all. They're like well we should have different floors in this club. Not with the poker rooms over here. So we have different floors. We have like the pop floor the country floor the you know industrial floor or whatever. And so they have now they have so they have music and they have poker and then they have like now we only have poker so we need other games. So we have poker we have shuffleboard we have whatever. Now they have this gaming center they have a dancing center. They have a media center where you can go and watch YouTube videos because you could just go to YouTube but OK. By the way this product has not launched at this point. It's still just one big flash by all that takes about 20 minutes to download before you ever actually get to do anything. And finally we said and this is I think the best piece of feedback we gave them we said you should launch you should put that out there and see what people think of this thing that you've made. And they said no no no we're not done. We have one more thing we need to put into this. We said I don't even have anything to tell you I can't imagine what else you could put into this club that you've made. And they say the guy looks me straight in the eye big South African guy he's like books. I said you want people to read books in your club. Well people up playing poker and dancing. Say hey you want to stop dancing go read a book. And he said back to me. No I don't want people to read books. I want people to write books. I was like don't they have Microsoft Word. He said yes but they could write a book chapter and then like you could write a chapter and then you could write a chapter and you could collaborate on these books together and have a library inside the club. And I said I am firing myself from this car. Because you're not listening to anything I'm saying or anyone they didn't launch they lost all the money in the game. And so I think if they had actually put it out there and you know what the poker game thing was lame it's true. But if I would have learned a lot by actually putting something out there and listening to people his feedback works both ways. You know what like I have a team of people. They work for me but they really work with me right like we sit together every day we wipe board and I tell them like look come at me like tell me everything I'm doing wrong both of them as a manager as a boss but also like as a designer. I tell my clients the same thing every kickoff meeting we are going to challenge you you're going to challenge us we might get into it a little bit. But it's for the better betterment of the product. And that should be true with everything we do it's like why are we giving feedback why are we helping each other. We're doing it because we want to make the thing we're making better solve the problem that we're trying to solve better. Except it's really hard. It's really hard to give good feedback and you know what it is and I don't think anyone's ever taught to give feedback. Does anyone is anyone here ever taught to give feedback. No. Yes. No. You have to be back. Yeah. Coaching. Yeah. That's a good point and the coaching is actually a good idea. So I learned how to give feedback when I was in college. I was a creative writing minor because I wanted to be successful. And and in creative writing workshops the way it works is you have like 10 people in a workshop and everyone writes a story every week except for one person. No. Everyone everyone writes a story every week and everyone gets critiqued every week. So they'll have to read all of my all the stories in the class and he has to say here's my critique of every story and I have to read all the stories too. So I'm going to read his story. He's going to read my story. We're going to be telling each other about our stories or what we think of them. The problem is is that most people in college who are writing minors or majors are not good writers. We think we're good writers. We think we're the best writers but we're terrible writers. And so you I read your story and I'm like oh my god. That was 10 pages of just utter shit. And what do I do. How do I tell you that. You know maybe it's not going to get much better but it could get a little better. Right. And so what we were taught the way that we're taught to get feedback is you know you start with a positive. You started with a positive thing. Thank you. You start with a positive like there's got to be something in here like I read some stories in college that I was like I don't know like there's not there. That was a good adjective. You know but really it's hard. You've got to find those things because I think our knee jerk reaction is to want to fix something. It's want to tell people like this is what you need to do right. Not in the collaborative way that we were doing it earlier with the Facebook drawings which I think was nice that you guys are helping each other find the details. But in like the this is what works here because first of all someone spent some time working on this whether it's engineering whether it's design whether it's writing you know put we do this for work but you know it does take energy it does take thinking and so people should be at least told that like we acknowledge that you did this and this is needed well. The other thing is all you know be actionable. Oftentimes we give feedback and it's not particularly actionable it's just an opinion that we put out in the world and it doesn't actually give you any direction to go in. I'll give you some other examples of feedback that doesn't work. I don't want to see it. Not helpful. I don't know what you I don't know what you're seeing I don't know what you want to see. A little more detail would be useful. Also I hate that. That's unfortunate you know but some context why do you hate it. Is there some weird like psychological thing inside of you that made you hate it or is it just wrong because it doesn't achieve the business goals. Any of those things could be helpless to figuring out how we get to the next step. I just don't like blue. That's OK. You know what maybe you're colorblind maybe you had a bad experience with blue as a child. I don't know but that doesn't help me move forward with the next step. And actually any noises at all are not useful feedback. We had a client once. This is amazing. She came in and looked at designs in my business partner stand there side by side and like excited it's first design review and she looks at it she goes you know it's a but I kind of want it to be a my business partner Chris looks at it this one and he's like I'm sorry that is a single worst piece of feedback I've ever gotten my entire life. It's not helpful. The thing that's helpful is is offering solutions if you can find a way to translate what's bothering you about that particular design into what could make it better. Now that doesn't mean you have to find that like don't have to solve the problem right like our job is can go and like go back and like listen to what you said and figure out how do we translate that into an actual like design piece but to understand why it's not working for you. For example you know I really wish this could flow a little better because it seems to me that it would take too long for someone to fill this out that would be a good piece of feedback or something like you know I really want this to this this piece of content to be more prioritized and attract someone's eye more than that one those kind of things are much more useful than you know I just don't like this sir. Also there is a fine line between being honest and being insulting and I mean that in the you know the most friendly way possible because I think oftentimes when we're trying to help people we are being honest but what we can cross the lines and I have certainly been guilty of this and been called out on it by my by my my team I'll say something like you know I think this is actually a step backwards but by the way is not useful piece of feedback it's like great now not even telling I did good work is saying I did worse work than the work I did before that's not really setting someone up for being motivated to do something better so thinking about ways that we can say instead of this is a step backwards like you know I think we want to re-examine how we're approaching this because maybe you know we might have it might want to start a new direction and here's a direction we can try also being clear and being contextual you know not everything that we do or put in front of someone can have context my whole thing of like I don't like blue like okay if you don't like blue that's fine but give me give me some reasons why that doesn't work for the rest of the world as opposed to just you unless I'm making this just for you which case doing like blue I'll make a red whatever works also having a real one-on-one conversation is so much better than playing a game of telephone if you guys play telephone you're telling one person tell someone else tell someone else gets distorted right it's like making a clone of something even copy of a copy it's not as good and so making sure that you are the one giving the feedback or being clear about your feedback is it's going to help make that change more effective and finally you know it's it is a conversation like not every piece of feedback is I'm telling you this now go do it I think a lot of clients in my case would like that to be the case but I don't believe in that like I believe in listening to what they have to say and and having a conversation just see what like if I can understand why they want to do something and they may will do it you know that's what we tried to do on Hulu from that earlier conversation and they didn't want to listen they didn't want to have that conversation they just want to be like make this and make this is not how great products are made speaking of making great products one more time who has some feedback for me good stuff bad stuff shirt off shirt on yeah I'm so glad you guys just positive up here yeah oh oh great I was I was afraid of trying this because I was like this could either totally fail or people will get it I don't know yeah sorry that's right yeah I think I think that's totally right finding the right moments and also that goes back to also who do you get feedback from at various points right so like let's say here's a good example so I'm working on a project right now yeah with a financial institution and they're not particularly visual people that's fine but we have to put some stuff in front of them so I have a UX designer working on these concepts that you know I'm in India she's in Hawaii so the time distance difference is pretty amazing so we're having phone conversations about this stuff and I'm asking her to show me stuff that's in progress and it's not in any way close to done right it's just like here's some ideas scribbled on a page but I know her well enough to know that like how she thinks so I can give her really good feedback on that on a let's say beginning of the day into the day process with a client I'm showing that client this thing once a week at best maybe you know if that's twice a week if it's a very fast project and they we know how to work together so I think it's not just about how often but it's how often you show it and to who on a different project I was talking about earlier how we put things in front of engineers will do a Tuesday review of wireframes or designs with the engineering team listen to their feedback revise it based on a Wednesday meeting with a client so that we are listening to the feedback from the right people at the right time before putting in front of the next set of people yeah yeah absolutely we're going to get to that kind of feedback in about a minute because because there's a surveys I think are challenging anyways but it's also important to know who you're sending it to and how you're segmenting it because if you just throw a survey out there it's kind of like that whole coffee shop example if I just throw something at you and say here's some questions you know maybe you're the right person to ask those questions maybe you're not so I think it's more about really defining who you need to be back from and the same thing is to internally and externally right if you're getting a survey every week that would seem to me unless it's narrowing down you know a particular feature or whatever service or whatever it seems to me that that would be too much you know so oh yeah sure sure that's true I don't know but if you really need to do something then there is something you need to do for the profit if you need anyone do you know why you say come on man something that I said that's okay that's totally fair but would you rather be just say look this isn't working or at least tell you like I think yeah I think it's even if you know it's coming it's good you know that there's something you did something right you know I often like whatever I'm doing a first concept review and I'm putting it in front of someone I'm like I hope that at least part of this is okay because I know that half of it's wrong you know I mean but let me let me hear something good you know otherwise the project could go south I think it's I think it's good I think positive feedback is actually useful because you know what maybe not what not to change but you know at least what's working so you can focus on the things that aren't working if you have this thing and it says nothing about this thing well all of a sudden my head's like well what do they think about this is this working is it not working and then I go down a path that it's just a nightmare and probably need to like relax a little bit but I do think that having a sense of like where you can focus on improvement is important so that's where positivity is really useful so we talked a lot about the different ways feedback works and you know I think the best balance is where they are to make the right decisions because there is no one right source and I think we all kind of kind of agree to that on this conversation I want to touch on one last thing we kind of talked a little bit about it which is the fact that we are not the user and I think a better way of putting that is you can't force people to do what you want and we all of us try to do this a lot in design you know what I mean I need to get these 20 pieces of information about a person before I can really do something with it and it's like okay well no one wants to give you that information so let's find a different way of doing it I've had that argument like 50 times like a year but but realizing that you can't get people to do what you want can open your eyes to what you can get them to do so I want to talk about one last company that I've worked with there's a company in the states which is Netflix for dresses so women can rent dresses for dates or for weddings or whatever any sort of event you can rent a nice dress for and look good and then you have to buy it you can just turn it and get a new one the next time which is very useful so we helped them launch their product in 2009 and it kind of worked with them straight from 2009 to 2016 this is their original homepage goal this homepage is to tell people we have a lot of time we have brands you've heard of here are some reasons you might want to do this that's all we need to do is get people to convince them that like maybe this is the service you want to take advantage of as soon as this project launched the founders of this company were obsessed with one thing and that one thing was why are you renting a dress we really want to know why you're renting it so that we can understand what dresses to put in front of you so within two months of launching it we redesigned to be about kind of getting you to kind of get into that renting process and tell us why there's a drop down there that says occasion and so what people do is they would come here and the goal was you're going to tell us when you're going to tell us where you are and why you're going and we're going to show you dresses that are right for you well people came to this page quite a bit and what they did was they went over here and clicked on dresses because they wanted to see all the dresses that were available to them no matter where they lived or what they were going for they really kind of frustrated the founders of this company and they were like we just want to know why they're renting it so we redesigned this page like six more times putting this form in different places and people still just want the dresses so then we redesigned it again and said okay look tell us what you're going for is it a wedding is it a date like just want to know why you're going here and I'll tell you a little secret you know what happened is people came to this page and they clicked on dresses because they wanted they wanted to know what was available to them so the thing was that like what's right for a dress what's the right dress for a wedding for you is going to be different than it is for you or you and so I don't want you to tell me what dresses I should wear I want to choose it myself and then you know and then go from there so finally of course yeah that means they're a customer service company they want you to ask them those questions absolutely so eventually here's something that the customer feedback they're getting they're getting email after email after email with people who rented dresses and they're like oh my god I rented this dress it was so perfect for this wedding I went to thank you so much here's a picture of me in the dress and they got all these pictures of people in dresses and someone was like you know it'd be interesting what if we put them on our blog so they put them on the blog and people start reading the blog and they're like where can I get that dress because they didn't link to the dress page you should talk to Billboard about that and but then one day they were like okay we'll add some links but what if we put that picture of that person in the dress instead on the blog on the actual dress page people were like wait what the models are on the dress page we can't do that and they were like we could try so they did an experiment where they put some pictures on on a page with the models and they put some that put some pages without the pictures and the pictures without with the regular people had a lift of 75% with the dresses they rented so they're like oh people really want this so this changed their business like literally changed their business almost overnight so they created a whole new section of the site called our runway which is like you could go and not just look at dresses like look at dresses in real people on the dress page you could just search by real people you could go to that page and you could search by body type by if you had bust size waist size whatever so you would feel you know like this will look good on me and in the end like their whole business is now about that and about getting you things that are right for you and that just just by listening to those people who finally email them as opposed to thinking that like we need to ask the user we need to get this from the user as opposed to you know the user want to tell us this so they've gone a little bit more minimal now on their on their home page anyways that was the last story there's about seven minutes if anyone has any more questions or answers I'd like them answers yeah so it depends on the feedback we're looking for so if it's customers the way we do it is we look for trends so if we see that like you know 80% of people are complaining about this thing or saying good things about this thing then it's something that we need to think about how how would we fix it now the follow up for that might be oh the the solution here is obvious we just need to do this like they're frustrated about this experience let's look at the analytics let's do some testing and then let's let's figure out what the solution is for client feedback it's a little bit different the way we do it with our clients especially if it's like a group of people is we assign one person to be the point of contact we say look you guys go back have a conversation because you guys might disagree about something and I don't want to sit there and listen to this guy say one thing and this guy says something else because I'm not if you guys can figure it out great if there is something you disagree on we can weigh in based on our experience and our recommendation but I don't want to sit there in the middle of an argument so figure it out first and that way we give them like 24 hours 48 hours to do that but generally that works pretty well and then once they give us the feedback we're like well you're wrong we're like okay here's what makes sense here's what we're going to address here's where we disagree and where we disagree let's have a conversation whether you're done for the business or is there a better solution or why are you thinking about it this way let's understand that and figure out maybe a better solution that we can all live with I wouldn't say which feedback is more important I think which feedback they all agree upon when it comes to prioritizing it that depends the way we do that is how will this feedback affect the timeline so if it's like this thing is going to change a whole lot of things and push this out by weeks do you want us to prioritize that or do you want us to focus on this and you know if they want us to they can do that but if they don't we'll put it over here for later and then work on the that little stuff any other questions yeah yeah right that's a good question so the way that I approach this now is I start by asking why questions because I think asking questions a little bit more open I did you start to have a conversation and so if you said to me like Dan I really like your presentation but I didn't like the fact that you used blue in that thanks I'd be like okay Bill is it is it the blue bothering you is it annoying you or what is it about the blue that is kind of affecting you like do you feel like it doesn't go with the colors and you might be I think it's too bright it's just really glaring to me and I'd be like okay so it's not the color blue it's just that it's a bright hue of blue what if I tried something softer you know I'd be like no I just hate blue but I'd really try to make it a conversation and then you know eventually that conversation leads to some sort of revelation usually usually unless someone's super like you know angry about blue which could happen I suppose yeah any last questions alright great thank you guys so much for coming in being part of this I've just been