 My name is Priscilla, head of design for SP Group, also better known as Singapore Power. I've got an approach to give a presentation today. And the first thing I did was to ask myself, what do I want to learn? And I feel like I've been to a lot of UX conferences. A lot of times it's about designing UX and I feel like as UX practitioner, we know that it always starts from the user. It always starts from understanding the user, but still there's some mystic towards it. And I still got recruiters calling me and asking me about my portfolio and design and visual design. So we need to change that. So here I want to bring awareness to user research in particular, quantitative and qualitative. And I also asked myself, who do I want to learn from? And this is where I learned that. What are you looking at? I'll do a brief introduction and then we will go straight into the question because I have a lot of questions. So first we have Kai Singh, head of Singapore, full proof. If you're a researcher in Singapore and you have never heard of Kai Singh, then, not, you need to do better. And then next we have Lishanth, co-managing director and co-founder for agency. It's a designed, human-centered design practice. Extensive experience in consulting. She was previously a design research specialist in IDO. And again, if you're a designer and doing design thinking and you have never heard of IDO then, seriously. Another veteran, we have Samantha Yuan, currently in GarfDec doing very important job of bringing research specialty into digital government agencies. And last but not least, Ramda, head of research in Gojek. Gojek is Indonesia's first unicorn. The company is valued at about five billion this year. Ramda, he's multidisciplinary team of UX researcher, both quantitative and qualitative. He's actually has background as a quantitative researcher. So we're happy to have him here. Again, if you're in tech and you have never heard of Gojek then, seriously. All right, we're going to the questions. We do have some format to give a little structure to our panel discussion because it's not just random questions. So here are the structure of the conversation. First, the difference between in-house versus agency because we have our agency researchers here and then we have our in-house product team researcher over there. So if you are in agencies, we hope we have questions that address your needs. If you are in a product team, we hope to answer some of your questions as well. The difference between qualitative versus quantitative research and how do you bring it into the organization? The future of research and making research strategic. So we have design leaders. We'll talk a little bit about how as a researcher, can you go into a strategic role? And finally, if we have time questions from the floor. Okay, so the first question for everyone. What is your favorite methodology in your research toolkit and why? Kai Singh. Hi, hi everybody. So I think this question is very difficult to answer. I guess because it depends on the need of the question that you want to really answer. So different methodologies suit different things. But my favorite one is borrowed from ethnography. So it's participant observation. So you're there as an observer, but you're also doing the thing together with the person you're researching with. And that for me is interesting because when you're doing it physically, you realize, oh, I really don't agree with how this is done. And that physical emotion that it brings gives you clues about why is your reality different from the user's reality? And you can sort of focus your curiosity on that. So there's an embodied understanding of this really quite big difference. How about Lishen? Okay, so my favorite, what was it? Methodology in the toolkit, in my research toolkit. Some background in that we work, my background's an IDO and now agency is a design studio. So we think about designing from day one. So research and design are actually coupled together. So my favorite, I think it's got a very long story short because it would go on all day about such things, is that we use this thing called sacrificial concepts. And what they are is actually drawing up what we think the future is. Because even as researchers, we get to imagine what we want, even though what could provoke responses from our participants. So if I say black cat, everyone in the image of a black cat and everyone's mind is different. Or if I said the word convenience, or if we heard the word convenience from anybody, convenience could mean a million different things. So how do we draw the right responses out from people? And I think we start that by making some of our questions tangible. I like to think that sacrificial concepts are questions in and of themselves. So that's why designers actually draw up these concepts from the very start to start that conversation. Rhonda? So for me, imagine the image of typical Southeast Asian mega cities with traffic jam, nonstop honking. And those are actually the most, the richest source of research. So I would have to go with field research. Coming from data, data science background. So I am actually value it a lot because there's no substitute in actually getting all those rich information by being one of them. When we do research to our drivers, then be a driver for a day. That every minute that you become a driver for a day, it brings you a lot of insight in how people actually decide things and ultimately that what matters. So satisfying with having all those field findings, being able to contradict all those data people in the back office and then see that, you know what, you're capturing things wrong. You're actually not capturing the things that actually matters. And the second plus is that it's nice to be able to travel around. Hi. My answer was actually going to be it depends as well because you have to see what the problem is and then you choose the right methodology. But if I had to pick one, I would say contextual inquiry, which is a kind of ethnographic technique as well in where you go into the person's environment and you do an interview to start with and then after that you ask them to show you how they would do something. And then they become the expert and you're learning from what they're doing. And in the same way, you can bring along your stakeholders to be part of research so they get involved and then they can see what the reality is really like for themselves. And I've seen a lot of stakeholders leave the participants home or the office with a very different perspective of who they're really designing for. Staying with the mentor, having moved between in-house and agencies, what's the biggest difference between in-house and agency researcher? So this is a bit tricky because I've only been in-house really for one year. Even though I'm in government now, we're really an agency serving the rest of GARF. So I'll speak from that one year experience at Property Guru. I think before you go into in-house, you think like, okay, as an agency designer, I wouldn't have much impact because I'm just limited to that timeframe that the project is on and then that's it. The reverse is actually the reality when you go into in-house that there is no limit. So the tendency is that you can actually do too much and then you have to be careful about not taking on and not everything is your responsibility. And I learned this the hard way. So I was doing qualitative, quantitative, I was doing UX advocacy, like doing talks and then I was getting pulled into branding, shaping the brand's voice and doing interaction design. It's like, hang on, actually, what is my role here? And especially if you're a team of one, which I think a lot of people are. So you just have to be careful, how can you best serve your team and your company at the end of the day? Having worked with clients with in-house researchers, what is their relationship like? This is for Kai Singh. That's interesting because probably about five years back, a lot of her foolproofs work was with the business units themselves. But it's great to see that there's a lot of in-house capabilities now in a lot of organizations. So I guess we have moved with the times as well. So right now we actually have seven ways we work with a client or organization partner. And one of the more interesting ones is what we call the augmented team. So it's where there's a team in-house, but maybe there's too much work for them or they are, for some reason, unable to hire all that they can. And then we bring our people in to support them for six months to 12 months. So that's quite a long time. I guess the difference is organizations move at the speed of trust. So I'll say that again. Organizations move at the speed of trust. And the key thing is being able to understand who trusts who in this organization and when we come in to help, what really is the link there. So generally, organizations move at the speed of the slowest person. And it's important that whether you're in-house or in-agency, you learn how to build that trust. And how we do that is we work with the in-house agencies to say, hey, we're actually the birds of the feather. We are from this same industry. We are all UX people. So let's help you to work with, convince your other departments internally. Very deep. Every time I talk to you, I learn something new. Okay, and then to Ramda, because Ramda has her own research team, have you worked with research agency before? So the answer is definitely yes. And it's not easy because imagine that a Gojek mostly works in transport and for the different business these days. And these are the type of businesses that doesn't exist three years ago. And so most of the agencies are not really catching up with that. So imagine that Indonesia is mostly a consumer market. It's like the ripe ground. If you want to sell things, there are 200 million people, blah, blah, blah. So a lot of people are just pushing, just people are just a lot of people are doing things to sell things, but not actually introducing new things. So when given like a new way of doing things like Gojek and people suddenly don't know what to expect, we have difficulties finding really good agencies actually. We need one that wants to brainstorm with us, not just this is my 10 year experience of how consumers behave and this is how it should be. Because then we now have internal data that probably prove otherwise. And those are the tension that usually arise when we talk to agencies. But these days I can say that it's getting better. Right now we find newer agencies that they don't really focus much on big names, they don't focus much on like years of experience, but they really are willing to try new methodologies and they really have good communication skills for us as a client and that's great. That's what we want to partner with. Ramda, I actually know two very good agencies here. What is the biggest pain point for you running a UX agency in Asia? Any advice on overcoming that? This is for Kaixing and Desan. I feel like I'm on Freud's couch. Okay, biggest pain point, let's see. I think, okay, let's start from what research actually has the power to do, right? So if you think about research, researchers have the power and good research done well, has the power to shape strategy. Now our role is to bring that voice of the customer to the boardroom, to whoever will listen so that they can make the changes that they need to make. Now, however, being all of us, being Asian and living in Asia today, all businesses here are set up to optimize and there's nothing wrong with that, right? But sometimes when you want, how do you do that and plan for the future at the same time? So how do you make a case for research? How do you make a case for design that shapes that kind of nebulous out there? How do you plan? How do you show the value of research bit by bit and the value of design from day one to build out to the future that at least the clients that we work for are looking to get to, right? So I think that would be, I mean, again, another thing that I could talk until the cows come home about, but helping clients to see that and to shift away from efficiency targets, optimized targets is the other one that they always come back to is we've already done all our research and I'm like, okay, and lots of research today tells you what the problem is and doesn't tell you how to solve it, right? Research done well shows you the how, highlights the opportunity in the short and the long term to enable clients to prioritize that but to make that case up front and to show that true value is one of those things that has proven challenging but we're still working on trying to solve. Yeah, I hear you there. Just to be additive, I think once you do have the research and it's great and you get to the C-suite or you get to your high level stakeholders, one of the pain points we have is the internal teams don't reveal organizational dynamics to us so we are maybe at the final presentation, we are speaking to the CEO and he's gonna give a go or no go and sometimes we get blindsided by something that's totally out of left field and a lot of times how teams, internal teams prepare us is do this, do that because I know my boss likes this and that but because that's their assessment of how it is it's difficult for the agency to read that and we may have a different or more creative way to sort of address those organizational dynamics so if that's not revealed to us we cannot help the in-house agency to do this or the in-house team to do this so things like power, what is the hard power in the organization's hierarchical? What is the soft power? Who are the people we need to bring into the room that's not there? Those things are very soft and I guess one reason is they cannot see it themselves so they can't help us so increasingly our team is being trained on how to see organizational dynamics just so that we can help better get people to do more research and yeah, do that. Sorry, I just wanted to build on that actually I think one thing in my personal journey in the field for the past eight to nine years I think one thing that has you know in the past we used to go just bring the user and the user's voice to the table but today if you do not understand organizational dynamics and if you don't for all of those of us in agencies who want to join agencies and also for all of us who are clients and looking to hire agencies I ask that you demand this of your agencies because I think as much time needs to be afforded to looking in as well as looking out but looking in is what we like to call client empathy you need to have as much empathy for your clients as you do your users and if you don't have that client empathy nothing that you do will ever hit the market so there's a set of suite of tools and the suite of questions there's a kind of time that we use to really understand the client that has actually proven very useful get them on board early get as senior and as many people in the room early figure out who is on the need to know basis figure out who is the core team who is the adjacent team and who is the check-in once in a while and kind of roadmap that in a really rigorous calendarized way Thank you Just short of hand how many people are actually in agencies in this room? Cool How many are in-house researchers or designers? Nice mix How many are qualitative researchers or qualitative? And quantitative? Nice You'll be very popular later Okay, so next segment we drive deeper into qualitative research so the series of questions will be qualitative research Let's start with a simple question first What is your favorite research question for user interview and why? For all If there's one question that you need to ask in user interview, what would it be? I'm calling this the 50 Shades of Why The baseline is why and there are different ways to ask why and it's to come up with your own little toolkit of how do you ask why, you know, like Oh, why did you do that? Or what led to that? Or how did you decide on that? You know, tell me more and it just builds on the fundamental question of why But for me, I think the intention of the question is more important than the phrasing sometimes just making sure that you understand why you're doing the research the outcomes that you'd like to uncover so that even if you lose your discussion guide or you forget what the questions are you can always go back to that intention and then just have a conversation with the participant Interesting Well, I will go with house quotes these days Everybody lies So the interesting thing is that we know that you know, every time we do an interview probably three out of 10 actually don't tell the truth and we can actually see it from data you know, there's one thing there's one moment where we grill one person to why did you do this, why did you do this and we actually know you don't do that but yeah, and then the person gives like answer that you know, it's basically, it's wired that people just want to give answers, they want to please people so as someone who asks these questions the interesting is actually not what the question is but how they answer the questions we actually especially interested what, you know, we know that a question that is impossible to answer like how often do you just go right in the last 30 days nobody remembers, nobody ever remembers nobody ever give a right but we still ask anyway because we want to know how they think or what they feel that this is the right answer and that's actually the most interesting things favorite research question, how was your day? I think it's really important to disarm the person that you're talking to because ultimately as a researcher you have a limited amount of time to win the trust of this person for this person to tell you everything about their lives because ultimately you're not you're designing for whatever it is to fit their lives not for them to shoehorn their lives in this thing that you're trying to build, right? So how was your day? And often you'd be surprised at how off guard that catches they'd be like, what? And that's how you're like, yeah, it's cool, yeah and then you start talking about that their weekend and you just get into a lot of different stories and tips of how they live rather than them telling you what they're doing I'm also learning a lot, I think all three are really good so I'm trying to figure out what's up not really Okay, okay, okay Yeah, yeah I think one other one, apart from the three is if I see something happening I'll say, I notice you are then fill in the blanks I notice you are frowning I notice you are hesitating and that brings a level of self-awareness for that person because they might be so busy doing whatever they wanna do whether it's a usability test or they're just walking around or whatever that gives them pause and it's like you're showing a mirror to them and then they're like, oh, yeah, I am frowning and then follow that journey of introspection but you do have to be aware of when they post-rationalize and there are ways to figure out when that happens but I think bringing a bit of self-awareness to them then following them on the journey sometimes also is quite interesting So for the next question is for Samantha and Avishan is there a difference in approach when researching for B2C versus B2B users? The thing that comes to mind is, I think, recruitment B2B recruitment is a lot harder because even though your client might have access to the customers, sometimes you have to go through like account managers to get the participants on board and then the account managers may not want to recruit for you or they may say, like, why are you bothering my clients? So there's a lot more education and a lot more hand-holding when it comes to helping your client's colleagues recruit for you in B2B projects and I think also you have a lot more stakeholders that come along with you doing such research like, for example, the account manager might say I want to come along with you to the interview because this is a really important client of mine and then when you get to that interview, you see like, oh, the client himself, the participant, has invited like his other colleagues to join in because they think that it's helpful and then it becomes more like a focus group discussion rather than a one-on-one interview so you have to be quite adaptable, I think, in doing B2B research because you never know what come up when you get to the person's office. In my experience doing B2B business, so I'll give an example. Worked for a very large social media company and they were building a business platform and they wanted more businesses to use it and they didn't really know how to create that relationship with the businesses they were trying to serve so essentially what we did for them is to go and speak to these businesses but these businesses, even though they're, my client's clients had clients of their own, it's client, client, client, they had no understand as in, and when I went to speak to all these businesses that were using the platform, they had so, I think the methods in which we use to understand them shed a lot of light for our client in seeing that actually you don't just split your businesses into how much value they bring to you which was how they had categorized low value kind of out there, mid value, high value, lots and lots of ad spend, we're gonna give them highly personalized treatment and they've had this big chunk in the middle and they didn't know how to reach out to them so in essence, actually not really because when you think about the needs, the motivations of behaviors of businesses, businesses are made out of a ton of people as well making decisions every day with bosses to please, with things to sell, with financial constraints and figuring all of those things out is really, really important when we were helping our client figure out how to serve theirs better. Now this is a question that I really wanna know. What are the challenges when you're targeting multiple markets in Southeast Asia and how do you manage it? Anyone can answer. I'm interested in how Gojek does this. But by the way, I'm waiting for you to come to Singapore. Well, we just started with one market so a lot of the difficult part is actually to make sure, is there anything useful from what we learned in Indonesia that can be applied to Vietnam, to Singapore? If you're like a typical pro-business analyst working in Bloomberg, you will think, so this Asia is similar, right? No, it's never the same. Even Vietnam having predominantly motorbike-based services is very different from how Jakarta actually do theirs. Very similar in looks only. If you take photo, it looks similar but how they self-organize, how communities are actually built, how do they communicate between drivers is totally different. So how to scale will definitely go local. Higher than the most local guy you've known, we literally have an object drivers in our team. Someone who literally rides the bikes eight hours a day just to build empathy because we cannot do that if we're just sitting behind. But so does that, do we need to do the same for every single market that we do? I would say yes. Sky Singh, you want to add on? Yeah, I think it's about, if you're doing research in individual countries, that's fine and it's easy to do a report for your product owner and bring the insights in. One of the challenges is around cross-analysis across markets and identifying what's similar and what's different. And that's been challenging because, especially if you work with each country's insights teams, and they have their own ways of doing this, and sometimes it's very different across countries, then how do you normalize and come up with a framework which all of them can sit under. And then you have issues where, oh Vietnam has a certain way of doing things and there's this new topic that they want to do but it's not part of the overall research and then Thailand also has this and then Taiwan has this and so it's about organizing that in a neat way. So in fact what we have done recently, so we are training seven countries in Africa how to do research on their own and what we've done is we've used mural.ly, so mural.ly if you know that, and we get all of the countries to put all of their research and insights into one huge board. And then so we just had a call with Zimbabwe and Rwanda and they could see each other's post-its and that was quite a nice way of getting the whole of Africa to come together. Very interesting. Speaking of research in Southeast Asia, actually I would say never underestimate and always pay for a good translator. Always do not stinge on that. The ability to not paraphrase what your customers are saying, to tell it to you word for word and not summarize what they're trying to say, so important. Definitely, definitely spend on that. Historically we've gotten translators that work for the UN to do it and we would justify that cost that way. We have also, we've also gone very much to what Rwanda was saying, cultural guides to take you into the field to speak with the driver. Like getting you immersed in the shortest time possible it should always be your goal. Going into Southeast Asian markets no matter which ones they may be. Right. I actually have a joke on that because I did research in Indonesia before so the users have been talking for like a minute and the translator is like, oh, he likes it. Yeah, I think to add on, sometimes you get clients saying, oh, like our colleagues can help to translate, no. Stay away. Be firm, hire a good simultaneous translator because translating is one thing, translating simultaneously is another. And the really good ones are worth their money. Another point I would say is to understand the culture of the people that you work with. So like, how people work in Indonesia is very different from how they work in Thailand and how they work in Hong Kong. And that's just basic like working with people really, yeah. Next question for our cementer. Stay with your microphone. How do you convey insights to organisations and ensure that it's thick? Coming from Gapdek. I think the obvious one but worth mentioning is make sure you're answering the right problem instead of the problem that they ask you to solve and making sure that you really get to the root of what the issue is because if you don't and you get all the way to research and synthesis and presentation and then the big boss comes in and he realises that actually this is not the problem, what. You may find yourself losing a lot of trust to perception of your team's work, maybe a factor as well. So that would be the first thing that I would do. I think it also helps to understand your organisation's culture and what sort of things your stakeholders respond to. Are they like numbers? Do they respond to numbers? But you're actually trying to give them stories then maybe they won't really listen to you up front. Are they more of a reports person or do they become more receptive if you actually walk them through the insights in a room that's set up with the pictures and the videos and things like that? Yeah, so I think just understanding your stakeholders as well. Okay, talking about numbers, we move on to quantitative research. So for Ramda and Lishan, how does quantitative data influence qualitative research and vice versa? One thing that it helps measureably is to find the needle in the haystack. Because the limitation, you might argue that it's the pros but sometimes it's seen as a con of the quality. So it's just it basically boils down to recruitment and quant analysis can actually help to improve that processes because ultimately the most mundane users can result in the most interesting behavior in our data and vice versa, the most interesting person in the whole data is actually like, doesn't have really interesting behavior. So it helps to inform each other, it cuts the overhead quite low and then it brings some more, arguably it's supposed to be imagined if every quality searcher can read data and vice versa, but it's actually harder than sit and then. But what it helps is that it helps to calibrate each others, it helps to inform each other biases. It can challenge each other assumptions, hopefully with the ultimate goal of reducing the total error and the total bias status present, although it can actually amplify each other. I'm not saying that it will always be good. It sometimes can just, if you're a quality guy and then you look at some numbers, it seems to confirm your intuition and was like, see, I'm right. Possibly both of them are wrong and that's actually where the challenge is. Lixian, I think maybe I'll answer this question with a story of a previous piece of research that I did and it was for a bank. And what they wanted to know was whether people knew how to manage their money well or not. So if you can imagine, the quantitative survey had a question about managing money. Do you, on a scale of one to 10, how well do you manage your money? And then they were like, you know what? Actually, our gut tells us that people don't really know how to manage their money life. You look at how people are using their bank accounts today, but how come everyone says they are so good at managing money? So what we did was we found this woman and we went to her house and she said, I manage my money really well, right? And we were like, okay, show us. Thinking that she might pull out some an Excel spreadsheet or some books and she's like, come, come, come to my kitchen. She opens a freezer and takes out a block of ice and her credit card is in the ice. This is a true story, okay? And then she goes, okay, so this is what I do. I'm going to take out my block of ice and I'm going to chuck it in the sink and I'm going to do my housework. So this is what I do every time I want to buy a dress. That's a bit above my budget and I'm going to see and I'm, you know, if I go and clean my house and then I come back and the credit card is still inside the ice, no, you have no hope. No need, no, not fated, not fated, not fated for me to buy the dress. But then if it melted, then, yeah, of course, ha, ha, off I go and now I'm going to buy my dress and that's how she manages her money. She clicks seven in terms of money management because this is how, as in, you know, this is in her mind, this is how she manages money and she's got a way to do it and the bank has no role in helping her manage her money whatsoever, right? So, yeah, and I'm not saying that Quant has no place, Quant has a place and honestly, anyone who works on the client side, if there are no numbers, there's no way your research will fly, right? You can't market size it, you don't, you know, people look at you what, you interview how many people again, huh? How is this representative? The whole slew of questions that have been asked have all been asked before. So I think the question is what questions are you asking and what kinds of research are you doing and when? So if you understand a couple of behaviors up front, you can use those behaviors to inform better questions that go into a quantitative survey that come up with a set of behavior types or a set of market size of, you know, who would go for what kind of product, how many people identify as this kind of person, how many people have those kinds of problems, take that and you've actually got a fantastic set of data that includes stories as well and we call this internally hybrid research. At IDEO we did the full suite, right now it's agency, there's so many very well qualified quantitative agencies out there who are also yearning for better briefs and I think that there's a huge opportunity for qual and quant to come together in a much, much more meaningful way than it is, that is happening at the moment. And I want to add, actually, a lot of people in quant research want to move away from those skills, one to five skills, one to 10, please rank 30 items in one go. We just, we don't like that, but it's sometimes the organization ego who wants to measure things that are unmeasurable and those require some very brave, basically a guard, you might say, a guardian for those ideals to say that, you know what, sorry, Mr. CEO, that cannot be done. Don't try, even try to measure it. And it takes some guts to actually to say that and that's actually where quant research should go next. For UX team with no quantitative research or data analysis, how do they get started? This is for Ramda. It depends on the org, is there any interest to actually do that? First and foremost, if the org doesn't have interest to do that, then it's very hard, because it requires some, imagine that the HR has to be brief on that, you know, you want people to have more skills and that's instead of being boxed into like one label. So that's actually hard, but the more important thing is actually, people think that by having UX qual team have more or less even, they will be more, much more well informed and very data informant, very, very basically just looking at data can, you know, can decide many things, but actually some leap of faith is actually okay. It's actually totally okay just to decide things without any data, because ultimately you cannot really measure everything anyway. And it's, I think the best way, if you really want to do it and you feel that the org can benefit more of it, one of the good ways to actually start the collaboration project with the data team, that's always first, but it shouldn't be a project that just be the, the, the, the as usual, right? These are these, these are the interviews, but with data, no, that's not enough. It requires some, probably some niche that is actually what exploring by combining quant and qual, then it becomes much more powerful. It yields not the intended consequence of your product, but an intended one that actually might even be business critical. And that's the way you start to make people see the value. And the, the best thing that can happen is that if people from the very data heavy team, let's say data scientist wants to join the UX research team, that would be awesome because that's actually one of the hardest jump to make. There are many companies that say that they are data driven. How do you know that they are doing it right? This is also from for Randa. It's not advisable actually. Saying that you're rather driven means that it seems like you only, you go where the data leads you. And imagine a company that always changed where the data actually is, you know, where the gold mine is. That's a very weird thing because usually you have a mission and then you have data to inform you. And one thing is that you can be data driven and your data can be wrong but you're still successful as a business. And that's the hard thing, right? To convince that you are going down the dark path because see, I don't know, we get a lot of money this way. Why it is all right, right? But no, I would say that a healthy organization actually has a healthy skepticism in their data that data actually captures the thing that probably critical for one part of the business but not for sustainability, for example, social impact, for example. So having all this skepticism on what we know so far is actually the way to go even though the cost is actually probably gross and economic profit. It probably cost that but I would say the trade-off is worthwhile. You can see how large tech company that data heavy company actually creates social impact that is not what they intended and this is actually something that can be prevented if you aware of this beforehand. And yeah, and it has to be democratized. Everyone should have access to the data. Okay, we are moving towards the future of research. This is for everyone with the rise of chief design officers, chief product officers, how can research move into a strategic position? So I think the question was with the rise of roles such as the chief design officer and chief product officer, how does research move into a much more strategic role? I think we start by thinking about what the role of a chief design officer and a chief product officer is, right? The chief design officer's role is to elevate design in an organization to the same level as the business, at same level as profit, making profit same level as what tech exists in the world, right? So when that, following that lead, research will definitely move into a more strategic role because first of all, design at a strategic level helps you frame, brings the human being back into the business which essentially what all businesses are about, even your own, right? And I lost my train of thought. I had many thoughts. Hang on, but also in terms of implementation, what does that mean, right? Design and research, I mean, we've talked a lot about research, exploration, these days, research, they think of it, many people still think of it as a bolt-on, or a tag-on to what's already going on in the company. I've got all this, I've got, I really know, I kind of know what the solution is, so I've kind of go find out whether people want to use it or not. That's not design in a strategic role. Design in a strategic role helps you set the course of where the business should go and what it should be doing. So I do encourage all the researchers in the room to be way more ambitious about what your roles are within the organization because when you bring that voice to the table, things can really, really change. And we've seen that on a level of bringing products to market in healthcare, in finance, we've also seen that transform organizations from the inside in terms of how people think and work with each other. Maybe just to add, I think, that's quite an interesting line of thought, and what's coming out for me is around, really, if there was such, I think as a chief research officer, chief insights officer, you really are the organizational self-awareness person because you are trying to discover what your customers are doing and how they are reacting to you and your decisions and then bringing that back internally. So you are inviting the organization to be more self-aware of itself and what they're trying to do and what outcomes they're trying to look for and whether they match or not. So I think self-awareness is quite a strategic position but you need to be able to sell this in. Yeah. So this is coming from a personal perspective, I think more from a practitioner's view and when you start out in research, you have this small fish and then you grow into a bigger fish and a very efficient fish. So you get very good at doing the research, right? But then now that I'm taking care of a team and taking care of a research practice, I find that I have to become an octopus. So to move from a big, efficient fish into an octopus requires a different kind of skill set, a different type of muscles and where maybe I only had to wriggle two, three feet legs before and now I have to bring out my eight in full force, not just looking at delivery of insights but also the team, team morale, practice and rigor, stakeholders, advocacy and a lot of horizontal things that maybe you didn't really have to think about before. So I would say dedicate that time to take care of your horizontal skills as well so that you can prepare yourself as you get more and more senior into fulfilling that kind of ambition. Ramna? I think it definitely needs more, especially in tech startups, it's going to be a very hard position to make. If you really have to make that position, it's very hard because the maturity and the experiences are not there, right? Research is infamously difficult to do in profit sector, much less in a very one-month prototyping to production startups. So I can totally imagine why there's no, like a very authoritative figure in the company or that dictates how rich it should be. What I think can justify is that if the position is actually focusing more not on the outcome of the research but more on the building the systems, how to make people know this insight, for example, across the org. How can someone, one people out of 20 who join the company every single week actually understand why are we here, why the company is here and what are the environment are we working in? What's, where is our place in the universe in those sort of insight? And that's very hard to do at scale. I think it's been mentioned that knowing the battlefield, knowing the stakeholder is very hard and the best thing that the in-house research lead can do is actually not only to advise the agencies on the battlefield but also to help manage the battlefield themselves. And when they manage to cultivate this culture and make systems that are scalable across the org, no matter how scalable, how fast growing the company is, then it is only then it's justified. But if you're looking at the outcome, how does it contribute to the bottom line? Maybe it's even possible that there shouldn't be any health of research and like on the sea level because then it's very far removed from the impact it generates and it is very hard to measure it. But someone who can perpetuate culture of research, that would be cool, I guess. Okay, the next question is just so open for all. What's next for research in Asia? Samantha and Kaising, actually, this is for both of you. Kaising, go first. Thanks. I think it's still a nascent market, most parts of Asia. I guess if you see how it follows the track of more developed industry fields in other countries, you just get more awareness first. People will know why they need it but they don't know how to implement. So the why is there and they will know I need this. It's like, oh, design thinking, right? So yeah, I need this but they don't know how to implement. So research, I guess one level down will be a bit more detailed. So the how is the, I think the next, what's really next? And I guess that's where conferences like this and any kind of skill sets you can build is critical because once you know, then you can explain it to your boss why it's important as well. And that explanation is also a skill we all need to do better in. At least that's my view. Samantha. I think now you see a lot of companies hiring researchers. So it feels like that's probably the next wave. There will be more specialist roles in research but to what kind of things it may not be necessarily the case that you're doing the research right or that you're actually doing enough generative research. Doing usability testing is not enough because you're just validating design. So how can we as an organization or as a team create more time and resource so that you can go out and do that generative research and make sure you're uncovering the right problems to solve? Okay, we have the last question before we open to the floor. What's the best way to continuously improve yourself in research? This is for all. Yes. Trying out new things separately. Be very open on your methodologies even though you have some skepticism or is this scientific enough? But ultimately what's more important is for yourself to open up, for me as a researcher, as a practitioner to open up to new way of doing things doesn't mean that you have to take it all. This is the greatest thing since last break. I'll adopt this immediately to everyone now. Doesn't have to be that. But there's some aspect of new methodologies that as you learn by, you can incorporate and actually strengthen the useful tools of trade that they use. I think for me, two things. To continuously improve your craft as a researcher. First of all is to continue to work with as many people as you can from different disciplines. I believe different disciplines. So for example, our team has everyone who trained in, who I've got architects. I've got interaction designers, graphic designers, product designers, engineers on our team. And then whenever they come for the interviews as well, they, the kinds of questions they ask and the way that they process the information really teaches me a lot about what we should be learning. The second one would be to really push all of ourselves to be far more rigorous about the kinds of research that we do. Like why is it we do what we do? How do you, when you are testing something in the field, how are you testing attitudes versus testing will someone buy something versus will someone actually use something over a period of time? How do you learn from that? How do you change that? And on the other hand, how do us as researchers talk about the future? How do you help organizations envision what the future could be and what kinds of questions you would ask for that? I think those are on two different ends of the spectrum but both very important as researchers to continue thinking about to improve our craft. Samantha, do you want to let Kai sing over us again? I recall a quote from Bruce Lee actually and he talks about, he fears not the person who knows 10,000 kicks, but he fears the person that has tried one kick 10,000 times. And it's really about rigor. So I would practice my one-to-one interviews constantly. It might feel like a drone, right? After like your 20th one in the same project, it might feel really automatic and it's important to reset back and say, yes, there's a reason why I'm practicing this and get better and better at the most basic things. Because once I think you get to the core first principles of it, you can then learn almost anything else. So I'll advise not to fetish over new techniques too much. Sam? How am I going to top a Bruce Lee quote? But pretty much what everyone has said, to add on to Kai sing's point, there was a time where I just did usability testing month after month, after month in the same country. And you can say that, oh, it's so boring or I'm not really learning anything. But you can shift your perspective into that experience and say, okay, can I take this opportunity to get better at, say, writing the best screener possible or facilitating usability testing in different ways, exploring how can I present usability testing results back in different ways. So it's really about perspective. And I would also say, look at lateral experiences and exposing yourself to different experiences and to help with your recommendations. Like when I go to museums, I look at how they do storytelling. When I go to a VR arcade, I look at how they do onboarding, how they onboard you with all the gear and how to use this stuff. So there's lots of experiences everywhere. You just have to keep your eyes open and observe. And then also to reflect on your week and how are you learning? What did you do more importantly? How did you feel? And why did you feel that way? And how can you do something different the next week? So reflection is a great part of learning. Thank you. I think we have good session today. I learn a lot from each and every one of you. We're gonna open it to the floor now and we're gonna use a system, right? Yeah, we're gonna use a system. It's called Slido. Great session, guys. I'm a researcher. I've been doing research for six plus years now, but it still felt like a crash course in research all over again. And that only says good things about our awesome, awesome panel today. So let's look at some of the questions that the room is asking. Our top most question is, what is your one wildcard or expert tip to convince the C-suite? And I think this goes back to the sentiment of how to get the importance of research communicated to the top level senior management. So if maybe one of you can take that so we can go through more questions in short time. Who wants to take this up? Any expert tip or what's the final wildcard that you pull out? Well, I do have a tip that I ask and when my team is presenting to C-suite to very, very high-up management, and I'm speaking to my team and we're presenting back what people in the field may have said. And usually I go, if you cannot tell me and make me cry with three things that you know about this person, then that story is not worth telling. And that's how we train ourselves, right? Because I'm just, I mean, we're quite, we speak very frankly at work and I'm just like, I want a stigmata moment. I want to be bleeding from my eyes and my palms when I hear the story of this person that you've gone out into the field to try and understand. And if no one can remember that story, then there's nothing to design. If you go an eye towards implementation, you have to feel the need of that person and the job of the researcher is to make that need felt and provide that line of sight from that to the solution that you're trying to design. Amazing, great tip. Another question, how do you as a researcher check and remove your own personal biases on the product or towards the interviewee to ensure objectivity? Who wants to take this? By having more diverse team, actually. You know, it's never like a singular effort, it's always a team effort and it's great where your team is hopefully diverse enough to be able to kind of counterbalance and you know, like a peer check of everyone else work because then with interviews, there's no way to remove 100% biases. It's always there, it's projection, it's your upbringing, it's inherent. So one way to do that is to basically at least help someone else in your team to calibrate that. Okay, so balance your metrics and then get someone else to wet them before you take that final call. Another question, how much is educational background important for being a successful UX researcher? I think that's a very interesting question. What's your background? Is there any school for your sister? No. No. No, I think so. I don't think so. I was a game designer and work in government and on research, I don't know what I am. So it's, you know, don't worry about it. We have intern who just graduated high school, which means zero experience in, you know, in education, even not even working experience, but she got good inquisitive minds which are probably more powerful than most people 10 years over her and have doing like my own consultancy. So that makes her a better researcher than the other guy. I mean, I come from hospitality management. Not really very related. I think like what Ramda said, inquisitive mind, curiosity, questioning why, maybe the types of people that like to break things apart just to see how it works. Traits, I guess, rather than education. There's a part of me that wishes there was actually a course that taught more of this because, because figuring out what people say and making sense of a hundred different voices or even eight different voices is really, really difficult. When everyone is telling you something, when everyone has a need and a want, when every, I mean, someone was telling me over the weekend, how do you cope with user creep? Which is when you keep, when someone, I'm sure all of you are familiar with that term, but when everyone just has a whole list of wants, how do you prioritise that? And how do you make sense of it? And I think I actually don't have an answer to that. I wish that synthesis process was actually something that could be taught more across any design course, across any, you know, any kind of module or school, like degree that we were all part of. Lisa, what educational background are you from? You don't know? Oh, really? I was trained as a journalist. So the way that I like to talk about my job a lot is, at some point in my life, what I was trying to produce was a piece, a written, a written word to be printed somewhere. And now, but the still is the same, you go in the field, you disarm the person, you ask them what they're about and what you want to do is bring that voice and that story to someone who's going to be, at that point, reading now someone using it. And I think I'm very privileged because right now I've got a whole team of designers that are coming up with solutions for that person, are these people that I'm talking to? So that was my background at least. But right now my other design researcher on my team is, used to be a video producer for Discovery. So her background is all about storytelling. It's about going to the field and understanding people and bringing those stories to life as well. So actually very, very diverse. One of the best ones I've worked with used to be a product designer that didn't want to actually design products as much more interested in how people were using them. And so she made the transition over into design research. To add on to Lixuan's point about user creep, I think it's not so much about falling back into the numbers, like, oh, nine participants said this. So yeah, that's the insight, right? But it's more about training your gut and your spider sense. And even though it might only be one person that has done that or said that, and you feel like there's something there, then you go forth and investigate rather than just relying on the numbers to give you that sense of security. Great, great tips. Any question from the floor? Okay, we have one. Let's do it again. Any questions on the floor? Hi, good afternoon. I actually would have typed this in Slido, but it only allows so many characters. So yeah, I actually just want to allude to a quote from Henry Ford that was brought up yesterday. So I'll read the quote, right? So, if I'd asked customers what they wanted, they would have told me a faster horse. And then he ended up inventing, like, a car. And then this was a firm by Steve Jobs who said that, people don't know what they want until you show it to them. That's why I never rely on market research. Our task is to read things that are not yet on the page. So my question would be, how do you make the case for the value of UX research to a boss like Henry Ford or Steve Jobs? So for user research, we actually don't ask what user want, because they don't know. But users are very eloquent about the problems that they have. So the solutions that we design usually address their problems. And I think there's a misconception with Steve Jobs. He has a lot of user researcher. They are also hiring researcher. So I think it was misquoted. So a lot of people, like, tell me about this quote. And then I went back to see what is the context. It's actually a different context. So not really true that they don't need to user researcher because they do. To add on to Priscilla's point, we also look at edge cases to see what people are doing to hack the system or to try and solve things that is currently not available to them. And that can be a powerful way to see how you can redesign that solution as well. Yeah, I think I agree. Nobody needed an iPad until it was here, right? We had smartphones, we had laptops, but when the iPad came, we were like, okay, let's just buy it, even though we didn't need it. So maybe to your point, we have one more question. Hi, my question is both for in-house and agency perspective. How do you explain a body content crafted? Because the business people, they say they want to talk like this product in this direction, this way. We have a method for cut sorting, for navigations and stuff, but the body content crafting, and how do you measure it after that? Some insights, tools, and how do you do the auditors? Yeah, so like say, for example, I have a new startup and they are producing our content for the website landing page, not a blog article, and that content has been crafted to be explained the product like this. But now this, whether this makes sense for the customer, how do you craft this thing based on the UX research, and how do you measure it at the end? How many iterations of this landing page have you got? Let's assume that it's coming from the business perspective, so the CEO says like three or four times. So, okay, so what I mean by that is when you want someone, when you are trying to figure out how someone responds to something, wouldn't you intuitively, wouldn't you create three or four versions of the same thing? Using different tone, using different colors, using different language, different characters. You're still selling the same product, but ultimately the way in which you talk about it, and I think what I'm trying to get to also is what do you lead with? There are a lot of questions. So what uncertainties do you have about this product that you've launched about this, about the way that you're talking about it? For example, if I'm launching a bank account for children, am I appealing to the parents? Am I appealing to the children? Mostly it's the parents, but is it about parents want to guide their children until they are old enough to manage their money? Is it about asking parents to identify who their child is, or what kinds of money habits their child has? How would you represent these different ways of expressing that content in order to understand what people respond most to? So maybe for your case, if you were to ask your management, what are you sure and unsure about, and can you test according to that? I think that would be something that I would... What's interesting is that this is what I love and hate about iteration, right? Because it sort of makes sense. You iterate things until it comes to the point that it's great enough or awesome enough to be launched and stuff. But at the same time, people who are doing this refuse the fact that there's always a level of ambiguity or level of, you know, I don't care about these things. It is present in every single user, you know? And that tension actually comes to the point where the desire or when to iterate is really down to the personal level, you know? When the person who has to create a nerf is like, you know what, I had enough work on this already, just stop, just launch, you know? So it's funny to see that, and people ask in which level and what's the best way. Because I think the humility and the acknowledgement that there are certain things that you cannot really design well, it's okay. You just... But if you don't buy, if there's no... The problem is when people don't buy into what's going to be launched with, you know? When different teams or different stakeholders have different level of confidence on the things that they want to launch, and that's actually the bigger problem that needs to be solved, not the process itself. I actually have a very good tip from IDEO. I learned from IDEO that you can test messaging very quickly in Facebook. So you create an ad campaign, and within that ad campaign, you can have different messaging, and then you check the conversion from... which has a bigger click-through, which has a bigger conversion, and with the data, it will help to convince your CEO. And the other thing is I also agree with Ramda that sometimes if you don't have alternative, you just run with it. There's no point arguing if there's no alternative, and you need to respect your CEO. The way to talk to C-suite is actually to respect them. So for us, I mean, our CEO is a visionary, and we do respect his opinions, and I think this is the best way to gain trust between the C-suite and the UX designers. The mutual respect. Awesome. So I guess progress over perfection is the answer. So let's thank our awesome panel for today. We do have a small token of appreciation, so I would like to welcome Kuldeep.