 Hi everyone and welcome to the in-depth workshop, in-depth interview workshop, excuse me, conducted by Margaret Roller. I'm Sue Boffman from ARL and very pleased to welcome project team members from the Research Library Impact Framework Initiative and colleagues from our project library. It's really great to have you join us today. With a quick introduction I'd like to share just a little bit about the Research Library Impact Framework. This is a grant initiative that has been underway for well over a year and our project teams have been very busy exploring a series of five questions. These questions relate to space, diversity, equity, and inclusion, special collections, and research or productivity. So our teams have been exploring these questions and using a variety of methods to do that. Our overarching goal of this initiative is to help us understand how to address some of the most pressing questions that you deal with in your libraries with regards to value and impact. And this initiative is funded by the IMLS grant for which we're very appreciative. So as part of this initiative our two consultants, Margaret Roller and Kevin Fomalant, have developed a series of workshops on qualitative and quantitative research methods. Our goal for the workshop series is to help our library staff develop their skills and expertise in conducting research at your libraries. The in-depth interview workshop is part of this series and Margaret will be offering it again on Thursdays the 18th. So again happy that you're here with us. The sessions are being recorded and we will share the recording slides and any other documentation with everyone. And you are very welcome to share the materials with your colleagues who couldn't be with us today if they were hoping to join us. So with that Margaret let me turn the podium, our virtual podium over to you. Thanks everyone. Okay great thank you Sue, thank you for that introduction. And welcome everybody. I'm going to, I don't know how many of you are at the focus group workshop we did last month. So I'm going to repeat myself here to say that I have a lot of content here. I know I have a lot of content and I've done that deliberately because you will have this slide deck and the recording of course at the end of all of this and I wanted you to have this information. And if there is anything we need to follow up on, if there's further questions or things we need to get into, fine let's do that. But I wanted, but again I wanted to keep the content I have in there so that you would have it, something for you to think about and again something for us to follow up on if we need to. Very broadly here are the four areas I'm going to cover today. Just a broad view of qualitative research. This is again of course a repeat of folks. This is actually talking about quality research upfront is a repeat of every discussion and talk I give about qualitative research because it's really important for you to understand what I am talking about when I talk about qualitative research. So it's within that context that you can think about some of the things we're going to talk about today. Talk very broadly about the in-depth interview method and then do more of a deep dive into the quality considerations and best practices, skills and techniques and then those three areas are going to help provide I think important context to the final item there which is a discussion of the modes including the online modes. So what is qualitative research? Again this is again going to be a repeat for many of you that were at the focus group workshop last month. But important for all of you to understand that when I'm talking about qualitative research I'm really talking about a set of methods that have very unique qualities very different than survey research. A qualitative research is about going beyond the obvious and the expedient and very much so focusing on context and the interconnections of human thought and behavior and which is another way of saying that a qualitative researcher understands that you know the the the answer to any single question really rests and is related to a whole bunch of other questions that adds meaning. Here are 10 unique attributes of qualitative research you'll see on the right side I've highlighted four of them because I believe that these four really serve to provide much of what is unique about qualitative research although all 10 of these are unique but it's the again context the importance of meeting and what we'll talk a lot about today the participant researcher relationship and the researcher as instrument are really fundamental fundamental and fundamentally unique to qualitative research. So the the in-depth interview method I I would assume we all know what that means but what I want to highlight here is the importance of what really singles out the in-depth interview method from all other methods and that's the interviewer-interview-y relationship which is at the heart of the method it makes in essence the in-depth interview method the most personal of all qualitative methods and we'll talk a lot about that today and actually it is going to be kind of the foundation by which from which we talk about all other aspects of the method that we're going to talk about today. There are three types basic types of interviews structured semi-structured and unstructured I have highlighted semi-structured simply because that is the most common interview type that is used which is not to say that not so much structured because then we get a little bit more into survey research but of course unstructured is used particularly in narrative research but semi-structured in-depth interviews is is probably the most common and is probably what I am thinking of the most as I go about with my comments today. So why conducting depth interviews which I will call IDIs for most of the rest of the discussion today focus group discussions as I talked about in the earlier workshop focus group discussions are really better at getting are definitely better at getting at focusing on dynamic conversation and understanding how people debate issues maybe looking at how people's attitudes shift during discussion which is really kind of an interesting phenomenon. It can also be a very supportive environment but the in-depth interview method which by the way is the more popular method of the two more people more qualitative researchers are conducting IDIs than they are focus groups for the most for the most part that's a very broad statement but the in-depth interview method provides very granular contextual type of data and information it can reduce response bias and here I'm thinking of something like social desirability which is a bias that can come in particularly in the focus group method and flexibility and we'll get into that later on in our discussion today about the flexibility of the IDI method when it comes to mode. More specifically the strengths of the in-depth interview method are right at the top the interviewer-interviewing relationship as I've already alluded to it can reduce bias again satisfying or social desirability bias non-risk it can reduce non-response meaning people just not responding to our questions and it can increase validity particularly a question and answer validity it's it's once again a wonderful and an effective way at understanding complex issues because of the contextual nature of in-depth interviews and its ability to leave the researcher with a very kind of rich nuanced story to tell it can be a very supportive environment the one-on-one environment which can strengthen data quality when it comes to these sensitive topics or certain segments of the population and things of that nature and and as I put on the the last item on this slide reaching hard to reach or vulnerable populations such as disabilities people with disabilities and things of that nature limitations now limitations right at the top of the list is the same thing was at the top of list of the strengths right which is the interviewer-interviewing relationship which can be a limitation because there can be interviewer effects that might appear because of personal characteristics or the values or beliefs that the interviewer brings to the interview and the social context or what Kevala called the power dynamics involved in the one-on-one in-depth interview where it may turn out to be more by one way dialogue where the interviewer is actually ruling the interview so so that can be clearly a limitation and a problem with in-depth interviews and it's through the the skills and techniques of the interviewer which are critical to overcome some of these interviewer effects in the social context problems and to mitigate potential bias and inconsistency in within and across interviews and we'll talk we'll talk a little bit more about that as we go on let's go on and talk about quality and qualitative research and I just as I did in the last workshop I'm going to draw on the the total quality framework which is a framework discussed in the book Paula Vakis and I published back in 2015 and it's it's based on kind of this idea that you see in front of you here which is you know if if quality research can serve worthwhile purposes then logically it would serve those purposes only to the degree that it's done well and so we propose something called the total quality framework consists of four components that you can see on the screen there the first component is credibility and that has to do with data collection and that's just the one I'm going to focus on right now to talk about quality and in-depth interviews so credibility that first component of the total quality framework has kind of two areas or components one is scope and scope has to do with coverage and sample design so in coverage we're looking at a lot of the same things we look at in survey research we're looking at the representativeness of participants to the population working with complete and accurate lists so if you're working with let's say lists of faculty members or students or things of that nature and sample design we usually work with purposive sampling rather than convenience or snowball sampling if we can get away with it and if we have lists that we can stratify we stratify them and we randomly select across lists this whole area of sample design includes the question of how many interviews do I conduct now I've read enough in the literature including the literature written by library researchers that there is some feeling out there that there is a magical number to how many interviews you should conduct and often the number revolves around six if you do six interviews you're good to go if you conduct 12 then you reach this thing called saturation and that's a whole other topic but all of which is to say that there is no magic number because you want to evaluate the number of interviews to conduct at least in two stages of the process at the design stage and at which point you'll be thinking about and looking at the diversity of your population you'll be looking at the breadth and depth of the topic initiative so for instance if you want to conduct interviews where the key research question or objective had to do with understanding specific terminology or let's say reactions to a new service idea you're going to those those interviews may be a little bit longer than than they would otherwise because you are going you the interviewer are going to be spending a very concerted time investigating that terminology or that concept which is going to add into the overall time of your interview and you're going to need to do that the same way for for interview to interview so it's going to to add in a little bit to how many interviews you need to do the expected variation of results and of course scheduling resources is always an issue but the other time to think about how many interviews you're going to conduct is during at the field stage when you're actually out there conducting interviews and when you're when you're in the field stage you're going to be reflecting on these kinds of questions you see in front of you now did it did all interviewees provide clear unambiguous answers do I understand what all the interviews said back to me in response to my questions our new ideas and themes emerging or are popping up in these interviews as I go along and so therefore should I keep on going in order to to understand if there are more ideas that I have an untapped and to help enrich the ones I have untapped do the data I've obtained tell a story so far is do I have a story I can do I think I have a story that I can come back with just based on the interviews I've got so it's so the point being is that for for me or anyone to say you need to conduct six interviews is really kind of missing the point of what you really need to be looking at when you think about how many interviews to conduct the other area within scope is a whole area of what we call non-response which is basically having to do with gaining access because why because the question is as always is are the participants who show up for your research for your in-depth interviews the same or different the ones that don't and that's kind of an important question so gaining cooperation and there's a lot of different ways you can try to gain cooperation I've highlighted on this list flexibility of mode which I have I touched on just briefly earlier this is unique to in-depth interviews unlike focus group discussions the in-depth interview offers the researcher the opportunity to to venture into a variety of different modes to conduct interviews and I would also add to that a variety of locations I can't tell you the kinds of locations I when I conduct in depth interviews I almost always if not always allow the participant to tell me where we are going to meet for in-person interview and simply for this reason to gain cooperation and make sure they show up and make sure we have an interview which has meant that you know I've conducted interviews on the back of pickup trucks and all kinds of various places which may be inconvenient for me but it's great in actually getting interviews done now part of the cooperation of gaining cooperation also means having effective confirmations and I'm not going to go through what you see on the screen right now except to say that when you are at this stage the confirmation stage it's very important for you and whoever you're working with to have kind of a process of protocol in place of how those confirmations are going to happen and what you are going to do when these various situations occur so for instance the last item on that screen is you know what happens when confirmation is judged weak well what is a weak confirmation a weak confirmation may be somebody who says oh right Friday at two o'clock well let me see I don't think I have anything else going on then that's not that's not a strong confirmation that's a weak confirmation so you need to to know how you're going to deal with those kinds of things when they come up okay so the other area of the credibility component of the TQF the total quality frame of data gathering and this has everything to do with validity in other words are we measuring what we think we are measuring in the in-depth interview and it boils down to looking at how we are how we are obtaining the content uh researcher effects again bias and inconsistency of the researcher and the participants willingness and ability to provide us information all of which affects the validity of our data so one as I just said is content how we obtain um how we obtain the information and I talked about this for the focus group in the focus group workshop and indeed for the for IDIs for the in-depth interview method uh we also need to be thinking very carefully about the guide development the overall flow the topics and issues main questions probing questions and remind remembering very important to remember this is not a script it is a guide and it has a funnel approach where we go from broad to narrow we start with introductions general information which is going to provide us what the context by which we will explore stages three and four and of course four stage four is where we get to what we really want to know the you know stage four is where we're getting to our real objective and key research question but to get there we have to go through these earlier stages so that so that we have the context I'm providing here one example of an idea something that I've worked on one of my studies an ide study for guide star I conducted 86 uh in-depth interviews for guide star guide star is the world's largest provider of nonprofit information and they asked me to conduct in-depth interviews with both users and non-users of their information to understand their needs as it relates to nonprofit information and to come back with what they needed to know to help develop new products and service concepts so I conducted 86 in-person and telephone in-depth interviews with corporate giving decision makers government entities foundations public charities and all kinds of folks who were using nonprofit information we began with in after introductions on stage two I gathered just some some some basic knowledge of how and what they are using for in the way to gather to gain nonprofit data and information along with that I gained their awareness and use and comparison of the existing nonprofit providers their existing nonprofit providers and then in stage three knowing all of that in stage three I could start focusing on guide star which of course in almost all cases came up in stage two because guide star is so prominent but in stage three is when I really focused on guide star their awareness and perception of guide star there's its strengths and weaknesses compared to other providers and things of that nature and then in stage four I got to what I really wanted to talk about which is how to improve guide star to assist in their future information needs and I asked them to paint me kind of a portrait of the ideal provider and then asked me asked them to tell me how guide star fit or didn't fit with their sense of ideal okay so the the other aspect that can influence the validity of our data and potentially weaken the validity of our data in-depth interviews is interviewer bias and that is because the among other things the interviewer may bias the data by not maintaining objectivity during the interview by losing track of the conversation and not identifying inconsistencies when in other words when participants contradict each other in different points in time in the interview the interviewer needs to pick up on that and question that because those kinds of inconsistencies or contradictions are just chock full of useful information that's going to help the interviewer understand what's really going on with that participant and their physical appearance in face to face mode as I stayed here and I stayed in the last workshop the reflexive journal can be a wonderful way to identify interviewer bias and that's simply a diary that the interviewer can go back to by way of the audio recording make the video recording and ask themselves you know what do I think I know from this participant and and it is ideally good to do this like right after the interview to ask themselves what do I think I know what assumptions do I did I make how do I think I know what I think I know and and I think probably most importantly how did my personal values and beliefs impact what went on in that interview questions I asked and things of that nature so very important to kind of get at any possible bias that's going on with the interviewer now the other the other area is inconsistency again which I've mentioned a few times now already and what I mean by inconsistency is or as an example is maybe when the interviewer for whatever reason does not cover the guide fully across all interviews or the interviewer is going back to something I talked about earlier about maybe a key objective of this research is to understand reactions to a particular concept let's say a new onsite library service or a new functionality on your website or something of that nature it's very important that the the interviewer talks about that introduces that to the participant in a consistent way from interview to interviewer if the interviewer doesn't follow up on really key topic areas across all interviews that's inconsistent and potentially a problem and that last point you see on the slide where I say the interview guide is not scaled appropriately unlike this is unlike focus group discussions because it's not unusual at all for IDIs to be conducted both in person and on the phone and that's fine and as I've already told you I've done that a lot but what's important here is that the interviewer the researcher scale the guide appropriately because you are not going to be able to have you you have to go into the phone interview assuming you're not going to have the time that you had in the in-person interview and so you're going to have to think very carefully what you need to scale back and your priorities and things of that nature so to mitigate these potential problems with validity having to do with interviewer bias in consistency what do you think about interviewer skills and here are four interviewer skills in front of you and we're going to just touch now on the first three we're building rapport as I mentioned earlier the end of the interview method is one of the most personal if not the most personal form of qualitative research so building rapport and trusting relationship is really key to the success of an interview it's important because it's by way of that rapport that participants are willing and able to share their candid thoughts and I would suggest to you that that includes the participants comfort in saying to you the interviewer I don't know that is just as critical as anything else that the interviewer the participant be comfortable in saying to the interviewer you know I just don't know the answer to that question rather than making something up it moves you so this willingness moves you closer to your objectives which in the end provides you with quality data and useful outcomes building rapport begins before the interview so it means it begins when you contact the participant prior to the interview to discuss and communicate the purpose of the study the basic content that's going to be covered the length the incentive is if there is one and hopefully there is and participants questions but it's also important of course during the interview and that's what most people think of when they think of building rapport during the interview and one way is actively listening actively listening is is being generally interested with whatever the participant has to say as Carl Rogers said it's about unconditional positive regard which is full acceptance and support irrespective of what the participant says actively listening means means actively listening when it's verbal or textual listening if and we'll get into modes in a minute when when the mode is text based as well as nonverbal or quiet listening which again is true really across all modes so everything I'm talking about here is really not mode specific and really true of all modes that we can actively listen it also means that during the interview that you're picking up on participants cues they kind of go hand in hand with what I just said but here the focus is on the participants cues that the participant is giving you the interviewer in terms of any verbal or textual cues again consistencies inconsistencies particular words particular repetition of words as well as nonverbal or what I call quiet cues in terms of facial expressions eye contact things of that nature for the in-person mode you know just the seating and the seating distance from between you and the participant sometimes can be a little clue that that the interviewer needs to pick up on and potentially use to the advantage of building building rapport so the fourth kind of really be broad key interviewer skills is staying focused and this is not unlike focus group discussions where you consistently probe keeping in mind always what the the most important research objectives are to your in-depth interview research being able to identify and assess the relevance unanticipated products topics is probably I think one of the more important and one of the more area that interviewers need skill on I think sometimes that skill is maybe sometimes missing in training but this is very important that the interviewer is able the interviewer understands the objectives of the research so well that that the interviewer can assess what is being said in the interview as it relates to the objectives and whether or not this is something that they need to learn more about even though it's nowhere on the guide question wording should not vary and I've talked I've already mentioned that a few times and being able to manage time which is always a big consideration in qualitative research just managing time now here is an area a skill that is not is unique to in-depth interviews was not discussed in focus group in the focus method and that is because the and the quote you see there on the screen is taken from an article I wrote and put on my blog devoted to note taking in the in-depth interview method it's very important what and I say handwritten notes I'm not talking about using your laptop or any other such thing but handwritten notes as you are and I would say particularly in the in-person interview online and all it becomes more difficult but in person particularly for the in-person interview taking handwritten notes as you are talking to the participant the interviewer helps you the interview fully engage with that participant and helps you become very reflective of what is going on in that interview and in the end of course maintains this all important participant research or relationship that is again kind of the foundation of of the in-depth interview method I think I can't stress enough there are some people that that want to talk about this idea may not agree with it but I think once we have that discussion they usually kind of see what I'm getting at here that that handwritten note taking for in-person interviews is very very important and really transforms the interviewer I will go so far to say I think the interviewer comes out of that interview more knowledgeable about what actually went on in that interview and what was learned in that interview then then someone who has has not that's pretty big statement but I think that's true I do think that's true okay in addition to those core skills we have to have effective ways to ask questions again this was covered in the focus group workshop but let me just go through this pretty briefly again this something that does not vary across modes this is not a mode thing this is true of really all in-depth interviews the three kind of very kind of in the simplest form the three basic types of questions are the context the comparison contrast and clarification the examples you see here for each of these types came were adapted from a study I found in literature by a library researcher who was conducting who conducted in-depth interviews with dance faculty and and shared the guide and and what you see here are some again some adaptations of what I saw from the guide so context is there to help kind of ground us in and help us the interviewer probe effectively moving on in the interview because it gives us the context and the grounding that we need the comparison contrast is is really an important type of question because to not ask a comparison contrast type question questions would leave the interviewer with a misunderstanding that all issues are are equal so in this case I've used the very simple idea of how would you compare the information you gain from printed books from that with that from ebooks the clarification question is kind of central to why we conduct qualitative research in the first place right which is explanation again to not ask participants to clarify themselves or clarify something they've talked about is really missing the an opportunity and missing the whole point of conducting qualitative research the example I use here is you stated that you consult magazines and journal articles and electronic format but for printed books what makes one format better than another for certain sources of information so all of these are are very basic but important types of questions to ask now by the way we all ask the why question again this is something else I covered I covered in the focus group workshop but let me just repeat this briefly we all ask the why question I asked the why question but my whole point here is to strongly suggest that when you're sitting down to write the why question that you think very very carefully about that and and and maybe even think about maybe another way to ask instead of asking why ask how so in what way tell me about a time give me an example what are you thinking of you know things of that nature and the reason is is that what you see in front of you are just four reasons that I question the why question one is rationality in essence we're asking participants to explain themselves to and what I say I use the word justify which is more appropriate to justify themselves and in the process of that we're stifling the conversation in the process of that they're trying to say okay well you know how am I going to justify this to the interviewer which has effectively kind of stopped the flow of conversation it can also I propose to you that it also clouds meaning and I use the example here of you know the question why is the library important to your research I would suggest is a pretty difficult and confusing question you know that's like asking somebody why are you happy it's just a very confusing kind of kind of question but asking what are the specific aspects of the library that make it important to your research well that's something that's much more answerable I can get my head around that and and give you an answer to that I also think that there are many times when that why question is actually asking a question you hadn't intended to ask and I use the example here of why do you use Google scholar and I'm suggesting that that's a very different question than maybe what you really wanted to ask but you really want to ask is what are the benefits you derive from using Google scholar compared to other databases that's all so in addition to direct questions there are indirect questions and we can use enabling techniques to modify direct questions I cannot tell you how many times many many times I have been conducting interviews or group discussions and have asked a question and gotten blank stares back at me for whatever reason participant couldn't answer but as soon as I put a sentence put a sentence completion and I use the example the examples here again come from the dance faculty study I mentioned to you a minute ago I have found that the most effective way to manage the information I obtain is to and all of a sudden the participant can answer I don't understand all the psychology of that but it's I just know it's true because it's happened so many times there's word association and storytelling you know tell me a story about a time when that kind of thing and just like focus group discussions there are participant types in the IDI method and I list a few of them here there may be more these are the ones that I am all too familiar with I would just suggest to you and there's ways to we're not going to go into it today there are ways to deal with these situations I would also just mentioned that towards the bottom I have fast talker and I would just suggest to you that the fast talker is not necessarily the same person who wants to rush through the interview which is the last item on that on the screen and by the way the person who wants to rush through the interview is more often than not someone I'm interviewing on the phone and that's it goes back to why what why we adapt the guide for the phone when we've been conducting in person interviews that's one reason because we're invariably going to get somebody who says I got to go I got to go how many questions do you have that kind of thing okay I'm going to go on to modes what you see here is kind of a broad graphic of synchronous and asynchronous modes if nothing else all of this is to say to you what I've already said which is what that the IDI method really lends itself to a lot of different modes definitely more modes than we have in the folks group discussion method you'll see here by the way that I have mobile the mobile mode both in synchronous and asynchronous and indeed that's because we can conduct mobile let's say in the moment research um um you know synchronously when when they are actually reporting to us and we are talking to them real time about what it is they are experiencing but you could also do that almost in a in a diary fashion in an asynchronous mode so traditional modes in person and phone and you probably already know this but there are a number of strengths to both of these modes in person is obviously more in line with a natural conversation it facilitates building rapport you're able to recognize and react to cues and it can be a more complete and in-depth interview which goes back to what I just said a minute ago about the phone and someone on the phone wanting to rush through the interview the phone on the other hand is a good mode in in the sense that in terms of scope you can get a wider coverage don't have trips around the US doing in-depth interviews which I've done it can help in gaining cooperation because it can be more convenient for participants and it can mitigate bias if again the various visual cues are going to cause possible possible bias the limitations of these modes are kind of a flip-flop of each other the in-person limitation of in-person obviously is the scope cooperation and potential bias interviewer bias and participant effects the phone weakened rapport I've already alluded to that the absence of cues and the potentially less in-depth and shorter interview which I've also alluded to so what about the online modes I've I've broken out three here the online video mobile and the email now the email slash what it's called discussion board discussion boards are basically quality of research platforms that can be used for focus group discussions which I talked about last time or in-depth interviews so so each of these types of modes are particularly good for the various types of research that you see there those are just those are merely examples obviously there's there's more to it but but they are particularly good for obviously online video to substitute for in-person usability research to understand you know how someone uses something concurrent mixed methods in other words using combining survey and qualitative research when they're they are being conducting concurrently the mobile research in moment research a mobile mode for in the moment research which I'm going to talk more about in a minute and the email and discussion board mode which can be very good for sensitive topics for for again people with researchers who have been conducted IDIs with people with disabilities have used the email mode it's good for professionals and things of that nature so as just a kind of a blanket broad statement across all these online modes the the key strengths lie in the representation and cooperation you know coverage convenience they may be more relevant for and just more appropriate for certain types of the population and the data accuracy and the depth again because you can in these online modes it's not just audio it's not just visual it's not it's just a real combination of what you can gather in terms of the data which I'll show in a minute can also be a problem because because it's just too much data but it so it allows for a lot of detail and it can mitigate recall error recall error and the asynchronous mode now you'll notice at the bottom here I say it may be efficient use of resource let me just say it may it may not and I use the word may because you will need to determine for yourself whether or not you're saving money simply because there are costs involved in using the online platforms and depending on what you're doing the the costs may or may not be better than doing it in a more traditional way so you just need to be careful about that limitations again broad statement on the limitations of these online modes um and as we discussed last time in the last workshop there is cues that may be missing you know even in the video mode picking up cues can be difficult and certainly not like when you're in person so they may be missing analysis as I just alluded to there's lots and lots of data potentially that you need to take into account particularly in terms of your time and energy going when when it's when your interviewing is done you're going to need to give yourself a lot of a lot of space a lot of time in order to do the analysis fraud and security of course is always a consideration to keep in mind there always can be technical glitches of course and for text-based like such as mobile and the email mode can be a limitation I talked earlier um about the total quality framework and the and the credibility component of the tqf and the two aspects of credibility which is scope and data gathering what I'm providing here is simply a graphic to give you an idea of what I've already talked about which is the the green boxes here have depict strength of the quality of the data and the red boxes is a limitation or weakness and um you can you can see here that among other things the validity this is for online video that the validity can be weakened in terms of the ability to establish rapport and you will notice that that is true for the other online modes as well here's the mobile mode and again a weakened ability to establish rapport also notice that I have over on the right side of that screen under participant effects I I I mentioned in the red boxes something that I call I'm calling selection bias and altered behavior and what I mean by that is that and and you're going to see this in a minute but what I mean by that is that if we ask a participant for instance to conduct an interview with us in the moment that they're doing something so that we can see and talk to them as they are going through a process let's say we the interviewer researcher need to keep in mind that that there may be some selection bias going on here the not necessarily consciously but the participant may be selecting what to show or not to show us for instance um the participant may be altering behavior in some way again not necessarily consciously but it can happen and so the same kind of thing for the email discussion board mode again in terms of validity a weakened ability to establish rapport over on the participant effects side as I've mentioned earlier or alluded to anyway this idea of low text skills which can come in many different shapes and form so anyway that just kind of gives you a again a graphic a visual to to think about to look at and to think about as it relates to these online modes within the within the realm of the framework the credibility component of the framework there are many qualitative dedicated platforms these are platforms that are very are specific to conducting qualitative research here are just some of them they offer a variety of features and and things that the interviewer can do to help engage participants this is from 2020 research I did an EPA study this is just one example of a markup where the participant can go in and as you're talking to the participant markup something that you're showing this is also a markup tool this one's from Focus Vision they offer heat maps they offer observed chat this is from iTracks they offer a number of different features they also allow you to engage a participant with collages and mind maps and visual elicitation and things of that nature also what these online platforms do these qualitative research dedicated platforms do is they provide full text support automatic transcripts editable video machine translations they have I don't know most are all have panels that they will do the recruiting from so they're really multi-dimensional I guess is what I'm trying to my points I'm trying to make here is screenshots from Indemo which is another provider and specific to their mobile platform and here is an example of the in the moment research where they have in this case they have their they have as you can see they're asking participants to answer questions as they actually do something and again it helps kind of understand what the participant is thinking and doing in the moment that they are thinking or doing it so it's obviously mitigating recall so all so this and the last thing I'm going to to mention is the idea of mobile diaries that this what you're seeing here from Indemo is this is something that can be done this in the moment research can be something that is done an interview that is conducted synchronously real time or it can be asynchronous so I might ask the participant to actually engage in some kind of activity whatever it is over time and then report back to me in the in the form of a diary mobile diaries are something that is something that has kind of been spearheaded by marketing researchers they have kind of used the mobile diary approach to to get at what they call consumer journeys such as understanding for example the online to offline purchase experience it allows them to to get a participant's thoughts and feelings and behavior at the moment that they are happening and I'm just going out on a limb here now maybe this is something you folks already do but this may be something that may could be useful to library assessment again going on a limb I don't know if it is or not but it just got me thinking that it could be useful for instance to understand how users are using library resources and understanding kind of the before during and after experience the strengths of a mobile library mobile diary approach is the is the fact that you can participants you you're kind of you're gathering information from participants from multiple locations doing multiple activities over various time periods it can be a good way at getting at kind of these the secret life of participants in a way kind of these hidden aspects of participants that you might not get to otherwise and they might not talk freely about in just a single shot interview the limitation is limitations are really well down to what I've already mentioned is that it has the potential of disturbing disrupting their natural behavior and experiences and again it's the participant not you who is controlling what is and what is not shared that can jeopardize the quality so okay I'm done those the those are my slides here is my blog I've included some links here that relate to what I've talked about today the total quality framework and the in-depth interview method that you might want to take a look at and this is my I have some references here and this is my contact information so um do we have questions how did I do on time okay too close thank you Margaret this is Sue so colleagues if you have a question you can add it into the chat box then Margaret and I can find them there or raise your hand or just jump shout out because we aren't able to see everyone at the same time okay I'm reading a comment here Margaret that long comment was probably for me as you were talking about you know building rapport and you know other you know key factors of success it just really reminded me of what just once told me and it's like and part of that rapport and the trust that they built is they trusted enough to say no you're wrong yeah which helped him actually come up with you know a much better analysis yeah that's great that's terrific and you know what I thought you were going to you know there is something in quality of research called member checking when the researcher during analysis actually goes back to the participants and says you know this is what I think I'm learning or I've learned in the research what do you think and I have to tell you Steve I um I'm not a big fan of that only because in the work I do anyway I don't have the luxury of then modifying because it's it's not unusual at all for the participants to say no you know that's not what I meant that's not you know you got it all wrong and I typically don't have the luxury of going back and like re re doing the whole thing you know what I mean so um so you so a researcher and we'll get the into this when we do analysis but has to um I think take ownership of the fact to understand that that quality of data is co-created and the researcher and the participant are creating this data and that at some point then it becomes the researcher's job to make sense of that and do the analysis and all so anyway I think I got off your comment but thank you for that comment it really struck me. Margaret there is a question from Claire and at the end of the chat question about in-person um do you do you want to read it out loud or paraphrase it so the everyone hear of it? Yes and and Ava thank you for the question for the statement on the I know that's your favorite slide I know you like that slide um for in-person can you comment on having a clock in a place that both interview and if you oh this is interesting sometimes it's awkward without that element in place and recognized at the outset there's a concern for running late or losing track of time when they are in grossing conversation you know not everybody agrees with me on this but you know what I think of Claire when I when I read this and this is this has to with focus group discussions but it's the same point when I walk into a focus group facility it's not totally unusual to see a clock in the focus group room somewhere on the wall or whatever the very first thing I do it's really I put down my stuff and then the next thing I do is I cut I remove the clock and the reason is is I don't want and this is true of of uh in-depth in-depth interviews as well I don't want the participants to be focused on that I want the participant to be focused on me just as I am focused on the participant and and I and I I'm really do not want the participant to to to see this clock here um or to see something that or something on the table that is always on and showing the time uh because it just highlights the fact like oh okay gee we only have 10 more minutes of this interview or gee I I need to be somewhere else in five minutes you know and uh and it just interrupts the entire flow and and process of of the interview so I'm reading your your comment again sometimes it's awkward without the element in place and recognize the outset there's a concern for running late or losing track of time when they are engrossed in conversation yeah and I just want to this is clear I just want to say that um when I was doing some interviews the second person when it happened that's when I started saying oh and we have a clock here just um you know and saying it at the beginning because uh they as they were talking um otherwise they were saying do you know what time it is I hate to interrupt and they just stopped everything to tell me that they had a class because when they really got into answering things they totally had no concept of what time it was which is great I think that's great Margaret if I could follow up because I think Claire has for me has raised another another thought and as I was listening to the two of you chat and what as an as the interviewer what are your strategies that you use to make sure that you stay on time because if you've asked someone to you know will you give me an hour of your time or 90 minutes of your time to have a conversation or a discussion I want to honor that so what are the strategies you use to keep yourself on time and ensure that the colleague you're interviewing is able to leave at the appropriate and what at the time you promised yeah I do it so so this is going to sound contradictory um but so when I'm doing an in person interview I will actually um wear a real watch you know those old-fashioned things you put on your wrist and what I will do is as we are having conversation I will hopefully very discreetly look down at my watch and I will usually pull up so before we start the interview I will actually pull up my sleeves a little bit depending on what I'm wearing so that my watch is always exposed now that may sound contradictory but it's uh but I don't I don't think it is it's it's one thing to have this clock that's sitting on a wall or is mounted on the table next to you or something of that nature which is just screaming the time all the time which I think is very disruptive and and really is is missing the point of of what we need to be focusing on but when I when I'm wearing a watch it's just almost like even though these days it's like it's a little bit odd but there are people who wear these wristwatch and and it's almost like just a piece of jewelry or something and I as and I as I mentioned I'm always taking notes um I take lots of notes and I as I'm taking notes I'm just you know blah blah blah and my eyes are going over into my watch just to make sure that I know that I'm on track uh uh on on time and where I am in the guide and where I am with with the interview and to the point where I will say if um if we only have let's say 10 or 15 minutes left of the interview time and this is a particularly interesting informative participant and I know and we're not we're not as far along in the guys we should be probably maybe because this person is so informative um I will even just say I'll do like a little time out and I will say to the inner the participant I say listen you know I I just noticed that we have you know 10 minutes left during our time slot uh you have just been a wealth of information I've just really enjoyed talking to you could um do you think we could extend our time here a little bit for the interview and if not maybe maybe you could return or I could come back and interview a little bit more at another time so I'm I'm getting that permission up front so then we can now put that behind us we've had that conversation we've had that conversation about time we know what's going to happen we know that 10 minutes are going to go by and either this person is going to give me another 10 15 minutes 20 minutes whatever it is or I'm going to come back in another time and we're going to finish the interview we can put that away and now we can continue having our conversation so I don't know if that answered sue your comment or not did that it did thank you I I tend to wear a watch myself all the time so that was that was what I was thinking but I just wanted to hear what your thoughts were yeah so colleagues we I'm just going to say we have a couple minutes left if there's one last question anyone has and if not we will we can bring our session to a close but let me give you a chance to ask one last question if anyone has one please don't hesitate to contact me about any of this we're going to be doing this again on Thursday of course but if there's anything else you want to follow up on just let me know I think sue has provided you links to my calendar for office hours and we could talk that way or whatever just let me know thank you Margaret well I don't see any more questions so colleagues thank you again for joining us today if you know of some colleagues that weren't able to join today and we're maybe on the fence for Thursday session please encourage them to join it's the same login information or let me know and we can make sure everyone has the link as I noted at the beginning we will share the recordings and materials etc after Thursday's presentation so thank you and yes do reach out to Margaret with questions and any other further comments so with that Margaret I'll give you the last word if there's anything you wanted to add to to say goodbye just thank you all for for being here it's good to see all of your profile images and all and and again don't be don't be afraid to to reach out if need be happy to discuss any of this further if you like so thanks again all right thanks everyone take care bye